Which local model is worth running on one 32 GB GPU?
Seven language models, five public benchmarks, one Intel Arc Pro B70.
The project
The models here are not being tested in the abstract. They serve a Hermes agent: a personal assistant that reads, plans, calls tools and hands parts of a job to sub-agents it starts itself.
That sets three requirements at once. Several requests must run concurrently, so a parent agent and its sub-agents are not queued behind each other. Each must be fast enough to hold a conversation. And accuracy, tool calling above all, must survive whatever was traded away for the first two.
The standing set-up is one language model in three parallel slots sharing 128,000 tokens (a parent and its sub-agents), a second, faster language model that reads pictures, and an image model, all on the one card. The benchmark, which measured the first of these in four 64,000-token slots, asks whether a different model would do that job better.
The machine
- System
- Dell Precision T5820 workstation
- GPU
- Intel Arc Pro B70, 32 GB, used for models only: no monitor is connected to it
- Display card
- NVIDIA GeForce GT 1030, a small single-slot card that drives the display so the B70 carries none of that load
- CPU
- Intel Core i9-9940X, 14 cores / 28 threads, 3.3 GHz
- Memory
- 64 GB RAM
- Storage
- NVMe and SATA solid-state drives
- Software
- Linux; llama.cpp (SYCL) behind llama-swap; ComfyUI on Intel PyTorch for images
- Agent
- Hermes, on a separate small machine, calling this one over the network
Energy used and what it costs
Only the GPU is metered: the card reports its own power draw every 30 seconds. The processor, memory, drives, fans and power-supply losses are not measured, so the whole machine uses more than this. Energy per answer is that model's average GPU draw while it was being benchmarked, multiplied by its average seconds per answer.
Accuracy against time per answer
Thinking on, every scored answer pooled. Up is more accurate, left is faster. The vertical line through each point is its 95% interval: where intervals overlap, the models are not yet separated.
Pooled accuracy weights each benchmark by how many questions it has (tool calling has the most), so it is a summary, not a ranking. The Results page splits it by benchmark.
What has been learned so far
Written up by hand as the work goes. The full list is on the Findings page.
Latest changes, logged automatically
Computed from the latest numbers
Progress
Share of the planned thinking-on answers scored for each model. Three rounds were run (six were planned at first; see Findings); a round is 120 questions per model.
Every night: the standing model's research shift
Through the day an equity-research app collects new company filings, economic data and news. From 10 pm to 6 am it has the standing model, Bonsai 2 + MTP head, read each new item and classify it, one item at a time. This table tracks how long that backlog takes to clear and is brought up to date each morning.
Started late: minutes after 10 pm before the first item ran; the app waits whenever another job holds the GPU. Accepted: the model's answer passed the app's checks (for example, a quote it gives as evidence must appear word for word in the source). Later in the night: new items that arrived after the backlog was cleared, handled as they came in. Model time is how long at least one model call was running. From 14 October the median time per item fell from about 48 seconds to about 19, and on the next three nights the backlog cleared in about an hour and a half. On 17 October the app was widened to read far more filings each day and to give items rated urgent a second check: that night's backlog was 660 items, 200 of them second checks, about three and a half times the night before, and it took 4 h 13 min to clear at a median of 18 seconds per item. The size of the backlog follows what came in during the day: the next night it was 232 items and cleared in 1 h 36 min. On 19 October the app began sending two items to the model at once and added a new kind of item, about 600 of that night's 870. The backlog was not cleared by 6 am; the model was busy for 7 h 32 min of the eight hours. Per-item times from that night on are not comparable with earlier ones: with two items sharing the model, the kind of item handled every night took a median of 49 seconds, against 21 the night before. The app is still being developed, so the nights are not like for like.
Other runs on the same model
An agreement check runs a fixed set of 60 items through the app and compares the model's ratings with ratings made by hand, before a change to the app goes live. These run in the day, at most one at a time.
The models
.
Several of these are community fine-tunes with refusals removed; they were chosen for an agent that must follow instructions without declining routine work, and are named here as their authors named them. Licences are as stated on each repository.