Local LLM Hardware Comparison for what will be yet another search plugin
Not every selfhoster has a few H200s lying around, but a lot of us have some kind of GPU dedicated to video transcoding. Mine is a humble Tesla P4, which I originally bought for the sole purpose of bringing hardware acceleration to the video transcoding plugin of Filestash for on demand HLS streaming. That project is now complete, and the card was sitting in its box waiting for its next job, which got me wondering: what else could we do with a GPU?
The obvious answer is a RAG pipeline to power things like semantic search, but over the last few weeks, I have been toying with another, simpler architecture, which consists in asking an LLM to turn a user query into SQL executed on a user owned index to power features like smart folders and semantic search. Would we need a frontier model for that? Probably not, and since I’m very much into selfhosting, the question I have been trying to answer is twofold:
- how little hardware can we get away with when running things locally?
- which LLM you can selfhost today could serve as a basis for fine tuning, with actual numbers on size, speed and accuracy, to power features like semantic search and smart folders?
So we benchmarked a whole bunch of small models on the kind of GPU that fits in a low profile server, the sort of card you can get your hands on for anywhere from $100 for the P4 to over $5k for the L4, with the T4 sitting in between at around $800 a pop, and threw in an RTX 3090 as a reference point since it is widely considered today’s value king.
Methodology
Everything runs through bare ollama without any tweaks or configuration shenanigans, using this script. The index is a prefilled sqlite database, each model are asked after a warmup run to make 2 queries to find how many docx files we have and showing files larger than 5MB.
Results
| Model | P4 | T4 | L4 | RTX 3090 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| test | time | tok/s | test | time | tok/s | test | time | tok/s | test | time | tok/s | |
| qwen3:0.6b | ✓ ✓ | 1.20s | 149 | ✓ ✗ | 0.57s | 244 | ✓ ✗ | 0.53s | 333 | ✓ ✗ | 0.41s | 434 |
| qwen3:1.7b | ✓ ✗ | 0.84s | 82 | ✓ ✗ | 0.39s | 145 | ✓ ✓ | 0.29s | 180 | ✓ ✓ | 0.22s | 325 |
| qwen3:4b | ✗ ✗ | 54.5s | 39 | ✗ ✗ | 2.60s | 68 | ✗ ✗ | 39.0s | 77 | ✗ ✗ | 12.3s | 174 |
| qwen3:8b | ✓ ✓ | 1.41s | 24 | ✓ ✓ | 0.65s | 43 | ✓ ✓ | 0.52s | 51 | ✓ ✓ | 0.29s | 131 |
| qwen3.5:0.8b | ✗ ✗ | 1.20s | 72 | ✗ ✗ | 0.91s | 104 | ✗ ✗ | 0.56s | 181 | ✗ ✗ | 0.62s | 254 |
| qwen3.5:2b | ✗ ✓ | 1.39s | 42 | ✓ ✗ | 0.86s | 72 | ✓ ✓ | 0.65s | 98 | ✓ ✗ | 0.40s | 185 |
| qwen3.5:9b | ✓ ✓ | 1.74s | 20 | ✓ ✓ | 1.32s | 34 | ✓ ✓ | 0.78s | 45 | ✓ ✓ | 0.55s | 107 |
| qwen3.6:27b | ✓ ✓ | 7.72s | 3 | ✓ ✓ | 4.58s | 5 | ✓ ✓ | 1.52s | 15 | ✓ ✓ | 0.96s | 41 |
| qwen3.6:35b | ✓ ✓ | 4.16s | 13 | ✓ ✓ | 1.67s | 25 | ✓ ✓ | 0.70s | 71 | ✓ ✓ | 0.64s | 83 |
| qwen2.5:0.5b | ✗ ✓ | 0.91s | 110 | ✓ ✓ | 0.44s | 147 | ✓ ✗ | 0.30s | 321 | ✗ ✓ | 0.25s | 387 |
| qwen2.5:1.5b | ✓ ✓ | 0.90s | 73 | ✓ ✓ | 0.48s | 99 | ✓ ✓ | 0.31s | 185 | ✓ ✓ | 0.26s | 279 |
| qwen2.5:3b | ✓ ✓ | 1.04s | 44 | ✓ ✓ | 0.51s | 80 | ✓ ✓ | 0.40s | 111 | ✓ ✓ | 0.28s | 208 |
| qwen2.5:7b | ✓ ✓ | 1.50s | 25 | ✓ ✓ | 0.73s | 44 | ✓ ✓ | 0.57s | 55 | ✓ ✓ | 0.32s | 137 |
| gemma4:12b | ✓ ✓ | 2.95s | 10 | ✓ ✓ | 1.49s | 25 | ✓ ✓ | 1.10s | 32 | ✓ ✓ | 0.72s | 75 |
| gemma4:e2b | ✓ ✓ | 1.45s | 42 | ✓ ✓ | 0.99s | 77 | ✓ ✓ | 0.70s | 115 | ✓ ✓ | 0.58s | 161 |
| gemma4:e4b | ✗ ✓ | 1.68s | 28 | ✓ ✓ | 1.13s | 49 | ✗ ✓ | 0.81s | 67 | ✓ ✓ | 0.62s | 123 |
| gemma3:270m | ✓ ✓ | 0.68s | 68 | — | — | — | — | — | — | — | — | — |
| gemma3:1b | ✗ ✗ | 1.50s | 60 | ✓ ✗ | 1.02s | 103 | ✗ ✗ | 0.66s | 178 | ✓ ✗ | 0.57s | 229 |
| gemma2:9b | ✓ ✓ | 1.64s | 19 | ✓ ✓ | 0.96s | 35 | ✓ ✓ | 0.78s | 42 | ✓ ✓ | 0.45s | 101 |
| gemma4:26b | ✓ ✓ | 3.15s | 14 | ✓ ✓ | 2.30s | 23 | ✓ ✓ | 0.87s | 72 | ✓ ✓ | 0.68s | 115 |
| deepseek-r1:1.5b | ✓ ✗ | 2.25s | 70 | ✓ ✗ | 1.64s | 117 | ✗ ✗ | 1.26s | 176 | ✗ ✗ | 0.77s | 285 |
| deepseek-r1:7b | ✓ ✓ | 4.94s | 24 | ✓ ✓ | 2.73s | 42 | ✓ ✓ | 2.88s | 52 | ✓ ✓ | 1.13s | 130 |
| deepseek-r1:8b | ✓ ✓ | 1.45s | 22 | ✓ ✓ | 2.17s | 39 | ✓ ✓ | 1.72s | 50 | ✓ ✓ | 1.26s | 124 |
| granite4.1:3b | ✓ ✓ | 0.88s | 45 | ✓ ✓ | 0.37s | 78 | ✓ ✓ | 0.29s | 101 | ✓ ✓ | 0.22s | 196 |
| granite4.1:8b | ✓ ✓ | 1.24s | 22 | ✓ ✓ | 1.30s | 38 | ✓ ✓ | 0.43s | 48 | ✓ ✓ | 0.24s | 119 |
| lfm2.5 | ✗ ✓ | 5.46s | 69 | ✗ ✓ | 1.59s | 117 | ✗ ✓ | 2.46s | 171 | ✗ ✗ | 1.07s | 305 |
| lfm2.5-thinking:1.2b | ✗ ✓ | 10.1s | 103 | ✗ ✗ | 8.25s | 188 | ✗ ✗ | 4.97s | 259 | ✗ ✗ | 1.18s | 459 |
| llama3.1:8b | ✓ ✓ | 3.13s | 23 | ✓ ✓ | 0.74s | 43 | ✓ ✓ | 0.60s | 52 | ✓ ✓ | 0.35s | 135 |
| mistral | ✓ ✓ | 1.82s | 26 | ✓ ✓ | 0.67s | 47 | ✓ ✓ | 0.69s | 56 | ✓ ✓ | 0.29s | 147 |
| tinyllama:1.1b | ✗ ✗ | 1.04s | 129 | ✗ ✗ | 0.64s | 233 | ✗ ✗ | 0.36s | 311 | ✗ ✗ | 0.28s | 495 |
| devstral-small-2 | ✓ ✓ | 5.84s | 5 | ✓ ✓ | 2.72s | 10 | ✓ ✓ | 1.41s | 18 | ✓ ✓ | 0.65s | 51 |
Takeaways
I will admit, before actually running this experiment, I had a bias that bigger means smarter and slower, but that is not what the numbers show. qwen3:4b is absolutely useless while its smaller siblings qwen3:1.7b and qwen3:0.6b show surprising results that make them prime candidates for fine tuning. I would also have expected deepseek-r1:1.5b to perform a lot better than gemma3:270m, a model so small it was not even part of the initial benchmark because I had the wrong preconception that there was no way a 270m model could do anything useful. Another surprise is IBM’s granite4.1:3b, which, despite its relatively heavy size, answered accurately in under 1 second even on the cheapest Tesla P4.
In the coming sprints, we will be experimenting with fine tuning those models for our workload, opening up yet another path for those like me who are curious whether an LLM can make finding your data easier.