The $3 Model That Beat Production

TLDR; this is both a “how to” guide to get started with fine-tuning as well as a detailed explanation on how I spent $3 to fine tune a small open source model that beat a production system using more expensive models. I walk through each step in the process and share my results and learnings. The takeaway is that anyone can post-train their own models and this is the first step towards sovereign intelligence.

The setup

I run hermesatlas.com, a community resource covering everything related to Hermes Agent by Nous Research. The site catalogs 200+ community projects, contains comprehensive guides, docs, newsletter, and a bunch of other resources.

The site has a feature called Ask the Atlas, a chatbot that answers questions about anything related to Hermes Agent. Ask it whatever you want and it uses a RAG pipeline to retrieve chunks from a large curated knowledge base, then it feeds those chunks to a model on Openrouter and gives you back a cited answer.

I run the site out of my own pocket and typically pay $10-20 a month in inference costs via OpenRouter depending on the waterfall of models that I have setup.

The question that kicked off this post-training exercise: can a small open model, post-trained on my own data (the curated knowledge base), beat the more expensive model setup?

This is a pretty simple use case that applies to anyone with a knowledge base that isn’t baked into the training set of any base models - if you want to ask a model questions about your data, you either setup a RAG system (like I did) or you post-train a model on your data, or for best results do both.

I learn by doing, so I ran the entire post-training loop myself with my Hermes Agent as my copilot handling evals, data generation, and judging while I made the calls and ran the GPU jobs.

Total cost for everything below, including a second training run on a managed platform to double check the results: under $10, less than one month’s worth of production inference cost.

The post-training loop

The hard work is in steps 1-3 and where you need to put in the most thinking, but once you set up your evals, baselines, and training data, the rest of the steps are pretty easy.

Also, that last step “keep measuring” wasn’t in my original plan but it produced a real interesting finding that I describe later.

Step 1: evals first!

If you take away anything from this article, let it be this: always start with evals! If you don’t structure evals correctly up front you’ll just be shooting in the dark as you go.

Evals are what help you decide if the change you are testing is actually better (or worse) than your baseline. Screw this up and your output is compromised.

So before touching a GPU we developed a 100 question eval set. 50 about the community catalog (content that is covered on the hermesatlas.com site), and 50 about the official docs from Nous Research that the site ingests as part of its knowledge base.

Why 100 questions? It’s a nice round number, but also big enough to differentiate among the results. This part is subjective, you decide what size of a test is good enough.

Each set of 50 questions has facts, a difficulty tag, and a category, including trap categories: false premises, projects that don’t exist, two real projects sharing one name, etc. Gotta throw some tricky ones in there.

Grading is rubric with five dimensions (accuracy, grounding, completeness, scope, style), 0-2 points on each, 10 points per answer. The judge model comes from a different family than the model being judged, and repeat judging runs helped us measure the “noise floor” at about 0.4 points, so anything closer than that is statistically a tie.

Important note: the eval questions never enter training data otherwise you’ve essentially just benchmark-maxxed your own model. Also, for hygiene they should never go in a public repo. Public text gets scraped into future model training runs, and then your eval becomes worthless.

Here’s the learning that surprised me most: the eval paid for itself before any training happened. Grading my own production system found 5 outright hallucinations in 100 answers, some inaccurate data from a stale index, and answers truncated by a forgotten token cap.

This is what makes evals so important - knowing what “good” is and being able to measure it helps you uncover issues you can fix before you even need to touch model setups or get into post-training.

On our first production run, the setup scored 8.16 (out of 10) and that became the number to beat.

Step 2: baseline everything

We then ran the same 100 question eval set through every configuration that could teach us something. Same questions, same judges, same rubric.

Models only, no RAG:

Claude Opus 4.8: 8.09
Claude Sonnet 5: 7.90
gemma-4-31b (my production backup model): 5.90
deepseek-v4-flash: 5.76
Qwen3.5-9B (the base I’d later train): 5.47
gemini-3-flash (my production model): 4.97

Three findings from this baseline:

The RAG uplift on my stack is +3.19 points (4.97 with just gemini-3-flash vs 8.16 in production). The RAG retrieval pipeline is doing almost all of the work.
None of these models actually know my knowledge base. The 3-point gap between Opus and Gemini Flash is not knowledge, it’s calibration. The frontier models produced zero hallucinations in 200 answers, they just said “I don’t know” gracefully. The cheap models hallucinated on answers, so the more expensive models buy calibration, not facts.
My production system first tries to run the free tier of gemma-4-31b, and if that isn’t available, the fallback is gemini-3-flash. In our eval run, the free tier failed 91 of 100 direct API calls, so almost every production answer was falling back to the paid Gemini Flash which turned out to be the worst base model I tested. Remember this fact as it comes back later in the story.

Frontier models, with RAG:

We also ran the frontier models WITH my retrieval to find the ceiling:

Opus+RAG scored 9.24
Sonnet+RAG 9.20.

Two takeaways here. The ceiling is not 10, because no model can answer from context that the retrieval system can’t surface. And Sonnet basically tied Opus, so for grounded Q&A the expensive model buys you almost nothing which is a good thing to know and highlights the importance of running evals to make sure you’re not paying for an expensive model when something cheaper will give you the same result.

Step 3: manufacture the training data

After running the baseline it was easy to conclude that the goal of training was not to teach the model facts. Facts can go stale, the hermesatlas.com catalog changes weekly, and retrieval already works well for this type of information.

The goal of training was to teach behavior: answer richly when the context supports it and say “the Atlas doesn’t have that” when it doesn’t. Basically don’t hallucinate which is exactly what we found the production models doing on the evals.

The eval set had 100 questions, but for proper training we need more data, so we built a dataset with 1,786 examples across three categories - grounded answers, partial answers, and honest abstentions.

The 1,786 questions came from agents reading the actual knowledge base, and 60% were deliberately aimed at information where we found the production models hallucinating the most.

Answers came from a teacher model (Opus) under strict rules: ground every claim in the record’s context, cite sources, and when the context doesn’t contain the answer, write the actual miss. The final mix: 59% grounded answers, 13% partial, 27% honest abstentions. That 27% is the key to the behavior we want to train.

One lesson learned. To build the training data set, we used 40 subagents and found that one drifted and started answering from general knowledge instead of the knowledge base. A review pass caught it but just remember that synthetic data pipelines can fail so you need to budget for QC. Just another example of why I say the work mostly happens in steps 1-3.

Step 4: run the training

I did two training runs, the first on a rented GPU and the second using a hosted fine-tuning platform. I did this for two reasons: first is that I wanted to try both methods and have the experience, and second was to double check the results to make sure they both aligned.

The model I chose to train was Qwen3.5 9B. Why? It’s small, cheap, can be run locally on my hardware, and I just wanted to see how something in that range would compete.

Run 1: a rented RTX A6000 on Vast.ai using Unsloth and QLoRA, a common approach to fine-tuning LLMs.

Setting up the run was pretty easy. Select the GPU instance on the Vast UI, add your SSH key on the instance, remote in and install Unsloth, then upload your training files. Total setup time was about 15 minutes. Don’t let this part intimidate you, any LLM can walk you through it.

It did however take three attempts to get a successful training run working. The first two failed runs were on my side (user error, good learnings). If you’re doing this for the first time, expect to run into some issues and keep your patience. Both issues were config tweaks that me and my agent figured out by reviewing the logs.

Third time was a charm. During training you watch two numbers, and both are error scores, so lower is better. One is how the model does on a test set of questions we held back (material it never gets to study) so we can track its learning progress. The other is how it does on the material it’s being trained on, so we cant track how well it’s learning that specific material.

The first pass through the data delivered a 9.3% improvement on the held-back questions. The second pass delivered 1%. Meanwhile the score on the study material kept dropping another 31% indicating the model was crushing it. The training was working.

Total cost including both failed starts: $2.60!

Step 5: the result

Now with my post-trained Qwen3.5 9B, it was time to run the eval using that same 100 questions, same judges, and same rubric from step 2.

Result: my Qwen3.5 9B model scored 8.58 against the production’s baseline of 8.16!

Pretty impressive. We took a small open weight model, ran it through ~$3 worth of post-training and it came out scoring better than my production setup using gemini-3-flash! It also scored well above what Opus/Sonnet scored (without RAG) and it held up relatively well against Opus/Sonnet (with RAG).

The best part is that every catastrophic production failure we found during the baseline runs flipped to either a correct answer or an honest miss - no hallucinations.

On one question, production had fabricated a description for a project by assigning a real feature from a different document onto the wrong name. Our 9B model untangled it, flagged the project as not in its context, and correctly attributed the feature to its actual source.

Step 6: replicate the results, then isolate the results

As I mentioned above, I ran the training twice using different setups to make sure the results were consistent.

Run 2: identical dataset, identical base model, but using a managed fine-tuning API from Together AI.

A couple quick observations on the rented GPU on Vast vs the hosted option on Together:

The setup and time-to-training on Together was much faster and easier. Together makes it quite easy to run everything via CLI with an API key so no need to setup SSH keys and install tools.
Together was over 2x more expensive at $5.60 vs. $2.60 on the rented GPU. Not a big deal for what we did, but something to consider for much larger runs.

Run 2 scored 8.62 vs. Run 1 score of 8.58, so basically the same result which made me feel good about the integrity of the training results.

The training platforms don’t matter, it’s really just a preference of cost vs convenience. The dataset is what matters and that’s the work we put into steps 1-3.

But we’re not done yet - time to run some isolation tests.

We ran the untrained Qwen3.5 9B base model through the exact same RAG pipeline that my production system uses and it scored 8.07.

Now we have a full picture of how the improvements stack up:

Base model alone scores a dismal 5.47, and when you score the trained model it improves to 6.39 - not bad, but still not production grade. If you take just the base model with retrieval it jumps to 8.07, but when you take the post-trained model with retrieval we get an impressive 8.58.

Retrieval buys 2.6 points. Training buys 0.51 which is more than just noise, so it’s real.

But if we look closer at where that half point lives, the untrained model with my retrieval still hallucinated in the evals which is something that most production systems cannot afford. What looks like a modest incremental training bump is what stops the hallucination so now we have something that can actually be trusted at the edges where my production system breaks.

Another big finding: an untrained open 9B model with my RAG retrieval statistically ties my production system using a much larger and more expensive model. The moat is the custom retrieval pipeline with a post-trained model on my data - this is sovereign intelligence!

Step 7: keep measuring because things change

Here’s the part I didn’t plan. A few days after the first production measurement, we re-ran the identical production configuration offline as a control: same prompt, same contexts, same token cap, same Gemini Flash. July 17 it scored 8.16 but on July 21 the identical setup scored 9.14.

A full point of drift in four days, with zero action on my side. We re-judged the archived July 17 answers with fresh judges to rule out grader drift: the old answers scored the same. Conclusion: Google updated gemini-3-flash-preview and the fabrication cluster that defined my baseline largely vanished. It’s a preview model so these things can change.

So that means my production system should have jumped a full point, right? We measured it live again on July 22: 8.28. Statistically identical to the original baseline of 8.16.

Here’s the difference, on July 22: 91 of 100 answers came from the free-tier gemma-4-31b model and the other 9 from gemini-3-flash. Remember the free tier that was failing 91% of calls during the original baseline? It recovered and now it worked on 91% of requests. The paid model got a silent upgrade but the free tier model negated those gains.

The point here is that production models change and that means whatever production systems you have that rely on those models can also change (for better or worse) without you noticing unless you’re running constant evals. The one mitigation - owning your intelligence.

My fine-tuned model scored 8.58 and 8.62 and it will score that next month too, because the weights are mine and they cannot move unless I move them. That’s the real argument for owned weights.

The final scoreboard

Here’s where the work that went into the evals pays off. I was able to run the eval across a full range of models, including my post-trained model, and production setups (with RAG, without RAG) to see how they all perform.

As you may expect, Opus/Sonnet with RAG comes out on top, but if I made those the default models in my production system, my monthly inference cost could easily go from $10-20 to $200+ so that’s not a practical option.

But what this exercise proved is that with a small open model (Qwen-3.5 9B) and $3 of post-training, I can achieve close to parity with those models and exceed the existing production setup that is far more expensive.

And on top of the cost savings, I now own this model - it won’t change underneath me and I’ll never get locked out of it.

I have a cheaper solution and I actually own it. That’s the power of sovereign intelligence and this is the playbook that anyone can use to do it.

4 simple takeaways

Build the evals first - everything else follows from it. Even if you don’t plan on post-training anything, build evals and test your current setup with various models and configs so you can measure the results and quantify if you’re paying extra for intelligence that doesn’t make a difference.
Don’t be intimidated to post-train models. It may sound technically complex but the process itself was fairly easy. The hardest part is just putting thought into the evals and training data.
The models are commodities now for many use cases. This project proved it three ways: my retrieval system equalized frontier and cheap models, two different training platforms produced the same model from the same dataset, and an untrained 9B tied my production system.
Not your data, not your flywheel. The evals, the data pipelines, and the system you assemble around them are your moat. This is the IP that you need to build on and protect.

I hope this article was helpful, if you ever want to jam on topics like this my DMs are open.