Public leaderboards tell you how a model does on someone else's test. At SOFTCON 2026, I shared how we choose models at ChatGenie instead: with evals built on our own customer conversations, scored against what our enterprise clients actually need. Here is the talk in short, with the slides.

‍

Why leaderboards weren't enough

ChatGenie designs, builds and deploys agentic AI systems for enterprises, and most of our deployments automate chat support. A customer writes in through email or a messaging app. Our system answers from approved knowledge, calls the client's tools through MCP, and hands harder cases to the client's team through their existing ticketing system.

It looks simple: pass the customer's message to a model and send back the reply. But once every reply must stay grounded in the client's knowledge base, and an attached screenshot can trigger its own workflow, the reply is only the last step of a longer chain. We needed a signal that the whole chain works. That signal is evals.

If you already write automated tests, you already know most of what evals are. They add to your test suite rather than replace it. They cover the part of an AI system that doesn't give the same answer twice: did the classifier pick the right label, did the agent pass the right argument to the tool, did the new prompt break something that used to pass?

You don't need to be a frontier AI lab to do this. Product companies such as Descript, Bolt and Sierra run evaluation suites built around their own workflows, and we started ours early in our product's life.

‍

The migration that forced the question

Our evals paid off when we moved from Microsoft Azure to AWS. Leaving Azure meant losing Azure Foundry, where our inference ran. So we had one urgent question: which model on AWS Bedrock could replace our GPT-5.2 baseline and still follow our enterprise workflows?

There are three practical ways to evaluate a model, and they complement each other. Public benchmarks measure general reasoning, coding and knowledge. Manual probes, like Matt Berman's 3D Rubik's Cube test, make a model's capability visible but aren't repeatable. Workload-specific evals run repeatable tests against real tasks and business rules.

We used public benchmarks only to build a shortlist from the many models on Bedrock. Because we already had evals for our own workflow, we skipped manual testing and went straight to them.

‍

How we ran the benchmark

The first comparison put five models, including the GPT-5.2 baseline, through 102 general scenarios and 100 Tagalog and Cebuano scenarios, scored on four weighted dimensions. Because we have live deployments, the test cases came from real customer messages and real business rules. Synthetic cases still have a place: they're privacy-safe and fast for early proofs of concept.

Grading is hybrid, with humans in the loop:

1. Run the same cases through each candidate.
2. Check deterministic assertions wherever possible.
3. Let a frontier LLM judge grade the open-ended replies.
4. Have humans audit the judge's passes and fails.
5. Revise the prompt and rerun the regressions.

The judge asks four questions of every reply. Did it answer the question? Did it promise an action it had no tool for? Did it invent a fact, price, contact or policy? Did it include the required information?

It caught real problems:

• A reply said "Para ma-check ko…" ("So I can check…"), implying a live status check with no API or MCP call behind it.
• A reply said "Walang dagdag na bayad" ("No extra charge") for an add-on the knowledge base priced at ₱49.
• A reply said it didn't have the information when a related FAQ already answered the question. Missing a known fact is an error too.

The process isn't fully automated. Models are trained and rewarded differently, so our engineers retrofit the prompts for each model, audit the results and iterate without overfitting. That effort is why we kept the shortlist small.

Two scope notes. This benchmark is fitted to our workflow, so it isn't an industry standard. It also left out tone and brand voice, and AI security testing, which deserves its own talk.

‍

Audit the judge, and don't trust one run

An LLM judge needs its own evals. Our first multilingual run scored GPT-5.2 at 64% on language match. The model hadn't suddenly gotten worse; the judge was applying the wrong language labels. Once we recalibrated it, GPT-5.2 averaged 92.29%, and the best Bedrock model reached 100% across two runs.

Human review is where we usually find errors like this. A model that scores over 95% on the general eval is very unlikely to score 64% on the multilingual one, so the anomaly pointed us at the grader, not the model.

A single run misleads too. On 102 cases, one verdict moves the score by about a point. Sonnet scored 93.14% on one prompt iteration and 95.10% on another. In August, six Qwen runs on the same day ranged from 92.2% to 98.0%. One run is a sample, not a score, so freeze the configuration and rerun the same cases.

The same caution applies to prompt work. Our GPT-5.2 baseline went from 93% to 99% on the development set over two prompt fixes. That's good engineering evidence on familiar cases, not proof that it generalizes.

‍

What the May benchmark found

Before showing any scores, I ran a blind poll. Three anonymized candidates from the same benchmark: A scored 95.10% at 23–24 seconds and $0.028 per interaction. B scored 93.14% at 12–13 seconds and $0.0011. C scored 88.24% at about 15 seconds and $0.011.

A was Claude Sonnet 4.5, B was Qwen 3 Next and C was Claude Haiku 4.5. There's no winner without a deployment objective. If you run hundreds of thousands of workflows a day and need low latency with decent quality, B is the clear answer. If your volume is low and cost isn't a concern, A is.

The full results show that each model has its own ceiling. GPT-5.2 reached 99% with prompt work; Amazon Nova Pro couldn't get there through prompting at all.

To rank the models, we weighted instruction adherence at 45%, multilingual performance at 30%, latency at 15% and inference cost at 10%. Accuracy carried the most weight because it was the point of the exercise. Latency and cost still mattered a lot at our volume.

Qwen 3 Next 80B ranked first at 94.9, ahead of our GPT-5.2 reference at 93.1. Claude Sonnet 4.5 came second at 90.3, and Amazon Nova Pro edged out Claude Haiku 4.5 for third on cost.

A weighted score screens candidates; it doesn't make the decision. Qwen led on value, yet we still recommended Sonnet for accuracy-critical use cases. Block critical failures whatever the total score, and verify outcomes before you ship.

‍

Price the decision per resolution

At production volume, unit economics decide whether the business works. If your inference stack (embedding models, LLMs and everything around them) costs more than a reply earns, you have a big problem. Before we moved off GPT-based models, we were losing ₱1.20 on every chat response.

But cheap per token isn't the same as cheap per resolved issue. A weaker model adds follow-up turns, retries and escalations to human specialists. The number that matters is the cost per successfully resolved issue.

In customer support, a healthy deflection rate is above 60%: most tickets should be resolved by the agentic workflow without a human. So choose the cheapest model that is capable enough, not simply the cheapest.

‍

October update: frontier vs open-weight models

We've since moved our inference to Fireworks so we can use the latest open-weight models, and in October we tested three new candidates against them, including two frontier models. Claude Fable 5.1 scored 96.9% and GPT-6 Astra 96.1%. But they would cost about 39 and 23 times more than Qwen3.7-Plus, our production model, for accuracy that sits inside our run-to-run noise.

Between the open-weight models, DeepSeek V4 Flash would cost about half as much as Qwen3.7-Plus. Its average accuracy was about two points higher, but that's within our noise, so I call accuracy a tie. Two caveats: the two groups were graded by different judges and prompt versions, and GPT-6 Astra graded its own replies.

For the leaderboard, we kept our latency and cost weights and moved multilingual's 30% to accuracy, since we didn't measure multilingual in October. Open-weight models lead on value. Qwen3.7-Plus scored 94.9 and DeepSeek V4 Flash 94.7, effectively a tie. GPT-6 Astra (86.1), Claude Fable 5.1 (84.0) and Muse Spark Open (83.3) trail on latency and cost.

‍

Beyond chat replies: evals for the agents behind them

In October we extended evals from the final reply to the agents that run before it. From first run to best run, Guard went from 86.7% to 96%, sentiment from 87% to 89%, escalation from 93.5% to 94%, image analysis from 59% to 95% and message categorization from 51% to 65%. Each agent has its own test, so the numbers don't compare across agents, and this work is still in progress.

The image analyzer taught the clearest lesson. One day of eval-driven fixes on 100 images (78 real customer images, 22 synthetic) took it from 59% to 95%. The biggest jump, to 82%, came from adding business context to the prompt and fixing the gold labels. A dip to 77% turned out to be the judge grading too literally. The last ten points came from defining "correct": a pass means the image was read correctly, and the judge grades meaning, not wording.

Categorization taught a different lesson. Some client categories overlapped in meaning, and no prompt can reliably separate two categories that mean the same thing. We asked our clients to revisit their categories so we can run that eval properly.

We tested the judges too, on 131 cases that included 29 deliberately bad replies. Claude Fable 5.1, GPT-6 Astra and GPT-5.2 caught every bad reply. Jev 1.13 graded all 131 cases for $0.04, about 35 times cheaper than GPT-5.2, but caught only 92.6% of the bad replies. These are best runs on one benchmark, not proof of zero misses in production. Our conclusion: reasoning models make better open-ended judges, and fast, cheap models fit narrower tasks like sentiment analysis and categorization.

‍

The leaderboard is only the starting point

Our evaluation program isn't finished, and we keep tightening the evidence behind every score. Next, we're separating tuning sets from held-out test sets, freezing configurations and reporting the mean and range of repeated runs, and checking the actual task state instead of the final text. We're also calibrating judges with human review and fixed weight anchors, and turning every production incident into a regression test.

The message I repeated throughout the talk: a public leaderboard is where model selection starts, not where it ends. Evaluate your own workload. Audit your judge. Verify the outcome. Then choose the model that works for your users and your economics.

Thank you to everyone who joined the session at SOFTCON 2026.

‍

Further reading

• The Agentic AI Evaluation Playbook, ChatGenie: our full write-up of the May comparison
• Demystifying evals for AI agents, Anthropic Engineering
• Evaluation best practices, OpenAI

‍

Download the slides

The full 40-slide deck from the talk is free, including the slides this article doesn't show. Fill out the short form and we'll email you the download link.

Download Form

‍