The first question most teams ask when they start building with AI is which model to use. It is a reasonable question with a short shelf life. The leading models trade places every few months, prices fall steadily, and the model that is best for your task today will probably not be the one you run a year from now.
After deploying agents across education, government and enterprise, our view is that the choice of model is the smallest of the decisions that determine whether an agent works. These are the ones that matter more.
1. Build the evaluation before you choose the model
You cannot pick the right model for a task you have not measured. Before comparing providers, collect a set of real examples from the job itself — a hundred past support tickets with the correct resolutions, fifty invoices with the correct extracted fields, a batch of grievances with the department each should have gone to.
That set becomes your test. Every candidate model runs against it, and the scores are compared on your data rather than on a public benchmark written for someone else's problem. The same set is re-run every time a prompt changes or a new model is released, which turns "should we upgrade?" from a debate into a measurement.
We apply the same idea inside a product. In AI-assisted marking of descriptive answers, an examiner first marks a sample of 20–25% of responses by hand. Those human judgements, alongside the model answers, are what the AI marks the rest against — and what its marking is checked against.
A team with a good evaluation set can change models in an afternoon. A team without one cannot safely change a prompt.
2. Decide where the data is allowed to go
For many of our customers — universities handling student records, government bodies handling citizen grievances, enterprises under India's DPDP Act — the constraint that narrows the field is not quality. It is where the data may be processed and who may retain it.
That decision sorts the options into three groups: hosted models used through an enterprise agreement with no training on your data, the same models served from a cloud region you choose, and open-weight models run on infrastructure you control. Settle this first. It eliminates more options than any benchmark will.
For our document verification platform, that decision is written down: primary hosting in Azure Central India (Pune), disaster recovery in Azure South India (Chennai), the database in the same region, and all data stored and processed within India under the DPDP Act, with a recovery-point objective of 15 minutes and a recovery-time objective of two hours. Every model choice has to fit inside that boundary, not the other way round.
3. Measure cost per task, not cost per token
Token prices are the number vendors publish, and they are the wrong number to budget with. What matters is the cost of completing one unit of work: one ticket resolved, one document verified, one report compiled.
A cheaper model that needs three attempts and a longer prompt can cost more per task than a more expensive one that succeeds first time. A larger model that handles a task in one call can be cheaper than a chain of small calls. Only the evaluation set, run with real prompts, tells you which.
4. Use more than one model
Very few production agents should run on a single model. Most jobs split naturally into steps with different demands:
- Routing and classification — small, fast, inexpensive models do this well
- Extraction from documents — mid-sized models with strong structured output
- Reasoning, planning and drafting for a customer — the most capable model you can justify
Routing each step to the cheapest model that passes its part of the evaluation usually cuts cost substantially without any loss in quality the user can see.
A second pass is often worth more than a bigger model. In our question-paper platform, every batch of generated questions goes through a separate AI review for ambiguity, leaked answers and duplicates before a faculty member sees it — and nothing enters the question bank without a human accepting it.
5. Keep the model behind your own interface
The practical consequence of everything above is architectural. The agent should never be written against one provider's SDK in a way that makes changing providers a rewrite. Put the model behind a thin interface of your own, keep prompts in version control alongside the evaluation set, and log every call with the model and version that produced it.
We built our question-paper platform this way. The LLM provider is a platform-level switch between OpenAI and Anthropic's Claude, usage is metered in credits per institution, and every call is logged with its tokens, cost, duration and payload. Changing provider is a setting, not a rewrite.
That is what lets you take advantage of the next price drop or the next better model — and what lets you move a workload in-house if a regulator or a customer contract requires it.
The short version
Choose the evaluation first, the data boundary second, and the model last. Treat the model as a replaceable component, because it is one. The durable asset in an AI system is not the model you picked. It is the evaluation set, the guardrails and the operational knowledge of what your agent gets wrong — and those belong to you whichever model you run.