Skip to content

LLM fine-tuning and evaluation: smaller models that are right more often

By Vishal · published 25 October 2026

Fine-tuning adapts an open-weight language model to your domain, tone and tasks, so a smaller, cheaper model performs like a large one on your work. VAI Five builds the evaluation set first, fine-tunes with parameter-efficient methods on your data, and delivers the model, the evals and the serving setup you can run yourself.

When fine-tuning beats prompting

Most teams should start with a good prompt and retrieval. Fine-tune when one of these is true:

  • The task is repetitive and high-volume — classification, extraction, structured output, summarisation in a fixed format — and API costs are a line item you can see.
  • The model must speak your domain: medical, legal, insurance, manufacturing terminology, or Indian-language and code-mixed text that general models handle poorly.
  • Latency matters, or the model must run on your hardware or on-device.
  • Prompts have grown to pages of instructions and still fail on edge cases.

Fine-tuning does not reliably teach a model new facts. For facts, use retrieval (see private-data agents); for behaviour, fine-tune; for both, combine them.

Evaluation first

We will not fine-tune without an evaluation set, because without one nobody can say whether the new model is better. The set is 200–1,000 real examples with the right outputs, split by the cases that matter to you (common, rare, adversarial). It is scored automatically where possible and by your domain experts where not. The same set gates every release afterwards, which is how you avoid shipping a regression.

How we fine-tune

  1. Data — curate from your logs, documents and expert annotations; fill gaps with reviewed synthetic examples; remove anything you would not want the model to repeat.
  2. Base model — an open-weight model sized to the task (often 1–8 billion parameters for classification and extraction; larger for generation), chosen by licence and by baseline score on your eval set.
  3. Method — parameter-efficient fine-tuning (LoRA and variants) for most work; full fine-tuning when the budget and the gain justify it; preference tuning when the output is a judgement rather than a fact.
  4. Guardrails — input and output checks for the failures you cannot accept: leaking personal data, unsafe advice, off-task answers.
  5. Release — the model, its eval scores against the base and against the hosted model you use today, and a short note on where it still fails.

Serving and cost

A fine-tuned 7-billion-parameter model on a single GPU instance in an Indian cloud region handles the throughput of most SME workloads. For bursty workloads, serverless GPU endpoints keep idle cost near zero. For data that cannot leave the building, we have run 30-billion-parameter models with retrieval on one high-memory workstation. The feasibility note includes a cost comparison against your current API spend over twelve months, so the decision is financial as well as technical.

How a project runs

  1. Feasibility note (free) — the task, your current approach, a first look at the data, and whether fine-tuning is worth it at all.
  2. Two-week pilot (fixed price) — evaluation set, a first fine-tuned model, scores against the baseline and the hosted model.
  3. Production — serving setup in your account, guardrails, monitoring, a retraining recipe and handover.

Questions people ask

Which models can you fine-tune?

Open-weight families with permissive licences — the current Llama, Qwen, Gemma and Mistral lines and their Indian-language derivatives — chosen per task by licence and baseline score. Hosted-provider fine-tuning is an option when data terms allow.

How much data do we need?

For classification or extraction, a few hundred to a few thousand labelled examples usually suffice with parameter-efficient methods. For generation tasks, more. The feasibility call looks at what you have before promising anything.

Will a fine-tuned model hallucinate less?

It will follow your format and domain more reliably, which removes a class of errors. It does not stop a model from inventing facts; retrieval and guardrails do that, and we combine them.

Can we keep the model if we stop working with you?

Yes. You own the weights, the training data, the eval set and the serving code.

Tell us the problem, not the model.

A 30-minute call and a one-page feasibility note, free. If a two-week pilot makes sense, we scope it with a fixed price. If it does not, we say so.

Start a conversation See how a pilot runs

Related