Fine-tuning smaller models can lift startup margins by moving repeat work off costly frontier models. Fine-tuning means further training a small model on your own task examples so it handles one job well for less. Margin pressure grows when customers use the product more. Each extra query adds inference cost, so founders need a way to serve growth without paying frontier prices for every task.
Table of Contents
- Why inference squeezes margins
- What fine-tuning a smaller model does
- How one legal startup cut query cost
- Which tasks should move first
Why inference squeezes margins
AI products carry a cost structure that traditional software does not. Andreessen Horowitz analysis reported via TechCrunch found AI app companies spend roughly 20-40% of revenue on inference, with gross margins near 50-60% versus 60-80% for comparable SaaS in TechCrunch summary of the Andreessen Horowitz analysis.
That gap matters when usage rises fast. Agentic workflows can multiply tokens per customer action. If pricing stays flat, the extra tokens come straight from margin.
What fine-tuning a smaller model does
A smaller fine-tuned model is a specialist. It learns your repeated format, vocabulary, and quality bar, then handles that narrow task without the full power of a frontier model.
OpenAI defines this approach as distillation, where a smaller cost-efficient model is fine-tuned on outputs from a larger model to match it on specific tasks at much lower cost in OpenAI's explanation of API model distillation. The founder keeps the large model for hard or changing work. The small model owns stable, high-volume work.
How one legal startup cut query cost
Usage growth can turn a healthy account into a loss. Bloomberg reporting found legal AI startup Harvey saw gross margins fall from about +50% to -50% by June 2026 after agentic use drove a roughly 20x surge in token consumption in Bloomberg report relayed by The State of AI.
Harvey answered by post-training open-weight Kimi K3 into a system called Tenet with Fireworks. Harvey reports roughly 90% lower cost per query on a review-table feature and nearly double the completed holdout legal tasks in Harvey's post-training update. The lesson is direct: own the repeatable task with a smaller trained model.
Which tasks should move first
Move work that is frequent, stable, and easy to grade. Good candidates include classification, extraction, drafting from a template, review tables, and support answers with clear correct outcomes. Start with one stable workflow, measure cost per completed task each week, and expand only after the small model holds quality.
- Pick one high-volume workflow with steady inputs.
- Save strong outputs as training and test examples.
- Train the small model, then compare accuracy and cost per finished task.
- Keep edge cases and new requests on the frontier model.
- Recheck quality after product or data changes.