Serving costs can exceed marketing spend because inference fees grow with every user action. Marketing spend is often fixed per campaign, while inference sits in cost of goods sold and climbs with usage.
For founders, that flips the old software math. Serving costs here mean the compute bill for running the model on each request. Heavy use can therefore overtake the budget meant to win those users.
Table of Contents
- Why does inference hit margins directly?
- When does serving overtake marketing?
- Why haven't cheaper tokens fixed it?
- How can founders control serving costs?
Why does inference hit margins directly?
Traditional software scales cheaply because serving one more user costs little. Generative AI breaks that pattern because each prompt, summary, or agent run triggers paid compute. Andreessen Horowitz found many generative-AI apps earn only 50-60% gross margins for this reason, as described in its 2023 platform analysis.
That margin pressure shapes pricing and fundraising. A startup cannot treat inference as overhead. It must forecast it like materials in a physical product.
When does serving overtake marketing?
The crossover often comes with free trials, viral growth, or generous usage limits. Acquisition looks cheap while existing users keep generating inference bills every day. OnlyCFO describes this shift as "COGS is the new CAC," with firms such as Cursor, Lovable and Replit treating inference as the distribution budget, as explained in its finance analysis.
The warning is direct. Startups cannot sustain high inference costs plus high sales-and-marketing spend at once. Founders must choose which lever buys growth.
Why haven't cheaper tokens fixed it?
Per-token prices have fallen sharply, but total consumption has risen faster. Equivalent performance that cost $20-$60 per million tokens in 2021-2023 fell to about $0.40 per million, yet enterprise AI spending still rose about 320%. Agentic workflows can use up to 1,000 times more tokens than single-turn chat, according to Auton AI News summarizing a16z and arXiv research in its June 2026 summary.
Declines also vary widely by task, from 9x to 900x per year. Teams cannot rely on cheaper tokens alone. Architecture decides whether savings reach the income statement.
How can founders control serving costs?
Start with product design, not vendor talks. Define which actions justify large model calls and which can use retrieval, cache, or smaller models.
Add usage caps, batch background work, and separate prices for writing-heavy features. Useful controls to review now: Track cost per active user alongside acquisition cost. Cut the workflow that never pays back its inference bill.
- cache repeat prompts and retrieval results
- route simple tasks to smaller models
- batch background and evaluation work
- set per-user compute budgets and alerts
- tie paid tiers to output-heavy actions