A token budget per customer is a fixed allowance for AI use tied to one account. It stops that account when input plus output tokens pass the cap.
Startups use it to protect flat-price plans from heavy users. Token volume decides cost more than request count. A per-customer cap turns that risk into an enforceable number.
Table of Contents
- Why equal requests cost different amounts
- How does a customer cap work?
- Why provider limits are not enough
- Where budgets leak
Why equal requests cost different amounts
Providers bill by tokens, with separate input and output rates per model. One customer can therefore cost far more than another on the same request count. Output tokens often cost much more than input tokens.
Opstree reports that output tokens often cost 3-5x input tokens. Agentic tool-call loops, retries, and growing context silently multiply tokens. Equal activity can then produce very different bills.
How does a customer cap work?
The control sits between your product and the model. Each customer gets a key, an allowance, and a reset period.
Use passes until the balance runs out, then further calls fail. According to the LiteLLM proxy docs, admins mint virtual keys with max_budget, budget_duration, plus TPM/RPM caps, and requests fail once a key crosses its budget LiteLLM proxy documentation. According to Agentgateway, the same pattern debits input plus output tokens, refills on an interval, and rejects exhausted budgets with 429 Agentgateway budget documentation.
- Give each paying account its own key or user record
- Set a token or spend allowance plus a reset interval, such as 30 days
- Add TPM and RPM caps to slow bursts before the budget burns
- Fail closed with a clear 429, then offer upgrade, reset, or add-on
Why provider limits are not enough
OpenAI enforces RPM, TPM, daily, and other limits per organization and project rather than per end user. According to OpenAI, startups must therefore add their own per-customer quota layer OpenAI rate-limit guide.
A workspace spend cap also works at the wrong level. OpenAI says an organization or project can set a monthly hard limit that fails calls with an organization or project spend error. Alerts alone only notify and never stop traffic.
Where budgets leak
Hard limits are not exact. OpenAI says enforcement is not instantaneous, so spend can slightly overshoot. Agentgateway counts guardrail-rejected requests against quota.
That means a blocked prompt can still consume budget. Track rejected, retried, and tool-loop tokens in customer usage. As one example, Anthropic lists Sonnet 5.5 at $2 input and $10 output per million tokens, Opus 5.5 at $4 and $20, and Haiku 5.5 at $0.10 and $0.50 for prompts up to 100K tokens. Route long drafts, retries, and loops to cheaper models first.