Most LLM APIs bill input and output tokens separately, commonly per million tokens. Input is everything you send: system instructions, chat history, documents, tool descriptions, and the current user message. Output is every generated token. Because output rates are often higher, unbounded answers can be a larger cost driver than long prompts.
A simple cost estimate
request cost = (input tokens × input rate) + (cached input tokens × cache rate) + (output tokens × output rate)Use one unit for each term. If rates are quoted per million tokens, divide token counts by one million first. Multiply the average request cost by expected volume. This reveals which part of the workflow is expensive instead of leaving you to guess from a monthly bill.
Why cached input matters
Some models price reused prompt prefixes at a lower cached-input rate. This matters when a system prompt, policy, long document, or tool schema is sent repeatedly. Caching is not an excuse to send unlimited context: a smaller relevant prompt is still easier to follow and often faster. But stable prefix design can improve high-volume economics.
Four costs teams miss
- Conversation replay: sending the whole chat on every turn grows input cost over time.
- Retries: malformed JSON or brittle tool calls can double the cost of a successful action.
- Over-generation: a response without a length limit can use more output than a user needs.
- Premium by default: simple extraction does not require the most capable route.
A practical control loop
- Log tokens, cache hits, model, latency, and a task-success signal.
- Set output limits and use structured responses where appropriate.
- Summarize or retrieve history instead of replaying every message.
- Route routine work economically and escalate exceptions.
- Review cost per successful task weekly, not just total spend.