Efficient Token
Usage

Who This Is For

This guide is for two people who share a problem. The first is a regular user of AI chat tools — Claude, ChatGPT, Gemini — who gets mediocre responses, burns through message limits, or spends money on API access without getting full value. The second is a beginning engineer building on LLM APIs, wondering why costs are higher than expected and context degrades over long conversations.

Both of you have the same underlying gap: a mental model of tokens that treats them as a cost to minimize rather than a medium to work with. Fix that model and most of the practical problems sort themselves out.

You'll leave with a working understanding of what tokens actually are and how models use them, the habits that produce consistently better responses with less back-and-forth, and — for engineers — the concrete patterns for managing context in real applications.

Why This Matters Now

Token economics used to be an engineer problem. You paid per token, so you cared about token counts. Regular users didn't think about it because the interface abstracted it away.

That's changed. Most consumer AI tools now have message limits, context window limits, or usage tiers that you hit in practice. As more people use AI for longer, more complex tasks — multi-step projects, document analysis, extended coding sessions — the question of what to put in a prompt, how long conversations should run, and when to start fresh has real consequences for both quality and cost.

The deeper reason this matters: token efficiency and response quality are the same problem, not competing priorities. Prompts that use tokens well produce better responses. Prompts that waste tokens produce worse responses. The skill is identical in both directions.

Tokens Are a Medium, Not a Meter

Here's the reframe that makes everything else make sense: stop thinking about tokens as a resource to conserve and start thinking about them as a medium to shape.

A token is roughly a word fragment — about 3/4 of a word on average in English. "Tokenization" is how text gets converted into the numeric sequences that models actually process. Every word in your prompt is tokens. Every word in the response is tokens. The model reads your tokens, does its reasoning, and generates response tokens. That's the whole loop.

The Wrong Optimization

The meter framing says: tokens cost money, so use fewer. This leads to prompts that are aggressively short, ambiguous, and missing context — which produces worse responses that require more rounds of clarification, which uses more tokens overall.

The medium framing says: tokens are how you communicate intent to a system that has no other way to know what you want. The question isn't "how few can I use" but "am I using them to communicate the right things?"

The Practical Test

Before you send a message, ask not "is this too long?" but "does this contain what the model needs to do what I want?" Usually the answer is no, and the gap is specific, relevant information the model doesn't have.

What Lives in the Context Window

The context window is the model's working memory. Everything the model can "see" when generating a response lives there. Everything outside it doesn't exist.

The Components

System prompt — instructions that define how the model should behave for the entire conversation. In consumer products, this is set by the product. In API applications, you write it.

Conversation history — every message so far, both user and assistant. This grows as the conversation progresses.

Current user message — what you just sent.

Retrieved or attached context — documents, files, search results, tool outputs. In Claude.ai, this includes uploaded files. In API applications, whatever you inject programmatically.

The Signal-to-Noise Problem

Most token waste isn't from prompts being too long. It's from prompts and conversations that contain a lot of words but not much signal.

Signal is context that helps the model understand exactly what you want. Noise is everything else — filler, restated instructions, conversational padding, irrelevant history, hedges, and pleasantries that don't affect the output.

Two Forms of Noise

Restating what the model already knows. If you've already told the model it's helping you write marketing copy for a SaaS product, you don't need to restate that in every message. The conversation history already contains it. Repeating it wastes tokens and sometimes confuses the model about whether something has changed.

Vague instructions that require guessing. "Make this better" is low signal. "Make this more conversational, shorter, and cut the jargon in the second paragraph" is high signal. Same intention, completely different efficiency because the precise version produces what you want in one pass.

What High-Signal Prompts Contain

The specific task, stated precisely. The relevant context the model doesn't have (but only that context). The format or structure you want. The criteria by which you'll judge the output. An example of what good looks like, when that's hard to describe.

What they don't contain: your reasoning about why you want this, extensive background the model doesn't need for this specific task, restatements of things already established, or lengthy preambles before the actual request.

The Five Habits

These five habits, applied consistently, will change the quality of your AI interactions more than any other single change. They also happen to be token-efficient — but that's secondary. They work because they give the model what it needs.

1. Front-Load the Task

The model starts generating based on early context. If your message begins with three paragraphs of background and states the actual task at the end, the generation is already shaped before the task specification fully arrives. Lead with what you want, then provide context.

I've been working on this proposal for a few weeks now
and I'm a bit stuck. I'm trying to address some concerns
that came up in a recent meeting. There were a few people
who raised questions about the budget and the timeline.
Can you help me revise the executive summary?

Revise this executive summary to address two specific
concerns: (1) the budget seems underestimated for the
scope, and (2) the timeline doesn't account for regulatory
approval. Here's the current draft: [draft]

2. Specify the Output, Not Just the Input

Most prompts describe the situation. Fewer describe the desired output. Both matter.

"Here's my data, what do you think?" produces an open-ended response. "Here's my data — give me three specific findings and a recommended next action for each, in bullet form" produces something you can actually use.

Format specification in particular produces dramatically better output: prose or bullets, long or short, formal or casual, numbered steps or flowing text.

3. Use Roles Strategically — But Sparingly

"You are a [role]" instructions work because they activate a coherent cluster of knowledge, style, and approach. "You are an experienced product manager reviewing this spec" produces different output than "give me feedback on this spec" — not because the model becomes someone else, but because the framing narrows the space of relevant considerations.

Use this when the role genuinely changes what good looks like. Don't reflexively append a role to every prompt. "You are a helpful assistant" adds nothing because it's what the model already is.

4. Give Examples of What Good Looks Like

Describing quality is hard. Showing it is easy. If you want responses in a particular style, format, or level of detail — show an example. One well-chosen example is worth many paragraphs of description.

This applies especially when you've built a communication style over multiple conversations and want new outputs to match. Paste in the example. Don't try to describe it.

5. Know When to Start Fresh

Long conversations accumulate noise. Exploration that changed direction, completed tasks whose history remains, questions no longer relevant — all of this takes up context window space and can pull outputs in irrelevant directions.

Context Degradation: Why Long Conversations Get Worse

One of the most common complaints about AI tools is that responses get worse in long conversations. This is real, and understanding why helps you manage it.

As conversations grow, the proportion of your current message to total context shrinks. Early context that established important framing gets pushed further back. If the model's attention is imperfect — and it is, models attend to recent content somewhat more reliably than older content in very long windows — important context from earlier matters less.

Additionally, long conversations often contain drift: you asked one thing, explored a direction, decided against it, explored another. The context now includes multiple contradictory framings, half-formed ideas, and abandoned paths. The model tries to reconcile all of this, and the reconciliation is imperfect.

The Token Budget

When you're building on an LLM API, tokens are a literal budget with a literal dollar cost. As of early 2026, frontier model APIs cost roughly $3-15 per million input tokens and $15-75 per million output tokens.

The Quadratic Trap

The math most beginning engineers get wrong: every message in a multi-turn conversation includes the full conversation history. If you have a 10-message conversation and each message averages 200 tokens, message 10 includes roughly 2,000 tokens of history plus the system prompt plus the current message. By message 20, you're sending 4,000+ tokens of history on every request. The cost curve is quadratic, not linear.

Total input tokens = system_prompt
                   + conversation_history
                   + current_message
                   + injected_context

Total output tokens = generated_response

Input tokens are generally cheaper than output tokens because generation is more compute-intensive than reading. This means long system prompts are less expensive than verbose response formatting — but both matter at scale.

Model Tier Selection

This is the highest-leverage cost decision. Frontier models (Claude Opus, GPT-4) are 5-10x more expensive than capable mid-tier models (Claude Sonnet, GPT-4o-mini). For tasks that don't require frontier reasoning — classification, extraction, straightforward summarization, simple Q&A — mid-tier models perform nearly identically at a fraction of the cost.

Context Management Patterns

Managing context in an LLM application is the engineering equivalent of the conversation management problem for casual users — but you're doing it programmatically, at scale, and the decisions are consequential for both cost and quality.

Pattern 1: Rolling Window

Fixed system prompt, managed conversation history. Keep the most recent messages that fit within a token budget.

MAX_HISTORY_TOKENS = 4000

def trim_history(messages, max_tokens):
    """Keep most recent messages that fit within token budget."""
    total = 0
    trimmed = []
    for msg in reversed(messages):
        msg_tokens = count_tokens(msg["content"])
        if total + msg_tokens > max_tokens:
            break
        trimmed.insert(0, msg)
        total += msg_tokens
    return trimmed

This keeps the most recent history. The problem: it may drop important context from early in the conversation. The fix is summarization.

Pattern 2: Conversation Summarization

When history approaches the token budget, summarize old messages instead of dropping them:

def summarize_old_messages(messages, model):
    """Compress older conversation into a summary."""
    summary_prompt = f"""Summarize this conversation
    concisely, preserving key decisions, facts, and
    context needed to continue:

    {format_messages(messages)}"""

    summary = call_llm(summary_prompt, model=model)
    return [{"role": "system",
             "content": f"Previous conversation summary: {summary}"}]

This works well for conversational applications. It loses some fidelity but preserves semantic content. The cost of the summarization call is usually small compared to savings from reduced history tokens.

Pattern 3: RAG (Retrieval-Augmented Generation)

For applications that need to reference a large knowledge base — documentation, product catalogs, research papers — store the content in a vector database, retrieve only relevant chunks at query time, and inject those as context.

User query → Embedding model → Vector search → Top-K chunks
                                                     ↓
System prompt + Retrieved chunks + Query → LLM → Response

Your context window contains only what's relevant to this specific query. Cost per query stays roughly constant regardless of knowledge base size. The overhead: you need a vector database, an embedding model, and a retrieval pipeline.

Pattern 4: Tool Calls for Dynamic Context

The model requests specific information by calling functions you define, rather than having all information pre-loaded.

tools = [
    {
        "name": "lookup_order",
        "description": "Look up an order by ID",
        "parameters": {
            "type": "object",
            "properties": {
                "order_id": {"type": "string"}
            }
        }
    }
]

Instead of loading every order into context, the model requests the specific order it needs. Context grows only by the specific information retrieved, not all possible information. See the APIs field guide.

Prompt Design at Scale

In application development, prompts aren't one-off text you type — they're code. They need to be designed, tested, versioned, and maintained.

The System Prompt Is Infrastructure

In consumer chat, a bad system prompt means a slightly off response. In a production application, a bad system prompt is a systematic defect affecting every user. Treat it accordingly: version it, test it, review changes before deploying.

You are a helpful AI assistant. You are knowledgeable,
friendly, and professional. You always try to be helpful
and provide accurate information. You care about user
satisfaction and always do your best to help. Remember
to be helpful!

You are a support agent for Acme subscription billing.
Help users with:
- Invoice questions and payment history
- Subscription changes (upgrade, downgrade, cancel)
- Billing errors and refund requests

For anything outside billing, say: "For [topic], please
visit acme.com/help or contact general support."

Confirm the account email before discussing details.
Respond concisely — 1-3 sentences unless more is needed.

Output Format Specification

When your application parses output programmatically, instruct the model to produce structured output:

system_prompt = """
Analyze sentiment and key topics of customer feedback.
Respond only with valid JSON in this exact format:
{
  "sentiment": "positive" | "negative" | "neutral",
  "confidence": 0.0-1.0,
  "topics": ["topic1", "topic2"],
  "summary": "one sentence summary"
}
No preamble, no explanation, just the JSON object.
"""

The "no preamble, no explanation, just the JSON" instruction matters. Without it, models often wrap JSON in prose that breaks parsing. Modern APIs support structured outputs / JSON mode — use these when available.

Temperature and Sampling

Temperature controls how deterministic the output is. Lower = more deterministic. Higher = more varied.

Low temperature (0.0-0.3) — classification, extraction, structured data. Consistent, predictable output.

Higher temperature (0.7-1.0) — creative writing, brainstorming, generating multiple options.

The efficiency angle: high temperature produces more varied outputs, which means more retries when validating against a schema. For extraction tasks, low temperature reduces formatting failures that require a second API call.

Caching: The Token You Don't Pay For Twice

Prompt caching stores the computed state of a prompt prefix. If you send the same prefix again, the API uses the cached computation at a significantly lower rate.

For applications with a long, stable system prompt, this is material savings. A 2,000-token system prompt sent with every request at 1,000 requests/day is 2 million tokens per day. With caching, those tokens are read from cache at roughly 10% of the input token cost.

The Engineering Requirement

The cacheable portion must be at the start of your prompt, and it must be identical across requests. A system prompt that varies — includes the current date, the user's name, or any dynamic content — is not cacheable.

The Measurement Habit

You can't optimize what you don't measure. For both casual users and engineers, the habit of measuring produces improvements that intuition alone doesn't.

For Casual Users

The metric is rounds to a good answer. If you regularly need four or five rounds to get what you want, something is wrong with how you're framing the initial prompt. Keep informal notes of prompts that worked well — they're templates for similar future tasks.

For Engineers

The metrics that matter: cost per successful completion, token distribution across prompt components, and rate of validation failures requiring retries.

import anthropic
import logging

def tracked_completion(prompt, **kwargs):
    client = anthropic.Anthropic()
    response = client.messages.create(
        model=kwargs.get("model", "claude-sonnet-4-6"),
        max_tokens=kwargs.get("max_tokens", 1024),
        messages=prompt
    )

    logging.info({
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
        "model": response.model,
        "cost_estimate": estimate_cost(response.usage, response.model)
    })

    return response

Once you're logging this, patterns emerge. Certain tasks consume 3x the tokens you expected. Your system prompt has grown to 4,000 tokens because you kept adding without removing. One query type produces extremely long responses that aren't necessary.

Where This Breaks Down

The honest limits of token efficiency as a practice.

The compression trap. Taken too far, context compression loses signal. A conversation summary that's too aggressive loses the specific wording of constraints that mattered. A system prompt pruned to the minimum starts missing edge case behavior. There's a floor below which compression costs more than it saves.

Long-document processing has different economics. "Analyze this 200-page contract" is a fundamentally different regime — you need a large context window and you're going to pay for it. Use models designed for this and price accordingly.

Creative tasks legitimately need more tokens. Constraining token usage for brainstorming, creative writing, or exploratory research produces worse output. The efficiency principles here apply most strongly to tasks with well-defined outputs.

Caching fails on dynamic prompts. If your application has highly variable context per user or session, you get minimal cache benefit. Don't distort your architecture just to enable caching.

Context length doesn't substitute for context quality. More tokens in doesn't mean better answers out. A window full of tangentially relevant documents produces worse output than a smaller context with precisely relevant content. Long context is a tool for specific cases, not a substitute for prompt design.

Quick Reference

Cheat sheets for token efficiency.

Token Fundamentals

~1 token = ~0.75 words (English)
Context window = system prompt + history + current message + injected context
Everything outside the window doesn't exist to the model

Signal-to-Noise Checklist

□ Task stated at the beginning, not the end
□ Output format specified explicitly
□ Relevant context included, irrelevant excluded
□ No restatement of things already in history
□ Example of good output included when format is hard to describe

When to Start Fresh

□ Task has genuinely changed from the original intent
□ Conversation has long abandoned paths and contradictions
□ Response quality is degrading and conversation is 20+ turns
□ You can summarize key context in a paragraph and get better results

Context Management Patterns

Fixed window     → Keep last N messages by token count
Summarization   → Compress old messages into a rolling summary
RAG             → Retrieve only relevant chunks per query
Tool calls      → Model requests specific data as needed
Caching         → Stable prompt prefix → reduced cost on repeat

System Prompt Quality

Good: defines scope, constraints, output format, out-of-scope handling
Bad:  generic encouragement ("be helpful!"), restated defaults, vague personas
Rule: every sentence should change behavior for at least one case

Model Selection by Task

Frontier model needed:   complex reasoning, nuanced judgment, hard coding
Mid-tier sufficient:     classification, extraction, summarization, simple Q&A
Use frontier by default: expensive mistake

Cost Levers (Highest to Lowest Impact)

What You Can Do Now

Pick the last AI conversation you had that required four or more rounds to get what you wanted.

Find the first message you sent. Write it again from scratch — this time, lead with the specific task, state the output format you wanted, include the relevant context upfront, and add one example of what a good response would look like.

You don't need to send it anywhere. Just write it, then compare the two versions. The gap between them is the gap between your current prompting practice and the one this guide describes. That gap is where your token waste lives, and it's also exactly where your response quality improvements live.