Tokens and Context: The GTM AI Mental Model

Every GTM AI workflow runs on tokens. Every dollar you spend on AI is a dollar spent on tokens. And whether your AI agent produces useful output or expensive hallucination comes down, more than almost anything else, to what tokens you put in the context window and how you put them there.

This guide builds the mental model: the working intuition that lets you make good architectural decisions, avoid the most expensive mistakes, and understand why some AI outputs are sharp and others are garbage. Think of it as the reasoning layer beneath the API documentation.

What a Token Actually Is

A token is the fundamental unit of text that language models process. It is not a word and it is not a character. It sits somewhere in between.

The practical conversion:

• 1 token = approximately 4 characters of English text
• 1 token = approximately 0.75 words
• 750 words = approximately 1,000 tokens
• To estimate tokens from a word count: multiply by 1.3 to 1.4

For common words, one word is often one token. For less common words, longer words, and technical terminology, a single word might become two or three tokens. Code and structured data tokenize less efficiently still. Non-English text is often worse.

Why it matters: Every model has a context window measured in tokens. Every API call is priced per token. Once you internalize these conversion rates, you can estimate the size and cost of any workflow before you build it.

Token Counts for Common GTM Documents

Before you can reason about context, you need a feel for how large typical GTM documents actually are in token terms.

A useful calibration: a complete, well-researched account brief for a single prospect sits around 4,000-6,000 tokens. A single AI SDR workflow that researches a prospect and generates a personalized email sequence consumes roughly 3,000-5,000 tokens per prospect, all in.

Context Windows: What the Numbers Mean

Every model comes with an advertised context window: the maximum number of tokens it can process in a single call. These numbers have grown dramatically. Models that topped out at 4,096 tokens in 2022 now commonly offer 128K, 200K, or 1 million tokens.

Large context windows are genuinely useful. But the number the model advertises and the context the model actually uses effectively are two different things.

The Gap Between Advertised and Effective Context

This is where most GTM AI builders go wrong.

Lost in the Middle

In 2023, a Stanford research team published "Lost in the Middle: How Language Models Use Long Contexts"↗ in the Transactions of the Association for Computational Linguistics. The finding was counterintuitive and has since become one of the most referenced results in applied AI.

Models exhibit a U-shaped performance curve across their context window. Performance is highest when relevant information appears at the beginning or end of the context. Performance degrades significantly when relevant information is buried in the middle.

Performance can degrade by more than 30% when the same fact moves from the start or end of a context window to the middle. This holds even for models explicitly designed for long-context tasks.

What this means in practice: load an account brief, three past call transcripts, a competitor analysis, and a list of product features into a large context window. If the specific fact that should drive your email hook is buried in transcript number two, the model is substantially more likely to miss it, misremember it, or ignore it entirely.

The most critical information belongs at the top of your context, or at the very end. Always the edges.

Context Rot

In July 2025, Chroma published "Context Rot: How Increasing Input Tokens Impacts LLM Performance,"↗ a study covering 18 state-of-the-art models including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3.

The central finding: every single model tested degraded in performance as input context length increased, even on simple tasks like retrieval and text replication.

The mechanism is attention. Every token competes with every other token for the model's focus. As context grows, each token gets proportionally less of it. The model's working memory is finite. Adding tokens beyond what the task needs dilutes the signal.

In practice, most models effectively utilize only 10-20% of their theoretical context window for complex tasks. Models in the Chroma study "fell far short of their Maximum Context Window by as much as >99%" in real task performance.

Context Stuffing: The Anti-Pattern

With million-token context windows now available and prices falling, there is a tempting architectural pattern: dump everything into the context and let the model sort it out. Every CRM record, every call transcript, every piece of company research, every product document. All of it, in one prompt.

This is called context stuffing. Production AI research identifies it as the single largest source of wasted compute, hallucinated outputs, and unreliable agent behavior in GTM pipelines.

Why context stuffing fails:

1. The Lost in the Middle effect. Critical information in the interior of a large context gets missed or misrepresented. The problem scales with context size.

2. Context rot. Performance degrades as tokens accumulate, even before you approach the limit.

3. Attention dilution. The model's attention budget is finite. More noise means less focus on the signal that actually matters.

4. Hallucination amplification. More context means more potentially conflicting information. Conflicts and distractors increase hallucination rates.

5. Cost and latency. A 200K token prompt for a task that actually needed 8K tokens costs 25 times more and takes substantially longer.

In production, these effects compound.

Research comparing context stuffing directly to retrieval-augmented generation (RAG) found that RAG achieved equivalent output quality with less than half the tokens and roughly half the latency.

Context Quality vs. Context Quantity

The most important insight in applied AI for GTM is this: the quality of your context matters more than the size of it.

A tightly selected 5,000-token context with the right account signals, the relevant call transcript, and a clear ICP definition will outperform a bloated one. Even 50,000 tokens performs worse if 40,000 of them are loosely related noise.

This has direct implications for how you build GTM AI workflows:

RAG vs. Long Context: What the Research Says

A January 2025 paper directly comparing long-context LLMs against retrieval-augmented generation, "Long Context vs. RAG for LLMs"↗, produced findings worth understanding before you architect GTM AI pipelines.

For most day-to-day GTM AI use cases (outreach generation, lead scoring, call summaries, account briefs), RAG or selective context injection outperforms bulk context loading. The exception is synthesis-heavy tasks (full document analysis, contract review, earnings call synthesis) where the model needs to reason across the entire scope. That's a narrow category.

Token Economics: What Things Actually Cost

The cost of AI has collapsed. As of March 2026, the price per token has declined roughly 280x from GPT-3.5 launch in November 2022. A workload that cost $10,000 per month in model API fees in 2023 likely costs $50-200 today.

Price ranges across models are wide. Routing decisions matter:

Per-call cost: 10,000 input tokens, 1,000 output tokens

Per-call cost: 100,000 input tokens, 5,000 output tokens

At 100K input tokens, the spread between the cheapest and most expensive options is more than 30x. Context architecture is cost architecture.

This is another reason to avoid context stuffing. Every token you trim from the context window is a direct cost reduction. Output quality often improves at the same time, because of context rot.

A Framework for GTM Token Budgets

Not every task needs the same context depth. A practical mental model:

What This Means for GTM AI Pipelines

The research points toward a consistent architectural principle: structured, targeted context outperforms bulk context at every price point.

The GTM AI workflows that perform best in production thought carefully about what the model needs for each task. They retrieve exactly that and structure it with the most important signals first.

The practical version of this principle looks like:

• An AI SDR tool that pulls a prospect's recent job change, one relevant news item, and the ICP definition for that segment beats one that dumps the full LinkedIn history, every news article from the past year, and all prior email history into a single prompt.

• A call summary workflow that feeds the transcript plus the CRM schema beats one that also injects the company profile, the contact's LinkedIn, and the rep's full account history.

• A lead scoring system that retrieves the ten most predictive firmographic signals plus one or two behavioral indicators beats one that loads every available data field.

The model is capable. The question is whether you are giving it the right raw material to work with. Fewer, better-selected tokens consistently outperform more tokens of loosely relevant context.

---

Research sources: Liu et al. (2023), "Lost in the Middle: How Language Models Use Long Contexts,"↗ Stanford / TACL. Chroma Research (July 2025), "Context Rot: How Increasing Input Tokens Impacts LLM Performance."↗ "Long Context vs. RAG for LLMs: An Evaluation and Revisits"↗ (January 2025). Google Research, "Chain of Agents: Large Language Models Collaborating on Long-Context Tasks"↗ (NeurIPS 2024). Token pricing data from pricepertoken.com↗ and official provider pricing pages, March 2026. Token economics via Stanford AI Index 2025↗.