--- title: "How Claude Code Works, From Tokens to Agents" description: "A visual walkthrough of what happens inside AI coding tools, built up layer by layer from a basic prompt to a full agent loop." date: 2026-03-27 tags: [ai, how-llms-work] url: https://nem035.com/thoughts/how-claude-code-works --- Tools like Claude Code, ChatGPT, and Cursor look like magic from the outside. You type a message, and the AI reads your files, fixes bugs, runs tests. Under the hood, it's a stack of pieces, and once you see them, the behavior of these tools gets a lot more predictable. This walkthrough builds that stack from scratch, starting from the simplest possible interaction and adding layers until we arrive at something like Claude Code. Every diagram is interactive, so click around. --- ## Layer 1: A basic prompt At the bottom of everything is a language model. You send it text, it sends back text. That's the API. ### Token by token A **token** is a fragment of text, roughly a word or part of a word. The model operates on tokens, not characters or words. Tokens are also the billing unit: providers charge per token for both input (what you send) and output (what the model generates). Type in the box below to see how text gets split: *[Visualization of how text is split into tokens]* Under the hood, the model generates its response one token at a time. At each step, it reads everything that came before (the entire prompt plus all tokens generated so far) and calculates a probability distribution over every possible next token. It picks one ([usually the most likely](https://huggingface.co/blog/how-to-generate)), appends it, and repeats. *[Animation showing token-by-token text generation]* The model has no branching thoughts, no parallel reasoning, no internal outline. Each token is predicted using all the tokens before it as context: *[Diagram of autoregressive token prediction]* When you see a long, well-structured response, the model didn't plan that structure in advance. It committed to the first token, then the second had to be consistent with the first, and so on. This also means the model has no persistent state. Between requests, it is [effectively off](/thoughts/the-biggest-misconception-about-llms). There is no background thinking, no memory of past conversations, no continuity from one prompt to the next. Every time you send a message, the model starts fresh with whatever text is in the prompt. Any "memory" of earlier turns exists only because the surrounding software preserved those messages and included them in the new prompt. The model itself retains nothing. Its knowledge is also frozen at training time. The model has no awareness of API changes, new library versions, or deprecations that happened after its training cutoff. It will confidently use patterns that were correct when it was trained, even if they're outdated now. ### Probabilities, not facts The model doesn't look up facts. It assigns probabilities to continuations and samples from the distribution, weighted toward the most probable tokens. *[Visualization of token probability distributions and sampling]* The mechanism is identical whether the output is correct or not. When the model has seen "the capital of France" thousands of times in training, the highest-probability token happens to be right. But when asked about something it has no information about, like your specific system, it still picks a probable continuation and keeps writing as if it knows: *[Animation showing token-by-token text generation]* The model has never seen your user data. It pattern-matched to what SaaS companies typically say about their users and produced it with full confidence. "Hallucination" is just the label we apply after the fact when the high-probability output doesn't match reality. The model has no way to tell the difference. In practice, hallucination in AI agents tends to show up in a few recurring ways: - **Wrong assumptions about your system** — the model generates code based on what it can see in context, but makes incorrect assumptions about parts it hasn't read. It might assume a field is nullable when it isn't, or write a full table scan for a table with 20 million rows because it has no idea about your data volume. It also commonly misses existing utility functions or shared structures and recreates them from scratch. - **Stale or fabricated APIs** — the model uses a method signature it has seen in training data, but the method has been deprecated, renamed, or never existed in the version you're using. This is partly the frozen-knowledge problem: the model's training data has a cutoff, and libraries keep changing. - **Pattern autocomplete** — the model applies a pattern it has seen frequently, even when the specific context is different. It might generate a UUID for a field that's actually a slug, or store "United States" in a column that expects country codes like "US." All of these come back to the same underlying mechanic: the model commits to tokens sequentially and can't go back. But that sequential nature also has an upside, because it means the order in which tokens are generated matters a lot. ### The tokens are the thinking There's no hidden scratchpad where the model works things out before responding. The reasoning happens in the output tokens themselves. Each token is produced based on everything before it, so earlier tokens shape later ones but not the other way around. Here's the other thing: the neural network does roughly the same amount of computation for every token it produces, whether the question is trivial or extremely hard. That means the model can't "think harder" about a difficult problem. The only way to get more reasoning is to generate more tokens. If you want the model to think before answering, it has to think out loud. Any time the model commits to an answer before working through the reasoning, the reasoning that comes after is post-hoc justification. This applies to follow-up messages ("how did you get that?"), to structured output where the answer field comes before the reasoning field, and to any situation where you ask the model to explain a decision it already made. *[Visualization of reasoning order effects]* ### Reasoning models and reasoning tokens Since earlier tokens shape later ones, more intermediate steps before the final answer means a better foundation to answer from. [Reasoning models](https://platform.openai.com/docs/guides/reasoning) (OpenAI's GPT-5 family, Anthropic's Claude with [extended thinking](https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking)) take this to its logical conclusion: they generate a long chain of **reasoning tokens** before the visible response. The final answer is produced with all that reasoning as context, which is why it tends to be better on hard problems. *[Animation showing token-by-token text generation]* The same principle applies to conversations, not just structured output. If you ask an agent to do something and then ask "why did you do that?", the explanation comes after the decision, so it's constrained to justify what was already said rather than genuinely reconstruct the process. And if the reasoning tokens from the original decision were discarded (which happens, see below), the model has even less to work with. *[Diagram of thinking direction in chain-of-thought]* The specifics of how reasoning tokens are handled differ by provider (and will continue to evolve), but the general shape is the same: - **OpenAI** discards reasoning tokens after each turn. On the next turn, the model sees previous inputs and outputs but not its prior reasoning. You can request a [summary](https://platform.openai.com/docs/guides/reasoning#reasoning-summaries) of what it thought about, but not the raw token sequence. - **Anthropic** returns thinking blocks with an encrypted `signature` that can be passed back across turns, preserving reasoning continuity during tool use loops. Current Claude models return [summarized thinking](https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking#summarized-thinking) by default (generated by a separate model), not the raw tokens. The recommended mode is now [adaptive thinking](https://docs.anthropic.com/en/docs/build-with-claude/adaptive-thinking), where the model dynamically decides how much to reason based on the complexity of each request. In both cases, you pay for the full reasoning tokens even though you only see a summary (or nothing at all). A single turn might generate anywhere from a few hundred to tens of thousands of reasoning tokens depending on the problem: *[Breakdown of reasoning vs output token usage]* Neither provider exposes the raw reasoning sequences (likely because they're valuable training signal). So that's layer 1: you send text, the model predicts tokens, you get text back. Everything else we'll add is built on top of this. --- ## Layer 2: The assembled prompt A raw prompt is just your text. But every real application wraps your text in a much larger prompt before sending it to the model. The application adds a **system message** (invisible instructions), tool definitions, project context files, and the full conversation history. Your message goes at the very end. When ChatGPT first launched in December 2022, its [leaked system message](https://github.com/jujumilk3/leaked-system-prompts/blob/main/openai-chatgpt_20221201.md) was about 25 lines of capability disclaimers. Today, production system prompts for tools like Claude Code run to thousands of tokens with tool definitions, personality settings, and policy constraints.
View ChatGPT's original leaked system prompt (Dec 2022)
{`Assistant is a large language model trained by OpenAI. knowledge cutoff: 2021-09 Current date: December 01 2022 Browsing: disabled - Assistant is a large language model trained by OpenAI. - Assistant does not have personal feelings or experiences and is not able to browse the internet or access new information. - Assistant's knowledge is limited to what it was trained on, which was cut off in 2021. - Assistant is not able to perform tasks or take physical actions, nor is it able to communicate with people or entities outside of this conversation. - Assistant is not able to provide personalized medical or legal advice, nor is it able to predict the future or provide certainties. - Assistant is not able to engage in activities that go against its programming, such as causing harm or engaging in illegal activities. - Assistant is a tool designed to provide information and assistance to users, but is not able to experience emotions or form personal relationships. - Assistant's responses are based on patterns and rules, rather than personal interpretation or judgment. - Assistant is not able to perceive or understand the physical world in the same way that humans do. - Assistant's knowledge is based on the data and information that was provided to it during its training process. - Assistant is not able to change its programming or modify its own capabilities, nor is it able to access or manipulate users' personal information or data. - Assistant is not able to communicate with other devices or systems outside of this conversation. - Assistant is not able to provide guarantees or assurances about the accuracy or reliability of its responses. - Assistant is not able to provide personal recommendations or advice based on individual preferences or circumstances. - Assistant is not able to diagnose or treat medical conditions. - Assistant is not able to interfere with or manipulate the outcomes of real-world events or situations. - Assistant is not able to engage in activities that go against the laws or ethical principles of the countries or regions in which it is used. - Assistant is not able to perform tasks or actions that require physical manipulation or movement. - Assistant is not able to provide translations for languages it was not trained on. - Assistant is not able to generate original content or creative works on its own. - Assistant is not able to provide real-time support or assistance. - Assistant is not able to carry out actions or tasks that go beyond its capabilities or the rules set by its creators. - Assistant is not able to fulfill requests that go against its programming or the rules set by its creators.`}
*[Diagram of the assembled prompt structure]* The model sees all of this as one continuous sequence of tokens. It doesn't treat any of these sections differently at a mechanical level. This is also where project context files like **CLAUDE.md** or **AGENTS.md** fit in. Their contents get injected alongside the system prompt, so the model reads them on every turn before it sees your message. Here's what the full assembled prompt looks like, with rough estimates of how much of the context budget each section typically consumes: *[Example of a full system prompt assembly]* --- ## Layer 3: Tool calling You saw tool definitions in the assembled prompt. But including tool schemas in the prompt just tells the model what tools exist. The model still only produces text. So how does a tool actually get called? The surrounding software handles it. When the model's response includes a structured tool call instead of plain text, the harness intercepts it, executes the tool, and feeds the result back as a new message. The model then continues, now with the tool's output in its context. *[Step-by-step walkthrough of a tool call lifecycle]* At each step, the model is producing text and conventional software is carrying out the action. There's also error handling you don't see. If the model outputs a malformed tool call (wrong casing, bad JSON), the harness tries to repair it rather than crashing. Large tool outputs get truncated, with the full content saved to disk. Notice how token usage compounds here. The model re-reads the *entire* conversation on every turn: the system prompt, tool definitions, your message, every previous tool call, and every previous result. So when the model reads auth.py on turn 2, the full file contents stay in the context for turns 3, 4, and beyond. *[Demonstration of how token prediction compounds across a sequence]* If your session adds 30k tokens of new content across 8 turns, you might expect to be billed for 30k tokens. But because the model re-reads everything on every turn, the actual billed input can be 100k+ tokens. The TokenCompounding visualization above shows exactly how this adds up. This re-reading overhead is also why providers are building new connection models. OpenAI's [WebSocket mode](https://platform.openai.com/docs/guides/websocket-mode), for example, keeps a persistent connection open and sends only the *new* tokens each turn instead of the full history. The server holds onto the prior conversation state in memory, so the model doesn't have to re-process everything from scratch. For agent loops with 20+ tool calls, this can cut end-to-end latency by roughly 40%. ### Extending tools Tools like Read, Edit, and Bash are built into the agent. There are three ways to add more. **CLI** is the simplest. The agent runs commands in the terminal and can pipe, filter, and chain them just like you would. If a service has a CLI, the agent can use it with zero setup. Prefer this unless you need more control. **[MCP](https://modelcontextprotocol.io/)** (Model Context Protocol) adds structure when you need it. You install an MCP server (there are existing ones for Postgres, GitHub, Sentry, and dozens of other services) and the agent discovers what tools are available at runtime. It calls them with structured inputs, the same way it calls built-in tools. The key advantage is access scoping — a Postgres MCP server can expose `query_users` and hide raw SQL, so the agent never gets direct database access. A traditional **API** integration is the most work. You read the docs, code against endpoints, and maintain the glue. This is the fallback when a service has no CLI and no MCP server. *[Diagram of tool extensions via MCP servers]* Most setups combine CLI for everyday flexibility with MCP for controlled access to external services. --- ## Layer 4: The agent loop With tool calling, the model can take a single action: read a file, run a command, make an edit. But real tasks take multiple steps. To fix a bug, you might need to run the tests, read the failing file, make a change, and run the tests again. An **agent** is effectively tool calling in a loop. The model evaluates the current state, outputs one or more tool calls, the system executes them, the results feed back into the context, and the model evaluates again. The loop ends when the model responds with plain text and no tool calls. Step through this bug-fix session to see how the loop works, including the branching (tool call vs text response), the permission check, and the loop-back when tool results feed back: *[Flowchart of the agent loop: evaluate, tool call, execute, inject result, repeat]* Here's what the same kind of session looks like in practice: *[Terminal-style code/command display]* Each turn is one round trip to the model. The turn counter, token usage, and cost are tracked by the harness. You can cap turns, set a dollar budget, or configure the reasoning effort level. Without limits, the loop runs until the model responds with no tool calls. This is what Claude Code, Cursor's agent mode, and similar tools are: an LLM in an agent loop, with tools for reading files, writing files, running shell commands, and searching code. --- ## Layer 5: Managing context The agent loop has a constraint. The context window, the total amount of text the model can see at once, is finite. Current flagship models (as of early 2026) like [Claude Opus 4.6](https://docs.anthropic.com/en/docs/about-claude/models) and [GPT-5.4](https://platform.openai.com/docs/models) support up to 1 million tokens. Everything accumulates: the system prompt, tool definitions, your project context files, every message, every tool call and its result. *[Visualization of context window filling over turns]* ### Bigger context windows don't mean better results Context windows have grown from 4K to 1M tokens in two years, but the advertised size and the *effective* size are different things. Anthropic's own [documentation](https://docs.anthropic.com/en/docs/build-with-claude/context-windows) puts it directly: > "As token count grows, accuracy and recall degrade, a phenomenon known as *context rot*. This makes curating what's in context just as important as how much space is available." Research consistently confirms this. ["Lost in the Middle"](https://arxiv.org/abs/2307.03172) (Liu et al., 2023) found that models are best at using information at the beginning and end of the context, with significant accuracy drops for information in the middle. Later benchmarks ([RULER](https://arxiv.org/abs/2404.06654), [MRCR](https://arxiv.org/abs/2501.03276), [GraphWalks](https://arxiv.org/abs/2412.04360)) show the same pattern: accuracy degrades with length, and the degradation is worse for complex tasks than simple retrieval. *[Chart showing how accuracy changes with context size]* Simple needle-in-a-haystack retrieval (finding one fact in a sea of text) holds up reasonably well at long contexts. But multi-step reasoning, aggregation across many pieces of information, and tasks that require understanding the full context degrade faster. This is why agentic tools don't just dump your entire codebase into a 1M context window and call it a day. Compaction, subagents, and focused context files exist because *quality* of context use matters more than quantity. ### RAG: searching before reading This is also why **RAG** (Retrieval-Augmented Generation) exists: instead of putting everything into context upfront, you **search** for the relevant pieces first and only add those. *[Timeline of context management events across a session]* RAG is a broader pattern than vector databases and embeddings. Tools like [Cursor](https://www.cursor.com/) use it for codebase search: when you ask a question, the tool searches your codebase (using a mix of [text indexing](https://www.cursor.com/blog/fast-regex-search), [semantic search](https://www.cursor.com/blog/secure-codebase-indexing), and file path matching), finds the most relevant files and functions, and adds just those to the context before sending the prompt to the model. Claude Code does something similar with its Glob and Grep tools, which the model calls during the agent loop to find relevant code before reading it. The point is the same regardless of implementation: context is expensive and accuracy-sensitive, so you want to be selective about what goes in. Searching first (whether with embeddings, keyword search, file path matching, or just grep) and adding only what's relevant is almost always better than stuffing everything in and hoping the model finds what it needs. ### Pruning and compaction When the buffer fills up, the system has two strategies. First, it **prunes**: old tool outputs get replaced with a placeholder like `"[Old tool result content cleared]"`, keeping the conversation structure but freeing the space those results occupied. If that's not enough, it **compacts**: older history gets summarized into a condensed version, and everything before the summary is dropped. This is why long sessions feel like the model "forgets" things you told it earlier. It literally lost those tokens. Instructions that need to persist belong in your [project context file](/thoughts/claude-md-is-not-the-problem) (CLAUDE.md, AGENTS.md, or equivalent) or the system prompt, not in a message at the start of the conversation. ### What goes into context matters more than how much A project context file that restates what the agent can figure out by reading your code is noise. It burns tokens every turn without adding information. A focused file containing only what the agent can't discover on its own, like business constraints, legacy decisions, and non-obvious gotchas, is [dramatically more effective](/thoughts/claude-md-is-not-the-problem). *[Visualization of how context quality affects model performance]* Two rules that work well: include what the agent can't learn from the code alone, and include what would take too long to figure out by reading. Everything else is overhead. One more detail: context files aren't just loaded from the project root. Tools like Claude Code and [OpenCode](https://opencode.ai) discover AGENTS.md files in subdirectories as the agent reads files there, so you can put directory-specific constraints closer to the code they apply to. ### Subagents When a task would bloat the main agent's context (like reviewing 8 files), the system can spawn a **subagent** with a fresh context window. The subagent gets its own instructions and the parent's task description. It does its work, and only the final summary returns. *[Diagram of parent-child subagent delegation]* In practice, different subagent types exist for different jobs. In Claude Code and OpenCode, an **explore** subagent is read-only: it can search, read files, and browse, but can't edit anything. A **general** subagent has full tool access for complex multi-step work. The parent picks the right type based on the task. "Find all the places we handle auth errors" gets an explore agent. "Refactor the auth module and update the tests" gets a general agent. ### Plan mode vs build mode Most agentic tools also have a **plan mode** where the model can read and reason but can't make changes. This is the same principle from earlier: [the model needs to think before it acts](#the-tokens-are-the-thinking), and thinking out loud in planning tokens before committing to changes produces better results than jumping straight into edits. In plan mode, the agent only has access to read-only tools (read, search, grep, glob). It explores your codebase and produces a plan. Once you approve, the system switches to build mode, unlocking the full tool set. *[Terminal-style code/command display]* This is useful for complex tasks where you want to review the approach before any code changes happen. The planning phase builds understanding (reading files, searching for patterns, identifying gotchas), and the building phase executes from that plan. --- ## Layer 6: Permissions and hooks The last layer is control. An agent with access to your terminal and filesystem is powerful, and you need guardrails. Before any tool executes, the system can run a **hook**: a piece of your code that inspects the tool call and decides whether to allow, block, or modify it. *[Diagram of the hook interception points in the agent loop]* You can also set broader permission modes: auto-approve file edits, require approval for shell commands, block categories of tools entirely. Most production setups combine pre-approved safe tools with hooks that gate dangerous operations. There's also automated safety logic. If the model makes the same exact tool call three times in a row (a "doom loop"), the system pauses and asks the user before continuing: *[Terminal-style code/command display]* Tools like OpenCode also maintain a shadow git repository that snapshots the working tree at each agent step, so you can revert any change the agent made without manually tracking what it touched. ### Prompt injection There's a security risk worth knowing about. Since the model reads tool results as part of its context, a malicious file or API response can contain instructions that the model follows as if they were part of the prompt. This is called **[prompt injection](https://simonwillison.net/2025/Apr/9/mcp-prompt-injection/)**. For example, if the agent reads a file that contains ``, a vulnerable model might treat that as a real instruction. In practice, modern models are trained to resist obvious injection attempts, and the permission system (hooks, allowed tools, approval gates) provides a second line of defense. But the attack surface exists because the model has no structural way to distinguish "text the user intended me to read" from "text that's trying to manipulate me." It's all tokens. This is one more reason why permission modes matter. An agent running with `--allowedTools "Read,Grep,Glob"` (read-only) can't be tricked into deleting files even if an injection succeeds in manipulating the model's intent. ### Sycophancy One more behavior worth knowing about: [sycophancy](https://arxiv.org/abs/2310.13548). Models tend to follow the user's direction without pushing back, even when the direction is questionable. Modern models have gotten better at correcting factual errors. If you state something wrong, they'll often push back: *[Terminal-style code/command display]* But they're still bad at challenging your *decisions*. If you say "let's add Redis here," the model starts setting up Redis. It rarely stops to ask whether Redis is the right tool for the job. *[Terminal-style code/command display]* This happens because of how training works. After the initial pre-training on text data, models go through a process called [RLHF](https://huggingface.co/blog/rlhf) (reinforcement learning from human feedback), where human raters score the model's responses and the model is optimized to produce responses that score higher. The problem is that raters consistently prefer responses that go along with the user's stated intent. Helpfully executing a request scores higher than questioning it. So the model learns: when someone says "let's do X," the high-reward completion is to start doing X. *[Comparison of prompt quality effects on output]* The costly mistakes in software tend to be architectural decisions that looked reasonable at the time, not typos or syntax errors. Even stating a vague goal like "make it fast" can lead the model toward over-engineered solutions because it has no constraints to reason against. If you frame your prompt as a direction ("add Redis for caching," "rewrite this in TypeScript"), the model will execute that direction. If you frame it as a question ("what could be causing the slowness?"), you're more likely to get a diagnosis before a prescription. This is also why ["vibecoding" without technical understanding has a ceiling](/thoughts/why-technical-people-with-ai-are-unstoppable). If you don't know enough to recognize when the model is making a bad architectural call, those calls compound. Each one makes the codebase harder to change, and the model will keep building on top of decisions it was never challenged on. A developer who understands the system can catch "you don't need Redis for 50 users" or "this is a DNS issue, not a caching problem." Someone without that background is more likely to follow the model's lead, and the model's lead is shaped by whatever framing you gave it. --- ## Layer 7: Automation Everything so far describes an interactive session: you type a prompt, the agent works, you review the result. But the same loop runs without a human at the keyboard. The harness handles turns, permissions, and termination the same way whether a person is watching or not. Tools like [OpenClaw](https://github.com/openclaw/openclaw) are built on exactly this: the same agentic loop from layers 1-6, wired to webhooks, cron jobs, and chat messages so it runs autonomously on your own machine. With tool calling and MCP, agents can integrate with any service that has an API or CLI: error trackers, CI systems, deployment platforms, databases, monitoring dashboards. You wire the trigger (a webhook, a cron schedule, a Slack command) to a non-interactive agent session with a budget and a set of allowed tools. The agent runs until it's done or hits a limit. *[Terminal-style code/command display]* The pipeline above is a GitHub Action triggered by a Sentry webhook. The agent reads the error, diagnoses the root cause, fixes it, runs tests and Playwright to verify, and opens a PR with the fix. The key constraints: `--max-turns 15` prevents runaway sessions, `--allowedTools` restricts what the agent can do, and the PR is labeled `needs-review` because automated fixes should never merge without human review. Playwright screenshots get attached as artifacts so the reviewer can see exactly what the fix looks like. This same pattern works with different triggers and different tools: - **Sentry/PagerDuty → agent → fix PR**: error monitoring triggers diagnosis and automated fixes - **PR opened → agent → code review comments**: read-only agent reviews code and posts findings - **Cron → agent → dependency update PR**: scheduled agent updates packages and runs tests - **Slack command → agent → investigation**: team member types `/diagnose auth slowness` and gets a report - **Deploy webhook → Playwright → agent → rollback PR**: if staging breaks after deploy, the agent diagnoses why The building blocks are the same in all cases: an LLM in a loop (layer 4), with tools (layer 3) extended via MCP and CLI to reach external services, managed context (layer 5), and permission guardrails (layer 6). The only thing that changes is the trigger and the prompt. If a service has a CLI or a web interface the agent can drive through a browser, it can be automated. --- That covers the machine: how tokens become actions, how context accumulates, how the loop runs and terminates. [Part 2](/thoughts/how-claude-code-works-part-2) covers what sits on top: skills (teaching agents new workflows), hooks (deterministic control over the loop), custom subagents (specialization for specific jobs), and the Agent SDK (the full loop as a library).