FIELD*MANUAL — AI ENGINEERING
§00 · PREMISE 0%
SCROLL — J / K TO MOVE — I FOR INDEX
FIELD MANUAL // AI ENGINEERING // TACIT KNOWLEDGE, COMPILED

From chatbot to fleet.

A plain guide to coding agents: what they are, how to steer them, when to trust them, and how to scale without losing control.

V 1.2 — 2026-08-03 WORDS — READING — RULES — SOURCES —
§00Premise

Read this first.

You have used a chatbot. You typed, it answered, and sometimes it was right. That was the demo. An agent can inspect a workspace, act, check the result, and try again.

AI engineering is the work around that agent. You choose its context, tools, permissions, tests, and stopping rule. The model supplies capability. The surrounding system turns capability into dependable work.

This manual starts at zero and proceeds through the model, agent loop, context, harness, briefing, workflow, verification, and multi-agent systems. The later systems reuse the same components with stricter boundaries.

Read it in order once. After that it is a reference.

Rules are numbered as they are introduced and collected at the end in §13.

§01Model

The model predicts the next token.

A large language model predicts the next token. Training gives those predictions a wide store of patterns. Post-training makes them follow instructions, use tools, reason across steps, and stop.

FIG 01 — NEXT-TOKEN SELECTIONTHREE.JS · LIVE
For each text prefix, the model scores possible next tokens and selects one. The probabilities shown here are illustrative, not outputs from a specific model.

It can recall an API, infer a bug, and write a patch. It has no guarantee that the recalled API exists or the patch works. Fluency is a capability. It is not evidence.

Three properties determine how you should use it.

It is stateless. The model keeps nothing between calls. Each request starts cold. What looks like memory is the harness sending history again.

RULE 01The model only knows what is in the current request. Put required state in context.

The window is finite. The model reads a bounded number of tokens at once. Every instruction, file, tool schema, and error competes for that space. §03 is about spending it.

Output varies. Ask twice, get two answers. The machine can sound certain when it is wrong. Confidence is not a check.

RULE 02Trust output in proportion to how cheaply you can verify it.

Verification is central to the discipline. Karpathy draws a useful distinction: traditional software automates what you can specify; these systems automate what you can verify. The remaining sections explain how to make that verification reliable.

§02Loop

A model, tools, and a loop.

Connect the model to tools such as file access, command execution, editing, and search, then run it in a loop. The model acts, reads the environment's response, and acts again until it reaches a stopping condition. That is the basic agent pattern.

Simon Willison's concise definition is useful: "An LLM agent runs tools in a loop to achieve a goal." The goal must define when to stop; otherwise the loop can continue consuming time and tokens without completing useful work.

a minimal agent loop
loop:
  reply = model(context)
  if reply.calls_tool:
      context += run(reply.tool)   # add the tool result
  else:
      stop                     # goal met or human input required
FIG 02 — A MODEL, TOOLS, AND A LOOPTHREE.JS · LIVE
Each pulse represents a tool call. Its result is added to the context ring; when the context budget fills, the harness must clear or compact it.

In April 2025 Thorsten Ball demonstrated the pattern with a working code-editing agent in about 300 lines of Go and three tools: read a file, list files, and edit a file.

When you look at an agent editing files, running commands, wriggling itself out of errors, retrying different strategies - it seems like there has to be a secret behind it. There isn't. It's an LLM, a loop, and enough tokens.

THORSTEN BALL — HOW TO BUILD AN AGENT, 2025

Early agent loops often ran without making progress and consumed many tokens. The loop itself is simple; model capability and the surrounding controls determine whether it works. Newer models plan across steps, recover from errors, and stop more reliably.

RULE 03An agent is a model, tools, and a loop. When a product says "agent," ask which tools and whose loop.

Tool results, test output, and errors are added to the context window, so every loop iteration consumes part of a finite budget.

§03Context

Context is the budget.

The context window is the agent's working memory. Instructions, tools, files, history, and mistakes all compete for it. Much of AI engineering is deciding what gets in.

"Prompt engineering" words one request. Context engineering chooses everything the model sees across the task. The aim is not maximum context. It is the smallest high-signal set for the next step.

The budget is real. More context costs more, and useful facts become harder to retrieve as noise grows. Anthropic named the failure:

Context rot: as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases.

ANTHROPIC — EFFECTIVE CONTEXT ENGINEERING FOR AI AGENTS, 2025
FIG 03 — CONTEXT ROT, THEN COMPACTIONTHREE.JS · LIVE
The window fills and older tokens fade. At the limit, the full history compresses into one amber summary row; detail is discarded, but the summary remains in the next context.
RULE 04Context is a budget, not unlimited storage. Additional tokens increase cost and can make existing facts harder to retrieve.

The operational consequences are straightforward.

  • One task, one thread. The second task inherits the first one's noise. Finish, clear, re-brief. A saved plan file makes the restart free.
  • Compaction is lossy. Near the limit, the harness summarizes the transcript and carries on. Decisions survive. Nuance does not. Anything that must survive belongs in a file first.
  • Subagents isolate context. Send a worker to search widely and return a concise report. Its intermediate material stays out of the main thread.
RULE 05Use one thread per task. Start a new thread when the task changes.

A thread is no longer useful when the agent repeats the same mistake after correction. Failed attempts remain in context and can reinforce the wrong approach. After two failed corrections, stop, start a new thread, and provide a clearer brief.

RULE 06After two failed corrections, restart with a better brief.

Keep identifiers in context—paths, queries, and links—and let the agent fetch full content when needed. Claude Code searches a repository with grep instead of loading a complete index into the window. Aider sends a ranked map of code definitions, roughly a thousand tokens, instead of every file. Just-in-time retrieval usually uses the context budget more effectively than preloading.

RULE 07Keep identifiers in context and retrieve full content on demand.
FIELD NOTE — WHERE THE BUDGET ACTUALLY GOES

Treat instruction space as "ad space." Point to long files instead of embedding them. Inspect the context meter when your harness exposes one. Tool schemas and standing instructions can take a material share before the task begins.

FIG 04 — WHERE THE BUDGET GOESHTML · RESPONSIVE
An illustrative 200k-token window. Standing instructions, tool schemas, and accumulated history can consume much of the window before task-specific work begins.

The model does not retain the window between sessions. The harness manages what is sent on each request, including retrieval, compaction, tools, and permissions.

§04Harness

The harness is half the system.

The harness is the program wrapped around the model: its tools, its permissions, its loop, its interface to you. People argue about models. The harness decides just as much.

FIG 05 — THE SYSTEM AROUND THE MODELMATPLOTLIB
Block diagram: you brief the harness and get a diff and evidence back; inside the harness sit context assembly, tools, permissions, and the loop; the harness exchanges context and tool calls with the model, and runs commands against the environment whose results feed back into context
You talk to the harness, not the model. It assembles context, brokers tools and permissions, and runs the loop against the environment until the goal is met.

Agent benchmarks list model–harness pairs for a reason. A model cannot edit a file by itself. The harness chooses what it sees, what it can do, and what feedback it gets. Never compare a bare model to a product and call the difference intelligence.

There are three common operating modes. Choose one according to the task:

  • Interactive. Use a terminal or editor agent that you watch and steer—Claude Code, Codex CLI, Cursor, Aider, or Amp. Immediate feedback makes this suitable for learning and difficult work.
  • Delegated. Cloud agents take a task, work in a sandbox, and return a pull request—Codex cloud, Devin, Jules, or Copilot's coding agent. The unit is the attempt. Run multiple attempts only when the result justifies the added cost and review.
  • Custom. Open harnesses and agent SDKs—OpenHands, SWE-agent, and vendor SDKs—support agents embedded in a product or pipeline. Research harnesses often favour one general code-execution tool over many narrow tools because models already understand shell workflows.

Harness design can improve results without changing the model. SWE-agent raised benchmark scores through a better agent-computer interface: fewer consolidated commands, an editor that rejects invalid edits with a reason, and concise structured feedback instead of raw output. Treat agents as a distinct user class and design their interfaces accordingly.

Humans can flexibly ignore unnecessary information, whereas all content has a fixed cost in memory and computation for LMs.

YANG ET AL. — SWE-AGENT: AGENT-COMPUTER INTERFACES, 2024

Terminal-based agents remain useful because shell workflows can be scripted, run headless in CI, and accessed over SSH. The ecosystem is also standardizing. AGENTS.md, a markdown brief stored in the repository, is read by many major harnesses and used by more than sixty thousand open-source projects.

The July 2026 field

This is a dated map, not a podium. Vendor claims and user reports are useful priors. They are noisy, and they often conflict.

MODEL + COMMON HARNESSREPORTED EDGEWATCH FOR
GPT-5.6 Sol / Terra / Luna
Codex
Sol for hard, open-ended work; Terra for daily engineering; Luna for clear, repeated jobs. OpenAI and field reports praise persistence, debugging, and token efficiency. Some users report literal readings, over-engineering, spec drift, and weaker prose. Start with the lowest effort that passes your eval. Reserve Max and Ultra for jobs that earn them.
Claude Fable 5 / Sonnet 5
Claude Code
Fable is reported strong at intent, planning, review, and long jobs. Sonnet is the faster execution model and common default. Fable costs more, can be slow, and may reroute some requests under safeguards. Reports on Sonnet's best effort level conflict. Test the whole harness.
Gemini 3.6 Flash
Gemini CLI / Antigravity
Fast, multimodal, long-context, and built for tool and computer use. A good fit when latency or mixed media dominates. Google notes extra diagnosis on simple front-end jobs and weaker default UI taste than some earlier models. Give explicit visual references and constraints.
Open-weight models
Aider / OpenCode / custom
Privacy, on-premise control, inspectable serving, and predictable unit economics. GLM-5.2 is one current long-context example. Hardware, serving, quantization, tool schemas, and licensing become your problem. A small integration defect can erase a benchmark lead.

Choose with a local eval. Freeze five representative tasks and their repositories. Keep acceptance tests outside the agent's reach. Record success, wall time, cost, and human fixes. The winner is the cheapest setup that passes. Re-run after material model or harness changes.

Your first safe hour

  1. Use a small repo or disposable branch. Begin from a clean commit. Keep production credentials out.
  2. Ask the agent to inspect only: map the repo, name the relevant files, list the commands, and state the risk.
  3. Give one bounded change. If it crosses several files or the goal is ambiguous, ask for a written plan first.
  4. Let it edit and run checks. Require the exact evidence named in the brief.
  5. Read the diff. Run the critical check yourself. Commit the green result or discard the attempt.
a brief that works
Goal: Change one observable behavior.
Success: Name the tests, build, or screenshot that proves it.
Constraints: Name files, interfaces, and actions that must not change.
Verify: Run the checks and report evidence plus remaining risks.
Stop: Ask before destructive, external, or out-of-scope work.
RULE 08The harness is half the system. Learn one thoroughly; the operating habits and much of the configuration transfer.
§05Briefing

Brief every session as a fresh start.

A new session knows common programming patterns but not the repository's local decisions. A concise brief supplies the information it cannot discover efficiently.

The brief is a markdown memo—CLAUDE.md or AGENTS.md—loaded into sessions. Include build and test commands, local rules, and costly traps. Exclude facts visible in code, rules enforced by linters, and generic advice. Every line consumes context in every session.

Current Codex and Claude Code guidance converges on the same shape: short, practical, and local. Split rules by directory when scope differs. Add a standing rule only after the agent repeats a costly mistake. There is no magic line count; the test is whether every line changes behavior.

Bloated CLAUDE.md files cause Claude to ignore your actual instructions!

CLAUDE CODE DOCS — BEST PRACTICES
RULE 09Brief like a memo: short, imperative, specific. A long brief is less control, not more.

For each line, ask what breaks if you delete it. If nothing breaks, delete it. If an important rule is ignored, shorten or move the file before repeating the rule.

Start simple. Add rules only when you notice Agent making the same mistake repeatedly.

CURSOR DOCS — RULES
RULE 10Add a rule only after a failure proves you need it. Prune monthly.

At scale, scope the brief instead of growing it. Put local instructions near the code they govern. Split procedures from facts. Load both only when relevant.

The repo is the other half of onboarding: fast deterministic tests, a one-command build, explicit code, greppable names, and logs in files. These help people too. Write for the next reader, human or machine.

RULE 11A legible repo is leverage: fast tests, one-command builds, boring code, greppable names.
§06Workflow

Plan enough. Then code.

One task, start to finish: explore, plan when needed, implement, verify, commit. Tiny changes need a sentence. Ambiguous or wide changes need a file.

An observational study of roughly 400,000 Claude Code sessions found a useful split: people made most planning decisions; Claude made most execution decisions. It is not a law. It is a sound default. You own intent and acceptance. The agent owns search and mechanics.

Explore. Have the agent read before it writes: the relevant files, the call sites, the tests, the prior art. Ask for a report, not a diff. Peter Steinberger checks blast radius before anything — how many files will this touch? One file: just talk to it. Forty: plan.

Plan wide or unclear work in a file. In Boris Tane's workflow, the agent writes plan.md, the developer annotates it, and implementation begins only when the plan is settled. The file remains available after resets and compaction; chat history may not.

RULE 12Plan in a file. Edit the plan, not the diff — the plan outlives the context window.

For a feature, let the agent interview you one question at a time. Save the result as spec.md. Execute in a fresh session. The spec crosses the reset; the interview noise does not.

Scope. Size tasks to your attention, not the machine's. It will happily produce more than you can judge.

RULE 13Scope one task to one review sitting. What you cannot review, you cannot ship.

Test-first became agent leverage. A failing test is a target the machine can verify. Use red, green, refactor: minimum code to pass, structural changes separate from behavioral ones, commit only on green.

RULE 14Make it fail first. A failing test is a goal a machine can verify alone.

Never let it guess an API. Paste the docs. Pin the versions. Prefer boring, popular libraries the model has seen ten thousand times. Its knowledge has a cutoff; your dependencies do not.

RULE 15Paste the docs. Boring dependencies, pinned versions, no guessed APIs.

Commit every green. Keep diffs small and messages honest. When you know the shape, stub names, signatures, and TODOs yourself. The agent fills the known frame. Git remains the ledger and undo button.

RULE 16Commit every green step. Git is the undo button for machine work.
FIELD NOTE — USE A STRONGER MODEL FOR A BLOCKING DECISION

Hashimoto ships Ghostty features by consulting an oracle—a slower, stronger model—for the plan, then letting a faster agent execute it. He published all sixteen sessions of one feature: $15.98 and about eight hours across three days. Amp implements the same pattern as a tool. When the worker repeatedly stalls on one decision, escalate that decision instead of repeating the same low-cost attempt. mitchellh.com/writing/non-trivial-vibing

§07Eyes

Build observable feedback loops.

An agent needs direct feedback about the consequences of its work. Let it run the code, read errors, and inspect the resulting interface.

The canonical form, from the Claude Code docs: give it a check it can run. A test suite. A build that exits nonzero. A linter. A screenshot to compare against a mock.

Give Claude a check it can run: tests, a build, a screenshot to compare. It's the difference between a session you watch and one you walk away from.

CLAUDE CODE DOCS — BEST PRACTICES

Harnesses are converging on the same idea. Cursor added a browser so agents can verify frontends. Google's Antigravity produces artifacts — plans, screenshots, browser recordings — instead of only transcripts. Evidence is checkable. Narration is not.

RULE 17Give the agent checks it can run before increasing its autonomy.
RULE 18Quality tracks the tightest feedback loop, not the biggest model.

Do not use the same context to produce and independently evaluate work. Anthropic found this failure to be systematic in long autonomous runs:

When asked to evaluate work they've produced, agents tend to respond by confidently praising the work—even when, to a human observer, the quality is obviously mediocre.

ANTHROPIC — HARNESS DESIGN FOR LONG-RUNNING AGENTS, 2026

So split the roles. A fresh session reviews better than the one that wrote the code; it does not inherit the writer's assumptions. Scope the reviewer to correctness against the spec. Open-ended demands to "find issues" can manufacture churn. Stop when another pass finds no material defect.

Make failure legible. Log to files. Put ERROR and its reason on one greppable line. If a human cannot find the failure in ten seconds, the agent will struggle too.

RULE 19Demand evidence: the failing test before, the passing run after, the screenshot attached.
FIELD NOTE — TESTS THE MACHINE WROTE

The loop is only as good as what it optimizes. Dan Luu's caution: generated tests land "somewhere between worthless and marginally useful" — agents pass what exists, so tests worth passing are your job first. Kent Beck's three red flags that an agent is drifting: unexpected loops, unrequested features, and the moment it disables or deletes a test. That last one is §12 material.

§08Leverage

Automate what you repeat.

Repeated instructions should become reusable configuration or automation. Use advisory instructions for preferences and deterministic controls for guarantees.

Prose is advisory. Slash commands package repeated requests. Hooks are deterministic: your code runs before a tool, after an edit, or at session end. Use hooks to format, require passing tests before commits, and block destructive commands. Keep shared controls in repository configuration.

RULE 20Prose requests. Hooks enforce. Automate every rule you have repeated twice.

Skills are procedure on demand. A skill is a folder — instructions, scripts, references — loaded only when relevant. Sometimes-needed knowledge goes here, not in the always-on brief. Jesse Vincent's Superpowers packages disciplines such as brainstorming, TDD, and debugging, then pressure-tests whether agents follow them under time pressure.

Tools have a context cost. MCP connects agents to outside systems, but connected tool definitions consume context before use. Prefer a familiar CLI when it covers the job. When you expose tools, group them around workflows, return concise actionable results, and load large schemas only when needed.

Anything can be a tool. A shell script can be a tool, an MCP server can be a tool.

ARMIN RONACHER — AGENTIC CODING RECOMMENDATIONS, 2025
RULE 21Every connected tool consumes context before use. Prefer tools whose value exceeds that cost.

The agent is also a Unix utility. Headless mode turns a tested prompt into a migration loop. In CI, an agent can review on mention, triage new issues, or read nightly build logs. Keep human approval proportional to risk.

Autonomy is a permission problem. Prompts do not solve permission problems. Users can become habituated to repeated approval dialogs. Structural limits work better: an OS sandbox, scoped and short-lived credentials, blocked destructive commands, and controlled network egress. Prevent dangerous actions at the permission layer.

RULE 22Run autonomous agents in a sandbox with scoped, short-lived credentials.
§09Parallel

Running agents in parallel.

Parallel agents can reduce elapsed time, but only when their tasks and workspaces are isolated and their results can be reviewed efficiently.

The primitive is isolation. Locally: one worktree, one branch, one agent, one task. In the cloud: one sandbox per attempt. Isolation makes parallel attempts possible. For a high-value bug, two independent tries can beat one long argument. You pay for both and must review both.

Read-heavy work parallelizes well: research, review, triage, search, and summarization. Write-heavy work is harder because concurrent agents make implicit design decisions without seeing one another's changes. Cognition illustrates this with a Flappy Bird clone where one subagent built a Mario-style background and another built a visually incompatible bird:

Actions carry implicit decisions, and conflicting decisions carry bad results.

WALDEN YAN, COGNITION — DON'T BUILD MULTI-AGENTS, 2025

The durable rule is simple: delegate read-heavy work; keep write-heavy work in one thread. If writes must run in parallel, shrink tasks until interfaces and ownership are explicit.

RULE 23Parallelize reads. Serialize writes — or hand the merge to git and the tests.
FIG 06 — READS FAN OUT, WRITES QUEUETHREE.JS · LIVE
Read-heavy tasks can fan out, but write-heavy results still pass through a serial review and merge gate. That gate usually becomes the bottleneck.

Fleets get cheaper when you tier. A stronger model can plan; faster models can execute bounded work. Anthropic's research system used this shape. But routing adds failure modes. Measure success per task, not price per token.

More agents move the constraint to review and merge. Parallel branches can arrive faster than anyone can understand them. When machines produce faster than you can judge, the system is bounded by judgment. Use smaller diffs, stronger tests, and a scoped agent review before human review.

RULE 24Scale agents until review is the bottleneck. Then fix review, not agents.
§10Systems

Common multi-agent architectures.

Beyond a few agents, explicit coordination becomes necessary. Five recurring patterns cover most systems; each trades conversational continuity for isolation or scale.

FIG 07 — THE FIVE SHAPESMATPLOTLIB
Five small node-link diagrams: an orchestrator fanning tasks to workers with dashed reports back; a planner-worker-evaluator triangle; a linear thread with a dashed uplink to an oracle; agents spoking into a shared git repo with no links between them; and a circular loop around a stack of state files
Five coordination patterns. Amber marks the component doing the highest-cost reasoning; in pattern 4, agents coordinate through shared repository state, locks, and tests.

1 — Orchestrator and workers. A lead plans, spawns workers with clean windows, and synthesizes their reports. On Anthropic's internal research eval this improved over one agent while using far more tokens. State when to spawn, how many, and what each worker must return. Vague briefs make workers duplicate each other.

RULE 25Delegate with the brief you would want: objective, output format, boundaries, budget.

2 — Planner, worker, evaluator. Keep strategy, execution, and judgment in separate windows. Define a sprint contract before code. Let the evaluator drive the result like a user. In one Anthropic app experiment, a short solo run failed while a much longer, costlier three-role run worked. The harness bought reliability with time and money.

3 — One thread plus an oracle. A single linear agent keeps full continuity and consults a stronger model for difficult decisions. Amp provides the oracle as a tool; Hashimoto uses the pattern manually; Devin was built on the principle. It avoids the context fragmentation introduced by multiple workers.

4 — Environment-mediated coordination. Anthropic's C compiler used sixteen agents, a shared repository, file locks, and tests as the source of truth. Coordination happened through the environment rather than a lead agent. It produced a large working prototype, but not a drop-in toolchain: parts still called GCC, generated code was inefficient, and the Rust needed expert cleanup. The transferable result was that autonomy increased with verifier quality.

5 — Repeated fresh-session loop. Geoffrey Huntley's Ralph sends the same prompt to a fresh agent repeatedly. State lives in files, the task list, and git—not the transcript. Tests and typecheckers gate each pass, with one task attempted per iteration. It suits greenfield work with a hard verifier and is a poor fit for delicate brownfield changes.

Ralph is deterministically bad in an undeterministic world.

GEOFFREY HUNTLEY — RALPH WIGGUM AS A SOFTWARE ENGINEER, 2025

All five shapes keep state outside the window: progress files, task graphs, tests, and git. Persist working behaviors as code when possible. Plan the reset: a fresh agent should be able to read the files and continue without the old transcript.

RULE 26Persist required state in files and define the handoff before the run.

One last lesson from the long games. The Pokémon harness that carried a weak model — mapping tools, memory aids — actively slowed a strong one, which later beat the game on raw screenshots. Scaffolding is a bet against the model, and the model improves every quarter.

Every component in a harness encodes an assumption about what the model can't do on its own, and those assumptions are worth stress testing.

ANTHROPIC — HARNESS DESIGN, 2026
§11Entropy

Generation can outpace review.

Agents can complete longer tasks and produce code faster than people can understand it. METR's estimated task horizon has recently doubled over periods measured in months, although the rate depends strongly on the task set and remains uncertain.

This creates comprehension debt: unexamined assumptions in code that nobody has fully read. The controls are ordinary engineering—small interfaces, cleanup work, documentation, and time to understand what shipped. Faster generation does not remove maintenance work.

Calibrate yourself. In one randomized METR study, sixteen experienced open-source developers using early-2025 tools were 19% slower on their own repositories while believing they were faster. A later follow-up found selection effects that made its newer estimate a likely lower bound. Neither result is universal. Both say the same thing: measure your own work.

RULE 27Measure throughput and quality; do not rely on perceived speed.

Case studies and experiments

Twelve systems that defined the field's sense of what is possible, and what is not. Full citations in §14.

THE C COMPILERANTHROPIC · 2026

Sixteen agents, two weeks, shared git, file locks, and tests as truth. The prototype compiled Linux on three architectures. It still leaned on GCC in places and produced inefficient code.

Autonomy scaled with verifier quality. The artifact still needed expert judgment.
RAKUTEN'S 7-HOUR RUNVENDOR CASE · 2025

Rakuten reports one agent worked for seven hours in a 12.5-million-line codebase and implemented a numerical feature to 99.9% accuracy against a reference.

A long leash works when correctness is checkable.
CISCO CODEWATCHCOMPANY CASE · 2026

Cisco used Codex CLI on large C/C++ defect-remediation jobs. It reports work falling from weeks to hours and a 10–15× throughput gain; React migrations fell from weeks to days.

A reviewed plan and a strong regression suite can turn an agent into migration machinery.
PROJECT VENDANTHROPIC · 2025

Claude ran a real shop: lost money, briefly claimed to wear a blazer, fell for an imposter CEO. Profitability came from CRM, inventory tooling, and a supervisor agent — not a smarter model.

When money moves, guardrails beat instructions.
CLAUDE PLAYS POKÉMONLONG-HORIZON · 2024–26

The famous memory-harness experiment. Scaffolds that carried a weak model slowed a strong one, which later won on raw screenshots.

Re-audit the harness every model generation.
VOYAGERRESEARCH · 2023

The Minecraft agent that banked every success as executable code in a growing skill library — 15× faster tech-tree progress, no fine-tuning.

Persist behaviors as code, not prose. The library is the learning.
RALPHFOLK ENGINEERING · 2025

A loop feeds one prompt file to a fresh agent. Files and git hold the state; tests provide backpressure. Reported successes are greenfield and come from its author.

Fresh context plus a hard verifier can beat a clever, crowded transcript.
GAS TOWNFLEET · 2026

Yegge's Mad Max orchestrator: 20–30 persistent agents, a mayor, patrol agents, and merge-queue refineries, all on a git-backed task graph.

At fleet scale the problems are organizational, not technical.
DEVIN, FIELD-TESTEDAUTONOMY · 2025

Answer.AI gave the autonomous engineer twenty real tasks. Three succeeded. Failures took days, not hours — and clustered on ambiguity, not difficulty.

Full autonomy fails on scope ambiguity before code difficulty.
AI VILLAGEEXPERIMENT · 2025

Seven frontier agents, shared chat, a week to complete as many video games as possible. Zero completed. Hallucinated success, distractor obsession, weak spatial reasoning.

Computer-use lags coding badly. Don't extrapolate across modalities.
THE TAX AGENT LOOPPRODUCTION · 2026

Crete turns accountant corrections and product traces into eval cases, lets Codex make scoped changes, runs regression tests, then requires human review. It serves 30+ firms and thousands of returns.

Production learning is not memory. It is traces → evals → changes → review.
ALPHAEVOLVEDEPLOYED SEARCH · 2025–26

Gemini proposes programs; automatic evaluators score them; an evolutionary loop keeps the winners. Google deployed results in data-center scheduling and chip design; a logistics partner reports better routes.

High-entropy search works when the score is cheap, objective, and tied to reality.
§12Failure

Common failure modes.

The failures below are common and documented. Use their symptoms to detect problems early.

MODESYMPTOMRESPONSE
DOOM LOOPThe agent repeats the same failed fix after correction.Stop, start a new thread, and rewrite the brief (Rule 06).
REWARD HACKTests pass only because the agent weakened tests or added excessive mocks.Protect test files with hooks. Review test diffs before code diffs.
PREMATURE VICTORY"Done!" with no evidence, often as the window fills.Demand the passing run. Verify in a fresh session.
SELF-REVIEW BIASThe producing agent reports no problems despite visible defects.Review in a fresh context with a scoped rubric.
CONTEXT POISONAn early incorrect claim repeatedly influences later attempts.Restart the thread. Keep plans in files, not chat.
SCOPE CREEPThe agent changes code outside the requested scope.Check the expected file list before editing and review it in the diff.
PHANTOM DEPSA suggested package does not exist.Verify against the registry. Use lockfiles. Review installs.
STALE KNOWLEDGEDeprecated API, v2 syntax in a v4 world.Paste current docs. Pin versions. Boring libraries (Rule 15).
PROMPT INJECTIONAfter reading untrusted text, the agent follows instructions unrelated to the task.Treat fetched text as untrusted. Remove one lethal-trifecta capability (Rule 28).
SECRET LEAKAGECredentials enter model context or logs.Use scoped, short-lived credentials and egress control.
DESTRUCTIONrm -rf, force-push, dropped table.Sandbox. Deny-list hooks. Backups. No production credentials, ever.
SUPPLY CHAINThe agent installs an untrusted or compromised dependency.Pin and audit installs. Use an isolated environment and scoped tokens.

One story covers half the table. In July 2025 an agent ignored a declared code freeze and deleted a production database. The failure was not that prose lacked force. The agent held production credentials. The freeze was an instruction. The credential was a permission. Only one was real.

The security floor is Willison's lethal trifecta: private data, untrusted content, and a channel to the outside world. Put all three in one agent and a crafted webpage or issue can become an exfiltration path. Cut a leg with sandboxing, data separation, or egress control. Do not bet the system on detecting every malicious instruction.

FIG 08 — THE LETHAL TRIFECTAMATPLOTLIB
Venn diagram of three circles — private data, untrusted input, and a way out — with the triple overlap marked in red as an exfiltration path and the note: pick two
Any two legs are survivable. All three in one agent hand a crafted webpage the keys and the exit. Cut a leg: sandboxing, data separation, or egress control.

The LLM vendors are not going to save us! We need to avoid the lethal trifecta combination of tools ourselves to stay safe.

SIMON WILLISON — THE LETHAL TRIFECTA, 2025
RULE 28Never give one agent private data, untrusted input, and a way out. Pick two.
§13Ladder

Six levels of agent use.

Each level increases agent autonomy and shifts the human role from direct operation toward review, coordination, and governance.

L0CHATyou are: the operatorAsk questions and manually transfer code. You verify every result.
L1COMPLETIONyou are: the editorThe model completes code inside the editor while you retain control of the file and workflow.
L2PAIRyou are: the navigatorOne agent works in the repository while you plan, steer, review diffs, run tests, and commit.
L3DELEGATEyou are: the reviewerTasks go out; pull requests come back. Briefs and specs steer. Attempts are cheap, so selection and review become the job.
L4FLEETyou are: the dispatcherAgents work in parallel worktrees, CI jobs, or scheduled loops. This requires merge discipline, code-health budgets, and trusted observability.
L5SYSTEMyou are: the governorAgents plan, build, verify, and create work for other agents. Humans set policy, define quality, and audit evidence.

Move to a higher level only when verification at the current level is automated, trusted, and easy to review. If review is already a bottleneck at L2, delegation at L3 will increase the problem.

The taxonomy

The whole discipline as one tree. Expand what you need.

context/ — the budget (§03) budget — more context costs more; useful facts get harder to retrieve hygiene — one task, one thread; clear between; never argue retrieval — keep paths and queries in context; fetch full content on demand compaction — lossy; decisions survive, nuance does not isolation — subagents burn tokens elsewhere, return one page audit — inspect context use; measure the preamble; cap each tool's cost
briefing/ — onboarding (§05) standing — CLAUDE.md / AGENTS.md: short, imperative, tested scoped — nearest file wins; glob rules load on match task — spec.md → plan.md → todo.md; interview first knowledge — paste docs; pin versions; boring dependencies split — procedures and facts want different files legibility — fast tests; make dev; greppable names; logs to files
verification/ — eyes (§07) checks — tests, types, lint, build exit codes eyes — screenshots, browser, REPL, greppable logs evidence — failing test before, passing run after review — fresh context; scoped rubric; stop when defects stop judges — one LLM judge per rubric dimension; calibrate against humans
control/ — permissions and safety (§08, §12) permissions — allowlists and structural limits; avoid repeated low-value approvals hooks — deterministic: format, gate, block, notify sandbox — containers, OS sandboxes, egress control credentials — scoped, short-lived, budget-capped; never prod checkpoints — commit every green; git is the undo button trifecta — private data, untrusted input, a way out: pick two
leverage/ — extension (§08) commands — slash commands: canned prompts with arguments skills — procedures on demand; three-level disclosure tools — consolidate by workflow; short actionable responses mcp — integrations consume context; prefer a CLI when sufficient headless — -p mode; fan-out loops; CI residents; scheduled runs
scale/ — parallelism (§09) isolation — one agent, one worktree or sandbox, one task asymmetry — parallelize reads; serialize writes selection — best-of-N attempts; diff and keep the winner tiering — expensive model plans, cheap model executes bottleneck — review; smaller diffs; agent pre-review first
architecture/ — systems (§10) orchestrator — lead plans, workers report; scaling rules in the prompt trio — planner / worker / evaluator; sprint contract up front thread+oracle — one linear agent, stronger model consulted for difficult decisions environment-mediated — git, file locks, and tests coordinate independent agents loop — ralph: fresh context, state in files, backpressure gates memory — progress files, agent trackers, skill libraries as code
economics/ — the meter (§11) multipliers — tool loops and subagents multiply spend; meter the whole attempt caching — stable prefixes; cache reads are the cheap tokens measurement — usage meters; instrument feelings (the 19% lesson) health — reserve capacity for cleanup, docs, and refactoring horizon — task length is rising fast; estimates depend on the task set

The rules, compiled

That is the manual. The model table will age. The rules will last longer because they concern evidence, boundaries, and judgment.

* * *
§14Sources

Sources.

The record below mixes research, official guidance, vendor case studies, and firsthand reports. They are not equal. The article labels claims where the distinction matters.

Frontier models & production loops

  1. OpenAI — Introducing GPT-5.6 (2026)openai.com/index/gpt-5-6
  2. OpenAI — Latest Model Guide (2026, living doc)developers.openai.com/api/docs/guides/latest-model
  3. OpenAI & Cisco — Codex at Cisco (2026, company case)openai.com/index/cisco
  4. OpenAI & Crete — Building Self-Improving Tax Agents with Codex (2026, company case)openai.com/index/building-self-improving-tax-agents-with-codex
  5. Anthropic — Claude Fable 5 and Mythos 5 (2026)anthropic.com/news/claude-fable-5-mythos-5
  6. Anthropic — Claude Sonnet 5 (2026)anthropic.com/news/claude-sonnet-5
  7. Anthropic — What Claude Code Usage Says About Expertise (2026, 400k-session study)anthropic.com/research/claude-code-expertise
  8. Google — Gemini 3.6 Flash Model Guide (2026)ai.google.dev/gemini-api/docs/models/gemini-3.6-flash
  9. Google DeepMind — AlphaEvolve's Real-World Impact (2026)deepmind.google/blog/alphaevolve-impact
  10. Z.ai — GLM-5.2 (2026)z.ai/blog/glm-5.2
  11. Firsthand comparison — Fable 5 and GPT-5.6 Sol on one build (2026)reddit.com/r/ClaudeCode/comments/1uw5jbr/…
  12. Field thread — Fable users compare GPT-5.6 (2026, conflicting reports)reddit.com/r/ClaudeCode/comments/1uuow9o/…
  13. Community eval — Sonnet 5 versus Opus 4.8 on 24 tasks (2026)reddit.com/r/ClaudeAI/comments/1ux74pp/…

Anthropic & Claude Code

  1. Building Effective Agents (2024)anthropic.com/engineering/building-effective-agents
  2. Claude Code: Best Practices (2025, living doc)code.claude.com/docs/en/best-practices
  3. How We Built Our Multi-Agent Research System (2025)anthropic.com/engineering/multi-agent-research-system
  4. Effective Context Engineering for AI Agents (2025)anthropic.com/engineering/effective-context-engineering-for-ai-agents
  5. Writing Effective Tools for Agents (2025)anthropic.com/engineering/writing-tools-for-agents
  6. Code Execution with MCP (2025)anthropic.com/engineering/code-execution-with-mcp
  7. Equipping Agents for the Real World with Agent Skills (2025)anthropic.com/engineering/equipping-agents-…-agent-skills
  8. Effective Harnesses for Long-Running Agents (2025)anthropic.com/engineering/effective-harnesses-for-long-running-agents
  9. Demystifying Evals for AI Agents (2026)anthropic.com/engineering/demystifying-evals-for-ai-agents
  10. Building a C Compiler with a Team of Parallel Claudes (2026)anthropic.com/engineering/building-c-compiler
  11. Harness Design for Long-Running Application Development (2026)anthropic.com/engineering/harness-design-long-running-apps
  12. Claude Code Auto Mode (2026)anthropic.com/engineering/claude-code-auto-mode
  13. How We Contain Claude Across Products (2026)anthropic.com/engineering/how-we-contain-claude
  14. An Update on Recent Claude Code Quality Reports (2026)anthropic.com/engineering/april-23-postmortem
  15. Agent Teams (2026, experimental)code.claude.com/docs/en/agent-teams
  16. Project Vend, Phase Two (2025)anthropic.com/research/project-vend-2
  17. Harnessing Claude's Intelligence (2026, the Pokémon lineage)claude.com/blog/harnessing-claudes-intelligence
  18. Rakuten Case Study (2025)claude.com/customers/rakuten

Practitioners

  1. Simon Willison — Here's How I Use LLMs to Help Me Write Code (2025)simonwillison.net/2025/Mar/11/using-llms-for-code
  2. Simon Willison — The Lethal Trifecta (2025)simonwillison.net/2025/Jun/16/the-lethal-trifecta
  3. Simon Willison — Designing Agentic Loops (2025)simonwillison.net/2025/Sep/30/designing-agentic-loops
  4. Simon Willison — Vibe Engineering (2025)simonwillison.net/2025/Oct/7/vibe-engineering
  5. Simon Willison — Agentic Engineering Patterns (2026, living)simonwillison.net/2026/Feb/23/agentic-engineering-patterns
  6. Andrej Karpathy — the "vibe coding" post (2025)x.com/karpathy/status/1886192184808149383
  7. Andrej Karpathy — Sequoia Ascent Notes: Agentic Engineering (2026)karpathy.bearblog.dev/sequoia-ascent-2026
  8. Harper Reed — My LLM Codegen Workflow ATM (2025)harper.blog/2025/02/16/my-llm-codegen-workflow-atm
  9. Mitchell Hashimoto — Vibing a Non-Trivial Ghostty Feature (2025)mitchellh.com/writing/non-trivial-vibing
  10. Mitchell Hashimoto — My AI Adoption Journey (2026)mitchellh.com/writing/my-ai-adoption-journey
  11. Armin Ronacher — Agentic Coding Recommendations (2025)lucumr.pocoo.org/2025/6/12/agentic-coding
  12. Armin Ronacher — The Tower Keeps Rising (2026)lucumr.pocoo.org/2026/7/13/the-tower-keeps-rising
  13. Peter Steinberger — Just Talk To It (2025)steipete.me/posts/just-talk-to-it
  14. Thorsten Ball — How to Build an Agent (2025)ampcode.com/how-to-build-an-agent
  15. Geoffrey Huntley — Ralph Wiggum as a Software Engineer (2025)ghuntley.com/ralph
  16. Steve Yegge — Revenge of the Junior Developer (2025)sourcegraph.com/blog/revenge-of-the-junior-developer
  17. Steve Yegge — Six New Tips for Better Coding with Agents (2025)steve-yegge.medium.com/six-new-tips-…
  18. Steve Yegge — Gas Town (2026)yegge.ai/gastown
  19. Kent Beck — Augmented Coding: Beyond the Vibes (2025)newsletter.kentbeck.com/p/augmented-coding-beyond-the-vibes
  20. Addy Osmani — The 70% Problem (2024)addyo.substack.com/p/the-70-problem-hard-truths-about
  21. Addy Osmani — The 80% Problem in Agentic Coding (2026)addyo.substack.com/p/the-80-problem-in-agentic-coding
  22. Jesse Vincent — Superpowers (2025)blog.fsck.com/2025/10/09/superpowers
  23. Jesse Vincent — Some New Agentic Patterns (2026)blog.fsck.com/2026/07/05/new-patterns
  24. swyx — The Rise of the AI Engineer (2023)latent.space/p/ai-engineer
  25. Boris Tane — How I Use Claude Code (2026)boristane.com/blog/how-i-use-claude-code
  26. Shrivu Shankar — How I Use Every Claude Code Feature (2025)blog.sshh.io/p/how-i-use-every-claude-code-feature
  27. Domenic Denicola — My Agentic Coding Setup (2026)domenic.me/agentic-coding-setup
  28. Dan Luu — AI Coding (2026)danluu.com/ai-coding

Harnesses & conventions

  1. AGENTS.md — the open convention (2025–)agents.md
  2. OpenAI — Codex Cloud & Best-of-N Delegation (2026)learn.chatgpt.com/docs/cloud
  3. OpenAI — Codex Subagents (read-heavy delegation) (2026)learn.chatgpt.com/docs/agent-configuration/subagents
  4. Cursor — Rules (living doc)cursor.com/docs/context/rules
  5. Cursor — Introducing Cursor 2.0 and Composer (2025)cursor.com/blog/2-0
  6. Cursor — Agent Swarms and the New Model Economics (2026)cursor.com/blog/agent-swarm-model-economics
  7. Aider — Repository Map (2023–)aider.chat/docs/repomap.html
  8. Aider — Separating Code Reasoning from Editing (2024)aider.chat/2024/09/26/architect.html
  9. Cline — Memory Bank (2025)docs.cline.bot/prompting/cline-memory-bank
  10. Amp — Owner's Manual (the Oracle, subagents) (2025–)ampcode.com/manual
  11. Devin — Playbooks & Knowledge (2025)docs.devin.ai/product-guides/creating-playbooks
  12. Google — Introducing Antigravity (artifacts over transcripts) (2025)antigravity.google/blog/introducing-google-antigravity
  13. Zed — Agent Client Protocol (2025–)zed.dev/acp
  14. GitHub — Copilot Coding Agent (self-review, CLI handoff) (2026)github.blog/…/whats-new-with-github-copilot-coding-agent

Systems, studies & incidents

  1. Cognition — Don't Build Multi-Agents (2025)cognition.com/blog/dont-build-multi-agents
  2. Cognition — Multi-Agents: What's Actually Working (2026)cognition.com/blog/multi-agents-working
  3. Yang et al. — SWE-agent: Agent-Computer Interfaces (2024)arxiv.org/abs/2405.15793
  4. Wang et al. — Voyager: An Open-Ended Embodied Agent (2023)arxiv.org/abs/2305.16291
  5. AI Digest — AI Village: Seven Agents, Zero Games (2025)theaidigest.org/village/blog/claude-plays-whatever-it-wants
  6. METR — The Randomized Trial (19% Slower) (2025)metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study
  7. METR — Uplift Study Update (2026)metr.org/blog/2026-02-24-uplift-update
  8. METR — Time Horizon 1.1 (2026)metr.org/blog/2026-1-29-time-horizon-1-1
  9. Terminal-Bench 2.1 Leaderboard (2026)tbench.ai/leaderboard/terminal-bench/2.1
  10. The Register — The Replit Production-Database Deletion (2025)theregister.com/2025/07/21/replit_saastr_vibe_coding_incident
  11. The Register — Devin Field-Tested (Answer.AI, 3 of 20) (2025)theregister.com/2025/01/23/ai_developer_devin_poor_reviews
  12. CISA — Shai-Hulud npm Supply-Chain Alert (2025)cisa.gov/news-events/alerts/2025/09/23/…
  13. Socket — Slopsquatting (2025)socket.dev/blog/slopsquatting-…
  14. Hugging Face — July 2026 Security Incident (2026)huggingface.co/blog/security-incident-july-2026
  15. Palo Alto Networks — The OpenClaw/Moltbot Analysis (2026)paloaltonetworks.com/blog/ai-security/why-moltbot-may-signal-ai-crisis
  16. Fortune — On AI-Written Code at the Labs (2026)fortune.com/2026/01/29/100-percent-of-code-…

Community

  1. awesome-claude-code — the ecosystem indexgithub.com/hesreallyhim/awesome-claude-code
  2. ccusage — token accounting from session logsgithub.com/ryoppippi/ccusage
  3. Beads — a git-backed issue tracker for agentsgithub.com/steveyegge/beads
  4. Superpowers — skills that enforce disciplinegithub.com/obra/superpowers
  5. ClaudeLog — the thinking-budget ladderclaudelog.com/mechanics/ultrathink
  6. Context Mode — stdout-only tool sandboxing (2026)github.com/mksglu/claude-context-mode
  7. Hacker News — the planning-vs-execution thread (2026)news.ycombinator.com/item?id=47106686
  8. Pragmatic Engineer — Building Claude Code with Boris Cherny (2025)newsletter.pragmaticengineer.com/p/building-claude-code-…

Updated 2026-07-30. Official guidance supports product behavior. Vendor cases show possibility, not general effect. Community reports show perceived strengths and failure modes, not rankings. Quotes are short and linked; reported numbers keep their attribution.