If you mock the shell, you aren’t testing the agent.

Rooklight is the name we gave the extension suite we actually run: prepared navigation over a local project, a shell that compresses its own logs, web retrieval that knows the difference between a document and a rumour, a tool locker that does not blow the prompt cache every time a model epoch changes. It is a Pi coding-agent suite. It is pre-release. It is not on npm. Those are not modesties; they are the current facts.

The suite exists because we got tired of watching an agent fail in ways that our tests could not see. An agent that writes a file you never asked for, or skips the README and guesses an API, or starts a two-minute install and then gets killed by a test runner that assumed thirty seconds was plenty — those are not “flaky LLM” stories. They are harness stories. This issue is the harness.

The problem#

Tests want to be fast, deterministic, and isolated. An agent is none of those things. Ask it to “build a static site” and it does not return a string. It lists a directory, notices a missing dependency, runs a real install, waits, reads a config file, edits it, starts a server, and only then produces anything you could screenshot.

A naïve test hides all of that:

test('agent builds site', async () => {
  await agent.execute('build me a site');
  expect(existsSync('./dist/index.html')).toBe(true);
});

agent.execute is not one call. It is a loop of observations and actions. One of those actions might be a shell command that spends forty-five seconds fetching Chromium. Another might hang because a prompt cache expired. Another might succeed, quietly, after the test has already marked itself failed.

The second problem is worse, and it is the one in the headline. The first instinct of a software person building a harness is to mock the world: stub fs, intercept bash, return a fixture stdout. That instinct is correct for a unit test of your code. It is fatal for a test of an agent. Agents fail because of file encodings, permission bits, unexpected prompts, pager output, CRLF, a tool that writes to stderr and still exits 0. If you mock the shell, you are not testing the agent. You are testing the story you told yourself about the agent.

The constraints#

We wrote them down so we could not wriggle out later.

1. Real I/O. The agent executes real commands and writes real files, on a real disk, in a temporary directory we throw away. No memfs. No fake child_process.
2. Wait exactly as long as the work. A four-second task takes four seconds. A twenty-five-second install takes twenty-five. Nothing waits out a clock “just in case.”
3. Assert on the trajectory. What the model said will change every run. What it did — which tool, which path, which flag — is the thing we can stand on.
4. Observable without a debugger. When a test fails we need the tool log, the job log, and the files. Not a screenshot of a spinner.

Rooklight’s Shell extension already pointed at this. It has background jobs, recoverable logs, and reusable scratch cells. The harness had to live at that layer, not above it in a testing daydream.

What we tried first#

We wrapped the whole run in a timeout and went to lunch.

Spin up a temp directory. Inject a prompt. sleep 30_000. Look at the disk. That was the first harness. It failed constantly, which we treated as a calibration problem.

Sometimes npm install took five seconds and we sat there for twenty-five more, congratulating ourselves on being thorough. Sometimes it took thirty-four, the test failed, and then — this is the part that still makes us angry — the agent finished in the background and left a dist/ that the next test inherited. We had flakes and pollution. We had invented a way to be wrong twice per run.

So we did what everyone does. We made the sleep longer. Thirty became sixty. Sixty became two minutes. The suite crossed an hour. At that point the tests stopped being a tool and became a ritual you run in CI so you can say you ran them. Locally, people (we) prompted the agent by hand and looked at the output, which is how you know the harness has died even if the badge is green.

We also tried the mock. A fake bash that returned recorded transcripts. The tests went green and stayed green while the real agent was failing on a permission denied that the fixture had never seen. That week taught the sentence we now write on the inside cover: if you mock the shell, you aren’t testing the agent, you’re testing your own assumptions.

We already knew this, because the missing dependency took a week to surface — and because a suite that takes an hour is not a test, it is a vigil.

The turn#

We stopped treating the agent as a request with a deadline, and started treating it as a process with a job queue.

The harness does not sleep. It polls. While a background job is running, or the agent is still inside a tool loop, we wait for the next tick. The moment the run loop is idle and the queue is empty, we assert. Immediately. No padding.

while (agent.isActive() || system.hasRunningJobs()) {
  await waitForTick();
}
assert(existsSync('./dist/index.html'));

That is the whole trick, and it is almost embarrassing how long it took to write it down. It cut wall-clock time by something we measured on our own fixtures as roughly seventy percent — a four-second task now takes four seconds; a heavy install takes what the install takes; nothing flakes because a clock expired while a compiler was still working.

Polling is not free. A tick that is too tight burns CPU and log lines. A tick that is too loose wastes the savings. We settled on a wait that follows the Shell extension’s own job machinery rather than a second clock we invented. One source of truth for “is anything running.” Two clocks was how we got the hour-long suite.

The other half of the turn was admitting that the model’s prose is not an API. Wording shifts. Hedging shifts. The same successful run will say “done” or “built” or nothing, just a path. Asserting on the string is how you write a test that fails when the model becomes more concise.

The artifact#

Trajectory assertions. The harness intercepts the raw tool calls and appends them to a structured timeline. We do not assert on what the agent said. We assert on what it did.

expect(trajectory).toHaveAction('read', { path: 'docs/api.md' });
expect(trajectory).toHaveAction('bash', {
  command: expect.stringMatching(/npm install/),
});

Negative tests fall out of this for free. We do not want the agent to rewrite package.json from scratch when it only needs one dependency. We assert that edit was used, not write. We do not want it to skip the README. We assert that a read happened before the first bash. These are tests of method, which is the only thing worth testing in a creature whose sentences you do not own.

The timeline is also the failure report. When a test dies you get the sequence, the job logs, and the leftover files. That is the difference between “the agent failed” and “the agent ran rm in the wrong directory at step 6.” We have had both. Only one of them is debuggable.

Rooklight itself is the wider artifact this harness sits inside. Deepfield is the local-project evidence layer — read, then mutate, with a hash so an edit cannot silently target a different buffer. Shell is the compressed, logged, job-aware terminal. Stow-tools is the locker that keeps late tool activation from invalidating a prompt cache. Web is retrieval that will tell you when a page is not enough. None of that is in this issue because it is not the failure story. The failure story is the thirty-second sleep.

We will say again, because the README says it and we mean it: Rooklight is pre-release. The public repository identity, the first immutable release ref, and the repository-wide licence are not selected. Do not treat this checkout as a product. Treat it as the thing we run.

Deepfield belongs in a footnote of this harness even if it is not the plot. Hash-authorized mutation — read a span, get a hash, write only if the hash still matches — is the same instinct as trajectory assertions: do not trust a name, trust a fact you just observed. Agents edit the wrong buffer when the file changed under them. Tests that mock fs never see that race. Tests that use a real temp directory and a hash do. We still lose that race sometimes. We can at least see it.1

The Shell extension’s LeanCTX compression is the other reason the harness can afford real I/O. Unbounded stdout from npm install will blow a context window faster than a timeout will save you. Compact logs with an exact recoverable file on disk means the agent sees a summary and we still have the truth when the assertion fails. That pairing — small in the prompt, full on disk — is the only way we have found to keep the tests both honest and runnable.

What’s still open#

Caching. A clean temporary directory is honest, and honesty here is expensive. If a fixture needs a 300 MB dependency, we currently download it every time. We have a half-built copy-on-write layer that pre-warms node_modules and it is brittle in the exact ways copy-on-write is always brittle: leftover file owners, broken symlinks, a cache that is slightly the wrong Node version. We do not have a version of this we would defend in a review.

Visual assertions. The agent can build a page. Declaring that the page “looks right” currently means driving a browser from inside the same loop that we just stopped stuffing with timeouts. We will not bring the thirty-second sleep back under a different name. So for now we assert on files and trajectories, and we look at the page ourselves. That is a hole, and it is the hole this site’s own verification protocol keeps reminding us about.

Publication. There is no npm package. There is no public repo name we can print without lying. Follow the work here; do not install it yet.

The harness does the job we actually had: it forces the agent to prove competence in the world, without making us wait an hour for the proof. Everything else on this page is still open, and we would rather print that than round it off.