# How Do I Check That an Agent's Change Improved a Metric? > Prepare the repo and a scoring script once in a sandbox, then fork it. Score the unchanged code and the agent's change on the same forked disk, several rounds each, interleaved. Treat a drop in a deterministic score as real. Count a speed change only if it beats the baseline's own spread. Keep the diff, delete the forks. Source: https://sandboxer.com/guides/compare-an-agents-change-against-a-baseline-with-forks. Updated 2026-10-28. **Short answer:** Prepare the repo and a scoring script once in a sandbox, then fork it. Score the unchanged code and the agent's change on the same forked disk, several rounds each, interleaved. Treat a drop in a deterministic score as real. Count a speed change only if it beats the baseline's own spread. Keep the diff, delete the forks. Tests tell you the code runs. They don't tell you whether a "make search better" or "make this faster" change did what it claims. An agent can ship a change that passes every test, runs 2.6 times faster and ranks results worse. You only catch that if you measure the metric, before and after, on the same machine and the same data. This page shows that check end to end. It ran on prod on 2026-10-28 on `small` sandboxes (2 vCPU, 4 GB). No model ran: the agent's two attempts are scripted files, so you can see one rejected and one kept. To run many attempts at once and keep the one that passes the most tests, see [best-of-N agent attempts with parallel forks](/guides/best-of-n-agents-with-parallel-forks). To check a UI change in a browser, see [let a coding agent test its UI change](/guides/let-a-coding-agent-test-its-ui-change-in-a-sandbox). This page is about numbers that aren't pass or fail. ## The pattern | Step | Command | Measured on prod | | - | - | - | | Create the source, copy in the repo, build the data, commit a base | `sbx new`, `sbx exec` | 1.2 s create, 0.5 s setup | | Freeze the source | `sbx stop --wait` | 2.2 s | | A/A check: score the base in its own fork | `sbx fork --wait`, `./bench.sh 5 base=.` | 3.3 s to ready, 6.3 s fork to deleted | | A/B: base and the agent's attempts in one fork, interleaved | `git worktree add`, `./bench.sh 5 base=../base try1=../try1 try2=.` | 3.3 s to ready, 5.2 s to scored | | Decide, keep the diff, delete everything | `decide.py`, `git diff`, `sbx delete --yes` | 2.1 s for the fork's delete | The commands took about 21 seconds in total. Wall clock from the first `sbx new` to the last delete was 57 seconds, because we paused between steps to read the output. These are single runs, not benchmarks. ## 1. A metric worth checking The example repo ranks Python standard library modules for a plain-English query, such as "read and write zip archive files". The corpus is each module's docstring, built once in the sandbox. There are 30 labeled queries, each with the module that should come first. `search.py` is the file the agent is asked to improve. The baseline counts matching words: ```python # search.py: rank documents for a query. This is the file the agent is asked to improve. import re def tokens(text): return re.findall(r"[a-z0-9]+", text.lower()) class Index: def __init__(self, docs): self.docs = {name: tokens(name + " " + text) for name, text in docs.items()} def search(self, query, k=10): q = set(tokens(query)) scores = {name: sum(1 for t in toks if t in q) for name, toks in self.docs.items()} return sorted(scores, key=lambda n: (-scores[n], n))[:k] ``` `eval.py` scores it. It reports two kinds of number, and they need different treatment: - **Quality**: mean reciprocal rank (`mrr`), and how many queries have the right module first (`hit1`) or in the top 3 (`hit3`). For a fixed corpus and code, these are the same on every run. - **Speed**: milliseconds per query, the median of 15 timed passes. This changes from run to run. ```python # eval.py: score search.py on the labeled queries. Prints one JSON line. import json, os, statistics, time from search import Index docs = json.load(open("data/docs.json")) queries = json.load(open("data/queries.json")) # [["read and write zip archive files", "zipfile"], …] t = time.perf_counter(); index = Index(docs); build_ms = (time.perf_counter() - t) * 1000 rr, hit1, hit3 = [], 0, 0 for q, want in queries: ranked = index.search(q) rank = ranked.index(want) + 1 if want in ranked else None rr.append(1 / rank if rank else 0.0) hit1 += rank == 1 hit3 += bool(rank and rank <= 3) passes = [] for _ in range(int(os.environ.get("REPEATS", "15"))): t = time.perf_counter() for q, _want in queries: index.search(q) passes.append((time.perf_counter() - t) * 1000 / len(queries)) print(json.dumps({"mrr": round(sum(rr) / len(rr), 3), "hit1": hit1, "hit3": hit3, "n": len(queries), "msPerQuery": round(statistics.median(passes), 3), "buildMs": round(build_ms, 1)})) ``` `bench.sh` runs `eval.py` in several checkouts, taking turns, for N rounds. Taking turns matters: if the machine slows down for a moment, every arm feels it, not just one. ```bash #!/bin/bash # bench.sh ... Run eval.py in each checkout, interleaved, for N rounds. rounds=$1; shift for r in $(seq 1 "$rounds"); do for arm in "$@"; do label=${arm%%=*}; dir=${arm#*=} (cd "$dir" && python3 eval.py) | sed "s/^{/{\"arm\":\"$label\",\"round\":$r,/" done done ``` ## 2. Prepare the source once and freeze it ```bash sbx new --name search-src --type small --ttl 3600 src=sb_imdxffhhfg # the id sbx new printed tar czf repo.tgz repo sbx exec $src "echo $(base64 -w0 repo.tgz) | base64 -d | tar xz" sbx exec $src --timeout 120 "cd repo && python3 --version && python3 build_corpus.py \ && git init -q && git add -A && git -c user.name=agent -c user.email=agent@example.com commit -qm base \ && git status --short && python3 eval.py" ``` ```text Python 3.12.3 79 docs, 50859 chars {"mrr": 0.44, "hit1": 10, "hit3": 16, "n": 30, "msPerQuery": 0.177, "buildMs": 1.0} ``` Build the data inside the sandbox and commit it with the code. Here the corpus comes from the installed Python's docstrings. The same script on a laptop with Python 3.14 gave `mrr` 0.458, not 0.44, because the docstrings differ. If the baseline and the candidate don't read the same bytes, the comparison is meaningless. Then stop the source: ```bash sbx stop $src --wait ``` A stopped sandbox has no running processes, so nothing can change its disk between forks. You can still fork it: `sbx fork` copies the source's latest snapshot, and a stop always takes one. Ours forked 3.3 s after each request. ## 3. The A/A check: how noisy is the metric? Before you trust a "2x faster", score the unchanged code twice and see how much it moves on its own. The first fork scores only the base: ```bash id=$(sbx fork $src --name search-aa --type small --ttl 900 --wait --json \ | jq -r 'select(.event=="ready") | .id') sbx exec $id --timeout 120 "cd ~/repo && ./bench.sh 5 base=." | tee -a runs.jsonl sbx delete $id --yes ``` ```text {"arm":"base","round":1,"mrr": 0.44, "hit1": 10, "hit3": 16, "n": 30, "msPerQuery": 0.172, "buildMs": 1.1} {"arm":"base","round":2,"mrr": 0.44, "hit1": 10, "hit3": 16, "n": 30, "msPerQuery": 0.178, "buildMs": 1.0} {"arm":"base","round":3,"mrr": 0.44, "hit1": 10, "hit3": 16, "n": 30, "msPerQuery": 0.172, "buildMs": 2.7} {"arm":"base","round":4,"mrr": 0.44, "hit1": 10, "hit3": 16, "n": 30, "msPerQuery": 0.179, "buildMs": 2.6} {"arm":"base","round":5,"mrr": 0.44, "hit1": 10, "hit3": 16, "n": 30, "msPerQuery": 0.18, "buildMs": 2.6} ``` What this tells you: - `mrr` was 0.44 in all five rounds, and in the second fork below too. A quality score that is deterministic needs one run. Any change in it is caused by the code. - `msPerQuery` moved between 0.172 and 0.18 inside one fork, even though each figure is already a median of 15 passes. The base in the second fork had a median of 0.174, against 0.178 here. That gap is smaller than either fork's own spread, so the two forks agree. - `buildMs` swung from 1.0 to 2.7 ms. A single run of a number this small tells you almost nothing. A fork copies the disk only. It boots fresh, with no memory, page cache or warm processes from the source, and it can land on a different machine. Speed figures from two different forks are two different machines. That's why the next step puts the base and the attempts in the same fork. ## 4. The A/B run: base and attempts in one fork The second fork holds three checkouts of the same commit, made with `git worktree`. `~/base` stays untouched. The agent's two attempts go into `~/try1` and `~/repo`: ```bash id=$(sbx fork $src --name search-ab --type small --ttl 900 --wait --json \ | jq -r 'select(.event=="ready") | .id') sbx exec $id "cd ~/repo && git worktree add -q ../base HEAD && git worktree add -q ../try1 HEAD \ && echo $(base64 -w0 try1.py) | base64 -d > ../try1/search.py \ && echo $(base64 -w0 try2.py) | base64 -d > search.py && git worktree list" sbx exec $id --timeout 120 "cd ~/repo && ./bench.sh 5 base=../base try1=../try1 try2=." > runsB.jsonl ``` ```text /home/user/repo ecbe4c3 [master] /home/user/base ecbe4c3 (detached HEAD) /home/user/try1 ecbe4c3 (detached HEAD) ``` The two attempts: | Attempt | What the "agent" did | Does it run? | | - | - | - | | `try1` | A speed-up: one compiled regex, a `Counter` per document, and it dropped `.lower()` because "queries are lowercase already" | Yes, no errors | | `try2` | BM25 ranking with IDF weights, a short stopword list and plain suffix stemming | Yes, no errors | The first and last of the 15 lines in `runsB.jsonl`, trimmed: ```text {"arm":"base","round":1,"mrr": 0.44, "hit1": 10, "hit3": 16, "n": 30, "msPerQuery": 0.174, …} {"arm":"try1","round":1,"mrr": 0.407, "hit1": 9, "hit3": 15, "n": 30, "msPerQuery": 0.067, …} {"arm":"try2","round":1,"mrr": 0.906, "hit1": 26, "hit3": 28, "n": 30, "msPerQuery": 0.057, …} … {"arm":"base","round":5,"mrr": 0.44, "hit1": 10, "hit3": 16, "n": 30, "msPerQuery": 0.174, …} {"arm":"try1","round":5,"mrr": 0.407, "hit1": 9, "hit3": 15, "n": 30, "msPerQuery": 0.067, …} {"arm":"try2","round":5,"mrr": 0.906, "hit1": 26, "hit3": 28, "n": 30, "msPerQuery": 0.057, …} ``` ## 5. Decide with a written rule Write the rule down before you see the numbers, so neither you nor the agent argues with it later. `decide.py` compares one candidate with the base from the same fork. Any drop in quality rejects it. A speed change only counts if it is bigger than the spread of the base's own rounds. ```python # decide.py : keep or discard the candidate. import json, statistics, sys runs = [json.loads(l) for l in open(sys.argv[1]) if l.strip()] base = [r for r in runs if r["arm"] == sys.argv[2]] cand = [r for r in runs if r["arm"] == sys.argv[3]] med = lambda rs, k: statistics.median(r[k] for r in rs) spread = max(r["msPerQuery"] for r in base) - min(r["msPerQuery"] for r in base) b_mrr, c_mrr = med(base, "mrr"), med(cand, "mrr") b_ms, c_ms = med(base, "msPerQuery"), med(cand, "msPerQuery") quality = "better" if c_mrr > b_mrr else "worse" if c_mrr < b_mrr else "same" speed = "faster" if c_ms < b_ms - spread else "slower" if c_ms > b_ms + spread else "within noise" keep = quality != "worse" and speed != "slower" and (quality == "better" or speed == "faster") print(f"mrr {b_mrr} -> {c_mrr} ({quality}); ms/query {b_ms} -> {c_ms}, baseline spread {spread:.3f} ({speed})") print("KEEP" if keep else "DISCARD") sys.exit(0 if keep else 1) ``` ```bash python3 repo/decide.py runsB.jsonl base try1 python3 repo/decide.py runsB.jsonl base try2 ``` ```text mrr 0.44 -> 0.407 (worse); ms/query 0.174 -> 0.067, baseline spread 0.011 (faster) DISCARD mrr 0.44 -> 0.906 (better); ms/query 0.174 -> 0.057, baseline spread 0.011 (faster) KEEP ``` `try1` is the change a test suite waves through. It runs, it is 2.6 times faster, and it ranks worse. Without `.lower()`, the tokenizer skips uppercase letters, so a word like "SMTP" vanishes from both the query and the docs. `try2` puts the right module first for 26 of 30 queries instead of 10. The same script also gives the A/A result from step 3. Fed the base rounds from both forks, it reports `mrr 0.44 -> 0.44 (same)` and `ms/query 0.178 -> 0.174, baseline spread 0.008 (within noise)`, then `DISCARD`. That is the right answer: nothing changed. The exit code is the verdict, so an agent loop or a CI job can act on it directly. ## 6. Keep the diff, delete the forks ```bash sbx exec $id "cd ~/repo && git diff --stat" sbx exec $id "cd ~/repo && git diff" > try2.diff sbx delete $id --yes sbx delete $src --yes ``` ```text search.py | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) ``` `try2.diff` is a normal `git diff` against the base commit. Apply it in your checkout with `git apply try2.diff`. All three sandboxes returned `404 not_found` afterwards. ## Make the comparison honest - **Freeze the data with the code.** Commit the corpus, fixtures or sample inputs in the base. The 0.44 versus 0.458 gap in step 2 came only from a different Python version. - **Repeat timed runs and take turns.** Five interleaved rounds were enough here because the gap was large. For a change of a few percent, run more rounds and expect some to come out "within noise". - **Compare speed inside one fork.** Each fork boots fresh and can run on a different machine. Compare quality across forks if you like, since it is deterministic. Compare speed only between arms in the same fork. - **Watch the size of the test set.** With 30 queries, one query moves `hit1` by 1 and `mrr` by up to 0.033. The rule rejects any drop, which is right for a fixed set, but a small gain on 30 queries may not hold on real traffic. Grow the set before trusting small wins. - **Hide the labels if the agent could game them.** An agent that can read `queries.json` can tune for those 30 queries. Copy the labels in with `sbx exec` after the attempt is finished. - **Pass a short `--ttl` to every fork.** A fork gets one hour by default, not the source's TTL. A short TTL stops it if your script dies before `sbx delete`. ## FAQ ### Why not compare the agent's numbers with a baseline I measured last week? Last week's number came from a different machine, maybe a different dependency version, and maybe different data. Score the base again next to the candidate, in the same fork, in the same run. It costs one more checkout. ### Does the fork carry over the source's warm cache or running services? No. A fork copies the disk only and boots fresh. Start anything the scorer needs, such as a database, from a script or an enabled systemd unit. Expect the first round in a new fork to be a cold start, and leave it out if your metric is sensitive to that. ### Can I keep the prepared source around for the next change? Yes. Save it as a template with `sbx snapshot search-base-v1` and start later rounds with `sbx new --from search-base-v1`. That works after the source is deleted. Re-save it when the base commit changes. ### How much does one comparison cost? Here, each of the three `small` sandboxes existed for under a minute, counting our pauses between steps. At $0.018 per hour for `small`, that is a fraction of a cent. Your scorer's run time is what sets the real cost. ## Related - [Best-of-N agent attempts with parallel forks](/guides/best-of-n-agents-with-parallel-forks) - [Let a coding agent test its UI change in a sandbox](/guides/let-a-coding-agent-test-its-ui-change-in-a-sandbox) - [Build RL and eval environments with snapshots and forks](/guides/build-rl-and-eval-environments-with-snapshots-and-forks) - [Snapshots, forks and templates](/docs/snapshots)