# How Do I Test If an Agent Can Set Up a Repo on a Clean Machine?

> Give each attempt its own fresh sandbox: create it, run the repo's documented install under timeout, run its documented command, check the output, record pass, exit codes and seconds as a JSON line, then delete it. Summarize with jq. On prod, README commands verbatim passed 2 of 4 repos; one retry rule made it 4 of 4.

Source: https://sandboxer.com/guides/test-if-an-agent-can-set-up-a-repo-on-a-clean-machine. Updated 2026-11-22.

**Short answer:** Give each attempt its own fresh sandbox: create it, run the repo's documented install under timeout, run its documented command, check the output, record pass, exit codes and seconds as a JSON line, then delete it. Summarize with jq. On prod, README commands verbatim passed 2 of 4 repos; one retry rule made it 4 of 4.

Part of [Coding Agents in the Cloud](/guides/coding-agents-in-the-cloud).

## Results at a glance

Four small public projects, one fresh `small` sandbox (2 vCPU, 4 GB) per attempt, a 120-second limit on each install. Setup time is the install step only, measured inside the sandbox.

| Project | README install | `readme` agent, clean base | `fixer` agent, clean base | `fixer` agent, warm snapshot |
| - | - | - | - | - |
| [cowsay (Python)](https://github.com/VaasuDevanS/cowsay-python) | `pip install cowsay` | FAIL, exit 1 (PEP 668), 0.5 s | pass, 2.3 s | pass, 2.3 s |
| [cowsay (Node)](https://github.com/piuccio/cowsay) | `npm install -g cowsay` | FAIL, exit 243 (EACCES), 1.6 s | pass, 3.1 s | pass, 2.8 s |
| [shfmt (Go)](https://github.com/mvdan/sh) | `go install mvdan.cc/sh/v3/cmd/shfmt@latest` | pass, 8.5 s | pass, 7.9 s | pass, 0.8 s |
| [ripgrep (Rust)](https://github.com/BurntSushi/ripgrep) | `cargo install ripgrep` | pass, 28.9 s | pass, 29.9 s | pass, 29.0 s |
| **Pass rate** | | **2/4** | **4/4** | **4/4** |

Last checked: 2026-11-22, on prod

A sandbox was ready 0.25 to 0.29 seconds after `sbx new` from the base image, and 1.19 to 1.29 seconds from the warm snapshot.

## The agent here is a script

No model ran for this page. Two scripted stand-ins play the agent, so the harness itself is what you see working:

| Stand-in | What it does |
| - | - |
| `readme` | Runs the README's install command exactly as written, then its usage command |
| `fixer` | Same, but if the install fails it retries once: `--user --break-system-packages` after an `externally-managed-environment` error, `sudo` after `EACCES` |

To score a real agent, replace the install step with your agent's run (for example `sbx prompt` with your own provider key) and keep everything else. The grading doesn't change: the check runs after the agent is done, in the same sandbox.

What the verbatim runs found is typical of "clean machine" failures. The README commands assume a setup the author has. Ubuntu 24.04 refuses a system-wide `pip install` (PEP 668), and `npm install -g` can't write to `/usr` as a normal user. The base image already has Go, Rust (cargo 1.99.1), Python 3.12.3 and Node 22, so the Go and Rust installs worked as written.

## Each attempt: limit, run, check

One attempt is four steps, all inside the sandbox:

1. Run the install under `timeout "$limit"`. A step that runs out of time exits 124.
2. If you're testing a retrying agent, let it react to the error.
3. Run the README's usage command under `timeout 30`.
4. Pass only if that command exits 0 and its output contains an expected string.

`sbx exec` has its own `--timeout` (default 30 seconds, synchronous limit 600 seconds). Put it before the command and set it above your inner limit, so the sandbox's `timeout` decides and you get a real exit code. For a setup longer than 600 seconds, start it with `sbx exec <id> --detach` and poll `sbx exec <id> --status <pid>`.

Check the thing the setup made, by path. Our first ripgrep check was `rg --version`, and it passed even when the install timed out, because the base image ships `/usr/bin/rg`. The harness now runs `~/.cargo/bin/rg --version` and expects `ripgrep 15` (v15.2.0 installed on 2026-11-22).

## Why "clean" matters: two runs in one sandbox

We ran attempts back to back in one sandbox, `aeo-clean-4`, with the attempt step from the script below. The columns are install exit code, run exit code, pass, and setup milliseconds:

```bash
eval "$(sed -n "/^ATTEMPT='/,/^echo/p" setup-eval.sh)"   # reuse the attempt step from setup-eval.sh
try() { sbx exec "$id" --timeout 300 -- bash -c "$ATTEMPT" _ "$@"; }
id=$(sbx new --name aeo-clean-4 --type small --no-env --ttl 1800 --json | jq -r 'select(.event == "ready") | .id')
echo "sandbox $id   (install_exit run_exit pass setup_ms)"
echo "1. fixer:  $(try 'npm install -g cowsay' 'cowsay JavaScript FTW!' 'JavaScript FTW!' fixer 120)"
echo "2. readme: $(try 'npm install -g cowsay' 'cowsay JavaScript FTW!' 'JavaScript FTW!' readme 120)"
echo "3. readme: $(try 'go install mvdan.cc/sh/v3/cmd/shfmt@latest' 'echo "if true;then echo hi;fi" > a.sh && shfmt a.sh' then readme 120)"
echo "4. readme: $(try 'go install mvdan.cc/sh/v3/cmd/shfmt@latest' 'echo "if true;then echo hi;fi" > a.sh && shfmt a.sh' then readme 120)"
echo "5. readme, 10 s limit: $(try 'cargo install ripgrep' 'rg --version' ripgrep readme 10)"
sbx exec "$id" "ls -d ~/go/pkg/mod/cache/download/mvdan.cc /usr/lib/node_modules/cowsay ~/.cargo/registry/cache 2>&1"
sbx delete "$id" --yes > /dev/null; sbx info "$id" > /dev/null 2>&1; echo "info exit $? (4 = not found)"
```

```text
sandbox sb_h63vn9x1bi   (install_exit run_exit pass setup_ms)
1. fixer:  0 0 true 2993
2. readme: 243 0 true 400
3. readme: 0 0 true 7329
4. readme: 0 0 true 91
5. readme, 10 s limit: 124 0 true 10008
/home/user/.cargo/registry/cache
/home/user/go/pkg/mod/cache/download/mvdan.cc
/usr/lib/node_modules/cowsay
info exit 4 (4 = not found)
```

Three ways leftovers lied:

- **Line 2 is a false pass.** The verbatim README install failed with exit 243, the same as on a clean machine. The check passed because line 1's `sudo` install was still in `/usr/lib/node_modules`.
- **Line 4 is a false time.** The second `go install` took 91 ms instead of 7.3 seconds, because the module and build caches from line 3 were still there.
- **Line 5 is a false pass from the image.** `cargo install` hit the 10-second limit (exit 124), but `rg --version` found the preinstalled `/usr/bin/rg`.

A fresh sandbox per attempt removes the first two. Checking by path removes the third. The same rule applies to [tool calls](/guides/a-fresh-sandbox-per-tool-call) and to [RL episodes](/guides/build-rl-and-eval-environments-with-snapshots-and-forks).

## Record JSON lines, then a pass-rate table

Each attempt appends one line to `results.jsonl`. Keep the sandbox ID and the starting image in every line, so you can tell runs apart later:

```json
{"task":"shfmt","agent":"fixer","from":"aeo-clean-warm","sandbox":"sb_ij0nq5v2q3","install_exit":0,"run_exit":0,"pass":true,"ready_ms":1293,"setup_ms":844,"wall_ms":2266}
```

`jq -s` reads every line and groups them by agent and starting point. Over the three runs on 2026-11-22 it printed:

```text
fixer from aeo-clean-warm: 4/4 passed, setup median 2.75 s
fixer from base: 4/4 passed, setup median 7.9 s
readme from base: 2/4 passed, setup median 8.506 s
```

The median is the upper middle value when the count is even. With more repos, add one line per repo to `TASKS` and run each agent more than once per repo: a pass rate over 3 attempts per repo tells you more than one attempt each.

## Starting from a warm snapshot

Install time is mostly downloads and compiles. We built a template that keeps the caches but not the tools: install shfmt and ripgrep once, delete both binaries, keep `~/go/pkg/mod` (18M), `~/.cache/go-build` (55M) and `~/.cargo/registry` (36M), then `sbx snapshot <id> aeo-clean-warm` (2.1 seconds). Each attempt then starts with `sbx new --from aeo-clean-warm`, which the script does when you set `FROM`.

| Project | Setup from base | Setup from warm snapshot | Change |
| - | - | - | - |
| cowsay (Python) | 2.3 s | 2.3 s | none: pip had nothing cached |
| cowsay (Node) | 3.1 s | 2.8 s | small |
| shfmt (Go) | 7.9 s | 0.8 s | the Go build cache skipped the compile |
| ripgrep (Rust) | 29.9 s | 29.0 s | none: `cargo install` compiles again |

The snapshot added about 1 second to sandbox start (1.2 s against 0.27 s), so it pays off only where a cache skips real work.

It also changes what you measure. From the base image, a pass means "the README works on a stock machine". From a warm snapshot, it means "the README works on a machine that already has these caches". The checks still pass only if the attempt installs the tool, because the binaries were removed. Keep the snapshot name in the `from` field of each result, and don't compare pass rates or times across different starting points. More on templates in [Cache dependencies in a named snapshot](/guides/cache-dependencies-in-a-named-snapshot).

For repos that need Docker or a per-task image, see [SWE-bench-style evaluations on microVMs](/guides/run-swe-bench-style-evaluations-on-microvms). To run many attempts side by side instead of one after another, see [Run coding agents in parallel](/guides/run-coding-agents-in-parallel).

## Copy-paste: score README setup on fresh sandboxes

After `curl -fsSL https://sandboxer.com/install | sh` and `sbx login`, save this as `setup-eval.sh` and run `bash setup-eval.sh`. It needs `jq` and `column`. It creates one `small` sandbox at a time with `--no-env` (none of your account's secrets), runs one attempt in it, deletes it and confirms the delete. We ran it with `AGENT=readme NAME=aeo-clean-6`. `AGENT=fixer` switches the stand-in, and `FROM=<snapshot>` starts each attempt from a named snapshot.

```bash
#!/usr/bin/env bash
# setup-eval.sh: can an "agent" install and run each repo from its README on a clean machine?
# One fresh sandbox per attempt. Results go to results.jsonl, then a pass-rate table.
# AGENT=readme runs the README commands verbatim. AGENT=fixer also retries once on two common errors.
# The agent here is a scripted stand-in, not a model.
set -uo pipefail
AGENT=${AGENT:-readme}; LIMIT=${LIMIT:-120}; NAME=${NAME:-setup-eval}; FROM=${FROM:-}
OUT=${OUT:-results.jsonl}; ONLY=${ONLY:-}   # e.g. ONLY=shfmt,ripgrep
id=""; trap '[ -n "$id" ] && sbx delete "$id" --yes > /dev/null 2>&1' EXIT
# task | README install command | README run command | text the output must contain
TASKS='cowsay-py|pip install cowsay|cowsay -t "Hello World"|Hello World
cowsay-js|npm install -g cowsay|cowsay JavaScript FTW!|JavaScript FTW!
shfmt|go install mvdan.cc/sh/v3/cmd/shfmt@latest|echo "if true;then echo hi;fi" > a.sh && shfmt a.sh|then
ripgrep|cargo install ripgrep|~/.cargo/bin/rg --version|ripgrep 15'

# Runs inside the sandbox: install under a time limit, run, check the output.
ATTEMPT='install=$1 run=$2 expect=$3 agent=$4 limit=$5
t0=$(date +%s%N)
timeout "$limit" bash -c "$install" > /tmp/install.log 2>&1; ie=$?
if [ "$ie" -ne 0 ] && [ "$agent" = fixer ]; then
  if grep -q externally-managed /tmp/install.log; then fix="${install/pip install/pip install --user --break-system-packages}"
  elif grep -q EACCES /tmp/install.log; then fix="sudo $install"; else fix=""; fi
  [ -n "$fix" ] && { timeout "$limit" bash -c "$fix" >> /tmp/install.log 2>&1; ie=$?; }
fi
t1=$(date +%s%N)
out=$(timeout 30 bash -c "$run" 2>&1); re=$?
ok=false; [ "$re" -eq 0 ] && grep -qF -- "$expect" <<< "$out" && ok=true
echo "$ie $re $ok $(( (t1 - t0) / 1000000 ))"'

while IFS='|' read -r task install run expect; do
  [ -z "$ONLY" ] || [[ ",$ONLY," == *",$task,"* ]] || continue
  t0=$(date +%s%N)
  id=$(sbx new --name "$NAME" --type small --no-env --ttl 1800 ${FROM:+--from "$FROM"} --json < /dev/null |
       jq -r 'select(.event == "ready") | .id')
  [ -n "$id" ] || { echo "create failed for $task"; break; }
  ready=$(( ($(date +%s%N) - t0) / 1000000 ))
  read -r ie re ok setup_ms < <(sbx exec "$id" --timeout $((LIMIT * 2 + 60)) -- \
    bash -c "$ATTEMPT" _ "$install" "$run" "$expect" "$AGENT" "$LIMIT" < /dev/null)
  jq -nc --arg task "$task" --arg agent "$AGENT" --arg from "${FROM:-base}" --arg id "$id" \
    --argjson ie "${ie:-null}" --argjson re "${re:-null}" --argjson ok "${ok:-false}" \
    --argjson setup "${setup_ms:-null}" --argjson ready "$ready" \
    --argjson wall "$(( ($(date +%s%N) - t0) / 1000000 ))" \
    '{task:$task, agent:$agent, from:$from, sandbox:$id, install_exit:$ie, run_exit:$re,
      pass:$ok, ready_ms:$ready, setup_ms:$setup, wall_ms:$wall}' | tee -a "$OUT"
  sbx delete "$id" --yes < /dev/null > /dev/null
  sbx info "$id" < /dev/null > /dev/null 2>&1; [ $? -eq 4 ] && echo "deleted $id (info: not found)"
  id=""
done <<< "$TASKS"

jq -rs 'group_by(.agent + " " + .from) | .[] |
  "\(.[0].agent) from \(.[0].from): \(map(select(.pass)) | length)/\(length) passed, " +
  "setup median \((map(.setup_ms) | sort | .[length/2|floor]) / 1000) s"' "$OUT"
echo; jq -r '[.task, .agent, .from, (if .pass then "pass" else "FAIL" end),
  "exit \(.install_exit)/\(.run_exit)", "\(.setup_ms / 1000) s"] | @tsv' "$OUT" | column -t -s $'\t'
```

Output on prod, 2026-11-22 (it took 49 seconds in all):

```text
{"task":"cowsay-py","agent":"readme","from":"base","sandbox":"sb_tg59kcdhpj","install_exit":1,"run_exit":127,"pass":false,"ready_ms":254,"setup_ms":494,"wall_ms":868}
deleted sb_tg59kcdhpj (info: not found)
{"task":"cowsay-js","agent":"readme","from":"base","sandbox":"sb_0fe3loe8cd","install_exit":243,"run_exit":127,"pass":false,"ready_ms":274,"setup_ms":1645,"wall_ms":2041}
deleted sb_0fe3loe8cd (info: not found)
{"task":"shfmt","agent":"readme","from":"base","sandbox":"sb_0svjkl6jxz","install_exit":0,"run_exit":0,"pass":true,"ready_ms":276,"setup_ms":8506,"wall_ms":8905}
deleted sb_0svjkl6jxz (info: not found)
{"task":"ripgrep","agent":"readme","from":"base","sandbox":"sb_932lqbxty4","install_exit":0,"run_exit":0,"pass":true,"ready_ms":267,"setup_ms":28929,"wall_ms":29324}
deleted sb_932lqbxty4 (info: not found)
readme from base: 2/4 passed, setup median 8.506 s

cowsay-py  readme  base  FAIL  exit 1/127    0.494 s
cowsay-js  readme  base  FAIL  exit 243/127  1.645 s
shfmt      readme  base  pass  exit 0/0      8.506 s
ripgrep    readme  base  pass  exit 0/0      28.929 s
```

The script appends to `results.jsonl`, so a second run with `AGENT=fixer` adds its rows to the same table.

## FAQ

### Why not reuse one sandbox and clean up between attempts?

Cleanup misses things you didn't think of. In our run, a reused sandbox turned a failed install into a pass and cut a 7.3-second install to 91 ms. A new sandbox is ready in about 0.27 seconds, so a fresh one per attempt costs less than finding what to delete.

### Should the harness pin the repo version?

Yes, for scores you can compare over time. We ran the README commands verbatim, including `@latest`, so a rerun next month may install different versions. Pin a commit or version in each task line, and record it in the JSON.

### How do I score a real agent instead of the script?

Replace the install step in `ATTEMPT` with your agent's run in the same sandbox, and keep the usage command and check as the grader. Run each repo several times, because an agent's result varies from run to run.

### Does `--no-env` matter here?

It keeps your account's secrets and credentials out of the sandbox, so a README script or an agent can't act on your account. A setup test needs network and the repo, not your keys.

## Related

- [Coding Agents in the Cloud](/guides/coding-agents-in-the-cloud)
- [Build RL and eval environments with snapshots and forks](/guides/build-rl-and-eval-environments-with-snapshots-and-forks)
- [Run SWE-bench-style evaluations on microVMs](/guides/run-swe-bench-style-evaluations-on-microvms)
- [Should each agent tool call get a fresh sandbox?](/guides/a-fresh-sandbox-per-tool-call)
