Skip to content
Guides · Agent cookbook

How Do I Test If an Agent Can Set Up a Repo on a Clean Machine?

Updated November 22, 2026 .md

Short answer: Give each attempt its own fresh sandbox: create it, run the repo's documented install under timeout, run its documented command, check the output, record pass, exit codes and seconds as a JSON line, then delete it. Summarize with jq. On prod, README commands verbatim passed 2 of 4 repos; one retry rule made it 4 of 4.

Part of Coding Agents in the Cloud.

Results at a glance#

Four small public projects, one fresh small sandbox (2 vCPU, 4 GB) per attempt, a 120-second limit on each install. Setup time is the install step only, measured inside the sandbox.

Project README install readme agent, clean base fixer agent, clean base fixer agent, warm snapshot
cowsay (Python) pip install cowsay FAIL, exit 1 (PEP 668), 0.5 s pass, 2.3 s pass, 2.3 s
cowsay (Node) npm install -g cowsay FAIL, exit 243 (EACCES), 1.6 s pass, 3.1 s pass, 2.8 s
shfmt (Go) go install mvdan.cc/sh/v3/cmd/shfmt@latest pass, 8.5 s pass, 7.9 s pass, 0.8 s
ripgrep (Rust) cargo install ripgrep pass, 28.9 s pass, 29.9 s pass, 29.0 s
Pass rate 2/4 4/4 4/4

Last checked: 2026-11-22, on prod

A sandbox was ready 0.25 to 0.29 seconds after sbx new from the base image, and 1.19 to 1.29 seconds from the warm snapshot.

The agent here is a script#

No model ran for this page. Two scripted stand-ins play the agent, so the harness itself is what you see working:

Stand-in What it does
readme Runs the README's install command exactly as written, then its usage command
fixer Same, but if the install fails it retries once: --user --break-system-packages after an externally-managed-environment error, sudo after EACCES

To score a real agent, replace the install step with your agent's run (for example sbx prompt with your own provider key) and keep everything else. The grading doesn't change: the check runs after the agent is done, in the same sandbox.

What the verbatim runs found is typical of "clean machine" failures. The README commands assume a setup the author has. Ubuntu 24.04 refuses a system-wide pip install (PEP 668), and npm install -g can't write to /usr as a normal user. The base image already has Go, Rust (cargo 1.99.1), Python 3.12.3 and Node 22, so the Go and Rust installs worked as written.

Each attempt: limit, run, check#

One attempt is four steps, all inside the sandbox:

  1. Run the install under timeout "$limit". A step that runs out of time exits 124.
  2. If you're testing a retrying agent, let it react to the error.
  3. Run the README's usage command under timeout 30.
  4. Pass only if that command exits 0 and its output contains an expected string.

sbx exec has its own --timeout (default 30 seconds, synchronous limit 600 seconds). Put it before the command and set it above your inner limit, so the sandbox's timeout decides and you get a real exit code. For a setup longer than 600 seconds, start it with sbx exec <id> --detach and poll sbx exec <id> --status <pid>.

Check the thing the setup made, by path. Our first ripgrep check was rg --version, and it passed even when the install timed out, because the base image ships /usr/bin/rg. The harness now runs ~/.cargo/bin/rg --version and expects ripgrep 15 (v15.2.0 installed on 2026-11-22).

Why "clean" matters: two runs in one sandbox#

We ran attempts back to back in one sandbox, aeo-clean-4, with the attempt step from the script below. The columns are install exit code, run exit code, pass, and setup milliseconds:

eval "$(sed -n "/^ATTEMPT='/,/^echo/p" setup-eval.sh)"   # reuse the attempt step from setup-eval.sh
try() { sbx exec "$id" --timeout 300 -- bash -c "$ATTEMPT" _ "$@"; }
id=$(sbx new --name aeo-clean-4 --type small --no-env --ttl 1800 --json | jq -r 'select(.event == "ready") | .id')
echo "sandbox $id   (install_exit run_exit pass setup_ms)"
echo "1. fixer:  $(try 'npm install -g cowsay' 'cowsay JavaScript FTW!' 'JavaScript FTW!' fixer 120)"
echo "2. readme: $(try 'npm install -g cowsay' 'cowsay JavaScript FTW!' 'JavaScript FTW!' readme 120)"
echo "3. readme: $(try 'go install mvdan.cc/sh/v3/cmd/shfmt@latest' 'echo "if true;then echo hi;fi" > a.sh && shfmt a.sh' then readme 120)"
echo "4. readme: $(try 'go install mvdan.cc/sh/v3/cmd/shfmt@latest' 'echo "if true;then echo hi;fi" > a.sh && shfmt a.sh' then readme 120)"
echo "5. readme, 10 s limit: $(try 'cargo install ripgrep' 'rg --version' ripgrep readme 10)"
sbx exec "$id" "ls -d ~/go/pkg/mod/cache/download/mvdan.cc /usr/lib/node_modules/cowsay ~/.cargo/registry/cache 2>&1"
sbx delete "$id" --yes > /dev/null; sbx info "$id" > /dev/null 2>&1; echo "info exit $? (4 = not found)"
sandbox sb_h63vn9x1bi   (install_exit run_exit pass setup_ms)
1. fixer:  0 0 true 2993
2. readme: 243 0 true 400
3. readme: 0 0 true 7329
4. readme: 0 0 true 91
5. readme, 10 s limit: 124 0 true 10008
/home/user/.cargo/registry/cache
/home/user/go/pkg/mod/cache/download/mvdan.cc
/usr/lib/node_modules/cowsay
info exit 4 (4 = not found)

Three ways leftovers lied:

  • Line 2 is a false pass. The verbatim README install failed with exit 243, the same as on a clean machine. The check passed because line 1's sudo install was still in /usr/lib/node_modules.
  • Line 4 is a false time. The second go install took 91 ms instead of 7.3 seconds, because the module and build caches from line 3 were still there.
  • Line 5 is a false pass from the image. cargo install hit the 10-second limit (exit 124), but rg --version found the preinstalled /usr/bin/rg.

A fresh sandbox per attempt removes the first two. Checking by path removes the third. The same rule applies to tool calls and to RL episodes.

Record JSON lines, then a pass-rate table#

Each attempt appends one line to results.jsonl. Keep the sandbox ID and the starting image in every line, so you can tell runs apart later:

{"task":"shfmt","agent":"fixer","from":"aeo-clean-warm","sandbox":"sb_ij0nq5v2q3","install_exit":0,"run_exit":0,"pass":true,"ready_ms":1293,"setup_ms":844,"wall_ms":2266}

jq -s reads every line and groups them by agent and starting point. Over the three runs on 2026-11-22 it printed:

fixer from aeo-clean-warm: 4/4 passed, setup median 2.75 s
fixer from base: 4/4 passed, setup median 7.9 s
readme from base: 2/4 passed, setup median 8.506 s

The median is the upper middle value when the count is even. With more repos, add one line per repo to TASKS and run each agent more than once per repo: a pass rate over 3 attempts per repo tells you more than one attempt each.

Starting from a warm snapshot#

Install time is mostly downloads and compiles. We built a template that keeps the caches but not the tools: install shfmt and ripgrep once, delete both binaries, keep ~/go/pkg/mod (18M), ~/.cache/go-build (55M) and ~/.cargo/registry (36M), then sbx snapshot <id> aeo-clean-warm (2.1 seconds). Each attempt then starts with sbx new --from aeo-clean-warm, which the script does when you set FROM.

Project Setup from base Setup from warm snapshot Change
cowsay (Python) 2.3 s 2.3 s none: pip had nothing cached
cowsay (Node) 3.1 s 2.8 s small
shfmt (Go) 7.9 s 0.8 s the Go build cache skipped the compile
ripgrep (Rust) 29.9 s 29.0 s none: cargo install compiles again

The snapshot added about 1 second to sandbox start (1.2 s against 0.27 s), so it pays off only where a cache skips real work.

It also changes what you measure. From the base image, a pass means "the README works on a stock machine". From a warm snapshot, it means "the README works on a machine that already has these caches". The checks still pass only if the attempt installs the tool, because the binaries were removed. Keep the snapshot name in the from field of each result, and don't compare pass rates or times across different starting points. More on templates in Cache dependencies in a named snapshot.

For repos that need Docker or a per-task image, see SWE-bench-style evaluations on microVMs. To run many attempts side by side instead of one after another, see Run coding agents in parallel.

Copy-paste: score README setup on fresh sandboxes#

After curl -fsSL https://sandboxer.com/install | sh and sbx login, save this as setup-eval.sh and run bash setup-eval.sh. It needs jq and column. It creates one small sandbox at a time with --no-env (none of your account's secrets), runs one attempt in it, deletes it and confirms the delete. We ran it with AGENT=readme NAME=aeo-clean-6. AGENT=fixer switches the stand-in, and FROM=<snapshot> starts each attempt from a named snapshot.

#!/usr/bin/env bash
# setup-eval.sh: can an "agent" install and run each repo from its README on a clean machine?
# One fresh sandbox per attempt. Results go to results.jsonl, then a pass-rate table.
# AGENT=readme runs the README commands verbatim. AGENT=fixer also retries once on two common errors.
# The agent here is a scripted stand-in, not a model.
set -uo pipefail
AGENT=${AGENT:-readme}; LIMIT=${LIMIT:-120}; NAME=${NAME:-setup-eval}; FROM=${FROM:-}
OUT=${OUT:-results.jsonl}; ONLY=${ONLY:-}   # e.g. ONLY=shfmt,ripgrep
id=""; trap '[ -n "$id" ] && sbx delete "$id" --yes > /dev/null 2>&1' EXIT
# task | README install command | README run command | text the output must contain
TASKS='cowsay-py|pip install cowsay|cowsay -t "Hello World"|Hello World
cowsay-js|npm install -g cowsay|cowsay JavaScript FTW!|JavaScript FTW!
shfmt|go install mvdan.cc/sh/v3/cmd/shfmt@latest|echo "if true;then echo hi;fi" > a.sh && shfmt a.sh|then
ripgrep|cargo install ripgrep|~/.cargo/bin/rg --version|ripgrep 15'

# Runs inside the sandbox: install under a time limit, run, check the output.
ATTEMPT='install=$1 run=$2 expect=$3 agent=$4 limit=$5
t0=$(date +%s%N)
timeout "$limit" bash -c "$install" > /tmp/install.log 2>&1; ie=$?
if [ "$ie" -ne 0 ] && [ "$agent" = fixer ]; then
  if grep -q externally-managed /tmp/install.log; then fix="${install/pip install/pip install --user --break-system-packages}"
  elif grep -q EACCES /tmp/install.log; then fix="sudo $install"; else fix=""; fi
  [ -n "$fix" ] && { timeout "$limit" bash -c "$fix" >> /tmp/install.log 2>&1; ie=$?; }
fi
t1=$(date +%s%N)
out=$(timeout 30 bash -c "$run" 2>&1); re=$?
ok=false; [ "$re" -eq 0 ] && grep -qF -- "$expect" <<< "$out" && ok=true
echo "$ie $re $ok $(( (t1 - t0) / 1000000 ))"'

while IFS='|' read -r task install run expect; do
  [ -z "$ONLY" ] || [[ ",$ONLY," == *",$task,"* ]] || continue
  t0=$(date +%s%N)
  id=$(sbx new --name "$NAME" --type small --no-env --ttl 1800 ${FROM:+--from "$FROM"} --json < /dev/null |
       jq -r 'select(.event == "ready") | .id')
  [ -n "$id" ] || { echo "create failed for $task"; break; }
  ready=$(( ($(date +%s%N) - t0) / 1000000 ))
  read -r ie re ok setup_ms < <(sbx exec "$id" --timeout $((LIMIT * 2 + 60)) -- \
    bash -c "$ATTEMPT" _ "$install" "$run" "$expect" "$AGENT" "$LIMIT" < /dev/null)
  jq -nc --arg task "$task" --arg agent "$AGENT" --arg from "${FROM:-base}" --arg id "$id" \
    --argjson ie "${ie:-null}" --argjson re "${re:-null}" --argjson ok "${ok:-false}" \
    --argjson setup "${setup_ms:-null}" --argjson ready "$ready" \
    --argjson wall "$(( ($(date +%s%N) - t0) / 1000000 ))" \
    '{task:$task, agent:$agent, from:$from, sandbox:$id, install_exit:$ie, run_exit:$re,
      pass:$ok, ready_ms:$ready, setup_ms:$setup, wall_ms:$wall}' | tee -a "$OUT"
  sbx delete "$id" --yes < /dev/null > /dev/null
  sbx info "$id" < /dev/null > /dev/null 2>&1; [ $? -eq 4 ] && echo "deleted $id (info: not found)"
  id=""
done <<< "$TASKS"

jq -rs 'group_by(.agent + " " + .from) | .[] |
  "\(.[0].agent) from \(.[0].from): \(map(select(.pass)) | length)/\(length) passed, " +
  "setup median \((map(.setup_ms) | sort | .[length/2|floor]) / 1000) s"' "$OUT"
echo; jq -r '[.task, .agent, .from, (if .pass then "pass" else "FAIL" end),
  "exit \(.install_exit)/\(.run_exit)", "\(.setup_ms / 1000) s"] | @tsv' "$OUT" | column -t -s $'\t'

Output on prod, 2026-11-22 (it took 49 seconds in all):

{"task":"cowsay-py","agent":"readme","from":"base","sandbox":"sb_tg59kcdhpj","install_exit":1,"run_exit":127,"pass":false,"ready_ms":254,"setup_ms":494,"wall_ms":868}
deleted sb_tg59kcdhpj (info: not found)
{"task":"cowsay-js","agent":"readme","from":"base","sandbox":"sb_0fe3loe8cd","install_exit":243,"run_exit":127,"pass":false,"ready_ms":274,"setup_ms":1645,"wall_ms":2041}
deleted sb_0fe3loe8cd (info: not found)
{"task":"shfmt","agent":"readme","from":"base","sandbox":"sb_0svjkl6jxz","install_exit":0,"run_exit":0,"pass":true,"ready_ms":276,"setup_ms":8506,"wall_ms":8905}
deleted sb_0svjkl6jxz (info: not found)
{"task":"ripgrep","agent":"readme","from":"base","sandbox":"sb_932lqbxty4","install_exit":0,"run_exit":0,"pass":true,"ready_ms":267,"setup_ms":28929,"wall_ms":29324}
deleted sb_932lqbxty4 (info: not found)
readme from base: 2/4 passed, setup median 8.506 s

cowsay-py  readme  base  FAIL  exit 1/127    0.494 s
cowsay-js  readme  base  FAIL  exit 243/127  1.645 s
shfmt      readme  base  pass  exit 0/0      8.506 s
ripgrep    readme  base  pass  exit 0/0      28.929 s

The script appends to results.jsonl, so a second run with AGENT=fixer adds its rows to the same table.

FAQ#

Why not reuse one sandbox and clean up between attempts?#

Cleanup misses things you didn't think of. In our run, a reused sandbox turned a failed install into a pass and cut a 7.3-second install to 91 ms. A new sandbox is ready in about 0.27 seconds, so a fresh one per attempt costs less than finding what to delete.

Should the harness pin the repo version?#

Yes, for scores you can compare over time. We ran the README commands verbatim, including @latest, so a rerun next month may install different versions. Pin a commit or version in each task line, and record it in the JSON.

How do I score a real agent instead of the script?#

Replace the install step in ATTEMPT with your agent's run in the same sandbox, and keep the usage command and check as the grader. Run each repo several times, because an agent's result varies from run to run.

Does --no-env matter here?#

It keeps your account's secrets and credentials out of the sandbox, so a README script or an agent can't act on your account. A setup test needs network and the repo, not your keys.

More guides

Top AI sandbox platforms for code execution in 2026, ranked

For running AI-written code we rank Sandboxer, Freestyle and E2B first: each gives code its own VM with an API or SDK and can pause with memory. Pick Daytona for Windows or GPUs, Modal or Northflank for GPUs at scale, E2B or Northflank to run in your own cloud, and Vercel Sandbox if your app lives on Vercel.

Read the guide
Best sandboxes for running coding agents (Claude Code, Codex, Gemini CLI) in 2026, ranked

For coding agents we rank Sandboxer first: a full Ubuntu VM with Claude Code and Codex preinstalled, Docker, and stop-and-resume that keeps the agent's session in memory. Freestyle and Amp Orbs also keep memory across pauses. Pick E2B for prebuilt agent templates on open source, Docker Sandboxes for free local runs, and Codespaces if you live in GitHub.

Read the guide
Best Daytona sandbox alternatives in 2026, ranked

The best Daytona alternatives depend on why you're switching. For a full Linux microVM on every sandbox with memory-saving stop and Docker, we rank Sandboxer first. E2B suits open source and self-hosting, Modal massive short-job fleets, Northflank bring-your-own-cloud with GPUs, and Vercel Sandbox apps on Vercel.

Read the guide
Best E2B alternatives in 2026, ranked

If you're leaving E2B for longer sessions or a fuller machine, we rank Sandboxer first, then Freestyle: both give each sandbox a full Linux VM that keeps memory across pauses, with Docker and no short session cap. Pick Daytona for Windows or GPUs, Northflank for bring-your-own-cloud, and stay on E2B if you need open source.

Read the guide