# How Do I Run SWE-bench-Style Evaluations on MicroVMs? > Run the official SWE-bench harness inside a Sandboxer sandbox. Each sandbox is a microVM with its own Docker daemon, so the harness pulls the per-task image, applies the candidate patch, runs the task's tests and grades FAIL_TO_PASS and PASS_TO_PASS. Save one prepared sandbox as a snapshot and start one VM per task. Source: https://sandboxer.com/guides/run-swe-bench-style-evaluations-on-microvms. Updated 2026-10-27. **Short answer:** Run the official SWE-bench harness inside a Sandboxer sandbox. Each sandbox is a microVM with its own Docker daemon, so the harness pulls the per-task image, applies the candidate patch, runs the task's tests and grades FAIL_TO_PASS and PASS_TO_PASS. Save one prepared sandbox as a snapshot and start one VM per task. ## What a SWE-bench task is A [SWE-bench](https://www.swebench.com/) task is a real GitHub issue with the repository frozen at the commit before the fix. The [dataset card](https://huggingface.co/datasets/SWE-bench/SWE-bench_Lite) defines the fields that grading uses: | Field | What it holds | | - | - | | `base_commit` | The repository state before the fix | | `patch` | The gold patch: the fix the real pull request made, without its test changes | | `test_patch` | The tests that pull request added | | `FAIL_TO_PASS` | Tests tied to the issue: they fail before the fix and must pass after it | | `PASS_TO_PASS` | Tests that pass before the fix and must still pass after it | A candidate patch, usually from a model, **resolves** the task only if every `FAIL_TO_PASS` test passes and no `PASS_TO_PASS` test breaks. The [evaluation guide](https://www.swebench.com/SWE-bench/guides/evaluation/) describes the harness that checks this. It runs each task in its own Docker container, so the machine running it needs a working Docker daemon. That is why a microVM fits. A Sandboxer sandbox has Docker preinstalled and its own kernel, so the harness runs unchanged, and nothing a patch or test does reaches your machine or another task. ## What we ran We used one real SWE-bench Lite task, `sympy__sympy-20590`. Its `test_patch` adds `test_immutable` to `sympy/core/tests/test_basic.py`, which checks that a SymPy `Basic` object has no `__dict__` and refuses new attributes. That is the task's one `FAIL_TO_PASS` test. The 21 other tests in that file are its `PASS_TO_PASS` tests. We didn't call a model for this page. We graded three candidate patches instead: | Candidate | Where it came from | | - | - | | `gold` | The dataset's own fix (`-p gold`): it adds `__slots__ = ()` to the `Printable` mixin | | `handmade` | Written by hand: a `__setattr__` guard on `Printable` that refuses attributes not declared in any `__slots__` | | `noop` | Written by hand: adds one comment line and changes nothing | In a real run, the candidate is your agent's diff. Everything else ran on prod as shown. Every time on this page is a single run on 2026-10-27, not a benchmark. ## 1. Install the harness in a sandbox Use `large` (8 vCPU, 16 GB, 125 GB disk; the free trial runs only `small` and `default`). Task images are a few GB each, and the harness runs several containers at once. ```bash sbx new --name aeo-swe-1 --type large --ttl 7200 id=sb_2y6ove8cl5 # the id sbx new printed sbx exec $id "python3 --version; docker --version; df -h / | tail -1" sbx exec $id --timeout 600 "python3 -m venv ~/swe && ~/swe/bin/pip install -q swebench && ~/swe/bin/pip show swebench | head -2" ``` ```text Python 3.12.3 Docker version 29.9.0, build f415da8 /dev/root 130G 5.5G 125G 5% / Name: swebench Version: 5.0.2 ``` The sandbox was ready 3.2 s after `sbx new`, and the install took 41 s. ## 2. Grade the gold patch The harness can run for minutes on bigger tasks, so start it detached and read the log: ```bash sbx exec $id --detach "mkdir -p ~/eval && cd ~/eval && s=\$(date +%s); \ ~/swe/bin/python -m swebench.harness.run_evaluation \ --dataset_name SWE-bench/SWE-bench_Lite --instance_ids sympy__sympy-20590 \ --predictions_path gold --max_workers 1 --run_id gold1 > run1.log 2>&1; \ echo \"exit=\$? seconds=\$((\$(date +%s)-s))\" >> run1.log" sbx exec $id "tail -c 1500 ~/eval/run1.log" ``` ```text … All instances run. Total instances: 1 Instances submitted: 1 Instances completed: 1 Instances incomplete: 0 Instances resolved: 1 Instances unresolved: 0 … Report written to gold.gold1.json exit=0 seconds=66 ``` The per-task report shows the grading: ```bash sbx exec $id "tail -20 ~/eval/logs/run_evaluation/gold1/gold/sympy__sympy-20590/run_instance.log" ``` ```text … 2026-10-27 02:35:21,569 - INFO - Test runtime: 3.44 seconds … 2026-10-27 02:35:21,618 - INFO - report: {'sympy__sympy-20590': {'patch_is_None': False, 'patch_exists': True, 'patch_successfully_applied': True, 'resolved': True, … 'tests_status': {'FAIL_TO_PASS': {'success': ['test_immutable'], 'failure': []}, 'PASS_TO_PASS': {'success': ['test__aresame', 'test_structure', 'test_equality', …], 'failure': []}, …}}} Result for sympy__sympy-20590: resolved: True ``` `docker images` inside the sandbox then listed `swebench/sweb.eval.x86_64.sympy_1776_sympy-20590`, 2.77 GB. The harness pulled a prebuilt image for the task instead of building one, which took 38 s of the 66. ## 3. Grade your own patches A predictions file is a JSON list with one entry per task. To get a diff that applies cleanly, make the edit inside the task's own image (the repository is at `/testbed`) and print `git diff`: ```bash sbx exec $id "cd ~/eval && docker run --rm swebench/sweb.eval.x86_64.sympy_1776_sympy-20590 bash -c \"cd /testbed && sed -i \\\"s/^class Printable:/# Printing mixin for Basic and friends.\\nclass Printable:/\\\" sympy/core/_print_helpers.py && git diff\" > noop.diff && cat noop.diff" ``` ```diff diff --git a/sympy/core/_print_helpers.py b/sympy/core/_print_helpers.py index 96d7e72676..69c96086f9 100755 --- a/sympy/core/_print_helpers.py +++ b/sympy/core/_print_helpers.py @@ -5,6 +5,7 @@ but cannot be defined there without causing circular imports. """ +# Printing mixin for Basic and friends. class Printable: """ The default implementation of printing for SymPy classes. ``` Wrap the diff in a predictions file and grade it: ```bash sbx exec $id "cd ~/eval && python3 -c \"import json; json.dump([{\\\"instance_id\\\": \\\"sympy__sympy-20590\\\", \\\"model_name_or_path\\\": \\\"noop\\\", \\\"model_patch\\\": open(\\\"noop.diff\\\").read()}], open(\\\"noop.json\\\", \\\"w\\\"))\"" sbx exec $id --timeout 600 "cd ~/eval && ~/swe/bin/python -m swebench.harness.run_evaluation \ -d SWE-bench/SWE-bench_Lite -i sympy__sympy-20590 -p noop.json --max_workers 1 --run_id noop1 \ 2>&1 | grep -E '^Instances (resolved|unresolved)'" ``` ```text Instances resolved: 0 Instances unresolved: 1 ``` Reading `report.json` for each candidate gives the whole picture: | Candidate | `FAIL_TO_PASS` | `PASS_TO_PASS` | Resolved | Task time | | - | - | - | - | - | | `gold` (cold, image pulled) | 1 of 1 passed | 21 of 21 passed | Yes | 66 s | | `gold` (image cached) | 1 of 1 passed | 21 of 21 passed | Yes | 27 s | | `noop` | `test_immutable` failed | 21 of 21 passed | No | 27 s | | `handmade` | No tests ran | No tests ran | No | 23 s | The `noop` row is the case `FAIL_TO_PASS` exists for: nothing broke, but the issue wasn't fixed. The `handmade` patch applied cleanly (`Applied patch sympy/core/_print_helpers.py cleanly.` in the log), but SymPy itself sets plain attributes on its objects, so `import sympy` failed before any test ran: ```text File "/testbed/sympy/assumptions/assume.py", line 154, in __new__ obj.name = name File "/testbed/sympy/core/_print_helpers.py", line 23, in __setattr__ raise AttributeError(name) AttributeError: name ``` With no test results to parse, the report says `"patch_successfully_applied": false` and has no `tests_status`. Check `test_output.txt` before you blame the diff format. ## Where the time goes From `run_instance.log` for the cached `gold` run (27.2 s from the client): | Step | Time | | - | - | | Harness start and dataset load | About 6 s | | Create the container and apply the patch | 0.5 s | | Run the tests (`bin/test sympy/core/tests/test_basic.py`) | 5.1 s | | Stop and remove the container | 15.2 s | More than half of a cached run is the container shutting down. With `--max_workers` above 1, those waits overlap, so run several tasks per harness call when you batch them in one VM. The harness's own help says to keep `--max_workers` at or below 75% of CPU cores, which is 6 on `large`. ## 4. Fan out: one task per VM Once a sandbox has the harness installed, save it as a named snapshot. Every task then starts from the same bytes, already set up: ```bash sbx snapshot $id aeo-swe-ready sbx delete $id --yes ``` This script runs two tasks at once, each in its own sandbox, and deletes each one as soon as its verdict is read: ```bash #!/bin/bash snap=aeo-swe-ready inst=sympy__sympy-20590 run_task() { # $1 = run id, $2 = predictions (gold or a file in ~/eval), $3 = sandbox name t0=$(date +%s.%N) id=$(sbx new --from "$snap" --name "$3" --type large --ttl 1800 --json | jq -r 'select(.event=="ready") | .id') t1=$(date +%s.%N) sbx exec "$id" --timeout 600 "cd ~/eval && ~/swe/bin/python -m swebench.harness.run_evaluation \ -d SWE-bench/SWE-bench_Lite -i $inst -p $2 --max_workers 1 --run_id $1 > run.log 2>&1" t2=$(date +%s.%N) verdict=$(sbx exec "$id" "python3 -c \"import json,glob; r=json.load(open(glob.glob('eval/logs/run_evaluation/$1/*/$inst/report.json')[0]))['$inst']; t=r['tests_status']; print(json.dumps({'resolved': r['resolved'], 'f2p_failed': t['FAIL_TO_PASS']['failure'], 'p2p_failed': t['PASS_TO_PASS']['failure']}))\"") sbx delete "$id" --yes > /dev/null printf '%s %s start=%.1fs eval=%.1fs %s\n' "$1" "$id" "$(echo "$t1-$t0" | bc)" "$(echo "$t2-$t1" | bc)" "$verdict" } run_task gold gold aeo-swe-2 & run_task noop noop.json aeo-swe-3 & wait ``` The output, from prod: ```text noop sb_hg1olybk8t start=3.2s eval=27.8s {"resolved": false, "f2p_failed": ["test_immutable"], "p2p_failed": []} gold sb_5httmzzxf1 start=3.2s eval=28.3s {"resolved": true, "f2p_failed": [], "p2p_failed": []} ``` Both tasks finished in 33 s of wall time, against about 62 s back to back. Each VM was ready 3.2 s after `sbx new --from`, and neither pulled the image, because the 2.77 GB image was already on the snapshot's disk. In a real run, `$inst` changes per task and the predictions file holds your agent's patch for it. ## Scale it | | One task per VM | Many tasks per VM | | - | - | - | | Isolation | Full: one kernel and one Docker daemon per task | Tasks share one VM, each in its own container | | Starts used | One per task | One per batch | | Disk | One task's image | Every image in the batch, so watch the 125 GB on `large` | | Use it for | Untrusted agent patches, RL rollouts, reruns of one task | Large sweeps on a tight start budget | - **Starts are rate limited.** Creates, resumes and forks each count as one start. The $20 plan allows 12 a minute, 60 an hour and 200 a day, and 100 sandboxes at once. The $100 plan allows 30, 210 and 840, and 300 at once ([billing](/docs/billing#plans)). The 300-task SWE-bench Lite test split, one VM per task, is 300 starts. That is more than the $20 plan's 200 a day, so batch several tasks per VM there. On the $100 plan, 210 starts an hour gets through it in under 2 hours. Check what's left before a burst with `sbx limits --json | jq '.starts'`. A `429 rate_limited` reply carries `Retry-After` and `details.resetsAt`. - **Cost is per second.** `large` costs $0.072/hour, so a one-minute task costs about $0.0012 of machine time. `xlarge` (16 vCPU, 32 GB) needs the $100 plan or higher. - **Pre-pull images into the snapshot.** A cold task here spent 38 s pulling its image, and VMs started from the snapshot skipped that. `docker pull` the images for a batch before `sbx snapshot`. Images count toward the snapshot's disk, so group tasks by repository rather than putting every image in one template. - **Always clean up.** Delete each sandbox after reading its report, and set a short `--ttl` as a backstop. When the sweep is done, run `sbx snapshot rm aeo-swe-ready`. ## FAQ ### Do I need to build the SWE-bench Docker images myself? Not for this task. With swebench 5.0.2, the harness pulled a prebuilt image for `sympy__sympy-20590` from Docker Hub (2.77 GB, 38 s) instead of building it. Building images yourself takes much longer and more disk. ### Why does a task need Docker inside the sandbox? The official harness runs every task in its own container built for that repository and commit. A Sandboxer sandbox is a full VM with its own Docker daemon, so the harness runs as it would on a Linux server, with no host Docker socket shared. ### Which sandbox size should I use? `large` (8 vCPU, 16 GB, 125 GB disk) ran this task with room to spare. Use `default` (4 vCPU, 8 GB, 50 GB) for one small task at a time, and `xlarge` for many workers in one VM. ### How do I grade a model's patch instead of the gold patch? Write a predictions JSON list of `instance_id`, `model_name_or_path` and `model_patch` (a unified diff against `base_commit`), and pass it with `-p`. The harness applies the diff, runs the tests and writes `report.json`. ### Can one task affect another? Not with one task per VM. Each sandbox has its own kernel, disk and Docker daemon, and each starts from the same snapshot, so nothing one task writes reaches the next. ## Related - [Build RL and eval environments with snapshots and forks](/guides/build-rl-and-eval-environments-with-snapshots-and-forks) - [Docker and nested KVM inside an AI agent sandbox](/guides/docker-and-kvm-inside-an-agent-sandbox) - [Snapshots, forks and named templates](/docs/snapshots) - [Limits and errors](/docs/limits)