Skip to content
sandboxer
Start free trial
Guides · Use cases

How Do I Run SWE-bench-Style Evaluations on MicroVMs?

Updated October 27, 2026 .md

Short answer: Run the official SWE-bench harness inside a Sandboxer sandbox. Each sandbox is a microVM with its own Docker daemon, so the harness pulls the per-task image, applies the candidate patch, runs the task's tests and grades FAIL_TO_PASS and PASS_TO_PASS. Save one prepared sandbox as a snapshot and start one VM per task.

What a SWE-bench task is#

A SWE-bench task is a real GitHub issue with the repository frozen at the commit before the fix. The dataset card defines the fields that grading uses:

Field What it holds
base_commit The repository state before the fix
patch The gold patch: the fix the real pull request made, without its test changes
test_patch The tests that pull request added
FAIL_TO_PASS Tests tied to the issue: they fail before the fix and must pass after it
PASS_TO_PASS Tests that pass before the fix and must still pass after it

A candidate patch, usually from a model, resolves the task only if every FAIL_TO_PASS test passes and no PASS_TO_PASS test breaks. The evaluation guide describes the harness that checks this. It runs each task in its own Docker container, so the machine running it needs a working Docker daemon.

That is why a microVM fits. A Sandboxer sandbox has Docker preinstalled and its own kernel, so the harness runs unchanged, and nothing a patch or test does reaches your machine or another task.

What we ran#

We used one real SWE-bench Lite task, sympy__sympy-20590. Its test_patch adds test_immutable to sympy/core/tests/test_basic.py, which checks that a SymPy Basic object has no __dict__ and refuses new attributes. That is the task's one FAIL_TO_PASS test. The 21 other tests in that file are its PASS_TO_PASS tests.

We didn't call a model for this page. We graded three candidate patches instead:

Candidate Where it came from
gold The dataset's own fix (-p gold): it adds __slots__ = () to the Printable mixin
handmade Written by hand: a __setattr__ guard on Printable that refuses attributes not declared in any __slots__
noop Written by hand: adds one comment line and changes nothing

In a real run, the candidate is your agent's diff. Everything else ran on prod as shown. Every time on this page is a single run on 2026-10-27, not a benchmark.

1. Install the harness in a sandbox#

Use large (8 vCPU, 16 GB, 125 GB disk; the free trial runs only small and default). Task images are a few GB each, and the harness runs several containers at once.

sbx new --name aeo-swe-1 --type large --ttl 7200
id=sb_2y6ove8cl5      # the id sbx new printed

sbx exec $id "python3 --version; docker --version; df -h / | tail -1"
sbx exec $id --timeout 600 "python3 -m venv ~/swe && ~/swe/bin/pip install -q swebench && ~/swe/bin/pip show swebench | head -2"
Python 3.12.3
Docker version 29.9.0, build f415da8
/dev/root       130G  5.5G  125G   5% /
Name: swebench
Version: 5.0.2

The sandbox was ready 3.2 s after sbx new, and the install took 41 s.

2. Grade the gold patch#

The harness can run for minutes on bigger tasks, so start it detached and read the log:

sbx exec $id --detach "mkdir -p ~/eval && cd ~/eval && s=\$(date +%s); \
  ~/swe/bin/python -m swebench.harness.run_evaluation \
    --dataset_name SWE-bench/SWE-bench_Lite --instance_ids sympy__sympy-20590 \
    --predictions_path gold --max_workers 1 --run_id gold1 > run1.log 2>&1; \
  echo \"exit=\$? seconds=\$((\$(date +%s)-s))\" >> run1.log"
sbx exec $id "tail -c 1500 ~/eval/run1.log"
…
All instances run.
Total instances: 1
Instances submitted: 1
Instances completed: 1
Instances incomplete: 0
Instances resolved: 1
Instances unresolved: 0
…
Report written to gold.gold1.json
exit=0 seconds=66

The per-task report shows the grading:

sbx exec $id "tail -20 ~/eval/logs/run_evaluation/gold1/gold/sympy__sympy-20590/run_instance.log"
…
2026-10-27 02:35:21,569 - INFO - Test runtime: 3.44 seconds
…
2026-10-27 02:35:21,618 - INFO - report: {'sympy__sympy-20590': {'patch_is_None': False, 'patch_exists': True, 'patch_successfully_applied': True, 'resolved': True, … 'tests_status': {'FAIL_TO_PASS': {'success': ['test_immutable'], 'failure': []}, 'PASS_TO_PASS': {'success': ['test__aresame', 'test_structure', 'test_equality', …], 'failure': []}, …}}}
Result for sympy__sympy-20590: resolved: True

docker images inside the sandbox then listed swebench/sweb.eval.x86_64.sympy_1776_sympy-20590, 2.77 GB. The harness pulled a prebuilt image for the task instead of building one, which took 38 s of the 66.

3. Grade your own patches#

A predictions file is a JSON list with one entry per task. To get a diff that applies cleanly, make the edit inside the task's own image (the repository is at /testbed) and print git diff:

sbx exec $id "cd ~/eval && docker run --rm swebench/sweb.eval.x86_64.sympy_1776_sympy-20590 bash -c \"cd /testbed && sed -i \\\"s/^class Printable:/# Printing mixin for Basic and friends.\\nclass Printable:/\\\" sympy/core/_print_helpers.py && git diff\" > noop.diff && cat noop.diff"
diff --git a/sympy/core/_print_helpers.py b/sympy/core/_print_helpers.py
index 96d7e72676..69c96086f9 100755
--- a/sympy/core/_print_helpers.py
+++ b/sympy/core/_print_helpers.py
@@ -5,6 +5,7 @@
 but cannot be defined there without causing circular imports.
 """
 
+# Printing mixin for Basic and friends.
 class Printable:
     """
     The default implementation of printing for SymPy classes.

Wrap the diff in a predictions file and grade it:

sbx exec $id "cd ~/eval && python3 -c \"import json; json.dump([{\\\"instance_id\\\": \\\"sympy__sympy-20590\\\", \\\"model_name_or_path\\\": \\\"noop\\\", \\\"model_patch\\\": open(\\\"noop.diff\\\").read()}], open(\\\"noop.json\\\", \\\"w\\\"))\""
sbx exec $id --timeout 600 "cd ~/eval && ~/swe/bin/python -m swebench.harness.run_evaluation \
  -d SWE-bench/SWE-bench_Lite -i sympy__sympy-20590 -p noop.json --max_workers 1 --run_id noop1 \
  2>&1 | grep -E '^Instances (resolved|unresolved)'"
Instances resolved: 0
Instances unresolved: 1

Reading report.json for each candidate gives the whole picture:

Candidate FAIL_TO_PASS PASS_TO_PASS Resolved Task time
gold (cold, image pulled) 1 of 1 passed 21 of 21 passed Yes 66 s
gold (image cached) 1 of 1 passed 21 of 21 passed Yes 27 s
noop test_immutable failed 21 of 21 passed No 27 s
handmade No tests ran No tests ran No 23 s

The noop row is the case FAIL_TO_PASS exists for: nothing broke, but the issue wasn't fixed. The handmade patch applied cleanly (Applied patch sympy/core/_print_helpers.py cleanly. in the log), but SymPy itself sets plain attributes on its objects, so import sympy failed before any test ran:

  File "/testbed/sympy/assumptions/assume.py", line 154, in __new__
    obj.name = name
  File "/testbed/sympy/core/_print_helpers.py", line 23, in __setattr__
    raise AttributeError(name)
AttributeError: name

With no test results to parse, the report says "patch_successfully_applied": false and has no tests_status. Check test_output.txt before you blame the diff format.

Where the time goes#

From run_instance.log for the cached gold run (27.2 s from the client):

Step Time
Harness start and dataset load About 6 s
Create the container and apply the patch 0.5 s
Run the tests (bin/test sympy/core/tests/test_basic.py) 5.1 s
Stop and remove the container 15.2 s

More than half of a cached run is the container shutting down. With --max_workers above 1, those waits overlap, so run several tasks per harness call when you batch them in one VM. The harness's own help says to keep --max_workers at or below 75% of CPU cores, which is 6 on large.

4. Fan out: one task per VM#

Once a sandbox has the harness installed, save it as a named snapshot. Every task then starts from the same bytes, already set up:

sbx snapshot $id aeo-swe-ready
sbx delete $id --yes

This script runs two tasks at once, each in its own sandbox, and deletes each one as soon as its verdict is read:

#!/bin/bash
snap=aeo-swe-ready
inst=sympy__sympy-20590
run_task() {   # $1 = run id, $2 = predictions (gold or a file in ~/eval), $3 = sandbox name
  t0=$(date +%s.%N)
  id=$(sbx new --from "$snap" --name "$3" --type large --ttl 1800 --json | jq -r 'select(.event=="ready") | .id')
  t1=$(date +%s.%N)
  sbx exec "$id" --timeout 600 "cd ~/eval && ~/swe/bin/python -m swebench.harness.run_evaluation \
    -d SWE-bench/SWE-bench_Lite -i $inst -p $2 --max_workers 1 --run_id $1 > run.log 2>&1"
  t2=$(date +%s.%N)
  verdict=$(sbx exec "$id" "python3 -c \"import json,glob; r=json.load(open(glob.glob('eval/logs/run_evaluation/$1/*/$inst/report.json')[0]))['$inst']; t=r['tests_status']; print(json.dumps({'resolved': r['resolved'], 'f2p_failed': t['FAIL_TO_PASS']['failure'], 'p2p_failed': t['PASS_TO_PASS']['failure']}))\"")
  sbx delete "$id" --yes > /dev/null
  printf '%s %s start=%.1fs eval=%.1fs %s\n' "$1" "$id" "$(echo "$t1-$t0" | bc)" "$(echo "$t2-$t1" | bc)" "$verdict"
}
run_task gold gold aeo-swe-2 &
run_task noop noop.json aeo-swe-3 &
wait

The output, from prod:

noop sb_hg1olybk8t start=3.2s eval=27.8s {"resolved": false, "f2p_failed": ["test_immutable"], "p2p_failed": []}
gold sb_5httmzzxf1 start=3.2s eval=28.3s {"resolved": true, "f2p_failed": [], "p2p_failed": []}

Both tasks finished in 33 s of wall time, against about 62 s back to back. Each VM was ready 3.2 s after sbx new --from, and neither pulled the image, because the 2.77 GB image was already on the snapshot's disk.

In a real run, $inst changes per task and the predictions file holds your agent's patch for it.

Scale it#

One task per VM Many tasks per VM
Isolation Full: one kernel and one Docker daemon per task Tasks share one VM, each in its own container
Starts used One per task One per batch
Disk One task's image Every image in the batch, so watch the 125 GB on large
Use it for Untrusted agent patches, RL rollouts, reruns of one task Large sweeps on a tight start budget
  • Starts are rate limited. Creates, resumes and forks each count as one start. The $20 plan allows 12 a minute, 60 an hour and 200 a day, and 100 sandboxes at once. The $100 plan allows 30, 210 and 840, and 300 at once (billing). The 300-task SWE-bench Lite test split, one VM per task, is 300 starts. That is more than the $20 plan's 200 a day, so batch several tasks per VM there. On the $100 plan, 210 starts an hour gets through it in under 2 hours. Check what's left before a burst with sbx limits --json | jq '.starts'. A 429 rate_limited reply carries Retry-After and details.resetsAt.
  • Cost is per second. large costs $0.072/hour, so a one-minute task costs about $0.0012 of machine time. xlarge (16 vCPU, 32 GB) needs the $100 plan or higher.
  • Pre-pull images into the snapshot. A cold task here spent 38 s pulling its image, and VMs started from the snapshot skipped that. docker pull the images for a batch before sbx snapshot. Images count toward the snapshot's disk, so group tasks by repository rather than putting every image in one template.
  • Always clean up. Delete each sandbox after reading its report, and set a short --ttl as a backstop. When the sweep is done, run sbx snapshot rm aeo-swe-ready.

FAQ#

Do I need to build the SWE-bench Docker images myself?#

Not for this task. With swebench 5.0.2, the harness pulled a prebuilt image for sympy__sympy-20590 from Docker Hub (2.77 GB, 38 s) instead of building it. Building images yourself takes much longer and more disk.

Why does a task need Docker inside the sandbox?#

The official harness runs every task in its own container built for that repository and commit. A Sandboxer sandbox is a full VM with its own Docker daemon, so the harness runs as it would on a Linux server, with no host Docker socket shared.

Which sandbox size should I use?#

large (8 vCPU, 16 GB, 125 GB disk) ran this task with room to spare. Use default (4 vCPU, 8 GB, 50 GB) for one small task at a time, and xlarge for many workers in one VM.

How do I grade a model's patch instead of the gold patch?#

Write a predictions JSON list of instance_id, model_name_or_path and model_patch (a unified diff against base_commit), and pass it with -p. The harness applies the diff, runs the tests and writes report.json.

Can one task affect another?#

Not with one task per VM. Each sandbox has its own kernel, disk and Docker daemon, and each starts from the same snapshot, so nothing one task writes reaches the next.

More guides

Top AI sandbox platforms for code execution in 2026, ranked

For running AI-written code we rank Sandboxer, Freestyle and E2B first: each gives code its own VM with an API or SDK and can pause with memory. Pick Daytona for Windows or GPUs, Modal or Northflank for GPUs at scale, E2B or Northflank to run in your own cloud, and Vercel Sandbox if your app lives on Vercel.

Read the guide
Best sandboxes for running coding agents (Claude Code, Codex, Gemini CLI) in 2026, ranked

For coding agents we rank Sandboxer first: a full Ubuntu VM with Claude Code and Codex preinstalled, Docker, and stop-and-resume that keeps the agent's session in memory. Freestyle and Amp Orbs also keep memory across pauses. Pick E2B for prebuilt agent templates on open source, Docker Sandboxes for free local runs, and Codespaces if you live in GitHub.

Read the guide
Best Daytona sandbox alternatives in 2026, ranked

The best Daytona alternatives depend on why you're switching. For a full Linux microVM on every sandbox with memory-saving stop and Docker, we rank Sandboxer first. E2B suits open source and self-hosting, Modal massive short-job fleets, Northflank bring-your-own-cloud with GPUs, and Vercel Sandbox apps on Vercel.

Read the guide
Best E2B alternatives in 2026, ranked

If you're leaving E2B for longer sessions or a fuller machine, we rank Sandboxer first, then Freestyle: both give each sandbox a full Linux VM that keeps memory across pauses, with Docker and no short session cap. Pick Daytona for Windows or GPUs, Northflank for bring-your-own-cloud, and stay on E2B if you need open source.

Read the guide