Skip to content
sandboxer
Start free trial
Guides · Agent cookbook

How Do I Shard a Test Suite Across Several Sandboxes?

Updated October 27, 2026 .md

Short answer: Install the project once, save it with sbx snapshot, then start one sandbox per shard with sbx new --from and run pytest-split's --splits N --group i in each. Copy each JUnit file back, merge them and delete the sandboxes. On prod, two small shards cut the networkx 3.5 suite from 96.9 s to 59.3 s.

Results#

We ran the full test suite of networkx at the networkx-3.5 release (commit 4fa222d2fb157e1b7f8c753c9f92e5907d1ddeb4) on prod on 2026-10-27, on small sandboxes (2 vCPU, 4 GB) in eu-central. One run per row.

Setup Wall time Sandbox time billed Cost at $0.018/hour
1 sandbox, plain pytest 96.9 s about 103 s (3.2 s create + 96.9 s + teardown) about $0.0005
1 sandbox, pytest -n 2 (pytest-xdist) 58.9 s about 65 s about $0.0003
2 sandboxes, one shard each 59.3 s, create to last delete 103.8 s (44.5 s + 59.3 s) $0.0005

What the numbers say:

  • Two shards cost about the same as one, and finish 39% sooner. Sharding adds sandboxes, not test work. The extra cost is each shard's fixed overhead: create, Python start-up, collection, copying results back, delete.
  • Use the vCPUs you already have first. A small sandbox has 2 vCPU, and plain pytest uses one. pytest -n 2 in a single sandbox matched two shards on wall time with about 40% less sandbox time (65 s against 103.8 s). Sharding across sandboxes pays off once each sandbox's cores are busy. We didn't measure the two combined (two shards, each with -n 2).
  • The shards were not balanced. Shard 1 ran 48.6 s of pytest, shard 2 ran 33.8 s, and the run waits for the slower one. Why, and the fix, is below.

The "1 sandbox" rows ran in the prepared sandbox before we saved the snapshot. Their billed time adds the 3.2 s create we measured for the shards; we didn't time a separate delete for them.

The pattern#

Step Command How often
Clone at a pinned commit, install, record test durations sbx new, sbx exec Once per lockfile change
Save the prepared disk sbx snapshot <id> <name> Once
One sandbox per shard sbx new --from <name> N times, in parallel
Run shard i of N pytest --splits N --group i --junitxml=… Once per shard
Keep the JUnit file, not the VM sbx exec … cat, then sbx delete <id> --yes Once per shard
Merge python3 merge.py results/junit-*.xml Once

Saving and reusing the prepared disk is covered in depth in cache dependencies in a named snapshot. This page is about what you do with it: split the work, collect the results and decide how many shards are worth it. If you want N different attempts at a fix rather than N slices of one test run, see best-of-N agents with parallel forks.

1. Prepare once and record durations#

Create a sandbox, clone networkx at the pinned commit and install it with its test extras, plus pytest-split for sharding:

sbx new --name test-shard-1 --type small --ttl 5400
sbx exec sb_ujjwkl4zej --timeout 600 "set -e; python3 --version; nproc; \
  git clone -q https://github.com/networkx/networkx && cd networkx \
  && git checkout -q cd59914a6e46b04f006633f203ee7b4878d53159 && git log -1 --format='%h %s' \
  && python3 -m venv .venv && .venv/bin/pip install -q -e .[default,test] pytest-split==0.10.0 junitparser \
  && .venv/bin/pip list | grep -iE '^(pytest|numpy|scipy|pandas|matplotlib|pytest-split|junitparser) '"
Python 3.12.3
2
4fa222d2f Designate 3.5 release
junitparser     5.0.3
matplotlib      3.11.2
numpy           2.5.4
pandas          3.1.0
pytest          8.4.2
pytest-split    0.10.0
scipy           1.18.1

cd59914 is the annotated tag object for networkx-3.5; it checks out commit 4fa222d. The install took 33 s. Pip resolved the newest numpy, scipy and pandas that day. For results you can repeat, install from a lockfile instead.

Then run the whole suite once with --store-durations. pytest-split writes each test's time to .test_durations, and it uses that file to balance the shards later:

sbx exec sb_ujjwkl4zej --detach "cd networkx && .venv/bin/pytest -q -p no:cacheprovider \
  --store-durations --junitxml=/home/user/junit-full.xml networkx > /home/user/full.log 2>&1"
6532 passed, 67 skipped, 38 warnings in 96.85s (0:01:36)

For the in-sandbox comparison we also installed pytest-xdist before saving the snapshot:

sbx exec sb_ujjwkl4zej --timeout 120 "cd networkx && .venv/bin/pip install -q pytest-xdist==3.8.0"

The durations file had 6,596 entries summing to 84.2 s; the other 12 s or so is start-up and collection. Because .test_durations is on the disk, the snapshot carries it to every shard. When you add tests, rerun this step and save the snapshot again.

2. The shard script#

shard.sh lives in the sandbox's home directory, so it's in the snapshot too. It runs one group, writes ~/junit-<group>.xml and prints one summary line:

#!/bin/bash
# shard.sh <group> <splits> [extra pytest args]: run one shard, write ~/junit-<group>.xml
cd ~/networkx
group=$1 splits=$2; shift 2
start=$(date +%s%N)
.venv/bin/pytest -q -p no:cacheprovider --splits "$splits" --group "$group" \
  --junitxml="$HOME/junit-$group.xml" "$@" networkx > "$HOME/shard-$group.log" 2>&1
rc=$?
echo "group $group/$splits exit=$rc wall=$(( ($(date +%s%N) - start) / 1000000 ))ms | $(tail -1 ~/shard-$group.log)"

With 1 1 it runs the whole suite. That gave the two one-sandbox rows in the results table:

sbx exec sb_ujjwkl4zej --timeout 300 "./shard.sh 1 1"
sbx exec sb_ujjwkl4zej --timeout 300 "./shard.sh 1 1 -n 2"
group 1/1 exit=0 wall=96855ms | 6532 passed, 67 skipped, 38 warnings in 90.13s (0:01:30)
group 1/1 exit=0 wall=58926ms | 6532 passed, 67 skipped, 44 warnings in 56.00s

Clean up the run's output, save the snapshot and delete the prepared sandbox. The snapshot outlives it:

sbx exec sb_ujjwkl4zej "rm -f junit-*.xml shard-*.log; ls"
sbx snapshot sb_ujjwkl4zej shard-nx-4fa222d
sbx delete sb_ujjwkl4zej --yes
networkx
shard.sh
{"ok":true,"type":"snapshot.named.info","snapshot":{…,"name":"shard-nx-4fa222d",…,"status":"ready","type":"small",…}}

Saving took 2.1 s. Put the commit in the snapshot name, so a new commit gets a new snapshot instead of silently reusing the old one.

3. Run the shards in parallel#

run-shards.sh runs on your laptop or CI runner and needs sbx and jq. It starts one background job per shard. Each job creates a sandbox from the snapshot, runs its group, copies the JUnit file back and deletes the sandbox:

#!/bin/bash
# run-shards.sh <snapshot> <N>: N sandboxes from one named snapshot, one shard each, in parallel.
snap=$1 N=$2
mkdir -p results && rm -f results/*
ms() { date +%s%3N; }
t0=$(ms)

run_shard() {
  local g=$1 id t
  t=$(ms)
  id=$(sbx new --from "$snap" --name "test-shard-$((g + 1))" --ttl 1800 \
       | jq -r 'select(.event == "ready") | .id')
  echo "shard $g/$N: $id ready in $(( $(ms) - t ))ms"
  sbx exec "$id" --timeout 600 "./shard.sh $g $N" | tee "results/$g.txt"
  sbx exec "$id" "gzip -c junit-$g.xml | base64 -w0" | base64 -d | gunzip > "results/junit-$g.xml"
  sbx delete "$id" --yes > /dev/null
  echo "shard $g/$N: deleted $id, $(( $(ms) - t ))ms from create to delete"
}

for g in $(seq 1 "$N"); do run_shard "$g" & done
wait
echo "all $N shards done in $(( $(ms) - t0 ))ms"
python3 merge.py results/junit-*.xml > junit.xml

Notes on the choices:

  • --ttl 1800. If the script dies between create and delete, the sandbox stops on its own after 30 minutes instead of running on.
  • gzip and base64 for the JUnit file. sbx exec returns the command's output, so the file comes back through it. Shard 1's XML was 520 KB; compressing it keeps the transfer small and binary-safe.
  • The sandbox goes as soon as its file is back. With per-second billing, a finished shard that stays up waiting for the others costs money and does nothing.
  • For suites longer than 10 minutes, synchronous sbx exec stops at --timeout 600. Start the shard with sbx exec --detach and poll sbx exec <id> --status <pid> instead.

The output from prod:

./run-shards.sh shard-nx-4fa222d 2
shard 1/2: sb_0puccgpvfo ready in 3187ms
shard 2/2: sb_afpu973qm7 ready in 3188ms
group 2/2 exit=0 wall=38658ms | 2798 passed, 48 skipped, 3753 deselected, 12 warnings in 33.80s
shard 2/2: deleted sb_afpu973qm7, 44519ms from create to delete
group 1/2 exit=0 wall=53311ms | 3734 passed, 22 skipped, 2843 deselected, 32 warnings in 48.63s
shard 1/2: deleted sb_0puccgpvfo, 59277ms from create to delete
all 2 shards done in 59282ms
merged 2 files: tests=6602, failures=0, errors=0, skipped=70, summed test time=82.4s

Both sandboxes were ready in 3.2 s. Passed tests add up exactly: 3,734 + 2,798 = 6,532, the same as the single run. The merged file shows 70 skips, not 67, because each shard collected all 6,599 tests and selected 3,756 + 2,846 = 6,602. The 3 extra are tests missing from the durations file (6,596 entries against 6,599 collected), which both shards ran; all 3 are skips. Count passes and failures, not totals, when you compare a sharded run with an unsharded one.

4. Merge the JUnit files#

Most CI systems accept several JUnit files, so you may not need to merge. When you want one file, merge.py uses only the Python standard library:

# merge.py: merge JUnit XML files from pytest into one <testsuites> document on stdout.
import sys
import xml.etree.ElementTree as ET

out = ET.Element("testsuites")
total = {"tests": 0, "failures": 0, "errors": 0, "skipped": 0}
time = 0.0
for path in sys.argv[1:]:
    root = ET.parse(path).getroot()
    for suite in root.iter("testsuite"):
        suite.set("name", f"{suite.get('name')}:{path}")
        out.append(suite)
        for k in total:
            total[k] += int(suite.get(k, 0))
        time += float(suite.get("time", 0))
for k, v in total.items():
    out.set(k, str(v))
out.set("time", f"{time:.2f}")
ET.ElementTree(out).write(sys.stdout.buffer, encoding="utf-8", xml_declaration=True)
print(f"merged {len(sys.argv) - 1} files: " + ", ".join(f"{k}={v}" for k, v in total.items())
      + f", summed test time={time:.1f}s", file=sys.stderr)
<?xml version='1.0' encoding='utf-8'?>
<testsuites tests="6602" failures="0" errors="0" skipped="70" time="82.39"><testsuite name="pytest:results/junit-1.xml" errors="0" failures="0" skipped="22" tests="3756" time="48.605" …

Make the CI job fail when failures or errors in the merged file is above zero, or when any shard's exit= is not 0.

Balance the shards#

Adding up the <testcase> times in each JUnit file gives 45.6 s for shard 1 and 31.1 s for shard 2. pytest-split's default algorithm, duration_based_chunks, cuts the test list into contiguous runs and keeps the original order. Its README says the other algorithm, least_duration, "should give more balanced groups":

pytest --splits 2 --group 1 --splitting-algorithm least_duration

We didn't rerun with it, so we have no number for the gain. Two other causes of imbalance are under your control: durations recorded on a different machine size than the shards, and tests added since the durations were recorded.

How many shards: a model, not a measurement#

We measured 1 and 2 shards only. What follows is arithmetic from the 2-shard run, not a result.

Each shard paid about 13.6 s that doesn't shrink with N: 3.2 s to create, about 7.6 s of pytest start-up and collection (shard wall time minus the sum of its test times), and about 2.8 s to copy the results back and delete. The test work was 76.7 s in total. With perfectly balanced shards:

  • wall time ≈ 13.6 + 76.7 / N seconds
  • billed sandbox time ≈ 13.6 × N + 76.7 seconds

For N = 2 the formula gives 52 s where we measured 59.3 s; the gap is the imbalance above. The formula says the test part shrinks with N while the fixed overhead is paid once per shard, so once 76.7 / N gets close to 13.6, more shards mostly add cost. We haven't measured where that point falls; run 4 or more shards on your own suite before you rely on it. A suite with more test time per shard, or a heavier setup that the snapshot already did, moves that point higher.

Two limits apply whatever the model says. Each sbx new counts as a start against your plan's per-minute, per-hour and per-day start limits, and the number of sandboxes running at once is capped per plan (two during the free trial). See limits and billing.

FAQ#

Why a named snapshot and not sbx fork?#

A fork needs its source sandbox to still exist. A named snapshot doesn't, so you can delete the prepared sandbox and start shards from the snapshot on every CI run, from any machine with the API key.

Can I shard without recording durations first?#

Yes. Without a .test_durations file, pytest-split splits by test count, which can be far off when a few tests are slow. Recording durations once and saving them in the snapshot costs one full run.

Does a failing shard stop the others?#

No. Each shard is its own sandbox and its own background job. shard.sh prints the pytest exit code, the merged JUnit file counts every failure, and your CI step decides whether to fail.

Should I use pytest-xdist inside each shard?#

Yes, if your tests are safe to run in parallel processes. On a small sandbox, pytest -n 2 alone matched two shards on wall time with about 40% less sandbox time. Use the cores in each sandbox first, then add sandboxes.

What happens to a shard sandbox if my laptop disconnects?#

It keeps running until its TTL ends, then stops; the script passes --ttl 1800 for that reason. A stopped sandbox isn't billed for time, but its disk counts toward your plan's included storage, so delete leftovers with sbx delete <id> --yes.

More guides

Top AI sandbox platforms for code execution in 2026, ranked

For running AI-written code we rank Sandboxer, Freestyle and E2B first: each gives code its own VM with an API or SDK and can pause with memory. Pick Daytona for Windows or GPUs, Modal or Northflank for GPUs at scale, E2B or Northflank to run in your own cloud, and Vercel Sandbox if your app lives on Vercel.

Read the guide
Best sandboxes for running coding agents (Claude Code, Codex, Gemini CLI) in 2026, ranked

For coding agents we rank Sandboxer first: a full Ubuntu VM with Claude Code and Codex preinstalled, Docker, and stop-and-resume that keeps the agent's session in memory. Freestyle and Amp Orbs also keep memory across pauses. Pick E2B for prebuilt agent templates on open source, Docker Sandboxes for free local runs, and Codespaces if you live in GitHub.

Read the guide
Best Daytona sandbox alternatives in 2026, ranked

The best Daytona alternatives depend on why you're switching. For a full Linux microVM on every sandbox with memory-saving stop and Docker, we rank Sandboxer first. E2B suits open source and self-hosting, Modal massive short-job fleets, Northflank bring-your-own-cloud with GPUs, and Vercel Sandbox apps on Vercel.

Read the guide
Best E2B alternatives in 2026, ranked

If you're leaving E2B for longer sessions or a fuller machine, we rank Sandboxer first, then Freestyle: both give each sandbox a full Linux VM that keeps memory across pauses, with Docker and no short session cap. Pick Daytona for Windows or GPUs, Northflank for bring-your-own-cloud, and stay on E2B if you need open source.

Read the guide