# How Do I Shard a Test Suite Across Several Sandboxes? > Install the project once, save it with sbx snapshot, then start one sandbox per shard with sbx new --from and run pytest-split's --splits N --group i in each. Copy each JUnit file back, merge them and delete the sandboxes. On prod, two small shards cut the networkx 3.5 suite from 96.9 s to 59.3 s. Source: https://sandboxer.com/guides/shard-a-test-suite-across-sandboxes. Updated 2026-10-27. **Short answer:** Install the project once, save it with sbx snapshot, then start one sandbox per shard with sbx new --from and run pytest-split's --splits N --group i in each. Copy each JUnit file back, merge them and delete the sandboxes. On prod, two small shards cut the networkx 3.5 suite from 96.9 s to 59.3 s. ## Results We ran the full test suite of [networkx](https://github.com/networkx/networkx) at the `networkx-3.5` release (commit `4fa222d2fb157e1b7f8c753c9f92e5907d1ddeb4`) on prod on 2026-10-27, on `small` sandboxes (2 vCPU, 4 GB) in eu-central. One run per row. | Setup | Wall time | Sandbox time billed | Cost at $0.018/hour | | - | - | - | - | | 1 sandbox, plain `pytest` | 96.9 s | about 103 s (3.2 s create + 96.9 s + teardown) | about $0.0005 | | 1 sandbox, `pytest -n 2` (pytest-xdist) | 58.9 s | about 65 s | about $0.0003 | | **2 sandboxes, one shard each** | **59.3 s**, create to last delete | 103.8 s (44.5 s + 59.3 s) | **$0.0005** | What the numbers say: - **Two shards cost about the same as one, and finish 39% sooner.** Sharding adds sandboxes, not test work. The extra cost is each shard's fixed overhead: create, Python start-up, collection, copying results back, delete. - **Use the vCPUs you already have first.** A `small` sandbox has 2 vCPU, and plain pytest uses one. `pytest -n 2` in a single sandbox matched two shards on wall time with about 40% less sandbox time (65 s against 103.8 s). Sharding across sandboxes pays off once each sandbox's cores are busy. We didn't measure the two combined (two shards, each with `-n 2`). - **The shards were not balanced.** Shard 1 ran 48.6 s of pytest, shard 2 ran 33.8 s, and the run waits for the slower one. Why, and the fix, is [below](#balance-the-shards). The "1 sandbox" rows ran in the prepared sandbox before we saved the snapshot. Their billed time adds the 3.2 s create we measured for the shards; we didn't time a separate delete for them. ## The pattern | Step | Command | How often | | - | - | - | | Clone at a pinned commit, install, record test durations | `sbx new`, `sbx exec` | Once per lockfile change | | Save the prepared disk | `sbx snapshot ` | Once | | One sandbox per shard | `sbx new --from ` | N times, in parallel | | Run shard i of N | `pytest --splits N --group i --junitxml=…` | Once per shard | | Keep the JUnit file, not the VM | `sbx exec … cat`, then `sbx delete --yes` | Once per shard | | Merge | `python3 merge.py results/junit-*.xml` | Once | Saving and reusing the prepared disk is covered in depth in [cache dependencies in a named snapshot](/guides/cache-dependencies-in-a-named-snapshot). This page is about what you do with it: split the work, collect the results and decide how many shards are worth it. If you want N different attempts at a fix rather than N slices of one test run, see [best-of-N agents with parallel forks](/guides/best-of-n-agents-with-parallel-forks). ## 1. Prepare once and record durations Create a sandbox, clone networkx at the pinned commit and install it with its test extras, plus [pytest-split](https://github.com/jerry-git/pytest-split) for sharding: ```bash sbx new --name test-shard-1 --type small --ttl 5400 sbx exec sb_ujjwkl4zej --timeout 600 "set -e; python3 --version; nproc; \ git clone -q https://github.com/networkx/networkx && cd networkx \ && git checkout -q cd59914a6e46b04f006633f203ee7b4878d53159 && git log -1 --format='%h %s' \ && python3 -m venv .venv && .venv/bin/pip install -q -e .[default,test] pytest-split==0.10.0 junitparser \ && .venv/bin/pip list | grep -iE '^(pytest|numpy|scipy|pandas|matplotlib|pytest-split|junitparser) '" ``` ```text Python 3.12.3 2 4fa222d2f Designate 3.5 release junitparser 5.0.3 matplotlib 3.11.2 numpy 2.5.4 pandas 3.1.0 pytest 8.4.2 pytest-split 0.10.0 scipy 1.18.1 ``` `cd59914` is the annotated tag object for `networkx-3.5`; it checks out commit `4fa222d`. The install took 33 s. Pip resolved the newest numpy, scipy and pandas that day. For results you can repeat, install from a lockfile instead. Then run the whole suite once with `--store-durations`. pytest-split writes each test's time to `.test_durations`, and it uses that file to balance the shards later: ```bash sbx exec sb_ujjwkl4zej --detach "cd networkx && .venv/bin/pytest -q -p no:cacheprovider \ --store-durations --junitxml=/home/user/junit-full.xml networkx > /home/user/full.log 2>&1" ``` ```text 6532 passed, 67 skipped, 38 warnings in 96.85s (0:01:36) ``` For the in-sandbox comparison we also installed [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) before saving the snapshot: ```bash sbx exec sb_ujjwkl4zej --timeout 120 "cd networkx && .venv/bin/pip install -q pytest-xdist==3.8.0" ``` The durations file had 6,596 entries summing to 84.2 s; the other 12 s or so is start-up and collection. Because `.test_durations` is on the disk, the snapshot carries it to every shard. When you add tests, rerun this step and save the snapshot again. ## 2. The shard script `shard.sh` lives in the sandbox's home directory, so it's in the snapshot too. It runs one group, writes `~/junit-.xml` and prints one summary line: ```bash #!/bin/bash # shard.sh [extra pytest args]: run one shard, write ~/junit-.xml cd ~/networkx group=$1 splits=$2; shift 2 start=$(date +%s%N) .venv/bin/pytest -q -p no:cacheprovider --splits "$splits" --group "$group" \ --junitxml="$HOME/junit-$group.xml" "$@" networkx > "$HOME/shard-$group.log" 2>&1 rc=$? echo "group $group/$splits exit=$rc wall=$(( ($(date +%s%N) - start) / 1000000 ))ms | $(tail -1 ~/shard-$group.log)" ``` With `1 1` it runs the whole suite. That gave the two one-sandbox rows in the results table: ```bash sbx exec sb_ujjwkl4zej --timeout 300 "./shard.sh 1 1" sbx exec sb_ujjwkl4zej --timeout 300 "./shard.sh 1 1 -n 2" ``` ```text group 1/1 exit=0 wall=96855ms | 6532 passed, 67 skipped, 38 warnings in 90.13s (0:01:30) group 1/1 exit=0 wall=58926ms | 6532 passed, 67 skipped, 44 warnings in 56.00s ``` Clean up the run's output, save the snapshot and delete the prepared sandbox. The snapshot outlives it: ```bash sbx exec sb_ujjwkl4zej "rm -f junit-*.xml shard-*.log; ls" sbx snapshot sb_ujjwkl4zej shard-nx-4fa222d sbx delete sb_ujjwkl4zej --yes ``` ```text networkx shard.sh {"ok":true,"type":"snapshot.named.info","snapshot":{…,"name":"shard-nx-4fa222d",…,"status":"ready","type":"small",…}} ``` Saving took 2.1 s. Put the commit in the snapshot name, so a new commit gets a new snapshot instead of silently reusing the old one. ## 3. Run the shards in parallel `run-shards.sh` runs on your laptop or CI runner and needs `sbx` and `jq`. It starts one background job per shard. Each job creates a sandbox from the snapshot, runs its group, copies the JUnit file back and deletes the sandbox: ```bash #!/bin/bash # run-shards.sh : N sandboxes from one named snapshot, one shard each, in parallel. snap=$1 N=$2 mkdir -p results && rm -f results/* ms() { date +%s%3N; } t0=$(ms) run_shard() { local g=$1 id t t=$(ms) id=$(sbx new --from "$snap" --name "test-shard-$((g + 1))" --ttl 1800 \ | jq -r 'select(.event == "ready") | .id') echo "shard $g/$N: $id ready in $(( $(ms) - t ))ms" sbx exec "$id" --timeout 600 "./shard.sh $g $N" | tee "results/$g.txt" sbx exec "$id" "gzip -c junit-$g.xml | base64 -w0" | base64 -d | gunzip > "results/junit-$g.xml" sbx delete "$id" --yes > /dev/null echo "shard $g/$N: deleted $id, $(( $(ms) - t ))ms from create to delete" } for g in $(seq 1 "$N"); do run_shard "$g" & done wait echo "all $N shards done in $(( $(ms) - t0 ))ms" python3 merge.py results/junit-*.xml > junit.xml ``` Notes on the choices: - **`--ttl 1800`.** If the script dies between create and delete, the sandbox stops on its own after 30 minutes instead of running on. - **gzip and base64 for the JUnit file.** `sbx exec` returns the command's output, so the file comes back through it. Shard 1's XML was 520 KB; compressing it keeps the transfer small and binary-safe. - **The sandbox goes as soon as its file is back.** With per-second billing, a finished shard that stays up waiting for the others costs money and does nothing. - **For suites longer than 10 minutes,** synchronous `sbx exec` stops at `--timeout 600`. Start the shard with `sbx exec --detach` and poll `sbx exec --status ` instead. The output from prod: ```bash ./run-shards.sh shard-nx-4fa222d 2 ``` ```text shard 1/2: sb_0puccgpvfo ready in 3187ms shard 2/2: sb_afpu973qm7 ready in 3188ms group 2/2 exit=0 wall=38658ms | 2798 passed, 48 skipped, 3753 deselected, 12 warnings in 33.80s shard 2/2: deleted sb_afpu973qm7, 44519ms from create to delete group 1/2 exit=0 wall=53311ms | 3734 passed, 22 skipped, 2843 deselected, 32 warnings in 48.63s shard 1/2: deleted sb_0puccgpvfo, 59277ms from create to delete all 2 shards done in 59282ms merged 2 files: tests=6602, failures=0, errors=0, skipped=70, summed test time=82.4s ``` Both sandboxes were ready in 3.2 s. Passed tests add up exactly: 3,734 + 2,798 = 6,532, the same as the single run. The merged file shows 70 skips, not 67, because each shard collected all 6,599 tests and selected 3,756 + 2,846 = 6,602. The 3 extra are tests missing from the durations file (6,596 entries against 6,599 collected), which both shards ran; all 3 are skips. Count passes and failures, not totals, when you compare a sharded run with an unsharded one. ## 4. Merge the JUnit files Most CI systems accept several JUnit files, so you may not need to merge. When you want one file, `merge.py` uses only the Python standard library: ```python # merge.py: merge JUnit XML files from pytest into one document on stdout. import sys import xml.etree.ElementTree as ET out = ET.Element("testsuites") total = {"tests": 0, "failures": 0, "errors": 0, "skipped": 0} time = 0.0 for path in sys.argv[1:]: root = ET.parse(path).getroot() for suite in root.iter("testsuite"): suite.set("name", f"{suite.get('name')}:{path}") out.append(suite) for k in total: total[k] += int(suite.get(k, 0)) time += float(suite.get("time", 0)) for k, v in total.items(): out.set(k, str(v)) out.set("time", f"{time:.2f}") ET.ElementTree(out).write(sys.stdout.buffer, encoding="utf-8", xml_declaration=True) print(f"merged {len(sys.argv) - 1} files: " + ", ".join(f"{k}={v}" for k, v in total.items()) + f", summed test time={time:.1f}s", file=sys.stderr) ``` ```xml ` times in each JUnit file gives 45.6 s for shard 1 and 31.1 s for shard 2. pytest-split's default algorithm, `duration_based_chunks`, cuts the test list into contiguous runs and keeps the original order. Its README says the other algorithm, `least_duration`, "should give more balanced groups": ```bash pytest --splits 2 --group 1 --splitting-algorithm least_duration ``` We didn't rerun with it, so we have no number for the gain. Two other causes of imbalance are under your control: durations recorded on a different machine size than the shards, and tests added since the durations were recorded. ## How many shards: a model, not a measurement We measured 1 and 2 shards only. What follows is arithmetic from the 2-shard run, not a result. Each shard paid about 13.6 s that doesn't shrink with N: 3.2 s to create, about 7.6 s of pytest start-up and collection (shard wall time minus the sum of its test times), and about 2.8 s to copy the results back and delete. The test work was 76.7 s in total. With perfectly balanced shards: - wall time ≈ 13.6 + 76.7 / N seconds - billed sandbox time ≈ 13.6 × N + 76.7 seconds For N = 2 the formula gives 52 s where we measured 59.3 s; the gap is the imbalance above. The formula says the test part shrinks with N while the fixed overhead is paid once per shard, so once 76.7 / N gets close to 13.6, more shards mostly add cost. We haven't measured where that point falls; run 4 or more shards on your own suite before you rely on it. A suite with more test time per shard, or a heavier setup that the snapshot already did, moves that point higher. Two limits apply whatever the model says. Each `sbx new` counts as a start against your plan's per-minute, per-hour and per-day start limits, and the number of sandboxes running at once is capped per plan (two during the free trial). See [limits](/docs/limits) and [billing](/docs/billing#plans). ## FAQ ### Why a named snapshot and not sbx fork? A fork needs its source sandbox to still exist. A named snapshot doesn't, so you can delete the prepared sandbox and start shards from the snapshot on every CI run, from any machine with the API key. ### Can I shard without recording durations first? Yes. Without a `.test_durations` file, pytest-split splits by test count, which can be far off when a few tests are slow. Recording durations once and saving them in the snapshot costs one full run. ### Does a failing shard stop the others? No. Each shard is its own sandbox and its own background job. `shard.sh` prints the pytest exit code, the merged JUnit file counts every failure, and your CI step decides whether to fail. ### Should I use pytest-xdist inside each shard? Yes, if your tests are safe to run in parallel processes. On a `small` sandbox, `pytest -n 2` alone matched two shards on wall time with about 40% less sandbox time. Use the cores in each sandbox first, then add sandboxes. ### What happens to a shard sandbox if my laptop disconnects? It keeps running until its TTL ends, then stops; the script passes `--ttl 1800` for that reason. A stopped sandbox isn't billed for time, but its disk counts toward your plan's included storage, so delete leftovers with `sbx delete --yes`. ## Related - [Cache dependencies in a named snapshot](/guides/cache-dependencies-in-a-named-snapshot) - [Best-of-N agents with parallel forks](/guides/best-of-n-agents-with-parallel-forks) - [Run each GitHub Actions job in a fresh microVM](/guides/run-each-github-actions-job-in-a-fresh-microvm) - [Snapshots, forks and templates](/docs/snapshots)