# How Do You Capture the Machine State When a Flaky Test Fails? > Loop the test inside a sandbox and write a marker file the moment it fails. Watch for the marker from outside, then fork the sandbox, copy the logs out, save a named snapshot and stop it. The stop keeps the failing processes alive in memory. The fork and the snapshot keep only the disk. Source: https://sandboxer.com/guides/capture-a-flaky-test-failure-with-a-snapshot. Updated 2026-10-30. **Short answer:** Loop the test inside a sandbox and write a marker file the moment it fails. Watch for the marker from outside, then fork the sandbox, copy the logs out, save a named snapshot and stop it. The stop keeps the failing processes alive in memory. The fork and the snapshot keep only the disk. This guide is about the one failing run: freezing it and replaying it. Measuring how often a test flakes, and why retries hide flakes, is covered in [sandboxes for QA and test automation](/guides/sandboxes-for-qa-and-test-automation). ## What each capture keeps We checked each row on prod on 2026-10-30. The details are in the run below. | Capture | Disk at the failure | Running processes and memory | The original after it | Use it for | | - | - | - | - | - | | `sbx fork` | Yes, as of the fork | No: the fork boots fresh | Keeps running | Poking at the files right away, without touching the original | | `sbx snapshot ` | Yes, as of the save | No | Keeps running | Starting any number of new sandboxes from the failing disk with `sbx new --from` | | `sbx stop --wait` | Yes | Yes, on resume (`memoryRestored: true`) | Stopped, no machine time | Coming back later to the live failing processes | | Artifacts copied out | Only what you collected | Only what you collected (`ps`, `lsof`, stacks, a core file) | Unchanged | Reading on your own machine, attaching to a bug | Do them in this order: fork, artifacts, snapshot, stop. The fork and the artifacts need the machine as it was at the failure. The stop comes last because it ends the live session. ## The flaky test The test starts a small HTTP service, sleeps a fixed 300 ms, then calls it. The service has a variable startup delay, so sometimes it isn't listening yet. A fixed sleep instead of a readiness check is a common real cause of flakes. The test also has a second bug that often comes with the first: it cleans up only when it passes, so a failure leaves the service running. ```python # server.py: a tiny service that keeps a session token and a hit count in memory import faulthandler, os, random, signal, sys, time, secrets from http.server import BaseHTTPRequestHandler, HTTPServer faulthandler.register(signal.SIGUSR1, file=sys.stderr) # kill -USR1 dumps Python stacks, process keeps running seed = int(os.environ.get("FLAKY_SEED", "0")) time.sleep(random.Random(seed).uniform(0.0, 0.4)) # slow, variable startup (cold caches, imports, a busy CPU) token, hits = secrets.token_hex(4), 0 class H(BaseHTTPRequestHandler): def do_GET(self): global hits hits += 1 body = f"token={token} hits={hits}\n".encode() self.send_response(200); self.end_headers(); self.wfile.write(body) def log_message(self, fmt, *args): print(f"{time.strftime('%H:%M:%S')} {fmt % args}", file=sys.stderr, flush=True) print(f"listening seed={seed} pid={os.getpid()}", file=sys.stderr, flush=True) HTTPServer(("127.0.0.1", 8765), H).serve_forever() ``` ```python # test_server.py: sleeps a fixed 300 ms instead of waiting until the server is ready import os, subprocess, sys, time, urllib.request log = open("logs/server.log", "a") srv = subprocess.Popen([sys.executable, "server.py"], stderr=log, env=os.environ) time.sleep(0.3) # the race body = urllib.request.urlopen("http://127.0.0.1:8765/", timeout=2).read().decode() assert body.startswith("token="), body srv.terminate(); srv.wait() # cleanup only runs when the test passes print("ok", body.strip()) ``` The seed makes the startup delay repeatable. Record the seed (or whatever randomness your test framework uses) on every run. It is what turns a flaky failure into one you can replay. ## Inside the sandbox: run until it fails, then capture The wrapper runs the test in a loop. On the first failure it collects the short-lived state, the processes, sockets and open files, and writes them under `~/failure`. Then it writes the marker file and exits. It writes to the home directory, not `/tmp`, because `/tmp` isn't kept across a cold boot ([what a snapshot contains](/docs/snapshots#what-a-snapshot-contains)). ```bash #!/bin/bash # run-until-fail.sh: run the flaky test until it fails, capture the machine state, then leave a marker cd ~/flaky && mkdir -p logs ~/failure max=${MAX_RUNS:-500} for n in $(seq 1 "$max"); do seed=$RANDOM echo "$n $seed" > ~/failure/progress if FLAKY_SEED=$seed python3 test_server.py > "logs/run.log" 2>&1; then continue; fi # Failed: capture before anything changes. Everything goes under ~/failure (not /tmp). f=~/failure date -u +%FT%T.%3NZ > $f/failed-at cp logs/run.log $f/test-output.log; cp logs/server.log $f/server.log ps -eo pid,ppid,etimes,stat,rss,args --forest > $f/ps.txt ss -ltnp > $f/listening.txt 2>&1 pid=$(pgrep -f 'python3 server.py' | head -1) if [ -n "$pid" ]; then echo "$pid" > $f/server.pid lsof -p "$pid" > $f/lsof.txt 2>&1 kill -USR1 "$pid"; sleep 0.2 # Python stacks into server.log; process keeps running cp logs/server.log $f/server.log command -v gcore >/dev/null && sudo gcore -o $f/core "$pid" > $f/gcore.log 2>&1 fi echo "run=$n seed=$seed" > $f/FAILED # the outside loop watches for this file exit 1 done echo "no failure in $max runs" > ~/failure/PASSED ``` `lsof` and `strace` are in the sandbox already. `gcore`, which writes a core file without killing the process, comes with `gdb`, which isn't. Installing it took 6.1 s: ```bash sbx exec $id --timeout 300 "sudo apt-get update -qq >/dev/null 2>&1; sudo apt-get install -y -qq gdb >/dev/null 2>&1; which gcore" ``` ```text /usr/bin/gcore ``` ## Outside: watch for the marker, then preserve the machine We ran this on prod on 2026-10-30 with one `small` sandbox (2 vCPU, 4 GB), after copying the `flaky` folder in with `sbx scp`. The script starts the wrapper with `--detach`, so the loop isn't bound by the 30 s default of `sbx exec`. Then it polls for the marker every 2 s. `sbx exec` returns the remote exit code, so `test -f` works as the check. ```bash #!/bin/bash # watch-and-capture.sh set -u id=$1 sbx exec $id --detach "bash ~/flaky/run-until-fail.sh" > /dev/null until sbx exec $id "test -f ~/failure/FAILED -o -f ~/failure/PASSED"; do sleep 2; done echo "$(date -u +%T) marker: $(sbx exec $id 'cat ~/failure/FAILED ~/failure/PASSED 2>/dev/null')" # 1. Fork: a copy of the disk right now, to debug on without touching the original. fork=$(sbx --json fork $id --name flaky-fork --ttl 3600 | jq -r '.id // .sandbox.id') echo "$(date -u +%T) fork: $fork" # 2. Artifacts out, while the leaked process is still running. sbx exec $id "tar czf ~/failure.tgz -C ~ failure flaky/logs" && sbx scp $id:/home/user/failure.tgz . echo "$(date -u +%T) artifacts: $(du -h failure.tgz | cut -f1)" # 3. A named snapshot of the failing disk, to start new sandboxes from later. sbx snapshot $id flaky-fail > /dev/null echo "$(date -u +%T) named snapshot saved" # 4. Stop: memory and processes are written to disk, the meter stops. sbx stop $id --wait > /dev/null echo "$(date -u +%T) stopped" ``` ```text 21:00:22 marker: run=3 seed=20005 21:00:22 fork: 21:00:25 artifacts: 2.5M 21:00:27 named snapshot saved 21:00:29 stopped ``` The test failed on run 3, at 21:00:21.426 UTC by the `failed-at` file. The script saw the marker within a second, and had the fork, the artifacts, the snapshot and the stop done 9.3 s after it started. The fork line is empty because our first run used a jq filter that didn't match what `sbx --json fork` prints without `--wait`: one `sandbox.forking` line whose `id` is the new fork's id. The script above reads that field; we checked it in a separate run at 21:18 UTC, where it printed the fork's id. The fork in this run was created all the same: `sbx --json list` showed it, `sb_8ab8yij3yj`, with a `createdAt` of 21:00:22.838. ## What the artifacts showed ```bash tar xzf failure.tgz && tail -1 failure/test-output.log && cat failure/server.log ``` ```text urllib.error.URLError: listening seed=22682 pid=1063 21:00:20 "GET / HTTP/1.1" 200 - listening seed=5225 pid=1070 21:00:21 "GET / HTTP/1.1" 200 - listening seed=20005 pid=1072 Current thread 0x00007fb3f42f4080 (most recent call first): File "/usr/lib/python3.12/selectors.py", line 415 in select File "/usr/lib/python3.12/socketserver.py", line 235 in serve_forever File "/home/user/flaky/server.py", line 20 in ``` That is the race in two files. The test got "connection refused". The server for seed 20005 did start, but too late: it has no request logged. The stack dump from `SIGUSR1` shows it idle in `serve_forever`. `ps.txt` and `listening.txt` show the leak: ```text PID PPID ELAPSED STAT RSS COMMAND 259 1 18310 Ssl 10332 /usr/local/bin/sbx-guest serve 1055 259 1 S 3780 \_ bash /home/user/flaky/run-until-fail.sh 1076 1055 0 R 4392 \_ ps -eo pid,ppid,etimes,stat,rss,args --forest … 1072 1 0 S 20612 /usr/bin/python3 server.py LISTEN 0 5 127.0.0.1:8765 0.0.0.0:* users:(("python3",pid=1072,fd=3)) ``` PID 1072's parent is 1: the test exited and left it behind, still holding port 8765. On a CI box, that leaked listener is often what makes the next test fail too. `lsof.txt` listed its open files, including `1w … /home/user/flaky/logs/run.log` and `3u IPv4 … TCP localhost:8765 (LISTEN)`. `gcore` wrote a 12 MB `core.1072`. It compressed well, so the whole tarball was 2.5 MB. A core file of a Python process is most useful for native code. `gdb` on it in the fork gave a C stack that ends in Python's eval loop, without Python frame names: ```bash sbx exec $fork --timeout 60 "cd ~/failure && gdb -batch -ex bt /usr/bin/python3 core.1072 2>&1 | grep -E '^#' | head -4" ``` ```text #0 0x00007fb3f411b764 in __GI___poll (fds=0x7fb3f3f787b0, nfds=1, timeout=454) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00000000006d4935 in ?? () #2 0x0000000000551922 in ?? () #3 0x00000000005499b5 in PyObject_Vectorcall () ``` For Python, the `faulthandler` dump above was the more readable stack. For a Node, Go or C service, the core file is the main artifact. ## What the fork had ```bash sbx exec $fork "cat ~/failure/FAILED; ls ~/failure; curl -s -m 2 127.0.0.1:8765/ || echo connection refused" ``` ```text run=3 seed=20005 core.1072 FAILED failed-at gcore.log listening.txt lsof.txt progress ps.txt server.log server.pid test-output.log connection refused ``` Every file written before the fork was there, including the core file written a second earlier. The leaked server was not: no `server.py` process, and nothing on port 8765. A fork copies the disk and boots fresh, as [the fork guide](/guides/fork-a-running-sandbox-with-memory) shows in more detail. The fork is where you can break things: we replayed the failing seed there 10 times and it failed 8. ## Resume the original: the failing process is still alive ```bash sbx resume $id sbx --json info $id | jq -c '.sandbox | {state, memoryRestored, resumeReason}' sbx exec $id 'pgrep -af "[p]ython3 server.py"; curl -s 127.0.0.1:8765/; curl -s 127.0.0.1:8765/' ``` ```text {"state":"ready","memoryRestored":true,"resumeReason":null} 1072 /usr/bin/python3 server.py token=d8d997d9 hits=1 token=d8d997d9 hits=2 ``` The stop took 2 s and the resume returned in 0.3 s. The leaked server came back as PID 1072. Its in-memory state was intact: `hits=1` on our first request proves the test never reached it. The token exists only in that process's memory. This is the one capture where you can attach `strace -p 1072` or `gdb -p 1072` to the actual failing process. Memory comes back only on a machine with the same CPU type. Otherwise the resume boots from disk: `memoryRestored` is `false`, the files are there and the processes are gone. Pass `sbx resume --require-memory` to wait for a matching machine ([resume](/docs/snapshots#resume)). ## Reproduce from the snapshot in a new sandbox ```bash sbx new --from flaky-fail --name flaky-repro --ttl 3600 sbx exec $repro "cat ~/failure/FAILED; pgrep -af '[p]ython3 server.py' || echo no server process" ``` ```text run=3 seed=20005 no server process ``` The new sandbox was ready 1.2 s after `sbx new` started, on the failing disk with `gdb` already installed. Then we replayed the failing seed and a passing seed 20 times each. When a run fails, the loop kills the leaked server so it can't affect the next run: ```bash sbx exec $repro --timeout 300 "cd ~/flaky && for s in 20005 5225; do printf 'seed %s: ' \$s; for i in \$(seq 20); do FLAKY_SEED=\$s python3 test_server.py >/dev/null 2>&1 && echo -n P || { echo -n F; pkill -f '[p]ython3 server.py'; sleep 0.3; }; done; echo; done" ``` ```text seed 20005: FFFFFFFFPFFFFFFFFFFF seed 5225: PPPPPPPPPPPPPPPPPPPP ``` Seed 20005 failed 19 of 20 times. A replay can still pass sometimes. That seed's startup delay is 0.297 s, and Python's own start time pushes it just past the 0.3 s sleep on most runs, not all. A seed or a recorded input makes a failure likely to come back. It can't remove the timing. Then the fix: wait until the server answers, instead of sleeping. ```python for _ in range(100): # wait until it answers, up to 5 s try: urllib.request.urlopen("http://127.0.0.1:8765/", timeout=0.5); break except OSError: time.sleep(0.05) ``` ```text PPPPPPPPPPPPPPPPPPPP ``` The same seed passed 20 of 20. Every sandbox and the named snapshot were then deleted (`sbx delete --yes`, `sbx snapshot rm flaky-fail`), and `sbx info` returned `not_found` for each. ## Timings Single runs on prod on 2026-10-30, from our client, so they include CLI and network time: | Step | Time | | - | - | | Install `gdb` (for `gcore`) | 6.1 s | | Failure to marker seen outside | under 1 s (2 s poll) | | Watch script, start to stopped (3 test runs, fork, artifacts, snapshot, stop) | 9.3 s | | Named snapshot from the running sandbox | about 2 s | | Stop with `--wait` | about 2 s | | Resume, memory restored | 0.3 s | | New sandbox from the named snapshot to ready | 1.2 s | | 40 replays of the test in the new sandbox | 22.2 s | ## What isn't captured - **The fork and the named snapshot keep no processes.** Only a stop and resume of the same sandbox keeps the live process. If you need the process in a second sandbox, a core file is the closest you get. - **The moment of failure isn't exact.** Our capture started within a second of the failure, but things can change in between: a timer fires, a log rotates, a leaked process exits. Collect the fast-changing state (`ps`, `ss`, `lsof`, stacks) inside the wrapper, before it writes the marker, as above. - **`/tmp` and other tmpfs mounts.** They aren't kept across a cold boot, so a fork or a sandbox from the snapshot won't have them. Write artifacts under `/home/user`. - **Outside services.** A database or API outside the sandbox keeps changing. A snapshot only freezes what's inside the VM. - **The CPU and the clock.** A replay may land on a different machine, and the resumed sandbox's clock is corrected to the real time. Timing-dependent failures reproduce less reliably. Record `lscpu` with the artifacts. - **Core files can hold secrets.** A core file has the process's memory, including any tokens it held. Treat it like a credential. - **Snapshots-off sandboxes.** A sandbox created with `--no-snapshots` can't be forked, saved or resumed ([snapshots off](/docs/snapshots#snapshots-off)). ## FAQ ### Does a fork keep the process that failed? No. In our run, the fork had every file, including a core file written a second before the fork, but the leaked server process was gone. Only stopping and resuming the same sandbox brought the process back, with the same PID and memory. ### Should I fork or stop when a test fails? Both, in that order. Fork first to get a disk copy you can change freely, then stop the original to keep its processes in memory. Resume the original only when you want to inspect the live process. ### How does the outside script know the test failed? The wrapper inside the sandbox writes `~/failure/FAILED` after it collects artifacts. The outside script polls `sbx exec $id "test -f ~/failure/FAILED"`, which exits 0 once the file exists, because `sbx exec` returns the remote command's exit code. ### Can I get a core dump in a sandbox? Yes. Install `gdb` with `sudo apt-get install -y gdb` and run `sudo gcore -o ~/failure/core `. It writes the core file and leaves the process running. Ours was 12 MB for a small Python server. ## Related - [Sandboxes for QA and test automation](/guides/sandboxes-for-qa-and-test-automation) - [Can you fork a running sandbox, memory and all?](/guides/fork-a-running-sandbox-with-memory) - [Pause and resume a sandbox with its memory](/guides/pause-and-resume-a-sandbox-with-memory) - [Snapshots: stop, resume, fork and templates](/docs/snapshots)