# How Do I Make a Long-Running Agent Resume After a Crash?

> Save the agent's progress to a state file after every step, writing it atomically (temp file, fsync, rename), and skip finished steps on start. Run it as an enabled systemd unit with Restart=on-failure. A kill or reboot then redoes only the step in progress, and a stop and resume loses nothing.

Source: https://sandboxer.com/guides/resume-a-long-running-agent-after-a-crash. Updated 2026-10-29.

**Short answer:** Save the agent's progress to a state file after every step, writing it atomically (temp file, fsync, rename), and skip finished steps on start. Run it as an enabled systemd unit with Restart=on-failure. A kill or reboot then redoes only the step in progress, and a stop and resume loses nothing.

Other guides cover keeping the *process* alive: [long jobs with `sbx exec --detach`](/guides/run-a-long-job-in-a-cloud-sandbox), [which start methods survive a resume, reboot or kill](/guides/run-a-service-that-comes-back-after-resume), and [telling a warm resume from a cold boot](/guides/detect-a-cold-boot-after-resume). This page covers the agent's own *progress*: what it writes down, so that a new process knows what's done.

## What we saw

We ran a 40-step worker on a `small` sandbox on prod on 2026-10-29 and broke it five ways. Each step takes 15 seconds; swap in a model call, a test run or a download.

| What happened | Process | Steps redone | What brought it back |
| - | - | - | - |
| `kill -9` on step 3 | New PID, 2 s later | 1 (step 3, from the start) | systemd, `Restart=on-failure` |
| `sbx stop --wait` on step 5, resume 90 s later | Same PID | 0 (step 5 finished mid-step) | Memory restore |
| `sudo reboot` on step 7 | New PID, 1 s after the reboot | 1 (step 7) | The unit is enabled |
| Worker hung (`SIGSTOP`) on step 8 | New PID | 1 (step 8) | Watcher saw a stale heartbeat |
| Auto-stop on step 35 | Same PID | 0 | Watcher resumed the sandbox |

The journal counted 43 step starts for 40 finished steps: one extra for each event that lost the process. Steps that were already checkpointed never ran twice.

## 1. Write the worker so it can restart

The worker reads its task list and its state file, skips what's done, and saves state after every step. The full script, as we ran it:

```python
#!/usr/bin/env python3
"""A resumable worker: works through tasks.txt and checkpoints after every step."""
import json, os, time
from pathlib import Path

DIR = Path("/home/user/agent")
STATE = DIR / "state.json"
BEAT = DIR / "heartbeat"
AWAKE = Path("/home/user/.sandboxer/keep-awake")
TASKS = (DIR / "tasks.txt").read_text().split()

def write_atomic(path, text):
    tmp = path.with_name(path.name + ".tmp")
    with open(tmp, "w") as f:
        f.write(text)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)          # readers see the old file or the new one, never half
    dfd = os.open(path.parent, os.O_RDONLY)
    os.fsync(dfd)                  # make the rename itself durable
    os.close(dfd)

def save(state):
    write_atomic(STATE, json.dumps(state))

def beat():
    write_atomic(BEAT, str(int(time.time())))

def log(msg):
    print(f"pid={os.getpid()} {msg}", flush=True)

state = json.loads(STATE.read_text()) if STATE.exists() else {"done": [], "status": "running"}
if state["status"] == "done":
    log("nothing to do")
    raise SystemExit(0)

AWAKE.parent.mkdir(exist_ok=True)
AWAKE.touch()                      # keep idle pause away while there is work
log(f"start: {len(state['done'])}/{len(TASKS)} done, resuming")
for task in TASKS:
    if task in state["done"]:
        continue                   # finished before the crash: skip it
    state["current"] = task
    save(state)
    log(f"working on {task}")
    for _ in range(15):            # the real step: a model call, a test run, a download…
        time.sleep(1)
        beat()
    write_atomic(DIR / "out" / f"{task}.txt", f"result of {task}\n")
    state["done"].append(task)
    state["current"] = None
    save(state)                    # checkpoint: this task is never redone
    log(f"finished {task}")

state["status"] = "done"
save(state)
AWAKE.unlink(missing_ok=True)      # let the sandbox idle-pause now
log("all tasks done")
```

Four choices make it restartable:

| Choice | Why |
| - | - |
| Temp file, `fsync`, then [`os.replace`](https://docs.python.org/3/library/os.html#os.replace) | A crash mid-write leaves the old state file whole, never a half-written one. The `fsync` calls make the data and the rename survive a reboot |
| Checkpoint after the result is written, not before | A step either counts as done with its output on disk, or runs again |
| Heartbeat from inside the step | A heartbeat thread would keep ticking while the main loop hangs. Beat from the code that does the work |
| `status: done` and an early exit | After the last step, a restart or reboot does nothing |

A step that was running when the process died runs again from its start. So make each step safe to repeat: write its output under a name derived from the task, as above. For side effects outside the sandbox, such as an API call that creates something, record what you sent in the state file before you send it, or use the service's idempotency key, so a repeat doesn't create a second one.

## 2. Run it as a systemd unit

```ini
[Unit]
Description=Resumable agent worker
After=network-online.target
Wants=network-online.target

[Service]
User=user
WorkingDirectory=/home/user/agent
ExecStart=/usr/bin/python3 /home/user/agent/worker.py
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
```

```bash
id=$(sbx new --type small --json | jq -r 'select(.event == "ready") | .id')
sbx exec $id 'mkdir -p ~/agent/out && seq -f task-%02g 1 40 > ~/agent/tasks.txt'
sbx scp worker.py $id:/home/user/agent/
sbx scp agent.service $id:/home/user/agent/
sbx exec $id 'sudo cp ~/agent/agent.service /etc/systemd/system/agent.service && sudo systemctl daemon-reload && sudo systemctl enable --now agent && systemctl is-active agent'
```

```text
active
Created symlink /etc/systemd/system/multi-user.target.wants/agent.service → /etc/systemd/system/agent.service.
```

Use [`Restart=on-failure`](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#Restart=), not `always`, for work that ends. A crash or a `SIGKILL` restarts the worker. A clean exit after the last step doesn't, so the worker isn't started again and again once the work is done.

## 3. Kill it mid-step

Step 3 was running. We killed the worker:

```bash
sbx exec $id 'sudo kill -9 $(systemctl show agent -p MainPID --value); sleep 4; systemctl show agent -p MainPID -p NRestarts'
sbx exec $id 'sudo journalctl -u agent --no-pager -o cat | tail -12'
```

```text
MainPID=713
NRestarts=1

…
pid=671 working on task-03
agent.service: Main process exited, code=killed, status=9/KILL
agent.service: Failed with result 'signal'.
agent.service: Scheduled restart job, restart counter is at 1.
Started agent.service - Resumable agent worker.
pid=713 start: 2/40 done, resuming
pid=713 working on task-03
```

Tasks 1 and 2 were not run again. Task 3 started over.

## 4. Stop and resume the sandbox

Step 5 was 8 seconds in when we ran `sbx stop $id --wait`. We resumed 90 seconds later:

```bash
sbx resume $id
sbx info $id --json | jq -c '.sandbox | {memoryRestored, resumeReason}'
sbx exec $id 'sudo journalctl -u agent --no-pager -o short-iso | tail -3'
```

```text
{"memoryRestored":true,"resumeReason":null}

11:03:44 pid=713 working on task-05
11:05:31 pid=713 finished task-05
11:05:31 pid=713 working on task-06
```

(Lines trimmed to the time and message.) Same PID, no restart, and the step finished where it froze: 107 seconds of wall time for 15 seconds of work. Nothing was redone. If a resume boots from disk instead (`memoryRestored: false`), you're in the reboot case below. The checkpoint covers both.

## 5. Reboot it

Step 7 was running. We ran `sbx exec $id 'sudo reboot'`, then read the new boot's journal:

```bash
sbx exec $id 'uptime -s; sudo journalctl -u agent --no-pager -o short-iso -b'
```

```text
2026-10-29 11:05:54
11:05:55 systemd[1]: Started agent.service - Resumable agent worker.
11:05:55 python3[235]: pid=235 start: 6/40 done, resuming
11:05:55 python3[235]: pid=235 working on task-07
```

The worker was back about a second after the reboot and resumed at step 7. A crash, a reboot and a cold-boot resume all end up here: the files on disk are all that's left.

## 6. Watch it from outside

An orchestrator outside the sandbox needs to answer three questions: is the sandbox running, is the worker making progress, and is it done? This script answers them with `sbx info` and one `sbx exec` per poll:

```bash
#!/bin/bash
# watch-agent.sh <sandbox-id>: wakes the sandbox if it stopped, restarts a hung worker, exits when the work is done.
id=$1
stale=45                                   # seconds without a heartbeat = stuck
while true; do
  read -r state reason < <(sbx info "$id" --json | jq -r '.sandbox | "\(.state) \(.stop.reason // "-")"')
  if [ "$state" = archived ]; then         # stopped: by you, by auto-stop or by idle pause
    sbx resume "$id" >/dev/null
    sbx extend "$id" --hours 1 >/dev/null  # new deadline: now + 1 hour
    restored=$(sbx info "$id" --json | jq -r .sandbox.memoryRestored)
    echo "$(date -u +%T) was stopped ($reason), resumed, memoryRestored=$restored"
    continue
  fi
  read -r status done age < <(sbx exec "$id" \
    'echo $(jq -r ".status, (.done | length)" ~/agent/state.json) $(( $(date +%s) - $(cat ~/agent/heartbeat) ))')
  echo "$(date -u +%T) $status done=$done/40 heartbeat=${age}s ago"
  if [ "$status" = done ]; then exit 0; fi
  if [ "$age" -gt "$stale" ]; then
    echo "$(date -u +%T) stuck: restarting the worker"
    sbx exec "$id" 'sudo systemctl restart agent'
  fi
  sleep 20
done
```

A stopped sandbox has `state` `archived`, and `stop.reason` says why (`user`, `auto_stop` or `idle`). To fake a hang, we froze the worker with `sudo kill -STOP` on step 8. systemd still saw it as running, so only the heartbeat showed the problem:

```text
11:06:27 running done=7/40 heartbeat=3s ago
11:06:47 running done=7/40 heartbeat=23s ago
11:07:07 running done=7/40 heartbeat=43s ago
11:07:28 running done=7/40 heartbeat=64s ago
11:07:28 stuck: restarting the worker
11:07:48 running done=8/40 heartbeat=0s ago
```

Set `stale` above your longest gap between heartbeats, not your longest step.

## 7. Auto-stop and idle pause

Auto-stop is a hard deadline that a running worker doesn't hold off (see [long jobs](/guides/run-a-long-job-in-a-cloud-sandbox)). We cut it to 60 seconds with `sbx extend $id --ttl 60`, with the watcher polling every 20 seconds. Don't rely on polling to keep it up. Set the TTL for the work. We stopped the watcher, and the sandbox auto-stopped soon after. Commands then fail until something resumes it:

```text
{"event":"error","error":"The sandbox is stopped. Resume it first.","code":"machine_not_running","status":409,…}
```

We started the watcher again. It resumed the sandbox, extended it, and ran to the end:

```text
11:14:31 was stopped (auto_stop), resumed, memoryRestored=true
11:14:31 running done=34/40 heartbeat=0s ago
…
11:15:52 done done=40/40 heartbeat=1s ago
```

A worker that sleeps between steps or waits on a model can look idle to the platform. That's why the worker creates `~/.sandboxer/keep-awake` while it has work and deletes it when it's done ([what counts as activity](/docs/long-running#idle-auto-pause-and-wake-on-use)). If the worker dies for good, delete the file yourself, or the sandbox stays up until auto-stop. With idle pause set to 1 minute (`sbx idle $id 1`), the sandbox stayed up through the whole run. After the last step it paused itself 74 seconds after the last activity, with `stop.reason` `idle`.

To hand it more work, wake it with any command. We added three tasks and started the unit again:

```bash
sbx exec $id 'seq -f task-%02g 41 43 >> ~/agent/tasks.txt && jq -c ".status = \"running\"" ~/agent/state.json > ~/agent/s.tmp && mv ~/agent/s.tmp ~/agent/state.json && sudo systemctl start agent && systemctl is-active agent'
```

```text
active
```

The command woke the paused sandbox (`memoryRestored: true`). The journal showed `start: 40/43 done, resuming`, and the worker ran only tasks 41 to 43. One more `sudo reboot` after that printed `nothing to do`, and the unit ended with `Result=success` and no restart.

## FAQ

### Should the state file live in /tmp?

No. Keep it in your home directory or elsewhere on disk. `/tmp` was empty after a cold boot in [our cold-boot test](/guides/detect-a-cold-boot-after-resume), and a checkpoint that disappears restarts the work from step 1.

### Why Restart=on-failure and not Restart=always?

`always` also restarts a worker that exited cleanly after the last step. With `on-failure` and an early exit when the state says `done`, the worker stops for good once the work is finished, even across reboots.

### What happens to the step that was running when the sandbox stopped?

If the resume restores memory, it carries on from where it froze, and nothing is redone. If the sandbox boots from disk, or the process was killed, that one step runs again from its start, so make each step safe to repeat.

### How does the orchestrator know the worker is stuck rather than slow?

Compare the heartbeat's age with a limit longer than any normal gap between beats. Beat from inside the work loop. In our run, a frozen process stayed `active` to systemd, and only the heartbeat age showed it.

## Related

- [Run a long job in a cloud sandbox](/guides/run-a-long-job-in-a-cloud-sandbox)
- [Which way of starting a service survives resume, reboot and kill](/guides/run-a-service-that-comes-back-after-resume)
- [Detect a cold boot after resume](/guides/detect-a-cold-boot-after-resume)
- [Long-running tasks](/docs/long-running)