# Are Sandboxer Sandboxes a Fit for QA and Test Automation Teams?

> Run each suite in a fresh microVM that already has Chrome, Node, Python and Docker. Copy JUnit, screenshots and traces out before you delete it. Repeat suspect tests in a clean VM to measure flake rates, keep a failing VM by stopping it instead of deleting it, and share the build on a private URL.

Source: https://sandboxer.com/guides/sandboxes-for-qa-and-test-automation. Updated 2026-10-27.

**Short answer:** Run each suite in a fresh microVM that already has Chrome, Node, Python and Docker. Copy JUnit, screenshots and traces out before you delete it. Repeat suspect tests in a clean VM to measure flake rates, keep a failing VM by stopping it instead of deleting it, and share the build on a private URL.

## The jobs, and what to use for each

A QA or test automation team asks six things of a sandbox. Each row gives the short answer, the feature that does it, and the guide with a full run on prod.

| Job | Short answer | Sandboxer feature | Worked guide |
| - | - | - | - |
| A clean machine per CI run | Create a VM, copy the checkout in, run the suite, delete it. Nothing from the last run leaks into the next one | `sbx new --ttl`, `sbx exec` (returns the test's exit code), `sbx delete` | [Run each GitHub Actions job in a fresh microVM](/guides/run-each-github-actions-job-in-a-fresh-microvm) |
| Browser end-to-end tests | Playwright's `channel: "chrome"` uses the Chrome that is already installed, so there is no browser download. For Firefox, run a Selenium Grid | Preinstalled Google Chrome, Docker with Compose | [Playwright with real Chrome](/guides/run-browser-use-or-playwright-with-real-chrome), [Selenium Grid](/guides/run-selenium-grid-in-a-sandbox) |
| Tests against real services | Start the real database and queues in the same VM with Compose, not mocks | Docker in every sandbox, `sbx host` per port | [Docker Compose with a URL per service](/guides/run-docker-compose-in-a-sandbox-with-a-url-per-service), [Postgres with pgvector](/guides/run-postgres-with-pgvector-in-a-sandbox) |
| A long suite | Install once, snapshot, then run one shard per sandbox and merge the JUnit files | `sbx snapshot`, `sbx new --from` | [Shard a test suite across sandboxes](/guides/shard-a-test-suite-across-sandboxes) |
| A build for manual QA | Serve the build and share a private HTTPS URL. Requests without its token get `401` | `sbx host --localhost`, idle pause | [Preview URL for every pull request](/guides/get-a-preview-url-for-every-pull-request) |
| Debugging a failure | Stop the failing VM instead of deleting it. Resume it later and look around, or SSH in | `sbx stop --wait`, `sbx resume`, `sbx ssh-config` | [SSH from VS Code or Cursor](/guides/ssh-into-a-cloud-sandbox-from-vs-code-or-cursor), [Pause and resume with memory](/guides/pause-and-resume-a-sandbox-with-memory) |

The example below covers the parts the other guides don't: failure artifacts, flake measurement, keeping a failed VM and handing the build to a person. Scoring models rather than testing a product is covered in [sandboxes for eval and research teams](/guides/sandboxes-for-eval-and-research-teams).

## One run, end to end

We ran this on prod on 2026-10-27 with one `small` sandbox (2 vCPU, 4 GB). The project is a one-page todo app with a planted bug: "Clear completed" removes every item, not only the checked ones. It also has a "Saved" message that appears after a random 200 to 1,500 ms, which makes one test flaky on purpose. There are three Playwright tests.

The config. Two settings do most of the QA work: `channel: "chrome"` skips the browser download, and `trace: "retain-on-failure"` keeps a trace only for the tests that fail.

```typescript
// playwright.config.ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  testDir: "tests",
  fullyParallel: true,
  reporter: [["list"], ["junit", { outputFile: "results/junit.xml" }]],
  use: {
    baseURL: "http://127.0.0.1:4173",
    channel: "chrome",            // the Chrome already in the sandbox
    trace: "retain-on-failure",
    screenshot: "only-on-failure",
  },
  webServer: {
    command: "python3 -m http.server 4173 --bind 127.0.0.1 --directory app",
    url: "http://127.0.0.1:4173",
    reuseExistingServer: true,
    stderr: "ignore",           // http.server logs every request to stderr
  },
});
```

```typescript
// tests/todo.spec.ts
import { test, expect } from "@playwright/test";

test.beforeEach(async ({ page }) => {
  await page.goto("/");
  for (const t of ["write tests", "ship it"]) {
    await page.fill("#new", t);
    await page.click("#add");
  }
});

test("adds items", async ({ page }) => {
  await expect(page.locator("#list li")).toHaveCount(2);
});

test("clear completed keeps active items", async ({ page }) => {
  await page.locator("#list li input").first().check();
  await page.click("#clear");
  await expect(page.locator("#list li")).toHaveText(["ship it"]);
});

test("shows the saved toast", async ({ page }) => {
  await expect(page.locator("#toast")).toBeVisible({ timeout: 1000 });
});
```

The app server binds to `127.0.0.1`. A sandbox has a public IPv6 address. Its firewall is closed except SSH, and binding to loopback keeps the server private even if you open a port later.

### 1. Create, copy in, install

```bash
id=$(sbx --json new --name qa-run-1 --type small --ttl 3600 | jq -r 'select(.event=="ready") | .id')
sbx scp proj.tgz $id:/home/user/
sbx exec $id "tar xzf proj.tgz && ls proj"
sbx exec $id --timeout 300 "cd proj && npm install --no-audit --no-fund && npx playwright --version"
```

```text
app
package.json
playwright.config.ts
tests
…
Version 1.64.1
```

The create took 3.2 s to ready. `npm install` of `@playwright/test` 1.64.1 plus the version check took 2.9 s, because there is no browser to download.

### 2. Run the suite and keep what failed

Run it with output to a file, so you keep the exit code and the full log:

```bash
sbx exec $id --timeout 300 "cd proj && npx playwright test --workers 2 > run.log 2>&1; rc=\$?; grep -E 'passed|failed|✘|✓' run.log; ls test-results/*; echo rc=\$rc"
```

```text
  ✓  2 tests/todo.spec.ts:11:5 › adds items (983ms)
  ✓  3 tests/todo.spec.ts:21:5 › shows the saved toast (1.2s)
  ✘  1 tests/todo.spec.ts:15:5 › clear completed keeps active items (6.5s)
    Error: expect(locator).toHaveText(expected) failed
    test-results/todo-clear-completed-keeps-active-items/test-failed-1.png
  1 failed
  2 passed (7.5s)
error-context.md
test-failed-1.png
trace.zip
rc=1
```

The whole call took 8.5 s from our machine. The failing test produced three files: a screenshot, a page snapshot as Markdown (`error-context.md`) and a 51 KB `trace.zip`. Tar them with the JUnit file and copy them out (3.1 s):

```bash
sbx exec $id "cd proj && tar czf qa-artifacts.tgz results test-results run.log"
sbx scp $id:/home/user/proj/qa-artifacts.tgz .
tar xzf qa-artifacts.tgz && head -2 results/junit.xml
```

```xml
<testsuites id="" name="" tests="3" failures="1" skipped="0" errors="0" time="7.546100999999999">
<testsuite name="todo.spec.ts" timestamp="2026-10-27T11:56:09.627Z" hostname="" tests="3" failures="1" skipped="0" time="8.677" errors="0">
```

Open the trace on your own machine with `npx playwright show-trace trace.zip`. It has the DOM, network and console for every step. Your CI can read the JUnit file directly. If you pipe the run's output (`npx playwright test | head`), the pipe's exit code replaces the test's. Our first attempt did that and reported success on a failing run.

### 3. Measure a flaky test instead of guessing

A clean VM is a good place to put a number on flakiness, because nothing else on the machine competes with the browser. We repeated the toast test 50 times with no retries:

```bash
sbx exec $id --timeout 300 "cd proj && npx playwright test -g toast --repeat-each 50 --workers 2 --retries 0 --reporter=dot --output flake-results > flake.log 2>&1; echo rc=\$?; grep -E '^  [0-9]+ (failed|passed|flaky)' flake.log"
```

```text
rc=1
  6 failed
  44 passed (1.0m)
```

That is a 12% failure rate from 50 runs, in 63 s of wall time. Then the same test with `--retries 2`, 20 times:

```text
rc=0
  1 flaky
  19 passed (27.0s)
```

With retries, the run exits 0 and only the summary line says "flaky". If your CI checks only the exit code, retries hide the problem. Fail the build on, or at least report, the `flaky` count. The fix here belongs in the test: wait for the save to finish (or raise the timeout to the app's real upper bound), not retry.

### 4. Hand the build to a person

Start the app in the background and give it a private URL. `--localhost` is needed because the server listens on `127.0.0.1` only:

```bash
sbx exec $id --detach "cd proj && python3 -m http.server 4173 --bind 127.0.0.1 --directory app"
sbx --json host $id 4173 --localhost --title "QA build"
```

```json
{"ok":true,"type":"port.hosted","sandboxId":"sb_3t6p0kvgxl","port":4173,"url":"https://sleek-elated-bamboo-4173.box.sandboxer.com?_token=…","access":"private","isProtected":true,"localhost":true}
```

From outside, the URL without its token returned `401` in 0.19 s. With the token it returned `302`, which sets a cookie and redirects, then `200` with the app's page. Send the full URL to the tester. Anyone who has it can open the app, so treat it like a password.

### 5. Keep the failing VM, then come back

A deleted VM can't be inspected. A stopped one costs no machine time and comes back as it was:

```bash
sbx stop $id --wait
sbx --json resume $id
sbx --json info $id | jq -c '.sandbox | {state, memoryRestored}'
sbx exec $id "pgrep -af http.server; ls proj/test-results/*/"
```

```text
{"state":"ready","memoryRestored":true}
10137 python3 -m http.server 4173 --bind 127.0.0.1 --directory app
error-context.md
test-failed-1.png
trace.zip
```

The stop took 2.2 s and the resume 1.4 s. The memory came back, so the app server kept its process ID (10137), the failure artifacts were still on disk, and the same private URL answered `200` again. A resume on a machine with a different CPU type boots from disk instead: the files stay, the processes don't, and `memoryRestored` is `false`. Pass `--require-memory` to wait for a matching machine, or start the processes again.

Then we fixed the bug in the VM and ran only that test:

```bash
sbx exec $id "cd proj && sed -i 's|list.querySelectorAll(\"li\")|list.querySelectorAll(\"li:has(input:checked)\")|' app/index.html"
sbx exec $id --timeout 300 "cd proj && npx playwright test --workers 2 -g 'clear completed' --reporter=line"
```

```text
[1/1] tests/todo.spec.ts:15:5 › clear completed keeps active items
  1 passed (1.4s)
```

Finally, `sbx delete $id --yes` took 2.1 s, and `sbx --json info $id` then returned `404 not_found`.

### Timings and cost

Single measurements from our client on 2026-10-27, so they include CLI and network time:

| Step | Time |
| - | - |
| Create a `small` sandbox to ready | 3.2 s |
| `npm install` of `@playwright/test`, no browser download | 2.9 s |
| Three tests, two workers, one failing | 8.5 s |
| Tar and copy artifacts out | 3.1 s |
| 50 repeats of the flaky test | 62.9 s |
| Stop, then resume with memory | 2.2 s, 1.4 s |
| Delete | 2.1 s |

`small` costs $0.018/hour, half the `default` rate of $0.036/hour, metered per second while the sandbox runs. This whole session ran for about 209 s: 209 / 3,600 × $0.018 = about $0.001. Some planning numbers:

- A 5-minute E2E suite on `small`: 300 / 3,600 × $0.018 = $0.0015 a run. 1,000 such runs cost $1.50. On `default`, $3.00.
- A failing VM you stop and keep for a day of debugging costs no machine time while it's stopped. Its disk counts toward your plan's included storage until you delete it ([storage](/docs/billing#storage)).
- A manual QA build left running for an 8-hour workday on `small`: 8 × $0.018 = $0.144. An idle sandbox pauses itself and wakes on the next request to its URL ([idle pause](/guides/idle-auto-pause-and-wake-on-use)), so the real number is usually lower.

## Limits to plan around

- **Concurrency and starts.** Every create, resume and fork counts as one start. The free trial allows 2 sandboxes at once (`small` and `default` only, TTL up to 8 hours). The $20 plan allows 100 at once and 12 starts per minute, and the $100 plan 300 and 30 ([plans](/docs/billing#plans)). A big matrix needs a semaphore and must pace its starts: see [rate limits and concurrency caps](/guides/handle-rate-limits-and-concurrency-caps).
- **CPU only, and no Safari.** There is no GPU. Chrome is preinstalled; Firefox runs through the Selenium Grid images; WebKit and Safari were not part of our tests. Mobile web is Chrome's device emulation, and native Android is the [emulator in a VM](/guides/run-the-android-emulator-in-a-cloud-vm).
- **Timing-sensitive tests.** In our runs, `small` gave 2 vCPU. Fifty browser runs on two workers took a minute. Use `default` (4 vCPU) or shard if your suite is CPU-bound, and don't compare timings across sizes or CPU models without recording `lscpu`.
- **`sbx exec` times out.** It waits 30 s by default and up to 600 s with `--timeout`. For longer suites, start with `--detach` and poll, or follow with `--attach` ([long-running jobs](/docs/long-running)).
- **Moving to a smaller size isn't supported yet.** Create a new, smaller sandbox and copy what you need into it.
- **The network is live.** A test that calls a real third-party API from the VM gets real answers and real rate limits. Pin dependencies, and stub external services you don't own.

## FAQ

### Do I need to install browsers for Playwright?

Not for Chrome. Set `channel: "chrome"` and Playwright uses the Google Chrome already in the sandbox. Our install of `@playwright/test` took 2.9 s with no `npx playwright install` step.

### How do I get screenshots and traces out of the sandbox?

Set `trace: "retain-on-failure"` and `screenshot: "only-on-failure"`, tar `test-results` and your JUnit file, and copy the archive out with `sbx scp <id>:/home/user/proj/qa-artifacts.tgz .`. Delete the sandbox only after the copy succeeds.

### How do I find out if a test is flaky?

Repeat it in a clean VM with `--repeat-each 50 --retries 0` and count the failures. Our planted flaky test failed 6 of 50 times. With `--retries 2` the same kind of run exited 0 and only reported "1 flaky", so read that count, not just the exit code.

### Can a tester see the app without a Sandboxer account?

Yes. `sbx host` gives a private HTTPS URL with a `_token`. Anyone who has the full URL can open it; without the token it returns `401`. Use `--public` only for builds that may be seen by anyone.

## Related

- [Run each GitHub Actions job in a fresh microVM](/guides/run-each-github-actions-job-in-a-fresh-microvm)
- [Shard a test suite across sandboxes](/guides/shard-a-test-suite-across-sandboxes)
- [Run Browser Use or Playwright with real Chrome](/guides/run-browser-use-or-playwright-with-real-chrome)
- [Host a port on an HTTPS URL](/docs/hosting)