Skip to content
sandboxer
Start free trial
Guides · Who it's for

Are Sandboxer Sandboxes a Fit for QA and Test Automation Teams?

Updated October 27, 2026 .md

Short answer: Run each suite in a fresh microVM that already has Chrome, Node, Python and Docker. Copy JUnit, screenshots and traces out before you delete it. Repeat suspect tests in a clean VM to measure flake rates, keep a failing VM by stopping it instead of deleting it, and share the build on a private URL.

The jobs, and what to use for each#

A QA or test automation team asks six things of a sandbox. Each row gives the short answer, the feature that does it, and the guide with a full run on prod.

Job Short answer Sandboxer feature Worked guide
A clean machine per CI run Create a VM, copy the checkout in, run the suite, delete it. Nothing from the last run leaks into the next one sbx new --ttl, sbx exec (returns the test's exit code), sbx delete Run each GitHub Actions job in a fresh microVM
Browser end-to-end tests Playwright's channel: "chrome" uses the Chrome that is already installed, so there is no browser download. For Firefox, run a Selenium Grid Preinstalled Google Chrome, Docker with Compose Playwright with real Chrome, Selenium Grid
Tests against real services Start the real database and queues in the same VM with Compose, not mocks Docker in every sandbox, sbx host per port Docker Compose with a URL per service, Postgres with pgvector
A long suite Install once, snapshot, then run one shard per sandbox and merge the JUnit files sbx snapshot, sbx new --from Shard a test suite across sandboxes
A build for manual QA Serve the build and share a private HTTPS URL. Requests without its token get 401 sbx host --localhost, idle pause Preview URL for every pull request
Debugging a failure Stop the failing VM instead of deleting it. Resume it later and look around, or SSH in sbx stop --wait, sbx resume, sbx ssh-config SSH from VS Code or Cursor, Pause and resume with memory

The example below covers the parts the other guides don't: failure artifacts, flake measurement, keeping a failed VM and handing the build to a person. Scoring models rather than testing a product is covered in sandboxes for eval and research teams.

One run, end to end#

We ran this on prod on 2026-10-27 with one small sandbox (2 vCPU, 4 GB). The project is a one-page todo app with a planted bug: "Clear completed" removes every item, not only the checked ones. It also has a "Saved" message that appears after a random 200 to 1,500 ms, which makes one test flaky on purpose. There are three Playwright tests.

The config. Two settings do most of the QA work: channel: "chrome" skips the browser download, and trace: "retain-on-failure" keeps a trace only for the tests that fail.

// playwright.config.ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  testDir: "tests",
  fullyParallel: true,
  reporter: [["list"], ["junit", { outputFile: "results/junit.xml" }]],
  use: {
    baseURL: "http://127.0.0.1:4173",
    channel: "chrome",            // the Chrome already in the sandbox
    trace: "retain-on-failure",
    screenshot: "only-on-failure",
  },
  webServer: {
    command: "python3 -m http.server 4173 --bind 127.0.0.1 --directory app",
    url: "http://127.0.0.1:4173",
    reuseExistingServer: true,
    stderr: "ignore",           // http.server logs every request to stderr
  },
});
// tests/todo.spec.ts
import { test, expect } from "@playwright/test";

test.beforeEach(async ({ page }) => {
  await page.goto("/");
  for (const t of ["write tests", "ship it"]) {
    await page.fill("#new", t);
    await page.click("#add");
  }
});

test("adds items", async ({ page }) => {
  await expect(page.locator("#list li")).toHaveCount(2);
});

test("clear completed keeps active items", async ({ page }) => {
  await page.locator("#list li input").first().check();
  await page.click("#clear");
  await expect(page.locator("#list li")).toHaveText(["ship it"]);
});

test("shows the saved toast", async ({ page }) => {
  await expect(page.locator("#toast")).toBeVisible({ timeout: 1000 });
});

The app server binds to 127.0.0.1. A sandbox has a public IPv6 address. Its firewall is closed except SSH, and binding to loopback keeps the server private even if you open a port later.

1. Create, copy in, install#

id=$(sbx --json new --name qa-run-1 --type small --ttl 3600 | jq -r 'select(.event=="ready") | .id')
sbx scp proj.tgz $id:/home/user/
sbx exec $id "tar xzf proj.tgz && ls proj"
sbx exec $id --timeout 300 "cd proj && npm install --no-audit --no-fund && npx playwright --version"
app
package.json
playwright.config.ts
tests
…
Version 1.64.1

The create took 3.2 s to ready. npm install of @playwright/test 1.64.1 plus the version check took 2.9 s, because there is no browser to download.

2. Run the suite and keep what failed#

Run it with output to a file, so you keep the exit code and the full log:

sbx exec $id --timeout 300 "cd proj && npx playwright test --workers 2 > run.log 2>&1; rc=\$?; grep -E 'passed|failed|✘|✓' run.log; ls test-results/*; echo rc=\$rc"
  ✓  2 tests/todo.spec.ts:11:5 › adds items (983ms)
  ✓  3 tests/todo.spec.ts:21:5 › shows the saved toast (1.2s)
  ✘  1 tests/todo.spec.ts:15:5 › clear completed keeps active items (6.5s)
    Error: expect(locator).toHaveText(expected) failed
    test-results/todo-clear-completed-keeps-active-items/test-failed-1.png
  1 failed
  2 passed (7.5s)
error-context.md
test-failed-1.png
trace.zip
rc=1

The whole call took 8.5 s from our machine. The failing test produced three files: a screenshot, a page snapshot as Markdown (error-context.md) and a 51 KB trace.zip. Tar them with the JUnit file and copy them out (3.1 s):

sbx exec $id "cd proj && tar czf qa-artifacts.tgz results test-results run.log"
sbx scp $id:/home/user/proj/qa-artifacts.tgz .
tar xzf qa-artifacts.tgz && head -2 results/junit.xml
<testsuites id="" name="" tests="3" failures="1" skipped="0" errors="0" time="7.546100999999999">
<testsuite name="todo.spec.ts" timestamp="2026-10-27T11:56:09.627Z" hostname="" tests="3" failures="1" skipped="0" time="8.677" errors="0">

Open the trace on your own machine with npx playwright show-trace trace.zip. It has the DOM, network and console for every step. Your CI can read the JUnit file directly. If you pipe the run's output (npx playwright test | head), the pipe's exit code replaces the test's. Our first attempt did that and reported success on a failing run.

3. Measure a flaky test instead of guessing#

A clean VM is a good place to put a number on flakiness, because nothing else on the machine competes with the browser. We repeated the toast test 50 times with no retries:

sbx exec $id --timeout 300 "cd proj && npx playwright test -g toast --repeat-each 50 --workers 2 --retries 0 --reporter=dot --output flake-results > flake.log 2>&1; echo rc=\$?; grep -E '^  [0-9]+ (failed|passed|flaky)' flake.log"
rc=1
  6 failed
  44 passed (1.0m)

That is a 12% failure rate from 50 runs, in 63 s of wall time. Then the same test with --retries 2, 20 times:

rc=0
  1 flaky
  19 passed (27.0s)

With retries, the run exits 0 and only the summary line says "flaky". If your CI checks only the exit code, retries hide the problem. Fail the build on, or at least report, the flaky count. The fix here belongs in the test: wait for the save to finish (or raise the timeout to the app's real upper bound), not retry.

4. Hand the build to a person#

Start the app in the background and give it a private URL. --localhost is needed because the server listens on 127.0.0.1 only:

sbx exec $id --detach "cd proj && python3 -m http.server 4173 --bind 127.0.0.1 --directory app"
sbx --json host $id 4173 --localhost --title "QA build"
{"ok":true,"type":"port.hosted","sandboxId":"sb_3t6p0kvgxl","port":4173,"url":"https://sleek-elated-bamboo-4173.box.sandboxer.com?_token=…","access":"private","isProtected":true,"localhost":true}

From outside, the URL without its token returned 401 in 0.19 s. With the token it returned 302, which sets a cookie and redirects, then 200 with the app's page. Send the full URL to the tester. Anyone who has it can open the app, so treat it like a password.

5. Keep the failing VM, then come back#

A deleted VM can't be inspected. A stopped one costs no machine time and comes back as it was:

sbx stop $id --wait
sbx --json resume $id
sbx --json info $id | jq -c '.sandbox | {state, memoryRestored}'
sbx exec $id "pgrep -af http.server; ls proj/test-results/*/"
{"state":"ready","memoryRestored":true}
10137 python3 -m http.server 4173 --bind 127.0.0.1 --directory app
error-context.md
test-failed-1.png
trace.zip

The stop took 2.2 s and the resume 1.4 s. The memory came back, so the app server kept its process ID (10137), the failure artifacts were still on disk, and the same private URL answered 200 again. A resume on a machine with a different CPU type boots from disk instead: the files stay, the processes don't, and memoryRestored is false. Pass --require-memory to wait for a matching machine, or start the processes again.

Then we fixed the bug in the VM and ran only that test:

sbx exec $id "cd proj && sed -i 's|list.querySelectorAll(\"li\")|list.querySelectorAll(\"li:has(input:checked)\")|' app/index.html"
sbx exec $id --timeout 300 "cd proj && npx playwright test --workers 2 -g 'clear completed' --reporter=line"
[1/1] tests/todo.spec.ts:15:5 › clear completed keeps active items
  1 passed (1.4s)

Finally, sbx delete $id --yes took 2.1 s, and sbx --json info $id then returned 404 not_found.

Timings and cost#

Single measurements from our client on 2026-10-27, so they include CLI and network time:

Step Time
Create a small sandbox to ready 3.2 s
npm install of @playwright/test, no browser download 2.9 s
Three tests, two workers, one failing 8.5 s
Tar and copy artifacts out 3.1 s
50 repeats of the flaky test 62.9 s
Stop, then resume with memory 2.2 s, 1.4 s
Delete 2.1 s

small costs $0.018/hour, half the default rate of $0.036/hour, metered per second while the sandbox runs. This whole session ran for about 209 s: 209 / 3,600 × $0.018 = about $0.001. Some planning numbers:

  • A 5-minute E2E suite on small: 300 / 3,600 × $0.018 = $0.0015 a run. 1,000 such runs cost $1.50. On default, $3.00.
  • A failing VM you stop and keep for a day of debugging costs no machine time while it's stopped. Its disk counts toward your plan's included storage until you delete it (storage).
  • A manual QA build left running for an 8-hour workday on small: 8 × $0.018 = $0.144. An idle sandbox pauses itself and wakes on the next request to its URL (idle pause), so the real number is usually lower.

Limits to plan around#

  • Concurrency and starts. Every create, resume and fork counts as one start. The free trial allows 2 sandboxes at once (small and default only, TTL up to 8 hours). The $20 plan allows 100 at once and 12 starts per minute, and the $100 plan 300 and 30 (plans). A big matrix needs a semaphore and must pace its starts: see rate limits and concurrency caps.
  • CPU only, and no Safari. There is no GPU. Chrome is preinstalled; Firefox runs through the Selenium Grid images; WebKit and Safari were not part of our tests. Mobile web is Chrome's device emulation, and native Android is the emulator in a VM.
  • Timing-sensitive tests. In our runs, small gave 2 vCPU. Fifty browser runs on two workers took a minute. Use default (4 vCPU) or shard if your suite is CPU-bound, and don't compare timings across sizes or CPU models without recording lscpu.
  • sbx exec times out. It waits 30 s by default and up to 600 s with --timeout. For longer suites, start with --detach and poll, or follow with --attach (long-running jobs).
  • Moving to a smaller size isn't supported yet. Create a new, smaller sandbox and copy what you need into it.
  • The network is live. A test that calls a real third-party API from the VM gets real answers and real rate limits. Pin dependencies, and stub external services you don't own.

FAQ#

Do I need to install browsers for Playwright?#

Not for Chrome. Set channel: "chrome" and Playwright uses the Google Chrome already in the sandbox. Our install of @playwright/test took 2.9 s with no npx playwright install step.

How do I get screenshots and traces out of the sandbox?#

Set trace: "retain-on-failure" and screenshot: "only-on-failure", tar test-results and your JUnit file, and copy the archive out with sbx scp <id>:/home/user/proj/qa-artifacts.tgz .. Delete the sandbox only after the copy succeeds.

How do I find out if a test is flaky?#

Repeat it in a clean VM with --repeat-each 50 --retries 0 and count the failures. Our planted flaky test failed 6 of 50 times. With --retries 2 the same kind of run exited 0 and only reported "1 flaky", so read that count, not just the exit code.

Can a tester see the app without a Sandboxer account?#

Yes. sbx host gives a private HTTPS URL with a _token. Anyone who has the full URL can open it; without the token it returns 401. Use --public only for builds that may be seen by anyone.

More guides

Top AI sandbox platforms for code execution in 2026, ranked

For running AI-written code we rank Sandboxer, Freestyle and E2B first: each gives code its own VM with an API or SDK and can pause with memory. Pick Daytona for Windows or GPUs, Modal or Northflank for GPUs at scale, E2B or Northflank to run in your own cloud, and Vercel Sandbox if your app lives on Vercel.

Read the guide
Best sandboxes for running coding agents (Claude Code, Codex, Gemini CLI) in 2026, ranked

For coding agents we rank Sandboxer first: a full Ubuntu VM with Claude Code and Codex preinstalled, Docker, and stop-and-resume that keeps the agent's session in memory. Freestyle and Amp Orbs also keep memory across pauses. Pick E2B for prebuilt agent templates on open source, Docker Sandboxes for free local runs, and Codespaces if you live in GitHub.

Read the guide
Best Daytona sandbox alternatives in 2026, ranked

The best Daytona alternatives depend on why you're switching. For a full Linux microVM on every sandbox with memory-saving stop and Docker, we rank Sandboxer first. E2B suits open source and self-hosting, Modal massive short-job fleets, Northflank bring-your-own-cloud with GPUs, and Vercel Sandbox apps on Vercel.

Read the guide
Best E2B alternatives in 2026, ranked

If you're leaving E2B for longer sessions or a fuller machine, we rank Sandboxer first, then Freestyle: both give each sandbox a full Linux VM that keeps memory across pauses, with Docker and no short session cap. Pick Daytona for Windows or GPUs, Northflank for bring-your-own-cloud, and stay on E2B if you need open source.

Read the guide