# Run Firecracker microVMs inside a cloud VM with nested KVM

> Use a cloud VM that gives you /dev/kvm. In a Sandboxer large sandbox, the official Firecracker v1.18.0 binary runs as the normal user without sudo. Check its SHA-256, fetch the kernel and Ubuntu rootfs from Firecracker's CI as its getting-started guide shows, and boot through the API socket. On 2026-11-18 a guest reached a login prompt in 2.2–2.7 s.

Source: https://sandboxer.com/guides/run-firecracker-inside-a-cloud-sandbox-with-nested-kvm. Updated 2026-11-18.

**Short answer:** Use a cloud VM that gives you /dev/kvm. In a Sandboxer large sandbox, the official Firecracker v1.18.0 binary runs as the normal user without sudo. Check its SHA-256, fetch the kernel and Ubuntu rootfs from Firecracker's CI as its getting-started guide shows, and boot through the API socket. On 2026-11-18 a guest reached a login prompt in 2.2–2.7 s.

Part of [Sandbox Isolation Compared: Containers, gVisor, Kata, MicroVMs, Wasm](/guides/sandbox-isolation-technologies-compared) and [Docker, Kubernetes and Nested VMs in a Cloud Sandbox](/guides/docker-and-virtualization-in-a-cloud-sandbox).

*Every command below ran on a `large` Sandboxer sandbox (8 vCPU, 16 GB) on 2026-11-18. Firecracker facts come from its own [getting-started guide](https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md) and repository, read the same day. To compare the two hypervisors themselves, see [Firecracker vs Cloud Hypervisor](/guides/firecracker-vs-cloud-hypervisor).*

## At a glance

| | Result on a `large` sandbox, 2026-11-18 |
| - | - |
| `/dev/kvm` | Present, `root:kvm` mode `0660`; `user` is in `kvm`, so no `sudo` and no `setfacl` |
| Firecracker | v1.18.0 release binary; the `.sha256.txt` checksum matched |
| Guest | Firecracker CI kernel 6.18.62 and Ubuntu 24.04 rootfs, 2 vCPU, 1024 MiB |
| Boot, InstanceStart to login prompt (5 runs each) | 2.62–2.71 s with the getting-started boot args; 2.20–2.32 s with Firecracker's default i8042 args added |
| Networking | A tap device with NAT, as in the docs: SSH, DNS and HTTPS from the guest all worked |
| Jailer | Worked under `sudo` with cgroup v2 limits |
| Firecracker snapshots | Full snapshot of a 1 GiB guest in 1.26 s; restore in a new process in 29 ms |
| 20 microVMs at once | All at a login prompt in 12.4 s, 2.0 GB of RSS in total |
| Sandbox stop and resume | A running inner microVM came back with its memory |

## Why you need a VM that passes through KVM

Firecracker [requires](https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md#kvm) read/write access to `/dev/kvm`. Its getting-started guide uses a bare-metal (`.metal`) cloud instance, because that cloud only offers KVM on bare-metal instance types. Its [development setup doc](https://github.com/firecracker-microvm/firecracker/blob/main/docs/dev-machine-setup.md) says the other way in: "a virtual machine that supports nested virtualization".

A Sandboxer sandbox is a VM with `/dev/kvm` available inside ([Machines](/docs/machines)). That makes it a nested setup: Firecracker's guest is a VM inside your sandbox's VM.

```bash
sbx new --type large --ttl 5400 --no-env
sbx exec sb_ropzl20q9c 'ls -l /dev/kvm; id -nG; [ -r /dev/kvm ] && [ -w /dev/kvm ] && echo OK || echo FAIL; grep -m1 -o -E "vmx|svm" /proc/cpuinfo; ls /sys/module | grep ^kvm'
```

```text
crw-rw---- 1 root kvm 10, 232 Nov  8 15:48 /dev/kvm
user sudo docker kvm systemd-journal
OK
svm
kvm
kvm_amd
kvm_intel
```

The access check is the one from Firecracker's docs. Their `lsmod | grep kvm` check prints nothing here, because KVM is built into the sandbox kernel rather than loaded as a module. `/sys/module` shows it's there. Our sandbox ran on an AMD EPYC 9254, hence `svm`.

We used `large`. `xlarge` (16 vCPU, 32 GB) isn't on every plan, and our test account's plan got `403 machine_class_plan_required` for it; see [Machines](/docs/machines). `--ttl` stops the sandbox after that many seconds ([auto-stop](/docs/long-running)); delete it with `sbx delete` when you're done.

## Download Firecracker and check the checksum

The [v1.18.0 release](https://github.com/firecracker-microvm/firecracker/releases/tag/v1.18.0) publishes a `.sha256.txt` file next to each tarball. This is the getting-started download with the checksum step added. Run it in the sandbox (`sbx ssh sb_ropzl20q9c`, or wrap it in `sbx exec`):

```bash
mkdir -p ~/fc && cd ~/fc
ARCH=$(uname -m)
latest=$(basename $(curl -fsSLI -o /dev/null -w %{url_effective} https://github.com/firecracker-microvm/firecracker/releases/latest))
base=https://github.com/firecracker-microvm/firecracker/releases/download/$latest
curl -fsSLO $base/firecracker-$latest-$ARCH.tgz
curl -fsSLO $base/firecracker-$latest-$ARCH.tgz.sha256.txt
sha256sum -c firecracker-$latest-$ARCH.tgz.sha256.txt
tar -xzf firecracker-$latest-$ARCH.tgz
cp release-$latest-$ARCH/firecracker-$latest-$ARCH firecracker
cp release-$latest-$ARCH/jailer-$latest-$ARCH jailer
./firecracker --version | head -1
```

```text
firecracker-v1.18.0-x86_64.tgz: OK
Firecracker v1.18.0
```

The tarball also has a `SHA256SUMS` file for the binaries inside. `sha256sum -c --ignore-missing SHA256SUMS` in the release folder said `OK` for `firecracker-v1.18.0-x86_64` and `jailer-v1.18.0-x86_64`.

## Get a guest kernel and rootfs

Firecracker's guide ([Getting a rootfs and Guest Kernel Image](https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md#getting-a-rootfs-and-guest-kernel-image)) takes the newest kernel and Ubuntu squashfs from its CI bucket. It then adds an SSH key and converts the image to ext4. We ran its script unchanged, apart from quieter flags and a `mkdir -p` for `/root/.ssh`. It needs `unsquashfs`, which isn't preinstalled:

```bash
sudo apt-get update -q && sudo apt-get install -y -q squashfs-tools
```

```text
CI prefix: firecracker-ci/20261118-6d7cfa9c6b53-0/
27M vmlinux-6.18.62
95M ubuntu-24.04.squashfs.upstream
1.0G ubuntu-24.04.ext4
```

The CI prefix is dated, so you get whatever Firecracker's CI built most recently. Ours was built the same day. Save the kernel and ext4 under `/home/user` so they survive a stop.

## Boot a microVM through the API socket

Firecracker's guide runs it with `sudo` in two terminals. Here it runs as `user`, because `user` can open `/dev/kvm`. This script starts Firecracker, sends the configuration over the socket, starts the VM and waits for the serial console's login prompt. The guest's serial console is Firecracker's stdin and stdout, so a FIFO on stdin lets the script type into it:

```bash
# ~/fc/boot.sh [net] ["command to type on the guest console"]
cd ~/fc
SOCK=~/fc/fc.sock
rm -f "$SOCK" serial.log fc.log in.fifo
mkfifo in.fifo
exec 3<>in.fifo   # serial console input
t0=$(date +%s%N)
./firecracker --api-sock "$SOCK" --enable-pci < in.fifo > serial.log 2>&1 &
FC=$!
until [ -S "$SOCK" ]; do sleep 0.01; done
api() { curl -fsS --unix-socket "$SOCK" -X PUT "http://localhost/$1" -H 'Content-Type: application/json' -d "$2"; }
api logger '{"log_path":"/home/user/fc/fc.log","level":"Info"}'
api machine-config '{"vcpu_count":2,"mem_size_mib":1024}'
api boot-source "{\"kernel_image_path\":\"$(ls ~/fc/vmlinux-* | tail -1)\",\"boot_args\":\"${BOOT_ARGS:-console=ttyS0 reboot=k panic=1}\"}"
api drives/rootfs '{"drive_id":"rootfs","path_on_host":"/home/user/fc/ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}'
if [ "${1:-}" = net ]; then
  api network-interfaces/net1 '{"iface_id":"net1","guest_mac":"06:00:AC:10:00:02","host_dev_name":"tap0"}'
fi
t1=$(date +%s%N)
api actions '{"action_type":"InstanceStart"}'
until grep -aq "login:" serial.log; do sleep 0.02; done
t2=$(date +%s%N)
echo "process start to login prompt: $(( (t2-t0)/1000000 )) ms; InstanceStart to login prompt: $(( (t2-t1)/1000000 )) ms"
echo $FC > fc.pid
if [ -n "${2:-}" ]; then
  sleep 1; printf '%s\n' "$2" >&3
  sleep ${WAIT:-2}; printf 'reboot\n' >&3
  wait $FC; echo "firecracker exited: $?"
fi
```

The CI rootfs logs `root` in on the serial console automatically. So the second argument runs as root in the guest:

```bash
bash ~/fc/boot.sh nonet 'echo MARK; uname -r; nproc; free -m | head -2; cat /proc/cmdline; systemd-analyze | head -1'
grep -a -E "Linux version|Ubuntu 24|login:" ~/fc/serial.log | head -4
grep -a -A6 MARK ~/fc/serial.log
```

```text
process start to login prompt: 2720 ms; InstanceStart to login prompt: 2668 ms
firecracker exited: 0
[    0.000000] Linux version 6.18.62+ (root@23ebf1a82c3e) (gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, …) #1 SMP PREEMPT_DYNAMIC Wed Nov 18 03:23:17 UTC 2026
Welcome to Ubuntu 24.04.5 LTS!
Ubuntu 24.04.5 LTS ubuntu-fc-uvm ttyS0
ubuntu-fc-uvm login: root (automatic login)
MARK
6.18.62+
2
               total        used        free      shared  buff/cache   available
Mem:             984         141         888           2          51         843
console=ttyS0 reboot=k panic=1 root=/dev/vda rw
Startup finished in 1.281s (kernel) + 922ms (userspace) = 2.204s
```

`reboot` in the guest ends the Firecracker process with exit code 0, as [the docs say](https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md#starting-firecracker): Firecracker doesn't implement guest power management.

## Boot time, and the boot args that cut it

Five boots with the getting-started boot args, then five with Firecracker's own default i8042 flags added:

```bash
cd ~/fc && for i in 1 2 3 4 5; do bash boot.sh nonet "systemd-analyze | head -1" | head -1; grep -a -m1 "Startup finished" serial.log; done
export BOOT_ARGS="console=ttyS0 reboot=k panic=1 i8042.noaux i8042.nomux i8042.dumbkbd swiotlb=noforce"
# …same loop again
```

| Guest: CI kernel 6.18.62, Ubuntu 24.04, 2 vCPU, 1024 MiB, PCI on; `large` sandbox, 2026-11-18 | InstanceStart to login prompt | systemd: kernel + userspace |
| - | - | - |
| `console=ttyS0 reboot=k panic=1` (getting-started, 5 runs) | 2.62–2.71 s | 1.27–1.31 s + 0.82–0.90 s |
| Same, plus `i8042.noaux i8042.nomux i8042.dumbkbd swiotlb=noforce` (5 runs) | 2.20–2.32 s | 0.82–0.91 s + 0.79–0.90 s |

Here's why the flags matter. Firecracker's [kernel policy](https://github.com/firecracker-microvm/firecracker/blob/main/docs/kernel-policy.md#kernel-command-line-parameters) lists a default command line with the i8042 flags. It notes they "save boot time". If you send your own `boot_args`, you replace that list. With the getting-started args, the guest kernel log showed a 0.41 s gap before `input: AT Raw Set 2 keyboard`. With the flags, the gap was gone.

These are times inside a nested VM on one machine type on one day. Firecracker's [integration tests](https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md#running-the-integration-test-suite) check its published performance targets as measured on bare-metal (`.metal`) cloud instances. Its docs say the startup-time test may fail elsewhere. Don't read our numbers as what Firecracker does on bare metal.

## Give the guest a network with a tap device

This is the network part of the getting-started guide, unchanged. It needs `sudo` for `ip` and `iptables`:

```bash
TAP_DEV="tap0"; TAP_IP="172.16.0.1"; MASK_SHORT="/30"
sudo ip link del "$TAP_DEV" 2> /dev/null || true
sudo ip tuntap add dev "$TAP_DEV" mode tap user "$(id -u)" group "$(id -g)"
sudo ip addr add "${TAP_IP}${MASK_SHORT}" dev "$TAP_DEV"
sudo ip link set dev "$TAP_DEV" up
sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
sudo iptables -P FORWARD ACCEPT
HOST_IFACE=$(ip -j route list default | jq -r '.[0].dev')   # eth0 in a sandbox
sudo iptables -t nat -D POSTROUTING -o "$HOST_IFACE" -j MASQUERADE 2>/dev/null || true
sudo iptables -t nat -A POSTROUTING -o "$HOST_IFACE" -j MASQUERADE
```

Then boot with `bash boot.sh net` and SSH in with the key the rootfs script made. We set the guest's default route and nameserver as the docs do:

```bash
SSH="ssh -q -i ~/fc/ubuntu-24.04.id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@172.16.0.2"
$SSH "ip route add default via 172.16.0.1 dev eth0; echo 'nameserver 8.8.8.8' > /etc/resolv.conf"
$SSH "ip -br addr show eth0; ping -c 2 -W 2 172.16.0.1 | tail -1; getent hosts github.com; curl -sS -m 15 -o /dev/null -w '%{http_code} %{time_total}s\n' https://github.com/"
```

```text
ssh ready 334 ms after login prompt
eth0             UP             172.16.0.2/30 fe80::400:acff:fe10:2/64
rtt min/avg/max/mdev = 0.181/0.242/0.303/0.061 ms
140.82.114.4    github.com
200 0.075411s
```

It worked. DNS also worked through the sandbox's own resolver (`nameserver 10.202.0.1`, from the sandbox's `/etc/resolv.conf`). Each SSH command printed a harmless `setlocale: LC_ALL` warning, because the CI rootfs has no `en_US.UTF-8` locale. The guest has only the NAT'd private address `172.16.0.2`. Processes in the sandbox can reach it directly. [`sbx host`](/docs/hosting) gives an HTTPS URL to a port on the sandbox itself, so a guest service would need a port forward in between; we didn't test that.

## The jailer works, under sudo

Firecracker's docs say production use runs it inside the [jailer](https://github.com/firecracker-microvm/firecracker/blob/main/docs/jailer.md), and the getting-started guide skips it. The jailer needs root, so it runs under `sudo`. Put the kernel and rootfs inside the chroot, which the jailer docs say you must do. Then give it cgroup v2 limits:

```bash
ID=fc-jail-1; JROOT=/srv/jailer/firecracker/$ID/root
sudo mkdir -p $JROOT && sudo cp vmlinux-6.18.62 ubuntu-24.04.ext4 $JROOT/ && sudo chown 1000:1000 $JROOT/vmlinux-6.18.62 $JROOT/ubuntu-24.04.ext4
sudo ./jailer --id $ID --exec-file ~/fc/firecracker --uid 1000 --gid 1000 --cgroup-version 2 \
  --cgroup memory.max=1610612736 --cgroup cpu.max="200000 100000" --daemonize -- --enable-pci
# configure through $JROOT/run/firecracker.socket with sudo curl, paths relative to the chroot (/vmlinux-6.18.62), then InstanceStart
sudo curl -fsS --unix-socket $JROOT/run/firecracker.socket http://localhost/
```

```text
{"id":"fc-jail-1","state":"Running","vmm_version":"1.18.0","app_name":"Firecracker"}
cgroup: /firecracker/fc-jail-1
1610612736
200000 100000
```

The jailer created `/dev/kvm`, `/dev/net/tun`, `/dev/urandom` and `/dev/userfaultfd` nodes inside the chroot. It ran Firecracker as uid 1000 in its own cgroup with both limits applied. The sandbox kernel uses cgroup v2 (`cpuset cpu io memory pids` controllers). With `--daemonize`, the serial console isn't captured, so we checked this one through the API state only.

## Firecracker snapshots inside the sandbox

Developing snapshot and restore tooling is a common reason to want a Firecracker lab. We paused a running guest and took a `Full` snapshot ([snapshot docs](https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md)). Then we killed the process and loaded the snapshot in a new Firecracker process. A value written to the guest's `/dev/shm` before the snapshot was still there:

```bash
curl -fsS --unix-socket $SOCK -X PATCH http://localhost/vm -d '{"state":"Paused"}'
curl -fsS --unix-socket $SOCK -X PUT http://localhost/snapshot/create -d '{"snapshot_type":"Full","snapshot_path":"/home/user/fc/snap.state","mem_file_path":"/home/user/fc/snap.mem"}'
# kill Firecracker, start a new one on a fresh socket, then:
curl -fsS --unix-socket $SOCK -X PUT http://localhost/snapshot/load -d '{"snapshot_path":"/home/user/fc/snap.state","mem_backend":{"backend_path":"/home/user/fc/snap.mem","backend_type":"File"},"resume_vm":true}'
```

```text
in-memory-1795023979
pause + full snapshot: 1259 ms
1.0G snap.mem
34K snap.state
new process + load + resume: 29 ms
restored: in-memory-1795023979 up 3.84s
first guest reply after restore start: 142 ms
```

## Many microVMs at once

To size a test fleet, we started microVMs together with `--config-file`, which configures and starts a VM with no API calls. Each had 1 vCPU, 512 MiB and its own copy of the rootfs:

| Started together on a `large` sandbox, 2026-11-18 | All at a login prompt | Firecracker RSS, total |
| - | - | - |
| 10 microVMs | 6.3 s | 1,021 MiB |
| 20 microVMs | 12.4 s | 2,035 MiB |

A guest's memory isn't all allocated at boot. Each of these idle 512 MiB guests used about 100 MiB at its login prompt. That grows as guests touch memory, and all of it comes out of the sandbox's 16 GB.

## Keep a microVM running across a sandbox stop

A sandbox stop saves memory as well as disk ([llms.txt](https://sandboxer.com/llms.txt)), and that includes the VMs inside it. We ran Firecracker with networking as a systemd unit, so it outlives the `sbx exec` call. We wrote a marker in the guest, then stopped and resumed the sandbox:

```bash
sudo systemd-run --unit fcvm -p User=user -p WorkingDirectory=/home/user/fc \
  /home/user/fc/firecracker --api-sock /home/user/fc/svc.sock --config-file /home/user/fc/vm-net.json --enable-pci
$SSH "echo before-stop > /dev/shm/mark; cat /proc/uptime"   # 3.84
sbx stop sb_ropzl20q9c --wait
sbx resume sb_ropzl20q9c          # "memoryRestored": true
$SSH "cat /dev/shm/mark /proc/uptime"
```

```text
before-stop
12.93 22.73
```

The guest kept its memory and its uptime, and the tap network still worked. If a resume boots from disk instead (`memoryRestored: false`), inner VMs are gone. The disk images are still there.

## What's limited

- **Nested is slower than one level.** One run each, on the same `large` sandbox, sandbox vs inner guest: `head -c 2000000000 /dev/zero | sha256sum` took 1.53 s vs 2.10 s. A 500 MiB direct-I/O write ran at 910 MB/s vs 509 MB/s. Use a sandbox for developing and functional testing, and bare metal for performance work that has to match production.
- **Memory is shared with the sandbox.** Every inner guest's RAM comes out of the sandbox's. `large` has 16 GB, and `xlarge` has 32 GB if your plan includes it.
- **The jailer and tap setup need sudo.** You have passwordless `sudo` in a sandbox, so this works. Plain `firecracker` doesn't need it.
- **Demonstration images.** Firecracker's guide says its CI kernel and rootfs are "for demonstration purposes and are not intended for production".
- **Idle pause.** A sandbox nobody is using pauses itself with memory after its `idleStopMinutes`. For long unattended runs, create it with `sbx new --idle 0` to turn idle pause off, and `--no-auto-stop` or a longer `--ttl` so the time-to-live doesn't stop it ([Long-running work](/docs/long-running), [CLI](/docs/cli)).

## FAQ

### Can Firecracker run inside a VM?

Yes, if the VM exposes `/dev/kvm`. Firecracker's own development docs list "a virtual machine that supports nested virtualization" as an option. On a Sandboxer `large` sandbox, v1.18.0 booted an Ubuntu guest in 2.2–2.7 s on 2026-11-18.

### Do I need sudo to run Firecracker in a sandbox?

Not for `firecracker` itself: `user` is in the `kvm` group and can open `/dev/kvm`. You need `sudo` for the jailer, tap devices and `iptables`.

### Why is my Firecracker guest slow to boot?

If you set `boot_args`, they replace Firecracker's default command line, which includes i8042 flags that save boot time. Adding Firecracker's default `i8042.noaux i8042.nomux i8042.dumbkbd swiotlb=noforce` took about 0.45 s off each boot in our test.

### Do Firecracker snapshots work in a nested VM?

Yes. A full snapshot of a 1 GiB guest took 1.26 s, and loading it in a new Firecracker process returned in 29 ms with the guest's memory intact.

### Which sandbox size should I use for Firecracker?

`large` (8 vCPU, 16 GB) ran 20 small microVMs at once in our test. Use `xlarge` if your plan includes it and your guests need more memory in total.

## Related

- [Sandbox Isolation Compared: Containers, gVisor, Kata, MicroVMs, Wasm](/guides/sandbox-isolation-technologies-compared)
- [Docker, Kubernetes and Nested VMs in a Cloud Sandbox](/guides/docker-and-virtualization-in-a-cloud-sandbox)
- [Docker and nested KVM inside an AI agent sandbox](/guides/docker-and-kvm-inside-an-agent-sandbox)
- [Firecracker vs Cloud Hypervisor](/guides/firecracker-vs-cloud-hypervisor)
- [What is a microVM?](/guides/what-is-a-microvm)
- [Machines](/docs/machines)
