Skip to content
Guides · Sandboxes and isolation

Run Firecracker microVMs inside a cloud VM with nested KVM

Updated November 18, 2026 .md

Short answer: Use a cloud VM that gives you /dev/kvm. In a Sandboxer large sandbox, the official Firecracker v1.18.0 binary runs as the normal user without sudo. Check its SHA-256, fetch the kernel and Ubuntu rootfs from Firecracker's CI as its getting-started guide shows, and boot through the API socket. On 2026-11-18 a guest reached a login prompt in 2.2–2.7 s.

Part of Sandbox Isolation Compared: Containers, gVisor, Kata, MicroVMs, Wasm and Docker, Kubernetes and Nested VMs in a Cloud Sandbox.

Every command below ran on a large Sandboxer sandbox (8 vCPU, 16 GB) on 2026-11-18. Firecracker facts come from its own getting-started guide and repository, read the same day. To compare the two hypervisors themselves, see Firecracker vs Cloud Hypervisor.

At a glance#

Result on a large sandbox, 2026-11-18
/dev/kvm Present, root:kvm mode 0660; user is in kvm, so no sudo and no setfacl
Firecracker v1.18.0 release binary; the .sha256.txt checksum matched
Guest Firecracker CI kernel 6.18.62 and Ubuntu 24.04 rootfs, 2 vCPU, 1024 MiB
Boot, InstanceStart to login prompt (5 runs each) 2.62–2.71 s with the getting-started boot args; 2.20–2.32 s with Firecracker's default i8042 args added
Networking A tap device with NAT, as in the docs: SSH, DNS and HTTPS from the guest all worked
Jailer Worked under sudo with cgroup v2 limits
Firecracker snapshots Full snapshot of a 1 GiB guest in 1.26 s; restore in a new process in 29 ms
20 microVMs at once All at a login prompt in 12.4 s, 2.0 GB of RSS in total
Sandbox stop and resume A running inner microVM came back with its memory

Why you need a VM that passes through KVM#

Firecracker requires read/write access to /dev/kvm. Its getting-started guide uses a bare-metal (.metal) cloud instance, because that cloud only offers KVM on bare-metal instance types. Its development setup doc says the other way in: "a virtual machine that supports nested virtualization".

A Sandboxer sandbox is a VM with /dev/kvm available inside (Machines). That makes it a nested setup: Firecracker's guest is a VM inside your sandbox's VM.

sbx new --type large --ttl 5400 --no-env
sbx exec sb_ropzl20q9c 'ls -l /dev/kvm; id -nG; [ -r /dev/kvm ] && [ -w /dev/kvm ] && echo OK || echo FAIL; grep -m1 -o -E "vmx|svm" /proc/cpuinfo; ls /sys/module | grep ^kvm'
crw-rw---- 1 root kvm 10, 232 Nov  8 15:48 /dev/kvm
user sudo docker kvm systemd-journal
OK
svm
kvm
kvm_amd
kvm_intel

The access check is the one from Firecracker's docs. Their lsmod | grep kvm check prints nothing here, because KVM is built into the sandbox kernel rather than loaded as a module. /sys/module shows it's there. Our sandbox ran on an AMD EPYC 9254, hence svm.

We used large. xlarge (16 vCPU, 32 GB) isn't on every plan, and our test account's plan got 403 machine_class_plan_required for it; see Machines. --ttl stops the sandbox after that many seconds (auto-stop); delete it with sbx delete when you're done.

Download Firecracker and check the checksum#

The v1.18.0 release publishes a .sha256.txt file next to each tarball. This is the getting-started download with the checksum step added. Run it in the sandbox (sbx ssh sb_ropzl20q9c, or wrap it in sbx exec):

mkdir -p ~/fc && cd ~/fc
ARCH=$(uname -m)
latest=$(basename $(curl -fsSLI -o /dev/null -w %{url_effective} https://github.com/firecracker-microvm/firecracker/releases/latest))
base=https://github.com/firecracker-microvm/firecracker/releases/download/$latest
curl -fsSLO $base/firecracker-$latest-$ARCH.tgz
curl -fsSLO $base/firecracker-$latest-$ARCH.tgz.sha256.txt
sha256sum -c firecracker-$latest-$ARCH.tgz.sha256.txt
tar -xzf firecracker-$latest-$ARCH.tgz
cp release-$latest-$ARCH/firecracker-$latest-$ARCH firecracker
cp release-$latest-$ARCH/jailer-$latest-$ARCH jailer
./firecracker --version | head -1
firecracker-v1.18.0-x86_64.tgz: OK
Firecracker v1.18.0

The tarball also has a SHA256SUMS file for the binaries inside. sha256sum -c --ignore-missing SHA256SUMS in the release folder said OK for firecracker-v1.18.0-x86_64 and jailer-v1.18.0-x86_64.

Get a guest kernel and rootfs#

Firecracker's guide (Getting a rootfs and Guest Kernel Image) takes the newest kernel and Ubuntu squashfs from its CI bucket. It then adds an SSH key and converts the image to ext4. We ran its script unchanged, apart from quieter flags and a mkdir -p for /root/.ssh. It needs unsquashfs, which isn't preinstalled:

sudo apt-get update -q && sudo apt-get install -y -q squashfs-tools
CI prefix: firecracker-ci/20261118-6d7cfa9c6b53-0/
27M vmlinux-6.18.62
95M ubuntu-24.04.squashfs.upstream
1.0G ubuntu-24.04.ext4

The CI prefix is dated, so you get whatever Firecracker's CI built most recently. Ours was built the same day. Save the kernel and ext4 under /home/user so they survive a stop.

Boot a microVM through the API socket#

Firecracker's guide runs it with sudo in two terminals. Here it runs as user, because user can open /dev/kvm. This script starts Firecracker, sends the configuration over the socket, starts the VM and waits for the serial console's login prompt. The guest's serial console is Firecracker's stdin and stdout, so a FIFO on stdin lets the script type into it:

# ~/fc/boot.sh [net] ["command to type on the guest console"]
cd ~/fc
SOCK=~/fc/fc.sock
rm -f "$SOCK" serial.log fc.log in.fifo
mkfifo in.fifo
exec 3<>in.fifo   # serial console input
t0=$(date +%s%N)
./firecracker --api-sock "$SOCK" --enable-pci < in.fifo > serial.log 2>&1 &
FC=$!
until [ -S "$SOCK" ]; do sleep 0.01; done
api() { curl -fsS --unix-socket "$SOCK" -X PUT "http://localhost/$1" -H 'Content-Type: application/json' -d "$2"; }
api logger '{"log_path":"/home/user/fc/fc.log","level":"Info"}'
api machine-config '{"vcpu_count":2,"mem_size_mib":1024}'
api boot-source "{\"kernel_image_path\":\"$(ls ~/fc/vmlinux-* | tail -1)\",\"boot_args\":\"${BOOT_ARGS:-console=ttyS0 reboot=k panic=1}\"}"
api drives/rootfs '{"drive_id":"rootfs","path_on_host":"/home/user/fc/ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}'
if [ "${1:-}" = net ]; then
  api network-interfaces/net1 '{"iface_id":"net1","guest_mac":"06:00:AC:10:00:02","host_dev_name":"tap0"}'
fi
t1=$(date +%s%N)
api actions '{"action_type":"InstanceStart"}'
until grep -aq "login:" serial.log; do sleep 0.02; done
t2=$(date +%s%N)
echo "process start to login prompt: $(( (t2-t0)/1000000 )) ms; InstanceStart to login prompt: $(( (t2-t1)/1000000 )) ms"
echo $FC > fc.pid
if [ -n "${2:-}" ]; then
  sleep 1; printf '%s\n' "$2" >&3
  sleep ${WAIT:-2}; printf 'reboot\n' >&3
  wait $FC; echo "firecracker exited: $?"
fi

The CI rootfs logs root in on the serial console automatically. So the second argument runs as root in the guest:

bash ~/fc/boot.sh nonet 'echo MARK; uname -r; nproc; free -m | head -2; cat /proc/cmdline; systemd-analyze | head -1'
grep -a -E "Linux version|Ubuntu 24|login:" ~/fc/serial.log | head -4
grep -a -A6 MARK ~/fc/serial.log
process start to login prompt: 2720 ms; InstanceStart to login prompt: 2668 ms
firecracker exited: 0
[    0.000000] Linux version 6.18.62+ (root@23ebf1a82c3e) (gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, …) #1 SMP PREEMPT_DYNAMIC Wed Nov 18 03:23:17 UTC 2026
Welcome to Ubuntu 24.04.5 LTS!
Ubuntu 24.04.5 LTS ubuntu-fc-uvm ttyS0
ubuntu-fc-uvm login: root (automatic login)
MARK
6.18.62+
2
               total        used        free      shared  buff/cache   available
Mem:             984         141         888           2          51         843
console=ttyS0 reboot=k panic=1 root=/dev/vda rw
Startup finished in 1.281s (kernel) + 922ms (userspace) = 2.204s

reboot in the guest ends the Firecracker process with exit code 0, as the docs say: Firecracker doesn't implement guest power management.

Boot time, and the boot args that cut it#

Five boots with the getting-started boot args, then five with Firecracker's own default i8042 flags added:

cd ~/fc && for i in 1 2 3 4 5; do bash boot.sh nonet "systemd-analyze | head -1" | head -1; grep -a -m1 "Startup finished" serial.log; done
export BOOT_ARGS="console=ttyS0 reboot=k panic=1 i8042.noaux i8042.nomux i8042.dumbkbd swiotlb=noforce"
# …same loop again
Guest: CI kernel 6.18.62, Ubuntu 24.04, 2 vCPU, 1024 MiB, PCI on; large sandbox, 2026-11-18 InstanceStart to login prompt systemd: kernel + userspace
console=ttyS0 reboot=k panic=1 (getting-started, 5 runs) 2.62–2.71 s 1.27–1.31 s + 0.82–0.90 s
Same, plus i8042.noaux i8042.nomux i8042.dumbkbd swiotlb=noforce (5 runs) 2.20–2.32 s 0.82–0.91 s + 0.79–0.90 s

Here's why the flags matter. Firecracker's kernel policy lists a default command line with the i8042 flags. It notes they "save boot time". If you send your own boot_args, you replace that list. With the getting-started args, the guest kernel log showed a 0.41 s gap before input: AT Raw Set 2 keyboard. With the flags, the gap was gone.

These are times inside a nested VM on one machine type on one day. Firecracker's integration tests check its published performance targets as measured on bare-metal (.metal) cloud instances. Its docs say the startup-time test may fail elsewhere. Don't read our numbers as what Firecracker does on bare metal.

Give the guest a network with a tap device#

This is the network part of the getting-started guide, unchanged. It needs sudo for ip and iptables:

TAP_DEV="tap0"; TAP_IP="172.16.0.1"; MASK_SHORT="/30"
sudo ip link del "$TAP_DEV" 2> /dev/null || true
sudo ip tuntap add dev "$TAP_DEV" mode tap user "$(id -u)" group "$(id -g)"
sudo ip addr add "${TAP_IP}${MASK_SHORT}" dev "$TAP_DEV"
sudo ip link set dev "$TAP_DEV" up
sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
sudo iptables -P FORWARD ACCEPT
HOST_IFACE=$(ip -j route list default | jq -r '.[0].dev')   # eth0 in a sandbox
sudo iptables -t nat -D POSTROUTING -o "$HOST_IFACE" -j MASQUERADE 2>/dev/null || true
sudo iptables -t nat -A POSTROUTING -o "$HOST_IFACE" -j MASQUERADE

Then boot with bash boot.sh net and SSH in with the key the rootfs script made. We set the guest's default route and nameserver as the docs do:

SSH="ssh -q -i ~/fc/ubuntu-24.04.id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@172.16.0.2"
$SSH "ip route add default via 172.16.0.1 dev eth0; echo 'nameserver 8.8.8.8' > /etc/resolv.conf"
$SSH "ip -br addr show eth0; ping -c 2 -W 2 172.16.0.1 | tail -1; getent hosts github.com; curl -sS -m 15 -o /dev/null -w '%{http_code} %{time_total}s\n' https://github.com/"
ssh ready 334 ms after login prompt
eth0             UP             172.16.0.2/30 fe80::400:acff:fe10:2/64
rtt min/avg/max/mdev = 0.181/0.242/0.303/0.061 ms
140.82.114.4    github.com
200 0.075411s

It worked. DNS also worked through the sandbox's own resolver (nameserver 10.202.0.1, from the sandbox's /etc/resolv.conf). Each SSH command printed a harmless setlocale: LC_ALL warning, because the CI rootfs has no en_US.UTF-8 locale. The guest has only the NAT'd private address 172.16.0.2. Processes in the sandbox can reach it directly. sbx host gives an HTTPS URL to a port on the sandbox itself, so a guest service would need a port forward in between; we didn't test that.

The jailer works, under sudo#

Firecracker's docs say production use runs it inside the jailer, and the getting-started guide skips it. The jailer needs root, so it runs under sudo. Put the kernel and rootfs inside the chroot, which the jailer docs say you must do. Then give it cgroup v2 limits:

ID=fc-jail-1; JROOT=/srv/jailer/firecracker/$ID/root
sudo mkdir -p $JROOT && sudo cp vmlinux-6.18.62 ubuntu-24.04.ext4 $JROOT/ && sudo chown 1000:1000 $JROOT/vmlinux-6.18.62 $JROOT/ubuntu-24.04.ext4
sudo ./jailer --id $ID --exec-file ~/fc/firecracker --uid 1000 --gid 1000 --cgroup-version 2 \
  --cgroup memory.max=1610612736 --cgroup cpu.max="200000 100000" --daemonize -- --enable-pci
# configure through $JROOT/run/firecracker.socket with sudo curl, paths relative to the chroot (/vmlinux-6.18.62), then InstanceStart
sudo curl -fsS --unix-socket $JROOT/run/firecracker.socket http://localhost/
{"id":"fc-jail-1","state":"Running","vmm_version":"1.18.0","app_name":"Firecracker"}
cgroup: /firecracker/fc-jail-1
1610612736
200000 100000

The jailer created /dev/kvm, /dev/net/tun, /dev/urandom and /dev/userfaultfd nodes inside the chroot. It ran Firecracker as uid 1000 in its own cgroup with both limits applied. The sandbox kernel uses cgroup v2 (cpuset cpu io memory pids controllers). With --daemonize, the serial console isn't captured, so we checked this one through the API state only.

Firecracker snapshots inside the sandbox#

Developing snapshot and restore tooling is a common reason to want a Firecracker lab. We paused a running guest and took a Full snapshot (snapshot docs). Then we killed the process and loaded the snapshot in a new Firecracker process. A value written to the guest's /dev/shm before the snapshot was still there:

curl -fsS --unix-socket $SOCK -X PATCH http://localhost/vm -d '{"state":"Paused"}'
curl -fsS --unix-socket $SOCK -X PUT http://localhost/snapshot/create -d '{"snapshot_type":"Full","snapshot_path":"/home/user/fc/snap.state","mem_file_path":"/home/user/fc/snap.mem"}'
# kill Firecracker, start a new one on a fresh socket, then:
curl -fsS --unix-socket $SOCK -X PUT http://localhost/snapshot/load -d '{"snapshot_path":"/home/user/fc/snap.state","mem_backend":{"backend_path":"/home/user/fc/snap.mem","backend_type":"File"},"resume_vm":true}'
in-memory-1795023979
pause + full snapshot: 1259 ms
1.0G snap.mem
34K snap.state
new process + load + resume: 29 ms
restored: in-memory-1795023979 up 3.84s
first guest reply after restore start: 142 ms

Many microVMs at once#

To size a test fleet, we started microVMs together with --config-file, which configures and starts a VM with no API calls. Each had 1 vCPU, 512 MiB and its own copy of the rootfs:

Started together on a large sandbox, 2026-11-18 All at a login prompt Firecracker RSS, total
10 microVMs 6.3 s 1,021 MiB
20 microVMs 12.4 s 2,035 MiB

A guest's memory isn't all allocated at boot. Each of these idle 512 MiB guests used about 100 MiB at its login prompt. That grows as guests touch memory, and all of it comes out of the sandbox's 16 GB.

Keep a microVM running across a sandbox stop#

A sandbox stop saves memory as well as disk (llms.txt), and that includes the VMs inside it. We ran Firecracker with networking as a systemd unit, so it outlives the sbx exec call. We wrote a marker in the guest, then stopped and resumed the sandbox:

sudo systemd-run --unit fcvm -p User=user -p WorkingDirectory=/home/user/fc \
  /home/user/fc/firecracker --api-sock /home/user/fc/svc.sock --config-file /home/user/fc/vm-net.json --enable-pci
$SSH "echo before-stop > /dev/shm/mark; cat /proc/uptime"   # 3.84
sbx stop sb_ropzl20q9c --wait
sbx resume sb_ropzl20q9c          # "memoryRestored": true
$SSH "cat /dev/shm/mark /proc/uptime"
before-stop
12.93 22.73

The guest kept its memory and its uptime, and the tap network still worked. If a resume boots from disk instead (memoryRestored: false), inner VMs are gone. The disk images are still there.

What's limited#

  • Nested is slower than one level. One run each, on the same large sandbox, sandbox vs inner guest: head -c 2000000000 /dev/zero | sha256sum took 1.53 s vs 2.10 s. A 500 MiB direct-I/O write ran at 910 MB/s vs 509 MB/s. Use a sandbox for developing and functional testing, and bare metal for performance work that has to match production.
  • Memory is shared with the sandbox. Every inner guest's RAM comes out of the sandbox's. large has 16 GB, and xlarge has 32 GB if your plan includes it.
  • The jailer and tap setup need sudo. You have passwordless sudo in a sandbox, so this works. Plain firecracker doesn't need it.
  • Demonstration images. Firecracker's guide says its CI kernel and rootfs are "for demonstration purposes and are not intended for production".
  • Idle pause. A sandbox nobody is using pauses itself with memory after its idleStopMinutes. For long unattended runs, create it with sbx new --idle 0 to turn idle pause off, and --no-auto-stop or a longer --ttl so the time-to-live doesn't stop it (Long-running work, CLI).

FAQ#

Can Firecracker run inside a VM?#

Yes, if the VM exposes /dev/kvm. Firecracker's own development docs list "a virtual machine that supports nested virtualization" as an option. On a Sandboxer large sandbox, v1.18.0 booted an Ubuntu guest in 2.2–2.7 s on 2026-11-18.

Do I need sudo to run Firecracker in a sandbox?#

Not for firecracker itself: user is in the kvm group and can open /dev/kvm. You need sudo for the jailer, tap devices and iptables.

Why is my Firecracker guest slow to boot?#

If you set boot_args, they replace Firecracker's default command line, which includes i8042 flags that save boot time. Adding Firecracker's default i8042.noaux i8042.nomux i8042.dumbkbd swiotlb=noforce took about 0.45 s off each boot in our test.

Do Firecracker snapshots work in a nested VM?#

Yes. A full snapshot of a 1 GiB guest took 1.26 s, and loading it in a new Firecracker process returned in 29 ms with the guest's memory intact.

Which sandbox size should I use for Firecracker?#

large (8 vCPU, 16 GB) ran 20 small microVMs at once in our test. Use xlarge if your plan includes it and your guests need more memory in total.

More guides

Top AI sandbox platforms for code execution in 2026, ranked

For running AI-written code we rank Sandboxer, Freestyle and E2B first: each gives code its own VM with an API or SDK and can pause with memory. Pick Daytona for Windows or GPUs, Modal or Northflank for GPUs at scale, E2B or Northflank to run in your own cloud, and Vercel Sandbox if your app lives on Vercel.

Read the guide
Best sandboxes for running coding agents (Claude Code, Codex, Gemini CLI) in 2026, ranked

For coding agents we rank Sandboxer first: a full Ubuntu VM with Claude Code and Codex preinstalled, Docker, and stop-and-resume that keeps the agent's session in memory. Freestyle and Amp Orbs also keep memory across pauses. Pick E2B for prebuilt agent templates on open source, Docker Sandboxes for free local runs, and Codespaces if you live in GitHub.

Read the guide
Best Daytona sandbox alternatives in 2026, ranked

The best Daytona alternatives depend on why you're switching. For a full Linux microVM on every sandbox with memory-saving stop and Docker, we rank Sandboxer first. E2B suits open source and self-hosting, Modal massive short-job fleets, Northflank bring-your-own-cloud with GPUs, and Vercel Sandbox apps on Vercel.

Read the guide
Best E2B alternatives in 2026, ranked

If you're leaving E2B for longer sessions or a fuller machine, we rank Sandboxer first, then Freestyle: both give each sandbox a full Linux VM that keeps memory across pauses, with Docker and no short session cap. Pick Daytona for Windows or GPUs, Northflank for bring-your-own-cloud, and stay on E2B if you need open source.

Read the guide