Skip to content
sandboxer
Start free trial
Guides · Recipes

How Do I Run Ollama on CPU in a Cloud Sandbox?

Updated October 27, 2026 .md

Short answer: Install Ollama with its Linux script in a default sandbox, pull a small model such as qwen2.5:0.5b, and bind it to 0.0.0.0 with a systemd override. Then sbx host the port for a private HTTPS URL. There is no GPU: expect about 30 to 55 tokens per second for 0.5B and 13 to 22 for 1B.

At a glance#

We ran everything below on October 27, 2026, on a default sandbox (4 vCPU, 8 GB RAM). For the last test we resumed it as large (8 vCPU, 16 GB RAM). Ollama 0.40.11. CPU only: a sandbox has no GPU, and Ollama runs every layer on the CPU (ollama ps shows 100% CPU).

Step What we saw
Install script 17 s, creates an enabled ollama systemd service on 127.0.0.1:11434
ollama pull qwen2.5:0.5b / llama3.2:1b 397 MB in 7 s / 1.3 GB in 16 s
Generation speed, default qwen2.5:0.5b 28 to 57 tokens/s, llama3.2:1b 13 to 19 tokens/s
Generation speed, large qwen2.5:0.5b 50 to 59 tokens/s, llama3.2:1b 17 to 22 tokens/s
sbx host --localhost with Ollama on 127.0.0.1 403 from Ollama
OLLAMA_HOST=0.0.0.0:11434, then sbx host 401 without the token, 200 with it
Short request through the hosted URL 0.3 to 0.55 s round trip
sbx stop --wait + sbx resume memoryRestored: true, same PID, model still loaded
Cold boot (resume as large) Service came back on its own, first request reloaded the model (1.4 s)

The speed ranges are real. The same prompt on the same sandbox varied by up to 2x between runs a minute apart. Plan for the low end.

1. Create a sandbox and install Ollama#

Use default. small has 4 GB of RAM, which is tight for anything above 1B. Install with the script from Ollama's Linux docs:

sbx new --name ollama --type default
id=sb_o32n3ltk3i    # the id that sbx new printed

sbx exec $id "curl -fsSL https://ollama.com/install.sh | sh"
WARNING: Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependencies.
>>> Installing ollama to /usr/local
>>> Downloading ollama-linux-amd64.tar.zst
…
>>> Creating ollama systemd service...
>>> Enabling and starting ollama service...
Created symlink /etc/systemd/system/default.target.wants/ollama.service → /etc/systemd/system/ollama.service.

The GPU warning is expected. Ignore it.

2. Pull a small model and run a prompt#

Pick models from the Ollama library that fit in RAM with room to spare. We used qwen2.5:0.5b and llama3.2:1b:

sbx exec $id "ollama pull qwen2.5:0.5b && ollama pull llama3.2:1b && ollama list"
sbx exec $id "ollama run qwen2.5:0.5b 'Write a haiku about a sandbox.'"
NAME            ID              SIZE      MODIFIED
llama3.2:1b     baf6a787fdff    1.3 GB    Less than a second ago
qwen2.5:0.5b    a8b0c5157701    397 MB    16 seconds ago

In the sandbox,
children play and laugh,
playground laughter flows.

3. Measure tokens per second#

Ollama's generate API returns eval_count (tokens generated) and eval_duration (nanoseconds). Tokens per second is eval_count / eval_duration * 1e9. Save this as bench.sh and run it in the sandbox with sbx exec $id "$(cat bench.sh)":

for m in qwen2.5:0.5b llama3.2:1b; do
  for i in 1 2 3; do
    curl -s http://127.0.0.1:11434/api/generate -d "{\"model\":\"$m\",\"prompt\":\"Explain what a microVM is in three sentences.\",\"stream\":false,\"options\":{\"num_predict\":128,\"temperature\":0}}" \
    | python3 -c "import json,sys; r=json.load(sys.stdin); print('$m', 'run $i', 'load_s=%.2f'%(r['load_duration']/1e9), 'prompt_tok_s=%.1f'%(r['prompt_eval_count']/r['prompt_eval_duration']*1e9), 'eval_count=%d'%r['eval_count'], 'tok_s=%.1f'%(r['eval_count']/r['eval_duration']*1e9))"
  done
done

On default:

qwen2.5:0.5b run 1 load_s=0.00 prompt_tok_s=575.4 eval_count=111 tok_s=56.3
qwen2.5:0.5b run 2 load_s=0.00 prompt_tok_s=1763.0 eval_count=111 tok_s=48.2
qwen2.5:0.5b run 3 load_s=0.00 prompt_tok_s=1458.7 eval_count=111 tok_s=57.4
llama3.2:1b run 1 load_s=0.00 prompt_tok_s=355.7 eval_count=107 tok_s=19.4
llama3.2:1b run 2 load_s=0.00 prompt_tok_s=648.7 eval_count=107 tok_s=16.3
llama3.2:1b run 3 load_s=0.00 prompt_tok_s=307.9 eval_count=107 tok_s=16.9

A pass a minute earlier gave 28 to 30 tokens/s for qwen2.5:0.5b and 13 to 15 for llama3.2:1b. The first request for a model also paid load_s=2.57 to load llama3.2:1b from disk.

On large, with the same script:

qwen2.5:0.5b run 2 load_s=0.00 prompt_tok_s=1460.5 eval_count=111 tok_s=51.3
qwen2.5:0.5b run 3 load_s=0.00 prompt_tok_s=1352.5 eval_count=111 tok_s=52.3
llama3.2:1b run 2 load_s=0.00 prompt_tok_s=669.0 eval_count=107 tok_s=18.0
llama3.2:1b run 3 load_s=0.01 prompt_tok_s=527.1 eval_count=107 tok_s=22.4
…

Doubling the vCPUs didn't double the speed. With "num_thread":8 in options, llama3.2:1b on large went from 17 to 19–22 tokens/s, against 17 with 4 threads. For one small model, default at $0.036 an hour is the better buy than large at twice that. Choose large when you need its 16 GB of RAM for a bigger model. large isn't available during the free trial.

4. Expose the API on a private URL#

Ollama listens on 127.0.0.1:11434 by default. We tried both ways to reach it from outside.

sbx host --localhost doesn't work for Ollama. The relay reaches the port, but Ollama rejects requests whose Host header isn't local. The proxy passed the request with the token, and Ollama answered 403. Inside the sandbox, curl -H "Host: <subdomain>-11434.box.sandboxer.com" http://127.0.0.1:11434/api/tags also got 403.

Binding to 0.0.0.0 works. Ollama's FAQ sets OLLAMA_HOST on the systemd service. Then host the port without --localhost:

sbx exec $id "sudo mkdir -p /etc/systemd/system/ollama.service.d && \
  printf '[Service]\nEnvironment=\"OLLAMA_HOST=0.0.0.0:11434\"\n' | sudo tee /etc/systemd/system/ollama.service.d/override.conf && \
  sudo systemctl daemon-reload && sudo systemctl restart ollama && ss -ltn | grep 11434"
sbx host $id 11434 --title Ollama
LISTEN 0      4096               *:11434            *:*
https://maple-glossy-dune-11434.box.sandboxer.com?_token=<token>

Before the override, the same URL returned 502 ("Nothing is answering on port 11434"). Ollama has no authentication of its own. The hosted URL's token is the lock, so keep the port private. Binding to 0.0.0.0 doesn't open the port to the internet: the sandbox firewall allows only SSH, and the hosting proxy reaches only ports you host. See what can reach a sandbox.

5. Call it from outside with curl#

Put the token in the query string. A GET with the token answers 302 and sets a cookie, so use -L and a cookie jar. A POST with the token goes straight through.

BASE=https://maple-glossy-dune-11434.box.sandboxer.com
TOKEN=<token>

curl -s -o /dev/null -w "%{http_code}\n" $BASE/api/tags                 # 401
curl -sL -c jar -b jar "$BASE/api/tags?_token=$TOKEN"                   # 200, the model list
curl -s "$BASE/api/generate?_token=$TOKEN" \
  -d '{"model":"qwen2.5:0.5b","prompt":"Reply with one word: ready?","stream":false}'
401
{"models":[{"name":"llama3.2:1b","model":"llama3.2:1b","modified_at":"2026-10-27T09:23:12.624999959Z","size":1321098329,…
{"model":"qwen2.5:0.5b","created_at":"2026-10-27T09:25:35.546740041Z","response":"ready","done":true,"done_reason":"stop",…,"load_duration":794796193,…}

Ollama's OpenAI-compatible endpoint works through the same URL:

curl -s "$BASE/v1/chat/completions?_token=$TOKEN" -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"Name one prime number. Reply with the number only."}]}'
{"id":"chatcmpl-685","object":"chat.completion",…,"choices":[{"index":0,"message":{"role":"assistant","content":"1"},"finish_reason":"stop"}],"usage":{"prompt_tokens":40,…,"completion_tokens":2,"total_tokens":42}}

That answer is wrong: 1 isn't prime. A 0.5B model is fast on a CPU and fine for classification, extraction or test fixtures. Don't rely on it for facts or reasoning. Five short requests (num_predict: 8) from our workstation to the sandbox in eu-central took 0.38 to 0.55 s each.

6. Stop, resume and cold boot#

sbx stop $id --wait
sbx resume $id
sbx exec $id "ollama ps; pgrep -f 'ollama serve'"

The resume took 1.4 s and reported memoryRestored: true. The server kept PID 947, qwen2.5:0.5b was still loaded, and the hosted URL answered at once with load_duration of 1.2 ms. The URL and token didn't change.

A resume that boots from disk loses running processes. That happens when the resume lands on a different CPU type, or when you change size. We stopped again and ran sbx resume $id --type large (memoryRestored: false, uptime 0 min). Ollama came back because the install script enabled its systemd service, and our OLLAMA_HOST override came back with it. The first request through the URL returned 502 for a moment, then 200 in 1.4 s while the model loaded from disk. Later requests took 0.3 s. Retry a 502 right after a cold boot. Don't start Ollama with ollama serve & in a sandbox you plan to stop: nothing would restart it after a cold boot.

FAQ#

Does a Sandboxer sandbox have a GPU?#

No. Sandboxes are CPU-only microVMs, and Ollama runs the whole model on the CPU. Use small models. In our test, 0.5B models ran at 28 to 59 tokens per second and 1B models at 13 to 22. Bigger models are slower.

Which model size fits which sandbox?#

The model has to fit in RAM next to everything else. llama3.2:1b used 1.5 GB loaded in our test, out of 8 GB on default. For models above about 3B, use large (16 GB) and expect lower tokens per second.

Why does sbx host --localhost return 403 for Ollama?#

Ollama checks the Host header when it listens on loopback and rejects names it doesn't know. Set OLLAMA_HOST=0.0.0.0:11434 on the service and host the port without --localhost.

Is my Ollama API open to the internet?#

Only through the hosted URL, and that URL is private by default. Requests without the token get 401 before they reach Ollama. Don't make the port --public, because Ollama has no login of its own.

Does the model stay loaded while the sandbox is stopped?#

Nothing runs while it is stopped, and nothing is billed. After a memory-restoring resume, the model was still in memory. After a cold boot, Ollama loads it again on the first request.

More guides

Top AI sandbox platforms for code execution in 2026, ranked

For running AI-written code we rank Sandboxer, Freestyle and E2B first: each gives code its own VM with an API or SDK and can pause with memory. Pick Daytona for Windows or GPUs, Modal or Northflank for GPUs at scale, E2B or Northflank to run in your own cloud, and Vercel Sandbox if your app lives on Vercel.

Read the guide
Best sandboxes for running coding agents (Claude Code, Codex, Gemini CLI) in 2026, ranked

For coding agents we rank Sandboxer first: a full Ubuntu VM with Claude Code and Codex preinstalled, Docker, and stop-and-resume that keeps the agent's session in memory. Freestyle and Amp Orbs also keep memory across pauses. Pick E2B for prebuilt agent templates on open source, Docker Sandboxes for free local runs, and Codespaces if you live in GitHub.

Read the guide
Best Daytona sandbox alternatives in 2026, ranked

The best Daytona alternatives depend on why you're switching. For a full Linux microVM on every sandbox with memory-saving stop and Docker, we rank Sandboxer first. E2B suits open source and self-hosting, Modal massive short-job fleets, Northflank bring-your-own-cloud with GPUs, and Vercel Sandbox apps on Vercel.

Read the guide
Best E2B alternatives in 2026, ranked

If you're leaving E2B for longer sessions or a fuller machine, we rank Sandboxer first, then Freestyle: both give each sandbox a full Linux VM that keeps memory across pauses, with Docker and no short session cap. Pick Daytona for Windows or GPUs, Northflank for bring-your-own-cloud, and stay on E2B if you need open source.

Read the guide