Skip to content
sandboxer
Start free trial
Guides · Recipes

How Do I Run llama.cpp's llama-server on CPU in a Cloud Sandbox?

Updated October 27, 2026 .md

Short answer: Download the prebuilt Linux x64 build from llama.cpp's GitHub releases, fetch a GGUF model from Hugging Face with curl, and run llama-server on 0.0.0.0 as a systemd service. Then sbx host the port for a private HTTPS URL that any OpenAI client can call. A 0.5B model generated 35 to 48 tokens per second on default.

At a glance#

We ran everything below on prod on October 27, 2026: a default sandbox (4 vCPU, 8 GB RAM), later resumed as large (8 vCPU, 16 GB RAM). llama.cpp build b11925. CPU only: a sandbox has no GPU. We used the plain ubuntu-x64 build, which picked its zen4 CPU backend on its own.

Step What we saw
Download and unpack the release tarball Under 1 s, no compiler needed
Download two GGUF files from Hugging Face 469 MB in 7 s, 1.1 GB in 12 s, no token
llama-bench, Qwen2.5 0.5B Q4_K_M, default, 4 threads 545 tokens/s prompt, 47 tokens/s generation
llama-bench, Qwen2.5 1.5B Q4_K_M, default, 4 threads 236 tokens/s prompt, 23 tokens/s generation
Same two models on large, 8 threads 0.5B: 525 / 60 tokens/s. 1.5B: 431 / 23 tokens/s
llama-server timings, 0.5B, real chat request 35 to 48 tokens/s on default, 57 to 61 on large
Hosted URL, no token / token but no API key / both 401 / 401 (from llama-server) / 200
Short request from our workstation through the URL 0.35 to 0.51 s round trip
sbx stop --wait + sbx resume memoryRestored: true, same PID, URL answered at once
Resume as large (cold boot) systemd started the server again; first request 502 (nothing listening yet), the retry 200

If you want a one-line install and a model library, run Ollama instead. Ollama is built on llama.cpp. Running llama.cpp directly gives you the exact build, any GGUF file on Hugging Face, llama-bench, and per-request timings in every response.

1. Download llama.cpp#

The llama.cpp releases page publishes a prebuilt CPU build for Linux on every build number. Pin one so a rebuild gives you the same binary:

sbx new --name llama --type default
id=sb_pgchnyueil    # the id that sbx new printed

sbx exec $id "curl -fsSL -o llama.tar.gz https://github.com/ggml-org/llama.cpp/releases/download/b11925/llama-b11925-bin-ubuntu-x64.tar.gz \
  && mkdir -p llama && tar xzf llama.tar.gz -C llama --strip-components=1 && ./llama/llama-server --version"
version: 0.7.0-dev (build 11925, commit fae9ac561)
built with GNU 11.4.0 for Linux x86_64

The tarball holds llama-server, llama-cli, llama-bench and one CPU backend library per CPU family (libggml-cpu-haswell.so, libggml-cpu-zen4.so and so on). At startup it loads the best one for the CPU it finds. Don't pick a cuda, rocm or vulkan asset: a sandbox has no GPU. To build from source instead, follow the repo's build guide.

2. Download a GGUF model#

llama-server reads one .gguf file. Public repos on Hugging Face need no token. We used Qwen's official 0.5B and 1.5B instruct models at Q4_K_M:

sbx exec $id "mkdir -p models && cd models \
  && curl -fsSL -O https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  && curl -fsSL -O https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF/resolve/main/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  && ls -lh"
-rw-r--r-- 1 user user 469M Oct 27 10:15 qwen2.5-0.5b-instruct-q4_k_m.gguf
-rw-r--r-- 1 user user 1.1G Oct 27 10:15 qwen2.5-1.5b-instruct-q4_k_m.gguf

The URL pattern is https://huggingface.co/<repo>/resolve/main/<file>. llama-server -hf <user>/<repo>:<quant> can also download for you, as the server README describes, but an explicit file path is easier to put in a systemd unit.

3. Measure tokens per second and pick a thread count#

llama-bench measures prompt processing (pp512, 512 prompt tokens) and generation (tg128, 128 new tokens) separately. -t 1,2,4 sweeps the thread count:

sbx exec $id --timeout 600 "./llama/llama-bench -m models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf -t 1,2,4 -p 512 -n 128 -r 3"
| model                    |       size | threads |  test |            t/s |
| ------------------------ | ---------: | ------: | ----: | -------------: |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       1 | pp512 | 153.85 ± 15.70 |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       1 | tg128 |   28.42 ± 0.44 |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       2 | pp512 |  318.03 ± 2.46 |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       2 | tg128 |   35.45 ± 1.12 |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       4 | pp512 | 544.90 ± 38.33 |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       4 | tg128 |   46.69 ± 0.87 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       1 | pp512 |   67.64 ± 0.61 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       1 | tg128 |   15.36 ± 0.39 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       2 | pp512 | 111.38 ± 14.43 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       2 | tg128 |   20.72 ± 1.00 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       4 | pp512 | 236.24 ± 13.00 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       4 | tg128 |   22.64 ± 2.00 |

(Columns trimmed. llama-bench labels the 0.5B model "1B"; it is the 469 MB file.) Later we resumed the sandbox as large and ran -t 2,4,8:

Model Test 4 threads 8 threads
Qwen2.5 0.5B pp512 403 525
Qwen2.5 0.5B tg128 47 60
Qwen2.5 1.5B pp512 242 431
Qwen2.5 1.5B tg128 22 23

What the numbers say:

  • Use one thread per vCPU. On both sizes, the most threads gave the most speed. We didn't test more threads than vCPUs.
  • Prompt speed scales with cores. Generation speed mostly doesn't. Doubling to 8 vCPUs nearly doubled prompt speed for the 1.5B model, but its generation moved from 22 to 23 tokens/s. Generation is usually limited by memory bandwidth, not core count.
  • large pays off for long prompts, such as RAG with big retrieved contexts, or for models that need its 16 GB of RAM. For short prompts and long answers, default at $0.036 an hour gives almost the same generation speed for half the price. large isn't available during the free trial.

llama-server takes -t for generation threads and -tb for prompt processing threads. We left both unset, and the server's own timings, below, were in line with the full-vCPU llama-bench rows.

4. Run llama-server as a systemd service#

A systemd service starts again after a cold boot. A process you start by hand doesn't. Bind to 0.0.0.0 so the hosting proxy can reach it, and give it an API key of its own:

sbx exec $id "umask 077 && openssl rand -hex 24 > ~/.llama-api-key"

sbx exec $id "sudo tee /etc/systemd/system/llama-server.service >/dev/null <<'EOF'
[Unit]
Description=llama.cpp server
After=network-online.target

[Service]
User=user
WorkingDirectory=/home/user
ExecStart=/home/user/llama/llama-server -m /home/user/models/qwen2.5-0.5b-instruct-q4_k_m.gguf --alias qwen2.5-0.5b --host 0.0.0.0 --port 8080 -c 4096 --api-key-file /home/user/.llama-api-key --metrics
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now llama-server"
  • --alias sets the model name clients send and /v1/models lists. Without it, the name is the file path.
  • --api-key-file makes every endpoint except /health require Authorization: Bearer <key>. Inside the sandbox, a request without it got 401.
  • Set --host and --port yourself. The server listens on 127.0.0.1 by default, which the hosting proxy can't reach.

5. Read prompt and generation speed from each response#

Every /v1/chat/completions response from llama-server includes a timings object. prompt_per_second and predicted_per_second are the two speeds. We ran this script in the sandbox (sbx exec $id "$(cat bench.sh)"):

KEY=$(cat ~/.llama-api-key)
for i in 1 2 3; do
  curl -s http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":"Explain what a microVM is in three sentences."}],"max_tokens":128,"temperature":0,"cache_prompt":false}' \
  | python3 -c "import json,sys; t=json.load(sys.stdin)['timings']; print('run $i prompt_n=%d prompt_tok_s=%.1f predicted_n=%d gen_tok_s=%.1f' % (t['prompt_n'], t['prompt_per_second'], t['predicted_n'], t['predicted_per_second']))"
done

On default:

run 1 prompt_n=40 prompt_tok_s=178.7 predicted_n=87 gen_tok_s=35.2
run 2 prompt_n=40 prompt_tok_s=302.5 predicted_n=87 gen_tok_s=36.9
run 3 prompt_n=40 prompt_tok_s=307.4 predicted_n=87 gen_tok_s=42.4

On large:

run 1 prompt_n=40 prompt_tok_s=233.6 predicted_n=87 gen_tok_s=61.3
run 2 prompt_n=40 prompt_tok_s=371.1 predicted_n=87 gen_tok_s=57.2
run 3 prompt_n=40 prompt_tok_s=420.8 predicted_n=87 gen_tok_s=60.7

Keep "cache_prompt":false when you benchmark. Our first pass left the default on, and runs 2 and 3 showed prompt_n=1: the server reused the cached prompt and processed one token, which makes prompt speed meaningless. In production, leave the cache on. A 40-token prompt is too short for a stable prompt speed. Use llama-bench for that number.

6. Give it a private HTTPS URL#

sbx host $id 8080 --title llama-server
https://broad-coastal-cove-8080.box.sandboxer.com?_token=<token>

Now there are two locks. The hosted URL's _token keeps everyone else from reaching the server at all. The llama-server API key is checked by the server itself. From our workstation:

BASE=https://broad-coastal-cove-8080.box.sandboxer.com
TOKEN=<token>          # from sbx host
LLAMA_KEY=<llama-key>  # the contents of ~/.llama-api-key

curl -s -o /dev/null -w "%{http_code}\n" $BASE/health                                   # 401 from the proxy
curl -sL -c jar -b jar "$BASE/v1/models?_token=$TOKEN"                                  # 401 from llama-server
curl -s "$BASE/v1/chat/completions?_token=$TOKEN" -H "Authorization: Bearer $LLAMA_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5-0.5b","messages":[{"role":"user","content":"Name the capital of France in one word."}],"max_tokens":16,"temperature":0}'
401
{"error":{"message":"Invalid API Key","type":"authentication_error","code":401}}
{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Paris."}}],…,"usage":{"completion_tokens":3,"prompt_tokens":38,"total_tokens":41,…},"timings":{…,"prompt_per_second":213.2,…,"predicted_per_second":52.6}}

A GET with ?_token= answers 302 and sets a cookie, so use -L and a cookie jar for GET requests. A POST with the token goes straight through. Five short requests (max_tokens: 8) from our workstation to the sandbox in eu-central took 0.35 to 0.51 s each. For more about private URLs, see Give a sandbox port an HTTPS URL.

7. Call it with the openai Python client#

Point base_url at the hosted URL plus /v1. Put the llama-server key in api_key, which the client sends as the Authorization header. Pass the hosted URL's token with default_query, so the client adds ?_token= to every request. Don't paste the token into base_url: the client appends paths to it.

import os, time
from openai import OpenAI

client = OpenAI(
    base_url="https://broad-coastal-cove-8080.box.sandboxer.com/v1",
    api_key=os.environ["LLAMA_API_KEY"],                    # llama-server's --api-key
    default_query={"_token": os.environ["SBX_HOST_TOKEN"]}, # the hosted URL's token
)

print([m.id for m in client.models.list()])

t = time.time()
r = client.chat.completions.create(
    model="qwen2.5-0.5b",
    messages=[{"role": "user", "content": "Write one sentence about microVMs."}],
    max_tokens=64,
    temperature=0,
)
print(r.choices[0].message.content)
print(r.usage.prompt_tokens, r.usage.completion_tokens, "%.2fs" % (time.time() - t))

stream = client.chat.completions.create(
    model="qwen2.5-0.5b",
    messages=[{"role": "user", "content": "Count from 1 to 5."}],
    max_tokens=32,
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
['qwen2.5-0.5b']
MicroVMs are small, lightweight virtual machines designed to run on low-end hardware, offering a more efficient and cost-effective alternative to traditional virtualization solutions.
37 32 1.02s
Sure, I can count from 1 to 5 for you.

1. 1
2. 2
3. 3
4.

We ran this with openai 3.39.0. Streaming works through the hosted URL. The answer about microVMs is loose, and the count stopped at max_tokens. A 0.5B model is fast on a CPU and fine for classification, extraction or test fixtures. Don't rely on it for facts.

8. Stop, resume and cold boot#

sbx stop $id --wait
sbx resume $id

The resume took 1.4 s and reported memoryRestored: true. llama-server kept its PID (750), the model stayed in memory, and the hosted URL answered at once with the same token.

Then we stopped again and ran sbx resume $id --type large. A size change boots from disk, so the resume reported memoryRestored: false with resumeReason: "resized". The systemd unit started llama-server again with a new PID. The first request through the URL got 502: nothing was answering on port 8080 yet, because llama-server was still starting and loading the model. The retry a moment later got 200. After a cold boot, retry a 502 for a few seconds until the server is up. The same cold boot happens when a resume lands on a machine with a different CPU type.

FAQ#

Does a Sandboxer sandbox have a GPU?#

No. Sandboxes are CPU-only microVMs. Use the plain ubuntu-x64 llama.cpp build, and pick small quantized models. In our test, a 0.5B model generated 35 to 61 tokens per second and a 1.5B model about 22.

What thread count should I use?#

One thread per vCPU: 4 on default, 8 on large. In our sweep, more threads were faster on both sizes. Extra cores help prompt processing far more than generation.

Do I need a Hugging Face token to download GGUF files?#

Not for public repos. A plain curl of the resolve/main URL worked for Qwen's GGUF repos, with no account.

Is the llama-server API open to the internet?#

Only through the hosted URL, which is private by default. Requests without the URL's token get 401 from the proxy before they reach llama-server. Its own --api-key is a second lock for anyone you share the URL with.

Which model sizes fit?#

The GGUF file has to fit in RAM with room for the context. A 1.1 GB file is easy on default (8 GB). Use large (16 GB) for bigger models. Bigger models generate more slowly: the 1.5B model ran at about half the speed of the 0.5B.

More guides

Top AI sandbox platforms for code execution in 2026, ranked

For running AI-written code we rank Sandboxer, Freestyle and E2B first: each gives code its own VM with an API or SDK and can pause with memory. Pick Daytona for Windows or GPUs, Modal or Northflank for GPUs at scale, E2B or Northflank to run in your own cloud, and Vercel Sandbox if your app lives on Vercel.

Read the guide
Best sandboxes for running coding agents (Claude Code, Codex, Gemini CLI) in 2026, ranked

For coding agents we rank Sandboxer first: a full Ubuntu VM with Claude Code and Codex preinstalled, Docker, and stop-and-resume that keeps the agent's session in memory. Freestyle and Amp Orbs also keep memory across pauses. Pick E2B for prebuilt agent templates on open source, Docker Sandboxes for free local runs, and Codespaces if you live in GitHub.

Read the guide
Best Daytona sandbox alternatives in 2026, ranked

The best Daytona alternatives depend on why you're switching. For a full Linux microVM on every sandbox with memory-saving stop and Docker, we rank Sandboxer first. E2B suits open source and self-hosting, Modal massive short-job fleets, Northflank bring-your-own-cloud with GPUs, and Vercel Sandbox apps on Vercel.

Read the guide
Best E2B alternatives in 2026, ranked

If you're leaving E2B for longer sessions or a fuller machine, we rank Sandboxer first, then Freestyle: both give each sandbox a full Linux VM that keeps memory across pauses, with Docker and no short session cap. Pick Daytona for Windows or GPUs, Northflank for bring-your-own-cloud, and stay on E2B if you need open source.

Read the guide