# How Do I Run llama.cpp's llama-server on CPU in a Cloud Sandbox?

> Download the prebuilt Linux x64 build from llama.cpp's GitHub releases, fetch a GGUF model from Hugging Face with curl, and run llama-server on 0.0.0.0 as a systemd service. Then sbx host the port for a private HTTPS URL that any OpenAI client can call. A 0.5B model generated 35 to 48 tokens per second on default.

Source: https://sandboxer.com/guides/run-llama-cpp-server-on-cpu-in-a-sandbox. Updated 2026-10-27.

**Short answer:** Download the prebuilt Linux x64 build from llama.cpp's GitHub releases, fetch a GGUF model from Hugging Face with curl, and run llama-server on 0.0.0.0 as a systemd service. Then sbx host the port for a private HTTPS URL that any OpenAI client can call. A 0.5B model generated 35 to 48 tokens per second on default.

## At a glance

We ran everything below on prod on October 27, 2026: a `default` sandbox (4 vCPU, 8 GB RAM), later resumed as `large` (8 vCPU, 16 GB RAM). llama.cpp build `b11925`. **CPU only:** a sandbox has no GPU. We used the plain `ubuntu-x64` build, which picked its `zen4` CPU backend on its own.

| Step | What we saw |
| - | - |
| Download and unpack the release tarball | Under 1 s, no compiler needed |
| Download two GGUF files from Hugging Face | 469 MB in 7 s, 1.1 GB in 12 s, no token |
| `llama-bench`, Qwen2.5 0.5B Q4_K_M, `default`, 4 threads | 545 tokens/s prompt, 47 tokens/s generation |
| `llama-bench`, Qwen2.5 1.5B Q4_K_M, `default`, 4 threads | 236 tokens/s prompt, 23 tokens/s generation |
| Same two models on `large`, 8 threads | 0.5B: 525 / 60 tokens/s. 1.5B: 431 / 23 tokens/s |
| `llama-server` timings, 0.5B, real chat request | 35 to 48 tokens/s on `default`, 57 to 61 on `large` |
| Hosted URL, no token / token but no API key / both | `401` / `401` (from llama-server) / `200` |
| Short request from our workstation through the URL | 0.35 to 0.51 s round trip |
| `sbx stop --wait` + `sbx resume` | `memoryRestored: true`, same PID, URL answered at once |
| Resume as `large` (cold boot) | systemd started the server again; first request `502` (nothing listening yet), the retry `200` |

If you want a one-line install and a model library, [run Ollama instead](/guides/run-ollama-on-cpu-in-a-cloud-sandbox). Ollama is built on llama.cpp. Running llama.cpp directly gives you the exact build, any GGUF file on Hugging Face, `llama-bench`, and per-request `timings` in every response.

## 1. Download llama.cpp

The [llama.cpp releases page](https://github.com/ggml-org/llama.cpp/releases) publishes a prebuilt CPU build for Linux on every build number. Pin one so a rebuild gives you the same binary:

```bash
sbx new --name llama --type default
id=sb_pgchnyueil    # the id that sbx new printed

sbx exec $id "curl -fsSL -o llama.tar.gz https://github.com/ggml-org/llama.cpp/releases/download/b11925/llama-b11925-bin-ubuntu-x64.tar.gz \
  && mkdir -p llama && tar xzf llama.tar.gz -C llama --strip-components=1 && ./llama/llama-server --version"
```

```text
version: 0.7.0-dev (build 11925, commit fae9ac561)
built with GNU 11.4.0 for Linux x86_64
```

The tarball holds `llama-server`, `llama-cli`, `llama-bench` and one CPU backend library per CPU family (`libggml-cpu-haswell.so`, `libggml-cpu-zen4.so` and so on). At startup it loads the best one for the CPU it finds. Don't pick a `cuda`, `rocm` or `vulkan` asset: a sandbox has no GPU. To build from source instead, follow the repo's [build guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md).

## 2. Download a GGUF model

`llama-server` reads one `.gguf` file. Public repos on Hugging Face need no token. We used Qwen's official [0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF) and [1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF) instruct models at `Q4_K_M`:

```bash
sbx exec $id "mkdir -p models && cd models \
  && curl -fsSL -O https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  && curl -fsSL -O https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF/resolve/main/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  && ls -lh"
```

```text
-rw-r--r-- 1 user user 469M Oct 27 10:15 qwen2.5-0.5b-instruct-q4_k_m.gguf
-rw-r--r-- 1 user user 1.1G Oct 27 10:15 qwen2.5-1.5b-instruct-q4_k_m.gguf
```

The URL pattern is `https://huggingface.co/<repo>/resolve/main/<file>`. `llama-server -hf <user>/<repo>:<quant>` can also download for you, as the [server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) describes, but an explicit file path is easier to put in a systemd unit.

## 3. Measure tokens per second and pick a thread count

`llama-bench` measures prompt processing (`pp512`, 512 prompt tokens) and generation (`tg128`, 128 new tokens) separately. `-t 1,2,4` sweeps the thread count:

```bash
sbx exec $id --timeout 600 "./llama/llama-bench -m models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf -t 1,2,4 -p 512 -n 128 -r 3"
```

```text
| model                    |       size | threads |  test |            t/s |
| ------------------------ | ---------: | ------: | ----: | -------------: |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       1 | pp512 | 153.85 ± 15.70 |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       1 | tg128 |   28.42 ± 0.44 |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       2 | pp512 |  318.03 ± 2.46 |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       2 | tg128 |   35.45 ± 1.12 |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       4 | pp512 | 544.90 ± 38.33 |
| qwen2 1B Q4_K - Medium   | 462.96 MiB |       4 | tg128 |   46.69 ± 0.87 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       1 | pp512 |   67.64 ± 0.61 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       1 | tg128 |   15.36 ± 0.39 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       2 | pp512 | 111.38 ± 14.43 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       2 | tg128 |   20.72 ± 1.00 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       4 | pp512 | 236.24 ± 13.00 |
| qwen2 1.5B Q4_K - Medium |   1.04 GiB |       4 | tg128 |   22.64 ± 2.00 |
```

(Columns trimmed. llama-bench labels the 0.5B model "1B"; it is the 469 MB file.) Later we resumed the sandbox as `large` and ran `-t 2,4,8`:

| Model | Test | 4 threads | 8 threads |
| - | - | - | - |
| Qwen2.5 0.5B | pp512 | 403 | 525 |
| Qwen2.5 0.5B | tg128 | 47 | 60 |
| Qwen2.5 1.5B | pp512 | 242 | 431 |
| Qwen2.5 1.5B | tg128 | 22 | 23 |

What the numbers say:

- **Use one thread per vCPU.** On both sizes, the most threads gave the most speed. We didn't test more threads than vCPUs.
- **Prompt speed scales with cores. Generation speed mostly doesn't.** Doubling to 8 vCPUs nearly doubled prompt speed for the 1.5B model, but its generation moved from 22 to 23 tokens/s. Generation is usually limited by memory bandwidth, not core count.
- **`large` pays off for long prompts**, such as RAG with big retrieved contexts, or for models that need its 16 GB of RAM. For short prompts and long answers, `default` at $0.036 an hour gives almost the same generation speed for half the price. `large` isn't available during the free trial.

`llama-server` takes `-t` for generation threads and `-tb` for prompt processing threads. We left both unset, and the server's own timings, below, were in line with the full-vCPU `llama-bench` rows.

## 4. Run llama-server as a systemd service

A systemd service starts again after a cold boot. A process you start by hand doesn't. Bind to `0.0.0.0` so the hosting proxy can reach it, and give it an API key of its own:

```bash
sbx exec $id "umask 077 && openssl rand -hex 24 > ~/.llama-api-key"

sbx exec $id "sudo tee /etc/systemd/system/llama-server.service >/dev/null <<'EOF'
[Unit]
Description=llama.cpp server
After=network-online.target

[Service]
User=user
WorkingDirectory=/home/user
ExecStart=/home/user/llama/llama-server -m /home/user/models/qwen2.5-0.5b-instruct-q4_k_m.gguf --alias qwen2.5-0.5b --host 0.0.0.0 --port 8080 -c 4096 --api-key-file /home/user/.llama-api-key --metrics
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now llama-server"
```

- `--alias` sets the model name clients send and `/v1/models` lists. Without it, the name is the file path.
- `--api-key-file` makes every endpoint except `/health` require `Authorization: Bearer <key>`. Inside the sandbox, a request without it got `401`.
- Set `--host` and `--port` yourself. The server listens on `127.0.0.1` by default, which the hosting proxy can't reach.

## 5. Read prompt and generation speed from each response

Every `/v1/chat/completions` response from `llama-server` includes a `timings` object. `prompt_per_second` and `predicted_per_second` are the two speeds. We ran this script in the sandbox (`sbx exec $id "$(cat bench.sh)"`):

```bash
KEY=$(cat ~/.llama-api-key)
for i in 1 2 3; do
  curl -s http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":"Explain what a microVM is in three sentences."}],"max_tokens":128,"temperature":0,"cache_prompt":false}' \
  | python3 -c "import json,sys; t=json.load(sys.stdin)['timings']; print('run $i prompt_n=%d prompt_tok_s=%.1f predicted_n=%d gen_tok_s=%.1f' % (t['prompt_n'], t['prompt_per_second'], t['predicted_n'], t['predicted_per_second']))"
done
```

On `default`:

```text
run 1 prompt_n=40 prompt_tok_s=178.7 predicted_n=87 gen_tok_s=35.2
run 2 prompt_n=40 prompt_tok_s=302.5 predicted_n=87 gen_tok_s=36.9
run 3 prompt_n=40 prompt_tok_s=307.4 predicted_n=87 gen_tok_s=42.4
```

On `large`:

```text
run 1 prompt_n=40 prompt_tok_s=233.6 predicted_n=87 gen_tok_s=61.3
run 2 prompt_n=40 prompt_tok_s=371.1 predicted_n=87 gen_tok_s=57.2
run 3 prompt_n=40 prompt_tok_s=420.8 predicted_n=87 gen_tok_s=60.7
```

Keep `"cache_prompt":false` when you benchmark. Our first pass left the default on, and runs 2 and 3 showed `prompt_n=1`: the server reused the cached prompt and processed one token, which makes prompt speed meaningless. In production, leave the cache on. A 40-token prompt is too short for a stable prompt speed. Use `llama-bench` for that number.

## 6. Give it a private HTTPS URL

```bash
sbx host $id 8080 --title llama-server
```

```text
https://broad-coastal-cove-8080.box.sandboxer.com?_token=<token>
```

Now there are two locks. The hosted URL's `_token` keeps everyone else from reaching the server at all. The llama-server API key is checked by the server itself. From our workstation:

```bash
BASE=https://broad-coastal-cove-8080.box.sandboxer.com
TOKEN=<token>          # from sbx host
LLAMA_KEY=<llama-key>  # the contents of ~/.llama-api-key

curl -s -o /dev/null -w "%{http_code}\n" $BASE/health                                   # 401 from the proxy
curl -sL -c jar -b jar "$BASE/v1/models?_token=$TOKEN"                                  # 401 from llama-server
curl -s "$BASE/v1/chat/completions?_token=$TOKEN" -H "Authorization: Bearer $LLAMA_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5-0.5b","messages":[{"role":"user","content":"Name the capital of France in one word."}],"max_tokens":16,"temperature":0}'
```

```text
401
{"error":{"message":"Invalid API Key","type":"authentication_error","code":401}}
{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Paris."}}],…,"usage":{"completion_tokens":3,"prompt_tokens":38,"total_tokens":41,…},"timings":{…,"prompt_per_second":213.2,…,"predicted_per_second":52.6}}
```

A `GET` with `?_token=` answers `302` and sets a cookie, so use `-L` and a cookie jar for `GET` requests. A `POST` with the token goes straight through. Five short requests (`max_tokens: 8`) from our workstation to the sandbox in eu-central took 0.35 to 0.51 s each. For more about private URLs, see [Give a sandbox port an HTTPS URL](/guides/give-a-sandbox-port-an-https-url).

## 7. Call it with the openai Python client

Point `base_url` at the hosted URL plus `/v1`. Put the llama-server key in `api_key`, which the client sends as the `Authorization` header. Pass the hosted URL's token with `default_query`, so the client adds `?_token=` to every request. Don't paste the token into `base_url`: the client appends paths to it.

```python
import os, time
from openai import OpenAI

client = OpenAI(
    base_url="https://broad-coastal-cove-8080.box.sandboxer.com/v1",
    api_key=os.environ["LLAMA_API_KEY"],                    # llama-server's --api-key
    default_query={"_token": os.environ["SBX_HOST_TOKEN"]}, # the hosted URL's token
)

print([m.id for m in client.models.list()])

t = time.time()
r = client.chat.completions.create(
    model="qwen2.5-0.5b",
    messages=[{"role": "user", "content": "Write one sentence about microVMs."}],
    max_tokens=64,
    temperature=0,
)
print(r.choices[0].message.content)
print(r.usage.prompt_tokens, r.usage.completion_tokens, "%.2fs" % (time.time() - t))

stream = client.chat.completions.create(
    model="qwen2.5-0.5b",
    messages=[{"role": "user", "content": "Count from 1 to 5."}],
    max_tokens=32,
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

```text
['qwen2.5-0.5b']
MicroVMs are small, lightweight virtual machines designed to run on low-end hardware, offering a more efficient and cost-effective alternative to traditional virtualization solutions.
37 32 1.02s
Sure, I can count from 1 to 5 for you.

1. 1
2. 2
3. 3
4. 
```

We ran this with `openai` 3.39.0. Streaming works through the hosted URL. The answer about microVMs is loose, and the count stopped at `max_tokens`. A 0.5B model is fast on a CPU and fine for classification, extraction or test fixtures. Don't rely on it for facts.

## 8. Stop, resume and cold boot

```bash
sbx stop $id --wait
sbx resume $id
```

The resume took 1.4 s and reported `memoryRestored: true`. `llama-server` kept its PID (750), the model stayed in memory, and the hosted URL answered at once with the same token.

Then we stopped again and ran `sbx resume $id --type large`. A size change boots from disk, so the resume reported `memoryRestored: false` with `resumeReason: "resized"`. The systemd unit started `llama-server` again with a new PID. The first request through the URL got `502`: nothing was answering on port 8080 yet, because `llama-server` was still starting and loading the model. The retry a moment later got `200`. After a cold boot, retry a `502` for a few seconds until the server is up. The same cold boot happens when a resume lands on a machine with a different CPU type.

## FAQ

### Does a Sandboxer sandbox have a GPU?

No. Sandboxes are CPU-only microVMs. Use the plain `ubuntu-x64` llama.cpp build, and pick small quantized models. In our test, a 0.5B model generated 35 to 61 tokens per second and a 1.5B model about 22.

### What thread count should I use?

One thread per vCPU: 4 on `default`, 8 on `large`. In our sweep, more threads were faster on both sizes. Extra cores help prompt processing far more than generation.

### Do I need a Hugging Face token to download GGUF files?

Not for public repos. A plain `curl` of the `resolve/main` URL worked for Qwen's GGUF repos, with no account.

### Is the llama-server API open to the internet?

Only through the hosted URL, which is private by default. Requests without the URL's token get `401` from the proxy before they reach llama-server. Its own `--api-key` is a second lock for anyone you share the URL with.

### Which model sizes fit?

The GGUF file has to fit in RAM with room for the context. A 1.1 GB file is easy on `default` (8 GB). Use `large` (16 GB) for bigger models. Bigger models generate more slowly: the 1.5B model ran at about half the speed of the 0.5B.

## Related

- [Run Ollama on CPU in a cloud sandbox](/guides/run-ollama-on-cpu-in-a-cloud-sandbox)
- [Give a sandbox port an HTTPS URL](/guides/give-a-sandbox-port-an-https-url)
- [Machines and sizes](/docs/machines)
- [Long-running tasks](/docs/long-running)