vLLM for Production LLM Serving: When It Beats Ollama

vLLM beats Ollama once you have a GPU and more than 5 to 10 requests arriving at the same moment, because vLLM is built as a production inference server. It manages the KV cache with PagedAttention so memory is not wasted, groups requests through continuous batching so one card serves dozens of users at several times the throughput of handling requests one by one, and exposes an OpenAI-compatible API out of the box. Ollama is the better choice for a small team of 3 to 15 people, or when there is no GPU at all, since it runs compressed GGUF models on the CPU with a single command and far less memory. The short rule: Ollama for experiments, small teams, and CPU servers, and vLLM once the model becomes a service that a product or hundreds of users depend on.

A Developer Tool Versus an Inference Server

Both run a language model and give you an HTTP API, but they aim at different jobs. Ollama was designed around a single developer's experience: you pull a model with ollama pull, run it, and it loads and unloads models from memory on demand. That is excellent on a developer machine or an internal team server, but it was not built to serve hundreds of concurrent requests efficiently.

vLLM takes the opposite view. It assumes the model stays loaded on the GPU all the time and that its only goal is to push out as many tokens per second as possible to as many users as possible. That is why it reserves a share of GPU memory up front (90% by default) and manages it itself.

Criterion	Ollama	vLLM
Ideal hardware	CPU or a single GPU	One or more GPUs (usually NVIDIA)
Model format	Compressed GGUF (Q4, Q8)	Hugging Face weights, plus AWQ, GPTQ, and FP8
Concurrency	A few parallel requests, then a queue	Dozens to hundreds via continuous batching
OpenAI API	Available at /v1	Available at /v1 as the primary interface
Getting started	Minutes	Half an hour to an hour with drivers and CUDA
Best fit	Internal team, prototypes, CPU server	Live product, customer-facing API, sustained load

PagedAttention and Continuous Batching in Plain Terms

The Memory Problem

During generation, the model keeps a memory called the KV cache for each conversation, and it grows with every token. Traditional servers reserve enough space per request for the longest possible reply, so if the limit is 4,096 tokens and the actual reply is 300, most of that reservation is wasted. The vLLM research paper measured that earlier systems wasted 60% to 80% of KV cache memory this way.

PagedAttention borrows the idea of virtual memory from operating systems. It splits memory into small fixed-size pages and hands each request pages as it actually needs them, bringing waste below 4%. The direct result is that the same card fits far more simultaneous conversations.

Continuous Batching

The older approach collects 8 requests into a batch and waits for the longest one to finish before accepting anything new, so the ninth user keeps waiting even after seven replies are done. vLLM admits new requests and releases finished ones at every generation step, so the card never sits idle. The original paper reported up to 24 times the throughput of serving through the standard Transformers library, and in production the practical gain over simpler servers usually lands between 2 and 4 times.

Running vLLM With an OpenAI-Compatible API

Once NVIDIA drivers and CUDA are installed, the basic launch is one command:

pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 64 \
  --api-key "$VLLM_API_KEY"

The options that matter most: --max-model-len sets the longest context, and lowering it raises the number of concurrent users because each conversation needs less memory. --max-num-seqs caps the requests in a single batch. With more than one card, --tensor-parallel-size 2 splits the model across them.

From there, any application written for the OpenAI API works by changing only the base URL:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="...")
r = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Summarize this contract in 5 points"}],
)

Never expose port 8000 directly to the internet. Put Nginx in front with TLS and a rate limit, and use the Prometheus metrics at /metrics to track queue length and time to first token.

Memory Requirements by Model Size

A rough rule: each billion parameters needs 2GB at FP16 and about 0.5 to 0.6GB with 4-bit quantization, and on top of that you add KV cache space, which decides how many users you can serve at once.

Model size	FP16 (weights only)	4-bit AWQ/GPTQ	Suitable GPU for vLLM
3B	About 6GB	About 2GB	16GB with plenty of headroom
7B to 8B	14 to 16GB	4 to 5GB	24GB for FP16, 16GB quantized
14B	About 28GB	About 9GB	48GB for FP16, 24GB quantized
32B	About 64GB	About 18 to 20GB	80GB, or two 48GB cards
70B	About 140GB	About 40GB	Two 80GB cards for FP16, or one 48GB card quantized with a short context

A worked example: an 8B model at FP16 on a 24GB card takes 16GB for weights and leaves about 5GB for the KV cache after the 90% reservation. With an 8,192-token context that covers roughly 10 to 15 long conversations at once, or many more if messages are short.

A Checklist Before Putting vLLM Into Production

Running the command is one thing, and running a service your customers depend on is another. These points separate an experiment from production:

• Run it as a systemd service with automatic restart, since running out of GPU memory on an unexpectedly long request can kill the process.
• Pin the model and library versions, because a vLLM upgrade or changed weights on Hugging Face can shift answer behavior without you noticing.
• Watch three numbers: time to first token (aim for under a second in chat), queue depth, and KV cache usage. If usage stays above 90%, you need a shorter context or another card.
• Cap request and reply length in the application so one user cannot reserve the memory of ten.
• Load test before launch with a tool that simulates 20 or 50 concurrent users, and record throughput in tokens per second so you know your real ceiling.

The checklist takes one working day, and it turns your first failure from a midnight surprise into an alert you handle before anyone notices.

When Ollama on a CPU Server Is Enough

Not every team needs a GPU. If usage is internal, such as summarizing documents, assisting the legal team, or classifying tickets, with fewer than 15 users who rarely ask in the same second, Ollama on a CPU server does the job at a fixed, predictable cost. Rough numbers on modern server processors:

• A 3B model at Q4 on 4 vCPU and 8GB: 10 to 20 tokens per second, enough for classification and short summaries.
• A 7B to 8B model at Q4 on 8 vCPU and 16GB: 4 to 8 tokens per second, acceptable for internal chat.
• Models of 14B and above on CPU are too slow for interactive use, so keep them for overnight background jobs.

The Enterprise plan on a cloud server, with 8 vCPU and 16GB at 18,000 SAR a year, runs a quantized 8B model for an internal team, and each extra 2GB of RAM costs 600 SAR a year. CLOUD's AI hosting ships with an Open WebUI and Ollama template preinstalled at no extra charge, so you can start real testing within minutes while your data stays on a server in Riyadh instead of going to an outside provider.

Tune Ollama for modest concurrency with environment variables:

OLLAMA_NUM_PARALLEL=4
OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_KEEP_ALIVE=24h

The last one keeps the model loaded, so the first user in the morning does not wait 20 seconds for it to load.

How to Decide

1. Measure real peak concurrency, meaning requests in the same second rather than registered users. Fifty employees often means only 2 to 5 concurrent requests.
2. If the peak is under 5 and a reply time of 10 to 30 seconds is acceptable, start with Ollama on CPU and measure.
3. If the queue regularly exceeds 10 seconds, or the model becomes part of a product your customers use, move to vLLM on a GPU.
4. Write your application against the OpenAI API from day one, so moving from Ollama to vLLM is a single URL change in your config file.

To find the right size and the cost of extra resources, see pricing and plans.