Model Quantization on Ollama: Running Bigger Models on Less RAM

Model quantization on Ollama means shrinking a model's weights from 16 bits (the full version) down to 4, 5, or 8 bits, which cuts the model's memory footprint sharply with a quality loss that is usually too small to notice in daily use. The direct benefit is running a bigger, smarter model on far less RAM. A 7-billion-parameter model (7B) in its full version needs about 14GB of memory, but at Q4 quantization it shrinks to just 4 to 5GB, so it runs comfortably on a server with 8GB of RAM. The practical rule is that Q4 is the most popular balance point between quality and size and covers most uses, Q5 gives slightly higher quality for more memory, and Q8 is very close to the full version but uses roughly double Q4. In practice, a 7B model fits an 8GB server, a 13B model fits 16GB, and 30B and larger models need 32GB or more. Ollama makes all of this easy because it pulls ready quantized versions with a single command, so you type the model name with the quantization tag and it handles the rest. This guide explains what quantization actually is and its effect on quality, gives a table linking model size to required RAM, covers when a CPU is enough and when you need a GPU, and shows practical Ollama commands on a CLOUD server with Riyadh servers at around 8ms response time.

What Quantization Is and Why It Saves Memory

A language model at its core is billions of numbers, the weights, that define its behavior. In its original version each number is stored at high precision taking 16 bits. Quantization reduces the number of bits per number, so instead of 16 bits it stores 8, 5, or 4, much like compressing a high-resolution image to a smaller size by reducing its colors. The result is that the model occupies far less memory. Roughly: the full version (16 bits) needs about twice the parameter count in gigabytes, so a 7B model needs around 14GB. Q8 quantization roughly halves that (about 7GB), and Q4 roughly quarters it (about 4GB). This difference is what turns a model impossible to run on your server into one that runs smoothly. Importantly, quantization also reduces the amount of data moved from memory during generation, which speeds up responses on a CPU, a double benefit.

The Effect on Quality: Where the Red Line Is

The logical question is whether you lose quality when you reduce precision. The answer is yes, but less than you would expect. The difference between the full version and Q8 is barely noticeable in any practical use. The difference between Q8 and Q5 is very small in most tasks. Q4 starts to show a slight difference in precise tasks like complex reasoning or coding, but it stays excellent for chat, summarization, and general questions. Quantization more aggressive than Q4, like Q3 and Q2, is where degradation becomes noticeable, and it is best avoided except for experimenting on very limited hardware. The practical rule is to start with Q4 as the best balance, and move up to Q5 or Q8 only if you notice a shortfall in your task's quality and the memory allows.

Table: Which Model Size Fits Your Server's RAM

This table links model size at Q4 quantization (the most common) to the practical RAM required, with the fitting CLOUD plan. Numbers are approximate and leave headroom for the system and context.

Model size	Required memory (Q4)	Fitting CLOUD plan	Good for
3B and smaller	About 3 to 4GB	Growth 4GB	Light tasks, classification, fast replies
7B to 8B	About 5 to 6GB	Business 8GB	General chat, summarization, internal assistant
13B to 14B	About 9 to 11GB	Enterprise 16GB	Higher quality, coding, better reasoning
30B to 34B	About 20 to 24GB	Enterprise 16GB + RAM add-on	Complex tasks needing precision

CLOUD plans annually with VAT: Growth (2vCPU/4GB) at 6,900 SAR, Business (4vCPU/8GB) at 11,500, Enterprise (8vCPU/16GB) at 18,000, and a 2GB RAM add-on at 600 SAR per year, so you raise your server to the model size you want with flexibility. See cloud server and plans and pricing to estimate the right size.

CPU or GPU: When a Processor Is Enough

A common question is whether you need a graphics card to run models. The answer depends on model size and workload. A CPU is perfectly enough for small and medium quantized models (up to around 13B) in single-user or light-load use, and gives acceptable speed for chat and summarization. But as the model grows or the number of concurrent users rises, the need for a GPU appears to speed up generation many times over (far higher tokens per second). The practical rule is to start with the CPU and a quantized model and test the speed on your real load. If it is enough you save the higher GPU cost, and if you need faster responses or to serve several users at once, then a GPU is worth its price. The principle is not to pay for pricier hardware before you prove you actually need it.

Practical Ollama Commands

Ollama makes quantization simple because it provides ready quantized versions. To run a model at a given quantization you write its name with the tag, for example you pull a 7B model at Q4 and run it immediately, and Ollama handles the download and memory. You can try more than one quantization level for the same model and compare quality and speed on your task before settling on one. Models are stored locally on your server, so your data stays with you without being sent to any external party, which is the core privacy benefit of self-hosting. AI hosting on CLOUD makes starting easy because it is a cloud server with an Open WebUI and Ollama template preinstalled and ready, so you reach your own chat interface in minutes. See AI hosting for details.

Beyond RAM: Other Factors That Shape the Experience

Memory is the most important constraint but not the only one. The context window size you run the model with consumes extra memory beyond the weight size, so the longer the context you request (to read large documents, for example) the more RAM you need on top of what the table shows, so always leave headroom. The number of concurrent users is a second factor, since a model serving one person differs from one serving a whole team at once, and concurrent load raises the need for processing and possibly a GPU. Disk speed affects the first-time model load, and an SSD makes this fast. Finally, the CPU core count (vCPU) directly affects generation speed on a CPU, so a four-core plan is noticeably faster than a two-core one for the same model. The practical rule is to estimate your need from the sum of these factors, not RAM alone, and start with a reasonable size then scale up when needed, since upgrades are instant with no downtime.

A Practical Summary

Start by choosing a model that fits your task, then pick Q4 quantization as a starting point for the balance between quality and memory. Match the model size to your server's RAM from the table above: 7B on 8GB, 13B on 16GB. Test on the CPU first and measure speed on your real load before paying for a GPU. If quality falls short on a precise task, move up to Q5 or Q8 or add RAM and step up to a bigger model, since upgrades on CLOUD are instant. The bottom line: quantization is what puts local AI within reach of an ordinary server, giving you your own assistant that respects your data privacy at a reasonable cost.