Linux Memory Explained: Swap, the OOM Killer and Why Your Service Died

If your database or application stopped suddenly with nothing in its own logs, the first command to run is dmesg -T | grep -i "killed process". If a line containing Out of memory: Killed process shows up, you know the culprit: the OOM killer, a Linux kernel mechanism that picks a process and terminates it by force when memory runs out completely, so the whole server does not freeze. Understanding what happened takes three numbers read correctly. The first is the available column in free -h, which is memory actually available to new workloads, not the free column, which always looks small because Linux uses empty memory as disk cache. The second is swap activity in vmstat 1, where sustained non zero values in the si and so columns mean the server is choking. The third is one process growing day after day, which is the signature of a leak. The fix depends on the cause: a small swap as a safety net, protecting the database from the OOM killer through systemd, sensible PHP-FPM worker limits, and sometimes the cheapest and fastest answer is an extra 2GB of RAM for 600 SAR per year instead of weeks of tuning. This article walks through each step with commands and numbers.

Reading free Correctly

Here is typical output from a server with 8GB of memory:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.8Gi       3.1Gi       312Mi       180Mi       4.4Gi       4.3Gi
Swap:          2.0Gi        64Mi       1.9Gi

Plenty of people look at the free column, see only 312MB and assume the server is about to fall over. In reality Linux puts unused memory to work as file cache (the buff/cache column) because empty memory is wasted memory, and when an application needs it the cache is released within a fraction of a second. The number that matters is available, 4.3GB here, which means this server is perfectly comfortable.

Available as share of total	State	What to do
Above 30%	Comfortable	Nothing, just watch the trend
10% to 30%	Needs attention	Review the biggest consumers and their limits
Below 10%	Danger	OOM or swap thrashing is close, act now

To find the biggest consumers ranked by resident memory (RSS):

ps -eo pid,user,comm,rss --sort=-rss | head -10

RSS is shown in kilobytes, so 1,048,576 is roughly 1GB.

Swap: Safety Net or a Cover for the Real Problem

Swap is disk space the system uses when memory fills up, moving rarely used pages out to it. It helps in two cases: absorbing a short, unexpected spike without the OOM killer stepping in, and moving genuinely idle pages (parts of services that run once a day, for example) out of the way so memory can serve as cache. It becomes a problem when the server leans on it constantly, because reading from disk is hundreds to thousands of times slower than memory, even on SSD.

This command shows the difference:

vmstat 1 10

Watch the si (swapped in) and so (swapped out) columns. A fixed amount of used swap such as 64MB with si and so at zero most of the time is perfectly normal. Sustained values in the hundreds or thousands every second are thrashing: the server spends its time shuffling pages instead of working, and a page that loaded in 200ms now takes 8 seconds. Swap did not protect you here. It hid a memory shortage and turned a quick crash into a long, painful slowdown.

Our recommendation for a cloud server with 2GB to 16GB of memory is a swap file of 1GB to 2GB as a safety net, with vm.swappiness lowered from the default of 60 to 10 so the kernel prefers dropping cache over pushing application memory to disk:

fallocate -l 2G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab

sysctl vm.swappiness=10
echo 'vm.swappiness=10' > /etc/sysctl.d/99-swappiness.conf

How the OOM Killer Picks Its Victim

When memory and swap are both exhausted, the kernel calculates an oom_score from 0 to 1000 for every process, based mainly on how much memory it uses, then kills the highest scorer. The problem is obvious. The database is usually the biggest memory user on the server, because InnoDB deliberately reserves a large buffer pool, so it is often the first victim even though it is the last thing you want to lose.

To see the current scores:

for p in $(pgrep -x mysqld) $(pgrep -f php-fpm | head -3); do
  echo "$p $(cat /proc/$p/comm) $(cat /proc/$p/oom_score)"
done

You influence the choice with oom_score_adj, ranging from -1000 to 1000. A value of -1000 means never kill this process, which is risky because it can push the kernel into killing essentials like sshd and locking you out. The right approach is to make the database a low priority target without full immunity, using a systemd drop in:

systemctl edit mysql

[Service]
OOMScoreAdjust=-800

Then run systemctl restart mysql. In the other direction you can raise the value for queue workers or PHP-FPM to something like 300, so if the kernel has to kill, it kills a worker that restarts in a second rather than a database that needs minutes to recover.

The best protection of all is never reaching OOM, which comes down to doing the arithmetic. On an 8GB server running a PHP application and MySQL together, a reasonable split is 3GB for innodb_buffer_pool_size, about 1GB for the OS and small services, and the remaining 4GB for PHP-FPM workers. If one worker averages 80MB, the safe ceiling for pm.max_children is 4,096 divided by 80, roughly 50 workers. The classic mistake is leaving it at 200 or more, and everything runs quietly until the first real peak.

Spotting a Slow Leak

The hardest OOM cases are the ones that hit every 9 or 12 days like clockwork with no traffic spike. That pattern is a leak: one process takes a little more memory every day and never gives it back. To catch it, log the top consumers every hour:

# in crontab
0 * * * * ps -eo pid,comm,rss --sort=-rss | head -5 | sed "s/^/$(date +\%F_\%H) /" >> /var/log/rss-track.log

After three days, open the file and look for a single PID whose RSS climbs in a straight line, say 400MB, then 520MB, then 650MB, never dropping. That is almost certainly a leak. The usual sources are:

• Long lived queue workers: a Laravel or Node.js worker running for weeks and accumulating objects. Cap jobs or memory and restart automatically, for example with --max-jobs=1000 or --memory=256.
• PHP-FPM: pm.max_requests = 500 recycles each worker after 500 requests, clearing any leak in PHP extensions.
• Unbounded in app caches: an array or Map that gains keys and never loses them. Set a size cap or move the cache to Redis with a maxmemory policy.

To confirm the fix, keep watching the same file for a full week. You want a flat line wobbling around a steady value.

When the Answer Is More Memory

Tuning matters, but it has an economic limit. Say your cloud server runs the 2 vCPU package with 4GB of memory, and available drops under 10% during working hours every day even though the settings are sensible and there is no leak. You have two paths:

Option	Cost	Time	Risk
Weeks of query tuning and trimming workers	Developer hours, possibly more than 10 days	Two to four weeks	Repeated outages while experimenting
+2GB RAM	600 SAR per year, about 1.64 SAR a day	Minutes	Close to none

The rule we use is simple. If the pressure comes from genuine growth rather than a leak, extra memory costs less than a single day of developer time. If it is a leak, more memory only pushes the crash from 9 days to 14, and the real fix is the source.

A Five Step Action Plan

1. Run dmesg -T | grep -i oom and journalctl -k | grep -i "out of memory" to learn whether OOM fired and who it killed.
2. Read available in free -h and watch si and so in vmstat 1 during peak hours.
3. Size the database, PHP-FPM and queue limits so their total stays under 85% of memory.
4. Protect the database with OOMScoreAdjust=-800, and add 1GB to 2GB of swap with swappiness set to 10.
5. Log RSS hourly for three days to rule out a leak, then choose between fixing the code and adding memory.

On a CLOUD cloud server you can add 2GB of RAM for 600 SAR per year without rebuilding the server or moving your data, with servers in Riyadh at around 8ms latency on SSD storage. Every package and add-on is on the pricing page.