What Is the OOM Killer?
When the Linux kernel runs out of memory, it invokes the OOM killer, which terminates processes to free memory. This often causes unexpected application crashes.
Detecting OOM Events
>_BASH
$# Check kernel logs for OOM events$dmesg | grep -i "out of memory"$dmesg | grep -i "oom"$$# Check system logs$grep -i "killed process" /var/log/syslog$journalctl -k | grep -i "oom"Example OOM log:
[12345.678] Out of memory: Kill process 1234 (python3) score 900 or sacrifice child
[12345.679] Killed process 1234 (python3) total-vm:8192000kB, anon-rss:7000000kB
Step 1 — Check Current Memory Usage
>_BASH
$# Overview$free -h$$# Detailed per-process$ps aux --sort=-%mem | head -20$$# Memory map of a process$pmap -x $PID | tail -5Step 2 — Identify Memory Leaks
>_BASH
$# Monitor memory growth over time$watch -n 5 'ps aux --sort=-%mem | head -10'$$# Valgrind (for C/C++ programs)$valgrind --leak-check=full ./myappStep 3 — Add Swap Space
>_BASH
$# Create a 4 GB swap file$fallocate -l 4G /swapfile$chmod 600 /swapfile$mkswap /swapfile$swapon /swapfile$$# Make persistent$echo '/swapfile none swap sw 0 0' >> /etc/fstab$$# Verify$swapon --showNote: Swap is much slower than RAM. It prevents crashes but degrades performance.
Step 4 — Tune OOM Killer
>_BASH
$# Protect a critical process from OOM killer$echo -1000 > /proc/$PID/oom_score_adj$$# Make a process more likely to be killed first$echo 1000 > /proc/$PID/oom_score_adjStep 5 — Set Memory Limits
>_BASH
$# Limit a service's memory usage$systemctl set-property myapp.service MemoryMax=2G$$# Docker container limit$docker run --memory=4g --memory-swap=4g myimageGPU OOM Errors
For CUDA out of memory errors, see Optimise GPU Memory Usage.