Class 53 of 55 • Advanced Linux Administration

Linux Performance Monitoring Tutorial: top, htop, vmstat, iostat & free

When an application slows down, web requests time out, or server memory exhausts, system administrators must rapidly identify whether the bottleneck is caused by CPU exhaustion, memory swapping, storage disk I/O saturation, or network congestion. Linux exposes an extraordinary wealth of kernel diagnostic metrics through the /proc virtual filesystem.

This guide teaches you how to systematically diagnose performance issues using the 5 essential monitoring pillars: top / htop (CPU & process activity), free (RAM & swap), vmstat (virtual memory & kernel context switches), iostat (disk transfer metrics), and sar (historical system analysis).

---

1. Understanding Linux Load Average

The first metric every sysadmin checks upon logging in is the Load Average, displayed by uptime or top:

$ uptime
 08:35:10 up 45 days,  2:14,  3 users,  load average: 0.85, 1.42, 2.10
							

The three numbers represent system load averaged over the last 1 minute, 5 minutes, and 15 minutes. In Linux, load average counts both:

     1. Processes actively executing or waiting in the CPU run queue (TASK_RUNNING).

     2. Processes blocked waiting for uninterruptible disk I/O (TASK_UNINTERRUPTIBLE).

How to interpret: Compare the load average to your total number of logical CPU cores (check with nproc):

• On a 4-core CPU, a load average of 4.00 means 100% core utilization with zero waiting queue.

• A load of 8.00 on 4 cores means the system is 100% overloaded (4 processes are queued and waiting for every process executing).

---

2. Real-Time CPU & Process Monitoring with top & htop

The standard tool for real-time process monitoring is top. Essential interactive shortcut keys:

Key	Action	Diagnostic Benefit
P	Sort by %CPU (Default)	Instantly surfaces the top CPU-consuming process.
M	Sort by %MEM	Finds memory-hungry applications and memory leaks.
1	Toggle per-CPU view	Reveals single-threaded bottlenecks across multi-core CPUs.
k	Kill process	Prompts for PID and signal (e.g. 15 for SIGTERM, 9 for SIGKILL).
c	Toggle command line	Shows full command path and execution arguments.

# Launch modern colorful interactive process viewer (install via apt/dnf)
$ htop
							

---

3. Memory & Swap Diagnostics with free

Linux manages RAM with sophisticated caching strategies. Inspect memory with free -h:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       4.2Gi       1.8Gi       350Mi       9.8Gi        10Gi
Swap:          4.0Gi       256Mi       3.7Gi
							

Key columns explained:

     • buff/cache: Memory allocated for disk buffers and filesystem page cache. This memory is NOT lost; Linux releases it to user processes instantaneously when requested.

     • available: The actual amount of memory available for starting new applications without swapping. Always monitor this column!

---

4. Virtual Memory & Context Switches with vmstat

vmstat reports kernel threads, memory pages, block I/O, and CPU activity in continuous samples:

# Sample system state every 1 second for 5 iterations:
$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0 262144 194560 382400 9850200   0    0    12    45  850 1200 14  4 81  1  0
							

Critical alert indicators in vmstat:

• b > 0: Processes blocked on disk I/O.

• si / so > 0: Swap-in / Swap-out activity! Non-zero values indicate the server is out of physical RAM and paging to disk, causing severe latency spikes.

• wa > 15%: CPU time spent waiting for slow disks.

---

5. Storage Disk I/O Diagnostics with iostat

From the sysstat package, iostat reveals disk latency and throughput bottlenecks:

$ sudo iostat -xz 1 3
Device     r/s     w/s     rkB/s     wkB/s  rrqm/s  wrqm/s  %rrqm  %wrqm  r_await w_await aqu-sz  %util
sda       4.00  120.00    160.00   4820.00    0.00   15.00   0.00  11.11     1.20    2.50   0.15  12.40
nvme0n1  85.00  650.00   3400.00  28400.00    0.00   40.00   0.00   5.80     0.45    0.80   0.22  24.80
							

Red flag metrics: If %util is near 100%, the storage controller is saturated. If await exceeds 15-20 milliseconds on SSDs, disk I/O queues are backing up.

---

Beginner Student Lab Practice Session

# Task 1: Check total number of CPU processor cores
$ nproc
$ lscpu

# Task 2: Check active memory and swappiness kernel parameter
$ cat /proc/sys/vm/swappiness  # 60 is default; servers often use 10

# Task 3: Find the top 5 memory-consuming processes
$ ps aux --sort=-%mem | head -n 6

# Task 4: Find the top 5 CPU-consuming processes
$ ps aux --sort=-%cpu | head -n 6

# Task 5: View network socket buffers and connection counts
$ ss -s
							

---

Frequently Asked Questions (FAQ)