Survival guide for troubleshooting degraded linux server performance

Exposing servers to the internet inevitably means exposing them to potential attackers, and waking up to an unresponsive server or extremely slow service. For the sake of this guide, we intentionally assume the issue is not caused by an attacker gaining access to the machine and executing commands on it directly. If you assume an attacker may have gotten access, treat the entire device as irrecoverably lost, as you cannot guarantee the integrity of stored data, nor can you ever be fully sure that an attacker has not left a backdoor or malicious script behind.

Is it still running?

If the server is not reachable over normal network protocols like SSH, you have to check if the server is still physically running and in a state that should be reachable. If you are experiencing a slow service or occasional errors but can otherwise reach anything on the server, be it ssh, hosted websites or databases, skip this step.


If none of the services on the server responds, there is a good chance the server is physically disconnected, off or dead.


For rented servers, no matter if dedicated or vps: Start by checking your emails. Your provider may be running maintenance, experience an outage or may have taken your server offline or deleted it because of an ongoing DDOS attack, suspected abuse or billing issues. These are surprisingly common in the real world.

If you have no emails, continue to the providers web interface/cloud console and verify that the server still exists, is connected to the expected network+ip/load balancer/floating ip and is currently running.


For self-hosted servers or colocation: Go and check the device physically or have an on-premise datacenter technician check the server for you. Verify that the LEDs in the front are on/blinking and the fans are spinning. Further verify that the server is connected to the correct router/switch and no cables were accidentally removed or connected incorrectly.

Getting a shell

Once you have confirmed that the server should be running and reachable, you can work on getting shell access to execute commands on it.


If enabled, try using SSH to access the machine as normal, but specify sh as your shell:

ssh user@sample.com sh

This ensures that no heavy scripts from .bashrc or similar run, reducing potential stress for the system caused by the connection.

If network access is not possible, you may be able to access the machine's serial console, either physically, via KVM console or through a hoster's web UI.

If nothing reacts and you cannot get an environment to execute commands in, you are stuck with having to (force) reboot the machine in recovery mode. While forcibly interrupting a running system can be dangerous, leaving it in a hanging state has little chance of self-recovery, and if it eventually crashes you are losing unflushed disk writes anyway. The only real difference is the time to recovery getting noticeably longer if you wait and hope the problem magically goes away.

Investigating hardware usage

Resist the urge to invoke heavy tooling like top or sar here and start with lightweight triaging tools. The first point to start at should always be:

uptime

The output may look like:

 19:14:55 up 5 days,  8:13,  1 user,  load average: 1.11, 1.12, 1.09

The sample gives us two important hints: up 5 days means it didn't just crash or restart, while the load average tells us that the CPU has not been under heavy pressure in the last few minutes.

Note that load average is an aggregate value of the last minutes, and the number must be interpreted using physical cpu core count, e.g. a dual core cpu would show 2.0 at 100% usage.


Next, check memory and swap:

free -mh

The output will include one row for memory, and another for swap if available:

               total        used        free      shared  buff/cache   available
Mem:            30Gi        12Gi        10Gi       203Mi       8.6Gi        18Gi
Swap:           31Gi       387Mi        30Gi

Only pay attention to the available column of the Mem: row. If this is near zero, your problem is memory pressure. You can pretty much ignore cache and swap usage, as they are not reliable indicators of problems on modern linux systems anymore.


If you still found nothing out of the ordinary, investigate virtual memory metrics:

vmstat -SM 1

The command will print realtime resource usage statistics once per second, which may look like

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 2  0 397180 10325980 1611892 7365660    0    0    21   125 1716    5  4  0 96  0  0  0
 2  0 397180 10326008 1611892 7365752    0    0     0  2328 2397 3320  9  0 91  0  0  0
 1  0 397180 10331272 1611892 7365752    0    0     0   260 2256 3140  9  0 91  0  0  0
 1  0 397180 10333060 1611892 7365752    0    0     0     0 2121 2968  9  0 91  0  0  0
 1  0 397180 10313316 1611892 7365752    0    0     0     0 2444 3376  9  0 91  0  0  0

Pay attention to the si/so columns indicating swap writes/reads, and bi/bo for block storage writes/reads. The -SM flag means all displayed values are in MiB, and since statistics are printed every second, rates in the i/o columns are therefore MiB/s.

Heavy block storage i/o indicates a disk either failing or running at or near capacity, which can be cause for performance issues. High values in the wa columns (cpu time spent waiting on i/o) would strengthen that suspicion.


To confirm whether a disk is at operational capacity, monitor /proc/diskstats:

watch -n 1 -x awk '{print $3 " - " $13}' /proc/diskstats

Don't forget to use the -x flag here to prevent dealing with awkward nested single quotes.

The output will print the total active time in ms per block device:

nvme0n1 - 930492
nvme0n1p1 - 116
nvme0n1p2 - 188
nvme0n1p3 - 953540
sda - 122080
dm-0 - 1052724
dm-1 - 1533000
dm-2 - 492
dm-3 - 124596

Since watch prints these stats once every second, a change of 1000 between two lines means the device was active 1000ms/1000ms, so 100%. Pay close attention to how much the values change every second to identify if a disk is maxed out of operations per second. An overloaded disk is typically caused by a degraded RAID array, rebalancing or failing hardware. Checking mdadm/LVM/ZFS/Btrfs and smartctl is advisable.


If disk bandwidth saturation is ruled out, you can check filesystem storage and inode usage:

df -sh
df -i

The first command prints the storage space usage for every filesystem/partition, the second the inode usage. If either one is at 100%, that is most likely your problem.


The last hardware category to check is networking. Start by triaging if you have many connections in non-active states:

ss -s

If the second row has any numeric value much larger than the others on the line, you are likely experiencing a networking problem or attack.


To diagnose it further, first find your network devices:

ip link show

Then track its metrics:

watch -n1 ip -s link show dev enp0s6

The output metrics are split into RX (received) and TX (transmitted/sent):

2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether f4:b5:20:70:4e:e0 brd ff:ff:ff:ff:ff:ff
    RX:   bytes  packets errors dropped  missed   mcast           
    21944041335 16618708      0       0       0   33643 
    TX:   bytes  packets errors dropped carrier collsns           
     5689076561 11163543      0       0       0       0 
    altname enxf4b520704ee0

If any errors or dropped column is increasing rapidly, you are likely experiencing network hardware failure, which could be your NIC, a cable, a switch, router or anything else.

Finding performance bottlenecks unrelated to hardware saturation

If none of the previous checks flagged any resource availability issue, your problem is likely not directly related to saturated hardware resources.

Since the underlying cause heavily depends on workload and machine, troubleshooting becomes a lot less streamlined.


Start by checking kernel/system logs:

sudo journalctl -k -b
sudo dmesg --level=emerg,alert,crit,err,warn

Check for failed systemd units:

systemctl list-unit-files

A service could be experiencing resource exhaustion without saturating the entire hardware if it is running with hardware limits or quotas. Depending on workloads, check resource usage of containers (docker, podman, incus etc), virtual machines and custom cgroups.


Some programs like databases also allow setting specific memory limits for caching or per-query storage, which may be at capacity and cause slowdowns as well.


If the slowdown can be traced down to a specific program or set of programs, start by viewing their most recent log entries, checking for unexpected restarts and selectively restarting them to see if the problem vanishes, persists or briefly vanishes but returns over time.

Resolving the problem

Depending on what issue(s) you identified during troubleshooting, the actual steps for resolution vary.

What remains important regardless of the core issue is to split the resolution into two steps.


Step one is to apply an immediate band-aid to keep the system afloat or get a temporary replacement up and running as quickly as possible. Growing a Btrfs filesystem using unused disk space or blocking an abusive IP address are examples of this kind of work.

Only once the server is back to a somewhat stable state should you spend time thinking about and implementing a long-term fix like adding more storage disks or network block storage volumes, rate-limiting web routes or detecting and banning brute-force attempts with fail2ban.

While it is tempting to start working on the final solution immediately, it will further delay the time until the system is operational again, and reduced availability is a lot better than complete unavailability for virtually any production system.

More articles

Git development strategies

Git history management for common project lifecycles

Running OCI containers on incus

And how it differs from docker/podman

Understanding the linux tr utility

Mapping characters one by one