# Linux CPU Performance Analysis with BPF Tools: A Practical Runbook

---

Recently, I got a chance to read *BPF Performance Tools* to brush up on my Linux skills. During my exploration, I came across some common CPU tools used for troubleshooting performance issues. I decided to create a runbook a practical guide that I (or anyone like me) can refer to whenever a CPU issue arises. This guide covers which tools to use, how to analyze their output, and how to draw actionable conclusions.

---

## Basics of CPU

A CPU consists of multiple **cores**, and each core can handle multiple tasks (instruction sets).

In cloud environments like AWS, when you say **1 CPU**, you are actually referring to **1 vCPU (virtual CPU)**.

✅ **Key points:**

* 1 vCPU = 1 hardware thread
    
* On most Intel and AMD processors, **1 physical core = 2 threads** (thanks to hyper-threading)
    

> Note: Here we are referring to **cores**, not the whole CPU. For example, a consumer laptop with an Intel i7 may have 7 cores. When we say 1 core, we mean 1 of those cores, not the entire CPU.

So the mapping between physical cores and vCPUs looks like this:

| Physical Core | vCPUs |
| --- | --- |
| 1 | 2 |
| 2 | 4 |
| 8 | 16 |

**Example:** AWS m5.xlarge → 2 vCPUs → 1 physical core with 2 threads

In Kubernetes, CPU limits are defined in **millicores**:

* 1 CPU = 1000m
    
* 500m = 50% of a CPU
    

---

## User Space vs Kernel Space

Processes in a CPU operate in two spaces:

1. **User Space** – where your applications and services run
    
2. **Kernel Space** – system-level operations managed by the OS
    

**Why this matters:** When troubleshooting high CPU usage, it’s critical to know whether the load comes from user space or kernel space. Kernel-level processes need to be handled carefully because stopping them can affect the entire system.

Additionally, processes can be in two states:

* **Runnable (ONPROC)** – ready and waiting for CPU execution
    
* **Sleeping (idle)** – waiting for resources or I/O
    

---

## Runbook: Debugging High CPU Usage

**Scenario:**

You’re an SRE for an e-commerce platform. You get an alert:

🚨 *High CPU utilization on web-server-03 — 95% for the last 10 minutes*

**Goal:** Find the cause of high CPU usage.

---

### Step 1: Check Load Average with “`uptime`"

```bash
uptime
```

This shows the average number of processes waiting to be executed over 1, 5, and 15 minutes.

**Example:**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1761035177611/071287a5-a46a-4178-845d-f95356637c38.png align="center")

**How to interpret:**

* The three numbers represent the **average number of processes waiting to run** over different time intervals. **These intervals are standard for uptime command**:
    
    * **6.45** → 1-minute average
        
    * **5.89** → 5-minute average
        
    * **4.12** → 15-minute average
        
    
    **What this tells you:**
    
    * If you have **2 CPUs**, any value above 2 means the system has **more processes waiting than CPUs available**.
        
    * In this example, all three averages are above 2 → your CPUs are **overloaded**, and processes are queuing up for execution.
        
    * A **high load average compared to CPU count** indicates that your system is experiencing CPU pressure, and further investigation is needed to identify the culprits.
        
    
    > Tip: The **1-minute average** is most reactive to recent spikes, while the **15-minute average** shows longer-term trends..
    

---

### Step 2: Identify CPU-Hungry Processes with “`top`"

```bash
top
```

**Example:**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1761035308096/83bc786e-6dff-4807-8082-a6bb5dffe37f.png align="center")

**What to look for:**

**Highlighted Field in Orange Color Defines:**

* **%us** → CPU usage in user space
    
* **%sy** → CPU usage in system/kernel space
    

This helps determine whether the CPU load is from user applications or kernel processes.

Sort processes by CPU usage for easier analysis:

```bash
top -o %CPU
```

**Dig deeper:**

```bash
pidstat -p <PID>
```

* You can use pidstat to get the detailed information about the specific Process id which you can get it from top command.
    

```bash
sudo strace -p <PID>
```

* This command will give you detailed report on the PID. If you really need to go deeper use strace command, mostly pidstat is sufficient enough for troubleshooting.
    

**Sample Output of Strace:**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1761035421661/3e019301-d350-4f76-8cd8-68fc927ae54c.png align="center")

**How to read:**

* Each line = 1 syscall.
    
* The last number (e.g. `= 17`) is the **return value**.
    
* If you see one syscall repeated rapidly (like `read()` or `epoll_wait()`), that’s the loop burning CPU.
    

If you suspect it’s looping too fast:

```bash
sudo strace -c -p <PID>
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1761035496112/640c3951-dd52-44ed-a7ce-6effc16e4560.png align="center")

**Possible remediation:**

1. Check and restart the service
    
2. Kill non-critical processes and restart later
    

---

### Step 3: Check CPU Distribution Across Cores using “`mpstat`“

Sometimes only a few CPUs are maxed out while others are idle. This can happen if workloads are:

* Single-threaded
    
* CPU-pinned (affinity set)
    
* Blocked by locks
    

Use `mpstat` to see **per-core utilization**:

```bash
mpstat -P ALL 2 3
```

* **2** → interval of 2 seconds
    
* **3** → run 3 times
    

This shows CPU usage per core and helps identify imbalances.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1761036126601/762cc6ba-f6ac-4bb0-993b-eabafe3aa5a9.png align="center")

---

### Step 4: Use BPF Tools for Deep Analysis

Sometimes `top` is not enough. Use BPF (Berkeley Packet Filter) tools when:

* High CPU is confirmed but the exact cause is unclear
    
* Kernel or syscall usage is high
    
* You suspect locks, spin loops, or scheduler delays
    

#### 1\. `profile`

Shows which functions consume CPU:

```bash
sudo /usr/share/bcc/tools/profile 5
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1761036183863/85bccf9d-207f-4901-9461-533d5cfc5065.png align="center")

**Interpretation:**

* Each block shows a call stack and the number of samples (e.g. `45` means CPU was in that stack 45 times).
    
* The higher the count, the more CPU time that function consumes.
    
* Helps pinpoint the exact code path burning CPU.
    

User-space only:

```bash
sudo /usr/share/bcc/tools/profile -U
```

#### 2\. `offcputime`

Shows where threads are waiting (blocked, sleeping, or I/O wait):

```bash
sudo /usr/share/bcc/tools/offcputime 5
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1761036250143/5becfca6-532a-49f6-9517-4d4bf18fd7eb.png align="center")

**Interpretation:**

* This means the thread is spending time *waiting* in a futex (a synchronisation lock).
    
* So CPU isn’t overloaded by raw computation — it’s waiting on something (like a lock or I/O).
    
* Combine this with `profile`:
    
    * `profile` → what’s *using* CPU
        
    * `offcputime` → what’s *waiting* for CPU
        

#### 3\. `runqlen`

Shows the run queue length per CPU:

```bash
sudo /usr/share/bcc/tools/runqlen
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1761036304140/affd2e55-63c8-4f13-b96c-703e58996401.png align="center")

**Interpretation:**

* Average 2.5 → 2.5 tasks waiting to run on CPU0 most of the time.
    
* High numbers mean **CPU contention** — more runnable tasks than CPUs.
    
* If this matches a high load average, you’ve confirmed CPU saturation.
    

---

### 🧠 Summary: How These Tools Fit Together

| **Tool** | **Purpose** | **When to Use** |
| --- | --- | --- |
| uptime | Shows load average (processes waiting) | Initial check to see CPU load vs available CPUs |
| top | Displays CPU usage per process and space | Identify high CPU processes and user/kernel usage |
| mpstat | Per-core CPU utilization | Detect load imbalance across cores |
| strace | Syscalls by a process | Process-level view |
| profile | Functions consuming CPU | High CPU in user/kernel |
| offcputime | Threads waiting (blocked, sleeping, I/O) | Performance stalls, I/O wait |
| runqlen | Threads waiting per CPU | Confirm CPU contention |

**Workflow:**

* `uptime` → check load average vs available CPUs
    
* `top` → confirm high CPU usage
    
* `mpstat` → verify per-core load distribution
    
* `profile` → find which functions burn CPU
    
* `offcputime` → find functions waiting off CPU
    
* `runqlen` → verify CPU contention
    
* `strace` → check syscalls causing delays
    

---

## 🧩 Mini Lab: Investigating High CPU Usage

Follow these steps to practice analyzing CPU issues on a test server:

1. **Simulate CPU load:**
    

```bash
# Stress CPU for 60 seconds
sudo apt install stress -y
stress --cpu 2 --timeout 60
```

2. **Check load averages:**
    

```bash
uptime
```

3. **Identify CPU-hungry processes:**
    

```bash
top -o %CPU
```

4. **Check per-core utilization:**
    

```bash
mpstat -P ALL 2 3
```

5. **Profile functions consuming CPU:**
    

```bash
sudo /usr/share/bcc/tools/profile 5
```

6. **Check where threads are blocked:**
    

```bash
sudo /usr/share/bcc/tools/offcputime 5
```

7. **Verify run queue length:**
    

```bash
sudo /usr/share/bcc/tools/runqlen
```

8. **Investigate syscalls for a process:**
    

```bash
pidstat -p <PID>
strace -p <PID>
sudo strace -c -p <PID>
```

By completing this mini-lab, you’ll have hands-on experience with CPU troubleshooting using both traditional and BPF tools.

---

*Happy Troubleshooting!*