CPU utilization is defined as the ratio of the total current utilization of a CPU over the total possible utilization of a CPU.

Therefore, we begin with the following simple formula:

CPU % = CPU_work / CPU_total

All operating systems, that we know of, define the CPU_ variables through a unit of time. It is the time the processor was doing any work (aka some instructions) or being in an idle state (aka not being assigned to process). The concept of an idle state here is slightly different from the idle state in a processor (aka being halted). An OS will typically also issue a HLT instruction when there is no process the scheduler needs to assign the processor to, but it should not be taken as guaranteed. Another wording that is often used is: “The CPU is put on the idle thread.”. This originates from the fact that usually at least one thread must be running to keep the scheduler working. Also if you feed a process NOP instructions this would not count as idle for the operating system.

Now we are looking further where Linux gets this metric from, and by looking in the source code for top we find that the values are captured from the procfs. More specifically system wide they come from /proc/stat.

The file entry looks like this:

cpu  887579 3095 335641 69621939 134839 0 11876 0 0 0
cpu0 887579 3095 335641 69621939 134839 0 11876 0 0 0
....

If we now do a man proc on Linux we get the man page that tells us what the numbers that follow the CPU id mean:

user   (1) Time spent in user mode.

 nice   (2) Time spent in user mode with low priority (nice).

 system (3) Time spent in system mode.

 idle   (4) Time spent in the idle task.  This value should be  USER_HZ  times  the
        second entry in the /proc/uptime pseudo-file.

 iowait (since Linux 2.5.41)
        (5)  Time waiting for I/O to complete.  This value is not reliable, for the
        following reasons:

        1. The CPU will not wait for I/O to complete; iowait is  the  time  that  a
           task  is  waiting  for I/O to complete.  When a CPU goes into idle state
           for outstanding task I/O, another task will be scheduled on this CPU.

        2. On a multi-core CPU, the task waiting for I/O to complete is not running
           on any CPU, so the iowait of each CPU is difficult to calculate.

        3. The value in this field may decrease in certain conditions.

 irq (since Linux 2.6.0)
        (6) Time servicing interrupts.

 softirq (since Linux 2.6.0)
        (7) Time servicing softirqs.

 steal (since Linux 2.6.11)
        (8)  Stolen  time,  which is the time spent in other operating systems when
        running in a virtualized environment

 guest (since Linux 2.6.24)
        (9) Time spent running a virtual CPU for guest operating systems under  the
        control of the Linux kernel.

 guest_nice (since Linux 2.6.33)
        (10) Time spent running a niced guest (virtual CPU for guest operating sys‐
        tems under the control of the Linux kernel).

We see now that our initial formula of CPU % = CPU_work / CPU_total was a bit too simple and modern operating systems collect more granular metrics.

For Linux the actual implementation how for instance htop, which is similar to top, reports the CPU utilization is:

CPU_total = idle + iowait_time + user_time + nice_time + system_time + irq_time + softirq_time + steal_time

and

CPU_work = user_time + nice_time + system_time + irq_time + softirq_time + steal_time

Source for htop calculation

All these values in the /proc/stat file are of the unit jiffies. This means it is a proxy for the actual time spent in that state. To get exact values we would need to multiply the jiffies with the USER_HZ variable of the operating system. If we just want to get the ratio though this step can be omitted.

We conclude that we now know where the CPU utilization comes from and that the resulting unit is a ratio of time.

So taking a normal Ubuntu 22.04 system on a Fujitsu Esprimo 956 4-Core machine we conduct the following test:

• Run a simple program that calculates CPU utilization and outputs a line every 10 ms to a file and sleeps in between
• Check how many lines were written
• check top while doing so and write down the CPU utilization of the system as well as the process
• check the frequency of all our cores on the system
• Repeat the test couple of times to check if values are stable

Now we introduce some load with running a stress process on two cores keeping these cores fully busy.

We see that the system wide CPU utilization climbs up to ~50% but the ratio attributed for the process now is at 4.3%.

This effect is nicely visualized here on a process level but the same issue happens with the system wide CPU utilization.

Most suprisingly: The throughput of the program has actually gone up! From 476 lines before to now 765 lines.

This behaviour comes from the fact that the CPU cycle count goes up and the process can actually do more work in the same time.

The throughput is not linear and 100% is not double of 50%.

But even if we are in a non-virtualized, non-hyper-threaded and fixed-frequency environment CPU utilization can still bite us if the CPU experiences memory congestion. A stalling time due to memory I/O is then falsely attributed to a non-idle state. Details are in Brendan Gregg’s Blog - CPU Utilization is Wrong

What however would happen if we fix the CPU frequency, so that it is always running on the maximum frequency and do not utilize features such as Turbo Boost, Hyper Threading and not use virtualization?

The latter is probably a lost cause, as in basically every cloud machine and even shared hosting systems are virtualized. We have here to rely on the fact that resources are not heavily oversubscribed and that hypervisors at least report some correct values of the CPU registers needed to derive metrics like utilzation.

Because if we would have system that has a fixed CPU frequency the load-performance curve actually becomes linear:

In this example we have run sysbench --cpu-max-prime=25000 --threads=1 --time=10 --test=cpu --events=0 --rate=0 and put a CPU % limiting on the process and increased that in 10% increments. The blue curve has been done with the schedutil CPU frequency govenor which dynamically scales the CPU frequency. And the red curve has been done with the performance scaling govenor which scales the CPU frequency to a maximum as soon as even a minimum amount of load happens on a core.

Let’s look at how different cloud vendors have this setting for CPU frequency and Turbo Boost set, so we can later use that to make some assumptions on estimations models for the cloud:

Here are the details, feel free to scroll to the bottom for the summary :)

AWS EC2 m5.metal (bare metal)

Frequency

We were running stress-ng -c 1 on one core to trigger different frequencies.

Every 2.0s: cat /proc/cpuinfo | grep MHz | uniq

cpu MHz         : 2609.666
cpu MHz         : 2500.000
cpu MHz         : 3500.001
cpu MHz         : 2500.000

CPU frequency is dynamic

P-States

....
core 95:
     TurboBoost (MSR):              enabled
     scaling_governor:              performance
     scaling_driver:              intel_pstate
     energy_performance_preference:      performance
CPU-wide settings
     P_State status (off,active,passive):      active
     P_State no_turbo (0=allowed,1=off):      0
     P_State max-turbo (%):          100

Available drivers for CPU DVFS:
performance powersave

CPUInfo

...
processor    : 95
vendor_id    : GenuineIntel
cpu family    : 6
model        : 85
model name    : Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
stepping    : 7
microcode    : 0x5003303
cpu MHz        : 2500.000
cache size    : 36608 KB
physical id    : 1
siblings    : 48
core id        : 29
cpu cores    : 24
...

AWS EC2 m5.large (virtualized)

Frequency

Every 2.0s: cat /proc/cpuinfo | grep MHz | uniq

cpu MHz         : 2499.998
cpu MHz         : 3099.774

CPU frequency is dynamic

P-states

Reading this info is blocked by the hypervisor :(

CPUInfo

...
processor    : 1
vendor_id    : GenuineIntel
cpu family    : 6
model        : 85
model name    : Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
stepping    : 7
microcode    : 0x5003501
cpu MHz        : 3097.292
cache size    : 36608 KB
physical id    : 0
siblings    : 2
core id        : 0
cpu cores    : 1
...

Github Actions (Linux)

Frequency

Every 2.0s: cat /proc/cpuinfo | grep MHz | uniq

cpu MHz         : 2793.437
cpu MHz         : 2793.437

CPU frequency does not change.

P-states

core 0:
         TurboBoost (MSR):                       enabled
core 1:
         TurboBoost (MSR):                       enabled

Other info is blocked by the hypervisor :(

CPUInfo

...
processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 106
model name      : Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
stepping        : 6
microcode       : 0xffffffff
cpu MHz         : 2793.437
cache size      : 49152 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
...

Gitlab Pipelines (Linux)

Source

Frequency

cat /proc/cpuinfo | grep MHz
cpu MHz        : 2249.998
cpu MHz        : 2249.998

CPU frequency does not change.

P-states

All info is blocked by the hypervisor :(

CPUInfo

...
processor    : 1
vendor_id    : AuthenticAMD
cpu family    : 23
model        : 49
model name    : AMD EPYC 7B12
stepping    : 0
microcode    : 0xffffffff
cpu MHz        : 2249.998
cache size    : 512 KB
physical id    : 0
siblings    : 2
core id        : 0
cpu cores    : 1
...

Google Cloud (N2)

Frequency

cat /proc/cpuinfo | grep MHz
Every 2.0s: cat /proc/cpuinfo | grep MHz

cpu MHz         : 2800.202
cpu MHz         : 2800.202
cpu MHz         : 2800.202
cpu MHz         : 2800.202
cpu MHz         : 2800.202
cpu MHz         : 2800.202
cpu MHz         : 2800.202
cpu MHz         : 2800.202

CPU frequency does not change.

P-states

All info is blocked by the hypervisor :(

CPUInfo

...
processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) CPU @ 2.80GHz
stepping        : 7
microcode       : 0xffffffff
cpu MHz         : 2800.202
cache size      : 33792 KB
physical id     : 0
siblings        : 8
core id         : 3
cpu cores       : 4
...

Hetzner

Frequency

cat /proc/cpuinfo | grep MHz
Every 2.0s: cat /proc/cpuinfo | grep MHz

cpu MHz        : 2294.606

CPU frequency does not change.

P-states

core 0:
     TurboBoost (MSR):              enabled

All other info is blocked by the hypervisor :(

CPUInfo

...
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 85
model name    : Intel Xeon Processor (Skylake, IBRS)
stepping    : 4
microcode    : 0x1
cpu MHz        : 2294.606
cache size    : 16384 KB
physical id    : 0
siblings    : 1
core id        : 0
cpu cores    : 1
...