Is it normal for the user and sys times in /proc/stat to move backwards with NO_HZ?

[Thomas Lindroth <thomas.lindroth@gmail.com>]

I have noticed that some of the values in /proc/stat, such as user, nice, and sys time,
can decrease on subsequent reads. The file Documentation/filesystems/proc.rst states:
"The value of iowait field in /proc/stat will decrease in certain conditions".
Since the documentation specifically mentions iowait I assume that the other values
should not decrease.

A simple way to test this is by running:
while true; do grep -F "cpu1 " /proc/stat;done| awk 'NR>1{diff=$2-p;if(diff<0)print diff}{p=$2}'

I believe this issue is related to the use of NO_HZ. I use nohz_full=1-23 on an
Intel i7-13700K CPU with 24 logical cores. I have never seen decreasing values on cpu0,
which is the housekeeping CPU with ticks always enabled. I have seen decreasing user,
nice, and sys times on all other cores on kernel 6.12.38 and an older 6.1.140 kernel.

I boot the kernel with the following arguments:
nohz_full=1-23 rcu_nocbs=1-23 intel_iommu=on kvm.ignore_msrs=1 kvm-intel.nested=0
kvm.report_ignored_msrs=0 vfio-pci.disable_vga=1 split_lock_detect=off

If this is expected behaviour it would be helpful if the documentation could be updated
to better clarify this.

/Thomas Lindroth