Tutorial/git's guide to c-state counters on Intel?

[Rick Jones <rick.jones2@hpe.com>]

I am presently trying to tease-out why a "slight" change in kernel 
(4.4.7 to 4.4.11) might result in a large change in some VM to VM 
netperf TCP_RR performance.  I've seen that if I set the system firmware 
to be in a static high performance mode rather than a dynamic power 
management (by the firmware) mode the gap narrows considerably.

I've come across:

   cstate_core/c3-residency/                          [Kernel PMU event]
   cstate_core/c6-residency/                          [Kernel PMU event]
   cstate_core/c7-residency/                          [Kernel PMU event]
   cstate_pkg/c2-residency/                           [Kernel PMU event]
   cstate_pkg/c3-residency/                           [Kernel PMU event]
   cstate_pkg/c6-residency/                           [Kernel PMU event]
   cstate_pkg/c7-residency/                           [Kernel PMU event]

and when I count those via perf comparing the two, back in the dynamic 
power managment mode, I can see more package c* residency on the slower 
setup (newer kernel).  That fits with the hypothesis that the slower 
request/response performance is coming from going into and out of sleep 
states more - something which has been known to affect the likes of a 
netperf TCP_RR test for years.   I don't see nearly as large a 
difference between the core c* counts though, which leaves me wondering 
why/how that might be.

The processors in question here are E5-2640:

model name	: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz

happy benchmarking,

rick jones