Re: [PATCH v1] cgroup/rstat: add cgroup_rstat_cpu_lock helpers and tracepoints

[Jesper Dangaard Brouer <hawk@kernel.org>]

On 02/05/2024 21.44, Shakeel Butt wrote:
> On Wed, May 01, 2024 at 07:22:26PM +0200, Jesper Dangaard Brouer wrote:
>>
> [...]
>>
>> More data, the histogram of time spend under the lock have some strange
>> variation issues with a group in 4ms to 65ms area. Investigating what
>> can be causeing this... which next step depend in these tracepoints.
>>
>> @lock_cnt: 759146
>>
>> @locked_ns:
>> [1K, 2K)             499 |      |
>> [2K, 4K)          206928
>> |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
>> [4K, 8K)          147904 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@      |
>> [8K, 16K)          64453 |@@@@@@@@@@@@@@@@      |
>> [16K, 32K)        135467 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@      |
>> [32K, 64K)         75943 |@@@@@@@@@@@@@@@@@@@      |
>> [64K, 128K)        38359 |@@@@@@@@@      |
>> [128K, 256K)       46597 |@@@@@@@@@@@      |
>> [256K, 512K)       32466 |@@@@@@@@      |
>> [512K, 1M)          3945 |      |
>> [1M, 2M)             642 |      |
>> [2M, 4M)             750 |      |
>> [4M, 8M)            1932 |      |
>> [8M, 16M)           2114 |      |
>> [16M, 32M)          1039 |      |
>> [32M, 64M)           108 |      |
>>
> 
> Am I understanding correctly that 1K is 1 microsecond and 1M is 1
> millisecond? 

Correct.

> Is it possible to further divide this table into update
> side and flush side?
>

This is *only* flush side.

You question indicate, that we are talking past each-other ;-)

Measurements above is with (recently) accepted tracepoints (e.g. not the
proposed tracepoints in this patch).  I'm arguing with existing
tracepoint that I'm seeing this data, and arguing I need per-CPU
tracepoints to dig deeper into this (as proposed in this patch).

The "update side" can only be measured once we apply this patch.

This morning I got 6 prod machines booted with new kernels, that contain 
this proposed per-CPU lock tracepoint patch.  And 3 of these machines 
have the Mutex lock change also.  No data to share yet...

--Jesper