Re: statistics infrastructure (in -mm tree) review

Martin Peschke
2006-06-22T11:43:42Z

Matthew Frost wrote:
> Martin Peschke wrote:
>> On Tue, 2006-06-20 at 09:50 -0700, Randy.Dunlap wrote:
>>> On Tue, 20 Jun 2006 17:40:01 +0200 Martin Peschke wrote:
>>>> Greg KH wrote:
>>>>>> 7) With regard to the delivery of statistic data to user land,
>>>>>> a library maintaining statistic counters, histograms or whatever
>>>>>> on behalf of exploiters doesn't need any help from the exploiter.
>>>>>> We can avoid the usual callbacks and code bloat in exploiters
>>>>>> this way.
>>>>> I don't really understand what you are stating here.
>>>> Sorry.
>>>> 1,$s/exploiter/client/g
>>>>
>>>> Any device driver or whatever gathering statistics data currently
>>>> has code dealing with showing the data. Usually, they have some
>>>> callbacks for procfs, sysfs or whatever.
>>>>
>>>> My point is that, if a library keeps track of statistics on behalf
>>>> of its clients, no client needs to be called back in order to
>>>> merge, format, copy, etc. data being shown to users. The library
>>>> can handle as a background operation without disturbing clients.
>>> That could be a good thing. OTOH, it means that the library
>>> has to be either all-ways flexible or willing to change to
>>> accommodate clients since you can't predict the universe of all
>>> clients' requirements.
>>
>> Right. I have made provisions for that to some degree.
>>
>>
>> First, I could imagine that the statistics data of a client requires
>> a new way its data should be aggregated and, therewith, requires
>> a new form of statistic result being shown to users.
>>
>> I have scanned through the kernel sources for ways of aggregating
>> and showing statistics data. The usual constructs appear to be:
>>
>> - counter
>> - histogram (for intervals), linear scale
>> - histogram (for intervals), logarithmic scale
>> - "histogram" for discrete and sparse values
>> - "utilisation indicator" or "fill level indicator" (num-min-avg-max)
>>
>> These are implemented in my patches. I would expect these to cover most
>> requirements of possible new clients.
>
> So you're saying, as regards "putting presentation format in ... the
> kernel", that we already have presentation formats specified pell-mell
> in the kernel. That should then be a non-issue, because you aren't
> introducing anything new, just centralizing an existing kernel behavior.
> Do I have you right?

Yes, there seem to be as many formats as statistics. See examples below.
My patches can help to improve usuability by providing some common basic
formats.

IMO, it would not make sense to "enhance" the statistics library in an
attempt to emulate all the preexisting statistic output formats.

[root@t2930041 ~]# cat /proc/dasd/statistics
31 dasd I/O requests
with 392 sectors(512B each)
__<4 ___8 __16 __32 __64 _128 _256 _512 __1k
__2k __4k __8k _16k _32k _64k 128k
_256 _512 __1M __2M __4M __8M _16M _32M _64M
128M 256M 512M __1G __2G __4G _>4G
Histogram of sizes (512B secs)
0 0 21 7 3 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
Histogram of I/O times (microseconds)
0 0 0 0 0 0 0 3 6
1 5 6 10 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
<snip>

[root@t2930041 ~]# cat /proc/diskstats
<snip>
94 0 dasda 67389 1478 1142520 281080 78260 461181 4326752 10280570
0 6392030 10565980
94 1 dasda1 68849 1142272 540838 4326704
94 4 dasdb 27 29 448 0 0 0 0 0 0 0 0
94 5 dasdb1 27 216 0 0
94 8 dasdc 28 29 456 40 0 0 0 0 0 30 40
94 9 dasdc1 28 224 0 0
9 0 md0 0 0 0 0 0 0 0 0 0 0 0
8 0 sda 35423 12268 4340826 284540 8605 275966 2276792 980810
0 219260 1265370
8 16 sdb 36741 12626 4588754 293140 10090 277678 2302400 440010
0 221990 733140
8 32 sdc 36621 11748 4548722 298170 10394 272680 2264736 303580
0 223110 601730

[root@t2930041 ~]# cat /proc/net/stat/arp_cache
entries allocs destroys hash_grows lookups hits res_failed
rcv_probes_mcast rcv_probes_ucast periodic_gc_runs forced_gc_runs
00000002 0000002c 000000cd 00000000 00000082 00000056
00000000 00000000 00000000 00017306 00000000
00000002 00000031 00000000 00000000 0000007d 0000004c
00000000 00000000 00000000 00000000 00000000
00000002 0000002f 00000000 00000001 00000074 00000045
00000000 00000000 00000000 00000000 00000000
00000002 00000043 00000000 00000000 00000084 00000041
00000000 00000000 00000000 00000000 00000000

>> If another construct would be needed anyway, it can be added to the
>> statistics library by implemententing about half a dozen routines
>> described by struct statistic_discipline. I might be wrong, but I don't
>> think we would see an inflationary growth there.
>>
>>
> -- elision --
>>
>> OTOH, I don't see a real need for allowing that. Data can be reformatted
>> and rearranged in any possible way in user space.
>
> Because you're just providing a range of basic output formats,
> standardized. So anybody can ask for statistics from the kernel in a
> preferred output to then massage as needed in userland. ACK? Am I
> oversimplifying?

Sounds reasonable.

Thanks, Martin