Re: [Patch 0/5] I/O statistics through request queues

["Frank Ch. Eigler" <fche@redhat.com>]

Hi -

On Thu, Oct 26, 2006 at 03:37:54PM +0200, Martin Peschke wrote:
> [...]
> lookup_table[key] = value	, or
> lookup_table[key]++
> 
> How does this scale?

It depends.  If one is interested in only aggregates as an end result,
then intermediate totals can be tracked individiaully per-cpu with no
locking contention, so this scales well.

> It must be someting else than an array, because key boundaries
> aren't known when the lookup table is created, right?
> And actual keys might be few and far between.

In systemtap, we use a hash table.

> What if the heap of intermediate results grows into thousands or
> more?  [...]

It depends whether you mean "rows" or "columns".

By "rows", if you need to track thousands of queues, you will need
memory to store some data for each of them.  In systemtap's case, the
maximum number of elements in a hash table is configurable, and is all
allocated at startup time.  (The default is a couple of thousand.)
This is of course still larger than enlarging the base structures the
way your code does.  But it's only larger by a constant amount, and
makes it unnecessary to patch the code.

By "columns", if you need to track statistical aggregates of thousands
of data points for an individual queue, then one can use a handful of
fixed-size counters, as you already have for histograms.


Anyway, my point was not that you should use systemtap proper, or that
you need to use the same techniques for managing data on the side.
It's that by using instrumentation markers, more things are possible.


- FChE