Re: [Patch][RFC] Disabling per-tgid stats on task exit in taskstats

[Jay Lan <jlan@engr.sgi.com>]

Shailabh Nagar wrote:
>Hi Andrew,
>
>Two developments on the tgid overhead issue:
>
>1. The latest results show that overhead is significant
>only when the exit rate exceeds roughly 1000 threads/second.
>  

I worked with Shailabh this week to run various testing and
debugging as he requested. I was pulled off to some urgent
task yesterday and surprising saw this coming this morning...

Let's slow it down please. My last testing (after your fix in
#2 below) still showed 109% overhead at system time. And, the
per-thread group processing also increase the rate of ENOBUFS
at the receiver.

I need to check with other guys to find out if 1000 threads/sec
indeed unrealistic at our customers' environments. A good
design should allow a mechanism to turn off the penalty due to
a feature that is not common to everybody. I do not understand
your objection.

Regards,
 - jay

>2. A new patch that modifies the locking used within taskstats,
>brings down the overhead of the extreme case quite a bit.
>I'll submit the patch along shortly in a separate mail.
>  
>To get back to the effect of exit rate, I modified the fork+exit
>benchmark to vary the rate at which exits happened and
>ran tests on a 4-way 1.4 GHz x86_64 box. The kernel was 2.6.17,
>uses the delay accounting/taskstat patches in 2.6.17-mm1 + the new
>locking patch mentioned in 2. above.
>
>The results show that differential between tgid on and off
>starts becoming significant once the exit rate crosses roughly 1000
>threads/second. Below that exit rate, the difference is negligible.
>Above it, the difference starts climbing rapidly.
>
>So I guess the question is whether this rate of exit is representative
>enough of real life to warrant making any more changes to the existing
>patchset, beyond the locking changes in 2. above.
>
>>From my limited experience, I think this is too high an exit rate
>to be worrying about overhead.
>
>
>        %ovhd of tgid on over off
>        (higher is worse)
>
>Exit     User     Sys     Elapsed
>Rate     Time     Time    Time
>
>2283      25.76  649.41   -0.14
>1193     -10.53   88.81   -0.12
>963      -11.90    3.28   -0.10
>806       -8.54   -0.84    0.16
>694       -4.41    2.38    0.03
>
>Exit Rate: units are threads exiting per second.
>Calculated by (#threads_forked+exited)/(elapsed_time)/2
>Since app pretty much does only thread create and exit for 10000
>threads (1000 threads, 10 iterations), this is a good measure
>for exit rate.
>
>%diff in user, sys, elapsed times calculated using
>(tgid_on - tgid_off)/tgid_off * 100
>where tgid_on/off times are reported by /usr/bin/time as before.
>
>Each data point for tgid_on and tgid_off was an average
>of 10 runs of the fork+exit benchmark.
>The rate of exits was controlled by delaying the individual
>threads through a usleep before being allowed to exit.
>
>Machine was 4-way 1.6GHz x86_64 Opteron.
>
>"exit_recv -w", the user program consuming the stats, was running
>on the side, reading the stats but not writing to a file or
>printing to screen.
>