Re: [Patch][RFC] Disabling per-tgid stats on task exit in taskstats

[Shailabh Nagar <nagar@watson.ibm.com>]

Andrew Morton wrote:

>On Thu, 29 Jun 2006 09:44:08 -0700
>Paul Jackson <pj@sgi.com> wrote:
>
>  
>
>>>You're probably correct on that model. However, it all depends on the actual
>>>workload. Are people who actually have large-CPU (>256) systems actually
>>>running fork()-heavy things like webservers on them, or are they running things
>>>like database servers and computations, which tend to have persistent
>>>processes?
>>>      
>>>
>>It may well be mostly as you say - the large-CPU systems not running
>>the fork() heavy jobs.
>>
>>Sooner or later, someone will want to run a fork()-heavy job on a
>>large-CPU system.  On a 1024 CPU system, it would apparently take
>>just 14 exits/sec/CPU to hit this bottleneck, if Jay's number of
>>14000 applied.
>>
>>Chris Sturdivant's reply is reasonable -- we'll hit it sooner or later,
>>and deal with it then.
>>
>>    
>>
>
>I agree, and I'm viewing this as blocking the taskstats merge.  Because if
>this _is_ a problem then it's a big one because fixing it will be
>intrusive, and might well involve userspace-visible changes.
>  
>
First off, just a reminder that this is inherently a netlink flow 
control issue...which was being exacerbated
earlier by taskstats decision to send per-tgid data (no longer the case).

But I'd like to know whats our target here ? How many messages per 
second do we want to be able to be sent
and received without risking any loss of data ? Netlink will lose 
messages at a high enough rate so the design point
will need to be known a bit.

For statistics type usage of the genetlink/netlink, I would have thought 
that userspace, provided it is reliably informed
about the loss of data through ENOBUFS, could take measures to just 
account for the missing data and carry on ?




>The only ways I can see of fixing the problem generally are to either
>
>a) throw more CPU(s) at stats collection: allow userspace to register for
>   "stats generated by CPU N", then run a stats collection daemon on each
>   CPU or
>  
>
>b) make the kernel recognise when it's getting overloaded and switch to
>   some degraded mode where it stops trying to send all the data to
>   userspace - just send a summary, or a "we goofed" message or something.
>  
>
One of the unused features of genetlink that's meant for high volume 
data output from the kernel is
the "dump" callback of a genetlink connection. Essentially kernel space 
keeps getting provided sk_buffs
to fill which the netlink layer then supplies to user space (over time I 
guess ?)

But whatever we do, there's going to be some limit so its useful to 
decide what the design point should be ?

Adding Jamal for his thoughts on netlink's flow control in the context 
of genetlink.

--Shailabh