Re: Issue perf attaching to processes creating many short-live threads

[David Ahern <dsahern@gmail.com>]

On 10/27/15 6:33 AM, Arnaldo Carvalho de Melo wrote:
>
>> Correlating data to user readable information is a key part of perf.
>
> Indeed, as best as it can.
>
>> One option that might be able to solve this problem is to have perf
>> kernel side walk the task list and generate the task events into the
>> ring buffer (task diag code could be leveraged). This would be a lot
>
> It would have to do this over multiple iterations, locklessly wrt the
> task list, in a non-intrusive way, which, in this case, could take
> forever, no? :-)

taskdiag struggles to keep up because netlink messages have a limited 
size, the skb's have to be pushed to userspace and ack'ed and then the 
walk proceeds to the next task.

Filenames for the maps are the biggest killer on throughput wrt kernel 
side processing.

With a multi-MB ring buffer you have a much larger buffer to fill. In 
addition perf userspace can be kicked at a low watermark so it is 
draining that buffer as fast as it can:

    kernel   --->   ring buffer  --->  perf  -->  what?

The limiter here is perf userspace draining the buffer such that the 
kernel side does not have to take much if any break.

If the "What" is a file (e.g., perf record) then file I/O becomes a 
limiter. If the "What" is processing the data (e.g., perf top) we should 
be able to come up with some design that at least pulls the data into 
memory so the buffer never fills.

Sure there would need to be some progress limiters put to keep the 
kernel side from killing a cpu but I think this kind of design has the 
best chance of getting the most information for this class of problem.

And then for all of the much smaller more typical perf use cases this 
kind of data collection is much less expensive than walking proc. 
taskdiag shows that and this design is faster and more efficient than 
taskdiag.

David