Re: Stacks leading into skb:kfree_skb

[David Ahern <dsahern@kernel.org>]

On 7/18/23 4:33 PM, Ivan Babrou wrote:
> On Fri, Jul 14, 2023 at 5:54 PM David Ahern <dsahern@kernel.org> wrote:
>>
>> On 7/14/23 4:13 PM, Ivan Babrou wrote:
>>> As requested by Jakub Kicinski and David Ahern here:
>>>
>>> * https://lore.kernel.org/netdev/20230713201427.2c50fc7b@kernel.org/
>>>
>>> I made some aggregations for the stacks we see leading into
>>> skb:kfree_skb endpoint. There's a lot of data that is not easily
>>> digestible, so I lightly massaged the data and added flamegraphs in
>>> addition to raw stack counts. Here's the gist link:
>>>
>>> * https://gist.github.com/bobrik/0e57671c732d9b13ac49fed85a2b2290
>>
>> I see a lot of packet_rcv as the tip before kfree_skb. How many packet
>> sockets do you have running on that box? Can you accumulate the total
>> packet_rcv -> kfree_skb_reasons into 1 count -- regardless of remaining
>> stacktrace?
> 
> Yan will respond regarding the packet sockets later in the day, he
> knows this stuff better than I do.
> 
> In the meantime, here are the aggregations you requested:
> 
> * Normal: https://gist.githubusercontent.com/bobrik/0e57671c732d9b13ac49fed85a2b2290/raw/ae8aa1bc3b22fad6cf541afeb51aa8049d122d02/flamegraph.normal.packet_rcv.aggregated.svg
> * Spike: https://gist.githubusercontent.com/bobrik/0e57671c732d9b13ac49fed85a2b2290/raw/ae8aa1bc3b22fad6cf541afeb51aa8049d122d02/flamegraph.spike.packet_rcv.aggregated.svg

For the spike, 97% are drops in packet_rcv. Each raw packet socket
causes every packet to be cloned which makes an N-factor on the number
of skbs to be freed. If this is tcpdump or lldp with a filter that would
be what Jakub mentioned in his response.

> 
> I just realized that Github links make flamegraphs non-interactive. If
> you download them and open a local copy, they should work better:

Firefox shows the graphs just fine.

> 
> * Expand to your screen width
> * Working search with highlights
> * Tooltips with counts and percentages
> * Working zoom