RFC: using NIC DMA ring as tracing buffer?

RFC: using NIC DMA ring as tracing buffer?

[Alexander Aring]

Hi all,

I have an idea about using the DMA rings of a networking card as a
tracing buffer. Hereby I want to thank the Linux community for the
discussions about how “basically” tracing buffers in Linux currently
works. In my minds I currently have two possible use-cases why doing
this:

1. A generic tracing buffer filtering and matching offloading engine
2. Directly transmit the tracing buffer to a remote tracer application

For example, my tracing use case DLM viewer [0], can use 1 to easily
“tag” each event according to specific trace event field key/values
pairs to completely offload this handling from the CPU. The use-case 2
can be used for distributed networking tracing e.g. my tracer is on a
remote machine running and expecting time synchronized traces. There
are probably more ideas around that can be taken from the networking
subsystem. Receiving the tracing buffer (after processed through the
NIC ASIC) AF_XDP could be used to get directly the data from the NIC
DMA. Bouncing buffers should be avoided as possible. I don’t want to
go more into the details but a NIC ASIC can be programmed, e.g.
matching is only a shift,mask and compare operation followed by an
action e.g. drop. How it will look in detail I am not sure yet, but I
hope the idea is clear so far. :)

I would simply start with trying introducing a tracing networking
interface (as my current software NIC), put trace events into skb,
loopback them into my tracing networking interface and run wireshark
on it. Then try to use software parts of the networking subsystem to
filter trace events out and see if it works in theory. Bringing it to
real hardware, doing it on real NIC DMA rings and offloading the
filter/matching operations is another future step. For my remote
tracer application and time synchronized tracing, I am sure. I can get
more metadata involved into my trace events that are coming from time
sensitive networking equipment - this is definitely something I would
like to look into.

So that's it about the idea that I have and I am asking for any comments.

Does this sound like a useful thing?

Can something already be done with splice(), etc. and direct the
tracing buffer to the NIC DMA over user space (without bouncing
buffers)?

Somebody already had a similar idea?

- Alex

[0] https://lwn.net/Articles/897257/

Re: RFC: using NIC DMA ring as tracing buffer?

[Mathieu Desnoyers]

On 2024-04-16 10:41, Alexander Aring wrote:
> Hi all,
> 
> I have an idea about using the DMA rings of a networking card as a
> tracing buffer. 

The first thing I'd be worried about is overhead caused by
false-sharing if you trace a large SMP/many-core system all
into the same global NIC buffer. The cache lines will bounce
left and right which is bad performance-wise.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: RFC: using NIC DMA ring as tracing buffer?

[Alexander Aring]

Hi Mathieu,

On Tue, Apr 16, 2024 at 10:49 AM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2024-04-16 10:41, Alexander Aring wrote:
> > Hi all,
> >
> > I have an idea about using the DMA rings of a networking card as a
> > tracing buffer.
>
> The first thing I'd be worried about is overhead caused by
> false-sharing if you trace a large SMP/many-core system all
> into the same global NIC buffer. The cache lines will bounce
> left and right which is bad performance-wise.
>

modern NICs have several tx and rx dma rings. There is an ethtool
command to query the number:

sudo ethtool -g $IFACE

so it depends on the hardware if you can pin one or more rings on a
per-cpu buffer basis.

I hope that somehow helps to make this idea less worrisome.

Thanks.

- Alex

Re: RFC: using NIC DMA ring as tracing buffer?

[Alexander Aring]

Hi,

On Wed, Apr 17, 2024 at 6:51 PM Alexander Aring <aahringo@redhat.com> wrote:
>
> Hi Mathieu,
>
> On Tue, Apr 16, 2024 at 10:49 AM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
> >
> > On 2024-04-16 10:41, Alexander Aring wrote:
> > > Hi all,
> > >
> > > I have an idea about using the DMA rings of a networking card as a
> > > tracing buffer.
> >
> > The first thing I'd be worried about is overhead caused by
> > false-sharing if you trace a large SMP/many-core system all
> > into the same global NIC buffer. The cache lines will bounce
> > left and right which is bad performance-wise.
> >
>
> modern NICs have several tx and rx dma rings. There is an ethtool
> command to query the number:
>
> sudo ethtool -g $IFACE
>
> so it depends on the hardware if you can pin one or more rings on a
> per-cpu buffer basis.
>
> I hope that somehow helps to make this idea less worrisome.
>

What I can think about that makes it worrisome is NUMA. In case of an
E-PCI connected NIC, the NICs buffer associated with a core needs to
be in the same NUMA node. You need to set up your system right, even
with 2 nics or more, to make it work "better".

- Alex