make XDP program aware of multi buffer

make XDP program aware of multi buffer

[Henning Fehrmann]

Hello,

we use XDP/BPF programs to redirect and record network data.
We run the 6.0.9 vanilla kernel with the delivered Mellanox driver.
We mounted MT28800 NICs.

As this scheme seems to work reliably we'd like to step forward and
enable multi buffer packets. We get the following message from the driver:

mlx5_core 0000:01:00.0 enp1s0f0np0: XDP is not allowed with striding RQ and MTU(9000) > 3498

We turned striding off and now we get:

mlx5_core 0000:01:00.0 enp1s0f0np0: MTU(9000) > 3498, too big for an XDP program not aware of multi buffer

It seems that we have to make the XDP program aware of multi buffer.
What would be the correct way to do it?

Thank you,
Henning

Re: make XDP program aware of multi buffer

[Toke Høiland-Jørgensen]

Henning Fehrmann <henning.fehrmann@aei.mpg.de> writes:

> Hello,
>
> we use XDP/BPF programs to redirect and record network data.
> We run the 6.0.9 vanilla kernel with the delivered Mellanox driver.
> We mounted MT28800 NICs.
>
> As this scheme seems to work reliably we'd like to step forward and
> enable multi buffer packets. We get the following message from the driver:
>
> mlx5_core 0000:01:00.0 enp1s0f0np0: XDP is not allowed with striding RQ and MTU(9000) > 3498
>
> We turned striding off and now we get:
>
> mlx5_core 0000:01:00.0 enp1s0f0np0: MTU(9000) > 3498, too big for an XDP program not aware of multi buffer
>
> It seems that we have to make the XDP program aware of multi buffer.
> What would be the correct way to do it?

You just replace:

SEC("xdp")

with

SEC("xdp.frags")

in your source file and, assuming your libbpf version is up-to-date,
that should be it as far as the kernel is concerned.

However, you obviously also need to make sure there are no semantic
issues in your program before doing this. I.e., data_end is no longer
the end of your packet, so if you are using data_end-data to calculate
the packet len, that will no longer be accurate. If you need to access
data beyond the first frag you'll need to use the
xdp_{load,store}_bytes() helpers.

Note that if you're using libxdp you also need to wait for this to be
resolved: https://github.com/xdp-project/xdp-tools/issues/235 - I'm
halfway through a fix for it, so hopefully that should land before too
long.

-Toke

Re: make XDP program aware of multi buffer

[Henning Fehrmann]

> You just replace:
> 
> SEC("xdp")
> 
> with
> 
> SEC("xdp.frags")
> 
> in your source file and, assuming your libbpf version is up-to-date,
> that should be it as far as the kernel is concerned.

Nice. It loads now. I appreciate the work done for multi buffer packets.

> 
> However, you obviously also need to make sure there are no semantic
> issues in your program before doing this. I.e., data_end is no longer
> the end of your packet, so if you are using data_end-data to calculate
> the packet len, that will no longer be accurate. If you need to access
> data beyond the first frag you'll need to use the
> xdp_{load,store}_bytes() helpers.

I guess, this is not the case or does bpf_redirect needs it if I have
multi frame packets?
I realized that redirection actually doesn't happen if the packets
exceed the page size.

Thank you,
Henning

Re: make XDP program aware of multi buffer

[Toke Høiland-Jørgensen]

Henning Fehrmann <henning.fehrmann@aei.mpg.de> writes:

>> You just replace:
>> 
>> SEC("xdp")
>> 
>> with
>> 
>> SEC("xdp.frags")
>> 
>> in your source file and, assuming your libbpf version is up-to-date,
>> that should be it as far as the kernel is concerned.
>
> Nice. It loads now. I appreciate the work done for multi buffer packets.

Cool!

>> However, you obviously also need to make sure there are no semantic
>> issues in your program before doing this. I.e., data_end is no longer
>> the end of your packet, so if you are using data_end-data to calculate
>> the packet len, that will no longer be accurate. If you need to access
>> data beyond the first frag you'll need to use the
>> xdp_{load,store}_bytes() helpers.
>
> I guess, this is not the case or does bpf_redirect needs it if I have
> multi frame packets?
> I realized that redirection actually doesn't happen if the packets
> exceed the page size.

No, XDP_REDIRECT is not supported at all for multi-frame packets:
https://elixir.bootlin.com/linux/latest/source/net/core/filter.c#L4278

Cc Lorenzo who was looking into enabling this at some point...

-Toke