mlx5 AF_XDP zero-copy queues

mlx5 AF_XDP zero-copy queues

[Srivats P]

Hi,

I'm seeing a behaviour difference between kernel 5.4 and 5.15 wrt the
AF_XDP queues used for zero copy. This is specific to the mlx5.

Assume you have 'n' queues configured on the interface.

In 5.4, you had to use n to 2*n -1 queue numbers for AF_XDP zc. Using
0 to n-1 queues would use copy mode.

In 5.15, you have to use 0 to n-1 for zero copy also. Socket creation
fails if you use queue >=n.

Also, with 5.4 netdev stats would not be incremented for AF_XDP
packets whereas with 5.15 they do get incremented?

Can someone confirm this behaviour change or point me to relevant
mailing list discussions or some other information on the same? My
google-fu is failing me.

When did this change go into the kernel? I can't find anything in the
kernelnewbies.org changelog between 5.4 and 5.15.

Srivats

Re: mlx5 AF_XDP zero-copy queues

[Jakub Kicinski]

On Sat, 13 Jul 2024 12:23:31 +0530 Srivats P wrote:
> I'm seeing a behaviour difference between kernel 5.4 and 5.15 wrt the
> AF_XDP queues used for zero copy. This is specific to the mlx5.
> 
> Assume you have 'n' queues configured on the interface.
> 
> In 5.4, you had to use n to 2*n -1 queue numbers for AF_XDP zc. Using
> 0 to n-1 queues would use copy mode.
> 
> In 5.15, you have to use 0 to n-1 for zero copy also. Socket creation
> fails if you use queue >=n.
> 
> Also, with 5.4 netdev stats would not be incremented for AF_XDP
> packets whereas with 5.15 they do get incremented?
> 
> Can someone confirm this behaviour change or point me to relevant
> mailing list discussions or some other information on the same? My
> google-fu is failing me.
> 
> When did this change go into the kernel? I can't find anything in the
> kernelnewbies.org changelog between 5.4 and 5.15.

Hm, can't find the relevant commit quickly either now. But it's
intentional. The new behavior is how all other drivers work.
Hopefully this change isn't a deal breaker for anyone because
long term the lack of uniformity among drivers is a PITA.

Re: mlx5 AF_XDP zero-copy queues

[Srivats P]

On Sat, Jul 13, 2024 at 9:34 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Sat, 13 Jul 2024 12:23:31 +0530 Srivats P wrote:
> > I'm seeing a behaviour difference between kernel 5.4 and 5.15 wrt the
> > AF_XDP queues used for zero copy. This is specific to the mlx5.
> >
> > Assume you have 'n' queues configured on the interface.
> >
> > In 5.4, you had to use n to 2*n -1 queue numbers for AF_XDP zc. Using
> > 0 to n-1 queues would use copy mode.
> >
> > In 5.15, you have to use 0 to n-1 for zero copy also. Socket creation
> > fails if you use queue >=n.
> >
> > Also, with 5.4 netdev stats would not be incremented for AF_XDP
> > packets whereas with 5.15 they do get incremented?
> >
> > Can someone confirm this behaviour change or point me to relevant
> > mailing list discussions or some other information on the same? My
> > google-fu is failing me.
> >
> > When did this change go into the kernel? I can't find anything in the
> > kernelnewbies.org changelog between 5.4 and 5.15.
>
> Hm, can't find the relevant commit quickly either now. But it's
> intentional. The new behavior is how all other drivers work.
> Hopefully this change isn't a deal breaker for anyone because
> long term the lack of uniformity among drivers is a PITA.

Matching how other drivers behave is a good thing. As an application
developer however I need to support both kernel versions. So I'm
trying to find out when this change happened so that I can add a
kernel version check in the code to modify the behaviour for mellanox
drivers.

As a developer/user visible change in behaviour, things like these
need more visibility. Information about XDP and AF_XDP is
unfortunately spread around too much. It would be really useful for
application developers if we could consolidate the information at one
place like DPDK does.

Srivats

Re: mlx5 AF_XDP zero-copy queues

[Srivats P]

On Mon, Jul 15, 2024 at 4:24 PM Srivats P <pstavirs@gmail.com> wrote:
>
> On Sat, Jul 13, 2024 at 9:34 PM Jakub Kicinski <kuba@kernel.org> wrote:
> >
> > On Sat, 13 Jul 2024 12:23:31 +0530 Srivats P wrote:
> > > I'm seeing a behaviour difference between kernel 5.4 and 5.15 wrt the
> > > AF_XDP queues used for zero copy. This is specific to the mlx5.
> > >
> > > Assume you have 'n' queues configured on the interface.
> > >
> > > In 5.4, you had to use n to 2*n -1 queue numbers for AF_XDP zc. Using
> > > 0 to n-1 queues would use copy mode.
> > >
> > > In 5.15, you have to use 0 to n-1 for zero copy also. Socket creation
> > > fails if you use queue >=n.
> > >
> > > Also, with 5.4 netdev stats would not be incremented for AF_XDP
> > > packets whereas with 5.15 they do get incremented?
> > >
> > > Can someone confirm this behaviour change or point me to relevant
> > > mailing list discussions or some other information on the same? My
> > > google-fu is failing me.
> > >
> > > When did this change go into the kernel? I can't find anything in the
> > > kernelnewbies.org changelog between 5.4 and 5.15.

Bump! Can anyone point to the commit or let me know the kernel version
where this behaviour change went in?

> >
> > Hm, can't find the relevant commit quickly either now. But it's
> > intentional. The new behavior is how all other drivers work.
> > Hopefully this change isn't a deal breaker for anyone because
> > long term the lack of uniformity among drivers is a PITA.
>
> Matching how other drivers behave is a good thing. As an application
> developer however I need to support both kernel versions. So I'm
> trying to find out when this change happened so that I can add a
> kernel version check in the code to modify the behaviour for mellanox
> drivers.
>
> As a developer/user visible change in behaviour, things like these
> need more visibility. Information about XDP and AF_XDP is
> unfortunately spread around too much. It would be really useful for
> application developers if we could consolidate the information at one
> place like DPDK does.
>
> Srivats

Srivats

Re: mlx5 AF_XDP zero-copy queues

[Srivats P]

> We previously had a similar problem with kernel 5.4, which resulted in us switching to a different vendor.
> The commit we found in the linux github repo that changes the XSK queues to be indexed from 0 is: https://github.com/torvalds/linux/commit/3db4c85cde7a514a5277070b32e776dbefcaa838
> Hopefully you can trace this back to a specific kernel version change or find reference to it elsewhere.

Thanks Robin! Looks like that commit went into Kernel version 6.1, but
I see the behaviour change in 5.15 itself.

And the 5.15 code includes two sets of receive groups -
https://elixir.bootlin.com/linux/v5.15/source/drivers/net/ethernet/mellanox/mlx5/core/en.h#L167

My app uses AF_XDP for TX only and I'm wondering if there's a
different commit for the TX queues in mlx5?

Srivats