[Intel-wired-lan] zero-copy between interfaces

[Intel-wired-lan] zero-copy between interfaces

[=?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?=]

On Mon, 13 Jan 2020 at 01:28, Ryan Goodfellow <rgoodfel@isi.edu> wrote:
>
[...]
>
> I could not get zero-copy to work with the i40e driver as it would crash. I've
> attached the corresponding traces from dmesg.

Thanks Ryan! I had a look at the crash, and it's in the XDP setup:

i40e_xdp_setup:
...
 for (i = 0; i < vsi->num_queue_pairs; i++)
     WRITE_ONCE(vsi->rx_rings[i]->xdp_prog, vsi->xdp_prog);

and the vsi->rx_ring[0] is NULL. This is clearly broken.

It would help with more lines from your dmesg: the cut i40e log hints
that something is really broken:

[  328.579154] i40e 0000:b7:00.2: failed to get tracking for 256
queues for VSI 0 err -12
[  328.579280] i40e 0000:b7:00.2: setup of MAIN VSI failed
[  328.579367] i40e 0000:b7:00.2: can't remove VEB 162 with 0 VSIs left

Is it possible to dig out the complete log?

Thanks!
Bj?rn

[Intel-wired-lan] zero-copy between interfaces

[Ryan Goodfellow]

On Fri, Jan 17, 2020 at 01:32:07PM +0100, Bj?rn T?pel wrote:
> On Mon, 13 Jan 2020 at 01:28, Ryan Goodfellow <rgoodfel@isi.edu> wrote:
> >
> [...]
> >
> > I could not get zero-copy to work with the i40e driver as it would crash. I've
> > attached the corresponding traces from dmesg.
> 
> Thanks Ryan! I had a look at the crash, and it's in the XDP setup:
> 
> i40e_xdp_setup:
> ...
>  for (i = 0; i < vsi->num_queue_pairs; i++)
>      WRITE_ONCE(vsi->rx_rings[i]->xdp_prog, vsi->xdp_prog);
> 
> and the vsi->rx_ring[0] is NULL. This is clearly broken.
> 
> It would help with more lines from your dmesg: the cut i40e log hints
> that something is really broken:
> 
> [  328.579154] i40e 0000:b7:00.2: failed to get tracking for 256
> queues for VSI 0 err -12
> [  328.579280] i40e 0000:b7:00.2: setup of MAIN VSI failed
> [  328.579367] i40e 0000:b7:00.2: can't remove VEB 162 with 0 VSIs left
> 
> Is it possible to dig out the complete log?

Hi Bj?rn,

I've linked a full dmesg log from an XDP setup crash. Note that there are 
two i40e cards on this machine. The X710 (0000:65:00.0, 0000:65:00.1) works 
fine, the X722 (0000:b7:00.0, 0000:b7:00.1, 0000:b7:00.2, 0000:b7:00.3) is the
one that is crashing on XDP setup.

https://gitlab.com/mergetb/tech/network-emulation/kernel/snippets/1931080

Some info that may be useful:

ry at turbine:~$ sudo ethtool -i eno7
driver: i40e
version: 2.8.20-k
firmware-version: 3.33 0x80001006 1.1747.0
expansion-rom-version:
bus-info: 0000:b7:00.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

The firmware version 3.33 was the latest I could find as of a few weeks ago.

ry at turbine:~$ sudo lspci -vvv | grep 722
b7:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04)
	DeviceName: Intel LAN X722 #1
	Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GBASE-T
b7:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04)
	DeviceName: Intel LAN X722 #2
	Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GBASE-T
b7:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04)
	DeviceName: Intel LAN X722 #3
	Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GbE SFP+
b7:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04)
	DeviceName: Intel LAN X722 #4
	Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GbE SFP+

ry at ryzen:~$ uname -a
Linux ryzen 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux

ry at turbine:~/kmoa/bpf-next$ git log -2
commit 60d71397d27e7859fdaaaaab6594e4d977ae46e2 (HEAD -> master)
Author: Ryan Goodfellow <rgoodfel@isi.edu>
Date:   Wed Jan 15 16:54:39 2020 -0500

    add xdpsock_multidev sample program

    This is a simple program that uses AF_XDP sockets to forward packets
    between two interfaces using a common memory region and no copying of
    packets.

    Signed-off-by: Ryan Goodfellow <rgoodfel@isi.edu>

commit 9173cac3b64e6785dd604f5075e6035b045a0026 (origin/master, origin/HEAD)
Author: Andrii Nakryiko <andriin@fb.com>
Date:   Wed Jan 15 11:08:56 2020 -0800

    libbpf: Support .text sub-calls relocations

    The LLVM patch https://reviews.llvm.org/D72197 makes LLVM emit function call
    relocations within the same section. This includes a default .text section,
    which contains any BPF sub-programs. This wasn't the case before and so libbpf
    was able to get a way with slightly simpler handling of subprogram call
    relocations.

    This patch adds support for .text section relocations. It needs to ensure
    correct order of relocations, so does two passes:
    - first, relocate .text instructions, if there are any relocations in it;
    - then process all the other programs and copy over patched .text instructions
    for all sub-program calls.

    v1->v2:
    - break early once .text program is processed.

    Signed-off-by: Andrii Nakryiko <andriin@fb.com>
    Signed-off-by: Alexei Starovoitov <ast@kernel.org>
    Acked-by: Yonghong Song <yhs@fb.com>
    Cc: Alexei Starovoitov <ast@kernel.org>
    Link: https://lore.kernel.org/bpf/20200115190856.2391325-1-andriin at fb.com

-- 
~ ry

[Intel-wired-lan] zero-copy between interfaces

[Ryan Goodfellow]

On Fri, Jan 17, 2020 at 12:16:37PM -0500, Ryan Goodfellow wrote:
> ry at ryzen:~$ uname -a
> Linux ryzen 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux

correction, that was a terminal on the wrong machine

ry at turbine:~$ uname -a
Linux turbine 5.5.0-rc4-moa+ #16 SMP Fri Jan 17 10:52:42 EST 2020 x86_64 GNU/Linux


-- 
~ ry

[Intel-wired-lan] zero-copy between interfaces

[Magnus Karlsson]

On Fri, Jan 17, 2020 at 6:16 PM Ryan Goodfellow <rgoodfel@isi.edu> wrote:
>
> On Fri, Jan 17, 2020 at 01:32:07PM +0100, Bj?rn T?pel wrote:
> > On Mon, 13 Jan 2020 at 01:28, Ryan Goodfellow <rgoodfel@isi.edu> wrote:
> > >
> > [...]
> > >
> > > I could not get zero-copy to work with the i40e driver as it would crash. I've
> > > attached the corresponding traces from dmesg.
> >
> > Thanks Ryan! I had a look at the crash, and it's in the XDP setup:
> >
> > i40e_xdp_setup:
> > ...
> >  for (i = 0; i < vsi->num_queue_pairs; i++)
> >      WRITE_ONCE(vsi->rx_rings[i]->xdp_prog, vsi->xdp_prog);
> >
> > and the vsi->rx_ring[0] is NULL. This is clearly broken.
> >
> > It would help with more lines from your dmesg: the cut i40e log hints
> > that something is really broken:
> >
> > [  328.579154] i40e 0000:b7:00.2: failed to get tracking for 256
> > queues for VSI 0 err -12
> > [  328.579280] i40e 0000:b7:00.2: setup of MAIN VSI failed
> > [  328.579367] i40e 0000:b7:00.2: can't remove VEB 162 with 0 VSIs left
> >
> > Is it possible to dig out the complete log?
>
> Hi Bj?rn,
>
> I've linked a full dmesg log from an XDP setup crash. Note that there are
> two i40e cards on this machine. The X710 (0000:65:00.0, 0000:65:00.1) works
> fine, the X722 (0000:b7:00.0, 0000:b7:00.1, 0000:b7:00.2, 0000:b7:00.3) is the
> one that is crashing on XDP setup.

Ryan,

I was wondering if you could run two small experiments since I cannot
reproduce this?

1: Run your program using the two ports on your X710 card. Does it
work? This is my setup and works for me.
2: On your Mellanox setup, insert a kick_tx() call for each of your
two sockets before the poll() call in your forward() function. Just to
see if it works when we explicitly wake up the driver.

Thanks: Magnus

> https://gitlab.com/mergetb/tech/network-emulation/kernel/snippets/1931080
>
> Some info that may be useful:
>
> ry at turbine:~$ sudo ethtool -i eno7
> driver: i40e
> version: 2.8.20-k
> firmware-version: 3.33 0x80001006 1.1747.0
> expansion-rom-version:
> bus-info: 0000:b7:00.2
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> The firmware version 3.33 was the latest I could find as of a few weeks ago.
>
> ry at turbine:~$ sudo lspci -vvv | grep 722
> b7:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04)
>         DeviceName: Intel LAN X722 #1
>         Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GBASE-T
> b7:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04)
>         DeviceName: Intel LAN X722 #2
>         Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GBASE-T
> b7:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04)
>         DeviceName: Intel LAN X722 #3
>         Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GbE SFP+
> b7:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04)
>         DeviceName: Intel LAN X722 #4
>         Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GbE SFP+
>
> ry at ryzen:~$ uname -a
> Linux ryzen 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux
>
> ry at turbine:~/kmoa/bpf-next$ git log -2
> commit 60d71397d27e7859fdaaaaab6594e4d977ae46e2 (HEAD -> master)
> Author: Ryan Goodfellow <rgoodfel@isi.edu>
> Date:   Wed Jan 15 16:54:39 2020 -0500
>
>     add xdpsock_multidev sample program
>
>     This is a simple program that uses AF_XDP sockets to forward packets
>     between two interfaces using a common memory region and no copying of
>     packets.
>
>     Signed-off-by: Ryan Goodfellow <rgoodfel@isi.edu>
>
> commit 9173cac3b64e6785dd604f5075e6035b045a0026 (origin/master, origin/HEAD)
> Author: Andrii Nakryiko <andriin@fb.com>
> Date:   Wed Jan 15 11:08:56 2020 -0800
>
>     libbpf: Support .text sub-calls relocations
>
>     The LLVM patch https://reviews.llvm.org/D72197 makes LLVM emit function call
>     relocations within the same section. This includes a default .text section,
>     which contains any BPF sub-programs. This wasn't the case before and so libbpf
>     was able to get a way with slightly simpler handling of subprogram call
>     relocations.
>
>     This patch adds support for .text section relocations. It needs to ensure
>     correct order of relocations, so does two passes:
>     - first, relocate .text instructions, if there are any relocations in it;
>     - then process all the other programs and copy over patched .text instructions
>     for all sub-program calls.
>
>     v1->v2:
>     - break early once .text program is processed.
>
>     Signed-off-by: Andrii Nakryiko <andriin@fb.com>
>     Signed-off-by: Alexei Starovoitov <ast@kernel.org>
>     Acked-by: Yonghong Song <yhs@fb.com>
>     Cc: Alexei Starovoitov <ast@kernel.org>
>     Link: https://lore.kernel.org/bpf/20200115190856.2391325-1-andriin at fb.com
>
> --
> ~ ry

[Intel-wired-lan] zero-copy between interfaces

[Ryan Goodfellow]

On Mon, Jan 20, 2020 at 09:24:05AM +0100, Magnus Karlsson wrote:
> 
> I was wondering if you could run two small experiments since I cannot
> reproduce this?
> 
> 1: Run your program using the two ports on your X710 card. Does it
> work? This is my setup and works for me.

The X710 card works without issue.

> 2: On your Mellanox setup, insert a kick_tx() call for each of your
> two sockets before the poll() call in your forward() function. Just to
> see if it works when we explicitly wake up the driver.

This did not have an effect on the observed behavior. Exactly N packets go
through the interface where N is equal to the size of the FQ/CQ rings and then
forwarding halts.

--
~ ry

[Intel-wired-lan] zero-copy between interfaces

[=?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?=]

On 2020-01-17 18:16, Ryan Goodfellow wrote:
[...]
> 
> https://gitlab.com/mergetb/tech/network-emulation/kernel/snippets/1931080
> 

Ryan, thanks a lot for the detailed report! Much appreciated!

Long story short, the i40e crash is that the drivers tries to allocate 
256 queues, but the HW is short on queues. The drivers enters a broken 
state, which triggers the crash.

I'll make sure we'll get a patch for this.


Bj?rn