Empty return from bond_eth_hash in 5.15

Empty return from bond_eth_hash in 5.15

[Ivan Babrou]

Hello,

We noticed an issue on Linux 5.15 where it sends packets from a single
connection via different bond members. Some of our machines are
connected to multiple TORs, which means that BGP can attract the same
connection to different servers, depending on which cable you
traverse.

On Linux 5.10 I can see bond_xmit_hash always return the same hash for
the same connection:

$ sudo bpftrace --include linux/ip.h -e 'kprobe:bond_xmit_hash {
@skbs[pid] = arg1 } kretprobe:bond_xmit_hash { $skb_ptr = @skbs[pid];
if ($skb_ptr) { $skb = (struct sk_buff *) $skb_ptr; $ipheader =
((struct iphdr *) ($skb->head + $skb->network_header)); printf("%s
%x\n", ntop($ipheader->daddr), retval); } }' | fgrep --line-buffered
x.y.z.205
x.y.z.205 9f24591
x.y.z.205 9f24591
x.y.z.205 9f24591
x.y.z.205 9f24591
x.y.z.205 9f24591
... many more of these

On Linux 5.10 I get fewer lines, mostly zeros for hash and one actual hash:

$ sudo bpftrace -e 'kprobe:bond_xmit_hash { @skbs[pid] = arg1 }
kretprobe:bond_xmit_hash { $skb_ptr = @skbs[pid]; if ($skb_ptr) { $skb
= (struct sk_buff *) $skb_ptr; $ipheader = ((struct iphdr *)
($skb->head + $skb->network_header)); printf("%s %x\n",
ntop($ipheader->daddr), retval); } }' | fgrep --line-buffered
x.y.z.205
x.y.z.205 0
x.y.z.205 0
x.y.z.205 215fec1b

As I mentioned above, this ends up breaking connections for us, which
is unfortunate.

I suspect that "net, bonding: Refactor bond_xmit_hash for use with
xdp_buff" commit a815bde56b1 has something to do with this. I don't
think we use XDP on the machines I tested.

Re: Empty return from bond_eth_hash in 5.15

[Jay Vosburgh]

Ivan Babrou <ivan@cloudflare.com> wrote:

>Hello,
>
>We noticed an issue on Linux 5.15 where it sends packets from a single
>connection via different bond members. Some of our machines are
>connected to multiple TORs, which means that BGP can attract the same
>connection to different servers, depending on which cable you
>traverse.
>
>On Linux 5.10 I can see bond_xmit_hash always return the same hash for
>the same connection:
>
>$ sudo bpftrace --include linux/ip.h -e 'kprobe:bond_xmit_hash {
>@skbs[pid] = arg1 } kretprobe:bond_xmit_hash { $skb_ptr = @skbs[pid];
>if ($skb_ptr) { $skb = (struct sk_buff *) $skb_ptr; $ipheader =
>((struct iphdr *) ($skb->head + $skb->network_header)); printf("%s
>%x\n", ntop($ipheader->daddr), retval); } }' | fgrep --line-buffered
>x.y.z.205
>x.y.z.205 9f24591
>x.y.z.205 9f24591
>x.y.z.205 9f24591
>x.y.z.205 9f24591
>x.y.z.205 9f24591
>... many more of these
>
>On Linux 5.10 I get fewer lines, mostly zeros for hash and one actual hash:

	Presumably you mean 5.15 here.

>$ sudo bpftrace -e 'kprobe:bond_xmit_hash { @skbs[pid] = arg1 }
>kretprobe:bond_xmit_hash { $skb_ptr = @skbs[pid]; if ($skb_ptr) { $skb
>= (struct sk_buff *) $skb_ptr; $ipheader = ((struct iphdr *)
>($skb->head + $skb->network_header)); printf("%s %x\n",
>ntop($ipheader->daddr), retval); } }' | fgrep --line-buffered
>x.y.z.205
>x.y.z.205 0
>x.y.z.205 0
>x.y.z.205 215fec1b
>
>As I mentioned above, this ends up breaking connections for us, which
>is unfortunate.
>
>I suspect that "net, bonding: Refactor bond_xmit_hash for use with
>xdp_buff" commit a815bde56b1 has something to do with this. I don't
>think we use XDP on the machines I tested.

	This sounds similar to the issue resolved by:

commit 429e3d123d9a50cc9882402e40e0ac912d88cfcf (HEAD -> master, origin/master, origin/HEAD)
Author: Moshe Tal <moshet@nvidia.com>
Date:   Sun Jan 16 19:39:29 2022 +0200

    bonding: Fix extraction of ports from the packet headers
    
    Wrong hash sends single stream to multiple output interfaces.

	which was just committed to net a few days ago; are you in a
position that you'd be able to test this change?

	-J

---
	-Jay Vosburgh, jay.vosburgh@canonical.com

Re: Empty return from bond_eth_hash in 5.15

[Ivan Babrou]

On Tue, Jan 18, 2022 at 4:08 PM Jay Vosburgh <jay.vosburgh@canonical.com> wrote:
> >On Linux 5.10 I get fewer lines, mostly zeros for hash and one actual hash:
>
>         Presumably you mean 5.15 here.

Yes, sorry about the confusion.

>         This sounds similar to the issue resolved by:
>
> commit 429e3d123d9a50cc9882402e40e0ac912d88cfcf (HEAD -> master, origin/master, origin/HEAD)
> Author: Moshe Tal <moshet@nvidia.com>
> Date:   Sun Jan 16 19:39:29 2022 +0200
>
>     bonding: Fix extraction of ports from the packet headers
>
>     Wrong hash sends single stream to multiple output interfaces.
>
>         which was just committed to net a few days ago; are you in a
> position that you'd be able to test this change?

Absolutely, I'll give it a try and report back.

Re: Empty return from bond_eth_hash in 5.15

[Ivan Babrou]

On Tue, Jan 18, 2022 at 4:15 PM Ivan Babrou <ivan@cloudflare.com> wrote:
> > commit 429e3d123d9a50cc9882402e40e0ac912d88cfcf (HEAD -> master, origin/master, origin/HEAD)
> > Author: Moshe Tal <moshet@nvidia.com>
> > Date:   Sun Jan 16 19:39:29 2022 +0200
> >
> >     bonding: Fix extraction of ports from the packet headers
> >
> >     Wrong hash sends single stream to multiple output interfaces.
> >
> >         which was just committed to net a few days ago; are you in a
> > position that you'd be able to test this change?
>
> Absolutely, I'll give it a try and report back.

Looks good:

$ sudo bpftrace -e 'kprobe:__bond_xmit_hash { @skbs[pid] = arg1 }
kretprobe:__bond_xmit_hash { $skb_ptr = @skbs[pid]; if ($skb_ptr) {
$skb = (struct sk_buff *) $skb_ptr; $ipheader = ((struct iphdr *)
($skb->head + $skb->network_header)); printf("%s %x\n",
ntop($ipheader->daddr), retval); } }' | fgrep --line-buffered
x.y.z.205
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b
x.y.z.205 215fec1b

Hope it lands into the 5.15 stable queue as well.