Packet capture and Bonding asymmetries

Packet capture and Bonding asymmetries

[Paul LeoNerd Evans]

[-- Attachment #1: Type: text/plain, Size: 1614 bytes --]

We use ethernet bonding to bond eth0 + eth1 into bond0, in an
active/standby failover pair. Given this is for redundancy, we put the
two physical ethernet links into different switches that follow
different paths in the data centre.

Given this topology, it can be really useful to know which physical
interface packets are received on. It seems the bonding driver doesn't
make this happen:

  # uname -r
  2.6.31.12

  # head -1 /proc/net/bonding/bond0 
  Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

  # pktdump -f icmp
  [15:27:12] RX(bond0): ICMP| 192.168.57.6->192.168.57.1 echo-request seq=1
  [15:27:12] TX(bond0): ICMP| 192.168.57.1->192.168.57.6 echo-reply seq=1
  [15:27:12] TX(eth0): ICMP| 192.168.57.1->192.168.57.6 echo-reply seq=1

I.e. when we transmit we see both the virtual bond0 interface and the
physical eth0 doing so; but when we receive only the virtual bond0
appears to do so.

I believe this should be fixable with a one-line patch; just adding a
call to netif_nit_deliver(skb) from within the bonding driver... though
just offhand I'm unable to find exactly the line where packets received
on slaves gets passed up to the master. :)

Can anyone advise on the sensibility or otherwise of this plan? I really
would like the behaviour where I can see how packets are received - is
this a good plan to acheive it?

I may sometime have a hack at writing a patch for this anyway, presuming
no major objections...

-- 
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: Packet capture and Bonding asymmetries

[Jay Vosburgh]

Paul LeoNerd Evans <leonerd@leonerd.org.uk> wrote:

>We use ethernet bonding to bond eth0 + eth1 into bond0, in an
>active/standby failover pair. Given this is for redundancy, we put the
>two physical ethernet links into different switches that follow
>different paths in the data centre.
>
>Given this topology, it can be really useful to know which physical
>interface packets are received on. It seems the bonding driver doesn't
>make this happen:
>
>  # uname -r
>  2.6.31.12
>
>  # head -1 /proc/net/bonding/bond0 
>  Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)
>
>  # pktdump -f icmp
>  [15:27:12] RX(bond0): ICMP| 192.168.57.6->192.168.57.1 echo-request seq=1
>  [15:27:12] TX(bond0): ICMP| 192.168.57.1->192.168.57.6 echo-reply seq=1
>  [15:27:12] TX(eth0): ICMP| 192.168.57.1->192.168.57.6 echo-reply seq=1
>
>I.e. when we transmit we see both the virtual bond0 interface and the
>physical eth0 doing so; but when we receive only the virtual bond0
>appears to do so.
>
>I believe this should be fixable with a one-line patch; just adding a
>call to netif_nit_deliver(skb) from within the bonding driver... though
>just offhand I'm unable to find exactly the line where packets received
>on slaves gets passed up to the master. :)

	This won't work, because bonding does not have a receive
function in the usual sense.  Instead, the slaves do their normal
receive logic, and then in __netif_receive_skb, packets are assigned to
the bonding master if the device is a slave.

	On the TX side, packet capture can happen at both the bonding
device and at the slave, because the packet will pass through both.

>Can anyone advise on the sensibility or otherwise of this plan? I really
>would like the behaviour where I can see how packets are received - is
>this a good plan to acheive it?

	For your own private testing, you could add a call to
__netif_nit_deliver in netif_receive_skb prior to this part:

        master = ACCESS_ONCE(orig_dev->master);
        if (master) {
                if (skb_bond_should_drop(skb, master))
                        null_or_orig = orig_dev; /* deliver only exact match */
                else
                        skb->dev = master;
        }

	This will give you multiple captures of the same packet, as is
seen for transmit (i.e., one on the slave, one on the bond).  For
non-bonding devices, tcpdump will see each packet twice on the same
device, so it's not really suitable for general use.

>I may sometime have a hack at writing a patch for this anyway, presuming
>no major objections...

	If merely knowing the traffic counts is sufficient, the slaves
do count their received packets individually, so, e.g., ifconfig will
show how many packets a particular slave has received, regardless of
what bonding does with them.  The packet counts for the bonding device
itself are merely a sum of all of its slaves.

	Also, generally speaking, IP protocol traffic that arrives on an
inactive bonding slave is not delivered.  If you're using active-backup
mode, and your traffic makes it through, it likely arrived on the active
slave.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: Packet capture and Bonding asymmetries

[Paul LeoNerd Evans]

[-- Attachment #1: Type: text/plain, Size: 3073 bytes --]

On Wed, Jun 09, 2010 at 03:52:31PM -0700, Jay Vosburgh wrote:
> >I believe this should be fixable with a one-line patch; just adding a
> >call to netif_nit_deliver(skb) from within the bonding driver... though
> >just offhand I'm unable to find exactly the line where packets received
> >on slaves gets passed up to the master. :)
> 
> 	This won't work, because bonding does not have a receive
> function in the usual sense.  Instead, the slaves do their normal
> receive logic, and then in __netif_receive_skb, packets are assigned to
> the bonding master if the device is a slave.

Ah; that would explain why I'm having trouble finding it then :)

> 	For your own private testing, you could add a call to
> __netif_nit_deliver in netif_receive_skb prior to this part:
<snip>

Rather than put it -before- that part, why not inside the if (master)
arm?

-----

diff -urp linux-2.6.33.2.orig/net/core/dev.c linux-2.6.33.2/net/core/dev.c
--- linux-2.6.33.2.orig/net/core/dev.c  2010-04-02 00:02:33.000000000 +0100
+++ linux-2.6.33.2/net/core/dev.c       2010-06-10 12:58:24.000000000 +0100
@@ -2443,6 +2443,7 @@ int netif_receive_skb(struct sk_buff *sk
        orig_dev = skb->dev;
        master = ACCESS_ONCE(orig_dev->master);
        if (master) {
+               netif_nit_deliver(skb);
                if (skb_bond_should_drop(skb, master))
                        null_or_orig = orig_dev; /* deliver only exact match */
                else

-----

(This patch doesn't quite apply cleanly to latest source but if that
approach seems OK I'll rebuild it and submit properly)

> 	This will give you multiple captures of the same packet, as is
> seen for transmit (i.e., one on the slave, one on the bond).  For
> non-bonding devices, tcpdump will see each packet twice on the same
> device, so it's not really suitable for general use.

I was hoping for a proper permanent solution - often it would have been
useful to me, that eth0 appears to receive traffic properly to tcpdump
et.al., as well as transmitting.

> 	If merely knowing the traffic counts is sufficient, the slaves
> do count their received packets individually, so, e.g., ifconfig will
> show how many packets a particular slave has received, regardless of
> what bonding does with them.  The packet counts for the bonding device
> itself are merely a sum of all of its slaves.

I had noticed the counters, yes. That has been useful on occasions. But
at other times we've really wanted to be able to see the actual traffic.

> 	Also, generally speaking, IP protocol traffic that arrives on an
> inactive bonding slave is not delivered.  If you're using active-backup
> mode, and your traffic makes it through, it likely arrived on the active
> slave.

Useful though that is, sometimes it's the fact that traffic appears on a
non-active slave, which is interesting, and we'd like to see what
traffic that was.

-- 
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: Packet capture and Bonding asymmetries

[Paul LeoNerd Evans]

[-- Attachment #1: Type: text/plain, Size: 2165 bytes --]

On Wed, Jun 09, 2010 at 03:52:31PM -0700, Jay Vosburgh wrote:
> 	For your own private testing, you could add a call to
> __netif_nit_deliver in netif_receive_skb prior to this part:
> 
>         master = ACCESS_ONCE(orig_dev->master);
>         if (master) {
>                 if (skb_bond_should_drop(skb, master))
>                         null_or_orig = orig_dev; /* deliver only exact match */
>                 else
>                         skb->dev = master;
>         }
> 
> 	This will give you multiple captures of the same packet, as is
> seen for transmit (i.e., one on the slave, one on the bond).  For
> non-bonding devices, tcpdump will see each packet twice on the same
> device, so it's not really suitable for general use.

As per my last post, I've just tested the following patch and found it
to work just fine:

# pktdump -f "icmp" -n
[13:04:30] RX(eth0): ICMP| 192.168.56.1->192.168.56.6 echo-request seq=1
[13:04:30] RX(bond0): ICMP| 192.168.56.1->192.168.56.6 echo-request seq=1
[13:04:30] TX(bond0): ICMP| 192.168.56.6->192.168.56.1 echo-reply seq=1
[13:04:30] TX(eth0): ICMP| 192.168.56.6->192.168.56.1 echo-reply seq=1

I'll resubmit the patch properly for latest kernel version; this being
2.6.31.12 doesn't apply cleanly to upstream:

-----

--- linux-2.6.31.12-router/net/core/dev.c       2010-01-18 18:30:45.000000000 +0000
+++ linux-2.6.31.12-router_leobonding/net/core/dev.c    2010-06-11 12:39:43.000000000 +0100
@@ -2265,6 +2265,7 @@
        null_or_orig = NULL;
        orig_dev = skb->dev;
        if (orig_dev->master) {
+               netif_nit_deliver(skb);
                if (skb_bond_should_drop(skb))
                        null_or_orig = orig_dev; /* deliver only exact match */
                else

-----

This patch quite deliberately includes packets arriving from non-active
bonding slaves, because the intention of tcpdump, pktdump, et.al., is to
see "close to the wire"; a view of what's happening down that physical
ethernet cable.

-- 
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]