quick help with bonding?

quick help with bonding?

[Chris Friesen]

I"m doing some experimenting with a new network driver that receives 
jumbo frames into multiple separate pages that are then joined together 
in a single sk_buff using skb_fill_page_desc().

It behaved fairly well with standard networking, but its behaving 
strangely with bonding added to the mix.

Could someone either point me to the bonding high level design document 
(couldn't find one at the sourceforge project page) or else give me a 
quick overview of the code path followed by an incoming packet when 
bonding is involved?

Thanks,

Chris Friesen

Re: [Bonding-devel] quick help with bonding?

[Jay Vosburgh]

Chris Friesen <cfriesen@nortel.com> wrote:

>Could someone either point me to the bonding high level design document 
>(couldn't find one at the sourceforge project page) or else give me a 
>quick overview of the code path followed by an incoming packet when 
>bonding is involved?

	There really isn't a high level design document.

	The input path goes from the driver, which (probably) calls
netif_receive_skb.  That function does its processing whatnot, the only
special step for bonding is the processing done by skb_bond() which
assigns the packet to the bonding device.  In the current mainline,
skb_bond() also does some stuff to drop traffic on inactive slaves as
the like.

	After that, the packet follows the regular input path in
netif_skb_receive.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: [Bonding-devel] quick help with bonding?

[Andy Gospodarek]

On Thu, Mar 29, 2007 at 10:14:40AM -0600, Chris Friesen wrote:
> 
> I"m doing some experimenting with a new network driver that receives 
> jumbo frames into multiple separate pages that are then joined together 
> in a single sk_buff using skb_fill_page_desc().
> 
> It behaved fairly well with standard networking, but its behaving 
> strangely with bonding added to the mix.


Can you elaborate on what isn't going well with this driver/hardware?  

Re: [Bonding-devel] quick help with bonding?

[Chris Friesen]

[-- Attachment #1: Type: text/plain, Size: 2054 bytes --]

Andy Gospodarek wrote:

> Can you elaborate on what isn't going well with this driver/hardware?  

I have a ppc64 blade running a customized 2.6.10.  At init time, two of 
our gigE links (eth4 and eth5) are bonded together to form bond0.  This 
link has an MTU of 9000, and uses arp monitoring.  We're using an 
ethernet driver with a modified RX path for jumbo frames[1].  With the 
stock driver, it seems to work fine.

The problem is that eth5 seems to be bouncing up and down every 15 sec 
or so (see the attached log excerpt).  Also, "ifconfig" shows that only 
3 packets totalling 250 bytes have gone out eth5, when I know that the 
arp monitoring code from the bond layer is sending 10 arps/sec out the link.


eth5      Link encap:Ethernet  HWaddr 00:03:CC:51:01:3E
           inet6 addr: fe80::203:ccff:fe51:13e/64 Scope:Link
           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:9000  Metric:1
           RX packets:119325 errors:90283 dropped:90283 overruns:90283 
frame:0
           TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:8978310 (8.5 MiB)  TX bytes:250 (250.0 b)
           Base address:0x3840 Memory:92220000-92240000


I had initially suspected that it might be due to the "u32 jiffies" 
stuff in bonding.h, but changing that doesn't seem to fix the issue.

If I boot the system and then log in and manually create the bond link 
(rather than it happening at init time) then I don't see the problem.

If it matters at all, normally the system boots from eth4.  I'm going to 
try booting from eth6 and see if the problem still occurs.


Chris




[1] I'm not sure if I'm supposed to mention the specific driver, as it 
hasn't been officially released yet, so I'll keep this high-level. 
Normally for jumbo frames you need to allocate a large physically 
contiguous buffer.  With the modified driver, rather than receiving into 
a contiguous buffer the incoming packet is split across multiple pages 
which are then reassembled into an sk_buff and passed up the link.

[-- Attachment #2: bond_log.txt --]
[-- Type: text/plain, Size: 3505 bytes --]

Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: ARP monitoring set to 100 ms with 2 target(s): 172.24.136.0 172.24.137.0
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: ARP monitoring set to 100 ms with 2 target(s): 172.25.136.0 172.25.137.0
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: Warning: failed to get speed/duplex from eth4, speed forced to 100Mbps, duplex forced to Full.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: enslaving eth4 as an active interface with an up link.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: Warning: failed to get speed/duplex from eth5, speed forced to 100Mbps, duplex forced to Full.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: enslaving eth5 as an active interface with an up link.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth4 to be reset in 30000 msec.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth4 is now down.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: now running without any active interface !
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: link status definitely up for interface eth5
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth4
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth4 is now up
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.
Mar 29 20:54:09 base0-0-0-5-0-11-1 kernel: bonding: interface eth4 reset delay set to 600 msec.
Mar 29 20:54:59 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5
Mar 29 20:54:59 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now up
Mar 29 20:54:59 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:54:59 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.
Mar 29 20:55:15 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5
Mar 29 20:55:15 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now up
Mar 29 20:55:15 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:55:15 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.
Mar 29 20:55:30 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5
Mar 29 20:55:30 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now up
Mar 29 20:55:30 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:55:30 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.
Mar 29 20:55:45 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5
Mar 29 20:55:45 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now up
Mar 29 20:55:46 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:55:46 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.

Re: [Bonding-devel] quick help with bonding?

[Jay Vosburgh]

Chris Friesen <cfriesen@nortel.com> wrote:
[...]
>I have a ppc64 blade running a customized 2.6.10.  At init time, two of
>our gigE links (eth4 and eth5) are bonded together to form bond0.  This
>link has an MTU of 9000, and uses arp monitoring.  We're using an ethernet
>driver with a modified RX path for jumbo frames[1].  With the stock
>driver, it seems to work fine.

	2.6.10 is pretty old, and there have been a number of fixes to
the bonding ARP monitor since then, so it may be that it is simply
misbehaving (presuming that you're running the 2.6.10 bonding driver).
Are you in a position to test against a more recent kernel (and/or
bonding driver)?  Does the miimon misbehave in a similar fashion?

>The problem is that eth5 seems to be bouncing up and down every 15 sec or
>so (see the attached log excerpt).  Also, "ifconfig" shows that only 3
>packets totalling 250 bytes have gone out eth5, when I know that the arp
>monitoring code from the bond layer is sending 10 arps/sec out the link.
[...]
>Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth4 to be reset in 30000 msec.
[...]
>Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5

	These two messages (which appear a number of times in your log
excerpt) are not from the standard mainline bonding driver, even in
2.6.10.  I don't know what this is all about.

>If I boot the system and then log in and manually create the bond link
>(rather than it happening at init time) then I don't see the problem.

	I would hazard to guess that it's an ARP monitor problem; older
versions of the ARP monitor had less than intelligent means to figure
out what the bond's IP address is (to use for the probes).  This, along
with some logic problems in the monitor code itself, led to various
problems with the ARP probes and the sort of "up / down" cycle of
behavior you seem to be seeing.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: [Bonding-devel] quick help with bonding?

[Mark Huth]

Jay Vosburgh wrote:
> Chris Friesen <cfriesen@nortel.com> wrote:
> [...]
>   
>> I have a ppc64 blade running a customized 2.6.10.  At init time, two of
>> our gigE links (eth4 and eth5) are bonded together to form bond0.  This
>> link has an MTU of 9000, and uses arp monitoring.  We're using an ethernet
>> driver with a modified RX path for jumbo frames[1].  With the stock
>> driver, it seems to work fine.
>>     
>
> 	2.6.10 is pretty old, and there have been a number of fixes to
> the bonding ARP monitor since then, so it may be that it is simply
> misbehaving (presuming that you're running the 2.6.10 bonding driver).
> Are you in a position to test against a more recent kernel (and/or
> bonding driver)?  Does the miimon misbehave in a similar fashion?
>
>   
>> The problem is that eth5 seems to be bouncing up and down every 15 sec or
>> so (see the attached log excerpt).  Also, "ifconfig" shows that only 3
>> packets totalling 250 bytes have gone out eth5, when I know that the arp
>> monitoring code from the bond layer is sending 10 arps/sec out the link.
>>     
> [...]
>   
>> Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth4 to be reset in 30000 msec.
>>     
> [...]
>   
>> Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5
>>     
>
> 	These two messages (which appear a number of times in your log
> excerpt) are not from the standard mainline bonding driver, even in
> 2.6.10.  I don't know what this is all about.
>
>   
>> If I boot the system and then log in and manually create the bond link
>> (rather than it happening at init time) then I don't see the problem.
>>     
>
> 	I would hazard to guess that it's an ARP monitor problem; older
> versions of the ARP monitor had less than intelligent means to figure
> out what the bond's IP address is (to use for the probes).  This, along
> with some logic problems in the monitor code itself, led to various
> problems with the ARP probes and the sort of "up / down" cycle of
> behavior you seem to be seeing.
>
> 	-J
>
> ---
> 	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com
> -
>   
I'll second what Jay said.  I support a version of the 2.6.10 kernel 
with bonding, and I needed to upgrade the bonding that was native to 
2.6.10 to get reasonable behavior.  You may also need a newer ifenslave.

It also looks like the mii interface is not well-behaved, because of the 
initialization messages related to link speed.

Re: [Bonding-devel] quick help with bonding?

[Chris Friesen]

Jay Vosburgh wrote:

> 	2.6.10 is pretty old, and there have been a number of fixes to
> the bonding ARP monitor since then, so it may be that it is simply
> misbehaving (presuming that you're running the 2.6.10 bonding driver).
> Are you in a position to test against a more recent kernel (and/or
> bonding driver)?  Does the miimon misbehave in a similar fashion?

Testing a more recent kernel is problematic.  A new bonding driver could 
be possible, assuming the code hasn't changed too much.

I just did another experiment.  Normally we boot via eth4 (which then 
becomes part of the bond  with eth5 at init time).  If I boot via eth6 
instead, it appears as though the problem doesn't show up.

Chris

Re: [Bonding-devel] quick help with bonding?

[Jay Vosburgh]

Chris Friesen <cfriesen@nortel.com> wrote:

>Jay Vosburgh wrote:
>
>> 	2.6.10 is pretty old, and there have been a number of fixes to
>> the bonding ARP monitor since then, so it may be that it is simply
>> misbehaving (presuming that you're running the 2.6.10 bonding driver).
>> Are you in a position to test against a more recent kernel (and/or
>> bonding driver)?  Does the miimon misbehave in a similar fashion?
>
>Testing a more recent kernel is problematic.  A new bonding driver could
>be possible, assuming the code hasn't changed too much.
>
>I just did another experiment.  Normally we boot via eth4 (which then
>becomes part of the bond  with eth5 at init time).  If I boot via eth6
>instead, it appears as though the problem doesn't show up.

	Well, if you're still inclined to investigate, you may want to
inspect the ARP probes generated by bonding in the "bad" situation.  I
don't really have any evidence to back it up, but one guess is that the
IP detection stuff in the ARP monitor is getting messed up. I'd check to
see if the ARP probes have the correct source IP address (which, in the
2.6.10 era bonding, is determined only once by inspection of outbound
ARP traffic, and never updated).  If you're not using active-backup mode
(you didn't say, and I can't tell from your log excerpt), then the ARP
monitor may not work at all (since it will send ARP probes with an IP
source of all zeros).

	If bad ARP probe source addresses are your problem, then that is
fixed in a later version of bonding, although the changes would require
some rework to backport to 2.6.10 (if they can be backported).

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: [Bonding-devel] quick help with bonding?

[Andy Gospodarek]

On Thu, Mar 29, 2007 at 05:42:54PM -0600, Chris Friesen wrote:
> Jay Vosburgh wrote:
> 
> > 	2.6.10 is pretty old, and there have been a number of fixes to
> > the bonding ARP monitor since then, so it may be that it is simply
> > misbehaving (presuming that you're running the 2.6.10 bonding driver).
> > Are you in a position to test against a more recent kernel (and/or
> > bonding driver)?  Does the miimon misbehave in a similar fashion?
> 
> Testing a more recent kernel is problematic.  A new bonding driver could 
> be possible, assuming the code hasn't changed too much.

If you are looking for a decent source for patches you could consider
downloading the latest source-rpm from RHEL4/CentOS4.  The bonding
driver in those releases have been updated to much later code and I can
tell you from personal experience they work pretty well.  You may need
to do some backporting to get the latest arp-monitoring features, but
let me know if you need a hand with that, I might have some laying
around. ;)
 
> I just did another experiment.  Normally we boot via eth4 (which then 
> becomes part of the bond  with eth5 at init time).  If I boot via eth6 
> instead, it appears as though the problem doesn't show up.

Does eth6 use the same hardware/driver as eth4/5?  (Sorry if I missed
that in the thread, but didn't see if you indicated that it did.)

Re: [Bonding-devel] quick help with bonding?

[Chris Friesen]

Andy Gospodarek wrote:

> If you are looking for a decent source for patches you could consider
> downloading the latest source-rpm from RHEL4/CentOS4.  The bonding
> driver in those releases have been updated to much later code and I can
> tell you from personal experience they work pretty well.  You may need
> to do some backporting to get the latest arp-monitoring features, but
> let me know if you need a hand with that, I might have some laying
> around. ;)

I'm just about to load a kernel with a backport of bonding from 2.6.14. 
  I'll try it out and if it doesn't help I'll try the RHEL4 one.

> Does eth6 use the same hardware/driver as eth4/5?  (Sorry if I missed
> that in the thread, but didn't see if you indicated that it did.)

No, eth6 is an AMD-8111.

Chris

Re: [Bonding-devel] quick help with bonding?

[Chris Friesen]

Chris Friesen wrote:
> Andy Gospodarek wrote:
> 
>> If you are looking for a decent source for patches you could consider
>> downloading the latest source-rpm from RHEL4/CentOS4.  The bonding
>> driver in those releases have been updated to much later code and I can
>> tell you from personal experience they work pretty well.

> I'm just about to load a kernel with a backport of bonding from 2.6.14. 
>  I'll try it out and if it doesn't help I'll try the RHEL4 one.

No joy on the 2.6.14 backport, so I guess I'll try the RHEL4 route.

Chris

Re: [Bonding-devel] quick help with bonding?

[Andy Gospodarek]

On Thu, Mar 29, 2007 at 07:26:07PM -0600, Chris Friesen wrote:
> Chris Friesen wrote:
> > Andy Gospodarek wrote:
> > 
> >> If you are looking for a decent source for patches you could consider
> >> downloading the latest source-rpm from RHEL4/CentOS4.  The bonding
> >> driver in those releases have been updated to much later code and I can
> >> tell you from personal experience they work pretty well.
> 
> > I'm just about to load a kernel with a backport of bonding from 2.6.14. 
> >  I'll try it out and if it doesn't help I'll try the RHEL4 one.
> 
> No joy on the 2.6.14 backport, so I guess I'll try the RHEL4 route.
> 

Ah, ok.  I'm not too sure how different the 2.6.9 and 2.6.10 bonding
code was, so it might take a little tweaking but I'm guessing there
won't be significant differences.

Re: [Bonding-devel] quick help with bonding?

[Chris Friesen]

Chris Friesen wrote:

> No joy on the 2.6.14 backport, so I guess I'll try the RHEL4 route.

Bonding driver from 2.6.9-42.0.8.EL doesn't help at all, at least with 
the module parms I was using before.

Switching to miimon doesn't help either.

Chris

Re: quick help with bonding?

[Chris Friesen]

Chris Friesen wrote:

> It behaved fairly well with standard networking, but its behaving 
> strangely with bonding added to the mix.

Thanks to everyone for the suggestions and help.

It turns out it was actually a driver problem that was more easily 
manifested by the timing of the way bonding changed the link MTU.

Chris