dhcp/bonding interaction question

dhcp/bonding interaction question

[Chris Friesen]

We've recently run into an interesting issue, and I was hoping to get 
some suggestions.

We've got a boot server configured with two bonded links using the XOR 
tx mode.  Another board comes up.  It's got two links which will 
eventually be bonded.  It issues a DHCP request to the server over one 
of the links, which arrives on one of the bonded slaves.  The server 
sends a reply, but the XOR results in the packet being sent back via the 
other slave.  On the booting blade the xid in the packet doesn't match 
the xid for the device on which the packet was received, so the packet 
is dropped.

What's the proper solution here?  Should dhcpd be forcing reply packets 
out the slave on which the packet was received?

Chris

Re: dhcp/bonding interaction question

[Jay Vosburgh]

Chris Friesen <cfriesen@nortel.com> wrote:

>We've got a boot server configured with two bonded links using the XOR tx
>mode.  Another board comes up.  It's got two links which will eventually
>be bonded.  It issues a DHCP request to the server over one of the links,
>which arrives on one of the bonded slaves.  The server sends a reply, but
>the XOR results in the packet being sent back via the other slave.  On the
>booting blade the xid in the packet doesn't match the xid for the device
>on which the packet was received, so the packet is dropped.

	Presumably the blade is matching xid on a per-interface basis.

	I'm guessing your problem is really due to the nature of the
intra-chassis network on the blade system.  If it's like the ones I'm
familiar with (eth0 of all blades to a single switch, eth1 of all blades
to a different switch, etc), then you can't configure the switches into
an etherchannel group as can be done with the usual configuration
(multiple ports on one switch).  If the other blades in the chassis
aren't also configured for etherchannel (balance-xor is an etherchannel
compatible mode), then they'll see the same MAC arriving on multiple
interfaces and not really be prepared to deal with it.

>What's the proper solution here?  Should dhcpd be forcing reply packets
>out the slave on which the packet was received?

	Why is the blade issuing DHCP before the bond is configured?

	I'm unsure as to whether dhcpd should force replies as
suggested, but recent changes to bonding/net core would probably permit
dhcpd to run directly on the slave (it might need tweaking, I'm not
sure) and accomplish that result.  I don't think it would work on older
versions of bonding, since there's no way to tell which slave a packet
arrived on.

	In that case, dhcpd would have to run on all slaves, since a
bond that uses DHCP for its address assignment might use any of its
interfaces to issue the request, and it would accept a response on any
interface.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: dhcp/bonding interaction question

[Chris Friesen]

Jay Vosburgh wrote:
> Chris Friesen <cfriesen@nortel.com> wrote:
> 
>> On the
>> booting blade the xid in the packet doesn't match the xid for the device
>> on which the packet was received, so the packet is dropped.
> 
> 	Presumably the blade is matching xid on a per-interface basis.

Yes, the dhcp code in the kernel does this.

> 	I'm guessing your problem is really due to the nature of the
> intra-chassis network on the blade system.  If it's like the ones I'm
> familiar with (eth0 of all blades to a single switch, eth1 of all blades
> to a different switch, etc), then you can't configure the switches into
> an etherchannel group as can be done with the usual configuration
> (multiple ports on one switch).

This has been handled already.  The two switches are a single 
etherchannel group and all blades (except the booting one) have 
etherchannel enabled.

>> What's the proper solution here?  Should dhcpd be forcing reply packets
>> out the slave on which the packet was received?
> 
> 	Why is the blade issuing DHCP before the bond is configured?

The blade is net-booting.  The BIOS does DHCP to download the kernel, 
and this works.  The kernel comes up and does DHCP to obtain an IP 
address so it can download the boot image, and this fails as described.

The etherchannel/bond link isn't set up until userspace is loaded and 
the bonding driver is insmodded (under multiple aliases with different 
sets of slaves).

> I'm unsure as to whether dhcpd should force replies as suggested, but
> recent changes to bonding/net core would probably permit dhcpd to run
> directly on the slave (it might need tweaking, I'm not sure) and
> accomplish that result. I don't think it would work on older versions
> of bonding, since there's no way to tell which slave a packet arrived
> on.

That's not a problem.  We've got an older version, but support for 
querying which slave a packet arrived on has been added.

Chris

Re: dhcp/bonding interaction question

[Jay Vosburgh]

Chris Friesen <cfriesen@nortel.com> wrote:

>Jay Vosburgh wrote:
>> Chris Friesen <cfriesen@nortel.com> wrote:
[...]
>Yes, the dhcp code in the kernel does this.

	Ah, I didn't know that.

>> 	I'm guessing your problem is really due to the nature of the
>> intra-chassis network on the blade system.  If it's like the ones I'm
>> familiar with (eth0 of all blades to a single switch, eth1 of all blades
>> to a different switch, etc), then you can't configure the switches into
>> an etherchannel group as can be done with the usual configuration
>> (multiple ports on one switch).
>
>This has been handled already.  The two switches are a single etherchannel
>group and all blades (except the booting one) have etherchannel enabled.

	Ok, so your problem (if I'm understanding things correctly) is
really a chicken-and-egg type of deal.  Your entire network is all
etherchannel, except for this blade, which needs to connect,
temporarily, in a non-etherchannel manner in order to boot up and become
etherchannel-ified (at which point it'll work).

	One possible problem with hacking the dhcpd to send back out on
the receiving interface is that (because of the etherchannel) there's no
general guarantee that the switch paths are symmetrical, particularly if
a port is down somewhere.

>That's not a problem.  We've got an older version, but support for
>querying which slave a packet arrived on has been added.

	If you're already rolling your own kernels, would it be easier
to remove the xid interface check from the kernel's dhcp client (i.e.,
accept the DHCP reply as if it had arrived on the proper interface for
its xid)?  You could make it into some kind of "netboot_on_etherchannel"
option.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: dhcp/bonding interaction question

[Chris Friesen]

Jay Vosburgh wrote:
> Chris Friesen <cfriesen@nortel.com> wrote:
> 	Ok, so your problem (if I'm understanding things correctly) is
> really a chicken-and-egg type of deal.  Your entire network is all
> etherchannel, except for this blade, which needs to connect,
> temporarily, in a non-etherchannel manner in order to boot up and become
> etherchannel-ified (at which point it'll work).

Basically, yes.

> 	One possible problem with hacking the dhcpd to send back out on
> the receiving interface is that (because of the etherchannel) there's no
> general guarantee that the switch paths are symmetrical, particularly if
> a port is down somewhere.

True.

> 	If you're already rolling your own kernels, would it be easier
> to remove the xid interface check from the kernel's dhcp client (i.e.,
> accept the DHCP reply as if it had arrived on the proper interface for
> its xid)?  You could make it into some kind of "netboot_on_etherchannel"
> option.

Yes, this is being considered as well.  Initially I was against it due 
to the possibility of introducing unexpected behaviour, but making it a 
boot argument makes a lot of sense, especially since it would allow for 
more robust behaviour on certain fault scenarios.

Thanks,

Chris