[RFT PATCH] Delay netroot mounting by 1 second

[RFT PATCH] Delay netroot mounting by 1 second

[Seewer Philippe]

While testing in our production environment, I discovered
that the time between interface configuration and trying 
to mount the nfs root is too short if the dhcp-server 
provides an mtu. Having to set a mtu means that netroot
executes 'ip link down', set's the mtu and the does
'ip link up' again.

This results in the mount error "No route to host", because
the network equipment has not had enough time to "shake hands". 
The patch below adds a sleep 1 after interface configuration, 
to give the equipment enough time.

Could anyone with a similar environment (and possibly with
a netroot server reachable only via default route) test this
with and without this patch to confirm?

Thank you,
Philippe


diff --git a/modules.d/40network/netroot b/modules.d/40network/netroot
index f719d93..81b99f7 100755
--- a/modules.d/40network/netroot
+++ b/modules.d/40network/netroot
@@ -88,7 +88,11 @@ for iface in $IFACES ; do
     . /tmp/net.$iface.up
 done
 
-[ -e /tmp/net.$netif.gw ]          && . /tmp/net.$netif.gw
+# Wait a bit until we continue to give the interface(s) time to
+# configure
+sleep 1
+
+[ -e /tmp/net.$netif.gw ]          && . /tmp/net.$netif.gw 
 [ -e /tmp/net.$netif.hostname ]    && . /tmp/net.$netif.hostname
 [ -e /tmp/net.$netif.resolv.conf ] && cp -f /tmp/net.$netif.resolv.conf /etc/resolv.conf
 
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Bill Nottingham]

Seewer Philippe (philippe.seewer-omB+W0Dpw2o@public.gmane.org) said: 
> While testing in our production environment, I discovered
> that the time between interface configuration and trying to mount the nfs 
> root is too short if the dhcp-server provides an mtu. Having to set a mtu 
> means that netroot
> executes 'ip link down', set's the mtu and the does
> 'ip link up' again.
>
> This results in the mount error "No route to host", because
> the network equipment has not had enough time to "shake hands". The patch 
> below adds a sleep 1 after interface configuration, to give the equipment 
> enough time.

Depending on your switch configuration, the time for the switch to start
passing traffic again on link bounce couuld be anywhere from 0.1 seconds
to 20 seconds...

Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Seewer Philippe]

Bill Nottingham wrote:
> Seewer Philippe (philippe.seewer-omB+W0Dpw2o@public.gmane.org) said: 
>> While testing in our production environment, I discovered
>> that the time between interface configuration and trying to mount the nfs 
>> root is too short if the dhcp-server provides an mtu. Having to set a mtu 
>> means that netroot
>> executes 'ip link down', set's the mtu and the does
>> 'ip link up' again.
>>
>> This results in the mount error "No route to host", because
>> the network equipment has not had enough time to "shake hands". The patch 
>> below adds a sleep 1 after interface configuration, to give the equipment 
>> enough time.
> 
> Depending on your switch configuration, the time for the switch to start
> passing traffic again on link bounce couuld be anywhere from 0.1 seconds
> to 20 seconds...

Quite correct. Up until now I've fared well with adding 1-5 second 
sleeps to my stuff. But yes it could even take longer. Any ideas how to 
solve this better?

Regards,
Philippe
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Bill Nottingham]

Seewer Philippe (philippe.seewer-omB+W0Dpw2o@public.gmane.org) said: 
>> Depending on your switch configuration, the time for the switch to start
>> passing traffic again on link bounce couuld be anywhere from 0.1 seconds
>> to 20 seconds...
>
> Quite correct. Up until now I've fared well with adding 1-5 second  
> sleeps to my stuff. But yes it could even take longer. Any ideas how to  
> solve this better?

Post-boot, we (in Fedora) we check every half second until a configurable
delay period (defualt: five seconds) has elapsed; if we don't have a link
by then, we punt. Of course, you probably don't want configuration
parameters in the initramfs.

Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Warren Togami]

On 06/19/2009 10:51 AM, Seewer Philippe wrote:
>
>
> Bill Nottingham wrote:
>> Seewer Philippe (philippe.seewer-omB+W0Dpw2o@public.gmane.org) said:
>>> While testing in our production environment, I discovered
>>> that the time between interface configuration and trying to mount the
>>> nfs root is too short if the dhcp-server provides an mtu. Having to
>>> set a mtu means that netroot
>>> executes 'ip link down', set's the mtu and the does
>>> 'ip link up' again.
>>>
>>> This results in the mount error "No route to host", because
>>> the network equipment has not had enough time to "shake hands". The
>>> patch below adds a sleep 1 after interface configuration, to give the
>>> equipment enough time.
>>
>> Depending on your switch configuration, the time for the switch to start
>> passing traffic again on link bounce couuld be anywhere from 0.1 seconds
>> to 20 seconds...
>
> Quite correct. Up until now I've fared well with adding 1-5 second
> sleeps to my stuff. But yes it could even take longer. Any ideas how to
> solve this better?

Any way to explicitly detect when the link is back?

Warren
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Victor Lowther]

On Jun 19, 2009, at 9:51 AM, Seewer Philippe <philippe.seewer-omB+W0Dpw2o@public.gmane.org>  
wrote:

>
>
> Bill Nottingham wrote:
>> Seewer Philippe (philippe.seewer-omB+W0Dpw2o@public.gmane.org) said:
>>> While testing in our production environment, I discovered
>>> that the time between interface configuration and trying to mount  
>>> the nfs root is too short if the dhcp-server provides an mtu.  
>>> Having to set a mtu means that netroot
>>> executes 'ip link down', set's the mtu and the does
>>> 'ip link up' again.
>>>
>>> This results in the mount error "No route to host", because
>>> the network equipment has not had enough time to "shake hands".  
>>> The patch below adds a sleep 1 after interface configuration, to  
>>> give the equipment enough time.
>> Depending on your switch configuration, the time for the switch to  
>> start
>> passing traffic again on link bounce couuld be anywhere from 0.1  
>> seconds
>> to 20 seconds...
>
> Quite correct. Up until now I've fared well with adding 1-5 second  
> sleeps to my stuff. But yes it could even take longer. Any ideas how  
> to solve this better?
>

Disable spanning tree on all edge switch ports. :)

Or you can include ethtool and check every half second until a timeout  
to see if the link is back.


> Regards,
> Philippe
> --
> To unsubscribe from this list: send the line "unsubscribe initramfs"  
> in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[David Dillow]

On Fri, 2009-06-19 at 11:08 -0500, Victor Lowther wrote:
> Or you can include ethtool and check every half second until a timeout  
> to see if the link is back.

With most switches, you'll get a link back almost immediately, but it
will be in blocking mode and won't transmit your packets for 20-30
seconds.
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Warren Togami]

On 06/19/2009 12:19 PM, David Dillow wrote:
> On Fri, 2009-06-19 at 11:08 -0500, Victor Lowther wrote:
>> Or you can include ethtool and check every half second until a timeout
>> to see if the link is back.
>
> With most switches, you'll get a link back almost immediately, but it
> will be in blocking mode and won't transmit your packets for 20-30
> seconds.

Our test suite can simulate this by using attaching qemu's -net tap 
device to a bridge to talk to the other virtual machine instead of using 
-net socket mcast mode.

qemu -net 
tap[,vlan=n][,name=name][,fd=h][,ifname=name][,script=file][,downscript=dfile]

script=can be a script that tells it to attach to a particular bridge.

#!/bin/sh
export PATH=/sbin:/usr/sbin:$PATH
BRIDGEDEV=br0
ifconfig $1 0.0.0.0 up
brctl addif ${BRIDGEDEV} $1

brctl setfd <bridge> <time> sets the bridge’s ’bridge forward delay’ to 
<time> seconds.

Then you can use brctl to make the bridge's forward delay super long.

Warren Togami
wtogami-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Hannes Reinecke]

David Dillow wrote:
> On Fri, 2009-06-19 at 11:08 -0500, Victor Lowther wrote:
>> Or you can include ethtool and check every half second until a timeout  
>> to see if the link is back.
> 
> With most switches, you'll get a link back almost immediately, but it
> will be in blocking mode and won't transmit your packets for 20-30
> seconds.

<And a huge grin spreads over my face>
Finally. I'm not alone anymore:-)
(I have been fighting the STP issue for years now.)

No, no, no. You don't want to add 'just' a sleep here because things
don't work as expected. Certainly not in this case.

The problem here is that most programs don't / can't distinguish
between an EHOSTUNREACH during startup (where we should retry as
the link / connection might not be up yet) and an EHOSTUNREACH
during normal operation (where we should return the error as this
is a genuine error condition).

The only safe way here is to send a ping to the host port we're
trying to connect to, and limit this with a timeout.
And start up the command once the ping succeeds.
Otherwise timeout and start error recovery.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare-l3A5Bk7waGM@public.gmane.org			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[David Dillow]

On Mon, 2009-06-22 at 09:50 +0200, Hannes Reinecke wrote:
> David Dillow wrote:
> > On Fri, 2009-06-19 at 11:08 -0500, Victor Lowther wrote:
> >> Or you can include ethtool and check every half second until a timeout  
> >> to see if the link is back.
> > 
> > With most switches, you'll get a link back almost immediately, but it
> > will be in blocking mode and won't transmit your packets for 20-30
> > seconds.
> 
> <And a huge grin spreads over my face>
> Finally. I'm not alone anymore:-)
> (I have been fighting the STP issue for years now.)

Yep, portfast is your friend if you have any control over your
network...

> No, no, no. You don't want to add 'just' a sleep here because things
> don't work as expected. Certainly not in this case.

Certainly we don't want to sleep if there is no reason to... why slow
down the boot if the network admin has the switch nicely configured for
us?

It may just be a matter of letting DHCP retry a sufficiency long time
before we look at the results and try to boot. As long as we don't down
the link after getting our information, the mount/login to the root
device should proceed fairly quickly. If we do down the link, the
blocking mode timer starts again and our very quick boot takes at least
40-60 seconds.

So, should we give dhclient 45-60 seconds to do its thing, and the admin
can use ip= lines to disable interfaces we know aren't going to get a
response?

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Seewer Philippe]

David Dillow wrote:
> On Mon, 2009-06-22 at 09:50 +0200, Hannes Reinecke wrote:
>> David Dillow wrote:
>>> On Fri, 2009-06-19 at 11:08 -0500, Victor Lowther wrote:
>>>> Or you can include ethtool and check every half second until a timeout  
>>>> to see if the link is back.
>>> With most switches, you'll get a link back almost immediately, but it
>>> will be in blocking mode and won't transmit your packets for 20-30
>>> seconds.
>> <And a huge grin spreads over my face>
>> Finally. I'm not alone anymore:-)
>> (I have been fighting the STP issue for years now.)
> 
> Yep, portfast is your friend if you have any control over your
> network...
> 
>> No, no, no. You don't want to add 'just' a sleep here because things
>> don't work as expected. Certainly not in this case.
> 
> Certainly we don't want to sleep if there is no reason to... why slow
> down the boot if the network admin has the switch nicely configured for
> us?

What if he hasn't?

> It may just be a matter of letting DHCP retry a sufficiency long time
> before we look at the results and try to boot. As long as we don't down
> the link after getting our information, the mount/login to the root
> device should proceed fairly quickly. If we do down the link, the
> blocking mode timer starts again and our very quick boot takes at least
> 40-60 seconds.

DHCP timeout already is 60 seconds. That's dhclient's default if you
don't provide other options.

The problem only manifests itself it we have to correct the mtu after dhcp,
which currently set's the interface down and up (paranoia I guess), or if
we do static configuration.

For dhcp the solution could be easy: Either we ignore the mtu or don't 
down/up the interface to set it.

> So, should we give dhclient 45-60 seconds to do its thing, and the admin
> can use ip= lines to disable interfaces we know aren't going to get a
> response?

Disabling interfaces is not needed I think. If you know which interfaces
are used/not used, you just tell dracut which interface(s) to use. All
other interfaces will be ignored.

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[David Dillow]

On Mon, 2009-06-22 at 18:17 +0200, Seewer Philippe wrote:
> David Dillow wrote:
> > Certainly we don't want to sleep if there is no reason to... why slow
> > down the boot if the network admin has the switch nicely configured for
> > us?
> 
> What if he hasn't?

That's where this part comes in:

> > It may just be a matter of letting DHCP retry a sufficiency long time
> > before we look at the results and try to boot. As long as we don't down
> > the link after getting our information, the mount/login to the root
> > device should proceed fairly quickly. If we do down the link, the
> > blocking mode timer starts again and our very quick boot takes at least
> > 40-60 seconds.
> 
> DHCP timeout already is 60 seconds. That's dhclient's default if you
> don't provide other options.

Ok, then we just need to set the udev settle time to 60 seconds in the
event of a netroot and avoid downing the link wherever possible. Given
that timeout, the only problem cases are static configs and MTU changes
on devices that cannot do that while UP'd.

> The problem only manifests itself it we have to correct the mtu after dhcp,
> which currently set's the interface down and up (paranoia I guess), or if
> we do static configuration.
>
> For dhcp the solution could be easy: Either we ignore the mtu or don't 
> down/up the interface to set it.

We shouldn't ignore the MTU, and we generally have to down/up the
interface to set it. Certainly, we should try to set it without downing
the interface, and fall back to the down/set/up sequence if that fails.
That would avoid further delay when good hardware/drivers allows it.

In the event of static config, or needing to down/up, I think it makes
sense to try to ping the server for a period of time and see if we are
able to communicate, before we try to get our root. You had even
suggested something similar some time back with your original network
scripts.

But that's not a 'sleep 60' just because we may happen be on a network
with a long blocking state. Perhaps I misread what we're talking about
and we're on the same page.

> > So, should we give dhclient 45-60 seconds to do its thing, and the admin
> > can use ip= lines to disable interfaces we know aren't going to get a
> > response?
> 
> Disabling interfaces is not needed I think. If you know which interfaces
> are used/not used, you just tell dracut which interface(s) to use. All
> other interfaces will be ignored.

That's what I meant by disabling.
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Seewer Philippe]

David Dillow wrote:
> On Mon, 2009-06-22 at 18:17 +0200, Seewer Philippe wrote:
>> David Dillow wrote:
>>> Certainly we don't want to sleep if there is no reason to... why slow
>>> down the boot if the network admin has the switch nicely configured for
>>> us?
>> What if he hasn't?
> 
> That's where this part comes in:
> 
>>> It may just be a matter of letting DHCP retry a sufficiency long time
>>> before we look at the results and try to boot. As long as we don't down
>>> the link after getting our information, the mount/login to the root
>>> device should proceed fairly quickly. If we do down the link, the
>>> blocking mode timer starts again and our very quick boot takes at least
>>> 40-60 seconds.
>> DHCP timeout already is 60 seconds. That's dhclient's default if you
>> don't provide other options.
> 
> Ok, then we just need to set the udev settle time to 60 seconds in the
> event of a netroot and avoid downing the link wherever possible. Given
> that timeout, the only problem cases are static configs and MTU changes
> on devices that cannot do that while UP'd.

I'd prefer not use an absolute settle time but instead "test/wait" the
queue. But that's a minor detail.

> 
>> The problem only manifests itself it we have to correct the mtu after dhcp,
>> which currently set's the interface down and up (paranoia I guess), or if
>> we do static configuration.
>>
>> For dhcp the solution could be easy: Either we ignore the mtu or don't 
>> down/up the interface to set it.
> 
> We shouldn't ignore the MTU, and we generally have to down/up the
> interface to set it. Certainly, we should try to set it without downing
> the interface, and fall back to the down/set/up sequence if that fails.
> That would avoid further delay when good hardware/drivers allows it.

Hmmm... I'm not sure that a failing 'ip set eth0 mtu xy' is an indicator
that we have to down/set mtu/up the interface. But to be honest, the 
hardware/driver combinations I'm able to test on seem to be working 
without down/up.

Are there any other methods to decide if we can set the mtu on the fly? 

> In the event of static config, or needing to down/up, I think it makes
> sense to try to ping the server for a period of time and see if we are
> able to communicate, before we try to get our root. You had even
> suggested something similar some time back with your original network
> scripts.
> 
> But that's not a 'sleep 60' just because we may happen be on a network
> with a long blocking state. Perhaps I misread what we're talking about
> and we're on the same page.

Actually, if I'm not mistaken STP can have a maximum timeout of 90
seconds. Add some paranoia and you're at 100. :-)

But yes, sleep 60 is a really bad idea for impatient people like you or me.

On the other hand, I'd really prefer other solutions to "ping" (if there
are any). Having used it myself doesn't mean I like it. First of all,
needing another utility enlarges the initrd even more, there's a possible
risk of having paranoid firewalls and we'd have to check the effective
service-port which isn't possible with standard iputils-ping.

Regards,
Philippe
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[David Dillow]

On Mon, 2009-06-22 at 20:47 +0200, Seewer Philippe wrote:
> David Dillow wrote:
> > Ok, then we just need to set the udev settle time to 60 seconds in the
> > event of a netroot and avoid downing the link wherever possible. Given
> > that timeout, the only problem cases are static configs and MTU changes
> > on devices that cannot do that while UP'd.
> 
> I'd prefer not use an absolute settle time but instead "test/wait" the
> queue. But that's a minor detail.

True, I'm just thinking that older udevs probably don't give us much
options here.

As an aside, we should keep in mind that we shouldn't wait for all of
the disks to become available if we don't have to -- I've got a few
systems with quite a few LUNs to scan, but we only need one for the
root. If it can find it and continue, then the rest can be done in
parallel later.

> Hmmm... I'm not sure that a failing 'ip set eth0 mtu xy' is an indicator
> that we have to down/set mtu/up the interface. But to be honest, the 
> hardware/driver combinations I'm able to test on seem to be working 
> without down/up.
> 
> Are there any other methods to decide if we can set the mtu on the fly?

Only by doing it. If it fails, it may be an indication that the MTU is
invalid for the device, or that it cannot be changed while the device is
up. If it is invalid, it fails then as well.

This method is a compromise; those with good hardware/drivers can use
large MTUs without taking a penalty, and it still works for those less
fortunate. Only the guys trying to set an invalid MTU pay a potentially
needless time penalty, but I'm OK with that. :)

> Actually, if I'm not mistaken STP can have a maximum timeout of 90
> seconds. Add some paranoia and you're at 100. :-)
> 
> But yes, sleep 60 is a really bad idea for impatient people like you or me.
> 
> On the other hand, I'd really prefer other solutions to "ping" (if there
> are any). Having used it myself doesn't mean I like it. First of all,
> needing another utility enlarges the initrd even more, there's a possible
> risk of having paranoid firewalls and we'd have to check the effective
> service-port which isn't possible with standard iputils-ping.

To check any of these, you'll need a separate utility. bash
has /dev/tcp/IP/port redirection, but I doubt dash supports that. And
even then you'll have some klunky trickery in the shell script to add a
timeout to it. Nor will you be able to work with anything over UDP.

I think that checking an IP is reachable is sufficient to verify that
traffic is flowing -- and we have choices there. We can use the server
IP if it is local, and the gateway IP if not. If we're concerned about
firewalls blocking ICMP, we could use 'arping' to send ARP requests. If
there's a way to kick off an arp request with 'ip' or something in /proc
or /sys, that'd avoid additional bloat, but I'm not aware of one.
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Seewer Philippe]

David Dillow wrote:
[snip]
> To check any of these, you'll need a separate utility. bash
> has /dev/tcp/IP/port redirection, but I doubt dash supports that. And
> even then you'll have some klunky trickery in the shell script to add a
> timeout to it. Nor will you be able to work with anything over UDP.
> 
> I think that checking an IP is reachable is sufficient to verify that
> traffic is flowing -- and we have choices there. We can use the server
> IP if it is local, and the gateway IP if not. If we're concerned about
> firewalls blocking ICMP, we could use 'arping' to send ARP requests. If
> there's a way to kick off an arp request with 'ip' or something in /proc
> or /sys, that'd avoid additional bloat, but I'm not aware of one.

Ha! You're brilliant. I completely forgot about ARP.

Including arping isn't such a bad idea. First of all, I tried using
arping to solve the problem some years ago but busybox' arping didn't
have the options required, iputils' arping does. Second, we should
include it anyway since doing DAD in dhcp-script isn't such a bad idea.

Now, I'd use the gateway for detection if possible. But if it isn't 
available, how to guess the server? That logic currently resides in
the root-handlers/-parsers and I'd prefer not to have the root-handler
do the checking.

Regards,
Philippe
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[David Dillow]

On Tue, 2009-06-23 at 09:31 +0200, Seewer Philippe wrote:
> David Dillow wrote:
> [snip]
> > To check any of these, you'll need a separate utility. bash
> > has /dev/tcp/IP/port redirection, but I doubt dash supports that. And
> > even then you'll have some klunky trickery in the shell script to add a
> > timeout to it. Nor will you be able to work with anything over UDP.
> > 
> > I think that checking an IP is reachable is sufficient to verify that
> > traffic is flowing -- and we have choices there. We can use the server
> > IP if it is local, and the gateway IP if not. If we're concerned about
> > firewalls blocking ICMP, we could use 'arping' to send ARP requests. If
> > there's a way to kick off an arp request with 'ip' or something in /proc
> > or /sys, that'd avoid additional bloat, but I'm not aware of one.
> 
> Ha! You're brilliant. I completely forgot about ARP.
> 
> Including arping isn't such a bad idea. First of all, I tried using
> arping to solve the problem some years ago but busybox' arping didn't
> have the options required, iputils' arping does. Second, we should
> include it anyway since doing DAD in dhcp-script isn't such a bad idea.
> 
> Now, I'd use the gateway for detection if possible. But if it isn't 
> available, how to guess the server? That logic currently resides in
> the root-handlers/-parsers and I'd prefer not to have the root-handler
> do the checking.

I've been thinking about this a bit, and I'd like to split the duties up
a bit, as I'd like to know the server IPs needed for other reasons. My
proposed split is that legacy handling -- ie nfsroot=blah to
netroot=nfs:blah conversion -- is done as it is now, and have the root
handler do syntax validation and stop, spit out the server IP(s), and do
the mount, depending on options handed to it (--validate, --servers, and
no options, respectively).

The sourced scripts could just call into their handler to do the final
validation, and netroot would get rid of its loop as it can call the
handler directly since it has the protocol field.

The only issue is handling the legacy NFS root-path options, everything
else is expected to have nfs:/nfs4:/nbd:/iscsi: or what not. I don't
think it would be the end of the world if the that handling lived in
netroot itself, as it will be the only root-path legacy handler we have.
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Seewer Philippe]

David Dillow wrote:
[snip]
> I've been thinking about this a bit, and I'd like to split the duties up
> a bit, as I'd like to know the server IPs needed for other reasons. My
> proposed split is that legacy handling -- ie nfsroot=blah to
> netroot=nfs:blah conversion -- is done as it is now, and have the root
> handler do syntax validation and stop, spit out the server IP(s), and do
> the mount, depending on options handed to it (--validate, --servers, and
> no options, respectively).

Hmmm... just so that I understand you correctly. Are you proposing to 
move the cmdline-parsers back into the root handlers and add arguments? 
If not maybe you could a more detailed example of your idea?

> The sourced scripts could just call into their handler to do the final
> validation, and netroot would get rid of its loop as it can call the
> handler directly since it has the protocol field.

Uhhh... which loop are you talking about?

> The only issue is handling the legacy NFS root-path options, everything
> else is expected to have nfs:/nfs4:/nbd:/iscsi: or what not. I don't
> think it would be the end of the world if the that handling lived in
> netroot itself, as it will be the only root-path legacy handler we have.

If possible, netroot should remain as generic as possible. You never 
know what the future holds.
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[David Dillow]

On Wed, 2009-06-24 at 08:56 +0200, Seewer Philippe wrote:
> David Dillow wrote:
> [snip]
> > I've been thinking about this a bit, and I'd like to split the duties up
> > a bit, as I'd like to know the server IPs needed for other reasons. My
> > proposed split is that legacy handling -- ie nfsroot=blah to
> > netroot=nfs:blah conversion -- is done as it is now, and have the root
> > handler do syntax validation and stop, spit out the server IP(s), and do
> > the mount, depending on options handed to it (--validate, --servers, and
> > no options, respectively).
> 
> Hmmm... just so that I understand you correctly. Are you proposing to 
> move the cmdline-parsers back into the root handlers and add arguments? 
> If not maybe you could a more detailed example of your idea?

Yes, something like that.

parse-nfsroot.sh, parse-nbdroot.sh, and parse-iscsiroot.sh would only
exist to convert legacy command line arguments to the
netroot={nfs,nbd,iscsi}: format. They would not validate the netroot=
format.

As part of the command line parsing, the appropriate handler would be
called for netroot to check its validity, such as:

/cmdline/98-check-netroot.sh:

if [ -n "$netroot" ]; then
    [ -e /sbin/${netroot%%:*}-root ] || die "No root handler for protocol ${netroot%%:*}"
    /sbin/${netroot%%:*}-root --validate "$netroot"
    case $? in
	0) rootok=1 ;; # All's good
	1) DHCPORSERVER=1 ;; # No server specified
	*) die "Invalid argument to root handler ${netroot%%:*}" ;;
    esac
fi

Checking the validity of the DHCP root-option becomes (modulo NFS legacy
support):

[ -e /sbin/${rootpath%%:*}-root ] || die "No root handler for protocol ${rootpath%%:*}"
/sbin/${rootpath%%:*}-root --validate "$rootpath" ||
	die "Invalid DHCP root-option for ${rootpath%%:*}"


Later, we can decide which IP to arping:

server=$(/sbin/${netroot%%:*}-root --server "$netroot")
destip=$gateway
is_ip_local $server $myip $netmask && destip=$server

while ! arping ... $server; do
    # count here to break out, etc.
    sleep 1
done


The variable names will likely change a bit, and the messages will need
to be massaged, but we have a need to decode the netroot= arguments
three times -- once for validation, once for extracting the server, and
once for the actual mounting. It makes sense to have the same parser for
all three uses. I think it would also clean things up to separate the
legacy handling from the supported mechanisms.

I haven't started coding this up; I wanted to try to get some consensus
first. NFSv4 would get its own handler, which I think makes sense --
I'll probably break it out into it's own module anyway.

> > The sourced scripts could just call into their handler to do the final
> > validation, and netroot would get rid of its loop as it can call the
> > handler directly since it has the protocol field.
> 
> Uhhh... which loop are you talking about?

/sbin/netroot:
# Source netroot hooks before we start the handler
source_all netroot

That would become the validity check above.

> > The only issue is handling the legacy NFS root-path options, everything
> > else is expected to have nfs:/nfs4:/nbd:/iscsi: or what not. I don't
> > think it would be the end of the world if the that handling lived in
> > netroot itself, as it will be the only root-path legacy handler we have.
> 
> If possible, netroot should remain as generic as possible. You never 
> know what the future holds.

As a general design principle, I tend to agree with you. However, the
only root-path we support that doesn't match the form protocol:arguments
is the [serverip:]/path syntax of NFS. I think it makes sense to either
pull that into netroot.

Of course, I just thought of an alternative solution -- maybe we could
pass both netroot and root-path in to the root handler, and it can do
the right thing. The initial cmdline parse would pass in "" for
root-path, or perhaps find another way to indicate we don't have the
root-path yet.

It's a wart, but I think we can get it cleaned up.

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[David Dillow]

[ Hannes, Victor, Bill: I'm loathe to cut cc's normally, but if this
discussion doesn't interest you, let me know privately and I'll trim you
from my future messages on it. ]

On Wed, 2009-06-24 at 23:30 -0400, David Dillow wrote:
> /sbin/netroot:
> # Source netroot hooks before we start the handler
> source_all netroot
> 
> That would become the validity check above.

Hmm, that source_all is vestigial, I think.

This is the loop I'm thinking of:

# Figure out the handler for root=dhcp by recalling all netroot cmdline 
# handlers
if [ "$netroot" = "dhcp" ] ; then
[snip]
    for f in ./cmdline/90*.sh; do
        [ -f "$f" ] && . "$f";
    done
fi

This could go away modulo the NFS issue I mentioned.


--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[David Dillow]

On Mon, 2009-06-22 at 18:17 +0200, Seewer Philippe wrote:
> David Dillow wrote:
> > Certainly we don't want to sleep if there is no reason to... why slow
> > down the boot if the network admin has the switch nicely configured for
> > us?
> 
> What if he hasn't?

Just to be sure I'm coming across clearly -- I want us to work properly
if the admin hasn't configured the network nicely for us, but I do not
want us to slow down at all when he has.

My networks will be configured correctly, and I'm an impatient bastard.
I don't want to wait if I don't have to. But not everyone will have the
luxury of control of their network.
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Bogdan Costescu]

On Mon, 22 Jun 2009, Seewer Philippe wrote:

> DHCP timeout already is 60 seconds. That's dhclient's default if you 
> don't provide other options.

On a network where I don't have control over the switch (some Cisco 
model) and people connect laptops, I've seen random cases of DHCP 
failing. That's because the STP can take 60 seconds and DHCP gives up 
just when the link becomes usable... For the workstations connected on 
another network where STP is also enabled, I've experimented with 
various solutions, the one using ping to the gateway (with a timeout 
of 100 seconds) worked best and was simplest for me to maintain.

> The problem only manifests itself it we have to correct the mtu 
> after dhcp, which currently set's the interface down and up 
> (paranoia I guess), or if we do static configuration.

No, it's not paranoia. Maybe things have changed in the meantime, but 
some years ago when I was involved in the Linux network stack some 
drivers couldn't change the MTU on the fly due to internal 
limitations. The only possibility to change it would be to bring the 
interface down then up. Not all drivers are limited this way, though. 
Maybe it's possible to try to set the MTU, check whether it has 
succeeded and only if it failed bring the interface down then up; 
don't know anymore if a failed MTU setting would make ifconfig return 
something... maybe just reading the MTU again and compare with desired 
would work too.

-- 
Bogdan Costescu

IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8240, Fax: +49 6221 54 8850
E-mail: bogdan.costescu-hEciA7+sKtudPOQpRHQ53DeJuz7u0hKX@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFT PATCH] Delay netroot mounting by 1 second

[Seewer Philippe]

Hannes Reinecke wrote:
> David Dillow wrote:
>> On Fri, 2009-06-19 at 11:08 -0500, Victor Lowther wrote:
>>> Or you can include ethtool and check every half second until a timeout  
>>> to see if the link is back.
>> With most switches, you'll get a link back almost immediately, but it
>> will be in blocking mode and won't transmit your packets for 20-30
>> seconds.
> 
> <And a huge grin spreads over my face>
> Finally. I'm not alone anymore:-)
> (I have been fighting the STP issue for years now.)

You can say that loud. Over the years I've tried various solutions,
from sleeping, to catching states via netlink, to guessing STP
timeouts (on newer switches probability of RSTP-like behaviour is
95%) to your mentioned ping... 

> No, no, no. You don't want to add 'just' a sleep here because things
> don't work as expected. Certainly not in this case.

Sleep isn't a good solution, I agree there. But it's simple.
Hence the RFT to see if someone can construct a real life case
where it actually isn't enough.

> The problem here is that most programs don't / can't distinguish
> between an EHOSTUNREACH during startup (where we should retry as
> the link / connection might not be up yet) and an EHOSTUNREACH
> during normal operation (where we should return the error as this
> is a genuine error condition).
> 
> The only safe way here is to send a ping to the host port we're
> trying to connect to, and limit this with a timeout.
> And start up the command once the ping succeeds.
> Otherwise timeout and start error recovery.

Yes. If we know that 1) We're using the correct interface and
2) provided server data is correct. Otherwise we just unnecessary
lengthen to boot process.

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html