Re: new ipdelay= option for faster netboot

Re: new ipdelay= option for faster netboot

[David Miller]

From: Tim Bird <tim.bird@am.sony.com>
Date: Mon, 17 Aug 2009 15:35:01 -0700

> Tim Bird wrote:
>> See the definitions of CONF_PRE_OPEN and CON_POST_OPEN
>> in net/ipv4/ipconfig.c
>> 
>> They are set to ridiculously long values.  In my experience,
>> you can cut them down considerably with no dangerous side
>> effects (but I haven't asked the network guys about the
>> possible downsides).
> 
> It turns out that others have seen this delay.  Simon
> Arlott recently posted a patch to make the delay avoidable
> at boot time from the kernel command line.
> 
> See http://patchwork.kernel.org/patch/31678/

"Rediculiously long" is a relative term.

I have card/switch combinations that take up to 10 seconds to
negotiate a proper link.

So what's there now is actually a quite agressive setting.

And BTW, discussions about stuff like this belong on
netdev@vger.kernel.org, which has been added to the CC:

Re: new ipdelay= option for faster netboot

[Tim Bird]

David Miller wrote:
> From: Tim Bird <tim.bird@am.sony.com>
> Date: Mon, 17 Aug 2009 15:35:01 -0700
> 
>> Tim Bird wrote:
>>> See the definitions of CONF_PRE_OPEN and CON_POST_OPEN
>>> in net/ipv4/ipconfig.c
>>>
>>> They are set to ridiculously long values.  In my experience,
>>> you can cut them down considerably with no dangerous side
>>> effects (but I haven't asked the network guys about the
>>> possible downsides).
>> It turns out that others have seen this delay.  Simon
>> Arlott recently posted a patch to make the delay avoidable
>> at boot time from the kernel command line.
>>
>> See http://patchwork.kernel.org/patch/31678/
> 
> "Rediculiously long" is a relative term.
No offense intended.  I could have phrased this
better.  The delays were a few orders of
magnitude longer than apparently needed, on my
embedded test systems with ethernet.  I didn't
try eliminating them completely, as in the Arlott patch.

1.5 seconds is a long time for me.  My bootup time budget for
the kernel ranges from 0.5 to 3.0 seconds, depending on the
product.

> I have card/switch combinations that take up to 10 seconds to
> negotiate a proper link.

What types of delays are these timeouts supposed to
cover?  Networking delays or hardware bring-up delays?
(Or both)?  If for networking delays, is this for all
types of networks, or just some (e.g. ones that create
virtual circuits)?

I'm trying to get a sense for whether the card/switch
combinations that would take this long would be encountered
in the types of embedded devices I code for.  (TVs, camcorders,
etc.)

>
> So what's there now is actually a quite agressive setting.
> 
> And BTW, discussions about stuff like this belong on
> netdev@vger.kernel.org, which has been added to the CC:

I was going to wait to see if this solved Robert's
problem, before widening the discussion.  But I'm happy
to find out more about these delays now.

Thanks,
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: new ipdelay= option for faster netboot

[David Miller]

From: Tim Bird <tim.bird@am.sony.com>
Date: Mon, 17 Aug 2009 18:24:26 -0700

> David Miller wrote:
>> I have card/switch combinations that take up to 10 seconds to
>> negotiate a proper link.
> 
> What types of delays are these timeouts supposed to
> cover?

The problem is that if you don't first give at least some time for the
link to come up, the remaining time it takes the link to come up will
end up chewing into the actual bootp/dhcp protocol timeouts.  And
that's what we're trying to avoid.

Re: new ipdelay= option for faster netboot

[Rick Jones]

David Miller wrote:
> I have card/switch combinations that take up to 10 seconds to
> negotiate a proper link.

Gotta love it when things adhere to specs...

rick jones
has also experienced nic/whatnot combinations that are far from IEEE specs... :(

Re: new ipdelay= option for faster netboot

[Tim Bird]

David Miller wrote:
> From: Tim Bird <tim.bird@am.sony.com>
> Date: Mon, 17 Aug 2009 18:24:26 -0700
>
>> David Miller wrote:
>>> I have card/switch combinations that take up to 10 seconds to
>>> negotiate a proper link.
>> What types of delays are these timeouts supposed to
>> cover?
>
> The problem is that if you don't first give at least some time for the
> link to come up, the remaining time it takes the link to come up will
> end up chewing into the actual bootp/dhcp protocol timeouts.  And
> that's what we're trying to avoid.

What link?  I'm not that familiar with networking.

Assuming I'm using ethernet, what link needs to come up?
Is this something to do with power propagation to the
physical wire?  Is there some MAC layer negotiation
between the card and the switch?  Is it the time for
the switch to do speed detection?

And, can any of this be more accurately determined
or guessed-at with knowledge of the onboard hardware?
Or is it dependent on external conditions?

Where would be a good place to find out more about
startup delays for networking chips and/or protocols?

Our usual solution is to kick the can down the road
and let user-space initialize anything that takes a long
time,  while we do other stuff like focus the camera or display
the TV picture.  It would be good to learn more about
this.
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: new ipdelay= option for faster netboot

[David Miller]

From: Tim Bird <tim.bird@am.sony.com>
Date: Mon, 17 Aug 2009 18:40:48 -0700

> David Miller wrote:
>> From: Tim Bird <tim.bird@am.sony.com>
>> Date: Mon, 17 Aug 2009 18:24:26 -0700
>>
>>> David Miller wrote:
>>>> I have card/switch combinations that take up to 10 seconds to
>>>> negotiate a proper link.
>>> What types of delays are these timeouts supposed to
>>> cover?
>>
>> The problem is that if you don't first give at least some time for the
>> link to come up, the remaining time it takes the link to come up will
>> end up chewing into the actual bootp/dhcp protocol timeouts.  And
>> that's what we're trying to avoid.
> 
> What link?  I'm not that familiar with networking.

The speed and duplex settings which are negotiated or forced between
the ethernet card and whatever is at the other end of the cable.

> Assuming I'm using ethernet, what link needs to come up?

All modern ethernet cards do autonegotiation of link parameters
with whatever is at the other end of the ethernet cable.  Cards
created ages ago which only support 10MB half-duplex typically
do not support autonegotiation at all.

This autonegotiation works like a protocol where the two link partners
go back and forth trying to figure out the best speed and duplex
settings to use.  There are advertisements of link capabilities and
stuff like that.

It should happen almost instantaneously, but there are millions
upon millions of cruddy parts out there, and some of them take
a long time to go through this negotiation process.

Re: new ipdelay= option for faster netboot

[david]

On Mon, 17 Aug 2009, Rick Jones wrote:

> David Miller wrote:
>> I have card/switch combinations that take up to 10 seconds to
>> negotiate a proper link.
>
> Gotta love it when things adhere to specs...

the default on Cisco switches is to wait 30 seconds before fully enabling 
the port so that it can listen for spanning tree broadcasts.

for windows systems this doesn't cause any problems (they take long enough 
to boot), but for a tightly configured linux box it can be fully booted 
long before the switch decides to enable the port (for that matter, I have 
a bare-metal install process that takes about 90 seconds from hitting 
power, from a CD. I could probably speed things up by switching to USB 
boot media)

David Lang

Re: new ipdelay= option for faster netboot

[Willy Tarreau]

On Mon, Aug 17, 2009 at 07:45:33PM -0700, david@lang.hm wrote:
> On Mon, 17 Aug 2009, Rick Jones wrote:
> 
> >David Miller wrote:
> >>I have card/switch combinations that take up to 10 seconds to
> >>negotiate a proper link.
> >
> >Gotta love it when things adhere to specs...
> 
> the default on Cisco switches is to wait 30 seconds before fully enabling 
> the port so that it can listen for spanning tree broadcasts.

And this causes a lot of trouble in high availability environments,
because the link is up but unusable. So if you're using it as a primary
bond link you can lose connectivity for that time. Fortunately, you can
configure the port in "switchport mode access", "portfast" mode to avoid
this annoying delay.

Willy

Re: new ipdelay= option for faster netboot

[Denys Vlasenko]

On Tuesday 18 August 2009 03:27, David Miller wrote:
> From: Tim Bird <tim.bird@am.sony.com>
> Date: Mon, 17 Aug 2009 18:24:26 -0700
> 
> > David Miller wrote:
> >> I have card/switch combinations that take up to 10 seconds to
> >> negotiate a proper link.
> > 
> > What types of delays are these timeouts supposed to
> > cover?
> 
> The problem is that if you don't first give at least some time for the
> link to come up, the remaining time it takes the link to come up will
> end up chewing into the actual bootp/dhcp protocol timeouts.  And
> that's what we're trying to avoid.

But in this case, they assign a static IP. They do not use DHCP.
So they pay this time penalty even if they wouldn't use networking
until sometime later (or never).

Since DHCP and any other networking activity like TCP connects
accomodate packet loss, things should work even without any delay
in kernel IP config code. The delay will be just shifted to the
moment when first DHCP/TCP/whatever negotiation happens.

If dropping delays altogether sounds too big a change,
then it makes sense at least to allow people to tweak it with
ipdelay=TIME_IN_MS

-- 
vda

Re: new ipdelay= option for faster netboot

[David Miller]

From: Denys Vlasenko <vda.linux@googlemail.com>
Date: Tue, 18 Aug 2009 06:56:53 +0200

> Since DHCP and any other networking activity like TCP connects
> accomodate packet loss, things should work even without any delay
> in kernel IP config code. The delay will be just shifted to the
> moment when first DHCP/TCP/whatever negotiation happens.

Until the link is up, the packet scheduler just holds onto packets
and queued them up.

When the link comes up, this queue is released and the packets
sent out.

That's why it's beneficial to wait some time until the link
comes up before we start sending stuff out.  Because otherwise
any timeouts used will be inaccurate.

This could code be spiffed up to wait for the link up indication
on devices it cares about.  Feel free to code that up :-)

Re: new ipdelay= option for faster netboot

[Jamie Lokier]

Tim Bird wrote:
> David Miller wrote:
> > From: Tim Bird <tim.bird@am.sony.com>
> > Date: Mon, 17 Aug 2009 18:24:26 -0700
> >
> >> David Miller wrote:
> >>> I have card/switch combinations that take up to 10 seconds to
> >>> negotiate a proper link.
> >> What types of delays are these timeouts supposed to
> >> cover?
> >
> > The problem is that if you don't first give at least some time for the
> > link to come up, the remaining time it takes the link to come up will
> > end up chewing into the actual bootp/dhcp protocol timeouts.  And
> > that's what we're trying to avoid.
> 
> What link?  I'm not that familiar with networking.
> 
> Assuming I'm using ethernet, what link needs to come up?

When you plug an ethernet cable in, you may have noticed it takes a
short time before the signal light comes on.  That's negotiation time.
Some are slower than others, but none of them do it instantly.

> Is this something to do with power propagation to the
> physical wire?

Not really.

> Is there some MAC layer negotiation between the card and the switch?
> Is it the time for the switch to do speed detection?

Yes and yes.

> And, can any of this be more accurately determined
> or guessed-at with knowledge of the onboard hardware?
> Or is it dependent on external conditions?

It can be accurately determined with most cards (all modern ones)
because you get a notification when it's done, or you can poll the card.

That's why on the desktop it's able to detect when you plug in an
ethernet cable and start DHCP as soon as link negotiation is complete.

So the right thing to do, as David Miller suggested too, isn't a fixed
timeout.  It should wait for link state UP and then start DHCP
immediately.

-- Jamie