Boot time: Kernel start parallelization issue?

Boot time: Kernel start parallelization issue?

[Dirk Behme]

There are some reports [1] [2] looking at the boot time of embedded 
(ARM?) systems using initcall debug [3]. Both reports seem to show 
that they have issues with the start up sequence of the kernel being 
completely single-threaded. In [2] Greg mentions that on a x86 box 
multi-threads are happening and that there he doesn't see this issue. 
On the other hand, both reports mention Arjan's async initcall patches 
[4] to help against the issue. I.e. introducing some parallelization 
(on ARM) does help, too.

With this, I wonder

- if anybody faces similar issues with single-threaded only kernel 
start on embedded (ARM?) systems? Or if this is known? Or if there are 
fixes for this?

- if we somehow should try to 're-activate' Arjan's async initcall 
patches?

Any ideas?

Many thanks and best regards

Dirk

P.S.: I couldn't find a mail address of Alex Gonzalez, the author of 
[1], to put him into CC. If anybody knows him, please feel free to add 
him to this thread. Thanks!

[1] 
http://www.lindusembedded.com/blog/2010/06/02/measuring-the-boot-time-of-an-embedded-linux-device/

[2] http://thread.gmane.org/gmane.linux.usb.general/41181 (mainly the 
last mail of this thread: 
http://article.gmane.org/gmane.linux.usb.general/41619)

[3] http://www.elinux.org/Initcall_Debug

[4] http://lwn.net/Articles/299591/

Re: Boot time: Kernel start parallelization issue?

[Arjan van de Ven]

On 1/15/2011 12:40 AM, Dirk Behme wrote:
>
> There are some reports [1] [2] looking at the boot time of embedded 
> (ARM?) systems using initcall debug [3]. Both reports seem to show 
> that they have issues with the start up sequence of the kernel being 
> completely single-threaded. In [2] Greg mentions that on a x86 box 
> multi-threads are happening and that there he doesn't see this issue. 
> On the other hand, both reports mention Arjan's async initcall patches 
> [4] to help against the issue. I.e. introducing some parallelization 
> (on ARM) does help, too.
>
> With this, I wonder
>
> - if anybody faces similar issues with single-threaded only kernel 
> start on embedded (ARM?) systems? Or if this is known? Or if there are 
> fixes for this?
>
> - if we somehow should try to 're-activate' Arjan's async initcall 
> patches?

those patches are obsoleted by the merged async_schedule() calls.

what kernel are you seeing issues on?
if it's something ancient, you should update first.

Re: Boot time: Kernel start parallelization issue?

[Dirk Behme]

On 15.01.2011 16:41, Arjan van de Ven wrote:
> On 1/15/2011 12:40 AM, Dirk Behme wrote:
>>
>> There are some reports [1] [2] looking at the boot time of embedded
>> (ARM?) systems using initcall debug [3]. Both reports seem to show
>> that they have issues with the start up sequence of the kernel being
>> completely single-threaded. In [2] Greg mentions that on a x86 box
>> multi-threads are happening and that there he doesn't see this
>> issue. On the other hand, both reports mention Arjan's async
>> initcall patches [4] to help against the issue. I.e. introducing
>> some parallelization (on ARM) does help, too.
>>
>> With this, I wonder
>>
>> - if anybody faces similar issues with single-threaded only kernel
>> start on embedded (ARM?) systems? Or if this is known? Or if there
>> are fixes for this?
>>
>> - if we somehow should try to 're-activate' Arjan's async initcall
>> patches?
>
> those patches are obsoleted by the merged async_schedule() calls.

Any link (patch name, git link) to be able to check for 
async_schedule() calls?

> what kernel are you seeing issues on?

[1] talks about 2.6.28, [2] talks about 2.6.34.

Best regards

Dirk

[1] 
http://www.lindusembedded.com/blog/2010/06/02/measuring-the-boot-time-of-an-embedded-linux-device/

[2] http://thread.gmane.org/gmane.linux.usb.general/41181 (mainly the 
last mail of this thread: 
http://article.gmane.org/gmane.linux.usb.general/41619)

Re: Boot time: Kernel start parallelization issue?

[Dirk Behme]

On 15.01.2011 17:18, Dirk Behme wrote:
> On 15.01.2011 16:41, Arjan van de Ven wrote:
>> On 1/15/2011 12:40 AM, Dirk Behme wrote:
>>>
>>> There are some reports [1] [2] looking at the boot time of embedded
>>> (ARM?) systems using initcall debug [3]. Both reports seem to show
>>> that they have issues with the start up sequence of the kernel being
>>> completely single-threaded. In [2] Greg mentions that on a x86 box
>>> multi-threads are happening and that there he doesn't see this
>>> issue. On the other hand, both reports mention Arjan's async
>>> initcall patches [4] to help against the issue. I.e. introducing
>>> some parallelization (on ARM) does help, too.
>>>
>>> With this, I wonder
>>>
>>> - if anybody faces similar issues with single-threaded only kernel
>>> start on embedded (ARM?) systems? Or if this is known? Or if there
>>> are fixes for this?
>>>
>>> - if we somehow should try to 're-activate' Arjan's async initcall
>>> patches?
>>
>> those patches are obsoleted by the merged async_schedule() calls.
>
> Any link (patch name, git link) to be able to check for
> async_schedule() calls?

Do you talk about

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=22a9d645677feefd402befd02edd59b122289ef1
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4ace92fc112c6069b4fcb95a31d3142d4a43ff2a
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=793180570ff2530d133343ceea85648de5f01b02
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=f29d3b23238e1955a8094e038c72546e99308e61

http://lwn.net/Articles/314808/

merged with 2.6.29?

With a recent .37 kernel the usage of async_schedule() seems to be 
implemented for

./drivers/acpi/battery.c
./drivers/s390/block/dasd.c
./drivers/base/power/main.c
./drivers/ata/libata-core.c
./drivers/scsi/sd.c
./drivers/md/raid5.c

With this, on an embedded system using none of this, the completely 
single-threaded start up reported in [2] seems to be reasonable, then?

This would mean that to improve the issues reported in [2], 
async_schedule() has to be implemented for the sub systems used, 
there, too? I.e. the USB init?

>> what kernel are you seeing issues on?
>
> [1] talks about 2.6.28, [2] talks about 2.6.34.

Thanks

Dirk

> [1]
> http://www.lindusembedded.com/blog/2010/06/02/measuring-the-boot-time-of-an-embedded-linux-device/
>
>
> [2] http://thread.gmane.org/gmane.linux.usb.general/41181 (mainly the
> last mail of this thread:
> http://article.gmane.org/gmane.linux.usb.general/41619)

Re: Boot time: Kernel start parallelization issue?

[Arjan van de Ven]

On 1/15/2011 11:33 PM, Dirk Behme wrote:
>
> Do you talk about
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=22a9d645677feefd402befd02edd59b122289ef1 
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4ace92fc112c6069b4fcb95a31d3142d4a43ff2a 
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=793180570ff2530d133343ceea85648de5f01b02 
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=f29d3b23238e1955a8094e038c72546e99308e61 
>
>
> http://lwn.net/Articles/314808/
>
> merged with 2.6.29?
>
> With a recent .37 kernel the usage of async_schedule() seems to be 
> implemented for

yep
>
> ./drivers/acpi/battery.c
> ./drivers/s390/block/dasd.c
> ./drivers/base/power/main.c
> ./drivers/ata/libata-core.c
> ./drivers/scsi/sd.c
> ./drivers/md/raid5.c
>
> With this, on an embedded system using none of this, the completely 
> single-threaded start up reported in [2] seems to be reasonable, then?

quite likely.

>
> This would mean that to improve the issues reported in [2], 
> async_schedule() has to be implemented for the sub systems used, 
> there, too? I.e. the USB init?

USB init is asynchronous internal to USB; this was one of the outcomes 
of a side discussion at the Kernel Summit a year and a half ago or so 
where Alan Stern visualized the problem... and solved it internal to USB.

Now, there is one 100 msec delay that is inherent to USB (mandatory 
delay required by the spec for voltage to stabilize) but that shouldn't 
be in the synchronous path anymore

>
>>> what kernel are you seeing issues on?
>>
>> [1] talks about 2.6.28, [2] talks about 2.6.34.

yeah 2.6.28 is a lost cause ;-)  2.6.34 should be not too bad.

Now, the one thing that's critical for all of this is that we need to do 
this data driven. Using async init is not always as easy as it sounds; 
there are device ordering things that need to be dealt with, and failure 
cases get much more complex as well.

So the first order of business ALWAYS is to get a bootgraph (from 
scripts/bootgraph.pl)... to see which things are big on the one hand, 
but also to see if there is gains to be had. There is absolutely no gain 
possible (in terms of making something async) if it is pretty much the 
last big thing in the boot that you are making asynchronous (since the 
end of kernel boot has to wait for it all anyway); there is only gains 
for async if the expensive operation, when made async, can run in 
parallel with something else expensive that comes later in the kernel init.

The bootgraph will show this opportunity / pitfall very clearly......