kernel/initrd loading delays

kernel/initrd loading delays

[Felix Miata]

On more and more installations since most distros have replaced mkinitrd with
dracut, on selecting a bootloader menu choice the "root
(hd0,...filesystemtype..[Linux-bzImage, setup, ... initrd /boot/initrd"
message stays on screen 40 seconds or more before the time-stamped boot
messages begin scrolling the screen. IIRC, these delays first began appearing
last January or February, and appear on multiple machines. All kernel and
initrd images are on smallish EXT3 or EXT4 (mostly the latter) filesystems
using 1024 blocksize on BIOS logical partitions on rotating rust (IIRC, all
with native 512b sector size, manufactured in 2011 or earlier). All
installations are configured with static IP. All have Plymouth either not
installed, or are booted with noplymouth included on kernel cmdline.

e.g, on my fastest system, host msi85, a 3.0 GHz dual core Haswell, total
time from boot stanza select on BIOS host msi85 to Rawhide 4.3.0-0.rc3.git4.1
(dracut initramfs.img 11,326,223) multi-user.target shell prompt ready is
about 80 seconds following that 40 second delay, with last time stamp showing
36.844484.

Same msi85 system booting LMDE 2 3.16.0-4 (non-dracut? initrd.img 27,362,964;
initramfs-tools 0.120) delays start of boot messages nearly 80 seconds,
reaching DM greeter ready nearly 140 seconds after stanza selection.

Same system booting openSUSE Leap 42.1 4.1.8 (dracut initrd 8,082,360)
exhibits no perceptible delay, reaching multi-user.target shell prompt in
under 47 seconds from boot stanza selection.

Same system booting openSUSE 13.2 3.16.7 (dracut initrd 5,351,996) delays
start of boot messages about 40 seconds, reaching multi-user.target bash
prompt about 90 seconds after stanza selection.

Same system booting Mageia 5 4.1.8 (dracut initrd.img 9,391,664) exhibits no
delay, reaching multi-user.target bash prompt in about 45 seconds.

Same system booting Wheezy 3.2.0 (non-dracut initrd.img 10,537,321) exhibits
no delay, reaching DM greeter in less than about 45 seconds.

Same system booting openSUSE 13.1 3.12.44 (non-dracut initrd 7,679,390)
exhibits no delay, reaching multi-user.target bash prompt in less than 45
seconds.

Same system booting Fedora 23 4.2.2 (dracut initramfs.img 11,255,214) reaches
multi-user.target shell prompt in about 75 seconds after unknown
kernel/initrd delay (puts display into sleep mode until boot messages appear).

Same system booting openSUSE Tumbleweed 4.2.1 (dracut initrd 8,970,760)
exhibits no delay, reaching multi-user.target bash prompt in less than 45
seconds.

Same system booting (second, on sda28, vs. other on sda23) openSUSE
Tumbleweed 4.2.1 (dracut initrd 8,965,124) exhibits ~40 second delay,
reaching multi-user.target bash prompt in about 100 seconds.

Same system booting Kubuntu 14.10 3.16.0 (non-dracut initrd 20,550,608)
exhibits no initial (linux/initrd) delay, reaching KDM greeter in about 90
seconds.

Same system booting Fedora 22 4.1.8 (dracut initrd 10,788,392) exhibits no
delay, reaching multi-user.target bash prompt in less than 50 seconds.


Any ideas what could cause these delays, or how to eliminate them?
-- 
"The wise are known for their understanding, and pleasant
words are persuasive." Proverbs 16:21 (New Living Translation)

 Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata  ***  http://fm.no-ip.com/

Re: kernel/initrd loading delays

[Dave Young]

On 10/12/15 at 12:29pm, Felix Miata wrote:
> On more and more installations since most distros have replaced mkinitrd with
> dracut, on selecting a bootloader menu choice the "root
> (hd0,...filesystemtype..[Linux-bzImage, setup, ... initrd /boot/initrd"
> message stays on screen 40 seconds or more before the time-stamped boot
> messages begin scrolling the screen. IIRC, these delays first began appearing
> last January or February, and appear on multiple machines. All kernel and
> initrd images are on smallish EXT3 or EXT4 (mostly the latter) filesystems
> using 1024 blocksize on BIOS logical partitions on rotating rust (IIRC, all
> with native 512b sector size, manufactured in 2011 or earlier). All
> installations are configured with static IP. All have Plymouth either not
> installed, or are booted with noplymouth included on kernel cmdline.
> 
> e.g, on my fastest system, host msi85, a 3.0 GHz dual core Haswell, total
> time from boot stanza select on BIOS host msi85 to Rawhide 4.3.0-0.rc3.git4.1
> (dracut initramfs.img 11,326,223) multi-user.target shell prompt ready is
> about 80 seconds following that 40 second delay, with last time stamp showing
> 36.844484.
> 
> Same msi85 system booting LMDE 2 3.16.0-4 (non-dracut? initrd.img 27,362,964;
> initramfs-tools 0.120) delays start of boot messages nearly 80 seconds,
> reaching DM greeter ready nearly 140 seconds after stanza selection.
> 
> Same system booting openSUSE Leap 42.1 4.1.8 (dracut initrd 8,082,360)
> exhibits no perceptible delay, reaching multi-user.target shell prompt in
> under 47 seconds from boot stanza selection.
> 
> Same system booting openSUSE 13.2 3.16.7 (dracut initrd 5,351,996) delays
> start of boot messages about 40 seconds, reaching multi-user.target bash
> prompt about 90 seconds after stanza selection.
> 
> Same system booting Mageia 5 4.1.8 (dracut initrd.img 9,391,664) exhibits no
> delay, reaching multi-user.target bash prompt in about 45 seconds.
> 
> Same system booting Wheezy 3.2.0 (non-dracut initrd.img 10,537,321) exhibits
> no delay, reaching DM greeter in less than about 45 seconds.
> 
> Same system booting openSUSE 13.1 3.12.44 (non-dracut initrd 7,679,390)
> exhibits no delay, reaching multi-user.target bash prompt in less than 45
> seconds.
> 
> Same system booting Fedora 23 4.2.2 (dracut initramfs.img 11,255,214) reaches
> multi-user.target shell prompt in about 75 seconds after unknown
> kernel/initrd delay (puts display into sleep mode until boot messages appear).
> 
> Same system booting openSUSE Tumbleweed 4.2.1 (dracut initrd 8,970,760)
> exhibits no delay, reaching multi-user.target bash prompt in less than 45
> seconds.
> 
> Same system booting (second, on sda28, vs. other on sda23) openSUSE
> Tumbleweed 4.2.1 (dracut initrd 8,965,124) exhibits ~40 second delay,
> reaching multi-user.target bash prompt in about 100 seconds.
> 
> Same system booting Kubuntu 14.10 3.16.0 (non-dracut initrd 20,550,608)
> exhibits no initial (linux/initrd) delay, reaching KDM greeter in about 90
> seconds.
> 
> Same system booting Fedora 22 4.1.8 (dracut initrd 10,788,392) exhibits no
> delay, reaching multi-user.target bash prompt in less than 50 seconds.

Wow, you are using so many distributions.

From your previous tests, I feel it should not be a dracut problem.
But you need first find where is the delay from, bootloader, kernel, or somewhere else.
I think you can select one distribution delaying happend, try older kernel
if the older kernel does not have the delay then it should be kernel problem.

As for boot loader, they may have some debug options but I do not know that
much.

Thanks
Dave

Re: kernel/initrd loading delays

[Harald Hoyer]

On 14.10.2015 11:04, Dave Young wrote:
> On 10/12/15 at 12:29pm, Felix Miata wrote:
>> On more and more installations since most distros have replaced mkinitrd with
>> dracut, on selecting a bootloader menu choice the "root
>> (hd0,...filesystemtype..[Linux-bzImage, setup, ... initrd /boot/initrd"
>> message stays on screen 40 seconds or more before the time-stamped boot
>> messages begin scrolling the screen. IIRC, these delays first began appearing
>> last January or February, and appear on multiple machines. All kernel and
>> initrd images are on smallish EXT3 or EXT4 (mostly the latter) filesystems
>> using 1024 blocksize on BIOS logical partitions on rotating rust (IIRC, all
>> with native 512b sector size, manufactured in 2011 or earlier). All
>> installations are configured with static IP. All have Plymouth either not
>> installed, or are booted with noplymouth included on kernel cmdline.
>>
>> e.g, on my fastest system, host msi85, a 3.0 GHz dual core Haswell, total
>> time from boot stanza select on BIOS host msi85 to Rawhide 4.3.0-0.rc3.git4.1
>> (dracut initramfs.img 11,326,223) multi-user.target shell prompt ready is
>> about 80 seconds following that 40 second delay, with last time stamp showing
>> 36.844484.
>>
>> Same msi85 system booting LMDE 2 3.16.0-4 (non-dracut? initrd.img 27,362,964;
>> initramfs-tools 0.120) delays start of boot messages nearly 80 seconds,
>> reaching DM greeter ready nearly 140 seconds after stanza selection.
>>
>> Same system booting openSUSE Leap 42.1 4.1.8 (dracut initrd 8,082,360)
>> exhibits no perceptible delay, reaching multi-user.target shell prompt in
>> under 47 seconds from boot stanza selection.
>>
>> Same system booting openSUSE 13.2 3.16.7 (dracut initrd 5,351,996) delays
>> start of boot messages about 40 seconds, reaching multi-user.target bash
>> prompt about 90 seconds after stanza selection.
>>
>> Same system booting Mageia 5 4.1.8 (dracut initrd.img 9,391,664) exhibits no
>> delay, reaching multi-user.target bash prompt in about 45 seconds.
>>
>> Same system booting Wheezy 3.2.0 (non-dracut initrd.img 10,537,321) exhibits
>> no delay, reaching DM greeter in less than about 45 seconds.
>>
>> Same system booting openSUSE 13.1 3.12.44 (non-dracut initrd 7,679,390)
>> exhibits no delay, reaching multi-user.target bash prompt in less than 45
>> seconds.
>>
>> Same system booting Fedora 23 4.2.2 (dracut initramfs.img 11,255,214) reaches
>> multi-user.target shell prompt in about 75 seconds after unknown
>> kernel/initrd delay (puts display into sleep mode until boot messages appear).
>>
>> Same system booting openSUSE Tumbleweed 4.2.1 (dracut initrd 8,970,760)
>> exhibits no delay, reaching multi-user.target bash prompt in less than 45
>> seconds.
>>
>> Same system booting (second, on sda28, vs. other on sda23) openSUSE
>> Tumbleweed 4.2.1 (dracut initrd 8,965,124) exhibits ~40 second delay,
>> reaching multi-user.target bash prompt in about 100 seconds.
>>
>> Same system booting Kubuntu 14.10 3.16.0 (non-dracut initrd 20,550,608)
>> exhibits no initial (linux/initrd) delay, reaching KDM greeter in about 90
>> seconds.
>>
>> Same system booting Fedora 22 4.1.8 (dracut initrd 10,788,392) exhibits no
>> delay, reaching multi-user.target bash prompt in less than 50 seconds.
> 
> Wow, you are using so many distributions.
> 
> From your previous tests, I feel it should not be a dracut problem.
> But you need first find where is the delay from, bootloader, kernel, or somewhere else.
> I think you can select one distribution delaying happend, try older kernel
> if the older kernel does not have the delay then it should be kernel problem.
> 
> As for boot loader, they may have some debug options but I do not know that
> much.
> 
> Thanks
> Dave

Also rawhide kernels have debugging enabled, which slows down significantly.

Adding "rd.debug" might help.

https://www.kernel.org/pub/linux/utils/boot/dracut/dracut.html#_troubleshooting

Also "debug" for systemd/udevd debug messages.

Re: kernel/initrd loading delays

[Felix Miata]

Felix Miata composed on 2015-10-12 12:29 (UTC-0400):

> On more and more installations since most distros have replaced mkinitrd with
> dracut, on selecting a bootloader menu choice the "root
> (hd0,...filesystemtype..[Linux-bzImage, setup, ... initrd /boot/initrd"
> message stays on screen 40 seconds or more before the time-stamped boot
> messages begin scrolling the screen. IIRC, these delays first began appearing
> last January or February, and appear on multiple machines. All kernel and
> initrd images are on smallish EXT3 or EXT4 (mostly the latter) filesystems
> using 1024 blocksize on BIOS logical partitions on rotating rust (IIRC, all
> with native 512b sector size, manufactured in 2011 or earlier). All
> installations are configured with static IP. All have Plymouth either not
> installed, or are booted with noplymouth included on kernel cmdline.

> e.g, on my fastest system, host msi85, a 3.0 GHz dual core Haswell, total
> time from boot stanza select on BIOS host msi85 to Rawhide 4.3.0-0.rc3.git4.1
> (dracut initramfs.img 11,326,223) multi-user.target shell prompt ready is
> about 80 seconds following that 40 second delay, with last time stamp showing
> 36.844484.

> Same msi85 system booting LMDE 2 3.16.0-4 (non-dracut? initrd.img 27,362,964;
> initramfs-tools 0.120) delays start of boot messages nearly 80 seconds,
> reaching DM greeter ready nearly 140 seconds after stanza selection.

> Same system booting openSUSE Leap 42.1 4.1.8 (dracut initrd 8,082,360)
> exhibits no perceptible delay, reaching multi-user.target shell prompt in
> under 47 seconds from boot stanza selection.

> Same system booting openSUSE 13.2 3.16.7 (dracut initrd 5,351,996) delays
> start of boot messages about 40 seconds, reaching multi-user.target bash
> prompt about 90 seconds after stanza selection.

> Same system booting Mageia 5 4.1.8 (dracut initrd.img 9,391,664) exhibits no
> delay, reaching multi-user.target bash prompt in about 45 seconds.

> Same system booting Wheezy 3.2.0 (non-dracut initrd.img 10,537,321) exhibits
> no delay, reaching DM greeter in less than about 45 seconds.

> Same system booting openSUSE 13.1 3.12.44 (non-dracut initrd 7,679,390)
> exhibits no delay, reaching multi-user.target bash prompt in less than 45
> seconds.

> Same system booting Fedora 23 4.2.2 (dracut initramfs.img 11,255,214) reaches
> multi-user.target shell prompt in about 75 seconds after unknown
> kernel/initrd delay (puts display into sleep mode until boot messages appear).

> Same system booting openSUSE Tumbleweed 4.2.1 (dracut initrd 8,970,760)
> exhibits no delay, reaching multi-user.target bash prompt in less than 45
> seconds.

> Same system booting (second, on sda28, vs. other on sda23) openSUSE
> Tumbleweed 4.2.1 (dracut initrd 8,965,124) exhibits ~40 second delay,
> reaching multi-user.target bash prompt in about 100 seconds.

> Same system booting Kubuntu 14.10 3.16.0 (non-dracut initrd 20,550,608)
> exhibits no initial (linux/initrd) delay, reaching KDM greeter in about 90
> seconds.

> Same system booting Fedora 22 4.1.8 (dracut initrd 10,788,392) exhibits no
> delay, reaching multi-user.target bash prompt in less than 50 seconds.

> Any ideas what could cause these delays, or how to eliminate them?

Booting host hpg33 openSUSE Tumbleweed 20160307 64 bit kernel-default 4.4.3, 
6194336 bytes, and initrd 38456340 bytes, just took 9.4 minutes for the last 
Grub message to clear, and extended that to ~10.2 minutes to reach a shell 
prompt. Prior 4.3.3-5 kernel-default's initrd is 9589212 bytes, 25% the size 
of the current. 4.3.3-5 took 2 2/3 minutes for the Grub message to clear, 
about 2/3 more minute to reach shell prompt, and 4.3.3-1 about the same. 3rd 
prior, kernel-desktop-4.2.1 cleared the Grub message in too few seconds to 
count, well under a minute to reach shell prompt. kernel-default-4.5.0 
cleared the Grub in 130 seconds, reached shell prompt at 174.

Mount reported /dev/mapper/* names with 4.4.3 and 4.5.0, while 4.2.1 and 
4.3.3 reported device names.

What's taking so long with kernel-default?
-- 
"The wise are known for their understanding, and pleasant
words are persuasive." Proverbs 16:21 (New Living Translation)

  Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata  ***  http://fm.no-ip.com/