* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
@ 2011-05-15 18:32 ` Tom Gundersen
2011-05-15 23:25 ` Kay Sievers
` (13 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Tom Gundersen @ 2011-05-15 18:32 UTC (permalink / raw)
To: linux-hotplug
On Sun, May 15, 2011 at 8:19 PM, Nix <nix@esperi.org.uk> wrote:
> I know that you're not supposed to rely on 'udevadm settle' anymore, but
> I rely on it across the board for systems with root filesystems that
> aren't expected to move around (i.e. all of them), because massively
> reengineering working systems' boot processes is generally considered a
> bad thing. And it's stopped working. Given how many things expect /dev
> to be populated, this has fairly serious effects.
>
> I can be certain that as of somewhere between udev 164 and 167, 'udevadm
> settle' has stopped waiting for block devices to appear (though I
> suspect others have vanished too). I'm booting udev as recommended in
> the release notes, via
>
> Â udevd --daemon
> Â udevadm trigger --actiond --type=subsystems
> Â udevadm trigger --actiond --typeÞvices
> Â udevadm settle
We are doing the same on Arch and today I started seeing bug reports
(after the upgrade to udev 168). So here are my two cents:
Most of the time the problem seems to be related to LVM, but I have
also seen regular block devices having problems. As would be expected
using devtmpfs greatly reduces (if not eliminates) the problem. My
guess was (like Nix said) that "udevadm settle" is somehow broken.
Some related bug reports:
Arch: <https://bugs.archlinux.org/task/24288>,
Debian: <http://bugs.debian.org/cgi-bin/bugreport.cgi?bugb4010>.
Cheers,
Tom
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
2011-05-15 18:32 ` Tom Gundersen
@ 2011-05-15 23:25 ` Kay Sievers
2011-05-15 23:33 ` Kay Sievers
` (12 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Kay Sievers @ 2011-05-15 23:25 UTC (permalink / raw)
To: linux-hotplug
On Sun, May 15, 2011 at 20:19, Nix <nix@esperi.org.uk> wrote:
> I know that you're not supposed to rely on 'udevadm settle' anymore, but
> I rely on it across the board for systems with root filesystems that
> aren't expected to move around (i.e. all of them), because massively
> reengineering working systems' boot processes is generally considered a
> bad thing. And it's stopped working. Given how many things expect /dev
> to be populated, this has fairly serious effects.
>
> I can be certain that as of somewhere between udev 164 and 167, 'udevadm
> settle' has stopped waiting for block devices to appear (though I
> suspect others have vanished too). I'm booting udev as recommended in
> the release notes, via
>
> Â udevd --daemon
> Â udevadm trigger --actiond --type=subsystems
> Â udevadm trigger --actiond --typeÞvices
> Â udevadm settle
>
> but this is what I now see:
>
> ,----
> | Creating initial device nodes...
> | [ Â Â 2.035253] <30>udevd[297]: starting version 168
> | udevd[297]: specified group 'audio' unknown
> |
> | [ Â Â 2.151279] <30>udevd[297]: converting old udev database
> | udevd[316]: failed to execute '/sbin/modprobe' '/sbin/modprobe -bv pci:v00001022d00002080sv00001022sd00002080bc06sc00i00': No such file or directory
> |
> | umount: /run: device is busy.
What's 'umount /run' during bootup? That sounds really strange.
> | Â Â Â Â (In some cases useful info about processes that use
> | Â Â Â Â Â the device is found by lsof(8) or fuser(1))
> | Cannot find device "gordianet"
> | fsck from util-linux 2.19
> | udevd[334]: failed to exec[ Â Â 2.457619] EXT2-fs (sda1): warning: mounting unchecked fs, running e2fsck is recommended
> | ute '/sbin/modprobe' '/sbin/modprobe -bv platform:cs5535-gpio': No such file or directory
> |
> | udevd[333]: failed to execute '/sbin/modprobe' '/sbin/modprobe -bv platform:cs5535-acpi': No such file or directory
> |
> | udevd[335]: failed to execute '/sbin/modprobe' '/sbin/modprobe -bv platform:cs5535-pms': No such file or directory
> |
> | udevd[336]: failed to execute '/sbin/modprobe' '/sbin/modprobe -bv sg': No such file or directory
> |
> | udevd[339]: failed to execute '/sbin/modprobe' '/sbin/modprobe -bv platform:rtc_cmos': No such file or directory
> |
> | fsck.ext2: No such file or directory while trying to open /dev/sda1
> | Possibly non-existent device?
> `----
>
> I have no clue why udev is trying to run modprobe when this is a
> non-modular kernel with all necessary devices built in. But that's not
> important.
Udev has no idea what the kernel supports, it calls modprobe for all
devices with a 'modalias' but no attached driver.
> By the time of that 'umount', 'udevadm settle' has returned. Shortly
> afterwards you see fsck claiming that devices don't exist. Look at
> it the filesystem after the resulting failed half-boot and you see:
>
> fold:~# ls -l /dev/sda1
> brw-rw---- 1 root disk 8, 1 May 15 18:56 /dev/sda1
>
> it's there. udevadm settle just didn't bother to wait for it to appear.
> (This is not a device on a bus for which enumeration takes some time:
> it's an SD card on an IDE-lookalike cs5535 bus. I've also seen this
> on LVM-atop-RAID-atop-SATA and on ordinary LVM-atop-SATA, so it doesn't
> require anything particularly unusual.)
Sounds weird. Settle should not return at that point. You are not
altering the content of /run/udev/ or /dev/.udev/ in any way right?
And you provide the /run tmpfs before you start udevd and don't touch
it again, right?
> Other things seem to have failed too. I have renaming rules:
>
> SUBSYSTEM="net", ATTR{address}="00:00:24:cb:c6:a0", NAME="gordianet"
> SUBSYSTEM="net", ATTR{address}="00:00:24:cb:c6:a1", NAME="adsl"
> SUBSYSTEM="net", ATTR{address}="00:00:24:cb:c6:a3", NAME="wireless"
>
> yet the devices were not renamed:
Hmm, that should be unrelated to the possible settle problem
> 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
> Â Â link/ether 00:00:24:cb:c6:a0 brd ff:ff:ff:ff:ff:ff
> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
> Â Â link/ether 00:00:24:cb:c6:a1 brd ff:ff:ff:ff:ff:ff
> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
> Â Â link/ether 00:00:24:cb:c6:a3 brd ff:ff:ff:ff:ff:ff
>
> Put a 'sleep 1' after the udev call, and everything works.
Which call? The trigger?
Kay
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
2011-05-15 18:32 ` Tom Gundersen
2011-05-15 23:25 ` Kay Sievers
@ 2011-05-15 23:33 ` Kay Sievers
2011-05-15 23:38 ` Nix
` (11 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Kay Sievers @ 2011-05-15 23:33 UTC (permalink / raw)
To: linux-hotplug
On Sun, May 15, 2011 at 20:32, Tom Gundersen <teg@jklm.no> wrote:
> On Sun, May 15, 2011 at 8:19 PM, Nix <nix@esperi.org.uk> wrote:
>> I know that you're not supposed to rely on 'udevadm settle' anymore, but
>> I rely on it across the board for systems with root filesystems that
>> aren't expected to move around (i.e. all of them), because massively
>> reengineering working systems' boot processes is generally considered a
>> bad thing. And it's stopped working. Given how many things expect /dev
>> to be populated, this has fairly serious effects.
>>
>> I can be certain that as of somewhere between udev 164 and 167, 'udevadm
>> settle' has stopped waiting for block devices to appear (though I
>> suspect others have vanished too). I'm booting udev as recommended in
>> the release notes, via
>>
>> Â udevd --daemon
>> Â udevadm trigger --actiond --type=subsystems
>> Â udevadm trigger --actiond --typeÞvices
>> Â udevadm settle
>
> We are doing the same on Arch and today I started seeing bug reports
> (after the upgrade to udev 168). So here are my two cents:
>
> Most of the time the problem seems to be related to LVM, but I have
> also seen regular block devices having problems. As would be expected
> using devtmpfs greatly reduces (if not eliminates) the problem. My
> guess was (like Nix said) that "udevadm settle" is somehow broken.
We are sure it's not related to /run? We keep the state there and need
a tmpfs there before udevd is started, and it must not be touched, or
cleaned by some other stuff, that thinks /var/run (bind mount or
symlink) needs to be cleaned during boot.
Devtmpfs solves a lot of settle races, yeah. We should run fine on
tmpfs, but it's the only config that is really tested these days, so
it might be a problem nobody else is really seeing.
The current settle wakes up in exactly the moment the last event is
gone, instead of it sleep()ing in a loop. It might be a bit earlier,
not really before stuff has settled though.
Does this work for you?
time (udevadm trigger; udevadm settle)
It should not return immediately. You can watch the events with:
udevadm monitor
at the same time and check if it really only returns after the last event.
Kay
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (2 preceding siblings ...)
2011-05-15 23:33 ` Kay Sievers
@ 2011-05-15 23:38 ` Nix
2011-05-15 23:47 ` Nix
` (10 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Nix @ 2011-05-15 23:38 UTC (permalink / raw)
To: linux-hotplug
On 16 May 2011, Kay Sievers stated:
> On Sun, May 15, 2011 at 20:19, Nix <nix@esperi.org.uk> wrote:
>> ,----
>> | Creating initial device nodes...
>> | [ Â Â 2.035253] <30>udevd[297]: starting version 168
>> | udevd[297]: specified group 'audio' unknown
>> |
>> | [ Â Â 2.151279] <30>udevd[297]: converting old udev database
>> | udevd[316]: failed to execute '/sbin/modprobe' '/sbin/modprobe -bv pci:v00001022d00002080sv00001022sd00002080bc06sc00i00': No such file or directory
>> |
>> | umount: /run: device is busy.
>
> What's 'umount /run' during bootup? That sounds really strange.
Hm, I didn't notice that. Yes, that does seem very odd indeed.
I'll have a look soon, if probably not on that system (it's a headless
system that does my firewalling so quite hard to reboot). I see the
symptoms on other systems too.
>> | udevd[339]: failed to execute '/sbin/modprobe' '/sbin/modprobe -bv platform:rtc_cmos': No such file or directory
>> |
>> | fsck.ext2: No such file or directory while trying to open /dev/sda1
>> | Possibly non-existent device?
>> `----
>>
>> I have no clue why udev is trying to run modprobe when this is a
>> non-modular kernel with all necessary devices built in. But that's not
>> important.
>
> Udev has no idea what the kernel supports, it calls modprobe for all
> devices with a 'modalias' but no attached driver.
Yeah, it's not terribly important (though why they even have a modalias
if this kernel does not have modules built in is also unclear. Anyway,
it can do no harm, since modprobe doesn't even exist on that system.)
>> fold:~# ls -l /dev/sda1
>> brw-rw---- 1 root disk 8, 1 May 15 18:56 /dev/sda1
>>
>> it's there. udevadm settle just didn't bother to wait for it to appear.
>> (This is not a device on a bus for which enumeration takes some time:
>> it's an SD card on an IDE-lookalike cs5535 bus. I've also seen this
>> on LVM-atop-RAID-atop-SATA and on ordinary LVM-atop-SATA, so it doesn't
>> require anything particularly unusual.)
>
> Sounds weird. Settle should not return at that point.
Agreed! (This is why I suspect the timeout stuff is simply timing out
immediately.)
> You are not
> altering the content of /run/udev/ or /dev/.udev/ in any way right?
Gods, no. Recipe for disaster.
> And you provide the /run tmpfs before you start udevd and don't touch
> it again, right?
ooo! possible bug. Here's my udev startup script:
mount -n /proc
mount -n /sys
mount -n /run
[...]
mount_tmpfs
mkdir -p $udev_root/.udev/db $udev_root/.udev/queue $udev_root/.udev/failed
echo "Creating initial device nodes..."
udevd --daemon
udevadm trigger --actiond --type=subsystems
udevadm trigger --actiond --typeÞvices
udevadm settle
sleep 1
*That* is going to cause udev to try to convert from the old to the new
udev database format every single time it starts, even though there *is*
no old udev database there, just skeletal directories. I wonder if
that's causing it?
(I've ripped that mkdir out. Let's see if that fixes things. If it does,
this suggests that udev needs further bulletproofing, because my udev
startup script was derived directly from one provided by earlier
versions of udev: a *lot* of people will have scripts like it.)
>> Other things seem to have failed too. I have renaming rules:
>>
>> SUBSYSTEM="net", ATTR{address}="00:00:24:cb:c6:a0", NAME="gordianet"
>> SUBSYSTEM="net", ATTR{address}="00:00:24:cb:c6:a1", NAME="adsl"
>> SUBSYSTEM="net", ATTR{address}="00:00:24:cb:c6:a3", NAME="wireless"
>>
>> yet the devices were not renamed:
>
> Hmm, that should be unrelated to the possible settle problem
Yeah. The rename-failure only seems to happen when the settle failure
happens, so perhaps it's related to parts of the startup script messing
with the interfaces and causing the kernel to ban renames because
they're in use. Obviously this doesn't happen if we're still sitting in
'udevadm settle' when this takes place.
>> 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
>> Â Â link/ether 00:00:24:cb:c6:a0 brd ff:ff:ff:ff:ff:ff
>> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
>> Â Â link/ether 00:00:24:cb:c6:a1 brd ff:ff:ff:ff:ff:ff
>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
>> Â Â link/ether 00:00:24:cb:c6:a3 brd ff:ff:ff:ff:ff:ff
>>
>> Put a 'sleep 1' after the udev call, and everything works.
>
> Which call? The trigger?
See above: it was right after the settle call.
Anyway, more tomorrow sometime after more testing. (It's past midnight
here.)
--
NULL && (void)
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (3 preceding siblings ...)
2011-05-15 23:38 ` Nix
@ 2011-05-15 23:47 ` Nix
2011-05-15 23:51 ` Kay Sievers
` (9 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Nix @ 2011-05-15 23:47 UTC (permalink / raw)
To: linux-hotplug
On 16 May 2011, Kay Sievers stated:
> We are sure it's not related to /run? We keep the state there and need
> a tmpfs there before udevd is started, and it must not be touched, or
> cleaned by some other stuff, that thinks /var/run (bind mount or
> symlink) needs to be cleaned during boot.
Just to confirm, I'm not cleaning out /run, or touching it at all except
to create /run/lock later in boot.
> Devtmpfs solves a lot of settle races, yeah. We should run fine on
> tmpfs, but it's the only config that is really tested these days, so
> it might be a problem nobody else is really seeing.
I could turn on devtmpfs, I suppose, except that I have an early
userspace and I'm not sure how I'd need to change it: devtmpfs gets
mounted on the rootfs, doesn't it? This problem isn't happening on the
rootfs, it's on the root filesystem that is overmounted over that. (I'm
using busybox mdev to populate the rootfs /dev because it works for such
a simple case, and never goes wrong because it's too simple to go wrong.
However it's also too simple to be much use for a running system.)
> The current settle wakes up in exactly the moment the last event is
> gone, instead of it sleep()ing in a loop. It might be a bit earlier,
> not really before stuff has settled though.
>
> Does this work for you?
> time (udevadm trigger; udevadm settle)
>
> It should not return immediately. You can watch the events with:
>
> udevadm monitor
> at the same time and check if it really only returns after the last event.
Hm, well, that seems to be working, at least once the system is all the
way up. But *something* plainly isn't.
On my next boot I'll time the trigger-and-settle pair and see how long
it takes...
--
NULL && (void)
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (4 preceding siblings ...)
2011-05-15 23:47 ` Nix
@ 2011-05-15 23:51 ` Kay Sievers
2011-05-15 23:57 ` Kay Sievers
` (8 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Kay Sievers @ 2011-05-15 23:51 UTC (permalink / raw)
To: linux-hotplug
On Mon, May 16, 2011 at 01:38, Nix <nix@esperi.org.uk> wrote:
> On 16 May 2011, Kay Sievers stated:
>> What's 'umount /run' during bootup? That sounds really strange.
>
> Hm, I didn't notice that. Yes, that does seem very odd indeed.
>
> I'll have a look soon, if probably not on that system (it's a headless
> system that does my firewalling so quite hard to reboot). I see the
> symptoms on other systems too.
>
>>> | udevd[339]: failed to execute '/sbin/modprobe' '/sbin/modprobe -bv platform:rtc_cmos': No such file or directory
>>>
>>> I have no clue why udev is trying to run modprobe when this is a
>>> non-modular kernel with all necessary devices built in. But that's not
>>> important.
>>
>> Udev has no idea what the kernel supports, it calls modprobe for all
>> devices with a 'modalias' but no attached driver.
>
> Yeah, it's not terribly important (though why they even have a modalias
> if this kernel does not have modules built in is also unclear. Anyway,
> it can do no harm, since modprobe doesn't even exist on that system.)
Yeah, you could #ifdef the modalias export if the kernel does not
support module loading. This is not optimized.
It' probably just easier to delete the default udev rule that calls
modprobe, if you don't need it. :)
>>> fold:~# ls -l /dev/sda1
>>> brw-rw---- 1 root disk 8, 1 May 15 18:56 /dev/sda1
>>>
>>> it's there. udevadm settle just didn't bother to wait for it to appear.
>>> (This is not a device on a bus for which enumeration takes some time:
>>> it's an SD card on an IDE-lookalike cs5535 bus. I've also seen this
>>> on LVM-atop-RAID-atop-SATA and on ordinary LVM-atop-SATA, so it doesn't
>>> require anything particularly unusual.)
>>
>> Sounds weird. Settle should not return at that point.
>
> Agreed! (This is why I suspect the timeout stuff is simply timing out
> immediately.)
Yeah, that could happen too, if /run/udev/queue.bin would be deleted
by some cleanup script, after udevd is started.
>> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â You are not
>> altering the content of /run/udev/ or /dev/.udev/ in any way right?
>
> Gods, no. Recipe for disaster.
Ok, fine.
>> And you provide the /run tmpfs before you start udevd and don't touch
>> it again, right?
>
> ooo! possible bug. Here's my udev startup script:
>
> mount -n /proc
> mount -n /sys
> mount -n /run
> [...]
> mount_tmpfs
> mkdir -p $udev_root/.udev/db $udev_root/.udev/queue $udev_root/.udev/failed
Yeah, never create any private directories of udevd. Most of them do
not even exist anymore in today's udev.
> echo "Creating initial device nodes..."
> udevd --daemon
> udevadm trigger --actiond --type=subsystems
> udevadm trigger --actiond --typeÞvices
> udevadm settle
> sleep 1
>
> *That* is going to cause udev to try to convert from the old to the new
> udev database format every single time it starts, even though there *is*
> no old udev database there, just skeletal directories. I wonder if
> that's causing it?
>
> (I've ripped that mkdir out. Let's see if that fixes things. If it does,
> this suggests that udev needs further bulletproofing, because my udev
> startup script was derived directly from one provided by earlier
> versions of udev: a *lot* of people will have scripts like it.)
Yeah, not a good idea to fiddle around with udev internals. The
existence of the old directory names indicates a need to convert. It
should not really break anything, just waste some time during udevd
startup.
>>> Other things seem to have failed too. I have renaming rules:
>>>
>>> SUBSYSTEM="net", ATTR{address}="00:00:24:cb:c6:a0", NAME="gordianet"
>>> SUBSYSTEM="net", ATTR{address}="00:00:24:cb:c6:a1", NAME="adsl"
>>> SUBSYSTEM="net", ATTR{address}="00:00:24:cb:c6:a3", NAME="wireless"
>>>
>>> yet the devices were not renamed:
>>
>> Hmm, that should be unrelated to the possible settle problem
>
> Yeah. The rename-failure only seems to happen when the settle failure
> happens, so perhaps it's related to parts of the startup script messing
> with the interfaces and causing the kernel to ban renames because
> they're in use. Obviously this doesn't happen if we're still sitting in
> 'udevadm settle' when this takes place.
Yeah, that could explain it. A soon as the netif is up, we can not
rename it anymore.
>>> Put a 'sleep 1' after the udev call, and everything works.
>>
>> Which call? The trigger?
>
> See above: it was right after the settle call.
I see. Hmm, no good idea why this would be.
> Anyway, more tomorrow sometime after more testing. (It's past midnight
> here.)
Sounds good, let me know, if you find something out.
Kay
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (5 preceding siblings ...)
2011-05-15 23:51 ` Kay Sievers
@ 2011-05-15 23:57 ` Kay Sievers
2011-05-16 5:05 ` DJ Lucas
` (7 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Kay Sievers @ 2011-05-15 23:57 UTC (permalink / raw)
To: linux-hotplug
On Mon, May 16, 2011 at 01:47, Nix <nix@esperi.org.uk> wrote:
> On 16 May 2011, Kay Sievers stated:
>> We are sure it's not related to /run? We keep the state there and need
>> a tmpfs there before udevd is started, and it must not be touched, or
>> cleaned by some other stuff, that thinks /var/run (bind mount or
>> symlink) needs to be cleaned during boot.
>
> Just to confirm, I'm not cleaning out /run, or touching it at all except
> to create /run/lock later in boot.
Ok, sounds good.
>> Devtmpfs solves a lot of settle races, yeah. We should run fine on
>> tmpfs, but it's the only config that is really tested these days, so
>> it might be a problem nobody else is really seeing.
>
> I could turn on devtmpfs, I suppose, except that I have an early
> userspace and I'm not sure how I'd need to change it: devtmpfs gets
> mounted on the rootfs, doesn't it?
If you ask the kernel to do that at compile time or with a parameter,
it does that, but only if the rootfs is mounted by the kernel itself,
not in any initramfs case.
> This problem isn't happening on the
> rootfs, it's on the root filesystem that is overmounted over that. (I'm
> using busybox mdev to populate the rootfs /dev because it works for such
> a simple case, and never goes wrong because it's too simple to go wrong.
> However it's also too simple to be much use for a running system.)
With devtmpfs you get the "early" /dev for free, and you preserve the
"early" /dev in the real root. Instead of mounting a 'tmpfs' just
mount a 'devtmpfs' at /dev, and it will be magically filled already,
that's the only difference.
When mounting the real root, just mount --move it over to the real
root before switching to the real root.
>> The current settle wakes up in exactly the moment the last event is
>> gone, instead of it sleep()ing in a loop. It might be a bit earlier,
>> not really before stuff has settled though.
>>
>> Does this work for you?
>> time (udevadm trigger; udevadm settle)
>>
>> It should not return immediately. You can watch the events with:
>>
>> udevadm monitor
>> at the same time and check if it really only returns after the last event.
>
> Hm, well, that seems to be working, at least once the system is all the
> way up. But *something* plainly isn't.
>
> On my next boot I'll time the trigger-and-settle pair and see how long
> it takes...
Good.
Maybe you can run:
udevadm monitor
in the background writing to a file, and get timestamps of your commands too.
For testing, you could also add test rules like:
KERNEL="null", PROGRAM="/bin/sleep 10"
Which should make settle definitely block for that amount of time.
Kay
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (6 preceding siblings ...)
2011-05-15 23:57 ` Kay Sievers
@ 2011-05-16 5:05 ` DJ Lucas
2011-05-16 6:23 ` DJ Lucas
` (6 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: DJ Lucas @ 2011-05-16 5:05 UTC (permalink / raw)
To: linux-hotplug
On 05/15/2011 01:32 PM, Tom Gundersen wrote:
> On Sun, May 15, 2011 at 8:19 PM, Nix<nix@esperi.org.uk> wrote:
>> I know that you're not supposed to rely on 'udevadm settle' anymore, but
>> I rely on it across the board for systems with root filesystems that
>> aren't expected to move around (i.e. all of them), because massively
>> reengineering working systems' boot processes is generally considered a
>> bad thing. And it's stopped working. Given how many things expect /dev
>> to be populated, this has fairly serious effects.
>>
>> I can be certain that as of somewhere between udev 164 and 167, 'udevadm
>> settle' has stopped waiting for block devices to appear (though I
>> suspect others have vanished too). I'm booting udev as recommended in
>> the release notes, via
>>
>> udevd --daemon
>> udevadm trigger --actiond --type=subsystems
>> udevadm trigger --actiond --typeÞvices
>> udevadm settle
>
> We are doing the same on Arch and today I started seeing bug reports
> (after the upgrade to udev 168). So here are my two cents:
>
> Most of the time the problem seems to be related to LVM, but I have
> also seen regular block devices having problems. As would be expected
> using devtmpfs greatly reduces (if not eliminates) the problem. My
> guess was (like Nix said) that "udevadm settle" is somehow broken.
>
> Some related bug reports:
>
> Arch:<https://bugs.archlinux.org/task/24288>,
> Debian:<http://bugs.debian.org/cgi-bin/bugreport.cgi?bugb4010>.
>
> Cheers,
>
> Tom
> --
> To unsubscribe from this list: send the line "unsubscribe linux-hotplug" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Just to add a "me too" regarding regular block devices. First noticed
with 168, coming from 165. /dev was tmpfs and swap is /dev/sda3 and
mounted immediately after udevadm --settle. Kernel is 2.6.36.2. In
addition to the udev change, I added the /run mount point to the mix
(started very early, directly in rc before any scripts are run). Moving
to devtmpfs 'fixed' it for me, but that was because devtmpfs has the raw
sd* nodes, as explained to me by another dev. If I flip to use by-id, it
breaks again. I plan to go back and see exactly where stuff broke, but
haven't had the time just yet, more pressing issues. Probably a
Wednesday evening project (CDT).
-- DJ Lucas
--
This message has been scanned for viruses and
dangerous content, and is believed to be clean.
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (7 preceding siblings ...)
2011-05-16 5:05 ` DJ Lucas
@ 2011-05-16 6:23 ` DJ Lucas
2011-05-16 10:01 ` Nix
` (5 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: DJ Lucas @ 2011-05-16 6:23 UTC (permalink / raw)
To: linux-hotplug
On 05/16/2011 12:05 AM, DJ Lucas wrote:
> Probably a Wednesday evening project (CDT).
>
Nevermind, found time. Tracked it down to this commit:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;hÿ2c503df091e6e4e9ab48cdb6df6ec8b7b525d0
-- DJ Lucas
--
This message has been scanned for viruses and
dangerous content, and is believed to be clean.
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (8 preceding siblings ...)
2011-05-16 6:23 ` DJ Lucas
@ 2011-05-16 10:01 ` Nix
2011-05-16 10:22 ` Marco d'Itri
` (4 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Nix @ 2011-05-16 10:01 UTC (permalink / raw)
To: linux-hotplug
On 16 May 2011, Kay Sievers outgrape:
> With devtmpfs you get the "early" /dev for free, and you preserve the
> "early" /dev in the real root. Instead of mounting a 'tmpfs' just
> mount a 'devtmpfs' at /dev, and it will be magically filled already,
> that's the only difference.
>
> When mounting the real root, just mount --move it over to the real
> root before switching to the real root.
I'll have to see what busybox's switch_root does. I know it deletes
the entire contents of the root directory it switches away from: I'll
have to make sure that doesn't somehow include moved-away mountpoints.
(I doubt it does.)
> Maybe you can run:
> udevadm monitor
> in the background writing to a file, and get timestamps of your commands too.
They're fine.
After a bunch of straces, I think the problem is different. As I
suspected from its intermittent nature, it's a race.
When udevd starts up, it looks for an old database and converts it into
a new-style /run database. This takes nonzero time, and is done *after*
daemonization. Unfortunately, in the same time window it is quite
possible to fit a pair of 'udevadm trigger's and a 'udevadm settle': and
when udevadm settle kicks up and finds that udev hasn't even initialized
yet, what does it do? Well, unless I misread the code
udev_queue_get_queue_is_empty() returns 1 in this situation, so 'udevadm
settle' returns EXIT_SUCCESS, and boom.
My evidence for this is very thin: it's no more than a hypothesis
really. My only evidence for this so far is that I haven't managed to
get the failure to happen unless I mkdir the old udev directories first
to force udev to do an old->new db conversion. It would be nice to
gather more data, but it's hard to do that without perturbing the race
:/ even an ls of /run slows it down enough that it doesn't happen
anymore. Hm, perhaps an echo would be fast enough, it's a shell
builtin after all...
If I'm right (and I'm not sure I am, but it feels plausible), then I
suspect the only fix is to move the db conversion so that it happens
before daemonization.
--
NULL && (void)
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (9 preceding siblings ...)
2011-05-16 10:01 ` Nix
@ 2011-05-16 10:22 ` Marco d'Itri
2011-05-16 10:36 ` Nix
` (3 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Marco d'Itri @ 2011-05-16 10:22 UTC (permalink / raw)
To: linux-hotplug
On May 16, Nix <nix@esperi.org.uk> wrote:
> My evidence for this is very thin: it's no more than a hypothesis
> really. My only evidence for this so far is that I haven't managed to
> get the failure to happen unless I mkdir the old udev directories first
> to force udev to do an old->new db conversion. It would be nice to
The Debian initramfs does not do this, so I do not think that this is
the cause.
--
ciao,
Marco
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (10 preceding siblings ...)
2011-05-16 10:22 ` Marco d'Itri
@ 2011-05-16 10:36 ` Nix
2011-05-16 10:51 ` Kay Sievers
` (2 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Nix @ 2011-05-16 10:36 UTC (permalink / raw)
To: linux-hotplug
On 16 May 2011, Marco d'Itri outgrape:
> On May 16, Nix <nix@esperi.org.uk> wrote:
>
>> My evidence for this is very thin: it's no more than a hypothesis
>> really. My only evidence for this so far is that I haven't managed to
>> get the failure to happen unless I mkdir the old udev directories first
>> to force udev to do an old->new db conversion. It would be nice to
> The Debian initramfs does not do this, so I do not think that this is
> the cause.
Curses. Oh well, back to the research drawing board.
--
NULL && (void)
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (11 preceding siblings ...)
2011-05-16 10:36 ` Nix
@ 2011-05-16 10:51 ` Kay Sievers
2011-05-16 11:28 ` Kay Sievers
2011-05-16 12:47 ` Nix
14 siblings, 0 replies; 16+ messages in thread
From: Kay Sievers @ 2011-05-16 10:51 UTC (permalink / raw)
To: linux-hotplug
On Mon, May 16, 2011 at 12:01, Nix <nix@esperi.org.uk> wrote:
> On 16 May 2011, Kay Sievers outgrape:
>> With devtmpfs you get the "early" /dev for free, and you preserve the
>> "early" /dev in the real root. Instead of mounting a 'tmpfs' just
>> mount a 'devtmpfs' at /dev, and it will be magically filled already,
>> that's the only difference.
>>
>> When mounting the real root, just mount --move it over to the real
>> root before switching to the real root.
>
> I'll have to see what busybox's switch_root does. I know it deletes
> the entire contents of the root directory it switches away from: I'll
> have to make sure that doesn't somehow include moved-away mountpoints.
> (I doubt it does.)
>
>> Maybe you can run:
>> Â udevadm monitor
>> in the background writing to a file, and get timestamps of your commands too.
>
> They're fine.
>
> After a bunch of straces, I think the problem is different. As I
> suspected from its intermittent nature, it's a race.
>
> When udevd starts up, it looks for an old database and converts it into
> a new-style /run database. This takes nonzero time, and is done *after*
> daemonization. Unfortunately, in the same time window it is quite
> possible to fit a pair of 'udevadm trigger's and a 'udevadm settle': and
> when udevadm settle kicks up and finds that udev hasn't even initialized
> yet, what does it do? Well, unless I misread the code
> udev_queue_get_queue_is_empty() returns 1 in this situation, so 'udevadm
> settle' returns EXIT_SUCCESS, and boom.
>
> My evidence for this is very thin: it's no more than a hypothesis
> really. My only evidence for this so far is that I haven't managed to
> get the failure to happen unless I mkdir the old udev directories first
> to force udev to do an old->new db conversion. It would be nice to
> gather more data, but it's hard to do that without perturbing the race
> :/ even an ls of /run slows it down enough that it doesn't happen
> anymore. Hm, perhaps an echo would be fast enough, it's a shell
> builtin after all...
>
> If I'm right (and I'm not sure I am, but it feels plausible), then I
> suspect the only fix is to move the db conversion so that it happens
> before daemonization.
Hmm, the trigger should increase the kernel's current uevent seqnum.
udev_queue_get_queue_is_empty() should see that there are events
pending, not seen by udevd, and should make settle block until they
are all handled.
But maybe udev_queue_export_new() needs to move above the daemonize?
It will create the initial queue file, that will make settle check for
the udevd vs. kernel number. That could be the reason, for the race.
Kay
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (12 preceding siblings ...)
2011-05-16 10:51 ` Kay Sievers
@ 2011-05-16 11:28 ` Kay Sievers
2011-05-16 12:47 ` Nix
14 siblings, 0 replies; 16+ messages in thread
From: Kay Sievers @ 2011-05-16 11:28 UTC (permalink / raw)
To: linux-hotplug
On Mon, May 16, 2011 at 12:51, Kay Sievers <kay.sievers@vrfy.org> wrote:
> On Mon, May 16, 2011 at 12:01, Nix <nix@esperi.org.uk> wrote:
>> On 16 May 2011, Kay Sievers outgrape:
>>> With devtmpfs you get the "early" /dev for free, and you preserve the
>>> "early" /dev in the real root. Instead of mounting a 'tmpfs' just
>>> mount a 'devtmpfs' at /dev, and it will be magically filled already,
>>> that's the only difference.
>>>
>>> When mounting the real root, just mount --move it over to the real
>>> root before switching to the real root.
>>
>> I'll have to see what busybox's switch_root does. I know it deletes
>> the entire contents of the root directory it switches away from: I'll
>> have to make sure that doesn't somehow include moved-away mountpoints.
>> (I doubt it does.)
>>
>>> Maybe you can run:
>>> Â udevadm monitor
>>> in the background writing to a file, and get timestamps of your commands too.
>>
>> They're fine.
>>
>> After a bunch of straces, I think the problem is different. As I
>> suspected from its intermittent nature, it's a race.
>>
>> When udevd starts up, it looks for an old database and converts it into
>> a new-style /run database. This takes nonzero time, and is done *after*
>> daemonization. Unfortunately, in the same time window it is quite
>> possible to fit a pair of 'udevadm trigger's and a 'udevadm settle': and
>> when udevadm settle kicks up and finds that udev hasn't even initialized
>> yet, what does it do? Well, unless I misread the code
>> udev_queue_get_queue_is_empty() returns 1 in this situation, so 'udevadm
>> settle' returns EXIT_SUCCESS, and boom.
>>
>> My evidence for this is very thin: it's no more than a hypothesis
>> really. My only evidence for this so far is that I haven't managed to
>> get the failure to happen unless I mkdir the old udev directories first
>> to force udev to do an old->new db conversion. It would be nice to
>> gather more data, but it's hard to do that without perturbing the race
>> :/ even an ls of /run slows it down enough that it doesn't happen
>> anymore. Hm, perhaps an echo would be fast enough, it's a shell
>> builtin after all...
>>
>> If I'm right (and I'm not sure I am, but it feels plausible), then I
>> suspect the only fix is to move the db conversion so that it happens
>> before daemonization.
>
> Hmm, the trigger should increase the kernel's current uevent seqnum.
> udev_queue_get_queue_is_empty() should see that there are events
> pending, not seen by udevd, and should make settle block until they
> are all handled.
>
> But maybe udev_queue_export_new() needs to move above the daemonize?
> It will create the initial queue file, that will make settle check for
> the udevd vs. kernel number. That could be the reason, for the race.
Commited a fix to git, that moves the creation of the queue file above
the daemonize. Hopefully that's the race you are seeing.
Kay
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: udevadm settle persistently failing
2011-05-15 18:19 udevadm settle persistently failing Nix
` (13 preceding siblings ...)
2011-05-16 11:28 ` Kay Sievers
@ 2011-05-16 12:47 ` Nix
14 siblings, 0 replies; 16+ messages in thread
From: Nix @ 2011-05-16 12:47 UTC (permalink / raw)
To: linux-hotplug
On 16 May 2011, Kay Sievers stated:
> Commited a fix to git, that moves the creation of the queue file above
> the daemonize. Hopefully that's the race you are seeing.
I'll try it later today, and scatter printf()s through the code to see
what path it's taking out if that doesn't work.
--
NULL && (void)
^ permalink raw reply [flat|nested] 16+ messages in thread