fastboot/async and initramfs: How am I supposed to know when devices are finished initializing?

fastboot/async and initramfs: How am I supposed to know when devices are finished initializing?

[Thomas Bächler]

Okay, I have this problem in 2.6.29 with the "fastboot" option and in 
today's linux-2.6.git.

In initramfs, we run the following commands:
     /sbin/udevd --daemon
     /sbin/udevadm trigger
     /sbin/udevadm settle
Among other things, this loads the ata_piix on my machine. On older 
kernels I could assume that after these commands, /dev/sda* existed and 
I could immediately access them (in my case, I run cryptsetup on 
/dev/sda6). But now, the devices don't exist here, but are only created 
a second or so later. As a result, the initramfs script has already 
bailed out as it couldn't find /dev/sda6 and assumed that the hard drive 
didn't exist.

So my question is, if udevadm settle doesn't wait for the devices 
anymore, how am I supposed to know when the devices have been created? 
If there is no way to know, I would consider this a regression, as it is 
no longer possible to cleanly write initramfs scripts.

Thanks for you replies
Thomas Bächler
Arch Linux Developer

Re: fastboot/async and initramfs: How am I supposed to know when devices are finished initializing?

[Arjan van de Ven]

On Sun, 29 Mar 2009 18:02:41 +0200
Thomas Bächler <thomas@archlinux.org> wrote:

> Okay, I have this problem in 2.6.29 with the "fastboot" option and in 
> today's linux-2.6.git.
> 
> In initramfs, we run the following commands:
>      /sbin/udevd --daemon
>      /sbin/udevadm trigger
>      /sbin/udevadm settle
> Among other things, this loads the ata_piix on my machine. On older 
> kernels I could assume that after these commands, /dev/sda* existed

not if you have CONFIG_SCSI_SCAN_ASYNC set though..

> and I could immediately access them (in my case, I run cryptsetup on 
> /dev/sda6). But now, the devices don't exist here, but are only
> created a second or so later. As a result, the initramfs script has
> already bailed out as it couldn't find /dev/sda6 and assumed that the
> hard drive didn't exist.

the CONFIG_SCSI_WAIT_SCAN method (basically loading that module to wait
for the scans to finish) will work for you......


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: fastboot/async and initramfs: How am I supposed to know when devices are finished initializing?

[Thomas Bächler]

[-- Attachment #1: Type: text/plain, Size: 998 bytes --]

Arjan van de Ven schrieb:
>> Among other things, this loads the ata_piix on my machine. On older 
>> kernels I could assume that after these commands, /dev/sda* existed
> 
> not if you have CONFIG_SCSI_SCAN_ASYNC set though..
> 
>> and I could immediately access them (in my case, I run cryptsetup on 
>> /dev/sda6). But now, the devices don't exist here, but are only
>> created a second or so later. As a result, the initramfs script has
>> already bailed out as it couldn't find /dev/sda6 and assumed that the
>> hard drive didn't exist.
> 
> the CONFIG_SCSI_WAIT_SCAN method (basically loading that module to wait
> for the scans to finish) will work for you......

Thanks, I will integrate that into our initramfs. Will that also work 
for USB mass storage (which was already problematic with older kernels, 
you load the module you don't know how long it takes until the sdX 
devices are created)?
Can I load that module before loading the actual SCSI/SATA/PATA driver?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: fastboot/async and initramfs: How am I supposed to know when devices are finished initializing?

[Kay Sievers]

On Sun, Mar 29, 2009 at 18:39, Thomas Bächler <thomas@archlinux.org> wrote:
> Arjan van de Ven schrieb:
>>>
>>> Among other things, this loads the ata_piix on my machine. On older
>>> kernels I could assume that after these commands, /dev/sda* existed
>>
>> not if you have CONFIG_SCSI_SCAN_ASYNC set though..
>>
>>> and I could immediately access them (in my case, I run cryptsetup on
>>> /dev/sda6). But now, the devices don't exist here, but are only
>>> created a second or so later. As a result, the initramfs script has
>>> already bailed out as it couldn't find /dev/sda6 and assumed that the
>>> hard drive didn't exist.
>>
>> the CONFIG_SCSI_WAIT_SCAN method (basically loading that module to wait
>> for the scans to finish) will work for you......
>
> Thanks, I will integrate that into our initramfs. Will that also work for
> USB mass storage (which was already problematic with older kernels, you load
> the module you don't know how long it takes until the sdX devices are
> created)?

USB is interrupt driven, there is and will never be such a thing as
"scanned" or "settled" -- anything can come and go at any time.

In initramfs, you have to wait until the device shows up, not for a
random module to initialize, or a bus to be scanned -- that can never
work correctly, it's pure luck, that your logic was always slower than
the kernel.

You need a block device -- so you should just wait for the block
device, instead of making assumptions about initialization of drivers
or buses. :)

Thanks,
Kay

Re: fastboot/async and initramfs: How am I supposed to know when devices are finished initializing?

[Thomas Bächler]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

Kay Sievers schrieb:
> In initramfs, you have to wait until the device shows up, not for a
> random module to initialize, or a bus to be scanned -- that can never
> work correctly, it's pure luck, that your logic was always slower than
> the kernel.
> 
> You need a block device -- so you should just wait for the block
> device, instead of making assumptions about initialization of drivers
> or buses. :)

I am planning to do that but there is one problem: Usually, the user 
specifies a root device, or a device that I want to open with 
cryptsetup, or anything similar and I can wait until it shows up.

For lvm however, the user does not specify a specific block device that 
I can wait for, instead lvm scans all available block devices. Now, how 
do I know that the block device that contains my physical volume is 
already there? Maybe I have several hard drives, and several volume 
groups, so should I now call vgchange -ay again and again until the 
right volume group shows up?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: fastboot/async and initramfs: How am I supposed to know when devices are finished initializing?

[Kay Sievers]

On Sun, Mar 29, 2009 at 19:52, Thomas Bächler <thomas@archlinux.org> wrote:
> Kay Sievers schrieb:
>>
>> In initramfs, you have to wait until the device shows up, not for a
>> random module to initialize, or a bus to be scanned -- that can never
>> work correctly, it's pure luck, that your logic was always slower than
>> the kernel.
>>
>> You need a block device -- so you should just wait for the block
>> device, instead of making assumptions about initialization of drivers
>> or buses. :)
>
> I am planning to do that but there is one problem: Usually, the user
> specifies a root device, or a device that I want to open with cryptsetup, or
> anything similar and I can wait until it shows up.

Right, you have to run the tools until the have found all what they
need to give you your root device. Please ask the device mapper
developers to port their stuff to the year 2009. :)

> For lvm however, the user does not specify a specific block device that I
> can wait for, instead lvm scans all available block devices. Now, how do I
> know that the block device that contains my physical volume is already
> there? Maybe I have several hard drives, and several volume groups, so
> should I now call vgchange -ay again and again until the right volume group
> shows up?

Yeah, today, you just call the "broken" tools in a loop until they
give you the device you wait for. :(

Assembling meta devices, or multi-volume devices should be triggered
directly when the underlying devices show up. If auto-assembly (which
kind of works today with md's --incremental) is properly implemented,
you should still be able to just wait for the specified root device to
show up, but the dm/lvm guys need to get the stuff done they talk
about for years now. :)

Thanks,
Kay

Re: fastboot/async and initramfs: How am I supposed to know when devices are finished initializing?

[Dan Williams]

On Sun, Mar 29, 2009 at 11:06 AM, Kay Sievers <kay.sievers@vrfy.org> wrote:
> Assembling meta devices, or multi-volume devices should be triggered
> directly when the underlying devices show up. If auto-assembly (which
> kind of works today with md's --incremental) is properly implemented,
> you should still be able to just wait for the specified root device to
> show up...

Incremental assembly mostly works but it would be nice to have some
event to signal "proceed with degraded assembly because I do not think
any more disks are coming".  With the below patch my thought is that
we can have a rule that:
1/ checks if incremental assembly is in progress (to prevent adding
timeouts to the non-raid path)
2/ checks if the udev event queue is empty
3/ upon finding an idle queue pause to allow last chance discovery
4/ assuming the queue remains idle proceed with degraded assembly

Thoughts?  Perhaps (3) can be eliminated after loading scsi_wait_scan.

Thanks,
Dan

---ws damaged patch---
settle: allow --timeout=0 to test for "queue busy"

From: Dan Williams <dan.j.williams@intel.com>

Allow udevadm settle to be used as a "queue busy" detector.  For use in
a early raid discovery where we want to hold off committing to the
current list of disks while devices might still be arriving.

Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
 udev/udevadm-settle.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/udev/udevadm-settle.c b/udev/udevadm-settle.c
index 867bbed..2fa85a4 100644
--- a/udev/udevadm-settle.c
+++ b/udev/udevadm-settle.c
@@ -85,6 +85,9 @@ int udevadm_settle(struct udev *udev, int argc, char *argv[])
 		usleep(1000 * 1000 / LOOP_PER_SECOND);
 	}

+	if (!udev_queue_get_queue_is_empty(udev_queue))
+		rc = 1;
+
 	/* if we reached the timeout, print the list of remaining events */
 	if (loop <= 0) {
 		struct udev_list_entry *list_entry;
@@ -97,7 +100,6 @@ int udevadm_settle(struct udev *udev, int argc, char *argv[])
 				       udev_list_entry_get_name(list_entry),
 				       udev_list_entry_get_value(list_entry));
 		}
-		rc = 1;
 	}
 exit:
 	udev_queue_unref(udev_queue);

Re: fastboot/async and initramfs: How am I supposed to know when devices are finished initializing?

[Kay Sievers]

On Mon, Mar 30, 2009 at 20:23, Dan Williams <dan.j.williams@intel.com> wrote:

> settle: allow --timeout=0 to test for "queue busy"

> +       if (!udev_queue_get_queue_is_empty(udev_queue))
> +               rc = 1;

You mean, we should not wait? Then we need to check for timeout == 0
and return immediately instead of entering the checking loop at all,
right?

Thanks,
Kay

Re: fastboot/async and initramfs: How am I supposed to know when devices are finished initializing?

[Dan Williams]

On Mon, Mar 30, 2009 at 11:42 AM, Kay Sievers <kay.sievers@vrfy.org> wrote:
> On Mon, Mar 30, 2009 at 20:23, Dan Williams <dan.j.williams@intel.com> wrote:
>
>> settle: allow --timeout=0 to test for "queue busy"
>
>> +       if (!udev_queue_get_queue_is_empty(udev_queue))
>> +               rc = 1;
>
> You mean, we should not wait? Then we need to check for timeout == 0
> and return immediately instead of entering the checking loop at all,
> right?
>

We already skip if loop==0...?

</me checks latest git>

Oh, this patch (against 138) needs to be reworked.

Thanks,
Dan

Re: fastboot/async and initramfs: How am I supposed to know when?devices are finished initializing?

[Alexander Clouter]

Thomas Bächler <thomas@archlinux.org> wrote:
> 
> Kay Sievers schrieb:
>> In initramfs, you have to wait until the device shows up, not for a
>> random module to initialize, or a bus to be scanned -- that can never
>> work correctly, it's pure luck, that your logic was always slower than
>> the kernel.
>> 
>> You need a block device -- so you should just wait for the block
>> device, instead of making assumptions about initialization of drivers
>> or buses. :)
> 
> I am planning to do that but there is one problem: Usually, the user 
> specifies a root device, or a device that I want to open with 
> cryptsetup, or anything similar and I can wait until it shows up.
> 
> For lvm however, the user does not specify a specific block device that 
> I can wait for, instead lvm scans all available block devices. Now, how 
> do I know that the block device that contains my physical volume is 
> already there? Maybe I have several hard drives, and several volume 
> groups, so should I now call vgchange -ay again and again until the 
> right volume group shows up?
> 
If the user is specifying the boot device, could you not 'teach' them to 
LABEL the boot device and live off something slightly more fancy (but 
akin to) the following:

----
while true
do
  mount LABEL=root /whatever

  if [ $? -eq 0 ]; then
    break
  fi

  # do some md and lvm magic dance here

  sleep 1
done
----

Your initramfs could do extra steps everytime the loop...erm...loops.

Now of course the downside is LABELing mounted devices, XFS for example, 
is not an easy thing to do... :-/

Cheers

-- 
Alexander Clouter
.sigmonster says: Am I ranting?  I hope so.  My ranting gets raves.

Re: fastboot/async and initramfs: How am I supposed to know when?devices are finished initializing?

[Thomas Bächler]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

Alexander Clouter schrieb:
> If the user is specifying the boot device, could you not 'teach' them to 
> LABEL the boot device and live off something slightly more fancy (but 
> akin to) the following:
> 
> ----
> while true
> do
>   mount LABEL=root /whatever
> 
>   if [ $? -eq 0 ]; then
>     break
>   fi
> 
>   # do some md and lvm magic dance here
> 
>   sleep 1
> done
> ----
> 
> Your initramfs could do extra steps everytime the loop...erm...loops.
> 
> Now of course the downside is LABELing mounted devices, XFS for example, 
> is not an easy thing to do... :-/

We do use UUIDs (or LABELs), but that doesn't solve the LVM problem, 
because the given device "/dev/somevg/somelv" only appears after LVM has 
been activated! I am thinking about using udev scan for LVM volume 
groups each time a hard drive appear that contains an LVM device. But 
that is a topic for another place.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: fastboot/async and initramfs: How am I supposed to know when devices are finished initializing?

[Thomas Bächler]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

Arjan van de Ven schrieb:
>> Thanks, I will integrate that into our initramfs. Will that also work 
>> for USB mass storage (which was already problematic with older
>> kernels, you load the module you don't know how long it takes until
>> the sdX devices are created)?
> 
> in part.
> 
> The problem with USB probing is that devices come online whenever they
> feel like it; there is no way to wait for that.

Yes, that is a problem when trying to make bootable USB pen drives.

>> Can I load that module before loading the actual SCSI/SATA/PATA
>> driver?
> 
> it's the loading of the driver that causes the wait. Not the act
> of having the driver loaded.
> 
> so the way to wait for things to settle is to load the driver...
> 
> ... I take it the answer to your question is "no". Yes you can load it
> earlier, but no it has no effect. You can however load/unload it
> multiple times.

I understand the logic now, works as expected on my machine. Many thanks 
for your quick help.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]