Re: Failover root devices

Re: Failover root devices

[Ortwin Glück]

 > I would like to see Linux support multiple root devices

You can do that completely in user space from an initramfs.

 From your init script you can do what you want. You may even parse /proc/cmdline and use the root= 
parameter as you propose. Then mount whatever root device you want by whatever method you like and 
finally
  exec switch_root /mnt/root /sbin/init "$@"

See here for example scripts for initramfs:
http://www.linuxfromscratch.org/blfs/view/svn/postlfs/initramfs.html

Above script actually makes it easy by supporting disk labels: root=LABEL=ROOT will boot the first 
available partition that is labelled ROOT. It is independent of the device name and works nice when 
you switch hardware vs. virtual machines for instance.

Ortwin

Re: Failover root devices

[Drew DeVault]

On 2015-09-17  1:40 PM, Ortwin Glück wrote:
> You can do that completely in user space from an initramfs.

Yep, I'm aware of that. I think it would still be useful for the kernel
to support it. Bonus - if the kernel supports it, there's a standard way
of doing it that would propegate down to the various initramfs designs
of the distros without having me write patches against all of them.
Right?

I'm working on an initramfs-based solution that meets my own needs, for
what it's worth.

> Above script actually makes it easy by supporting disk labels:
> root=LABEL=ROOT will boot the first available partition that is labelled
> ROOT. It is independent of the device name and works nice when you switch
> hardware vs. virtual machines for instance.

This is a neat idea, I'll check it out.

--
Drew DeVault

Re: Failover root devices

[Richard Weinberger]

On Thu, Sep 17, 2015 at 1:49 PM, Drew DeVault <sir@cmpwn.com> wrote:
> On 2015-09-17  1:40 PM, Ortwin Glück wrote:
>> You can do that completely in user space from an initramfs.
>
> Yep, I'm aware of that. I think it would still be useful for the kernel
> to support it. Bonus - if the kernel supports it, there's a standard way
> of doing it that would propegate down to the various initramfs designs
> of the distros without having me write patches against all of them.
> Right?

I really don't see why we need this feature in-kernel as it can be
done perfectly fine
in userspace. Every non-trivial system needs an initramfs anyway these days.

-- 
Thanks,
//richard

Re: Failover root devices

[Drew DeVault]

> I really don't see why we need this feature in-kernel as it can be
> done perfectly fine
> in userspace. Every non-trivial system needs an initramfs anyway these days.

Most initramfs systems will parse the root= line of your kernel command 
line and use it to remount root. The kernel seems like the best place to 
establish this, since the various initramfs solutions will implement the 
same format just to support parsing the kernel command line correctly. I 
mean, maybe that's not enough to justify putting it in the kernel, but 
that's why I see the kernel as the appropriate place to implement this 
feature.

--
Drew DeVault

Re: Failover root devices

[Richard Weinberger]

Am 17.09.2015 um 19:49 schrieb Drew DeVault:
>> I really don't see why we need this feature in-kernel as it can be
>> done perfectly fine
>> in userspace. Every non-trivial system needs an initramfs anyway these days.
> 
> Most initramfs systems will parse the root= line of your kernel command line and use it to remount root. The kernel seems like the best place to establish this, since the various
> initramfs solutions will implement the same format just to support parsing the kernel command line correctly. I mean, maybe that's not enough to justify putting it in the kernel,
> but that's why I see the kernel as the appropriate place to implement this feature.

Better send a patch to dracut folks. :-)
Major distros use it and if the feature is nice other initramfs implementations will adopt it too.

Thanks,
//richard

Re: Failover root devices

[Drew DeVault]

> Better send a patch to dracut folks. :-)
> Major distros use it and if the feature is nice other initramfs implementations will adopt it too.

dracut is the common one sure, but I'm still not confident that it's the 
right place to put this. How would that feature look? Would we have the 
root= parameter use a format that's specific to dracut and no longer a 
sane kernel parameter? Would we use a second parameter and discard the 
root= parameter? I think all of these are suboptimal solutions. No, the 
right way, I think, is to implement this in the kernel and let the init 
systems take it from there themselves.

--
Drew DeVault

Re: Failover root devices

[Richard Weinberger]

Am 17.09.2015 um 20:05 schrieb Drew DeVault:
>> Better send a patch to dracut folks. :-)
>> Major distros use it and if the feature is nice other initramfs implementations will adopt it too.
> 
> dracut is the common one sure, but I'm still not confident that it's the right place to put this. How would that feature look? Would we have the root= parameter use a format that's
> specific to dracut and no longer a sane kernel parameter? Would we use a second parameter and discard the root= parameter? I think all of these are suboptimal solutions. No, the
> right way, I think, is to implement this in the kernel and let the init systems take it from there themselves.

I'd patch dracut to support kernel command lines like "root=/dev/diskX root=/dev/diskY". initramfs is allowed to parse/use the command line.
So, you can do what you want.

Thanks,
//richard

Re: Failover root devices

[Drew DeVault]

> I'd patch dracut to support kernel command lines like "root=/dev/diskX root=/dev/diskY". initramfs is allowed to parse/use the command line.
> So, you can do what you want.

Is there precedent for using the same parameter name several times? That 
sounds dangerous and short-sighted.

--
Drew DeVault

Re: Failover root devices

[Richard Weinberger]

Am 17.09.2015 um 20:18 schrieb Drew DeVault:
>> I'd patch dracut to support kernel command lines like "root=/dev/diskX root=/dev/diskY". initramfs is allowed to parse/use the command line.
>> So, you can do what you want.
> 
> Is there precedent for using the same parameter name several times? That sounds dangerous and short-sighted.

Depends on how you patch dracut. ;-)
You can also add a "try_roots=/dev/diskX;/dev/diskY"...

Thanks,
//richard

Re: Failover root devices

[Drew DeVault]

> Depends on how you patch dracut. ;-)
> You can also add a "try_roots=/dev/diskX;/dev/diskY"...

That comes back around to my earlier email:

 >Would we use a second parameter and discard the root= parameter? I 
 >think all of these are suboptimal solutions.

I'm not a fan of this idea. What downside is there to implementing it in 
the kernel? It trickles down in a consistent way and bonus points for 
supporting systems that don't use an initramfs.

--
Drew DeVault

Re: Failover root devices

[Richard Weinberger]

Am 17.09.2015 um 20:21 schrieb Drew DeVault:
> I'm not a fan of this idea. What downside is there to implementing it in the kernel? It trickles down in a consistent way and bonus points for supporting systems that don't use an
> initramfs.

The kernel is already super complicated and has millions of lines of code. If it can be done nicely in userspace, let's do it there
and keep the kernel maintainable.

Thanks,
//richard

Re: Failover root devices

[Drew DeVault]

> The kernel is already super complicated and has millions of lines of code. If it can be done nicely in userspace, let's do it there
> and keep the kernel maintainable.

I don't think this is a strong argument against this feature. The 
implementation of this feature will be pretty straightfoward - only a 
small part of the code has to actually change its behavior and it can do 
without changing the interfaces it already relies on. On top of that, I 
don't think it can be done "nicely" in userspace anyway.

--
Drew DeVault

Re: Failover root devices

[Ortwin Glück]

On 17.09.2015 20:28, Drew DeVault wrote:
> I don't think this is a strong argument against this feature. The implementation of this feature
> will be pretty straightfoward - only a small part of the code has to actually change its behavior
> and it can do without changing the interfaces it already relies on. On top of that, I don't think it
> can be done "nicely" in userspace anyway.

I personally think this is opening a can of worms. Now it's just a list of alternative root devices. 
But the kernel knows absolutely nothing about these. When is it fine to try an alternative? Why did 
the first one not work? Did we just not wait long enough? Or is it a failed RAID device? Or is it an 
encrypted disk that needs setup? Or is it on NFS and the network is not available (or we are lacking 
driver firmware)?

It could actually introduce security problems: if I know that a device will fallback to an 
alternative root (under my control), I can try and DOS the primary root.

In short: if a simple root device doesn't work for you, you should *really* use an initramfs.

One could even argue to remove the boot= parameter altogether and always use initramfs :-)

Ortwin

Re: Failover root devices

[Drew DeVault]

> I personally think this is opening a can of worms. Now it's just a list
> of alternative root devices. But the kernel knows absolutely nothing
> about these. When is it fine to try an alternative? Why did the first
> one not work? Did we just not wait long enough? Or is it a failed RAID
> device? Or is it an encrypted disk that needs setup? Or is it on NFS and
> the network is not available (or we are lacking driver firmware)?

I don't think these are problems that aren't already inherent in a 
single root device via root=.

> It could actually introduce security problems: if I know that a device
> will fallback to an alternative root (under my control), I can try and
> DOS the primary root.

If you have physical access then the machine is yours to do with as you 
please.

--
Drew DeVault

Re: Failover root devices

[Ortwin Glück]

> If you have physical access then the machine is yours to do with as you please.

Thinking of ATMs or voting machines that is a bold statement :-)

Thinking of mobile phones it depends on your jurisdiction.

Ortwin

Re: Failover root devices

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1563 bytes --]

On 2015-09-18 11:04, Ortwin Glück wrote:
>> If you have physical access then the machine is yours to do with as
>> you please.
>
> Thinking of ATMs or voting machines that is a bold statement :-)
Many voting machines already have known ACE exploits already (I 
distinctly remember a while back some CS students demonstrated a 
'modern' voting machine playing PAC-Man without modifying any of the 
hardware at all), and those that have network access or other accessible 
peripheral connections are inherently insecure, period.

And most ATM's (at least in the US) run Windows (_shivers_) XP or 
eCommStation (the current commercial version of OS/2 (yes it still lives 
on), neither of which is particularly secure even when it comes to 
remote access to the system, and even then, the kind of access you need 
would involve3 directly tampering with the system.

Irrespective of that, neither one should be configured to work like 
that.  The intent is for custom setups primarily, if some company 
decides to use this in an insecure way, that's their problem, not ours 
(it's really easy to use a wide number of kernel features in ways that 
compromise security, that doesn't mean we should just rip those out).
>
> Thinking of mobile phones it depends on your jurisdiction.
This isn't a legal ruling, it's a simple statement of fact, if someone 
has physical access to a system, they effectively have root access, 
period.  While this is not probably what the above comment was directly 
referring to, it is an established fact.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Failover root devices

[Harald Hoyer]

On 17.09.2015 20:17, Richard Weinberger wrote:
> Am 17.09.2015 um 20:05 schrieb Drew DeVault:
>>> Better send a patch to dracut folks. :-)
>>> Major distros use it and if the feature is nice other initramfs implementations will adopt it too.
>>
>> dracut is the common one sure, but I'm still not confident that it's the right place to put this. How would that feature look? Would we have the root= parameter use a format that's
>> specific to dracut and no longer a sane kernel parameter? Would we use a second parameter and discard the root= parameter? I think all of these are suboptimal solutions. No, the
>> right way, I think, is to implement this in the kernel and let the init systems take it from there themselves.
> 
> I'd patch dracut to support kernel command lines like "root=/dev/diskX root=/dev/diskY". initramfs is allowed to parse/use the command line.
> So, you can do what you want.
> 
> Thanks,
> //richard


https://github.com/haraldh/dracut/blob/master/modules.d/95rootfs-block/rootfallback.sh

Re: Failover root devices

[Drew DeVault]

> https://github.com/haraldh/dracut/blob/master/modules.d/95rootfs-block/rootfallback.sh

Neat. Did you ever submit this upstream? Is there a discussion to read?

--
Drew DeVault

Re: Failover root devices

[Richard Weinberger]

Am 17.09.2015 um 20:29 schrieb Drew DeVault:
>> https://github.com/haraldh/dracut/blob/master/modules.d/95rootfs-block/rootfallback.sh
> 
> Neat. Did you ever submit this upstream? Is there a discussion to read?

Harald is the main developer...

Thanks,
//richard

Re: Failover root devices

[Drew DeVault]

> Harald is the main developer...

Hah, egg on my face. I still see one big issue with this - no waiting 
for the root device to come online. If the primary device is slow to get 
going, then it will be skipped in favor of the secondary device even if 
present.

--
Drew DeVault

Re: Failover root devices

[Richard Weinberger]

Am 17.09.2015 um 20:35 schrieb Drew DeVault:
>> Harald is the main developer...
> 
> Hah, egg on my face. I still see one big issue with this - no waiting for the root device to come online. If the primary device is slow to get going, then it will be skipped in
> favor of the secondary device even if present.

That's a good point. So, yeah I think a wait_for_dev is missing.
Harald?

Thanks,
//richard

Re: Failover root devices

[Richard Weinberger]

Am 17.09.2015 um 20:27 schrieb Harald Hoyer:
> On 17.09.2015 20:17, Richard Weinberger wrote:
>> Am 17.09.2015 um 20:05 schrieb Drew DeVault:
>>>> Better send a patch to dracut folks. :-)
>>>> Major distros use it and if the feature is nice other initramfs implementations will adopt it too.
>>>
>>> dracut is the common one sure, but I'm still not confident that it's the right place to put this. How would that feature look? Would we have the root= parameter use a format that's
>>> specific to dracut and no longer a sane kernel parameter? Would we use a second parameter and discard the root= parameter? I think all of these are suboptimal solutions. No, the
>>> right way, I think, is to implement this in the kernel and let the init systems take it from there themselves.
>>
>> I'd patch dracut to support kernel command lines like "root=/dev/diskX root=/dev/diskY". initramfs is allowed to parse/use the command line.
>> So, you can do what you want.
>>
>> Thanks,
>> //richard
> 
> 
> https://github.com/haraldh/dracut/blob/master/modules.d/95rootfs-block/rootfallback.sh

Awesome! :-)

Thanks,
//richard

Re: Failover root devices

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1009 bytes --]

On 2015-09-17 13:47, Richard Weinberger wrote:
> On Thu, Sep 17, 2015 at 1:49 PM, Drew DeVault <sir@cmpwn.com> wrote:
>> On 2015-09-17  1:40 PM, Ortwin Glück wrote:
>>> You can do that completely in user space from an initramfs.
>>
>> Yep, I'm aware of that. I think it would still be useful for the kernel
>> to support it. Bonus - if the kernel supports it, there's a standard way
>> of doing it that would propegate down to the various initramfs designs
>> of the distros without having me write patches against all of them.
>> Right?
>
> I really don't see why we need this feature in-kernel as it can be
> done perfectly fine
> in userspace. Every non-trivial system needs an initramfs anyway these days.
>
Ha, not unless you're using systemd.  I have more than 2 dozen servers 
with complex setups that boot just fine without an initramfs.  Yes there 
is more setup done in initramfs these days, but it's still not actually 
needed in most cases except complicated storage setups.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Failover root devices

[Richard Weinberger]

Am 17.09.2015 um 20:37 schrieb Austin S Hemmelgarn:
> On 2015-09-17 13:47, Richard Weinberger wrote:
>> On Thu, Sep 17, 2015 at 1:49 PM, Drew DeVault <sir@cmpwn.com> wrote:
>>> On 2015-09-17  1:40 PM, Ortwin Glück wrote:
>>>> You can do that completely in user space from an initramfs.
>>>
>>> Yep, I'm aware of that. I think it would still be useful for the kernel
>>> to support it. Bonus - if the kernel supports it, there's a standard way
>>> of doing it that would propegate down to the various initramfs designs
>>> of the distros without having me write patches against all of them.
>>> Right?
>>
>> I really don't see why we need this feature in-kernel as it can be
>> done perfectly fine
>> in userspace. Every non-trivial system needs an initramfs anyway these days.
>>
> Ha, not unless you're using systemd.  I have more than 2 dozen servers with complex setups that boot just fine without an initramfs.  Yes there is more setup done in initramfs
> these days, but it's still not actually needed in most cases except complicated storage setups.

I really don't count root=UUID... or root=LABEL... as complicated storage setup...

Thanks,
//richard

Re: Failover root devices

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1527 bytes --]

On 2015-09-17 14:40, Richard Weinberger wrote:
> Am 17.09.2015 um 20:37 schrieb Austin S Hemmelgarn:
>> On 2015-09-17 13:47, Richard Weinberger wrote:
>>> On Thu, Sep 17, 2015 at 1:49 PM, Drew DeVault <sir@cmpwn.com> wrote:
>>>> On 2015-09-17  1:40 PM, Ortwin Glück wrote:
>>>>> You can do that completely in user space from an initramfs.
>>>>
>>>> Yep, I'm aware of that. I think it would still be useful for the kernel
>>>> to support it. Bonus - if the kernel supports it, there's a standard way
>>>> of doing it that would propegate down to the various initramfs designs
>>>> of the distros without having me write patches against all of them.
>>>> Right?
>>>
>>> I really don't see why we need this feature in-kernel as it can be
>>> done perfectly fine
>>> in userspace. Every non-trivial system needs an initramfs anyway these days.
>>>
>> Ha, not unless you're using systemd.  I have more than 2 dozen servers with complex setups that boot just fine without an initramfs.  Yes there is more setup done in initramfs
>> these days, but it's still not actually needed in most cases except complicated storage setups.
>
> I really don't count root=UUID... or root=LABEL... as complicated storage setup...
>
> Thanks,
> //richard
>
That's not what I mean, I mean stuff like /usr and /var on separate 
filesystems, in a couple of cases self-assembling MD arrays, and in a 
couple of cases ATAoE or iSCSI backed root filesystems on hardware that 
doesn't natively support booting such devices.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Failover root devices

[Drew DeVault]

I would like to see Linux support multiple root devices, so that it can
attempt one and move on to the next if it is not present. I've reviewed
the relevant code during boot-up and it seems like a good place for me
to submit my first patch, but I want to bring it up for discussion here
on LKML first.

The design I had in mind is something like this:

root=device;device;device;...

Where 'device' follows the current format (/dev/sdX, UUIDs, and so on,
via name_to_dev_t). I would modify prepare_namespace to iterate through
each offered root device until one works.

My use-case for this feature is that I would like to be able to change
the hardware of my machine and boot up differently based on what's
present. In my case, I would like to install my system normally, with
/boot on its own partition, and keep a seperate userspace on a flash
drive. Then, during boot-up, if the flash drive is present, it would be
used as the root device. If it's not present, a partition on disk would
be selected.

The only potential roadblock with this feature that comes to mind is
figuring out how to handle time-outs between root devices. I think it
would be wise to choose a sensible default value, and provide another
cmdline parameter to tweak it. The prepare_namespace flow might end up
looking something like this:

1. Wait rootdelay seconds
2. Check 1st device, not present
3. Recheck 1st device until rootfailoverdelay seconds has passed
4. Move on to 2nd device, present -> boot

Or:

1. Wait rootdelay seconds
2. Check 1st device, not present
3. Recheck 1st device until rootfailoverdelay seconds has passed
4. Move on to 2nd device, not present
5. Recheck 2st device until rootfailoverdelay seconds has passed
6. GOTO 2

And so on.

I also need to research how the various init systems interact with this
part of the boot process. I suspect systemd probably does something
silly wrt waiting for the root device. Since this feature would (of
course) be backwards compatible, it might be wise to just implement it
here and let the init systems add support for the feature themselves.

Advice? Who should I send my patches to when they're ready? Please CC
me, I do not subscribe to LKML.

--
Drew DeVault

Re: Failover root devices

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 3496 bytes --]

On 2015-09-16 20:16, Drew DeVault wrote:
> I would like to see Linux support multiple root devices, so that it can
> attempt one and move on to the next if it is not present. I've reviewed
> the relevant code during boot-up and it seems like a good place for me
> to submit my first patch, but I want to bring it up for discussion here
> on LKML first.
>
> The design I had in mind is something like this:
>
> root=device;device;device;...
>
> Where 'device' follows the current format (/dev/sdX, UUIDs, and so on,
> via name_to_dev_t). I would modify prepare_namespace to iterate through
> each offered root device until one works.
>
> My use-case for this feature is that I would like to be able to change
> the hardware of my machine and boot up differently based on what's
> present. In my case, I would like to install my system normally, with
> /boot on its own partition, and keep a seperate userspace on a flash
> drive. Then, during boot-up, if the flash drive is present, it would be
> used as the root device. If it's not present, a partition on disk would
> be selected.
I think this is an excellent idea, in addition to the above use-case, it 
would allow for distros to automatically launch a recovery image if the 
main root device has failed for some reason.

That said, using the term failover for this is probably not the best 
idea, many people associate it almost exclusively with online failover 
and high-availability setups, and trying to do something like that with 
the root file system is just asking for trouble (I'll be happy to go 
into specifics as to why if someone asks).
> The only potential roadblock with this feature that comes to mind is
> figuring out how to handle time-outs between root devices. I think it
> would be wise to choose a sensible default value, and provide another
> cmdline parameter to tweak it. The prepare_namespace flow might end up
> looking something like this:
>
> 1. Wait rootdelay seconds
> 2. Check 1st device, not present
> 3. Recheck 1st device until rootfailoverdelay seconds has passed
> 4. Move on to 2nd device, present -> boot
>
> Or:
>
> 1. Wait rootdelay seconds
> 2. Check 1st device, not present
> 3. Recheck 1st device until rootfailoverdelay seconds has passed
> 4. Move on to 2nd device, not present
> 5. Recheck 2st device until rootfailoverdelay seconds has passed
> 6. GOTO 2
>
> And so on.
As for this, I'd say default to the first method, and then provide an 
option to switch to the second (both have practical uses).
> I also need to research how the various init systems interact with this
> part of the boot process. I suspect systemd probably does something
> silly wrt waiting for the root device. Since this feature would (of
> course) be backwards compatible, it might be wise to just implement it
> here and let the init systems add support for the feature themselves.
If you're using an initramfs (which is a requirement from what I 
understand for using systemd), then this could be done entirely in the 
initramfs.  The issue with that is that there is no standard syntax for 
doing it, and no way to do it without an initramfs (both of which would 
be nice to have).
> Advice? Who should I send my patches to when they're ready? Please CC
> me, I do not subscribe to LKML.
Use scripts/getmaintainer.pl (or just check the MAINTAINERS file 
directly) to determine this, but make sure to Cc at least LKML for the 
changes as well.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Failover root devices

[Drew DeVault]

> That said, using the term failover for this is probably not the best
> idea, many people associate it almost exclusively with online failover
> and high-availability setups, and trying to do something like that with
> the root file system is just asking for trouble (I'll be happy to go
> into specifics as to why if someone asks).

Do you have a suggestion for another name for this feature? Maybe we can 
just call it "multiple root devices". The issue comes with the 
associated command line options, like "rootfailoverdelay". Perhaps it 
could be called "rootcycledelay". "rootdelay" is the obvious one, but 
it's taken for another feature.

>> 1. Wait rootdelay seconds
>> 2. Check 1st device, not present
>> 3. Recheck 1st device until rootfailoverdelay seconds has passed
>> 4. Move on to 2nd device, present -> boot
>>
>> Or:
>>
>> 1. Wait rootdelay seconds
>> 2. Check 1st device, not present
>> 3. Recheck 1st device until rootfailoverdelay seconds has passed
>> 4. Move on to 2nd device, not present
>> 5. Recheck 2st device until rootfailoverdelay seconds has passed
>> 6. GOTO 2
>>
>> And so on.
> As for this, I'd say default to the first method, and then provide an
> option to switch to the second (both have practical uses).

Sorry to cause confusion - these are actually the same method, but 
handling different scenarios. The first is dealing with the first device 
being nonexistent, and the second device existing. The second is dealing 
with both being nonexistent, and cycling between them until one of them 
shows up. After further thought, though, I think the best solution is a 
bit different: a new command line option called "rootmultiwait" or 
similar, which is a maximum amount of time to wait for the user's first 
choice of root device to become available, then testing all devices 
until that time runs out or the first choice becomes available.

--
Drew DeVault

Re: Failover root devices

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 3177 bytes --]

On 2015-09-17 13:30, Drew DeVault wrote:
>> That said, using the term failover for this is probably not the best
>> idea, many people associate it almost exclusively with online failover
>> and high-availability setups, and trying to do something like that with
>> the root file system is just asking for trouble (I'll be happy to go
>> into specifics as to why if someone asks).
>
> Do you have a suggestion for another name for this feature? Maybe we can
> just call it "multiple root devices". The issue comes with the
> associated command line options, like "rootfailoverdelay". Perhaps it
> could be called "rootcycledelay". "rootdelay" is the obvious one, but
> it's taken for another feature.
Possibly 'multirootdelay'?

However, is there any case you can think of for wanting the values to be 
different between rootdelay and the per-device scan delay other than 
having the per-device scan delay be 0 and rootdelay be >0?

The way I'd probably write it would be:
1. Wait rootdelay seconds
2. Check for 1st device
3. If first device is not there, check for 2nd
4. If second device is not there, check next one
5. Repeat 4 until all devices are checked.
6. If a device wasn't found, check if we were told to loop until one is 
found, and if so, start at 1 again.
And then add an option to tell it to wait 'rootdelay' seconds between 
checking each device.
>
>>> 1. Wait rootdelay seconds
>>> 2. Check 1st device, not present
>>> 3. Recheck 1st device until rootfailoverdelay seconds has passed
>>> 4. Move on to 2nd device, present -> boot
>>>
>>> Or:
>>>
>>> 1. Wait rootdelay seconds
>>> 2. Check 1st device, not present
>>> 3. Recheck 1st device until rootfailoverdelay seconds has passed
>>> 4. Move on to 2nd device, not present
>>> 5. Recheck 2st device until rootfailoverdelay seconds has passed
>>> 6. GOTO 2
>>>
>>> And so on.
>> As for this, I'd say default to the first method, and then provide an
>> option to switch to the second (both have practical uses).
>
> Sorry to cause confusion - these are actually the same method, but
> handling different scenarios. The first is dealing with the first device
> being nonexistent, and the second device existing. The second is dealing
> with both being nonexistent, and cycling between them until one of them
> shows up. After further thought, though, I think the best solution is a
> bit different: a new command line option called "rootmultiwait" or
> similar, which is a maximum amount of time to wait for the user's first
> choice of root device to become available, then testing all devices
> until that time runs out or the first choice becomes available.
I think there's value in being able to tell it to go through each one 
exactly once, and halt like it does now if it can't find the filesystem 
on any of them.  That should probably be the default behavior in fact, 
as it's more similar to what's done now.

Secondarily, I've been thinking more about this, and I think it would be 
wonderful to have such functionality in the nfsroot code as well (and 
for that matter, also in any other built-in networked root filesystem 
support).


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Failover root devices

[Drew DeVault]

> Possibly 'multirootdelay'?

I had the same thought, but wanted to avoid using any prefix other than 
root*= since it would break tradition for this part of the kernel.

> However, is there any case you can think of for wanting the values to be
> different between rootdelay and the per-device scan delay other than
> having the per-device scan delay be 0 and rootdelay be >0?

rootdelay is not really a part of this flow, to be honest. It's a number 
of seconds that it just blocks before trying any devices at all. It 
might not be necessary to make them seperate, though. What's the 
"per-device scan delay" you're speaking of?

> The way I'd probably write it would be:
> 1. Wait rootdelay seconds
> 2. Check for 1st device
> 3. If first device is not there, check for 2nd
> 4. If second device is not there, check next one
> 5. Repeat 4 until all devices are checked.
> 6. If a device wasn't found, check if we were told to loop until one is
> found, and if so, start at 1 again.
> And then add an option to tell it to wait 'rootdelay' seconds between
> checking each device.

-snip-

 > I think there's value in being able to tell it to go through each one
 > exactly once, and halt like it does now if it can't find the filesystem
 > on any of them.  That should probably be the default behavior in fact,
 > as it's more similar to what's done now.

What is the behavior if we weren't told to loop, but reach the end of 
the list? I don't want to try one device for a while and move on - I'd 
prefer to try all devices, then loop. With the other strategy, what 
happens if you try the first device for a while, then move on to the 
second, and the first device comes up while you're waiting on the 
second? The way the user would want that to play out is to boot the 
first device (since it is their preference, after all), but the actual 
behavior will be to boot the second if it comes up during this time.

> Secondarily, I've been thinking more about this, and I think it would be
> wonderful to have such functionality in the nfsroot code as well (and
> for that matter, also in any other built-in networked root filesystem
> support).

I agree that it would be great, but there be dragons. I'm nervous to 
attempt that in my first patch (or even my first few).

--
Drew DeVault