module 90kernel-modules-loaded

module 90kernel-modules-loaded

[Harald Hoyer]

Added 90kernel-modules-loaded to be used instead or additionally to 
90kernel-modules, which just installs all currently loaded modules to initrd.

branch "merged" on:

http://git.surfsite.org/dracut.git
git://surfsite.org/pub/git/dracut.git
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Bill Nottingham]

Harald Hoyer (harald-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org) said: 
> Added 90kernel-modules-loaded to be used instead or additionally to  
> 90kernel-modules, which just installs all currently loaded modules to 
> initrd.
>
> branch "merged" on:
>
> http://git.surfsite.org/dracut.git
> git://surfsite.org/pub/git/dracut.git

Wasn't the entire point to make the initramfs generic?

Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Harald Hoyer]

Bill Nottingham wrote:
> Harald Hoyer (harald-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org) said: 
>> Added 90kernel-modules-loaded to be used instead or additionally to  
>> 90kernel-modules, which just installs all currently loaded modules to 
>> initrd.
>>
>> branch "merged" on:
>>
>> http://git.surfsite.org/dracut.git
>> git://surfsite.org/pub/git/dracut.git
> 
> Wasn't the entire point to make the initramfs generic?
> 
> Bill

if haven't removed the generic 90kernel-modules, which still is used by default
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Victor Lowther]

On Fri, 2009-03-06 at 11:08 -0500, Bill Nottingham wrote:
> Harald Hoyer (harald-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org) said: 
> > Added 90kernel-modules-loaded to be used instead or additionally to  
> > 90kernel-modules, which just installs all currently loaded modules to 
> > initrd.
> >
> > branch "merged" on:
> >
> > http://git.surfsite.org/dracut.git
> > git://surfsite.org/pub/git/dracut.git
> 
> Wasn't the entire point to make the initramfs generic?

No, to make the initramfs generator generic. A subtle but important
distinction.

> Bill
> --
> To unsubscribe from this list: send the line "unsubscribe initramfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Victor Lowther
RHCE# 805008539634727
LPIC-2# LPI000140019

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Bill Nottingham]

Victor Lowther (victor.lowther-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) said: 
> > > http://git.surfsite.org/dracut.git
> > > git://surfsite.org/pub/git/dracut.git
> > 
> > Wasn't the entire point to make the initramfs generic?
> 
> No, to make the initramfs generator generic. A subtle but important
> distinction.

Nope. The plan was (quoting earlier mails to the list, from the
creators...):

...
[I]nstead of scripts hard-coded to do various things, we depend on
udev to create device nodes for us and then when we have the
rootfs's device node, we mount and carry on
...
There's another reason this is really useful.
If something goes wrong, remotely debugging a users initrd right is
a lot easier if you know what it looks like.  Right now, in Fedora for eg,
where we generate an initrd for each users system at runtime, we need
to get a copy of the generated initrd, and pull it apart just to find
out what modules ended up in there, what didn't, and then somehow
try to work backwards to try and figure out how the generator got into
that state.  After doing this for five years, let me tell you it's
_really_ _really_ painful.
...

And now we've got large patchsets that:
- make the initramfs non-generic (in fact, machine-specific)
- handle various device types through shell snippets, not udev
  (raid, crypt, etc.)
- farm everything out to random modules, so no initramfs is alike

I think something's been lost here.

Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Victor Lowther]

On Fri, 2009-03-06 at 15:39 -0500, Bill Nottingham wrote:
> Victor Lowther (victor.lowther-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) said: 
> > > > http://git.surfsite.org/dracut.git
> > > > git://surfsite.org/pub/git/dracut.git
> > > 
> > > Wasn't the entire point to make the initramfs generic?
> > 
> > No, to make the initramfs generator generic. A subtle but important
> > distinction.
> 
> Nope. The plan was (quoting earlier mails to the list, from the
> creators...):
> 
> ...
> [I]nstead of scripts hard-coded to do various things, we depend on
> udev to create device nodes for us and then when we have the
> rootfs's device node, we mount and carry on

We do this just fine.  Udev is the engine that drives everything in a
dracut-generated initramfs -- the additional scripts are there in the
initramfs to handle things that udev does not handle gracefully. If you
see some functionality currently handles in a script that udev could
handle better, please implement it.

> ...
> There's another reason this is really useful.
> If something goes wrong, remotely debugging a users initrd right is
> a lot easier if you know what it looks like.  Right now, in Fedora for eg,
> where we generate an initrd for each users system at runtime, we need
> to get a copy of the generated initrd, and pull it apart just to find
> out what modules ended up in there, what didn't, and then somehow
> try to work backwards to try and figure out how the generator got into
> that state.  After doing this for five years, let me tell you it's
> _really_ _really_ painful.

Dracut should eventually be able to generate a single initramfs that
RHEL6 (for example) could distribute (per kernel, of course) that would
load on any supported server without having to be generated specifically
for that server. Assuming all you wanted to do was boot off a local
block device, we should be able to do that right now. 

> ...
> 
> And now we've got large patchsets that:
> - make the initramfs non-generic (in fact, machine-specific)

Yep.  Not everyone wants a generic does-everything initramfs. dracut is
flexible enough to handle their needs.

> - handle various device types through shell snippets, not udev
>   (raid, crypt, etc.)

We use shell snippets where udev is not flexible enough or where there
are other limitations we have to work around.  The entire process on the
initramfs is still udev driven, however.

> - farm everything out to random modules, so no initramfs is alike

That is up to whoever is generating the initramfs.  If someone wants to
tweak their initramfs, I have no problem letting them, and I do not
think dracut should get in their way.  At the same time, we want to be
able to generate an initrd that will work on darn near everything and
whose only customization can be through kernel parameters.

The current patching activity has been a little crazy, but we have
managed to take dracut from a Fedora-specifc proof of concept to
something that can build a working initramfs on fedora, suse, ubuntu and
(maybe) mandriva with minimal to no modification.  Not bad for a month's
mostly volunteer work.

> I think something's been lost here.
> 
> Bill
-- 
Victor Lowther
RHCE# 805008539634727
LPIC-2# LPI000140019

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Seewer Philippe]

Bill Nottingham wrote:
[snip]
> ...
> 
> And now we've got large patchsets that:
> - make the initramfs non-generic (in fact, machine-specific)
> - handle various device types through shell snippets, not udev
>   (raid, crypt, etc.)
> - farm everything out to random modules, so no initramfs is alike
> 
> I think something's been lost here.

Just my 50 cents: The past few weeks have really been focused on 
creating a basic dracut infrastructure and getting various crazy and 
convoluted system-configurations to boot with a dracut _generated_ 
initramfs. And we're not even finished with that yet.

I view the current state as "collecting information" and prototyping to 
know which things need to be done inside the initrd and test out various 
combinations in how that can be done. But we're still a few steps from 
"the initrd of total domination".

Regards,
Philippe

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Seewer Philippe]

Bill, a question: How far do you think the genericness of dracut should go?

For example if we really build the initrd of doom that builds once and 
boots anywhere we'd have to include booting from net, usb, etc. as well. 
But say, doing dhcp isn't always necessary and should really only be 
done if root is on a net device. That would mean adding tons of drivers 
and another layer of complex code to decide at runtime what parts of the 
initrd to run. For net-boots I couldn't care less about boot time and 
the size of the initrd. But for other cases? What do you think?

Thanks,
Philippe
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Jeremy Katz]

On Friday, March 06 2009, Victor Lowther said:
> On Fri, 2009-03-06 at 15:39 -0500, Bill Nottingham wrote:
> > Victor Lowther (victor.lowther-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) said: 
> > > > > http://git.surfsite.org/dracut.git
> > > > > git://surfsite.org/pub/git/dracut.git
> > > > 
> > > > Wasn't the entire point to make the initramfs generic?
> > > 
> > > No, to make the initramfs generator generic. A subtle but important
> > > distinction.
> > 
> > Nope. The plan was (quoting earlier mails to the list, from the
> > creators...):
> > 
> > ...
> > [I]nstead of scripts hard-coded to do various things, we depend on
> > udev to create device nodes for us and then when we have the
> > rootfs's device node, we mount and carry on
> 
> We do this just fine.  Udev is the engine that drives everything in a
> dracut-generated initramfs -- the additional scripts are there in the
> initramfs to handle things that udev does not handle gracefully. If you
> see some functionality currently handles in a script that udev could
> handle better, please implement it.

The problem is the direction seems to be "hm, udev can't do this, guess
we'll make a hook" rather than "fix udev and the underlying tools".
Yes, it means it's harder.  And yes it means that distros have to get
updates for everything to work.  But otherwise, we remain in the past
with piles of scripts doing most things.

[snip]
> > - handle various device types through shell snippets, not udev
> >   (raid, crypt, etc.)
> 
> We use shell snippets where udev is not flexible enough or where there
> are other limitations we have to work around.  The entire process on the
> initramfs is still udev driven, however.

Honestly, I really wouldn't say that "the entire process is still udev
driven".  The rootfs *NEEDS* to be being mounted from a udev rule --
otherwise, we're not being driven by events.  We're having events to do
some things and scripts for others/more

Jeremy
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Victor Lowther]

On Sun, 2009-03-08 at 23:11 -0400, Jeremy Katz wrote:
> On Friday, March 06 2009, Victor Lowther said:
> > On Fri, 2009-03-06 at 15:39 -0500, Bill Nottingham wrote:
> > > [I]nstead of scripts hard-coded to do various things, we depend on
> > > udev to create device nodes for us and then when we have the
> > > rootfs's device node, we mount and carry on
> > 
> > We do this just fine.  Udev is the engine that drives everything in a
> > dracut-generated initramfs -- the additional scripts are there in the
> > initramfs to handle things that udev does not handle gracefully. If you
> > see some functionality currently handles in a script that udev could
> > handle better, please implement it.
> 
> The problem is the direction seems to be "hm, udev can't do this, guess
> we'll make a hook" rather than "fix udev and the underlying tools".
> Yes, it means it's harder.  And yes it means that distros have to get
> updates for everything to work.  But otherwise, we remain in the past
> with piles of scripts doing most things.

udev and the underlying tools are not broken.  Udev names devices
according to rules and calls out to external programs or scripts for
everything else.  It does that job very well.

> [snip]
> > > - handle various device types through shell snippets, not udev
> > >   (raid, crypt, etc.)
> > 
> > We use shell snippets where udev is not flexible enough or where there
> > are other limitations we have to work around.  The entire process on the
> > initramfs is still udev driven, however.
> 
> Honestly, I really wouldn't say that "the entire process is still udev
> driven".  The rootfs *NEEDS* to be being mounted from a udev rule --
> otherwise, we're not being driven by events.  We're having events to do
> some things and scripts for others/more

OK, there are a few ways that spring to mind to do this:

1: Teach udev how to interpret all the kernel parameters that might be
involved in detecting, performing all the prerequisites for mounting,
and then mounting the root filesystem.  Package all this logic up into
something that can can be driven by a small number of new rules.

While that sounds tempting, it seems to add quite alot of magic to udev
that it currently does not have and that is well outside its core focus
of device naming -- or does udev perform any device configuration that
it does not call out to an external executable or a shell script to do?

2: Include enough logic on the initramfs to parse all the kernel
parameters that might be involved in booting the system and then
custom-write udev rules that will do all the work of making the root
filesystem available (assuming it is available at all) inside the
initramfs.

That sounds like loads of extra work -- we will have to write a script
or program written in C whose job is to parse kernel parameters,
custom-write udev rules based on the ones that might be involved in
booting, and inject them into udevd and hope for the best.

3: As above, but create our custom rules at initramfs generation time
instead of at boot time.

This makes a generic initrd impossible to create, so I will discard it
out of hand.

4: Add a few udev rules that call scripts to configure a few key device
types as they are detected, and have another process running that looks
for the root filesystem to appear running in tandem with udev.

This is what we currently have, and it seems to be doing very well.

Of all the approaches, only the first meets your *NEEDS* requirement,
and it involves tacking a shedload of extra capabilities onto udev that
it will only ever use while booting the system. The second sounds even
harder to debug than any of the current initramfs schemes, and the third
is too brittle and actually worse than the current redhat/fedora
initramfs scheme. We have made large advances using the fourth approach
-- sorry they were not in the direction you initially hoped for.


> Jeremy
> --
> To unsubscribe from this list: send the line "unsubscribe initramfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Victor Lowther
RHCE# 805008539634727
LPIC-2# LPI000140019

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Bogdan Costescu]

On Sat, 7 Mar 2009, Seewer Philippe wrote:

> That would mean adding tons of drivers and another layer of complex 
> code to decide at runtime what parts of the initrd to run.

I could argue the same for the drivers for the block devices and their 
tools. If the root is using NFS, there is no need for a block device 
detection; this however is still needed for iSCSI (and possibly FCoE 
in the future).

Maybe allow at build time to specify what boot drivers+tools to be 
included ? This would certainly reduce the generic-ness of the 
initramfs, but would also reduce the size and complexity.

> For net-boots I couldn't care less about boot time and the size of 
> the initrd.

I do. When I boot 100+ cluster nodes at once from a single server, I 
want to minimize the network traffic at a minimum, especially a part 
that happens over UDP.

-- 
Bogdan Costescu

IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8240, Fax: +49 6221 54 8850
E-mail: bogdan.costescu-hEciA7+sKtudPOQpRHQ53DeJuz7u0hKX@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Harald Hoyer]

Jeremy Katz wrote:
> On Friday, March 06 2009, Victor Lowther said:
>> On Fri, 2009-03-06 at 15:39 -0500, Bill Nottingham wrote:
>>> Victor Lowther (victor.lowther-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) said: 
>>>>>> http://git.surfsite.org/dracut.git
>>>>>> git://surfsite.org/pub/git/dracut.git
>>>>> Wasn't the entire point to make the initramfs generic?
>>>> No, to make the initramfs generator generic. A subtle but important
>>>> distinction.
>>> Nope. The plan was (quoting earlier mails to the list, from the
>>> creators...):
>>>
>>> ...
>>> [I]nstead of scripts hard-coded to do various things, we depend on
>>> udev to create device nodes for us and then when we have the
>>> rootfs's device node, we mount and carry on
>> We do this just fine.  Udev is the engine that drives everything in a
>> dracut-generated initramfs -- the additional scripts are there in the
>> initramfs to handle things that udev does not handle gracefully. If you
>> see some functionality currently handles in a script that udev could
>> handle better, please implement it.
> 
> The problem is the direction seems to be "hm, udev can't do this, guess
> we'll make a hook" rather than "fix udev and the underlying tools".
> Yes, it means it's harder.  And yes it means that distros have to get
> updates for everything to work.  But otherwise, we remain in the past
> with piles of scripts doing most things.
>

Problems so far:
	- dmraid (raid assembly)
	- device mapper handled /dev/ symlinks
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Hannes Reinecke]

Hi all,

Harald Hoyer wrote:
> Jeremy Katz wrote:
>> On Friday, March 06 2009, Victor Lowther said:
>>> On Fri, 2009-03-06 at 15:39 -0500, Bill Nottingham wrote:
>>>> Victor Lowther (victor.lowther-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) said:
>>>>>>> http://git.surfsite.org/dracut.git
>>>>>>> git://surfsite.org/pub/git/dracut.git
>>>>>> Wasn't the entire point to make the initramfs generic?
>>>>> No, to make the initramfs generator generic. A subtle but important
>>>>> distinction.
>>>> Nope. The plan was (quoting earlier mails to the list, from the
>>>> creators...):
>>>>
>>>> ...
>>>> [I]nstead of scripts hard-coded to do various things, we depend on
>>>> udev to create device nodes for us and then when we have the
>>>> rootfs's device node, we mount and carry on
>>> We do this just fine.  Udev is the engine that drives everything in a
>>> dracut-generated initramfs -- the additional scripts are there in the
>>> initramfs to handle things that udev does not handle gracefully. If you
>>> see some functionality currently handles in a script that udev could
>>> handle better, please implement it.
>>
>> The problem is the direction seems to be "hm, udev can't do this, guess
>> we'll make a hook" rather than "fix udev and the underlying tools".
>> Yes, it means it's harder.  And yes it means that distros have to get
>> updates for everything to work.  But otherwise, we remain in the past
>> with piles of scripts doing most things.
>>
> 
> Problems so far:
>     - dmraid (raid assembly)
>     - device mapper handled /dev/ symlinks

That's hardly a udev issue. We should rather be fixing device-mapper to
work together with udev.

But yes, dmraid (and multipath) are the hard problems.
dmraid is probably okay, as you can eg just add a '--export' to dmraid
to print out all found signatures. Then write some rules which latch
on these exported variables and start dmraid if all component devices
have been discovered. Easily done with eg the 'collect' utility from
udev extras.
Problem with multipath here is that you can't detect a multipathed
device a priori. It's basically a policy decision whether the user
wants to start multipath or not. So you basically have to write
some device-specific udev rules for enabling multipath on some
devices.
The hard part here is to remove the component device from the list
of eligible devices so that eg LVM doesn't erroneously tries to
scan from a multipath or dmraid component device.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare-l3A5Bk7waGM@public.gmane.org			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Seewer Philippe]

Bogdan Costescu wrote:
> On Sat, 7 Mar 2009, Seewer Philippe wrote:
> 
>> That would mean adding tons of drivers and another layer of complex 
>> code to decide at runtime what parts of the initrd to run.
> 
> I could argue the same for the drivers for the block devices and their 
> tools. If the root is using NFS, there is no need for a block device 
> detection; this however is still needed for iSCSI (and possibly FCoE in 
> the future).

Exactly.

> 
> Maybe allow at build time to specify what boot drivers+tools to be 
> included ? This would certainly reduce the generic-ness of the 
> initramfs, but would also reduce the size and complexity.

dracut can already do this. I was speaking about an initrd of doom which 
would include all manners of drivers and tools to boot from/to anything.

> 
>> For net-boots I couldn't care less about boot time and the size of the 
>> initrd.
> 
> I do. When I boot 100+ cluster nodes at once from a single server, I 
> want to minimize the network traffic at a minimum, especially a part 
> that happens over UDP.

He. point taken.

> 
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Jeremy Katz]

On Monday, March 09 2009, Victor Lowther said:
> On Sun, 2009-03-08 at 23:11 -0400, Jeremy Katz wrote:
> > On Friday, March 06 2009, Victor Lowther said:
> > > On Fri, 2009-03-06 at 15:39 -0500, Bill Nottingham wrote:
> > > > [I]nstead of scripts hard-coded to do various things, we depend on
> > > > udev to create device nodes for us and then when we have the
> > > > rootfs's device node, we mount and carry on
> > > 
> > > We do this just fine.  Udev is the engine that drives everything in a
> > > dracut-generated initramfs -- the additional scripts are there in the
> > > initramfs to handle things that udev does not handle gracefully. If you
> > > see some functionality currently handles in a script that udev could
> > > handle better, please implement it.
> > 
> > The problem is the direction seems to be "hm, udev can't do this, guess
> > we'll make a hook" rather than "fix udev and the underlying tools".
> > Yes, it means it's harder.  And yes it means that distros have to get
> > updates for everything to work.  But otherwise, we remain in the past
> > with piles of scripts doing most things.
> 
> udev and the underlying tools are not broken.  Udev names devices
> according to rules and calls out to external programs or scripts for
> everything else.  It does that job very well.

Certainly md and dm have their fair share of brokenness when it comes to
(lack of) udev integration. :)
 
> > [snip]
> > > > - handle various device types through shell snippets, not udev
> > > >   (raid, crypt, etc.)
> > > 
> > > We use shell snippets where udev is not flexible enough or where there
> > > are other limitations we have to work around.  The entire process on the
> > > initramfs is still udev driven, however.
> > 
> > Honestly, I really wouldn't say that "the entire process is still udev
> > driven".  The rootfs *NEEDS* to be being mounted from a udev rule --
> > otherwise, we're not being driven by events.  We're having events to do
> > some things and scripts for others/more
> 
> OK, there are a few ways that spring to mind to do this:
> 
> 1: Teach udev how to interpret all the kernel parameters that might be
> involved in detecting, performing all the prerequisites for mounting,
> and then mounting the root filesystem.  Package all this logic up into
> something that can can be driven by a small number of new rules.
> 
> While that sounds tempting, it seems to add quite alot of magic to udev
> that it currently does not have and that is well outside its core focus
> of device naming -- or does udev perform any device configuration that
> it does not call out to an external executable or a shell script to do?

There's nothing that says that you don't end up using callouts for some
of this stuff.  The rule syntax is a little brittle (you can't have
substitutions on both the LHS and the RHS of a conditional iirc, for one
example) but expanding some of those might well make sense for more
cases.

And udev *is* growing more capabilities as part of its integration with
halv2^W DeviceKit

> 2: Include enough logic on the initramfs to parse all the kernel
> parameters that might be involved in booting the system and then
> custom-write udev rules that will do all the work of making the root
> filesystem available (assuming it is available at all) inside the
> initramfs.
> 
> That sounds like loads of extra work -- we will have to write a script
> or program written in C whose job is to parse kernel parameters,
> custom-write udev rules based on the ones that might be involved in
> booting, and inject them into udevd and hope for the best.

I'm not a huge fan of this, it's been done in the past (see
mayflower/mkliveinitrd in Fedora-land, other things in other places).
That said, it actually worked surprisingly well.  The problem is
carrying it over to the real system as you'd like to have a consistent
set of rules for both

> 3: As above, but create our custom rules at initramfs generation time
> instead of at boot time.
> 
> This makes a generic initrd impossible to create, so I will discard it
> out of hand.

Indeed! :)

> 4: Add a few udev rules that call scripts to configure a few key device
> types as they are detected, and have another process running that looks
> for the root filesystem to appear running in tandem with udev.
> 
> This is what we currently have, and it seems to be doing very well.
> 
> Of all the approaches, only the first meets your *NEEDS* requirement,
> and it involves tacking a shedload of extra capabilities onto udev that
> it will only ever use while booting the system. The second sounds even
> harder to debug than any of the current initramfs schemes, and the third
> is too brittle and actually worse than the current redhat/fedora
> initramfs scheme. We have made large advances using the fourth approach
> -- sorry they were not in the direction you initially hoped for.

And I really am glad/pleased/impressed with all the progress.  I'm just
saying to step back and (... to pick on a red-headed step child ;),
before going off and saying "well, dmraid doesn't work with udev, so
we'll do it with a hook", the question really should be "how can we get
<foo> to work within udev to create us a /dev/root symlink" and then be
able to use that rather than having arbitrary hooks.  Even the bits to
do LABEL/UUID munging in the first drop of the code were intended to be
at best a short-term workaround... maybe having them at all ended up
leading things astray :/

But what is, is.  We'll see what davej says (since I've mostly passed
things on to him due to time and other commitments) since he was on
vacation the latter half of last week and certainly now has a nice pile
of mail to return to :-)

Jeremy
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Victor Lowther]

I apologize in advance for the quoting, iPhones have no bulk delete  
function.


On Mar 9, 2009, at 8:47 AM, Jeremy Katz <katzj-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On Monday, March 09 2009, Victor Lowther said:
>> On Sun, 2009-03-08 at 23:11 -0400, Jeremy Katz wrote:
>>> On Friday, March 06 2009, Victor Lowther said:
>>>> On Fri, 2009-03-06 at 15:39 -0500, Bill Nottingham wrote:
>>>>> [I]nstead of scripts hard-coded to do various things, we depend on
>>>>> udev to create device nodes for us and then when we have the
>>>>> rootfs's device node, we mount and carry on
>>>>
>>>> We do this just fine.  Udev is the engine that drives everything  
>>>> in a
>>>> dracut-generated initramfs -- the additional scripts are there in  
>>>> the
>>>> initramfs to handle things that udev does not handle gracefully.  
>>>> If you
>>>> see some functionality currently handles in a script that udev  
>>>> could
>>>> handle better, please implement it.
>>>
>>> The problem is the direction seems to be "hm, udev can't do this,  
>>> guess
>>> we'll make a hook" rather than "fix udev and the underlying tools".
>>> Yes, it means it's harder.  And yes it means that distros have to  
>>> get
>>> updates for everything to work.  But otherwise, we remain in the  
>>> past
>>> with piles of scripts doing most things.
>>
>> udev and the underlying tools are not broken.  Udev names devices
>> according to rules and calls out to external programs or scripts for
>> everything else.  It does that job very well.
>
> Certainly md and dm have their fair share of brokenness when it  
> comes to
> (lack of) udev integration. :)

Indeed.  They are not as hard to integrate as multipathing or nfsroot.

>
>
>>> [snip]
>>>>> - handle various device types through shell snippets, not udev
>>>>>  (raid, crypt, etc.)
>>>>
>>>> We use shell snippets where udev is not flexible enough or where  
>>>> there
>>>> are other limitations we have to work around.  The entire process  
>>>> on the
>>>> initramfs is still udev driven, however.
>>>
>>> Honestly, I really wouldn't say that "the entire process is still  
>>> udev
>>> driven".  The rootfs *NEEDS* to be being mounted from a udev rule --
>>> otherwise, we're not being driven by events.  We're having events  
>>> to do
>>> some things and scripts for others/more
>>
>> OK, there are a few ways that spring to mind to do this:
>>
>> 1: Teach udev how to interpret all the kernel parameters that might  
>> be
>> involved in detecting, performing all the prerequisites for mounting,
>> and then mounting the root filesystem.  Package all this logic up  
>> into
>> something that can can be driven by a small number of new rules.
>>
>> While that sounds tempting, it seems to add quite alot of magic to  
>> udev
>> that it currently does not have and that is well outside its core  
>> focus
>> of device naming -- or does udev perform any device configuration  
>> that
>> it does not call out to an external executable or a shell script to  
>> do?
>
> There's nothing that says that you don't end up using callouts for  
> some
> of this stuff.  The rule syntax is a little brittle (you can't have
> substitutions on both the LHS and the RHS of a conditional iirc, for  
> one
> example) but expanding some of those might well make sense for more
> cases.
>
> And udev *is* growing more capabilities as part of its integration  
> with
> halv2^W DeviceKit
>
>> 2: Include enough logic on the initramfs to parse all the kernel
>> parameters that might be involved in booting the system and then
>> custom-write udev rules that will do all the work of making the root
>> filesystem available (assuming it is available at all) inside the
>> initramfs.
>>
>> That sounds like loads of extra work -- we will have to write a  
>> script
>> or program written in C whose job is to parse kernel parameters,
>> custom-write udev rules based on the ones that might be involved in
>> booting, and inject them into udevd and hope for the best.
>
> I'm not a huge fan of this, it's been done in the past (see
> mayflower/mkliveinitrd in Fedora-land, other things in other places).
> That said, it actually worked surprisingly well.  The problem is
> carrying it over to the real system as you'd like to have a consistent
> set of rules for both.

Exactly. The additional rules we have added are just for their side  
effects - I don't want to inadvertently stomp all over someone's  
device naming scheme.

>
>
>> 3: As above, but create our custom rules at initramfs generation time
>> instead of at boot time.
>>
>> This makes a generic initrd impossible to create, so I will discard  
>> it
>> out of hand.
>
> Indeed! :)
>
>> 4: Add a few udev rules that call scripts to configure a few key  
>> device
>> types as they are detected, and have another process running that  
>> looks
>> for the root filesystem to appear running in tandem with udev.
>>
>> This is what we currently have, and it seems to be doing very well.
>>
>> Of all the approaches, only the first meets your *NEEDS* requirement,
>> and it involves tacking a shedload of extra capabilities onto udev  
>> that
>> it will only ever use while booting the system. The second sounds  
>> even
>> harder to debug than any of the current initramfs schemes, and the  
>> third
>> is too brittle and actually worse than the current redhat/fedora
>> initramfs scheme. We have made large advances using the fourth  
>> approach
>> -- sorry they were not in the direction you initially hoped for.
>
> And I really am glad/pleased/impressed with all the progress.  I'm  
> just
> saying to step back and (... to pick on a red-headed step child ;),
> before going off and saying "well, dmraid doesn't work with udev, so
> we'll do it with a hook", the question really should be "how can we  
> get
> <foo> to work within udev to create us a /dev/root symlink" and then  
> be
> able to use that rather than having arbitrary hooks.  Even the bits to
> do LABEL/UUID munging in the first drop of the code were intended to  
> be
> at best a short-term workaround... maybe having them at all ended up
> leading things astray :/

Well, it is not all bad.

How did you intend nfsroot (and other file systems that do not have a  
backing device) support to work in dracut? Several of my design  
decisions were based around making it easy to integrate them.

>
>
> But what is, is.  We'll see what davej says (since I've mostly passed
> things on to him due to time and other commitments) since he was on
> vacation the latter half of last week and certainly now has a nice  
> pile
> of mail to return to :-)
>
> Jeremy
> --
> To unsubscribe from this list: send the line "unsubscribe initramfs"  
> in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Bill Nottingham]

Seewer Philippe (philippe.seewer-omB+W0Dpw2o@public.gmane.org) said: 
> Bill, a question: How far do you think the genericness of dracut should go?

At a minimum, anything block-based that doesn't require network setup
should be part of the generic case, whether that's disk, CD, usb, MD,
DM, etc.

Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Adam Spragg]

Victor Lowther wrote:
> 4: Add a few udev rules that call scripts to configure a few key device
> types as they are detected, and have another process running that looks
> for the root filesystem to appear running in tandem with udev.
[snip]
> We have made large advances using the fourth approach

Something that worries me slightly ... ISTR that a large part of the goal of
udev was to get device naming policy out of the kernel tree and into an
externally maintained module.

But one of the goals of dracut seems to be to become part of the kernel
source tree.

So, if dracut imports udev naming rules and, therefore, device naming
policy, and is in turn imported into the main kernel source tree, doesn't
that kind of defeat one of the main reasons for udev's existence?

Is this something we need to worry about?

Adam Spragg.

-- 
The fundamental cause of the trouble is that in the modern world the
stupid are cocksure while the intelligent are full of doubt.
    -- Bertrand Russell
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Victor Lowther]

On Tue, 2009-03-10 at 08:06 +0000, Adam Spragg wrote:
> Victor Lowther wrote:
> > 4: Add a few udev rules that call scripts to configure a few key device
> > types as they are detected, and have another process running that looks
> > for the root filesystem to appear running in tandem with udev.
> [snip]
> > We have made large advances using the fourth approach
> 
> Something that worries me slightly ... ISTR that a large part of the goal of
> udev was to get device naming policy out of the kernel tree and into an
> externally maintained module.
> 
> But one of the goals of dracut seems to be to become part of the kernel
> source tree.
> 
> So, if dracut imports udev naming rules and, therefore, device naming
> policy, and is in turn imported into the main kernel source tree, doesn't
> that kind of defeat one of the main reasons for udev's existence?
> 
> Is this something we need to worry about?

Nope, we import our rules from the host system, so whatever policy
decisions are made are someone else's policy decisions.

> Adam Spragg.
> 
-- 
Victor Lowther
RHCE# 805008539634727
LPIC-2# LPI000140019

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Adam Spragg]

> Nope, we import our rules from the host system,

Ah, I misunderstood it as importing udev rules from the udev source repo
into dracut's repo.

My bad. Apologies for the confusion.

Adam
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Victor Lowther]

On Mar 10, 2009, at 6:01 AM, "Adam Spragg" <adam-2iSS7ArDF14@public.gmane.org> wrote:

>> Nope, we import our rules from the host system,
>
> Ah, I misunderstood it as importing udev rules from the udev source  
> repo
> into dracut's repo.
>

I did have a patch to do that, but it was withdrawn after Kay  
explained why it was a bad idea.

> My bad. Apologies for the confusion.
>
> Adam
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Dave Jones]

On Tue, Mar 10, 2009 at 08:06:33AM +0000, Adam Spragg wrote:
 
 > But one of the goals of dracut seems to be to become part of the kernel
 > source tree.

I'm beginning to have some second thoughts about the feasibility of this.
There are a number of use-cases where regeneration of the initramfs
is desirable, which would mean a standalone tool would be needed that
isn't shipped as part of the kernel package.

	Dave

-- 
http://www.codemonkey.org.uk
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: module 90kernel-modules-loaded

[Victor Lowther]

On Tue, 2009-03-10 at 11:14 -0400, Dave Jones wrote:
> On Tue, Mar 10, 2009 at 08:06:33AM +0000, Adam Spragg wrote:
>  
>  > But one of the goals of dracut seems to be to become part of the kernel
>  > source tree.
> 
> I'm beginning to have some second thoughts about the feasibility of this.
> There are a number of use-cases where regeneration of the initramfs
> is desirable, which would mean a standalone tool would be needed that
> isn't shipped as part of the kernel package.

True, but that does not mean it would not be useful to have a reference
initrd generation tool.

How about a round of testing, documentation, and bugfixing to hammer out
any weirdness in the current codebase, then call it 0.5 "may eat babies"
and try to get feedback from a wider audience?

Do we have a bugtracker set up for dracut yet?

> 	Dave
> 
-- 
Victor Lowther
RHCE# 805008539634727
LPIC-2# LPI000140019

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html