Booting a modular kernel through a multiple streams file

Booting a modular kernel through a multiple streams file

[Otto Wyss]

Well I don't think a kernel without anything compiled in is currently functional
but I'd rather like to have as less as possible. And I _don't_ like the solution
with initrd. So I've thought about other solutions to solve the hen-egg dilemma
when booting a modular kernel.

Well a simple solution would be if Linux supports the multiple streams file
format. Assume the kernel and all necessary modules for booting (if not all
modules) are combined into a single file. The boot loader (i.e. lilo) simply
loads this file and starts the first stream (the kernel). It doesn't need to
know the full multiple stream format (maybe nothing at all). The kernel of
course needs this functionality to load the rest of the modules for a minimal
working system. 

I assume it's no problem to integrate the building of this boot file into either
the Linux compilation or better into a separate setup phase (possibly together
with good hardware/module detection).

Advantages:
- A simple boot loader can handle it without much tweaking
- The multiple streams file format is a standard concept usable anywhere
- No ramdisk is necessary
- This concept needs possibly less kernel functionality than initrd
- No change in the current compilation process except for the additional setup phase

Disadvantages:
- Someone else has to do it, I'm not a kernel/driver developer

Why did nobody else have this simple idea? I don't know, maybe the multiple
streams file format isn't widely known in the Linux community. Who else does use
it? The MacOS runs nicely with it, supporting data and resource streams. Maybe
it is already ported from the MacOS X to Darwin. WindowsNT supports it but
doesn't use it.

O. Wyss

Re: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Sun, 16 Dec 2001, Otto Wyss wrote:

> Well a simple solution would be if Linux supports the multiple streams file
> format. Assume the kernel and all necessary modules for booting (if not all
> modules) are combined into a single file. The boot loader (i.e. lilo) simply
> loads this file and starts the first stream (the kernel). It doesn't need to
> know the full multiple stream format (maybe nothing at all). The kernel of
> course needs this functionality to load the rest of the modules for a minimal
> working system. 
> 
> I assume it's no problem to integrate the building of this boot file into either
> the Linux compilation or better into a separate setup phase (possibly together
> with good hardware/module detection).
> 
> Advantages:
> - A simple boot loader can handle it without much tweaking
> - The multiple streams file format is a standard concept usable anywhere

ITYM useless.

> - No ramdisk is necessary
> - This concept needs possibly less kernel functionality than initrd
> - No change in the current compilation process except for the additional setup phase
> 
> Disadvantages:
> - Someone else has to do it, I'm not a kernel/driver developer
> 
> Why did nobody else have this simple idea? I don't know, maybe the multiple
> streams file format isn't widely known in the Linux community.

Ewww....

"Forked files" crap _is_ known.  And not welcome.

There is a bog-standard way to combine several files in one - cpio.  Or tar.
No need to bring Apple Shit-For-Design(tm)(r) when standard tools are quite
enough.

Re: Booting a modular kernel through a multiple streams file

[Otto Wyss]

> > Well a simple solution would be if Linux supports the multiple streams file
> > format. 
> 
> "Forked files" crap _is_ known.  And not welcome.
> 
> There is a bog-standard way to combine several files in one - cpio.  Or tar.
> No need to bring Apple Shit-For-Design(tm)(r) when standard tools are quite
> enough.
>
I don't want to bring in any ...-Design if standard tools are enough! If
cpio/tar do suffice, then much better. I don't know if they are simple and
fitting enough to be handled by any boot loader and the kernel. Maybe you could
elaborate more on this.

You have to admit that a multiple streams file format (regardless which kind)
would be a good solution to the booting of a modular kernel. Anyway this format
has to be supported by the kernel itself and in some extend by any boot loader.
So anybody has to write a kernel module for the cpio/tar format and help with
implementing it into  boot loaders. Maybe you could give some help. 


O. Wyss

Re: Booting a modular kernel through a multiple streams file

[Andreas Dilger]

On Dec 17, 2001  19:49 +0100, Otto Wyss wrote:
> > Somebody not attributed wrote:
> > > Well a simple solution would be if Linux supports the multiple streams
> > > file format. 

Well "multiple streams file format" is purely a FILESYSTEM issue.  When it
is outside of the filesystem, it is an ARCHIVE (e.g. tar, cpio, zip, SHA, etc).
So this whole concept is bogus.

> I don't want to bring in any ...-Design if standard tools are enough! If
> cpio/tar do suffice, then much better. I don't know if they are simple and
> fitting enough to be handled by any boot loader and the kernel. Maybe you
> could elaborate more on this.
> 
> You have to admit that a multiple streams file format (regardless which kind)
> would be a good solution to the booting of a modular kernel. Anyway this
> format has to be supported by the kernel itself and in some extend by any
> boot loader.
> So anybody has to write a kernel module for the cpio/tar format and help with
> implementing it into  boot loaders. Maybe you could give some help. 

Well, the good news is that this whole discussion is moot.  Al Viro has
already written a kernel patch which does all of this, and I'm sure he
is just waiting to get it into the 2.5 kernel.  It creates an "initramfs"
which is populated from a cpio (or tar, can't remeber) archive attached
to the kernel image.

This does things like BOOTP/DHCP discovery, mount the rootfs, load
modules, etc, all before the root filesystem is mounted.  This allows
removing all sorts of junk from the kernel which was there because it
needed to be done before the root filesystem was mounted.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Booting a modular kernel through a multiple streams file

[Otto Wyss]

> Well "multiple streams file format" is purely a FILESYSTEM issue.  When it
> is outside of the filesystem, it is an ARCHIVE (e.g. tar, cpio, zip, SHA, etc).
> So this whole concept is bogus.
> 
It is a FILESYSTEM issue and an ARCHIVE is simply an implementation of the
"multiple streams file format" (MSFF), with archiving in mind. But it is not at
all bogus as long as it _isn't_ supported directly by the kernel. The MSFF
kernel support has more uses (i.e. transparently accessing anything inside as if
it were a directory). Since it is a kind of fs within an fs it can be easy
accessed from a boot loader regardless of the surrounding fs. I.e. Lilo can
access it regardless if the MSFF is on SCSI raid system, a FAT floppy or a Mac
Zip drive (on i386). 

> > So anybody has to write a kernel module for the cpio/tar format and help
> > with implementing it into  boot loaders. Maybe you could give some help.
> 
> Well, the good news is that this whole discussion is moot.  Al Viro has
> already written a kernel patch which does all of this, and I'm sure he
> is just waiting to get it into the 2.5 kernel.  It creates an "initramfs"
> which is populated from a cpio (or tar, can't remeber) archive attached
> to the kernel image.
> 
When I first read about initramfs I considered it just a slight improvement of
initrd and now after more reading about it, I still think it's not the best
solution. It still has to be loaded somewhere. Well initramfs might be usable
but it is just another special case for the booting process. 

With MSFF the kernel just sees it as an ordinary directory and accesses modules
as if they where ordinary files (maybe within sub directories). The bootloader
first just sees a file and tries to start the stream in it. Depending on how
smart the MSFF is the boot loader does not even need to know the MSFF itself as
long as it is able to start the stream. Otherwise if the MSFF is i.e. a tar
archive even Lilo doesn't need much intelligence to start the kernel within.

O. Wyss

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Followup to:  <3C1D060B.9475C9F8@bluewin.ch>
By author:    Otto Wyss <otto.wyss@bluewin.ch>
In newsgroup: linux.dev.kernel
> 
> Disadvantages:
> - Someone else has to do it, I'm not a kernel/driver developer
> 

By this you have pretty much shown yourself utterly unqualified to be
a kernel *designer*.  I won't even go into the various bogus
assumption you're making.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Booting a modular kernel through a multiple streams file

[antirez]

On Sun, Dec 16, 2001 at 01:06:15PM -0800, H. Peter Anvin wrote:
> Followup to:  <3C1D060B.9475C9F8@bluewin.ch>
> By author:    Otto Wyss <otto.wyss@bluewin.ch>
> In newsgroup: linux.dev.kernel
> > 
> > Disadvantages:
> > - Someone else has to do it, I'm not a kernel/driver developer
> > 
> 
> By this you have pretty much shown yourself utterly unqualified to be
> a kernel *designer*.  I won't even go into the various bogus

Agreed, but it should be also noted that to be able to do
kernel coding is insufficient to do kernel design.
The same for user space, to be able to write correct C (or .*)
will not stop people to write code with unsane design, organization
and data structures.

You can see it on freshmeat and in many popular kernels every day.
I see it with my code every time I read my last-month source code.

-- 
Salvatore Sanfilippo <antirez@invece.org>
http://www.kyuzz.org/antirez
finger antirez@tella.alicom.com for PGP key
28 52 F5 4A 49 65 34 29 - 1D 1B F6 DA 24 C7 12 BF

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

antirez wrote:

>>>
>>By this you have pretty much shown yourself utterly unqualified to be
>>a kernel *designer*.  I won't even go into the various bogus
> 
> Agreed, but it should be also noted that to be able to do
> kernel coding is insufficient to do kernel design.

 >

Agreed 100%.

	-hpa

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Sunday 16 December 2001 15:37, Otto Wyss wrote:

> modules) are combined into a single file. The boot loader (i.e. lilo)

LILO follows an outdated, broken concept and should once and for all
be layed to rest, preferably with a stake through it's heart. 

> simply loads this file and starts the first stream (the kernel).

The point to all these 'streams' escapes me. The proper way to implement
this has all ready been done. It's called the Multi Boot Standard
as implemented in GRUB bootloader. http://www.gnu.org/software/grub/

GRUB is similar to syslinux in that is can read directly from the the FS,
but unlike syslinux supports just about all of them instead of just FAT.

Basically what Grub does is loads the kernel modules from disk
into memory, and 'tells' the kernel the memory location to load
them from, very similar to how an initrd file is loaded. The problem
is Linux, is not MBS compilant and doesn't know to look for and load
the modules. 

HOWEVER I recieved patches from
	"ChristianK."@t-online.de (Christian Koenig)
last week that actually implments the module loading in Linux!
I'm trying to get Rusty interested in them, but thus far have not
heard back from him. (Hey Paul! Hit that reply button!)

Anyone interested mail me off list to go over this in more detail.

Dave

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[Eric W. Biederman]

Dave Cinege <dcinege@psychosis.com> writes:

> On Sunday 16 December 2001 15:37, Otto Wyss wrote:
> 
> > modules) are combined into a single file. The boot loader (i.e. lilo)
> 
> LILO follows an outdated, broken concept and should once and for all
> be layed to rest, preferably with a stake through it's heart. 

The way LILO does things is perfectly valid.  Not flexible, not dynamic
but valid.

 
> > simply loads this file and starts the first stream (the kernel).
> 
> The point to all these 'streams' escapes me. The proper way to implement
> this has all ready been done. It's called the Multi Boot Standard
> as implemented in GRUB bootloader. http://www.gnu.org/software/grub/

GRUB is a good proof of concept bootloader but I wouldn't call it
a good way to implement things.  As for the Multi Boot Specification
it is hardly a standard, and it got a lot of details wrong.  A big
one is who is supposed to do the BIOS calls.

It should be o.k. for a bootloader to be static, but still useful on multiple
machines for years.  What comes out of GRUB does not encourage a
static bootloader.  But the development of GRUB is slow enough that it
is practically static...
 
> GRUB is similar to syslinux in that is can read directly from the the FS,
> but unlike syslinux supports just about all of them instead of just FAT.
> 
> Basically what Grub does is loads the kernel modules from disk
> into memory, and 'tells' the kernel the memory location to load
> them from, very similar to how an initrd file is loaded. The problem
> is Linux, is not MBS compilant and doesn't know to look for and load
> the modules. 

So tell me how you make an MBS compliant alpha kernel again?

Eric

Re: Booting a modular kernel through a multiple streams file

[Alan Cox]

> Basically what Grub does is loads the kernel modules from disk
> into memory, and 'tells' the kernel the memory location to load
> them from, very similar to how an initrd file is loaded. The problem
> is Linux, is not MBS compilant and doesn't know to look for and load
> the modules. 

And vendors who've shipped GRUB still have to ship Lilo because Grub plain
doesn't work on some machines. Lilo has the virtue that its extremely simple
in what it does and how it does it. It works in a suprisingly large number
of cases and can handle interesting setups that GRUB really struggles with.

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Sunday 23 December 2001 8:48, Alan Cox wrote:
>
> And vendors who've shipped GRUB still have to ship Lilo because Grub plain
> doesn't work on some machines. 

This is an issue with how GRUB deals with a FUBAR BIOS, and not 
necessarily a core GRUB design flaw. 

Syslinux, seems to win hands down (IMO) when it comes to dealing with
flacky BIOS's, but it's not because it's a dynamic loading design; it's
because HPA knew enough and spent the time refining that 1% of the
program that deals with bad BIOSes.

Should HPA and Werner take a day of their time to review and
correct GRUB, I'm sure this issue would vanish overnight.

> Lilo has the virtue that its extremely
> simple in what it does and how it does it. It works in a suprisingly large
> number of cases and can handle interesting setups that GRUB really
> struggles with.

Where GRUB struggles LILO throws up brickwalls.

Coolest thing is the world you can do with grub:
	Load a *partition* as your initrd
With this in mind it makes using images/archives for initrd for
non-embedded systems seem rather clunky. It's much cleaner
to just have an 8/16/32MB partition you dedicate to initrd. Users of
GRUB have been able to do this for YEARS.

And something to say here: Yes I hate lilo, but not because I think
Werner coded it lousy. A boot loader that uses a static block offset
to find it's target is a bad idea, that should have died with M$-DOS.

That said, it's still a wonder to me why GRUB, or another loader similar
in approach, was not adopted as the 'prefered' linux loader years ago. 

Dave

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[Marcus Meissner]

In article <E16I8zQ-0000d9-00@the-village.bc.nu> you wrote:
>> Basically what Grub does is loads the kernel modules from disk
>> into memory, and 'tells' the kernel the memory location to load
>> them from, very similar to how an initrd file is loaded. The problem
>> is Linux, is not MBS compilant and doesn't know to look for and load
>> the modules. 

> And vendors who've shipped GRUB still have to ship Lilo because Grub plain
> doesn't work on some machines. Lilo has the virtue that its extremely simple
> in what it does and how it does it. It works in a suprisingly large number
> of cases and can handle interesting setups that GRUB really struggles with.

Apart that it moves the initrd somewhere unsafe on high memory machines
and some other odds ends we have fixed, I know of exactly 1 problem with a
hw raid controller, which we did not come around to debug yet.

All other machines, obscure as they may are, boot just fine and without  
problems with GRUB.

Ciao, Marcus

Re: Booting a modular kernel through a multiple streams file

[Alan Cox]

> Apart that it moves the initrd somewhere unsafe on high memory machines
> and some other odds ends we have fixed, I know of exactly 1 problem with a
> hw raid controller, which we did not come around to debug yet.
> 
> All other machines, obscure as they may are, boot just fine and without  
> problems with GRUB.

I can introduce you to machines that don't work with grub in my
test collection. Things like the keyboard not working with a USB
keyboard on one box.

Alan

Re: Booting a modular kernel through a multiple streams file

[Wilfried Weissmann]

Marcus Meissner wrote:

 > In article <E16I8zQ-0000d9-00@the-village.bc.nu> you wrote:
 >
 >>>Basically what Grub does is loads the kernel modules from disk
 >>>into memory, and 'tells' the kernel the memory location to load
 >>>them from, very similar to how an initrd file is loaded. The problem
 >>>is Linux, is not MBS compilant and doesn't know to look for and load
 >>>the modules.
 >>>
 >
 >>And vendors who've shipped GRUB still have to ship Lilo because Grub 
plain
 >>doesn't work on some machines. Lilo has the virtue that its extremely 
simple
 >>in what it does and how it does it. It works in a suprisingly large 
number
 >>of cases and can handle interesting setups that GRUB really struggles 
with.
 >>
 >
 > Apart that it moves the initrd somewhere unsafe on high memory machines
 > and some other odds ends we have fixed, I know of exactly 1 problem 
with a
 > hw raid controller, which we did not come around to debug yet.


I'm just wondering if you mean the (almost) RAID HPT370. You can fix
this by avoiding the "embed" command, which overwrites the raid setup
information. Just "dd" the file system module to a higher offset (e.g.
16) and run grub's "install" command manually.

Wilfried


-- 
Terorists crashed an airplane into the server room, have to remove
/bin/laden. (rm -rf /bin/laden)

RE: Booting a modular kernel through a multiple streams file

[Grover, Andrew]

> From: Otto Wyss [mailto:otto.wyss@bluewin.ch]
> You have to admit that a multiple streams file format 
> (regardless which kind)
> would be a good solution to the booting of a modular kernel. 
> Anyway this format
> has to be supported by the kernel itself and in some extend 
> by any boot loader.
> So anybody has to write a kernel module for the cpio/tar 
> format and help with
> implementing it into  boot loaders. Maybe you could give some help. 

I don't think multiple streams is a good idea, but did you all see the patch
by Christian Koenig to let the bootloader load modules? That seems to solve
the problem nicely.

Regards -- Andy

RE: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Mon, 17 Dec 2001, Grover, Andrew wrote:

> I don't think multiple streams is a good idea, but did you all see the patch
> by Christian Koenig to let the bootloader load modules? That seems to solve
> the problem nicely.

That puts an awful lot of smarts into bootloader and creates code duplication
for no good reason.

We _already_ have code for loading modules.  And it's going to stay, no
matter what happens on boot.  So why the hell duplicate that in $BIGNUM
unrelated pieces of software (LILO, SYSLINUX, MILO, SILO, etc.)?  Just
to create extra fun with debugging?

RE: Booting a modular kernel through a multiple streams file

[Grover, Andrew]

> From: Alexander Viro [mailto:viro@math.psu.edu]
> On Mon, 17 Dec 2001, Grover, Andrew wrote:
> > I don't think multiple streams is a good idea, but did you 
> all see the patch
> > by Christian Koenig to let the bootloader load modules? 
> That seems to solve
> > the problem nicely.
> 
> That puts an awful lot of smarts into bootloader and creates 
> code duplication
> for no good reason.
> 
> We _already_ have code for loading modules.  And it's going 
> to stay, no
> matter what happens on boot.  So why the hell duplicate that 
> in $BIGNUM
> unrelated pieces of software (LILO, SYSLINUX, MILO, SILO, etc.)?  Just
> to create extra fun with debugging?

OK so (correct me if I'm wrong) there are two parts to loading modules -
getting them in memory, and then resolving references. My understanding of
Christian's work is that the bootloader (e.g. GRUB) gets them in memory, but
then it is up to the kernel linker to resolve refs. So yes, there would be
an additional piece of code (marked __init), but it would not have to be
duplicated for each bootloader -- all they have to do is get the modules in
memory and indicate via the multiboot struct where they are.

I don't think this will obsolete any existing boot methods, but it seems
like an additional genuinely useful capability for the Linux kernel to have.

Regards -- Andy

RE: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Mon, 17 Dec 2001, Grover, Andrew wrote:

> 
> OK so (correct me if I'm wrong) there are two parts to loading modules -
> getting them in memory, and then resolving references. My understanding of
> Christian's work is that the bootloader (e.g. GRUB) gets them in memory, but
> then it is up to the kernel linker to resolve refs. So yes, there would be
> an additional piece of code (marked __init), but it would not have to be
> duplicated for each bootloader -- all they have to do is get the modules in
> memory and indicate via the multiboot struct where they are.

*shrug*  Your "all they have to do" is quite heavy.
 
> I don't think this will obsolete any existing boot methods, but it seems
> like an additional genuinely useful capability for the Linux kernel to have.

I've had a very dubious pleasure of dealing with our boot sequence lately.
Adding more cruft to it (including in-kernel linker, for fsck sake) is
_not_ a good idea.

Folks, whatever had happened with "if it can be done in userland - don't
put it into the kernel"?

That goes for a _lot_ of code.  Mounting root.  Detecting the type of
initrd contents.  Loading ramdisk from floppies.  Asking to press
key (you really ought to look what is done for _that_).  Speaking
DHCP - we have a kernel DHCP client, of all things.  All that stuff
can (and should) be done from userland process.  And there's much
more of the same kind.

There is a word for that and that word is "crap".

Let loader leave an archive to be unpacked into rootfs?  Sure.  Let kernel
exec /init on rootfs and leave the rest to it?  Absolutely.  But let's
stop adding userland stuff into the kernel.  Loading modules _can_ be
done from userland - insmod does it just fine.  And that's where it should
be done.

Re: Booting a modular kernel through a multiple streams file

[Craig Christophel]

On Monday 17 December 2001 21:31, Alexander Viro wrote:
> There is a word for that and that word is "crap".
>
> Let loader leave an archive to be unpacked into rootfs?  Sure.  Let kernel
> exec /init on rootfs and leave the rest to it?  Absolutely.  But let's
> stop adding userland stuff into the kernel. 


I agree 100%.  there are several other popular systems that try to do 
everything in-kernel and it adds to the instability overall, besides it gives 
less freedom to customize with standard tools.



Craig.


 Loading modules _can_ be
> done from userland - insmod does it just fine.  And that's where it should
> be done.

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Monday 17 December 2001 21:31, Alexander Viro wrote:

> *shrug*  Your "all they have to do" is quite heavy.

Lilo (broken), syslinux (not very difficult), GRUB, (already implemented).
The list is not long. It's little differeent then a boot loader supporting
initrd loading.

> > I don't think this will obsolete any existing boot methods, but it seems
> > like an additional genuinely useful capability for the Linux kernel to
> > have.
>
> I've had a very dubious pleasure of dealing with our boot sequence lately.

Yes the guilty are well known... 

> Adding more cruft to it (including in-kernel linker, for fsck sake) is
> _not_ a good idea.

Which is why all of that shit that loads multiple initrd floppies, etc,
etc should be gutting and replaced with a solid initrd system that
can load tar.gz's to tmpfs. Jeez...I happen to have such a patch RIGHT 
HERE....for 3 years now. (minix prior to tmpfs existance...)

Loading initrd and boot time kernel modules, from multiple sources,
belongs in 'pre-kernel' (prom/bootloader) land. Most importantly this
boot loader is already implemented for IA32.

> That goes for a _lot_ of code.  Mounting root.  Detecting the type of
> initrd contents.  Loading ramdisk from floppies.  Asking to press
> key (you really ought to look what is done for _that_).  Speaking
> DHCP - we have a kernel DHCP client, of all things.  All that stuff
> can (and should) be done from userland process. 

Already done (properly) in GRUB. Userland (after the kernel boots) is
no place for this. It's too late to be done cleanly. I agree the kernel
is no place either.
 
> Let loader leave an archive to be unpacked into rootfs?  Sure.  Let kernel
> exec /init on rootfs and leave the rest to it?  Absolutely.  But let's
> stop adding userland stuff into the kernel.  Loading modules _can_ be
> done from userland - insmod does it just fine.  And that's where it should
> be done.

This is a very much a failed concept. In theory it sounds nice. My
experience in
	1) implementing a Linux OS from scratch
	2) administering multiple networks of mission critical servers
says this doesn't work well.  

Dave

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Tuesday 18 December 2001 15:26, Alexander Viro wrote:
> On Tue, 18 Dec 2001, Grover, Andrew wrote:
> > I'm not arguing that the new initrd won't be better than the old initrd
> > (because obviously you are right) I'm arguing that no matter how whizzy
> > initrd is, it's still an unnecessary step, and it's one that other OSs
> > (e.g. FreeBSD) omit in favor of the approach I'm advocating.
>
> Learn to read.  You don't _have_ to have initrd.  At all.  There's nothing
> to stop your loader from putting whatever cpio archive it likes - it
> doesn't involve anything other than slapping files you want together
> putting their owner/group/size/timestamps/mode/name before each of them.
> Anything that puts a bunch of modules in core will have to do equivalent
> job.

Deja Vu: *shrug*  Your "all they have to do" is quite heavy.
(boot loader must implement full cpio/tar[/gzip}

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Followup to:  <E16Hwks-0001wV-00@schizo.psychosis.com>
By author:    Dave Cinege <dcinege@psychosis.com>
In newsgroup: linux.dev.kernel
> 
> Deja Vu: *shrug*  Your "all they have to do" is quite heavy.
> (boot loader must implement full cpio/tar[/gzip}
> 

cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
unacceptable, but should not be required.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On 22 Dec 2001, H. Peter Anvin wrote:

> Followup to:  <E16Hwks-0001wV-00@schizo.psychosis.com>
> By author:    Dave Cinege <dcinege@psychosis.com>
> In newsgroup: linux.dev.kernel
> > 
> > Deja Vu: *shrug*  Your "all they have to do" is quite heavy.
> > (boot loader must implement full cpio/tar[/gzip}
> > 
> 
> cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
> unacceptable, but should not be required.

tar is ugly as hell and not going to be supported on the kernel side.

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Saturday 22 December 2001 21:10, Alexander Viro wrote:

> > cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
> > unacceptable, but should not be required.
>
> tar is ugly as hell and not going to be supported on the kernel side.

Excellent! You've settled on using using an archiver format nobody uses,
instead of the defacto standard that's already been implemented by
atleast two people.
			G-E-N-I-U-S!

>IIRC, his objections back then were about linking archive into the kernel
>image.  s/disaster/ugly stuff that was nowhere near top-priority and got 
>fixed since then/ and I agree with that one.

Ahh good. Maybe you'll wise up to supporting tar as well. Then in
maybe a year and a half from now standard Linux will finally have
what I've had in production for 3 years...

Dave

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Sat, 22 Dec 2001, Dave Cinege wrote:

> On Saturday 22 December 2001 21:10, Alexander Viro wrote:
> 
> > > cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
> > > unacceptable, but should not be required.
> >
> > tar is ugly as hell and not going to be supported on the kernel side.
> 
> Excellent! You've settled on using using an archiver format nobody uses,
> instead of the defacto standard that's already been implemented by
> atleast two people.
> 			G-E-N-I-U-S!

OK, back into the killfile you go.

Hint: instead of wanking in public try to _think_ for a while.  Requirements
to archive format:
	* can be generated with minimum of code
	* can be parsed <ditto>
	* can be handled by standard utilities
That's it.  Both cpio(1) and tar(1) (or pax(1) that can do both) fit the last
one.  And tar loses on the first two - it's messier.  Not much, but enough
to make the choice obvious.  "Popular" is completely irrelevant here - as long
as it's handled by standard UNIX utilities it's OK.

Re: Booting a modular kernel through a multiple streams file

[Ryan Cumming]

On December 22, 2001 19:00, Dave Cinege wrote:
> Excellent! You've settled on using using an archiver format nobody uses,
Many file formats that require 'embedded' archives use cpio. RPM is one such 
file format. I think every single RPM distribution relying on cpio for the 
core of its package management system discounts your claim of 'nobody using 
it'. Just because you have not seen cpio directly used as an archive format 
doesn't mean it's unused.

> instead of the defacto standard that's already been implemented by
> atleast two people.
tar and cpio are both POSIX standards. I fail to see how one is more 
'defacto' than the other. 

As far as having tar already being implemented by at least two people, it 
really doesn't matter for a format as simple as cpio. Having a reference 
implementation for tar would be nice, as it is a hairy, complicated standard. 
The cpio format can be fully described in less than a page.

> G-E-N-I-U-S!
Grow up. Please.

-Ryan

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Saturday 22 December 2001 23:41, Ryan Cumming wrote:
> On December 22, 2001 19:00, Dave Cinege wrote:
> > Excellent! You've settled on using using an archiver format nobody uses,
>
> Many file formats that require 'embedded' archives use cpio. RPM is one
> such file format.

I know this, and have taken rpm's apart buy hand, and wrote 
a small util to take them apart using cpio. I consider it a good
example of Redhat's many screw up's. Deb's use a pair of .tgz's
and header. Much more sane.

> I think every single RPM distribution relying on cpio for

Hardly. It relies on rpm, which uses the cpio for the data archive.
It does not use the cpio command to make or unpackage archives. 

We're talking about a raw archive type, not something embedded in
a broader applicaiton. 

> the core of its package management system discounts your claim of 'nobody
> using it'. Just because you have not seen cpio directly used as an archive
> format doesn't mean it's unused.

How many times have you seen ANYTHING ditributed that way?

> As far as having tar already being implemented by at least two people, it
> really doesn't matter for a format as simple as cpio. Having a reference
> implementation for tar would be nice, as it is a hairy, complicated
> standard. 

You have no idea what you're talkig about, do you? Basic untar'ing is
quite simple. Now, GNU's tar as implented is pretty fucked up, but
that's not relevant to this.

> The cpio format can be fully described in less than a page.

Untar is implemented in less then 2. It doesn't get much smaller.

>From rd.dyn.c
---------------------
// This is called by flush_window to flush the gzip output
// buffer (containing part of a tar image).
UNTAR_WRITE
{
	int	i, scoop, err, file_written = 0;
	char	*to;
	kdev_t	fdev;
	struct	utimbuf ut;
	struct	untar_context *Untar;
#ifndef DEBUG_UNTAR
#ifndef CONFIG_ARCH_S390
	char rotator[4] = { '|' , '/' , '-' , '\\' };

	// pointer to our context structure is passed thru ioctl fop
	Untar = (struct untar_context*)fp->f_op->ioctl;

	printk("%c\b", rotator[Untar->rotate & 0x3]);
	Untar->rotate++;
#endif
#endif
	// Note: the gunzip routines call this only when
	//       a) the gunzip buffer is full -> count = WSIZE = 64 * TARBLOCKSIZE
	// or    b) to flush the last bytes out
	while(count) {	// What is our current state ?
		err = 0;	
		switch(Untar->state) {
			case READING_HEADER:
				// Get the tar header
				if(count < TARBLOCKSIZE) {
					count = 0;
					break;
				}
				to = (char*)&Untar->tarInfo;
				for(i = 0; i < TARBLOCKSIZE; i++) {
					*to++ = *buf++;
					--count;
				}
	
				// Tar usually pads the output byte to a multiple of it's block size usually
				// 10 KB, appending zeroes if necessary. Here we skip those zeroes:
				if(Untar->tarInfo.header.name[0] == '\0'  &&
				   Untar->tarInfo.header.mode[0] == '\0' &&
				   Untar->tarInfo.header.mode[0] == '\0') {
					if(count < TARBLOCKSIZE)
						count = 0;
					break;
				}
				// determine mode, size, uid, gid
				Untar->fmode = strtoul(Untar->tarInfo.header.mode,NULL,8);
				Untar->fsize = strtoul(Untar->tarInfo.header.size,NULL,8);
				Untar->uid = strtoul(Untar->tarInfo.header.uid,NULL,8);
				Untar->gid = strtoul(Untar->tarInfo.header.gid,NULL,8);
#ifdef DEBUG_UNTAR
				printk("\ntype: %c ",Untar->tarInfo.header.typeflag);   
				printk("name: %s ",Untar->tarInfo.header.name);
				printk("Untar->fsize: %ld ",Untar->fsize); 
				udelay(100000);
#endif
				// check the type
				switch(Untar->tarInfo.header.typeflag) {
					case AREGTYPE : 
					case REGTYPE  :
						Untar->out = sys_open(Untar->tarInfo.header.name,
					             O_CREAT|O_WRONLY|O_TRUNC, Untar->fmode);

						if(Untar->out == -1) {
							err = 1;
							break;
						}
						Untar->state = READING_DATA;
						break;
					case LNKTYPE  :
						err = sys_link(Untar->tarInfo.header.linkname,
						                  Untar->tarInfo.header.name);
						break;

					case SYMTYPE  :
						err = sys_symlink(Untar->tarInfo.header.linkname,
						                 Untar->tarInfo.header.name);
						break;

					case CHRTYPE  :	
					case BLKTYPE  :	
					case FIFOTYPE :	
						fdev = MKDEV(strtoul(Untar->tarInfo.header.devmajor,NULL,8),
						   strtoul(Untar->tarInfo.header.devminor,NULL,8));
						err = sys_mknod(Untar->tarInfo.header.name,Untar->fmode,fdev);
   	    				break;

					case DIRTYPE  :
						if((Untar->tarInfo.header.name[0] != '.' &&	// Skip if name is "" "./" "." ".." */
						    Untar->tarInfo.header.name[0] != '\0') &&
						   (Untar->tarInfo.header.name[1] != '/' &&
						    Untar->tarInfo.header.name[1] != '\0') &&
						   (Untar->tarInfo.header.name[1] != '\0')) {
							err = sys_mkdir(Untar->tarInfo.header.name,Untar->fmode);
						}
						break;
					default:
						printk(KERN_ERR "RAMDISK: corrupt tar archive\n");
						Untar->state = SKIPPING_REST;
						return(-1);
				}
				break;
			case READING_DATA:
				scoop = 0;
				while(count > 0 && Untar->fsize > 0) {
					scoop = Untar->fsize > TARBLOCKSIZE ? TARBLOCKSIZE : Untar->fsize;
					if(sys_write(Untar->out, buf, scoop) < scoop)
						err = 1;
					count -= scoop;
					buf   += scoop;
					Untar->fsize -= scoop;
				}
				if(Untar->fsize == 0) {	//skip to the next tar block
					sys_close(Untar->out);
					scoop = count % TARBLOCKSIZE;
					buf   += scoop;
					count -= scoop;
					Untar->state = READING_HEADER;
					file_written = 1;
				}
				break;

			case SKIPPING_REST:
				count = 0;
				break;
		}
		if(err)
			printk("!");
			//printk("\nError making %s", Untar->tarInfo.header.name);

		if(file_written) {
		
			err = sys_chown(Untar->tarInfo.header.name, Untar->uid, Untar->gid);
		
			if(Untar->tarInfo.header.typeflag != LNKTYPE &&
			   Untar->tarInfo.header.typeflag != SYMTYPE) {
				err += sys_chmod(Untar->tarInfo.header.name, Untar->fmode);
			}

			if(err)
				printk("\nError chown and/or chmod %s", Untar->tarInfo.header.name);
			ut.actime=sys_time(NULL);
			ut.modtime=strtoul(Untar->tarInfo.header.mtime,NULL,8);
			sys_utime(Untar->tarInfo.header.name,&ut);
			file_written = 0;
		}
	}
	return(0);
}

> > G-E-N-I-U-S!
>
> Grow up. Please.

Shut up Sit down!

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[Bernd Eckenfels]

In article <E16I1Yc-0003eD-00@schizo.psychosis.com> you wrote:
> I know this, and have taken rpm's apart buy hand, and wrote 
> a small util to take them apart using cpio. I consider it a good
> example of Redhat's many screw up's. Deb's use a pair of .tgz's
> and header. Much more sane.

Well, debian is using an "ar" archive with tar.gz contents.

> How many times have you seen ANYTHING ditributed that way?

AFAIK cpio is quite common on SysV Systems. The Problem with cpio and tar is,
that there are many incompatible versions. Even the Posix.1 format is quite
limited (255 char path limit). Some others fail short with symlinks or block
device nodes.

AFAIK SUS is supporting the use of pax.

http://www.opengroup.org/onlinepubs/7908799/xcu/pax.html 

Greetings
Bernd

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Followup to:  <E16I2rw-0006zi-00@sites.inka.de>
By author:    Bernd Eckenfels <usenet2001-12@lina.inka.de>
In newsgroup: linux.dev.kernel
> 
> > How many times have you seen ANYTHING ditributed that way?
> 
> AFAIK cpio is quite common on SysV Systems. The Problem with cpio and tar is,
> that there are many incompatible versions. Even the Posix.1 format is quite
> limited (255 char path limit).

This is the POSIX.1 *TAR* format.  The POSIX.1 *CPIO* format doesn't
have this limitation.

>  Some others fail short with symlinks or block
> device nodes.

Only utterly archaic versions, never used on Linux.

> AFAIK SUS is supporting the use of pax.
> 
> http://www.opengroup.org/onlinepubs/7908799/xcu/pax.html 

Pax is just a single utility which does cpio and tar.

Al is using a specific cpio format, which I believe is either the
"newc" or "odc" format... Al?

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Booting a modular kernel through a multiple streams file

[Kai Henningsen]

hpa@zytor.com (H. Peter Anvin)  wrote on 23.12.01 in <a0439g$7a5$1@cesium.transmeta.com>:

> Pax is just a single utility which does cpio and tar.

Well, pax is in POSIX 2001. Neither tar nor cpio are.

Of course, pax is specified to be able to work with both tar and cpio  
formats. And a new extended tar format which is what you get when you try  
to stuff a completely different format into tar.

MfG Kai

Re: Booting a modular kernel through a multiple streams file

[Erik Mouw]

On Sun, Dec 23, 2001 at 02:22:00PM +0200, Kai Henningsen wrote:
> hpa@zytor.com (H. Peter Anvin)  wrote on 23.12.01 in <a0439g$7a5$1@cesium.transmeta.com>:
> 
> > Pax is just a single utility which does cpio and tar.
> 
> Well, pax is in POSIX 2001. Neither tar nor cpio are.

If POSIX abandons tar and cpio, it doesn't mean that all
implementations are magically erased from all hard drives all over the
world.


Erik

-- 
J.A.K. (Erik) Mouw, Information and Communication Theory Group, Faculty
of Information Technology and Systems, Delft University of Technology,
PO BOX 5031, 2600 GA Delft, The Netherlands  Phone: +31-15-2783635
Fax: +31-15-2781843  Email: J.A.K.Mouw@its.tudelft.nl
WWW: http://www-ict.its.tudelft.nl/~erik/

Re: Booting a modular kernel through a multiple streams file

[James A Sutherland]

On Tuesday 18 December 2001 8:55 am, Alexander Viro wrote:
> On Tue, 18 Dec 2001, James A Sutherland wrote:
> > Not necessarily. You could, say, put the modules in a small filesystem
> > image - say, Minix, or maybe ext2. Then just have the loader put that
> > disk image into RAM, and have the kernel able to read disk images from
> > RAM initially.
> >
> > Of course, this revolutionary new features needs a name. Something like
> > initrd, perhaps?
>
> Had you actually looked at initrd-related code?  I had and "bloody mess"
> is the kindest description I've been able to come up with.  Even after
> cleanups and boy, were they painful...

With a choice between that, or teaching lilo, grub etc how to link modules - 
and how to read NTFS and XFS, and losing the ability to boot from fat, minix 
etc floppies, tftp or nfs servers - almost any level of existing nastiness 
would be preferable to that sort of insane codebloat!


James.

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Followup to:  <T57e612d0dbac1785e6169@pcow028o.blueyonder.co.uk>
By author:    James A Sutherland <james@sutherland.net>
In newsgroup: linux.dev.kernel
>
> On Tuesday 18 December 2001 8:55 am, Alexander Viro wrote:
> > On Tue, 18 Dec 2001, James A Sutherland wrote:
> > > Not necessarily. You could, say, put the modules in a small filesystem
> > > image - say, Minix, or maybe ext2. Then just have the loader put that
> > > disk image into RAM, and have the kernel able to read disk images from
> > > RAM initially.
> > >
> > > Of course, this revolutionary new features needs a name. Something like
> > > initrd, perhaps?
> >
> > Had you actually looked at initrd-related code?  I had and "bloody mess"
> > is the kindest description I've been able to come up with.  Even after
> > cleanups and boy, were they painful...
> 
> With a choice between that, or teaching lilo, grub etc how to link modules - 
> and how to read NTFS and XFS, and losing the ability to boot from fat, minix 
> etc floppies, tftp or nfs servers - almost any level of existing nastiness 
> would be preferable to that sort of insane codebloat!
> 

Note that Al is working on a replacement; he's not just bitching.  The
replacement is called "initramfs" which means populating a ramfs from
an archive or collection of archives passwd by the bootloader.  With
that in there, lots of things can be done in userspace.

I had my own idea for how to do this, but I don't really seem to be
able to get the time to work it...

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Booting a modular kernel through a multiple streams file

[James A Sutherland]

On Tuesday 18 December 2001 7:10 pm, H. Peter Anvin wrote:
> Followup to:  <T57e612d0dbac1785e6169@pcow028o.blueyonder.co.uk>
> By author:    James A Sutherland <james@sutherland.net>
> In newsgroup: linux.dev.kernel
>
> > On Tuesday 18 December 2001 8:55 am, Alexander Viro wrote:
> > > On Tue, 18 Dec 2001, James A Sutherland wrote:
> > > > Not necessarily. You could, say, put the modules in a small
> > > > filesystem image - say, Minix, or maybe ext2. Then just have the
> > > > loader put that disk image into RAM, and have the kernel able to read
> > > > disk images from RAM initially.
> > > >
> > > > Of course, this revolutionary new features needs a name. Something
> > > > like initrd, perhaps?
> > >
> > > Had you actually looked at initrd-related code?  I had and "bloody
> > > mess" is the kindest description I've been able to come up with.  Even
> > > after cleanups and boy, were they painful...
> >
> > With a choice between that, or teaching lilo, grub etc how to link
> > modules - and how to read NTFS and XFS, and losing the ability to boot
> > from fat, minix etc floppies, tftp or nfs servers - almost any level of
> > existing nastiness would be preferable to that sort of insane codebloat!
>
> Note that Al is working on a replacement; he's not just bitching. The
> replacement is called "initramfs" which means populating a ramfs from
> an archive or collection of archives passwd by the bootloader.  With
> that in there, lots of things can be done in userspace.

What I was suggesting is that using an initfs (whether initrd, initramfs or 
something else) is a better approach than trying to get the bootloader 
grovelling around in the kernel innards - initramfs strengthens this 
argument, I think. Just put the modules into archives, and use initramfs to 
access them and a copy of insmod...


James.

Re: Booting a modular kernel through a multiple streams file

[Erik Andersen]

On Wed Dec 19, 2001 at 09:27:29AM +0000, James A Sutherland wrote:
> 
> What I was suggesting is that using an initfs (whether initrd, initramfs or 
> something else) is a better approach than trying to get the bootloader 
> grovelling around in the kernel innards - initramfs strengthens this 
> argument, I think. Just put the modules into archives, and use initramfs to 
> access them and a copy of insmod...

Here is a simple example...  Take busybox, disable everything but
insmod and lash (the simplest of the busybox shells).  Staticly
link vs uClibc -- the result is 61k (far less then the 440k from
glibc).  Now make a root fs that contains just /bin/sh (the
busybox binary), /sbin/insmod (a symlink to /bin/sh), and then
fill up / of your root fs with all the modules you wish to load.
Add in a simple script as /linuxrc or /sbin/init with something
like:

    #!/bin/sh
    /sbin/insmod /foo.o

Now, gzip the result.  With about 60k worth of modules, the final
initrd will compress to about 60k.  Simple, easy, small, and all
in userspace,

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Tuesday 18 December 2001 14:10, H. Peter Anvin wrote:

> Note that Al is working on a replacement; he's not just bitching.  The
> replacement is called "initramfs" which means populating a ramfs from
> an archive or collection of archives passwd by the bootloader.  With
> that in there, lots of things can be done in userspace.

Already done for many months (actully years) now. If I thought it
would be accepted, I'd gut rd.c and fully merge it into a 2.5 patch.

ftp://ftp.psychosis.com/linux/initrd-dyn/

Initrd Dynamic supercedes Initrd Archive. This function subset was
first written in January 1998 and has progressed in features since
then. Developed for the Linux Router Project, it is in wide spread
use in various Linux embedded systems and thin OS's.

Initial RAM disk dynamic creation support
CONFIG_BLK_DEV_INITRD_DYN
  Initrd Dynamic allows you to use tar.gz archive(s) instead of those
  nasty raw  images. It works by dynamically creating a filesystem at
  boot, mounting it as '/', then extracting the archive(s) loaded into
  the initrd memory space. If your bootloader can load multiple archives
  sequentially (IE a patched GRUB) all archives will be extracted in the
  order they where loaded.

  Optionally Initrd Dynamic can create the /dev directory and some
  failsafe console files.

  You must select at least one of the available filesystems below.
  You probably want to use tmpfs if it's available in your kernel.

  The kernel parameter to specify options is:
  initrd_dyn=fstype[,fssize[k|m]][,mkdev]

  fstype == One of the supported dynamic mkfs filesystems.

  fssize ==  Limits the size of the filesystem created.
	tmpfs: If not speced it's equal to the real memory space.
  	minix: If not speced it's equal to ramdisk_size.

  mkdev  == /dev is created after archive untar. Useful even if devfs=mount.
	  
  Examples:
  initrd_dyn=tmpfs load_ramdisk=1
  initrd_dyn=tmpfs,4096k load_ramdisk=1
  initrd_dyn=tmpfs,16m,mkdev load_ramdisk=1
  
  initrd_dyn=minix,8m ramdisk_size=20480 root=/dev/ram0 load_ramdisk=1
  initrd_dyn=minix,,mkdev root=/dev/rd/0 load_ramdisk=1

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Dave Cinege wrote:

> On Tuesday 18 December 2001 14:10, H. Peter Anvin wrote:
> 
> 
>>Note that Al is working on a replacement; he's not just bitching.  The
>>replacement is called "initramfs" which means populating a ramfs from
>>an archive or collection of archives passwd by the bootloader.  With
>>that in there, lots of things can be done in userspace.
>>
> 
> Already done for many months (actully years) now. If I thought it
> would be accepted, I'd gut rd.c and fully merge it into a 2.5 patch.
> 


Viro's going quite a bit beyond this, though.  Also, a big key is the 
ability to have multiple images, some of which may not be compressed.

	-hpa

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Saturday 22 December 2001 19:42, H. Peter Anvin wrote:
> Dave Cinege wrote:
> > On Tuesday 18 December 2001 14:10, H. Peter Anvin wrote:
> >>Note that Al is working on a replacement; he's not just bitching.  The
> >>replacement is called "initramfs" which means populating a ramfs from
> >>an archive or collection of archives passwd by the bootloader.  With
> >>that in there, lots of things can be done in userspace.
> >
> > Already done for many months (actully years) now. If I thought it
> > would be accepted, I'd gut rd.c and fully merge it into a 2.5 patch.
>
> Viro's going quite a bit beyond this, though.

Disaster is the word that comes to mind when I think about the last
initramfs patch I looked at (be it several months ago)

> Also, a big key is the
> ability to have multiple images, some of which may not be compressed.

Initrd Dynamic already supports multiple tar.gz archives. If I thought
their was a need I could support uncompressed tar's with a few small
changes.

I've patched GRUB to support loading the mutiple images. It would also
be supported in syslinux, if you choose to implement it. Months
ago when I asked you to implement it, you told me the idea was
stupid.  : P   I will never let you forget this myopia. : >

Dave

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Dave Cinege wrote:

> 
> I've patched GRUB to support loading the mutiple images. It would also
> be supported in syslinux, if you choose to implement it. Months
> ago when I asked you to implement it, you told me the idea was
> stupid.  : P   I will never let you forget this myopia. : >
> 


Holding a grudge, eh?  How very mature of you.

Seriously, I made it very clear at the time that I thought supporting 
*multiple ramdisks* were a stupid idea.  Perhaps you misunderstood, but 
I was talking about what some people had been requesting of loading a 
filesystem into /dev/ram0, another filesystem into /dev/ram1, etc., 
including waiting for a disk replacement in between.  Doing that in the 
bootloader is just idiotic, for the same reason all the crap in the 
current kernel is equally stupid (although forgivable for historical 
reasons.)

The reason to support uncompressed images -- as well as gaps between 
images -- is to let synthesis happen in the bootloader.  For example, 
some people have requested passing the PXE configuration packets to the 
kernel, which currently is all but impossible.  Presenting them as files 
in the initramfs is the natural way to do it.

Viros changes are drastic, no question about it, but dismissing them as 
"disaster" without further motivation is a bloody awfully arrogant.

	-hpa

Re: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Sat, 22 Dec 2001, H. Peter Anvin wrote:

> Dave Cinege wrote:

[snip]

Umm... Peter, mind forwarding the sensible parts to me?  Sometimes
.procmailrc is a bit too drastic and /dev/null is hard to read...

> Viros changes are drastic, no question about it, but dismissing them as 
> "disaster" without further motivation is a bloody awfully arrogant.

IIRC, his objections back then were about linking archive into the kernel
image.  s/disaster/ugly stuff that was nowhere near top-priority and got fixed
since then/ and I agree with that one.

RE: Booting a modular kernel through a multiple streams file

[Grover, Andrew]

> From: Alexander Viro [mailto:viro@math.psu.edu]
> On Mon, 17 Dec 2001, Grover, Andrew wrote:
> all they have to do is 
> get the modules in
> > memory and indicate via the multiboot struct where they are.
> 
> *shrug*  Your "all they have to do" is quite heavy.

GRUB 0.90 does this today. All other bootloaders could also do it quite
easily, since this is just an extension of what they have to do for the
kernel and initrd images.

> I've had a very dubious pleasure of dealing with our boot 
> sequence lately.
> Adding more cruft to it (including in-kernel linker, for fsck sake) is
> _not_ a good idea.

I see this as cruft elimination. By adding a kernel linker (which can be
discarded after init) it allows one to increase the modularity of the kernel
- without using an initrd. Heck, you could make the initrd code
modularizable ;-)

> Folks, whatever had happened with "if it can be done in 
> userland - don't
> put it into the kernel"?

Yes, but this isn't an absolute rule. IIRC that rule also has an "unless it
makes things a lot simpler" clause, too.

> That goes for a _lot_ of code.  Mounting root.  Detecting the type of
> initrd contents.  Loading ramdisk from floppies.  Asking to press
> key (you really ought to look what is done for _that_).  Speaking
> DHCP - we have a kernel DHCP client, of all things.  All that stuff
> can (and should) be done from userland process.  And there's much
> more of the same kind.

> There is a word for that and that word is "crap".

These examples are either a direct result of initrd complexities, or not
related.

Initrd exists to allow a two-phase startup. My point is that why have a 2
phase startup when you can have a 1 phase startup? Also, I'm not advocating
ditching the initrd capability, but wouldn't bootloading modules be
preferable for the majority of the systems currently using initrd out of
necessity?

Regards -- Andy

RE: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Tue, 18 Dec 2001, Grover, Andrew wrote:

> GRUB 0.90 does this today. All other bootloaders could also do it quite
> easily, since this is just an extension of what they have to do for the
> kernel and initrd images.

... and I'm quite sure that EMACS could do it easily.  Let's not talk
about GNU bloatware, OK?

> > I've had a very dubious pleasure of dealing with our boot 
> > sequence lately.
> > Adding more cruft to it (including in-kernel linker, for fsck sake) is
> > _not_ a good idea.
> 
> I see this as cruft elimination. By adding a kernel linker (which can be
> discarded after init) it allows one to increase the modularity of the kernel
> - without using an initrd. Heck, you could make the initrd code
> modularizable ;-)

I will take it into userland, thank you very much.

> > Folks, whatever had happened with "if it can be done in 
> > userland - don't
> > put it into the kernel"?
> 
> Yes, but this isn't an absolute rule. IIRC that rule also has an "unless it
> makes things a lot simpler" clause, too.

Except that in this case it doesn't make anything simpler.
 
> > That goes for a _lot_ of code.  Mounting root.  Detecting the type of
> > initrd contents.  Loading ramdisk from floppies.  Asking to press
> > key (you really ought to look what is done for _that_).  Speaking
> > DHCP - we have a kernel DHCP client, of all things.  All that stuff
> > can (and should) be done from userland process.  And there's much
> > more of the same kind.
> 
> > There is a word for that and that word is "crap".
> 
> These examples are either a direct result of initrd complexities, or not
> related.

They are related to the fact that we enter userland too late.  Which can
(and should) be fixed.

Linker is not something special - it's yet another userland program.
And we _can_ get userland early - before any drivers are initialized.
Including ramdisk.  Including any block filesystems.  Including at
least large part of (if not entire) PCI initialization.

All we need is s/rootfs/ramfs/ (8Kb patch, mostly removing code) and
unpacking archive to absolute root.  End of story.  Loader should
leave that archive (or concatenation of several archives) in core and
I don't believe that creating cpio on the fly is more complex than
whatever structure you will use to tell the kernel where these modules
are left in core.

All you need on the kernel side is a pass through the area where loader
had left the damn thing and uncpio to /.  After that we free that area
and do execve("/init", ...).  End of story.

BTW, we don't need a special device to handle initrd after that.  Just
have your initrd image (gzipped, whatever) in the archive under /initrd.
After that /init will have the contents in that file (regular file, at
that) and can do whatever it bloody wants.

It kills a _lot_ of kernel code.  And works nicely with any existing
loader - if they want they can build archive on the fly, but they can
just get a pre-built archive using the same mechanism they currently
use for initrd images.  IOW, we are backwards compatible with old
loaders.

Re: Booting a modular kernel through a multiple streams file

[Greg KH]

On Tue, Dec 18, 2001 at 09:43:28AM -0800, Grover, Andrew wrote:
> Initrd exists to allow a two-phase startup. My point is that why have a 2
> phase startup when you can have a 1 phase startup? Also, I'm not advocating
> ditching the initrd capability, but wouldn't bootloading modules be
> preferable for the majority of the systems currently using initrd out of
> necessity?

Cool, how is GRUB going to do hotplug issues? :)

The 2 stage approach allows us to run /sbin/hotplug during kernel boot
so that we can load the proper pci/usb/firewire/whatever drivers for
whatever is currently plugged into the machine at boot time.

thanks,

greg k-h

RE: Booting a modular kernel through a multiple streams file

[Grover, Andrew]

> From: James A Sutherland [mailto:james@sutherland.net]
> > Had you actually looked at initrd-related code?  I had and 
> "bloody mess"
> > is the kindest description I've been able to come up with.  
> Even after
> > cleanups and boy, were they painful...
> 
> With a choice between that, or teaching lilo, grub etc how to 
> link modules - 
> and how to read NTFS and XFS, and losing the ability to boot 
> from fat, minix 
> etc floppies, tftp or nfs servers - almost any level of 
> existing nastiness 
> would be preferable to that sort of insane codebloat!

1) GRUB can already do this
2) Each bootloader doesn't need to link, the kernel includes the linker.
(which after it does its job can be discarded and insmod used later on)
3) Seeing how ugly everyone seems to think initrd is, this seems like a
worthwhile option to consider.

Regards -- Andy

RE: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Tue, 18 Dec 2001, Grover, Andrew wrote:

> 1) GRUB can already do this
> 2) Each bootloader doesn't need to link, the kernel includes the linker.
> (which after it does its job can be discarded and insmod used later on)
> 3) Seeing how ugly everyone seems to think initrd is, this seems like a
> worthwhile option to consider.

Sigh...  You end up adding more code to kernel _and_ keeping initrd
code in there.  And that's supposed to be an improvement?

Re: Booting a modular kernel through a multiple streams file

[David Weinehall]

On Tue, Dec 18, 2001 at 09:47:58AM -0800, Grover, Andrew wrote:
> > From: James A Sutherland [mailto:james@sutherland.net]
> > > Had you actually looked at initrd-related code?  I had and 
> > "bloody mess"
> > > is the kindest description I've been able to come up with.  
> > Even after
> > > cleanups and boy, were they painful...
> > 
> > With a choice between that, or teaching lilo, grub etc how to 
> > link modules - 
> > and how to read NTFS and XFS, and losing the ability to boot 
> > from fat, minix 
> > etc floppies, tftp or nfs servers - almost any level of 
> > existing nastiness 
> > would be preferable to that sort of insane codebloat!
> 
> 1) GRUB can already do this
> 2) Each bootloader doesn't need to link, the kernel includes the linker.
> (which after it does its job can be discarded and insmod used later on)
> 3) Seeing how ugly everyone seems to think initrd is, this seems like a
> worthwhile option to consider.

And GRUB is of course available for all platforms that Linux is
available for? Noooo? I didn't think so...


/David
  _                                                                 _
 // David Weinehall <tao@acc.umu.se> /> Northern lights wander      \\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </

RE: Booting a modular kernel through a multiple streams file

[Grover, Andrew]

> From: Alexander Viro [mailto:viro@math.psu.edu]
> On Tue, 18 Dec 2001, Grover, Andrew wrote:
> > GRUB 0.90 does this today.
> ... and I'm quite sure that EMACS could do it easily.  Let's not talk
> about GNU bloatware, OK?

I don't think this is bloatware, especially considering there really isn't
any cost for having a full-featured bootloader - all its footprint gets
reclaimed, after all. I respect lilo and its cousins, but they make things
harder than they have to be. Why maintain a reduced level of functionality
(software emaciation?) when better alternatives are available?

> Except that in this case it doesn't make anything simpler.

Implicit in the use of initrd is that you have to *make a ramdisk image*,
and then tell your bootloader to load it. If you have a bootloader that can
load multiple images (i.e. the modules themselves) you can skip the first
step.

> BTW, we don't need a special device to handle initrd after that.  Just
> have your initrd image (gzipped, whatever) in the archive 
> under /initrd.
> After that /init will have the contents in that file (regular file, at
> that) and can do whatever it bloody wants.

Again, even the new scheme will still involve the creation of an initrd. I'm
saying, as a user, it would be easier for me to not do this, and just modify
a .conf file to have the driver loaded early-on.

I'm not arguing that the new initrd won't be better than the old initrd
(because obviously you are right) I'm arguing that no matter how whizzy
initrd is, it's still an unnecessary step, and it's one that other OSs (e.g.
FreeBSD) omit in favor of the approach I'm advocating.

> IOW, we are backwards compatible with old
> loaders.

No progress will ever be made if we cater to the lowest common denominator.

Regards -- Andy

RE: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Tue, 18 Dec 2001, Grover, Andrew wrote:

> I'm not arguing that the new initrd won't be better than the old initrd
> (because obviously you are right) I'm arguing that no matter how whizzy
> initrd is, it's still an unnecessary step, and it's one that other OSs (e.g.
> FreeBSD) omit in favor of the approach I'm advocating.

Learn to read.  You don't _have_ to have initrd.  At all.  There's nothing
to stop your loader from putting whatever cpio archive it likes - it
doesn't involve anything other than slapping files you want together
putting their owner/group/size/timestamps/mode/name before each of them.
Anything that puts a bunch of modules in core will have to do equivalent
job.

Re: Booting a modular kernel through a multiple streams file

[M. R. Brown]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]

* Grover, Andrew <andrew.grover@intel.com> on Tue, Dec 18, 2001:

> > From: Alexander Viro [mailto:viro@math.psu.edu]
> > On Tue, 18 Dec 2001, Grover, Andrew wrote:
> > > GRUB 0.90 does this today.
> > ... and I'm quite sure that EMACS could do it easily.  Let's not talk
> > about GNU bloatware, OK?
> 
> I don't think this is bloatware, especially considering there really isn't
> any cost for having a full-featured bootloader - all its footprint gets
> reclaimed, after all. I respect lilo and its cousins, but they make things
> harder than they have to be. Why maintain a reduced level of functionality
> (software emaciation?) when better alternatives are available?
> 

Available for what?  SuperH?  MIPS?  IA-64?  Your precious GRUB and
Multiboot "standard" are only ever useful on the x86 architecture, so you
would propose a x86-specific global change that doesn't benefit the rest of
the other kernel archs?  Bloat for the sake of?

> 
> > IOW, we are backwards compatible with old
> > loaders.
> 
> No progress will ever be made if we cater to the lowest common denominator.
> 

The i386.

M. R.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Followup to:  <59885C5E3098D511AD690002A5072D3C42D804@orsmsx111.jf.intel.com>
By author:    "Grover, Andrew" <andrew.grover@intel.com>
In newsgroup: linux.dev.kernel
> 
> Implicit in the use of initrd is that you have to *make a ramdisk image*,
> and then tell your bootloader to load it. If you have a bootloader that can
> load multiple images (i.e. the modules themselves) you can skip the first
> step.
> 

initramfs will do this just fine.

> Again, even the new scheme will still involve the creation of an initrd. I'm
> saying, as a user, it would be easier for me to not do this, and just modify
> a .conf file to have the driver loaded early-on.

See above.

> I'm not arguing that the new initrd won't be better than the old initrd
> (because obviously you are right) I'm arguing that no matter how whizzy
> initrd is, it's still an unnecessary step, and it's one that other OSs (e.g.
> FreeBSD) omit in favor of the approach I'm advocating.

Doing things in multiple steps is often appropriate.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Booting a modular kernel through a multiple streams file

[James A Sutherland]

On Tuesday 18 December 2001 7:50 pm, Grover, Andrew wrote:
> > From: Alexander Viro [mailto:viro@math.psu.edu]
> >
> > On Tue, 18 Dec 2001, Grover, Andrew wrote:
> > > GRUB 0.90 does this today.
> >
> > ... and I'm quite sure that EMACS could do it easily.  Let's not talk
> > about GNU bloatware, OK?
>
> I don't think this is bloatware, especially considering there really isn't
> any cost for having a full-featured bootloader - all its footprint gets
> reclaimed, after all.

Not on disk it doesn't. When you're booting from a floppy (distro 
installation, recovery disk, whatever) you only have 1.something Mb to play 
with. Or a network. For that matter, you're complicating CD booting too.

> I respect lilo and its cousins, but they make things
> harder than they have to be. Why maintain a reduced level of functionality
> (software emaciation?) when better alternatives are available?

Not "better". More features/bloat - and when they are "features" most of us 
neither need nor want, why bother? Lilo does everything most of us need, and 
does it better than grub or others: same job with fewer resources.

> > Except that in this case it doesn't make anything simpler.
>
> Implicit in the use of initrd is that you have to *make a ramdisk image*,
> and then tell your bootloader to load it. If you have a bootloader that can
> load multiple images (i.e. the modules themselves) you can skip the first
> step.

Initramfs will do this, it seems. Alternatively, you might have to copy some 
files into a tarball - oh, the stress! Oh, wait - you just compiled 100+Mb of 
C source to make that kernel and the modules. Somehow, making a tarball out 
of the modules doesn't seem too stressful to me.

> > BTW, we don't need a special device to handle initrd after that.  Just
> > have your initrd image (gzipped, whatever) in the archive
> > under /initrd.
> > After that /init will have the contents in that file (regular file, at
> > that) and can do whatever it bloody wants.
>
> Again, even the new scheme will still involve the creation of an initrd.
> I'm saying, as a user, it would be easier for me to not do this, and just
> modify a .conf file to have the driver loaded early-on.

OK, make the conf file a shell script which copies the modules into a tarball 
or initrd image.

> I'm not arguing that the new initrd won't be better than the old initrd
> (because obviously you are right) I'm arguing that no matter how whizzy
> initrd is, it's still an unnecessary step, and it's one that other OSs
> (e.g. FreeBSD) omit in favor of the approach I'm advocating.

Feeping creaturism in the bootloader, you mean? Great... while we're at it, 
how about building gcc, make, tar and gzip into the bootloader, along with an 
FTP client, so it can download and compile the kernel to boot? Chuck in a 
copy of emacs, too, for any changes you want?

You're already compiling the modules from C into .o files; what's the big 
deal with copying them into a tarball?!

> > IOW, we are backwards compatible with old
> > loaders.
>
> No progress will ever be made if we cater to the lowest common denominator.

You're right. Let's drop x86 support, and require everyone to buy a nice new 
POWER4 box from IBM.


James.

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Wednesday 19 December 2001 4:34, James A Sutherland wrote:

> Initramfs will do this, it seems. Alternatively, you might have to copy
> some files into a tarball - oh, the stress! Oh, wait - you just compiled
> 100+Mb of C source to make that kernel and the modules. Somehow, making a
> tarball out of the modules doesn't seem too stressful to me.

Do it for 15 production systems, each with varying hardware.
Now do it again for kernel revision 2. Now again for rev 3. Now
again for rev 1 of 2.4.17...

OR

List the names of the modules for boot ONCE in the systems 'grub.conf'
file, and just create a 'current' symlink each time you install
a new kernel and modules. A simple user land util can parge depmod to
give you to right order...no bloat needed.

FYI I only ned create a 'vmlinuz' for new kernel install as it is.
If I could do the same for modules, I wouldn't have a 1.3MB
'catch all' bzImage.

> OK, make the conf file a shell script which copies the modules into a
> tarball or initrd image.

Oh that's nice, clean, and standardized! Again do it for 15 machines...

Many of you people spouting about 'kernel design' really need to get
a clue about putting these things into real world practise, across
mutiple machines, that must stay running...



-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[James A Sutherland]

On Sunday 23 December 2001 1:02 am, Dave Cinege wrote:
> On Wednesday 19 December 2001 4:34, James A Sutherland wrote:
> > Initramfs will do this, it seems. Alternatively, you might have to copy
> > some files into a tarball - oh, the stress! Oh, wait - you just compiled
> > 100+Mb of C source to make that kernel and the modules. Somehow, making a
> > tarball out of the modules doesn't seem too stressful to me.
>
> Do it for 15 production systems, each with varying hardware.
> Now do it again for kernel revision 2. Now again for rev 3. Now
> again for rev 1 of 2.4.17...

i.e. build kernel image and associated modules (you need to do that anyway). 
Then have a configuration file on each machine which drops the modules you 
need into a tarball.

Or have a configuration file which does precisely the same thing at boottime, 
with extra overhead in the boot loader.

> OR
>
> List the names of the modules for boot ONCE in the systems 'grub.conf'
> file, and just create a 'current' symlink each time you install
> a new kernel and modules. A simple user land util can parge depmod to
> give you to right order...no bloat needed.

Which you could also do with a standard shellscript in your initfs, thus 
avoiding any extra bootloader complexity at all.
 
> FYI I only ned create a 'vmlinuz' for new kernel install as it is.
> If I could do the same for modules, I wouldn't have a 1.3MB
> 'catch all' bzImage.

Eh? WTF does bzImage have to do with this? You can also standardise your set 
of modules: just build all the ones you need.

> > OK, make the conf file a shell script which copies the modules into a
> > tarball or initrd image.
>
> Oh that's nice, clean, and standardized! Again do it for 15 machines...

Remembering that in most cases loading drivers for hardware you don't have is 
harmless, I doubt the 15 all REQUIRE different sets of modules. You could 
probably get away with attempting to load a single set; the unneeded drivers 
will just fail to load.

> Many of you people spouting about 'kernel design' really need to get
> a clue about putting these things into real world practise, across
> mutiple machines, that must stay running...

If nonstop uptime is a requirement, you can't change kernel anyway, so what's 
the problem? :-)

(And here, at least, most of the hardware is deliberately homogenous: of the 
couple of thousand PCs in use, there are only half a dozen distinct hardware 
setups.)


James.

Re: Booting a modular kernel through a multiple streams file

[Erik Mouw]

On Tue, Dec 18, 2001 at 11:50:47AM -0800, Grover, Andrew wrote:
> > From: Alexander Viro [mailto:viro@math.psu.edu]
> > On Tue, 18 Dec 2001, Grover, Andrew wrote:
> > > GRUB 0.90 does this today.
> > ... and I'm quite sure that EMACS could do it easily.  Let's not talk
> > about GNU bloatware, OK?
> 
> I don't think this is bloatware, especially considering there really isn't
> any cost for having a full-featured bootloader - all its footprint gets
> reclaimed, after all. I respect lilo and its cousins, but they make things
> harder than they have to be. Why maintain a reduced level of functionality
> (software emaciation?) when better alternatives are available?

I think it's time to wake you up from your warm and fuzzy "all the
world is an x86" dream.

Al's initramfs works on *all* architectures, not only on an
architecture which happens to have a bootloader bloated enough to be an
operating system in itself.

Let me disturb you even further: you might think that a full featured
bootloader/firmware is a Good Thing, but in the ARM Linux world we have
learned that a simple bootloader is *much* better for maintainability.
I think the ACPI problems exactly show my point: depending on BIOS
vendors to get the ACPI tables right just doesn't work.


Erik

-- 
J.A.K. (Erik) Mouw, Information and Communication Theory Group, Faculty
of Information Technology and Systems, Delft University of Technology,
PO BOX 5031, 2600 GA Delft, The Netherlands  Phone: +31-15-2783635
Fax: +31-15-2781843  Email: J.A.K.Mouw@its.tudelft.nl
WWW: http://www-ict.its.tudelft.nl/~erik/

RE: Booting a modular kernel through a multiple streams file

[Torrey Hoffman]

Grover, Andrew wrote:

[...]

> I'm arguing that no matter 
> how whizzy
> initrd is, it's still an unnecessary step, and it's one that 
> other OSs (e.g.
> FreeBSD) omit in favor of the approach I'm advocating.

Well, I'm a linux user more than a developer but I disagree.

It isn't hard to make an initrd, and initrd will always be 
neccessary for some things.  Since we can't get rid of it 
and don't want to (*), I like the approach of making it better.

* I network boot (via pxelinux boot loader) a linux kernel and 
initrd.gz, total size <2 MB, which then proceeds to:

1 - load network drivers, do dhcp
2 - partition and format the hard disk
3 - mount the new filesystems
4 - download (with retry/resume), unpack, cryptographically 
    verify, and install a complete (~100 mb) embedded linux
    system.  
5 - while it does all the above, it shows pretty graphics and
    "please wait while we setup your system" messages on the
    attached television set via framebuffer support.

You may be able to get grub to do the first step but the next
four would be... challenging.  I agree that grub is cool, though.

Torrey Hoffman

Re: Booting a modular kernel through a multiple streams file

[Eric W. Biederman]

Alexander Viro <viro@math.psu.edu> writes:

> On 19 Dec 2001, Eric W. Biederman wrote:
> 
> > Alexander Viro <viro@math.psu.edu> writes:
> > 
> > > They are related to the fact that we enter userland too late.  Which can
> > > (and should) be fixed.
> > 
> > Agreed, we need to be careful with the userland we introduce as it can
> > easily get huge if we use standard components.
> 
> If by standard components you mean glibc - definitely.  It's not going
> to be there.

Actually any general purpose piece of software.  glibc is big and just
about everything else on a linux box is big for the same reasons, it
handles lots of cases.  glibc is extra big because it isn't
stripped. 

Just for credentials I have a static user space dhcp and tftp just
16KB (uncompressed).  If I knew what was involved in the nfs
mount code I'd contribute this and you would be feature complete.


> > Have you done the work to allow the cpio archive to be changed in
> > a binary kernel yet?
> 
> It can be left by the loader in the way we currently leave initrd.

O.k. but this isn't implemented currently correct?  The whole
bootloader leaving something, and having something included with
the kernel build is where I see unresolved technical details.

Probably the simplest solution is to allow files to come from both
locations with the additional bootloader files overwriting the
compiled in files.  And just dropping initrd support altogether.

Currently there is another issue of how to replace the compiled
in default files, but that can probably be resolved after we start
sucking in cpio archives.

The protocol details and the implementation issues are what really
fascinate me at this point.

> > > use for initrd images.  IOW, we are backwards compatible with old
> > > loaders.
> > 
> > Al what is the current status of your initramfs work?
> 
> nfsroot needs more work on it, other than that we can have it very
> soon - almost everything we need is already merged into the tree.

O.k. the kernel has all of your dependencies for this patch, cool.
  
> > So far I have concentrated a lot on the simple, and much less on full
> > featured boot loaders that load data off of filesystems etc.  I need
> > to look at dynamically composing and manipulating images.  Seeing the
> > direction you are taking with initramfs would be a great help there.
> 
> man cpio.

Here I am thinking protocol and logistics, and you think I'm asking
about data format.

> 
> > Portability across cpu architectures for a boot protocol is not a
> > silly goal.  If you stir in a linux booting linux patch plus a small
> > user space you get one bootloader across all of the linux platforms
> > with more potential than any bootloader currently out there.
> 
> You still need to get the original kernel in core...

Given that I haven't seen a bootloader without a stage1 I don't see
this a major problem, and since we run linux on the platform someone
has obviously figured out some way to boot it.   It is the all singing
all dancing all interacting bootloader that is worth standardizing.
Having an X App to select your boot image just might be worth it :)

The only real issue with using linux as a bootloader is there a
serious delay during kernel initialization.  Lately the 10 or 15
seconds of kernel time this takes has been 50% of my boot time.  The
other 45% going to stupid rc scripts.

Eric

Re: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On 19 Dec 2001, Eric W. Biederman wrote:

> Just for credentials I have a static user space dhcp and tftp just
> 16KB (uncompressed).  If I knew what was involved in the nfs
> mount code I'd contribute this and you would be feature complete.

I have _very_ crude userland nfs mount-by-hands.  It works for testing
other parts of the patch, but that's it - it's quick and dirty with
strong emphasis on the latter.

If you want to look at it - I've dropped it on ftp.math.psu.edu/pub/viro;
name's nfsroot.c and it's _crap_.  Don't read without a barf-bag.  Due to
intended use it takes server address and options from environment variables
(kernel passes options to /init that way).  I.e.
# root_server_addr=<address> root_server_path=<path> nfsroot
or
# root_server_addr=<address> root_server_path=<path>,<options> nfsroot
address is decimal, in network order.

Should mount on /mnt.  RPC over UDP by hands (all we need is three RPC
calls in raw, we wait for result before sending the next one, very limited
handling of timeouts).

Again, it's nowhere near anything I would consider suitable for real use.
Works for testing purposes, but that's it.
 
> Probably the simplest solution is to allow files to come from both
> locations with the additional bootloader files overwriting the
> compiled in files.  And just dropping initrd support altogether.

No need.  Pass a concatenation of several cpio/cpio.gz.  Instead of
compiled-in put that bunch first in the cat.  And there's no real
need to drop initrd support - wrap the image with cpio header saying
that it's /initrd and have /init use it instead of /dev/initrd.  End
of story (and minor 250 in rd.c, while we are at it).

I'm doing that variant right now (basically it's removal of compiled-in
array from uncpio, adding support for several concatenated archives
and divorcing initrd_start/initrd_end from CONFIG_BLK_DEV_INITRD).
It's easy, since both cpio and gzip are self-terminating and both
are guaranteed to have non-zero first byte, so we can even allow
arbitrary padding with NULs between the parts.

Re: Booting a modular kernel through a multiple streams file

[Eric W. Biederman]

Alexander Viro <viro@math.psu.edu> writes:

> On 19 Dec 2001, Eric W. Biederman wrote:
> 
> > Just for credentials I have a static user space dhcp and tftp just
> > 16KB (uncompressed).  If I knew what was involved in the nfs
> > mount code I'd contribute this and you would be feature complete.
> 
> I have _very_ crude userland nfs mount-by-hands.  It works for testing
> other parts of the patch, but that's it - it's quick and dirty with
> strong emphasis on the latter.
> 
> If you want to look at it - I've dropped it on
> ftp.math.psu.edu/pub/viro;

[snip]

Thanks I'll take a look, if all it takes is 3 RPC calls then it
doesn't sound too bad, to deal with.
  
> > Probably the simplest solution is to allow files to come from both
> > locations with the additional bootloader files overwriting the
> > compiled in files.  And just dropping initrd support altogether.
> 
> No need.  Pass a concatenation of several cpio/cpio.gz.  Instead of
> compiled-in put that bunch first in the cat.  And there's no real
> need to drop initrd support - wrap the image with cpio header saying
> that it's /initrd and have /init use it instead of /dev/initrd.  End
> of story (and minor 250 in rd.c, while we are at it).

Well that sounds like a reasonable approach.  It sounds like dropping
and re-adding initrd support in a slightly different form.

I guess from what you have described there could be a test ``if !cpio
archive assume old initrd''.  But I can't imagine how that would be
anything but ugly.
 
> I'm doing that variant right now (basically it's removal of compiled-in
> array from uncpio, adding support for several concatenated archives
> and divorcing initrd_start/initrd_end from CONFIG_BLK_DEV_INITRD).
> It's easy, since both cpio and gzip are self-terminating and both
> are guaranteed to have non-zero first byte, so we can even allow
> arbitrary padding with NULs between the parts.

O.k. This sounds quite reasonable on the boot protocol
side.  Bootloaders if they want to take advantage of the new
functionality need to be made aware of the cpio issue but otherwise
can remain blissfully ignorant.

Except for the issue of creating a kernel file with one or more of
these cpio archives attached to the kernel image.  This work seems
completely independent from what I'm doing.  And for simple network
booting it is required to have everything all in one file. 

I have alarm bells ringing in my gut saying there are pieces of your
proposal that are on the edge of being overly complex... But without
source I can't really say.  Arbitrary NULL padding between images is
cool but why?

Eric

Re: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On 19 Dec 2001, Eric W. Biederman wrote:

> I have alarm bells ringing in my gut saying there are pieces of your
> proposal that are on the edge of being overly complex... But without
> source I can't really say.  Arbitrary NULL padding between images is
> cool but why?

	Alignment that might be wanted by loaders.  Take that with hpa - for
all I care it's a non-issue.  while(!*p) p++; added before p = handle_part(p);
in the main loop...

Re: Booting a modular kernel through a multiple streams file

[Eric W. Biederman]

Alexander Viro <viro@math.psu.edu> writes:

> On 19 Dec 2001, Eric W. Biederman wrote:
> 
> > I have alarm bells ringing in my gut saying there are pieces of your
> > proposal that are on the edge of being overly complex... But without
> > source I can't really say.  Arbitrary NULL padding between images is
> > cool but why?
> 
> 	Alignment that might be wanted by loaders.  Take that with hpa - for
> all I care it's a non-issue.  while(!*p) p++; added before p = handle_part(p);
> in the main loop...

Right there are definitely cases where it makes sense, and it isn't too bad.

My basic design filter checks to see if there if there is one feature per
requirement.  At one feature per requirement it is a uninspired design.  At less
than one feature per requirement it approaches an elegant design.  At greater
than one feature per requirement it approaches a crap design.

The whole moving excess kernel code into user space, and use cpio
instead of a raw disk image part is elegant.  I just don't want the
idea to loose it in the small details.

And you have mentioned enough small features my reaction is to want to review
the code and think it through.  So I'll have to look it over very closely next
time you post a patch.

Eric

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Followup to:  <Pine.GSO.4.21.0112191153280.11104-100000@weyl.math.psu.edu>
By author:    Alexander Viro <viro@math.psu.edu>
In newsgroup: linux.dev.kernel
> 
> On 19 Dec 2001, Eric W. Biederman wrote:
> 
> > I have alarm bells ringing in my gut saying there are pieces of your
> > proposal that are on the edge of being overly complex... But without
> > source I can't really say.  Arbitrary NULL padding between images is
> > cool but why?
> 
> 	Alignment that might be wanted by loaders.  Take that with hpa - for
> all I care it's a non-issue.  while(!*p) p++; added before p = handle_part(p);
> in the main loop...
> 

Examples on allowing NULL padding makes life easier for the
bootloader, because it can pick its own alignment:

a) A bootloader uses INT 15h AH=87h to move things to high memory.
   This function can only move 16-bit words.

b) SYSLINUX' builting high bcopy routine can only move 32-bit words.

c) If a boot loader runs in protected mode, it may want to operate on
   4K pages only.

d) A block-oriented boot loader like LILO can simply concatentate the
   blocks of multiple files together (as long as it can make sure
   the slop is zeroed out.)

It's an insignificant addition to the kernel that allows the
bootloader to be potentially significantly simpler, by removing the
corner cases.  This is a Good Thing[TM].

	-hpa

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

RE: Booting a modular kernel through a multiple streams file

[Grover, Andrew]

> From: ebiederm@xmission.com [mailto:ebiederm@xmission.com]
> > Basically what Grub does is loads the kernel modules from disk
> > into memory, and 'tells' the kernel the memory location to load
> > them from, very similar to how an initrd file is loaded. The problem
> > is Linux, is not MBS compilant and doesn't know to look for and load
> > the modules. 
> 
> So tell me how you make an MBS compliant alpha kernel again?

1) Someone writes a MBS spec chapter for Alpha
2) Someone implements it.

Any volunteers? (Eric? ;-))

It's all about scratching an itch, right? Things don't become cross-platform
by themselves. Linux started out 386-only, after all.

Regards -- Andy

Re: Booting a modular kernel through a multiple streams file

[Eric W. Biederman]

"Grover, Andrew" <andrew.grover@intel.com> writes:

> > From: ebiederm@xmission.com [mailto:ebiederm@xmission.com]
> > > Basically what Grub does is loads the kernel modules from disk
> > > into memory, and 'tells' the kernel the memory location to load
> > > them from, very similar to how an initrd file is loaded. The problem
> > > is Linux, is not MBS compilant and doesn't know to look for and load
> > > the modules. 
> > 
> > So tell me how you make an MBS compliant alpha kernel again?
> 
> 1) Someone writes a MBS spec chapter for Alpha

Nope it isn't that simple.  Multiboot doesn't extend gracefully at all,
not to multiple architectures, not to new parameters, not to multiple
vendors.   If you do extend it you get a real mess.

Which is why it is not good to adopt.  

> 2) Someone implements it.
> 
> Any volunteers? (Eric? ;-))

I've looked and I'm not going there.  Multiboot is just plain nasty,
and quite poorly specified as well.

I'll do something better but not that.  In fact I have already started...

> It's all about scratching an itch, right? Things don't become cross-platform
> by themselves. Linux started out 386-only, after all.

GRUB allows some very neat things and it allows for it with the
multiboot stuff.  And everyone seems to assume that because what you
can do with GRUB is good, multiboot must be good as well.  But have
you ever wondered why no other bootloader implementor was interested?

Eric

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Followup to:  <m1r8pmqotz.fsf@frodo.biederman.org>
By author:    ebiederm@xmission.com (Eric W. Biederman)
In newsgroup: linux.dev.kernel
> 
> GRUB allows some very neat things and it allows for it with the
> multiboot stuff.  And everyone seems to assume that because what you
> can do with GRUB is good, multiboot must be good as well.  But have
> you ever wondered why no other bootloader implementor was interested?
> 

Very well said!!  I really think that an initramfs protocol which
allows (but does not require!!) synthesis in the boot loader is the
way to go.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Booting a modular kernel through a multiple streams file

[Eric W. Biederman]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Very well said!!  I really think that an initramfs protocol which
> allows (but does not require!!) synthesis in the boot loader is the
> way to go.

Agreed.  Just one more thought before I take off on vacation.  

The concept of a generic boot protocol is something I like.  And it
seems the ability to include at least one and in general multiple
files with a kernel is highly desireable. 

In concepts that are well understood we have several potential
candidates.  We have the concept of a single file.  We have the
concept of multiple files.  And we have the concept of object files.

A single file works but it really doesn't offer much flexibility in
those cases where you really need it.  Modules don't need any support
by the loaded kernel at all because they are linked in transparently.
The downside with modules is that because the bootloader must include
a linker this requires a fair amount of work in the bootloader.  It
needs a very particular kernel setup.  Plus it has issues with binary
only modules and a GPL'd kernel.  A filesystem in a stream format
(tar/cpio/pax...) gets the job done.  You can use a relatively stupid
bootloader, so verification for correctness is easier.  Plus a
bootloader can inject files into the filesystem with the expectation
that they will meat with a reasonable interpretation.  Something not
possible with modules as they are code first data second.

So in the generic sense I agree that a bootloader loaded filesystem is
the way to go.  The attributes given by the GRUB filesystem are
inadequate, you can't even describe a device node.  And except for
permissions something running in userspace at kernel init time is
going to exercise the system to it's limit.  So we really need a
filesystem format that can handle extended attributes and all kinds of
other weird and strange things.  Which is basically a requirement for
simple extensibility.

So we have the choice now of being picky in the format we select or
long term needing to support multiple stream filesystems.  Given that
we would like bootloaders to have the ability to inject individual
files being picky now is probably a good idea.  Since it would be
silly to support multiple filesystems in the bootloader.  Everyone
except GRUB seems reluctant to do that sort of thing.

Eric