RE: Booting a modular kernel through a multiple streams file

RE: Booting a modular kernel through a multiple streams file

[Grover, Andrew]

> From: Alexander Viro [mailto:viro@math.psu.edu]
> On Mon, 17 Dec 2001, Grover, Andrew wrote:
> > I don't think multiple streams is a good idea, but did you 
> all see the patch
> > by Christian Koenig to let the bootloader load modules? 
> That seems to solve
> > the problem nicely.
> 
> That puts an awful lot of smarts into bootloader and creates 
> code duplication
> for no good reason.
> 
> We _already_ have code for loading modules.  And it's going 
> to stay, no
> matter what happens on boot.  So why the hell duplicate that 
> in $BIGNUM
> unrelated pieces of software (LILO, SYSLINUX, MILO, SILO, etc.)?  Just
> to create extra fun with debugging?

OK so (correct me if I'm wrong) there are two parts to loading modules -
getting them in memory, and then resolving references. My understanding of
Christian's work is that the bootloader (e.g. GRUB) gets them in memory, but
then it is up to the kernel linker to resolve refs. So yes, there would be
an additional piece of code (marked __init), but it would not have to be
duplicated for each bootloader -- all they have to do is get the modules in
memory and indicate via the multiboot struct where they are.

I don't think this will obsolete any existing boot methods, but it seems
like an additional genuinely useful capability for the Linux kernel to have.

Regards -- Andy

RE: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Mon, 17 Dec 2001, Grover, Andrew wrote:

> 
> OK so (correct me if I'm wrong) there are two parts to loading modules -
> getting them in memory, and then resolving references. My understanding of
> Christian's work is that the bootloader (e.g. GRUB) gets them in memory, but
> then it is up to the kernel linker to resolve refs. So yes, there would be
> an additional piece of code (marked __init), but it would not have to be
> duplicated for each bootloader -- all they have to do is get the modules in
> memory and indicate via the multiboot struct where they are.

*shrug*  Your "all they have to do" is quite heavy.
 
> I don't think this will obsolete any existing boot methods, but it seems
> like an additional genuinely useful capability for the Linux kernel to have.

I've had a very dubious pleasure of dealing with our boot sequence lately.
Adding more cruft to it (including in-kernel linker, for fsck sake) is
_not_ a good idea.

Folks, whatever had happened with "if it can be done in userland - don't
put it into the kernel"?

That goes for a _lot_ of code.  Mounting root.  Detecting the type of
initrd contents.  Loading ramdisk from floppies.  Asking to press
key (you really ought to look what is done for _that_).  Speaking
DHCP - we have a kernel DHCP client, of all things.  All that stuff
can (and should) be done from userland process.  And there's much
more of the same kind.

There is a word for that and that word is "crap".

Let loader leave an archive to be unpacked into rootfs?  Sure.  Let kernel
exec /init on rootfs and leave the rest to it?  Absolutely.  But let's
stop adding userland stuff into the kernel.  Loading modules _can_ be
done from userland - insmod does it just fine.  And that's where it should
be done.

Re: Booting a modular kernel through a multiple streams file

[Craig Christophel]

On Monday 17 December 2001 21:31, Alexander Viro wrote:
> There is a word for that and that word is "crap".
>
> Let loader leave an archive to be unpacked into rootfs?  Sure.  Let kernel
> exec /init on rootfs and leave the rest to it?  Absolutely.  But let's
> stop adding userland stuff into the kernel. 


I agree 100%.  there are several other popular systems that try to do 
everything in-kernel and it adds to the instability overall, besides it gives 
less freedom to customize with standard tools.



Craig.


 Loading modules _can_ be
> done from userland - insmod does it just fine.  And that's where it should
> be done.

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[Christian Koenig]

Hi,

On Monday 17 December 2001 23:10, Grover, Andrew wrote:
> > From: Alexander Viro [mailto:viro@math.psu.edu]
> >
> > On Mon, 17 Dec 2001, Grover, Andrew wrote:
> > > I don't think multiple streams is a good idea, but did you
> >
> > all see the patch
> >
> > > by Christian Koenig to let the bootloader load modules?
> >
> > That seems to solve
> >
> > > the problem nicely.
> >
> > That puts an awful lot of smarts into bootloader and creates
> > code duplication
> > for no good reason.

I agree, but the problem I have with Initrd is that you could only have 
1 single initrd-image on your hard disk / loaded by the boot-loader.

Imaging my situation, I have some removeable hard-disks, changing it between 
a large amount of Systems all with individual Hardware-configuration.
(Some SCSI, some ide, and a cluster booting with pxegrub/Floppy-Disks).

With the last relleas of my patch I'm now capable to have a grub menu.lst 
looking like:

tile Kernel 2.4.14 Adaptec xyz/ne2000
kernel /boot/vmlinuz-2.4.14 root=/dev/sdxyz ...

# Loading SCSI-Drivers
modulesfromfile /etc/modules-adaptec /lib/modules/2.4.14/modules.dep 

# Loading Network-Drivers
modulesfromfile /etc/modules-ne2000 /lib/modules/2.4.14/modules.dep

tile Kernel 2.4.14 Tecram xyz / Inter EExpresPro 

# Loading SCSI-Drivers
modulesfromfile /etc/modules-tecram /lib/modules/2.4.14/modules.dep 

# Loading Network-Drivers
modulesfromfile /etc/modules-eepro /lib/modules/2.4.14/modules.dep 
......

Could you Imaging what work it would be to create an individual initrd-Image 
/ Kernel Image for every system I have. Beside this I sometimes have the 
Problem that I come to a system never seen bevore and have to boot with my 
Floppy Disk's. (I'm now looking forward that grub someday gets capable to use 
cd-roms directly).

I know this it isn't the best solution to add a module-loader to the Kernel, 
but did you have a better idea ?

MfG, Christian Koenig.

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[Jesse Pollard]

---------  Received message begins Here  ---------

> 
> Hi,
> 
> On Monday 17 December 2001 23:10, Grover, Andrew wrote:
> > > From: Alexander Viro [mailto:viro@math.psu.edu]
> > >
> > > On Mon, 17 Dec 2001, Grover, Andrew wrote:
> > > > I don't think multiple streams is a good idea, but did you
> > >
> > > all see the patch
> > >
> > > > by Christian Koenig to let the bootloader load modules?
> > >
> > > That seems to solve
> > >
> > > > the problem nicely.
> > >
> > > That puts an awful lot of smarts into bootloader and creates
> > > code duplication
> > > for no good reason.
> 
> I agree, but the problem I have with Initrd is that you could only have 
> 1 single initrd-image on your hard disk / loaded by the boot-loader.
> 
> Imaging my situation, I have some removeable hard-disks, changing it between 
> a large amount of Systems all with individual Hardware-configuration.
> (Some SCSI, some ide, and a cluster booting with pxegrub/Floppy-Disks).
> 
> With the last relleas of my patch I'm now capable to have a grub menu.lst 
> looking like:
> 
> tile Kernel 2.4.14 Adaptec xyz/ne2000
> kernel /boot/vmlinuz-2.4.14 root=/dev/sdxyz ...
> 
> # Loading SCSI-Drivers
> modulesfromfile /etc/modules-adaptec /lib/modules/2.4.14/modules.dep 
> 
> # Loading Network-Drivers
> modulesfromfile /etc/modules-ne2000 /lib/modules/2.4.14/modules.dep
> 
> tile Kernel 2.4.14 Tecram xyz / Inter EExpresPro 
> 
> # Loading SCSI-Drivers
> modulesfromfile /etc/modules-tecram /lib/modules/2.4.14/modules.dep 
> 
> # Loading Network-Drivers
> modulesfromfile /etc/modules-eepro /lib/modules/2.4.14/modules.dep 
> ......
> 
> Could you Imaging what work it would be to create an individual initrd-Image 
> / Kernel Image for every system I have. Beside this I sometimes have the 
> Problem that I come to a system never seen bevore and have to boot with my 
> Floppy Disk's. (I'm now looking forward that grub someday gets capable to use 
> cd-roms directly).
> 
> I know this it isn't the best solution to add a module-loader to the Kernel, 
> but did you have a better idea ?

How about a three step loader:
1. primary boot (PROM) loads secondary boot <any boot program>
2. the <any boot program> loads a "superboot"
3. superboot has filesystem knowlege of a boot device (this is different) and
   Loads base kernel
4. installs required modules from boot device (this is not done currently)
5. starts kernel

The kernel should not need any installed drivers. All of them could be
installed after the kernel is loaded, but before being started.

This used to be/is done by many systems (VAX, Sun, IBM, SGI...) The PROM loader
loads a boot loader which loads boot program.

There are several ways to implement this: The "superboot" program doesn't
even have to know more than one filesystem type. A very simple file system
using contigueous, variable length segments, each segment containing one file.
The only requirement is that the /boot filesystem be accessable to both
Linux AND the loader. and yes, the /lib/modules/... should be moved to
/boot/modules directory. The /boot filesystem should need be no larger
than 2-5 MB depending on how many kernels and modules are to be present
at one time.

And yes, a tar formatted file should be sufficient if it were on a partition
of it's own (below cylender 1024 for the antique systems). If desired, the
tar file(fs?) could even be compressed (in which case, the kernel wouldn't
need to be). Even an ISO9660 format might work, with some modifications
(it WOULD be desirable to be able to write directly to it, and it would
 have to allow for fragmentation of the free space)

I think it would be much easier to use a block device though.. first block
is a directory block, directory containing the minimum of 
"name:filetype:starting block:length (in blocks)". If the filetype is a
directory, then the blocks pointed to are a subdirectory, treated in the
same manner as the first directory. True, there are no inodes in this,
but they could be added as just a unique number in an additional field
in a directory entry. NO hard links, NO symbolic links, NO owner (mounted
ONLY as uid=0,gid=0) intended for /boot. The superblock would only need to
know where the first filesystem block is, the length of the file name array,
and the size of the partition (for limit error checking).

Even an ISO9660 format might work, with some modifications.

The block nature of the file boundaries would allow for a possible DMA load
of an entire file into memory directly. Installing a module might require a
copy operation though. The contigueous nature of the files eliminates the
need for an fsck (not possible to corrupt the files, and it is easy to
recreate the entire filesystem). This also simplifies the secondary boot -
there is only one starting block number and length needed (file starting
block number + FS offset).

It may not even be necessary that the filesystem be known to linux, provided
some tools similar to the "mtools" were available to put files into the
partition, defragment the filesystem (move files to put all contigueous free
space at the end), get files out, make directory listings... It would just
make support simpler to mount it.

Oh - another thing - this could eliminate the need for the kernel to include
a "decompress" startup; this should be done by the "superboot" loader,
allowing a choice of different kinds of compression without kernel modification.

Who knows - it may be possible to merge <any boot program> with the "superboot"
and produce the equivalent to what is being done now, but without requiring
any built-in modules. Personally, I don't think so because such a loader
would become more usefull as a "mini-os" capable of being used to run
diagnostics as well as booting the system.

Suggested contents of the boot fs:

boot	- the superboot program
vmlinux	- the kernel (one or more instances)
<version> - directory containing modules for kernel <version>
	    multiple instances, one per kernel version. The contents of the
	    directory could even be in the same as in /lib/modules/<vers>
	    with the necessary modules dependancy files and directories of
	    for the drivers.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[H. Peter Anvin]

Christian Koenig wrote:
> 
> I agree, but the problem I have with Initrd is that you could only have 
> 1 single initrd-image on your hard disk / loaded by the boot-loader.
> 

Al and I have already discussed an improved approach.  What we want is 
rather than initrd an initramfs.  I have specifically discussed the 
initramfs protocol with Al so it can be combined from multiple sources.

	-hpa

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[Eric W. Biederman]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Eric W. Biederman wrote:
> 
> > I have a personally dislike for using firmware calls to drive hardware
> > devices, so I won't be picking that up.  But I am interested in what
> > it requires to not burn bridges.  So I can make certain a linux kernel
> > loaded with a linux booting linux patch can requery the firmware.
> >
> 
> > My impression is that the linux kernel already does the important
> > things by not smashing firmware reserved memory, (assuming you aren't
> > loaded with loadlin).  So all that is required is to switch the idt
> > back to address 0, and switch the cpu back to 16bit real mode.
> > But if you know of other cases that need to be handled I would be
> > happy to hear about it.
> >
> 
> 
> Unfortunately that's not the case.  The big issue is "who owns the interrupt
> controller", and "who owns the interrupts." 
[snip]



> You can check out the BIOS extender I wrote for genesis at
> ftp://ftp.zytor.com/pub/linux/genesis/

Thanks.  I've got genesis-1.10 now looking and digesting to see if you
have any unexpected tricks takes a little longer.

For my purposes I intend to fully disable the BIOS and then after I
have done all of my work, reenable the BIOS.  Which should be a little
easier and have a slightly different set of issues. 

>From the 10,000 foot level it looks like I am pretty safe already
except for those BIOS functions that drive the hardware.  For those I
need to setup the legacy PIC back to it's default setting, and
possibly a few other hardware things.   I wonder just how sensitive
the an x86 BIOS really is to changing those things...

For the most part I find it perfectly acceptable if I break all of the
firmware hardware drivers.  As long as the information callbacks are
preserved.  But preserver enough so I could load dos from linux would
be nice.

Eric

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[H. Peter Anvin]

Eric W. Biederman wrote:
> 
>>From the 10,000 foot level it looks like I am pretty safe already
> except for those BIOS functions that drive the hardware.  For those I
> need to setup the legacy PIC back to it's default setting, and
> possibly a few other hardware things.   I wonder just how sensitive
> the an x86 BIOS really is to changing those things...
> 

You never know, especially since part of the BIOS might be an external 
SCSI or network card BIOS...

	-hpa

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[Eric W. Biederman]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Eric W. Biederman wrote:
> >
> >>From the 10,000 foot level it looks like I am pretty safe already
> > except for those BIOS functions that drive the hardware.  For those I
> > need to setup the legacy PIC back to it's default setting, and
> > possibly a few other hardware things.   I wonder just how sensitive
> > the an x86 BIOS really is to changing those things...
> >
> 
> You never know, especially since part of the BIOS might be an external SCSI or
> network card BIOS...

Which just goes to show what a fragile firmware design it is, to have
firmware callbacks doing device I/O.  I think the whole approach of
having firmware callbacks is fundamentally flawed but I'll do my best
to keep it working, for those things that care.  If it works over 50%
of the time I'm happy...

The much more interesting challenge is to get the linux kernel drivers
to shutdown their hardware well enough that they can still drive the
hardware when linux restarts.  And if the hardware can't do that it is
buggy...


Eric

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[H. Peter Anvin]

Followup to:  <m1zo4fursh.fsf@frodo.biederman.org>
By author:    ebiederm@xmission.com (Eric W. Biederman)
In newsgroup: linux.dev.kernel
> 
> Which just goes to show what a fragile firmware design it is, to have
> firmware callbacks doing device I/O.  I think the whole approach of
> having firmware callbacks is fundamentally flawed but I'll do my best
> to keep it working, for those things that care.  If it works over 50%
> of the time I'm happy...
> 

NAK.  You can make it perfectly robust thankyouverymuch, as long as
you don't try to *mix* firmware and poking directly at the
hardware... this is a classic "who owns what" class problem.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[Eric W. Biederman]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Followup to:  <m1zo4fursh.fsf@frodo.biederman.org>
> By author:    ebiederm@xmission.com (Eric W. Biederman)
> In newsgroup: linux.dev.kernel
> > 
> > Which just goes to show what a fragile firmware design it is, to have
> > firmware callbacks doing device I/O.  I think the whole approach of
> > having firmware callbacks is fundamentally flawed but I'll do my best
> > to keep it working, for those things that care.  If it works over 50%
> > of the time I'm happy...
> > 
> 
> NAK.  You can make it perfectly robust thankyouverymuch, as long as
> you don't try to *mix* firmware and poking directly at the
> hardware... this is a classic "who owns what" class problem.

I agree that I could keep it working as well as it ever would.  Not
that x86 firmware or any software is ever perfectly working.

At this point in time I live in a world where 99+% of the time the
hardware is owned by the operating system, and the firmware is just
there to get the operating system loaded, and to hold details about
the motherboard that the operating system can not find out by probing
the hardware.

For the cases I find important I get better reliability and
portability by never involving the firmware at all.  If there is a
problem with a driver I can fix it.  If I want to switch cpus I can.
Admittedly the cost for native drivers is high, but if I don't have to
pay that cost twice and can actually reuse my OS drivers.  It isn't a
price I mind paying.

I care about not trashing the firmware so a newer probe routine can
find out more precisely or robustly what is on a motherboard.  Having
a reasonable chance that the firmware can also still drive the
hardware is a plus. 

I criticize firmware designers, not to attack anyones dependence on
the firmware.  But more to make certain I never implement anything
like that.  

I don't think I have seen a firmware design where someone has designed
it with the assumption that humans mess up.  Instead every firmware
interface I have seen seems to be designed by asking how can I include
every possible desirable feature.  Since it is painful to fix or
replace firmware this is a real issue. 

I have seen alpha firmware getting confused when the operating system
uses the hardware, when rebooting on the alpha.  Which is why I am
sensitive to it.

Eric

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[H. Peter Anvin]

Eric W. Biederman wrote:

> 
>>Followup to:  <m1zo4fursh.fsf@frodo.biederman.org>
>>By author:    ebiederm@xmission.com (Eric W. Biederman)
>>In newsgroup: linux.dev.kernel
>>
>>>Which just goes to show what a fragile firmware design it is, to have
>>>firmware callbacks doing device I/O.  I think the whole approach of
>>>having firmware callbacks is fundamentally flawed but I'll do my best
>>>to keep it working, for those things that care.  If it works over 50%
>>>of the time I'm happy...
>>>
>>>
>>NAK.  You can make it perfectly robust thankyouverymuch, as long as
>>you don't try to *mix* firmware and poking directly at the
>>hardware... this is a classic "who owns what" class problem.
>>
> 
> I agree that I could keep it working as well as it ever would.  Not
> that x86 firmware or any software is ever perfectly working.
> 


True enough; I guess what I meant is you can design it so that it is *no 
less unreliable than the underlying firmware*.


> At this point in time I live in a world where 99+% of the time the
> hardware is owned by the operating system, and the firmware is just
> there to get the operating system loaded, and to hold details about
> the motherboard that the operating system can not find out by probing
> the hardware.


Right.  My point was that you want to treat the transition as modal -- 
you have "boot mode" and you have "runtime".  My work was to build a 
Linux kernel derivative which could run in "boot mode" and therefore not 
require drivers.  The intent was that this "boot mode" kernel would then 
load the real "runtime" kernel and transition to it.

My comment was mostly that there is a persistent myth that you can't use 
the x86 firmware from protected mode.  You can't, directly, but you can 
get the functionality through other means.  This is hardly a new 
technique; in fact, it's very similar to the DOS extenders of old times; 
in fact, it's somewhat simpler.

	-hpa

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[Eric W. Biederman]

"H. Peter Anvin" <hpa@zytor.com> writes:

> My comment was mostly that there is a persistent myth that you can't use the x86
> firmware from protected mode.  You can't, directly, but you can get the
> functionality through other means.  This is hardly a new technique; in fact,
> it's very similar to the DOS extenders of old times; in fact, it's somewhat
> simpler.

Agreed.  And to completely dispel the myth.  Etherboot has been doing
something similar for years.  It disables interrupts in 32bit mode so
it doesn't have quite as much work to do but otherwise it is pretty
much the same picture.

I finally tracked down the reason why Setup.S is run in real mode,
instead of being called from protected mode.  And that is in extremely
hostile environments (like loadlin works in) loading the kernel wrecks
the firmware callbacks.  So you must do your BIOS calls as a special
case before you switch to protected mode.

Eric

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[H. Peter Anvin]

Eric W. Biederman wrote:

> 
> Agreed.  And to completely dispel the myth.  Etherboot has been doing
> something similar for years.  It disables interrupts in 32bit mode so
> it doesn't have quite as much work to do but otherwise it is pretty
> much the same picture.
> 


If you disable interrupts in 32-bit mode a lot of things will not work.


> I finally tracked down the reason why Setup.S is run in real mode,
> instead of being called from protected mode.  And that is in extremely
> hostile environments (like loadlin works in) loading the kernel wrecks
> the firmware callbacks.  So you must do your BIOS calls as a special
> case before you switch to protected mode.


No, it's because it was easier to do it that way -- do all BIOS calls 
once and for all in the early part of the execution of the kernel, and 
then forget about it.

	-hpa

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[Eric W. Biederman]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Eric W. Biederman wrote:
> 
> > Agreed.  And to completely dispel the myth.  Etherboot has been doing
> > something similar for years.  It disables interrupts in 32bit mode so
> > it doesn't have quite as much work to do but otherwise it is pretty
> > much the same picture.
> >
> 
> 
> If you disable interrupts in 32-bit mode a lot of things will not work.

The basic technique use by etherboot is:
In 32bit mode handle no interrupts.  When you want to do a BIOS call
switch to 16bit mode enable interrupts do the call.  disable interrupts
and switch back to 32bit mode.

As far as I can tell this is a similar idea to what you are doing in
genesis.  etherboot doesn't need everything so it doesn't handle the
general case.  But given that it is public program, I mentioned it
because to help dispell the myth that you can do bios call from 32 bit
protected mode..

> > I finally tracked down the reason why Setup.S is run in real mode,
> > instead of being called from protected mode.  And that is in extremely
> > hostile environments (like loadlin works in) loading the kernel wrecks
> > the firmware callbacks.  So you must do your BIOS calls as a special
> > case before you switch to protected mode.
> 
> 
> No, it's because it was easier to do it that way -- do all BIOS calls once and
> for all in the early part of the execution of the kernel, and then forget about
> it.

That may have been the original reason.  But the reason to keep the
code that way is so we work with loadlin, (and any kin it might have).  

I have tested it and after you are loaded by loadlin you cannot go
back and make BIOS calls.  The problem is that dos TSR's and device
drivers have intercepted BIOS interrupts, and we stomp all over
those.

It may be possible to overcome this by saving the state of the entire
dos session but to be safe we would need to take a core dump of the
entire machine.  And for such little gain I don't think that is worth
it.

Eric

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[T. A.]

    What about considering one of the simpler filesystems or archive formats
instead?  How much "Unix"-ism is required to be retained in the archive?
(permissions, device files, etc?)

    As for the bigendianness... Is it really relevant since each kernel is
tied to its own platform?  And if it is may it be better to use the native
format of the 98% or so of the Linux machines out there which are
littleendian (performance and ease of general access on the majority of host
machines comes to mind).

----- Original Message -----
From: "H. Peter Anvin" <hpa@zytor.com>
To: "Alexander Viro" <viro@math.psu.edu>
Cc: <linux-kernel@vger.kernel.org>
Sent: Sunday, December 23, 2001 4:14 AM
Subject: Re: tar vs cpio (was: Booting a modular kernel through a multiple
streams file)


> I have looked through the various forms of tar and cpio, and I'm getting
> the feeling that both are ugly as hell for the purpose of initializing
> initramfs.  The block-based setup of tar (and of the new Austin group
> "pax" format -- really just a new rev of tar) is awful, but some of the
> tape-oriented aspects of cpio isn't much better.  What concerns me about
> cpio in particular:
> It seems to me that this application doesn't really have a particular
> need for backward compatibility, nor for the I/O blocking stuff of
> tar/cpio.  I would certainly be willing to write a set of portable
> utilities to create an archive in a custom format, if that would be more
> desirable.  We'd still use gzip for compression, of course, and have the
> buffer composed as a combination of ".rfs" and ".rfs.gz" files,
> separated by zero-padding.
>
> What I'm talking about would probably still look a lot like the cpio
> header, but probably would use bigendian binary (bigendian because it
> allows the use of the widely portable htons() and htonl() macros), have
> explicit support for hard links, and not require a trailer block.
>
> -hpa
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Monday 17 December 2001 21:31, Alexander Viro wrote:

> *shrug*  Your "all they have to do" is quite heavy.

Lilo (broken), syslinux (not very difficult), GRUB, (already implemented).
The list is not long. It's little differeent then a boot loader supporting
initrd loading.

> > I don't think this will obsolete any existing boot methods, but it seems
> > like an additional genuinely useful capability for the Linux kernel to
> > have.
>
> I've had a very dubious pleasure of dealing with our boot sequence lately.

Yes the guilty are well known... 

> Adding more cruft to it (including in-kernel linker, for fsck sake) is
> _not_ a good idea.

Which is why all of that shit that loads multiple initrd floppies, etc,
etc should be gutting and replaced with a solid initrd system that
can load tar.gz's to tmpfs. Jeez...I happen to have such a patch RIGHT 
HERE....for 3 years now. (minix prior to tmpfs existance...)

Loading initrd and boot time kernel modules, from multiple sources,
belongs in 'pre-kernel' (prom/bootloader) land. Most importantly this
boot loader is already implemented for IA32.

> That goes for a _lot_ of code.  Mounting root.  Detecting the type of
> initrd contents.  Loading ramdisk from floppies.  Asking to press
> key (you really ought to look what is done for _that_).  Speaking
> DHCP - we have a kernel DHCP client, of all things.  All that stuff
> can (and should) be done from userland process. 

Already done (properly) in GRUB. Userland (after the kernel boots) is
no place for this. It's too late to be done cleanly. I agree the kernel
is no place either.
 
> Let loader leave an archive to be unpacked into rootfs?  Sure.  Let kernel
> exec /init on rootfs and leave the rest to it?  Absolutely.  But let's
> stop adding userland stuff into the kernel.  Loading modules _can_ be
> done from userland - insmod does it just fine.  And that's where it should
> be done.

This is a very much a failed concept. In theory it sounds nice. My
experience in
	1) implementing a Linux OS from scratch
	2) administering multiple networks of mission critical servers
says this doesn't work well.  

Dave

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Tuesday 18 December 2001 15:26, Alexander Viro wrote:
> On Tue, 18 Dec 2001, Grover, Andrew wrote:
> > I'm not arguing that the new initrd won't be better than the old initrd
> > (because obviously you are right) I'm arguing that no matter how whizzy
> > initrd is, it's still an unnecessary step, and it's one that other OSs
> > (e.g. FreeBSD) omit in favor of the approach I'm advocating.
>
> Learn to read.  You don't _have_ to have initrd.  At all.  There's nothing
> to stop your loader from putting whatever cpio archive it likes - it
> doesn't involve anything other than slapping files you want together
> putting their owner/group/size/timestamps/mode/name before each of them.
> Anything that puts a bunch of modules in core will have to do equivalent
> job.

Deja Vu: *shrug*  Your "all they have to do" is quite heavy.
(boot loader must implement full cpio/tar[/gzip}

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Followup to:  <E16Hwks-0001wV-00@schizo.psychosis.com>
By author:    Dave Cinege <dcinege@psychosis.com>
In newsgroup: linux.dev.kernel
> 
> Deja Vu: *shrug*  Your "all they have to do" is quite heavy.
> (boot loader must implement full cpio/tar[/gzip}
> 

cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
unacceptable, but should not be required.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On 22 Dec 2001, H. Peter Anvin wrote:

> Followup to:  <E16Hwks-0001wV-00@schizo.psychosis.com>
> By author:    Dave Cinege <dcinege@psychosis.com>
> In newsgroup: linux.dev.kernel
> > 
> > Deja Vu: *shrug*  Your "all they have to do" is quite heavy.
> > (boot loader must implement full cpio/tar[/gzip}
> > 
> 
> cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
> unacceptable, but should not be required.

tar is ugly as hell and not going to be supported on the kernel side.

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Saturday 22 December 2001 21:10, Alexander Viro wrote:

> > cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
> > unacceptable, but should not be required.
>
> tar is ugly as hell and not going to be supported on the kernel side.

Excellent! You've settled on using using an archiver format nobody uses,
instead of the defacto standard that's already been implemented by
atleast two people.
			G-E-N-I-U-S!

>IIRC, his objections back then were about linking archive into the kernel
>image.  s/disaster/ugly stuff that was nowhere near top-priority and got 
>fixed since then/ and I agree with that one.

Ahh good. Maybe you'll wise up to supporting tar as well. Then in
maybe a year and a half from now standard Linux will finally have
what I've had in production for 3 years...

Dave

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Sat, 22 Dec 2001, Dave Cinege wrote:

> On Saturday 22 December 2001 21:10, Alexander Viro wrote:
> 
> > > cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
> > > unacceptable, but should not be required.
> >
> > tar is ugly as hell and not going to be supported on the kernel side.
> 
> Excellent! You've settled on using using an archiver format nobody uses,
> instead of the defacto standard that's already been implemented by
> atleast two people.
> 			G-E-N-I-U-S!

OK, back into the killfile you go.

Hint: instead of wanking in public try to _think_ for a while.  Requirements
to archive format:
	* can be generated with minimum of code
	* can be parsed <ditto>
	* can be handled by standard utilities
That's it.  Both cpio(1) and tar(1) (or pax(1) that can do both) fit the last
one.  And tar loses on the first two - it's messier.  Not much, but enough
to make the choice obvious.  "Popular" is completely irrelevant here - as long
as it's handled by standard UNIX utilities it's OK.

Re: Booting a modular kernel through a multiple streams file

[Ryan Cumming]

On December 22, 2001 19:00, Dave Cinege wrote:
> Excellent! You've settled on using using an archiver format nobody uses,
Many file formats that require 'embedded' archives use cpio. RPM is one such 
file format. I think every single RPM distribution relying on cpio for the 
core of its package management system discounts your claim of 'nobody using 
it'. Just because you have not seen cpio directly used as an archive format 
doesn't mean it's unused.

> instead of the defacto standard that's already been implemented by
> atleast two people.
tar and cpio are both POSIX standards. I fail to see how one is more 
'defacto' than the other. 

As far as having tar already being implemented by at least two people, it 
really doesn't matter for a format as simple as cpio. Having a reference 
implementation for tar would be nice, as it is a hairy, complicated standard. 
The cpio format can be fully described in less than a page.

> G-E-N-I-U-S!
Grow up. Please.

-Ryan

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Saturday 22 December 2001 23:41, Ryan Cumming wrote:
> On December 22, 2001 19:00, Dave Cinege wrote:
> > Excellent! You've settled on using using an archiver format nobody uses,
>
> Many file formats that require 'embedded' archives use cpio. RPM is one
> such file format.

I know this, and have taken rpm's apart buy hand, and wrote 
a small util to take them apart using cpio. I consider it a good
example of Redhat's many screw up's. Deb's use a pair of .tgz's
and header. Much more sane.

> I think every single RPM distribution relying on cpio for

Hardly. It relies on rpm, which uses the cpio for the data archive.
It does not use the cpio command to make or unpackage archives. 

We're talking about a raw archive type, not something embedded in
a broader applicaiton. 

> the core of its package management system discounts your claim of 'nobody
> using it'. Just because you have not seen cpio directly used as an archive
> format doesn't mean it's unused.

How many times have you seen ANYTHING ditributed that way?

> As far as having tar already being implemented by at least two people, it
> really doesn't matter for a format as simple as cpio. Having a reference
> implementation for tar would be nice, as it is a hairy, complicated
> standard. 

You have no idea what you're talkig about, do you? Basic untar'ing is
quite simple. Now, GNU's tar as implented is pretty fucked up, but
that's not relevant to this.

> The cpio format can be fully described in less than a page.

Untar is implemented in less then 2. It doesn't get much smaller.

>From rd.dyn.c
---------------------
// This is called by flush_window to flush the gzip output
// buffer (containing part of a tar image).
UNTAR_WRITE
{
	int	i, scoop, err, file_written = 0;
	char	*to;
	kdev_t	fdev;
	struct	utimbuf ut;
	struct	untar_context *Untar;
#ifndef DEBUG_UNTAR
#ifndef CONFIG_ARCH_S390
	char rotator[4] = { '|' , '/' , '-' , '\\' };

	// pointer to our context structure is passed thru ioctl fop
	Untar = (struct untar_context*)fp->f_op->ioctl;

	printk("%c\b", rotator[Untar->rotate & 0x3]);
	Untar->rotate++;
#endif
#endif
	// Note: the gunzip routines call this only when
	//       a) the gunzip buffer is full -> count = WSIZE = 64 * TARBLOCKSIZE
	// or    b) to flush the last bytes out
	while(count) {	// What is our current state ?
		err = 0;	
		switch(Untar->state) {
			case READING_HEADER:
				// Get the tar header
				if(count < TARBLOCKSIZE) {
					count = 0;
					break;
				}
				to = (char*)&Untar->tarInfo;
				for(i = 0; i < TARBLOCKSIZE; i++) {
					*to++ = *buf++;
					--count;
				}
	
				// Tar usually pads the output byte to a multiple of it's block size usually
				// 10 KB, appending zeroes if necessary. Here we skip those zeroes:
				if(Untar->tarInfo.header.name[0] == '\0'  &&
				   Untar->tarInfo.header.mode[0] == '\0' &&
				   Untar->tarInfo.header.mode[0] == '\0') {
					if(count < TARBLOCKSIZE)
						count = 0;
					break;
				}
				// determine mode, size, uid, gid
				Untar->fmode = strtoul(Untar->tarInfo.header.mode,NULL,8);
				Untar->fsize = strtoul(Untar->tarInfo.header.size,NULL,8);
				Untar->uid = strtoul(Untar->tarInfo.header.uid,NULL,8);
				Untar->gid = strtoul(Untar->tarInfo.header.gid,NULL,8);
#ifdef DEBUG_UNTAR
				printk("\ntype: %c ",Untar->tarInfo.header.typeflag);   
				printk("name: %s ",Untar->tarInfo.header.name);
				printk("Untar->fsize: %ld ",Untar->fsize); 
				udelay(100000);
#endif
				// check the type
				switch(Untar->tarInfo.header.typeflag) {
					case AREGTYPE : 
					case REGTYPE  :
						Untar->out = sys_open(Untar->tarInfo.header.name,
					             O_CREAT|O_WRONLY|O_TRUNC, Untar->fmode);

						if(Untar->out == -1) {
							err = 1;
							break;
						}
						Untar->state = READING_DATA;
						break;
					case LNKTYPE  :
						err = sys_link(Untar->tarInfo.header.linkname,
						                  Untar->tarInfo.header.name);
						break;

					case SYMTYPE  :
						err = sys_symlink(Untar->tarInfo.header.linkname,
						                 Untar->tarInfo.header.name);
						break;

					case CHRTYPE  :	
					case BLKTYPE  :	
					case FIFOTYPE :	
						fdev = MKDEV(strtoul(Untar->tarInfo.header.devmajor,NULL,8),
						   strtoul(Untar->tarInfo.header.devminor,NULL,8));
						err = sys_mknod(Untar->tarInfo.header.name,Untar->fmode,fdev);
   	    				break;

					case DIRTYPE  :
						if((Untar->tarInfo.header.name[0] != '.' &&	// Skip if name is "" "./" "." ".." */
						    Untar->tarInfo.header.name[0] != '\0') &&
						   (Untar->tarInfo.header.name[1] != '/' &&
						    Untar->tarInfo.header.name[1] != '\0') &&
						   (Untar->tarInfo.header.name[1] != '\0')) {
							err = sys_mkdir(Untar->tarInfo.header.name,Untar->fmode);
						}
						break;
					default:
						printk(KERN_ERR "RAMDISK: corrupt tar archive\n");
						Untar->state = SKIPPING_REST;
						return(-1);
				}
				break;
			case READING_DATA:
				scoop = 0;
				while(count > 0 && Untar->fsize > 0) {
					scoop = Untar->fsize > TARBLOCKSIZE ? TARBLOCKSIZE : Untar->fsize;
					if(sys_write(Untar->out, buf, scoop) < scoop)
						err = 1;
					count -= scoop;
					buf   += scoop;
					Untar->fsize -= scoop;
				}
				if(Untar->fsize == 0) {	//skip to the next tar block
					sys_close(Untar->out);
					scoop = count % TARBLOCKSIZE;
					buf   += scoop;
					count -= scoop;
					Untar->state = READING_HEADER;
					file_written = 1;
				}
				break;

			case SKIPPING_REST:
				count = 0;
				break;
		}
		if(err)
			printk("!");
			//printk("\nError making %s", Untar->tarInfo.header.name);

		if(file_written) {
		
			err = sys_chown(Untar->tarInfo.header.name, Untar->uid, Untar->gid);
		
			if(Untar->tarInfo.header.typeflag != LNKTYPE &&
			   Untar->tarInfo.header.typeflag != SYMTYPE) {
				err += sys_chmod(Untar->tarInfo.header.name, Untar->fmode);
			}

			if(err)
				printk("\nError chown and/or chmod %s", Untar->tarInfo.header.name);
			ut.actime=sys_time(NULL);
			ut.modtime=strtoul(Untar->tarInfo.header.mtime,NULL,8);
			sys_utime(Untar->tarInfo.header.name,&ut);
			file_written = 0;
		}
	}
	return(0);
}

> > G-E-N-I-U-S!
>
> Grow up. Please.

Shut up Sit down!

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[Bernd Eckenfels]

In article <E16I1Yc-0003eD-00@schizo.psychosis.com> you wrote:
> I know this, and have taken rpm's apart buy hand, and wrote 
> a small util to take them apart using cpio. I consider it a good
> example of Redhat's many screw up's. Deb's use a pair of .tgz's
> and header. Much more sane.

Well, debian is using an "ar" archive with tar.gz contents.

> How many times have you seen ANYTHING ditributed that way?

AFAIK cpio is quite common on SysV Systems. The Problem with cpio and tar is,
that there are many incompatible versions. Even the Posix.1 format is quite
limited (255 char path limit). Some others fail short with symlinks or block
device nodes.

AFAIK SUS is supporting the use of pax.

http://www.opengroup.org/onlinepubs/7908799/xcu/pax.html 

Greetings
Bernd

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Followup to:  <E16I2rw-0006zi-00@sites.inka.de>
By author:    Bernd Eckenfels <usenet2001-12@lina.inka.de>
In newsgroup: linux.dev.kernel
> 
> > How many times have you seen ANYTHING ditributed that way?
> 
> AFAIK cpio is quite common on SysV Systems. The Problem with cpio and tar is,
> that there are many incompatible versions. Even the Posix.1 format is quite
> limited (255 char path limit).

This is the POSIX.1 *TAR* format.  The POSIX.1 *CPIO* format doesn't
have this limitation.

>  Some others fail short with symlinks or block
> device nodes.

Only utterly archaic versions, never used on Linux.

> AFAIK SUS is supporting the use of pax.
> 
> http://www.opengroup.org/onlinepubs/7908799/xcu/pax.html 

Pax is just a single utility which does cpio and tar.

Al is using a specific cpio format, which I believe is either the
"newc" or "odc" format... Al?

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[H. Peter Anvin]

I have looked through the various forms of tar and cpio, and I'm getting 
the feeling that both are ugly as hell for the purpose of initializing 
initramfs.  The block-based setup of tar (and of the new Austin group 
"pax" format -- really just a new rev of tar) is awful, but some of the 
tape-oriented aspects of cpio isn't much better.  What concerns me about 
cpio in particular:

a) Several different formats; <cpio.h> only documents one of them; I 
have only found info on that one so some of these things may not apply.

b) No obvious ways to handle hard links, that doesn't require you 
keeping a table of the inode numbers already seen (at least for which 
c_nlink > 1) and hard link to them on the decompression side.  Since we 
have an assymetric setup, it seems like its done at the wrong end.

c) The use of TRAILER!!! as an end-of-archive marker (first, it's a 
valid name, and second, there shouldn't be a need for an end-of-archive 
marker in this application as long as each individual file is 
self-terminating thus returning the dearchiver to its ground state.  If 
we stick with cpio, I would like these entries to have no effect.

d) c_rdev, c_uid, c_gid, c_dev, and c_ino are too small, at least in the 
<cpio.h> format.

e) The use of octal ASCII numbers is somewhat ugly.

f) No ctime, no atime.

It seems to me that this application doesn't really have a particular 
need for backward compatibility, nor for the I/O blocking stuff of 
tar/cpio.  I would certainly be willing to write a set of portable 
utilities to create an archive in a custom format, if that would be more 
desirable.  We'd still use gzip for compression, of course, and have the 
buffer composed as a combination of ".rfs" and ".rfs.gz" files, 
separated by zero-padding.

What I'm talking about would probably still look a lot like the cpio 
header, but probably would use bigendian binary (bigendian because it 
allows the use of the widely portable htons() and htonl() macros), have 
explicit support for hard links, and not require a trailer block.

	-hpa

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[Ryan Cumming]

On December 23, 2001 01:14, H. Peter Anvin wrote:
> It seems to me that this application doesn't really have a particular
> need for backward compatibility, nor for the I/O blocking stuff of
> tar/cpio.  I would certainly be willing to write a set of portable
> utilities to create an archive in a custom format, if that would be more
> desirable.  We'd still use gzip for compression, of course, and have the
> buffer composed as a combination of ".rfs" and ".rfs.gz" files,
> separated by zero-padding.
>
> What I'm talking about would probably still look a lot like the cpio
> header, but probably would use bigendian binary (bigendian because it
> allows the use of the widely portable htons() and htonl() macros), have
> explicit support for hard links, and not require a trailer block.

What about recycling a big-endian version of cramfs? IIRC, the format is 
really clean, it already does compression, and is terse enough to be efficent 
for booting purposes. However, it might be too terse as it:
1) Doesn't support hard links
2) Doesn't store atime/mtime/ctime

I don't know if that'd be too restrictive for an initrd replacement, but 
having only one compressed-filesystemish filesystem for the kernel would be 
nice.

-Ryan

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[H. Peter Anvin]

Ryan Cumming wrote:

> 
> What about recycling a big-endian version of cramfs? IIRC, the format is 
> really clean, it already does compression, and is terse enough to be efficent 
> for booting purposes. However, it might be too terse as it:
> 1) Doesn't support hard links
> 2) Doesn't store atime/mtime/ctime
> 
> I don't know if that'd be too restrictive for an initrd replacement, but 
> having only one compressed-filesystemish filesystem for the kernel would be 
> nice.
> 


You don't want to use the cramfs format.  It does a lot of things wrong 
for this application (such as block compression.)

	-hpa

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[Wichert Akkerman]

In article <3C25A06D.7030408@zytor.com>,
H. Peter Anvin <hpa@zytor.com> wrote:
>What concerns me about cpio in particular:

g) It is impossible to extend without changing the magic at the beginning
of the archive which will make all other cpio-handling tools not accept it.
tar does this better by having per-file types with a nice room for new
types, and older tar implementations will just skip over types they can
not handle.

This is probably not very relevant for this application, but it is something
you might want to remember if you are thinking of using cpio.

Wichert.


-- 
  _________________________________________________________________
 /       Nothing is fool-proof to a sufficiently talented fool     \
| wichert@wiggy.net                   http://www.liacs.nl/~wichert/ |
| 1024D/2FA3BC2D 576E 100B 518D 2F16 36B0  2805 3CB8 9250 2FA3 BC2D |

Re: Booting a modular kernel through a multiple streams file

[Kai Henningsen]

hpa@zytor.com (H. Peter Anvin)  wrote on 23.12.01 in <a0439g$7a5$1@cesium.transmeta.com>:

> Pax is just a single utility which does cpio and tar.

Well, pax is in POSIX 2001. Neither tar nor cpio are.

Of course, pax is specified to be able to work with both tar and cpio  
formats. And a new extended tar format which is what you get when you try  
to stuff a completely different format into tar.

MfG Kai

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[Alexander Viro]

On Sun, 23 Dec 2001, H. Peter Anvin wrote:

> a) Several different formats; <cpio.h> only documents one of them; I 
> have only found info on that one so some of these things may not apply.

--format=newc and yes, some of them do not apply

> b) No obvious ways to handle hard links, that doesn't require you 
> keeping a table of the inode numbers already seen (at least for which 
> c_nlink > 1) and hard link to them on the decompression side.  Since we 
> have an assymetric setup, it seems like its done at the wrong end.

See below.

> c) The use of TRAILER!!! as an end-of-archive marker (first, it's a 
> valid name, and second, there shouldn't be a need for an end-of-archive 
> marker in this application as long as each individual file is 
> self-terminating thus returning the dearchiver to its ground state.  If 
> we stick with cpio, I would like these entries to have no effect.

No problem.
 
> d) c_rdev, c_uid, c_gid, c_dev, and c_ino are too small, at least in the 
> <cpio.h> format.

32 bits each (and it's major/minor both for dev and rdev, so in effect it's
64 bits).

> e) The use of octal ASCII numbers is somewhat ugly.

hexadecimal, not octal.
 
> f) No ctime, no atime.

For populating root?

> It seems to me that this application doesn't really have a particular 
> need for backward compatibility, nor for the I/O blocking stuff of 
> tar/cpio.  I would certainly be willing to write a set of portable 
> utilities to create an archive in a custom format, if that would be more 
> desirable.  We'd still use gzip for compression, of course, and have the 
> buffer composed as a combination of ".rfs" and ".rfs.gz" files, 
> separated by zero-padding.
> 
> What I'm talking about would probably still look a lot like the cpio 
> header, but probably would use bigendian binary (bigendian because it 
> allows the use of the widely portable htons() and htonl() macros), have 
> explicit support for hard links, and not require a trailer block.

As for the hardlinks...  I'm not sure.  All it takes in uncpio.c is
~30 lines of sparse C.

Trailer block can (and will) be silently ignored by unpacking side.
"TRAILER!!! is a valid name" is true, but hardly serious.

I doubt that extra utilities are worth the effort.  Said that, layout
of the damn thing is nowhere near the top of the list of things that
worry me - as long as it's well-defined, can be handled from userland
and can be easily unpacked I'm OK with it.  What _does_ worry me, though
	* we either need mini-libc of our own or uclibc becomes mandatory
for build (glibc is out of question, dietlibc doesn't support quite a few
architectures we need).
	* getting sane Makefile for init/*
	* getting rid of /{LINUX,DOS} kludge in UMSDOS (code relies on
ROOT_DEV, which adds a lot of PITA)
	* devfs=only needing in effect a search over entire devfs tree (and
yes, that's what it was doing all along)
	* RAID autoconf code (seriously entangled with regular parts of
md.c).

Frankly, holy wars over layout of archive look rather pointless.  _That_
is trivial to change at any point until the moment when it hits the main
tree.  Getting late-boot code clean is the real problem.

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[H. Peter Anvin]

T. A. wrote:

>     What about considering one of the simpler filesystems or archive formats
> instead?  How much "Unix"-ism is required to be retained in the archive?
> (permissions, device files, etc?)
> 


They're MUCH MUCH MUCH MUCH MUCH worse.  Don't even think abou it.


>     As for the bigendianness... Is it really relevant since each kernel is
> tied to its own platform?  And if it is may it be better to use the native
> format of the 98% or so of the Linux machines out there which are
> littleendian (performance and ease of general access on the majority of host
> machines comes to mind).


This was discussed recently... doing a nonportable format is begging for 
problems.  The only reason I'm suggesting bigendian is that conversion 
to bigendian macros are more widely available in the form of the 
standard hton macros.

	-hpa

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[H. Peter Anvin]

Alexander Viro wrote:
 >

> I doubt that extra utilities are worth the effort.  Said that, layout
> of the damn thing is nowhere near the top of the list of things that
> worry me - as long as it's well-defined, can be handled from userland
> and can be easily unpacked I'm OK with it.

   [...]

> 
> Frankly, holy wars over layout of archive look rather pointless.  _That_
> is trivial to change at any point until the moment when it hits the main
> tree.  Getting late-boot code clean is the real problem.

 >

Agreed, of course.  I just happen to be approaching this from this end, 
with considering what kind of boot loader changes are desirable.

	-hpa


P.S. As far as the umsdos pivot thing is concerned, I always considered 
it to be a bug that it was only applicable to the root.  I would like to 
suggest making it a mount option instead.

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[H. Peter Anvin]

Alexander Viro wrote:

> 
>>b) No obvious ways to handle hard links, that doesn't require you 
>>keeping a table of the inode numbers already seen (at least for which 
>>c_nlink > 1) and hard link to them on the decompression side.  Since we 
>>have an assymetric setup, it seems like its done at the wrong end.
>>
> 
> See below.
> 


What I perhaps am calling for more than anything else is very precisely 
defined semantics of when hard links are generated, and how to control 
that.  Anyway, I'm about to leave for the holidays, and won't have 
email, but I'll think a bit more about this over that time.

	-hpa

Re: Booting a modular kernel through a multiple streams file

[Erik Mouw]

On Sun, Dec 23, 2001 at 02:22:00PM +0200, Kai Henningsen wrote:
> hpa@zytor.com (H. Peter Anvin)  wrote on 23.12.01 in <a0439g$7a5$1@cesium.transmeta.com>:
> 
> > Pax is just a single utility which does cpio and tar.
> 
> Well, pax is in POSIX 2001. Neither tar nor cpio are.

If POSIX abandons tar and cpio, it doesn't mean that all
implementations are magically erased from all hard drives all over the
world.


Erik

-- 
J.A.K. (Erik) Mouw, Information and Communication Theory Group, Faculty
of Information Technology and Systems, Delft University of Technology,
PO BOX 5031, 2600 GA Delft, The Netherlands  Phone: +31-15-2783635
Fax: +31-15-2781843  Email: J.A.K.Mouw@its.tudelft.nl
WWW: http://www-ict.its.tudelft.nl/~erik/