RE: Booting a modular kernel through a multiple streams file

RE: Booting a modular kernel through a multiple streams file

[Grover, Andrew]

> From: Alexander Viro [mailto:viro@math.psu.edu]
> On Mon, 17 Dec 2001, Grover, Andrew wrote:
> > I don't think multiple streams is a good idea, but did you 
> all see the patch
> > by Christian Koenig to let the bootloader load modules? 
> That seems to solve
> > the problem nicely.
> 
> That puts an awful lot of smarts into bootloader and creates 
> code duplication
> for no good reason.
> 
> We _already_ have code for loading modules.  And it's going 
> to stay, no
> matter what happens on boot.  So why the hell duplicate that 
> in $BIGNUM
> unrelated pieces of software (LILO, SYSLINUX, MILO, SILO, etc.)?  Just
> to create extra fun with debugging?

OK so (correct me if I'm wrong) there are two parts to loading modules -
getting them in memory, and then resolving references. My understanding of
Christian's work is that the bootloader (e.g. GRUB) gets them in memory, but
then it is up to the kernel linker to resolve refs. So yes, there would be
an additional piece of code (marked __init), but it would not have to be
duplicated for each bootloader -- all they have to do is get the modules in
memory and indicate via the multiboot struct where they are.

I don't think this will obsolete any existing boot methods, but it seems
like an additional genuinely useful capability for the Linux kernel to have.

Regards -- Andy

RE: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Mon, 17 Dec 2001, Grover, Andrew wrote:

> 
> OK so (correct me if I'm wrong) there are two parts to loading modules -
> getting them in memory, and then resolving references. My understanding of
> Christian's work is that the bootloader (e.g. GRUB) gets them in memory, but
> then it is up to the kernel linker to resolve refs. So yes, there would be
> an additional piece of code (marked __init), but it would not have to be
> duplicated for each bootloader -- all they have to do is get the modules in
> memory and indicate via the multiboot struct where they are.

*shrug*  Your "all they have to do" is quite heavy.
 
> I don't think this will obsolete any existing boot methods, but it seems
> like an additional genuinely useful capability for the Linux kernel to have.

I've had a very dubious pleasure of dealing with our boot sequence lately.
Adding more cruft to it (including in-kernel linker, for fsck sake) is
_not_ a good idea.

Folks, whatever had happened with "if it can be done in userland - don't
put it into the kernel"?

That goes for a _lot_ of code.  Mounting root.  Detecting the type of
initrd contents.  Loading ramdisk from floppies.  Asking to press
key (you really ought to look what is done for _that_).  Speaking
DHCP - we have a kernel DHCP client, of all things.  All that stuff
can (and should) be done from userland process.  And there's much
more of the same kind.

There is a word for that and that word is "crap".

Let loader leave an archive to be unpacked into rootfs?  Sure.  Let kernel
exec /init on rootfs and leave the rest to it?  Absolutely.  But let's
stop adding userland stuff into the kernel.  Loading modules _can_ be
done from userland - insmod does it just fine.  And that's where it should
be done.

Re: Booting a modular kernel through a multiple streams file

[Craig Christophel]

On Monday 17 December 2001 21:31, Alexander Viro wrote:
> There is a word for that and that word is "crap".
>
> Let loader leave an archive to be unpacked into rootfs?  Sure.  Let kernel
> exec /init on rootfs and leave the rest to it?  Absolutely.  But let's
> stop adding userland stuff into the kernel. 


I agree 100%.  there are several other popular systems that try to do 
everything in-kernel and it adds to the instability overall, besides it gives 
less freedom to customize with standard tools.



Craig.


 Loading modules _can_ be
> done from userland - insmod does it just fine.  And that's where it should
> be done.

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Monday 17 December 2001 21:31, Alexander Viro wrote:

> *shrug*  Your "all they have to do" is quite heavy.

Lilo (broken), syslinux (not very difficult), GRUB, (already implemented).
The list is not long. It's little differeent then a boot loader supporting
initrd loading.

> > I don't think this will obsolete any existing boot methods, but it seems
> > like an additional genuinely useful capability for the Linux kernel to
> > have.
>
> I've had a very dubious pleasure of dealing with our boot sequence lately.

Yes the guilty are well known... 

> Adding more cruft to it (including in-kernel linker, for fsck sake) is
> _not_ a good idea.

Which is why all of that shit that loads multiple initrd floppies, etc,
etc should be gutting and replaced with a solid initrd system that
can load tar.gz's to tmpfs. Jeez...I happen to have such a patch RIGHT 
HERE....for 3 years now. (minix prior to tmpfs existance...)

Loading initrd and boot time kernel modules, from multiple sources,
belongs in 'pre-kernel' (prom/bootloader) land. Most importantly this
boot loader is already implemented for IA32.

> That goes for a _lot_ of code.  Mounting root.  Detecting the type of
> initrd contents.  Loading ramdisk from floppies.  Asking to press
> key (you really ought to look what is done for _that_).  Speaking
> DHCP - we have a kernel DHCP client, of all things.  All that stuff
> can (and should) be done from userland process. 

Already done (properly) in GRUB. Userland (after the kernel boots) is
no place for this. It's too late to be done cleanly. I agree the kernel
is no place either.
 
> Let loader leave an archive to be unpacked into rootfs?  Sure.  Let kernel
> exec /init on rootfs and leave the rest to it?  Absolutely.  But let's
> stop adding userland stuff into the kernel.  Loading modules _can_ be
> done from userland - insmod does it just fine.  And that's where it should
> be done.

This is a very much a failed concept. In theory it sounds nice. My
experience in
	1) implementing a Linux OS from scratch
	2) administering multiple networks of mission critical servers
says this doesn't work well.  

Dave

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Tuesday 18 December 2001 15:26, Alexander Viro wrote:
> On Tue, 18 Dec 2001, Grover, Andrew wrote:
> > I'm not arguing that the new initrd won't be better than the old initrd
> > (because obviously you are right) I'm arguing that no matter how whizzy
> > initrd is, it's still an unnecessary step, and it's one that other OSs
> > (e.g. FreeBSD) omit in favor of the approach I'm advocating.
>
> Learn to read.  You don't _have_ to have initrd.  At all.  There's nothing
> to stop your loader from putting whatever cpio archive it likes - it
> doesn't involve anything other than slapping files you want together
> putting their owner/group/size/timestamps/mode/name before each of them.
> Anything that puts a bunch of modules in core will have to do equivalent
> job.

Deja Vu: *shrug*  Your "all they have to do" is quite heavy.
(boot loader must implement full cpio/tar[/gzip}

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Followup to:  <E16Hwks-0001wV-00@schizo.psychosis.com>
By author:    Dave Cinege <dcinege@psychosis.com>
In newsgroup: linux.dev.kernel
> 
> Deja Vu: *shrug*  Your "all they have to do" is quite heavy.
> (boot loader must implement full cpio/tar[/gzip}
> 

cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
unacceptable, but should not be required.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On 22 Dec 2001, H. Peter Anvin wrote:

> Followup to:  <E16Hwks-0001wV-00@schizo.psychosis.com>
> By author:    Dave Cinege <dcinege@psychosis.com>
> In newsgroup: linux.dev.kernel
> > 
> > Deja Vu: *shrug*  Your "all they have to do" is quite heavy.
> > (boot loader must implement full cpio/tar[/gzip}
> > 
> 
> cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
> unacceptable, but should not be required.

tar is ugly as hell and not going to be supported on the kernel side.

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Saturday 22 December 2001 21:10, Alexander Viro wrote:

> > cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
> > unacceptable, but should not be required.
>
> tar is ugly as hell and not going to be supported on the kernel side.

Excellent! You've settled on using using an archiver format nobody uses,
instead of the defacto standard that's already been implemented by
atleast two people.
			G-E-N-I-U-S!

>IIRC, his objections back then were about linking archive into the kernel
>image.  s/disaster/ugly stuff that was nowhere near top-priority and got 
>fixed since then/ and I agree with that one.

Ahh good. Maybe you'll wise up to supporting tar as well. Then in
maybe a year and a half from now standard Linux will finally have
what I've had in production for 3 years...

Dave

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[Alexander Viro]

On Sat, 22 Dec 2001, Dave Cinege wrote:

> On Saturday 22 December 2001 21:10, Alexander Viro wrote:
> 
> > > cpio is trivial.  tar is a bit more painful, but not too bad.  gzip is
> > > unacceptable, but should not be required.
> >
> > tar is ugly as hell and not going to be supported on the kernel side.
> 
> Excellent! You've settled on using using an archiver format nobody uses,
> instead of the defacto standard that's already been implemented by
> atleast two people.
> 			G-E-N-I-U-S!

OK, back into the killfile you go.

Hint: instead of wanking in public try to _think_ for a while.  Requirements
to archive format:
	* can be generated with minimum of code
	* can be parsed <ditto>
	* can be handled by standard utilities
That's it.  Both cpio(1) and tar(1) (or pax(1) that can do both) fit the last
one.  And tar loses on the first two - it's messier.  Not much, but enough
to make the choice obvious.  "Popular" is completely irrelevant here - as long
as it's handled by standard UNIX utilities it's OK.

Re: Booting a modular kernel through a multiple streams file

[Ryan Cumming]

On December 22, 2001 19:00, Dave Cinege wrote:
> Excellent! You've settled on using using an archiver format nobody uses,
Many file formats that require 'embedded' archives use cpio. RPM is one such 
file format. I think every single RPM distribution relying on cpio for the 
core of its package management system discounts your claim of 'nobody using 
it'. Just because you have not seen cpio directly used as an archive format 
doesn't mean it's unused.

> instead of the defacto standard that's already been implemented by
> atleast two people.
tar and cpio are both POSIX standards. I fail to see how one is more 
'defacto' than the other. 

As far as having tar already being implemented by at least two people, it 
really doesn't matter for a format as simple as cpio. Having a reference 
implementation for tar would be nice, as it is a hairy, complicated standard. 
The cpio format can be fully described in less than a page.

> G-E-N-I-U-S!
Grow up. Please.

-Ryan

Re: Booting a modular kernel through a multiple streams file

[Dave Cinege]

On Saturday 22 December 2001 23:41, Ryan Cumming wrote:
> On December 22, 2001 19:00, Dave Cinege wrote:
> > Excellent! You've settled on using using an archiver format nobody uses,
>
> Many file formats that require 'embedded' archives use cpio. RPM is one
> such file format.

I know this, and have taken rpm's apart buy hand, and wrote 
a small util to take them apart using cpio. I consider it a good
example of Redhat's many screw up's. Deb's use a pair of .tgz's
and header. Much more sane.

> I think every single RPM distribution relying on cpio for

Hardly. It relies on rpm, which uses the cpio for the data archive.
It does not use the cpio command to make or unpackage archives. 

We're talking about a raw archive type, not something embedded in
a broader applicaiton. 

> the core of its package management system discounts your claim of 'nobody
> using it'. Just because you have not seen cpio directly used as an archive
> format doesn't mean it's unused.

How many times have you seen ANYTHING ditributed that way?

> As far as having tar already being implemented by at least two people, it
> really doesn't matter for a format as simple as cpio. Having a reference
> implementation for tar would be nice, as it is a hairy, complicated
> standard. 

You have no idea what you're talkig about, do you? Basic untar'ing is
quite simple. Now, GNU's tar as implented is pretty fucked up, but
that's not relevant to this.

> The cpio format can be fully described in less than a page.

Untar is implemented in less then 2. It doesn't get much smaller.

>From rd.dyn.c
---------------------
// This is called by flush_window to flush the gzip output
// buffer (containing part of a tar image).
UNTAR_WRITE
{
	int	i, scoop, err, file_written = 0;
	char	*to;
	kdev_t	fdev;
	struct	utimbuf ut;
	struct	untar_context *Untar;
#ifndef DEBUG_UNTAR
#ifndef CONFIG_ARCH_S390
	char rotator[4] = { '|' , '/' , '-' , '\\' };

	// pointer to our context structure is passed thru ioctl fop
	Untar = (struct untar_context*)fp->f_op->ioctl;

	printk("%c\b", rotator[Untar->rotate & 0x3]);
	Untar->rotate++;
#endif
#endif
	// Note: the gunzip routines call this only when
	//       a) the gunzip buffer is full -> count = WSIZE = 64 * TARBLOCKSIZE
	// or    b) to flush the last bytes out
	while(count) {	// What is our current state ?
		err = 0;	
		switch(Untar->state) {
			case READING_HEADER:
				// Get the tar header
				if(count < TARBLOCKSIZE) {
					count = 0;
					break;
				}
				to = (char*)&Untar->tarInfo;
				for(i = 0; i < TARBLOCKSIZE; i++) {
					*to++ = *buf++;
					--count;
				}
	
				// Tar usually pads the output byte to a multiple of it's block size usually
				// 10 KB, appending zeroes if necessary. Here we skip those zeroes:
				if(Untar->tarInfo.header.name[0] == '\0'  &&
				   Untar->tarInfo.header.mode[0] == '\0' &&
				   Untar->tarInfo.header.mode[0] == '\0') {
					if(count < TARBLOCKSIZE)
						count = 0;
					break;
				}
				// determine mode, size, uid, gid
				Untar->fmode = strtoul(Untar->tarInfo.header.mode,NULL,8);
				Untar->fsize = strtoul(Untar->tarInfo.header.size,NULL,8);
				Untar->uid = strtoul(Untar->tarInfo.header.uid,NULL,8);
				Untar->gid = strtoul(Untar->tarInfo.header.gid,NULL,8);
#ifdef DEBUG_UNTAR
				printk("\ntype: %c ",Untar->tarInfo.header.typeflag);   
				printk("name: %s ",Untar->tarInfo.header.name);
				printk("Untar->fsize: %ld ",Untar->fsize); 
				udelay(100000);
#endif
				// check the type
				switch(Untar->tarInfo.header.typeflag) {
					case AREGTYPE : 
					case REGTYPE  :
						Untar->out = sys_open(Untar->tarInfo.header.name,
					             O_CREAT|O_WRONLY|O_TRUNC, Untar->fmode);

						if(Untar->out == -1) {
							err = 1;
							break;
						}
						Untar->state = READING_DATA;
						break;
					case LNKTYPE  :
						err = sys_link(Untar->tarInfo.header.linkname,
						                  Untar->tarInfo.header.name);
						break;

					case SYMTYPE  :
						err = sys_symlink(Untar->tarInfo.header.linkname,
						                 Untar->tarInfo.header.name);
						break;

					case CHRTYPE  :	
					case BLKTYPE  :	
					case FIFOTYPE :	
						fdev = MKDEV(strtoul(Untar->tarInfo.header.devmajor,NULL,8),
						   strtoul(Untar->tarInfo.header.devminor,NULL,8));
						err = sys_mknod(Untar->tarInfo.header.name,Untar->fmode,fdev);
   	    				break;

					case DIRTYPE  :
						if((Untar->tarInfo.header.name[0] != '.' &&	// Skip if name is "" "./" "." ".." */
						    Untar->tarInfo.header.name[0] != '\0') &&
						   (Untar->tarInfo.header.name[1] != '/' &&
						    Untar->tarInfo.header.name[1] != '\0') &&
						   (Untar->tarInfo.header.name[1] != '\0')) {
							err = sys_mkdir(Untar->tarInfo.header.name,Untar->fmode);
						}
						break;
					default:
						printk(KERN_ERR "RAMDISK: corrupt tar archive\n");
						Untar->state = SKIPPING_REST;
						return(-1);
				}
				break;
			case READING_DATA:
				scoop = 0;
				while(count > 0 && Untar->fsize > 0) {
					scoop = Untar->fsize > TARBLOCKSIZE ? TARBLOCKSIZE : Untar->fsize;
					if(sys_write(Untar->out, buf, scoop) < scoop)
						err = 1;
					count -= scoop;
					buf   += scoop;
					Untar->fsize -= scoop;
				}
				if(Untar->fsize == 0) {	//skip to the next tar block
					sys_close(Untar->out);
					scoop = count % TARBLOCKSIZE;
					buf   += scoop;
					count -= scoop;
					Untar->state = READING_HEADER;
					file_written = 1;
				}
				break;

			case SKIPPING_REST:
				count = 0;
				break;
		}
		if(err)
			printk("!");
			//printk("\nError making %s", Untar->tarInfo.header.name);

		if(file_written) {
		
			err = sys_chown(Untar->tarInfo.header.name, Untar->uid, Untar->gid);
		
			if(Untar->tarInfo.header.typeflag != LNKTYPE &&
			   Untar->tarInfo.header.typeflag != SYMTYPE) {
				err += sys_chmod(Untar->tarInfo.header.name, Untar->fmode);
			}

			if(err)
				printk("\nError chown and/or chmod %s", Untar->tarInfo.header.name);
			ut.actime=sys_time(NULL);
			ut.modtime=strtoul(Untar->tarInfo.header.mtime,NULL,8);
			sys_utime(Untar->tarInfo.header.name,&ut);
			file_written = 0;
		}
	}
	return(0);
}

> > G-E-N-I-U-S!
>
> Grow up. Please.

Shut up Sit down!

-- 
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/

Re: Booting a modular kernel through a multiple streams file

[Bernd Eckenfels]

In article <E16I1Yc-0003eD-00@schizo.psychosis.com> you wrote:
> I know this, and have taken rpm's apart buy hand, and wrote 
> a small util to take them apart using cpio. I consider it a good
> example of Redhat's many screw up's. Deb's use a pair of .tgz's
> and header. Much more sane.

Well, debian is using an "ar" archive with tar.gz contents.

> How many times have you seen ANYTHING ditributed that way?

AFAIK cpio is quite common on SysV Systems. The Problem with cpio and tar is,
that there are many incompatible versions. Even the Posix.1 format is quite
limited (255 char path limit). Some others fail short with symlinks or block
device nodes.

AFAIK SUS is supporting the use of pax.

http://www.opengroup.org/onlinepubs/7908799/xcu/pax.html 

Greetings
Bernd

Re: Booting a modular kernel through a multiple streams file

[H. Peter Anvin]

Followup to:  <E16I2rw-0006zi-00@sites.inka.de>
By author:    Bernd Eckenfels <usenet2001-12@lina.inka.de>
In newsgroup: linux.dev.kernel
> 
> > How many times have you seen ANYTHING ditributed that way?
> 
> AFAIK cpio is quite common on SysV Systems. The Problem with cpio and tar is,
> that there are many incompatible versions. Even the Posix.1 format is quite
> limited (255 char path limit).

This is the POSIX.1 *TAR* format.  The POSIX.1 *CPIO* format doesn't
have this limitation.

>  Some others fail short with symlinks or block
> device nodes.

Only utterly archaic versions, never used on Linux.

> AFAIK SUS is supporting the use of pax.
> 
> http://www.opengroup.org/onlinepubs/7908799/xcu/pax.html 

Pax is just a single utility which does cpio and tar.

Al is using a specific cpio format, which I believe is either the
"newc" or "odc" format... Al?

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Booting a modular kernel through a multiple streams file

[Kai Henningsen]

hpa@zytor.com (H. Peter Anvin)  wrote on 23.12.01 in <a0439g$7a5$1@cesium.transmeta.com>:

> Pax is just a single utility which does cpio and tar.

Well, pax is in POSIX 2001. Neither tar nor cpio are.

Of course, pax is specified to be able to work with both tar and cpio  
formats. And a new extended tar format which is what you get when you try  
to stuff a completely different format into tar.

MfG Kai

Re: Booting a modular kernel through a multiple streams file

[Erik Mouw]

On Sun, Dec 23, 2001 at 02:22:00PM +0200, Kai Henningsen wrote:
> hpa@zytor.com (H. Peter Anvin)  wrote on 23.12.01 in <a0439g$7a5$1@cesium.transmeta.com>:
> 
> > Pax is just a single utility which does cpio and tar.
> 
> Well, pax is in POSIX 2001. Neither tar nor cpio are.

If POSIX abandons tar and cpio, it doesn't mean that all
implementations are magically erased from all hard drives all over the
world.


Erik

-- 
J.A.K. (Erik) Mouw, Information and Communication Theory Group, Faculty
of Information Technology and Systems, Delft University of Technology,
PO BOX 5031, 2600 GA Delft, The Netherlands  Phone: +31-15-2783635
Fax: +31-15-2781843  Email: J.A.K.Mouw@its.tudelft.nl
WWW: http://www-ict.its.tudelft.nl/~erik/

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[H. Peter Anvin]

I have looked through the various forms of tar and cpio, and I'm getting 
the feeling that both are ugly as hell for the purpose of initializing 
initramfs.  The block-based setup of tar (and of the new Austin group 
"pax" format -- really just a new rev of tar) is awful, but some of the 
tape-oriented aspects of cpio isn't much better.  What concerns me about 
cpio in particular:

a) Several different formats; <cpio.h> only documents one of them; I 
have only found info on that one so some of these things may not apply.

b) No obvious ways to handle hard links, that doesn't require you 
keeping a table of the inode numbers already seen (at least for which 
c_nlink > 1) and hard link to them on the decompression side.  Since we 
have an assymetric setup, it seems like its done at the wrong end.

c) The use of TRAILER!!! as an end-of-archive marker (first, it's a 
valid name, and second, there shouldn't be a need for an end-of-archive 
marker in this application as long as each individual file is 
self-terminating thus returning the dearchiver to its ground state.  If 
we stick with cpio, I would like these entries to have no effect.

d) c_rdev, c_uid, c_gid, c_dev, and c_ino are too small, at least in the 
<cpio.h> format.

e) The use of octal ASCII numbers is somewhat ugly.

f) No ctime, no atime.

It seems to me that this application doesn't really have a particular 
need for backward compatibility, nor for the I/O blocking stuff of 
tar/cpio.  I would certainly be willing to write a set of portable 
utilities to create an archive in a custom format, if that would be more 
desirable.  We'd still use gzip for compression, of course, and have the 
buffer composed as a combination of ".rfs" and ".rfs.gz" files, 
separated by zero-padding.

What I'm talking about would probably still look a lot like the cpio 
header, but probably would use bigendian binary (bigendian because it 
allows the use of the widely portable htons() and htonl() macros), have 
explicit support for hard links, and not require a trailer block.

	-hpa

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[T. A.]

    What about considering one of the simpler filesystems or archive formats
instead?  How much "Unix"-ism is required to be retained in the archive?
(permissions, device files, etc?)

    As for the bigendianness... Is it really relevant since each kernel is
tied to its own platform?  And if it is may it be better to use the native
format of the 98% or so of the Linux machines out there which are
littleendian (performance and ease of general access on the majority of host
machines comes to mind).

----- Original Message -----
From: "H. Peter Anvin" <hpa@zytor.com>
To: "Alexander Viro" <viro@math.psu.edu>
Cc: <linux-kernel@vger.kernel.org>
Sent: Sunday, December 23, 2001 4:14 AM
Subject: Re: tar vs cpio (was: Booting a modular kernel through a multiple
streams file)


> I have looked through the various forms of tar and cpio, and I'm getting
> the feeling that both are ugly as hell for the purpose of initializing
> initramfs.  The block-based setup of tar (and of the new Austin group
> "pax" format -- really just a new rev of tar) is awful, but some of the
> tape-oriented aspects of cpio isn't much better.  What concerns me about
> cpio in particular:
> It seems to me that this application doesn't really have a particular
> need for backward compatibility, nor for the I/O blocking stuff of
> tar/cpio.  I would certainly be willing to write a set of portable
> utilities to create an archive in a custom format, if that would be more
> desirable.  We'd still use gzip for compression, of course, and have the
> buffer composed as a combination of ".rfs" and ".rfs.gz" files,
> separated by zero-padding.
>
> What I'm talking about would probably still look a lot like the cpio
> header, but probably would use bigendian binary (bigendian because it
> allows the use of the widely portable htons() and htonl() macros), have
> explicit support for hard links, and not require a trailer block.
>
> -hpa
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[H. Peter Anvin]

T. A. wrote:

>     What about considering one of the simpler filesystems or archive formats
> instead?  How much "Unix"-ism is required to be retained in the archive?
> (permissions, device files, etc?)
> 


They're MUCH MUCH MUCH MUCH MUCH worse.  Don't even think abou it.


>     As for the bigendianness... Is it really relevant since each kernel is
> tied to its own platform?  And if it is may it be better to use the native
> format of the 98% or so of the Linux machines out there which are
> littleendian (performance and ease of general access on the majority of host
> machines comes to mind).


This was discussed recently... doing a nonportable format is begging for 
problems.  The only reason I'm suggesting bigendian is that conversion 
to bigendian macros are more widely available in the form of the 
standard hton macros.

	-hpa

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[Ryan Cumming]

On December 23, 2001 01:14, H. Peter Anvin wrote:
> It seems to me that this application doesn't really have a particular
> need for backward compatibility, nor for the I/O blocking stuff of
> tar/cpio.  I would certainly be willing to write a set of portable
> utilities to create an archive in a custom format, if that would be more
> desirable.  We'd still use gzip for compression, of course, and have the
> buffer composed as a combination of ".rfs" and ".rfs.gz" files,
> separated by zero-padding.
>
> What I'm talking about would probably still look a lot like the cpio
> header, but probably would use bigendian binary (bigendian because it
> allows the use of the widely portable htons() and htonl() macros), have
> explicit support for hard links, and not require a trailer block.

What about recycling a big-endian version of cramfs? IIRC, the format is 
really clean, it already does compression, and is terse enough to be efficent 
for booting purposes. However, it might be too terse as it:
1) Doesn't support hard links
2) Doesn't store atime/mtime/ctime

I don't know if that'd be too restrictive for an initrd replacement, but 
having only one compressed-filesystemish filesystem for the kernel would be 
nice.

-Ryan

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[H. Peter Anvin]

Ryan Cumming wrote:

> 
> What about recycling a big-endian version of cramfs? IIRC, the format is 
> really clean, it already does compression, and is terse enough to be efficent 
> for booting purposes. However, it might be too terse as it:
> 1) Doesn't support hard links
> 2) Doesn't store atime/mtime/ctime
> 
> I don't know if that'd be too restrictive for an initrd replacement, but 
> having only one compressed-filesystemish filesystem for the kernel would be 
> nice.
> 


You don't want to use the cramfs format.  It does a lot of things wrong 
for this application (such as block compression.)

	-hpa

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[Wichert Akkerman]

In article <3C25A06D.7030408@zytor.com>,
H. Peter Anvin <hpa@zytor.com> wrote:
>What concerns me about cpio in particular:

g) It is impossible to extend without changing the magic at the beginning
of the archive which will make all other cpio-handling tools not accept it.
tar does this better by having per-file types with a nice room for new
types, and older tar implementations will just skip over types they can
not handle.

This is probably not very relevant for this application, but it is something
you might want to remember if you are thinking of using cpio.

Wichert.


-- 
  _________________________________________________________________
 /       Nothing is fool-proof to a sufficiently talented fool     \
| wichert@wiggy.net                   http://www.liacs.nl/~wichert/ |
| 1024D/2FA3BC2D 576E 100B 518D 2F16 36B0  2805 3CB8 9250 2FA3 BC2D |

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[Alexander Viro]

On Sun, 23 Dec 2001, H. Peter Anvin wrote:

> a) Several different formats; <cpio.h> only documents one of them; I 
> have only found info on that one so some of these things may not apply.

--format=newc and yes, some of them do not apply

> b) No obvious ways to handle hard links, that doesn't require you 
> keeping a table of the inode numbers already seen (at least for which 
> c_nlink > 1) and hard link to them on the decompression side.  Since we 
> have an assymetric setup, it seems like its done at the wrong end.

See below.

> c) The use of TRAILER!!! as an end-of-archive marker (first, it's a 
> valid name, and second, there shouldn't be a need for an end-of-archive 
> marker in this application as long as each individual file is 
> self-terminating thus returning the dearchiver to its ground state.  If 
> we stick with cpio, I would like these entries to have no effect.

No problem.
 
> d) c_rdev, c_uid, c_gid, c_dev, and c_ino are too small, at least in the 
> <cpio.h> format.

32 bits each (and it's major/minor both for dev and rdev, so in effect it's
64 bits).

> e) The use of octal ASCII numbers is somewhat ugly.

hexadecimal, not octal.
 
> f) No ctime, no atime.

For populating root?

> It seems to me that this application doesn't really have a particular 
> need for backward compatibility, nor for the I/O blocking stuff of 
> tar/cpio.  I would certainly be willing to write a set of portable 
> utilities to create an archive in a custom format, if that would be more 
> desirable.  We'd still use gzip for compression, of course, and have the 
> buffer composed as a combination of ".rfs" and ".rfs.gz" files, 
> separated by zero-padding.
> 
> What I'm talking about would probably still look a lot like the cpio 
> header, but probably would use bigendian binary (bigendian because it 
> allows the use of the widely portable htons() and htonl() macros), have 
> explicit support for hard links, and not require a trailer block.

As for the hardlinks...  I'm not sure.  All it takes in uncpio.c is
~30 lines of sparse C.

Trailer block can (and will) be silently ignored by unpacking side.
"TRAILER!!! is a valid name" is true, but hardly serious.

I doubt that extra utilities are worth the effort.  Said that, layout
of the damn thing is nowhere near the top of the list of things that
worry me - as long as it's well-defined, can be handled from userland
and can be easily unpacked I'm OK with it.  What _does_ worry me, though
	* we either need mini-libc of our own or uclibc becomes mandatory
for build (glibc is out of question, dietlibc doesn't support quite a few
architectures we need).
	* getting sane Makefile for init/*
	* getting rid of /{LINUX,DOS} kludge in UMSDOS (code relies on
ROOT_DEV, which adds a lot of PITA)
	* devfs=only needing in effect a search over entire devfs tree (and
yes, that's what it was doing all along)
	* RAID autoconf code (seriously entangled with regular parts of
md.c).

Frankly, holy wars over layout of archive look rather pointless.  _That_
is trivial to change at any point until the moment when it hits the main
tree.  Getting late-boot code clean is the real problem.

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[H. Peter Anvin]

Alexander Viro wrote:
 >

> I doubt that extra utilities are worth the effort.  Said that, layout
> of the damn thing is nowhere near the top of the list of things that
> worry me - as long as it's well-defined, can be handled from userland
> and can be easily unpacked I'm OK with it.

   [...]

> 
> Frankly, holy wars over layout of archive look rather pointless.  _That_
> is trivial to change at any point until the moment when it hits the main
> tree.  Getting late-boot code clean is the real problem.

 >

Agreed, of course.  I just happen to be approaching this from this end, 
with considering what kind of boot loader changes are desirable.

	-hpa


P.S. As far as the umsdos pivot thing is concerned, I always considered 
it to be a bug that it was only applicable to the root.  I would like to 
suggest making it a mount option instead.

Re: tar vs cpio (was: Booting a modular kernel through a multiple streams file)

[H. Peter Anvin]

Alexander Viro wrote:

> 
>>b) No obvious ways to handle hard links, that doesn't require you 
>>keeping a table of the inode numbers already seen (at least for which 
>>c_nlink > 1) and hard link to them on the decompression side.  Since we 
>>have an assymetric setup, it seems like its done at the wrong end.
>>
> 
> See below.
> 


What I perhaps am calling for more than anything else is very precisely 
defined semantics of when hard links are generated, and how to control 
that.  Anyway, I'm about to leave for the holidays, and won't have 
email, but I'll think a bit more about this over that time.

	-hpa

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[Christian Koenig]

Hi,

On Monday 17 December 2001 23:10, Grover, Andrew wrote:
> > From: Alexander Viro [mailto:viro@math.psu.edu]
> >
> > On Mon, 17 Dec 2001, Grover, Andrew wrote:
> > > I don't think multiple streams is a good idea, but did you
> >
> > all see the patch
> >
> > > by Christian Koenig to let the bootloader load modules?
> >
> > That seems to solve
> >
> > > the problem nicely.
> >
> > That puts an awful lot of smarts into bootloader and creates
> > code duplication
> > for no good reason.

I agree, but the problem I have with Initrd is that you could only have 
1 single initrd-image on your hard disk / loaded by the boot-loader.

Imaging my situation, I have some removeable hard-disks, changing it between 
a large amount of Systems all with individual Hardware-configuration.
(Some SCSI, some ide, and a cluster booting with pxegrub/Floppy-Disks).

With the last relleas of my patch I'm now capable to have a grub menu.lst 
looking like:

tile Kernel 2.4.14 Adaptec xyz/ne2000
kernel /boot/vmlinuz-2.4.14 root=/dev/sdxyz ...

# Loading SCSI-Drivers
modulesfromfile /etc/modules-adaptec /lib/modules/2.4.14/modules.dep 

# Loading Network-Drivers
modulesfromfile /etc/modules-ne2000 /lib/modules/2.4.14/modules.dep

tile Kernel 2.4.14 Tecram xyz / Inter EExpresPro 

# Loading SCSI-Drivers
modulesfromfile /etc/modules-tecram /lib/modules/2.4.14/modules.dep 

# Loading Network-Drivers
modulesfromfile /etc/modules-eepro /lib/modules/2.4.14/modules.dep 
......

Could you Imaging what work it would be to create an individual initrd-Image 
/ Kernel Image for every system I have. Beside this I sometimes have the 
Problem that I come to a system never seen bevore and have to boot with my 
Floppy Disk's. (I'm now looking forward that grub someday gets capable to use 
cd-roms directly).

I know this it isn't the best solution to add a module-loader to the Kernel, 
but did you have a better idea ?

MfG, Christian Koenig.

Re: Booting a modular kernel through a multiple streams file / Making Linux multiboot capable and grub loading kernel modules at boot time.

[H. Peter Anvin]

Christian Koenig wrote:
> 
> I agree, but the problem I have with Initrd is that you could only have 
> 1 single initrd-image on your hard disk / loaded by the boot-loader.
> 

Al and I have already discussed an improved approach.  What we want is 
rather than initrd an initramfs.  I have specifically discussed the 
initramfs protocol with Al so it can be combined from multiple sources.

	-hpa