[RFC] klibc requirements

[RFC] klibc requirements

[Greg KH]

Hi all,

First off, I do not want to fork off yet another tiny libc
implementation, unless it is determined that we really need to do it.  I
posted the klibc implementation because I have found that in the past,
people can talk all they want, but the only way to actually get people
all riled up and start paying attention is to post code :)

Now that people are riled up, and want to talk about it, let's try to
describe the problem and see if any of the existing libc implementations
help solve them.  Here's what I see as a list of requirements for a
klibc like library that can be used by initramfs programs (please,
everyone else jump in here with their requirements too):
	- portable, runs on all platforms that the kernel currently
	  works on, but doesn't have to run on any non-Linux based OS.
	- tiny.  If we link statically it should be _small_.  Both
	  dietLibc and uClibc are good examples of the size goal.  We do
	  not need to support all of POSIX here, only what we really
	  need.
	- If we end up having a lot of different programs in initramfs,
	  a dynamic version of the library should be available.  This
	  shared library should be _small_ and only contain the symbols
	  that are needed by the programs to run.  This should be able
	  to be determined at build time.
	- It has to "not suck" :)  This is a lovely relative feeling
	  about the quality of the code base, ease at building the
	  library, ease at understanding the code, and responsiveness of
	  the developers of the library.

I don't think either dietHotplug or uClibc or glibc or any other
existing libc implementation meets these goals right now, right?

I had asked the dietLibc authors about the ability of tweaking their
library into something that resembles the above, but didn't get a
response.  Hence my post.  I would love to work with the authors of an
existing libc to build such a library, as I have other things I would
rather work on than a libc :)

Comments from the various libc authors?  Comments from other kernel
developers about requirements and goals they would like to see from such
a libc?

thanks,

greg k-h

Re: [RFC] klibc requirements

[Erik Andersen]

On Tue Jan 08, 2002 at 11:24:50AM -0800, Greg KH wrote:
> 	- portable, runs on all platforms that the kernel currently
> 	  works on, but doesn't have to run on any non-Linux based OS.

uClibc currently only runs on arm, i386, m68k, mips, powerpc, sh,
and v850, with full shared lib support on arm, i386, and powerpc.
Porting to a new arch typically involves writing just a few asm
files.  I don't bother with anything non-Linux...

> 	- tiny.  If we link statically it should be _small_.  Both
> 	  dietLibc and uClibc are good examples of the size goal.  We do
> 	  not need to support all of POSIX here, only what we really
> 	  need.

uClibc does tiny static stuff just fine.  Though these days,
uClibc passes POSIX conformance tests (with some exceptions for
stupid things which have been omitted).

> 	- If we end up having a lot of different programs in initramfs,
> 	  a dynamic version of the library should be available.  This
> 	  shared library should be _small_ and only contain the symbols
> 	  that are needed by the programs to run.  This should be able
> 	  to be determined at build time.

uClibc currently has shared lib support only on arm, i386, and powerpc.
There is a library reducer script (to include only needed
symbols) in uClibc/extra/libstrip/libstrip which does a fine job.

I personally think busybox + uClibc are ideal for building
initramfs stuff, since you can build everything you need into 
a single multi-call binary (eliminating the need for shared libs
in most cases).

> 	- It has to "not suck" :)  This is a lovely relative feeling
> 	  about the quality of the code base, ease at building the
> 	  library, ease at understanding the code, and responsiveness of
> 	  the developers of the library.

Well, I do my best.  :)

As for ease of building the library, most people can just copy
the Config file into place and compile.  I build a fake
gcc-wrapper toolchain, so using the library is as simple as 
'make install' then running 'CC=/usr/i386-linux-uclibc/bin/gcc make'

I think one big advantage uClibc has in the "not suck"
department, is that it uses the header files from glibc 2.2.4
(with minor changes), meaning that for most apps, if it compiles
with glibc it will compile with uClibc.

> response.  Hence my post.  I would love to work with the authors of an
> existing libc to build such a library, as I have other things I would
> rather work on than a libc :)

Ok.  here I am.   Work with me.  :) 

> Comments from the various libc authors?  Comments from other kernel
> developers about requirements and goals they would like to see from such
> a libc?

If folks have requirements and goals, I'm very interested in
hearing them...

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] klibc requirements

[Felix von Leitner]

Thus spake Greg KH (greg@kroah.com):
> First off, I do not want to fork off yet another tiny libc
> implementation,

It's good to hear that.

> Now that people are riled up, and want to talk about it, let's try to
> describe the problem and see if any of the existing libc implementations
> help solve them.  Here's what I see as a list of requirements for a
> klibc like library that can be used by initramfs programs (please,
> everyone else jump in here with their requirements too):

My understanding of what "initramfs programs" actually means is vague at
best.  Are these just programs that are intended to work in an initial
ram disk?  Or is it a special collection that is included in the kernel
distribution?

I don't see why we would need to include those programs with the kernel.
The kernel tries hard to provide backwards compatibility to old versions
of syscalls that have changed over the years.  That's why we _have_ a
standard.  Also, we don't ship glibc with the kernel sources, and we
didn't when our libc was Linux specific.

If we follow that argumentation and are talking here about programs that
aren't included in the kernel, why demand a specific libc for them at
all?  Of course glibc is out of the question, but both the kernel API
_and_ the libc API are standardized.  It does not make sense to write
code that demands a specific libc if it can be avoided.  If you need
any special syscall supported that is not yet part of the diet libc or
uClibc, Eric and I will probably be glad to add support for it.

> 	- portable, runs on all platforms that the kernel currently
> 	  works on, but doesn't have to run on any non-Linux based OS.
> 	- tiny.  If we link statically it should be _small_.  Both
> 	  dietLibc and uClibc are good examples of the size goal.  We do
> 	  not need to support all of POSIX here, only what we really
> 	  need.

When you link statically, it does not matter whether the libc also
supports the rest of POSIX.  The size of libc.a does not matter.  It
only matters which parts are referenced.  Since we are talking about new
software specifically written for Linux and with the goal to be small,
there are no legacy requirements to cater to.  We can write code without
printf and stdio, for example.  Also, we probably don't need regular
expressions or DNS.  Those are the big space hogs when linking
statically against a libc.  In the diet libc, all of the above are very
small, but avoiding them in the first place is better then optimizing
them for small size.

And if we don't use all that bloaty code, it does not matter if we use
diet libc, uClibc or tomorrow's next great small libc.

> 	- If we end up having a lot of different programs in initramfs,
> 	  a dynamic version of the library should be available.  This
> 	  shared library should be _small_ and only contain the symbols
> 	  that are needed by the programs to run.  This should be able
> 	  to be determined at build time.

This may look like a good idea, but dynamic linking should be avoided.
Trust me on this.  While it is possible to squeeze the dynamic loader
down to below 10k, 10k really is a lot of code.  And empty program with
the diet libc is way below 1k on x86.  So to reap the benefit of dynamic
linking, you would need a lot of programs.  Also please note that -fPIC
makes code larger.  And we need to keep symbols around, which makes up a
substantial part of the shared diet libc.

How many programs are we talking about here?  And what should they do?

You can make a libc.so that only contains the superset of the symbols
used by your initramdisk programs.  But then you have to remake the
libc.so if you add another program, so that's not very flexible.

What I want to say here is: don't make rash decisions.  Profile, don't
speculate.  Also, don't underestimate the amount of time dynamic linking
takes.  It may seem insignificant on today's monster CPUs, but it isn't.
I optimized my boot sequence to below 3 seconds from the start of init
to the shell prompt on tty1 (the other ttys are spawned immediately).
All the dynamic linking and shell script overhead adds up quickly.

Please read http://www.fefe.de/dietlibc/talk.pdf for some more info
(slightly less than 100k).

> 	- It has to "not suck" :)  This is a lovely relative feeling
> 	  about the quality of the code base, ease at building the
> 	  library, ease at understanding the code, and responsiveness of
> 	  the developers of the library.

While I am of course sure to be the libc developer that sucks least ;),
I again would advise not to rush to any hasty decisions.  Don't bind
yourself to a libc unless you have to or there are any benefits to be
gained.  I fail to see the benefits.  Currently, the diet libc produces
the smallest binaries by a healthy margin, but binding yourself to the
diet libc does not gain you anything over using APIs that are portable
across all libcs.  In fact, I avoided offering any diet libc specific
APIs at all (with the exception of write12.h).

> I had asked the dietLibc authors about the ability of tweaking their
> library into something that resembles the above, but didn't get a
> response.

Huh?
What email are you talking about here?

Your initial email asked whether you could use parts of the diet libc
for diethotplug and for tweaking, and I said OK.

> Hence my post.  I would love to work with the authors of an
> existing libc to build such a library, as I have other things I would
> rather work on than a libc :)

Before doing any real work, we need to get the specification straight.
What exactly do we need?  The initrd stuff is too vague for my taste.
How many programs do we want to have?  What should those programs do?

> Comments from the various libc authors?  Comments from other kernel
> developers about requirements and goals they would like to see from such
> a libc?

I think you need to ask initrd users.
My understanding was that people want to use the IP autoconfiguration
stuff from the kernel to initrd.  Is that still so?  What other programs
are needed?

Felix

initramfs programs (was [RFC] klibc requirements)

[Greg KH]

On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> 
> How many programs are we talking about here?  And what should they do?

Very good question that we should probably answer first (I'll follow up
to your other points in a separate message).

Here's what I want to have in my initramfs:
	- /sbin/hotplug
	- /sbin/modprobe
	- modules.dep (needed for modprobe, but is a text file)

What does everyone else need/want there?

greg k-h

Re: initramfs programs (was [RFC] klibc requirements)

[Greg KH]

On Tue, Jan 08, 2002 at 08:34:47PM -0800, Greg KH wrote:
> 
> Here's what I want to have in my initramfs:
> 	- /sbin/hotplug
> 	- /sbin/modprobe
> 	- modules.dep (needed for modprobe, but is a text file)

Forgot the modules themselves.  That would be helpful...

greg k-h

Re: initramfs programs (was [RFC] klibc requirements)

[H. Peter Anvin]

Followup to:  <20020109061037.GB18024@kroah.com>
By author:    Greg KH <greg@kroah.com>
In newsgroup: linux.dev.kernel
>
> On Tue, Jan 08, 2002 at 08:34:47PM -0800, Greg KH wrote:
> > 
> > Here's what I want to have in my initramfs:
> > 	- /sbin/hotplug
> > 	- /sbin/modprobe
> > 	- modules.dep (needed for modprobe, but is a text file)
> 
> Forgot the modules themselves.  That would be helpful...
> 

Sure, but those are data files as far as the (k)libc is concerned.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: initramfs programs (was [RFC] klibc requirements)

[Kai Germaschewski]

On Tue, 8 Jan 2002, Greg KH wrote:

> On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> > 
> > How many programs are we talking about here?  And what should they do?
> 
> Very good question that we should probably answer first (I'll follow up
> to your other points in a separate message).
> 
> Here's what I want to have in my initramfs:
> 	- /sbin/hotplug
> 	- /sbin/modprobe
> 	- modules.dep (needed for modprobe, but is a text file)
> 
> What does everyone else need/want there?

Provision to mount the real root, i.e.
- mount

Mount the real root via NFS
- ifconfig/dhcpcd/pump
- portmap?

Or did I get that wrong?

--Kai

Re: initramfs programs (was [RFC] klibc requirements)

[Erik Andersen]

On Tue Jan 08, 2002 at 08:34:47PM -0800, Greg KH wrote:
> On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> > 
> > How many programs are we talking about here?  And what should they do?
> 
> Very good question that we should probably answer first (I'll follow up
> to your other points in a separate message).
> 
> Here's what I want to have in my initramfs:
> 	- /sbin/hotplug
> 	- /sbin/modprobe
> 	- modules.dep (needed for modprobe, but is a text file)
> 
> What does everyone else need/want there?

I think you want a staticly linked multi-call binary, a 
trivial shell, and some kernel modules...
    http://www.uwsg.indiana.edu/hypermail/linux/kernel/0112.2/0681.html

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] klibc requirements

[Greg KH]

On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> My understanding of what "initramfs programs" actually means is vague at
> best.  Are these just programs that are intended to work in an initial
> ram disk?  Or is it a special collection that is included in the kernel
> distribution?

I don't know if they will be included in the kernel distribution or not.
But they will be part of the kernel build process, if only to copy them
to the ramfs image which is then added to the kernel image.

The dietHotplug program will need to be built each time against the
kernel that it is going to be bundled up with (that's how it gets the
size improvements.)

Maybe in the end, specialized programs like dietHotplug will become part
of the kernel source tree, due to them being so tightly bound to it.
Does anyone else have any thoughts about this?

> I don't see why we would need to include those programs with the kernel.
> The kernel tries hard to provide backwards compatibility to old versions
> of syscalls that have changed over the years.  That's why we _have_ a
> standard.  Also, we don't ship glibc with the kernel sources, and we
> didn't when our libc was Linux specific.

These programs are basically the movement of a lot of existing in-kernel
code, to userspace.  And as they need to be bundled up within the kernel
image itself, they don't need to have to run on any kernel but that one.

> If we follow that argumentation and are talking here about programs that
> aren't included in the kernel, why demand a specific libc for them at
> all?  Of course glibc is out of the question, but both the kernel API
> _and_ the libc API are standardized.  It does not make sense to write
> code that demands a specific libc if it can be avoided.  If you need
> any special syscall supported that is not yet part of the diet libc or
> uClibc, Eric and I will probably be glad to add support for it.

Great!  We are needing a small libc for these userspace programs.  A
limited number of libc functions, and a clean syscall interface.

> > 	- portable, runs on all platforms that the kernel currently
> > 	  works on, but doesn't have to run on any non-Linux based OS.

You didn't address this.  What are the future plans of porting dietLibc
to the platforms that are not currently supported by it (but are by
Linux)?

> When you link statically, it does not matter whether the libc also
> supports the rest of POSIX.  The size of libc.a does not matter.  It
> only matters which parts are referenced.  Since we are talking about new
> software specifically written for Linux and with the goal to be small,
> there are no legacy requirements to cater to.  We can write code without
> printf and stdio, for example.  Also, we probably don't need regular
> expressions or DNS.  Those are the big space hogs when linking
> statically against a libc.  In the diet libc, all of the above are very
> small, but avoiding them in the first place is better then optimizing
> them for small size.
> 
> And if we don't use all that bloaty code, it does not matter if we use
> diet libc, uClibc or tomorrow's next great small libc.

Agreed.

<good argument against dynamic linking snipped>

You don't have to convince me about not really wanting dynamic linking.
As I do not know how many programs other people want in their initramfs,
or the size of these programs, I don't know if the size of a loader and
the symbol tables will outweigh the size of statically linking all of
the individual programs themselves.

It's just something to be aware of that we might have to do in the
future, that's all.  And it's nice to see that dietLibc supports this :)

> > I had asked the dietLibc authors about the ability of tweaking their
> > library into something that resembles the above, but didn't get a
> > response.
> 
> Huh?
> What email are you talking about here?

It was sent and received on the dietlibc mailing list on Jan 04, 2001.
It looks like the mailing list archive for the mailing list doesn't have
any messages for Jan, 2001, otherwise I would point you at it.  I can
forward it to you offline if you want me to.

> Before doing any real work, we need to get the specification straight.
> What exactly do we need?  The initrd stuff is too vague for my taste.
> How many programs do we want to have?  What should those programs do?
> 
> I think you need to ask initrd users.
> My understanding was that people want to use the IP autoconfiguration
> stuff from the kernel to initrd.  Is that still so?  What other programs
> are needed?

These are great questions that need to be answered.  I've started a
different topic just for it.

thanks,

greg k-h

Re: [RFC] klibc requirements

[H. Peter Anvin]

Followup to:  <20020109045109.GA17776@kroah.com>
By author:    Greg KH <greg@kroah.com>
In newsgroup: linux.dev.kernel
>
> On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> > My understanding of what "initramfs programs" actually means is vague at
> > best.  Are these just programs that are intended to work in an initial
> > ram disk?  Or is it a special collection that is included in the kernel
> > distribution?
> 
> I don't know if they will be included in the kernel distribution or not.
> But they will be part of the kernel build process, if only to copy them
> to the ramfs image which is then added to the kernel image.
> 

Why should it be included in the kernel image?  That's not the current
plan, as far as I know.  It should be a separate file or set of files
loaded by the bootloader (using an enhanced initrd protocol backward
compatible with old bootloaders.)  Of course, it might be convenient
to have them come out of the same source distribution, but that's
really an unrelated issue.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: [RFC] klibc requirements

[Greg KH]

On Tue, Jan 08, 2002 at 09:01:16PM -0800, H. Peter Anvin wrote:
> 
> Why should it be included in the kernel image?  That's not the current
> plan, as far as I know.  It should be a separate file or set of files
> loaded by the bootloader (using an enhanced initrd protocol backward
> compatible with old bootloaders.)

Hm, missed those messages.  I remember talk of adding the initramfs
image to the kernel image itself, which keeps from having to change any
bootloaders.  But if this has changed, that's ok with me.

Is any of this written down all in one place?

> Of course, it might be convenient
> to have them come out of the same source distribution, but that's
> really an unrelated issue.

If the files contain kernel modules, they need to come out of the same
source distribution :)

thanks,

greg k-h

Re: [RFC] klibc requirements

[H. Peter Anvin]

Followup to:  <20020109060951.GA18024@kroah.com>
By author:    Greg KH <greg@kroah.com>
In newsgroup: linux.dev.kernel
>
> On Tue, Jan 08, 2002 at 09:01:16PM -0800, H. Peter Anvin wrote:
> > 
> > Why should it be included in the kernel image?  That's not the current
> > plan, as far as I know.  It should be a separate file or set of files
> > loaded by the bootloader (using an enhanced initrd protocol backward
> > compatible with old bootloaders.)
> 
> Hm, missed those messages.  I remember talk of adding the initramfs
> image to the kernel image itself, which keeps from having to change any
> bootloaders.  But if this has changed, that's ok with me.
> 
> Is any of this written down all in one place?
> 

This is a *draft* of the specification; I was hoping to get Viro to
comment on it before releasing it publically -- there still may need
to be some tightening of the definition especially with respect to the
definition of alignment, but the "big picture" should be correct:


		       initramfs buffer format
		       -----------------------

		       Al Viro, H. Peter Anvin
		      Last revision: 2002-01-03

  ** DRAFT ** DRAFT ** DRAFT ** DRAFT ** DRAFT ** DRAFT ** DRAFT **

Starting with kernel 2.5.x, the old "initial ramdisk" protocol is
getting {replaced/complemented} with the new "initial ramfs"
(initramfs) protocol.  The initramfs contents is passed using the same
memory buffer protocol used by the initrd protocol, but the contents
is different.  The initramfs buffer contains an archive which is
expanded into a ramfs filesystem; this document details the format of
the initramfs buffer format.

The initramfs buffer format is based around the "newc" CPIO format,
and can be created with the cpio(1) utility.  The cpio archive can be
compressed using gzip(1).  The simplest form of the initramfs buffer
is thus a single .cpio.gz file.

The full format of the initramfs buffer is defined by the following
grammar, where:
	*	is used to indicate "0 or more occurrences of"
	(|)	indicates alternatives
	+	indicates concatenation
	GZIP()	indicates the gzip(1) of the operand
	PAD(n)	means padding with null bytes to an n-byte boundary
	[QUESTION: is the padding relative to the start of the
	previous header, or is it an absolute address?  Is it at all
	legal to have a header start on a non-multiple of 4?]

	initramfs  := ("\0" | cpio_archive | cpio_gzip_archive)*

	cpio_gzip_archive := GZIP(cpio_archive)

	cpio_archive := cpio_file* + (<nothing> | cpio_trailer)

	cpio_file := cpio_header + filename + "\0" + PAD(4) + data + PAD(4)

	cpio_trailer := cpio_header + "TRAILER!!!\0" + PAD(4)


In human terms, the initramfs buffer contains a collection of
compressed and/or uncompressed cpio archives (in the "newc" format);
arbitrary amounts zero bytes (for padding) can be added between
members.

The cpio "TRAILER!!!" entry (cpio end of file) is optional, but is not
ignored; see "handling of hard links" below.

The structure of the cpio_header is as follows (all 8-byte entries
contain 32-bit hexadecimal ASCII numbers):

Field name    Field size	 Meaning
c_magic	      6 bytes		 The string "070701" or "070702"
c_ino	      8 bytes		 File inode number
c_mode	      8 bytes		 File mode and permissions
c_uid	      8 bytes		 File uid
c_gid	      8 bytes		 File gid
c_nlink	      8 bytes		 Number of links
c_mtime	      8 bytes		 Modification time
c_filesize    8 bytes		 Size of data field
c_maj	      8 bytes		 Major part of file device number
c_min	      8 bytes		 Minor part of file device number
c_rmaj	      8 bytes		 Major part of device node reference
c_rmin	      8 bytes		 Minor part of device node reference
c_namesize    8 bytes		 Length of filename, including final \0
c_chksum      8 bytes		 CRC of data field if c_magic is 070702

The c_mode field matches the contents of st_mode returned by stat(2)
on Linux, and encodes the file type and file permissions.

The c_filesize should be zero for any non-regular file.

If the filename is "TRAILER!!!" this is actually an end-of-file
marker; the c_filesize for an end-of-file marker must be zero.


*** Handling of hard links

When a nondirectory with c_nlink > 1 is seen, the (c_maj,c_min,c_ino)
tuple is looked up in a tuple buffer.  If not found, it is entered in
the tuple buffer and the entry is created as usual; if found, a hard
link rather than a second copy of the file is created.  It is not
necessary (but permitted) to include a second copy of the file
contents; if the file contents is not included, the c_filesize field
should be set to zero to indicate no data section follows.

When a "TRAILER!!!" end-of-file marker is seen, the tuple buffer is
reset.  This permits archives which are generated independently to be
concatenated.

To combine file data from different sources (without having to
regenerate the (c_maj,c_min,c_ino) fields), therefore, either one of
the following techniques can be used:

a) Separate the different file data sources with a "TRAILER!!!"
   end-of-file marker, or

b) Make sure c_nlink == 1 for all nondirectory entries.

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: [RFC] klibc requirements

[Rusty Russell]

On 8 Jan 2002 23:26:46 -0800
"H. Peter Anvin" <hpa@zytor.com> wrote:
> *** Handling of hard links
> 
> When a nondirectory with c_nlink > 1 is seen, the (c_maj,c_min,c_ino)
> tuple is looked up in a tuple buffer.  If not found, it is entered in
> the tuple buffer and the entry is created as usual; if found, a hard
> link rather than a second copy of the file is created.  It is not

HPA,
	gnu cpio (v 2.4.2) actually puts the contents in the *last*
entry, for hardlinks in "newc" format.  This probably means you should
specify that if it's a found tuple, and c_filesize is non-zero,
overwrite the contents of the file.

Cheers!
Rusty.
-- 
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: [RFC] klibc requirements

[H. Peter Anvin]

Followup to:  <20020110173029.7616f752.rusty@rustcorp.com.au>
By author:    Rusty Russell <rusty@rustcorp.com.au>
In newsgroup: linux.dev.kernel
> 
> HPA,
> 	gnu cpio (v 2.4.2) actually puts the contents in the *last*
> entry, for hardlinks in "newc" format.  This probably means you should
> specify that if it's a found tuple, and c_filesize is non-zero,
> overwrite the contents of the file.
> 

Check.  I must admit this greatly surprises me (since it means that it
needs to do more than one pass), but it certainly isn't making the
spec any more difficult.  As I said, it's a draft.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: [RFC] klibc requirements

[Matthias Kilian]

On 8 Jan 2002, H. Peter Anvin wrote:

> 		       initramfs buffer format
[...]
> compressed using gzip(1).  The simplest form of the initramfs buffer
> is thus a single .cpio.gz file.

FYI: I've already implemented this *simplest* form for .tar.gz files. A
patch can be found here:

http://www.escape.de/users/outback/linux/patch-2.4.17-inittar.gz

(I could also prepare a patch against 2.5.x)

So, before starting to implement the initramfs stuff, may be you should
have a look on my version.

Changing it from .tar to .cpio (if required) shouldn't be too difficult.

Bye,
	Kili

Re: [RFC] klibc requirements

[H. Peter Anvin]

Matthias Kilian wrote:

> 
> FYI: I've already implemented this *simplest* form for .tar.gz files. A
> patch can be found here:
> 
> http://www.escape.de/users/outback/linux/patch-2.4.17-inittar.gz
> 
> (I could also prepare a patch against 2.5.x)
> 
> So, before starting to implement the initramfs stuff, may be you should
> have a look on my version.
> 


I think Al has already implemented it.

	-hpa

Re: [RFC] klibc requirements

[Felix von Leitner]

Thus spake Greg KH (greg@kroah.com):
> > > 	- portable, runs on all platforms that the kernel currently
> > > 	  works on, but doesn't have to run on any non-Linux based OS.
> You didn't address this.  What are the future plans of porting dietLibc
> to the platforms that are not currently supported by it (but are by
> Linux)?

We will attempt to port the diet libc to every Linux platform that we
have access to.  I won't go out and buy hardware to port my software to,
so if anyone needs the diet libc ported somewhere, I need an account on
a test box somewhere.

We are currently missing sh-linux, ia64-linux and m68k-linux and
probably a few architectures that I have never heard of ;)

> It was sent and received on the dietlibc mailing list on Jan 04, 2001.
> It looks like the mailing list archive for the mailing list doesn't have
> any messages for Jan, 2001, otherwise I would point you at it.  I can
> forward it to you offline if you want me to.

Please do so!

Felix

Re: [RFC] klibc requirements

[Lars Brinkhoff]

Felix von Leitner <felix-dietlibc@fefe.de> writes:
> Thus spake Greg KH (greg@kroah.com):
> > > > 	- portable, runs on all platforms that the kernel currently
> > > > 	  works on, but doesn't have to run on any non-Linux based OS.
> > What are the future plans of porting dietLibc to the platforms
> > that are not currently supported by it (but are by Linux)?
> We will attempt to port the diet libc to every Linux platform that
> we have access to.  We are currently missing sh-linux, ia64-linux
> and m68k-linux and probably a few architectures that I have never
> heard of ;)

For the record: cris, parisc, s390, vax, x86-64.

-- 
Lars Brinkhoff          http://lars.nocrew.org/     Linux, GCC, PDP-10
Brinkhoff Consulting    http://www.brinkhoff.se/    programming

Re: [RFC] klibc requirements

[Erik Andersen]

On Wed Jan 09, 2002 at 03:15:14PM +0100, Felix von Leitner wrote:
> Thus spake Greg KH (greg@kroah.com):
> > > > 	- portable, runs on all platforms that the kernel currently
> > > > 	  works on, but doesn't have to run on any non-Linux based OS.
> > You didn't address this.  What are the future plans of porting dietLibc
> > to the platforms that are not currently supported by it (but are by
> > Linux)?
> 
> We will attempt to port the diet libc to every Linux platform that we
> have access to.  I won't go out and buy hardware to port my software to,
> so if anyone needs the diet libc ported somewhere, I need an account on
> a test box somewhere.
> 
> We are currently missing sh-linux, ia64-linux and m68k-linux and
> probably a few architectures that I have never heard of ;)

uClibc runs on both sh and ,68k, so the only arch missing 
coverage with either dietlibc or uClibc seems to be ia64.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: initramfs programs (was [RFC] klibc requirements)

[Anton Altaparmakov]

At 04:34 09/01/02, Greg KH wrote:
>On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> >
> > How many programs are we talking about here?  And what should they do?
>
>Very good question that we should probably answer first (I'll follow up
>to your other points in a separate message).
>
>Here's what I want to have in my initramfs:
>         - /sbin/hotplug
>         - /sbin/modprobe
>         - modules.dep (needed for modprobe, but is a text file)
>
>What does everyone else need/want there?

It is planned to move partition discovery to userspace which would then 
instruct the kernel of the positions of various partitions.

The program(s) to do this will need to be in pretty much everyones initramfs...

Best regards,

         Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: initramfs programs (was [RFC] klibc requirements)

[Pavel Machek]

Hi!

> >> How many programs are we talking about here?  And what should they do?
> >
> >Very good question that we should probably answer first (I'll follow up
> >to your other points in a separate message).
> >
> >Here's what I want to have in my initramfs:
> >        - /sbin/hotplug
> >        - /sbin/modprobe
> >        - modules.dep (needed for modprobe, but is a text file)
> >
> >What does everyone else need/want there?
> 
> It is planned to move partition discovery to userspace which would then 
> instruct the kernel of the positions of various partitions.

Hmm, and when you insert PCMCIA harddrive, what happens?

-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.

Re: initramfs programs (was [RFC] klibc requirements)

[Patrick Mochel]

On Wed, 9 Jan 2002, Pavel Machek wrote:

> Hi!
>
> > >> How many programs are we talking about here?  And what should they do?
> > >
> > >Very good question that we should probably answer first (I'll follow up
> > >to your other points in a separate message).
> > >
> > >Here's what I want to have in my initramfs:
> > >        - /sbin/hotplug
> > >        - /sbin/modprobe
> > >        - modules.dep (needed for modprobe, but is a text file)
> > >
> > >What does everyone else need/want there?
> >
> > It is planned to move partition discovery to userspace which would then
> > instruct the kernel of the positions of various partitions.
>
> Hmm, and when you insert PCMCIA harddrive, what happens?

If you're booting from that hard drive, you make sure you have the modules
to talk to the drive. In general, you only need to modules to mount the
real root partition, right?

Will we need cardmgr in the future, or will be able to get away with
/sbin/hotplug?

	-pat

Re: initramfs programs (was [RFC] klibc requirements)

[Greg KH]

On Wed, Jan 09, 2002 at 08:04:44AM -0800, Patrick Mochel wrote:
> 
> Will we need cardmgr in the future, or will be able to get away with
> /sbin/hotplug?

Hopefully only /sbin/hotplug if the pcmcia port to the kernel is ever
finished (some people were working on it.)  If not, then we will need
cardmgr for some cards.

greg k-h

Re: initramfs programs (was [RFC] klibc requirements)

[Pavel Machek]

Hi!

> > > >> How many programs are we talking about here?  And what should they do?
> > > >
> > > >Very good question that we should probably answer first (I'll follow up
> > > >to your other points in a separate message).
> > > >
> > > >Here's what I want to have in my initramfs:
> > > >        - /sbin/hotplug
> > > >        - /sbin/modprobe
> > > >        - modules.dep (needed for modprobe, but is a text file)
> > > >
> > > >What does everyone else need/want there?
> > >
> > > It is planned to move partition discovery to userspace which would then
> > > instruct the kernel of the positions of various partitions.
> >
> > Hmm, and when you insert PCMCIA harddrive, what happens?
> 
> If you're booting from that hard drive, you make sure you have the modules
> to talk to the drive. In general, you only need to modules to mount the
> real root partition, right?

But that means that I'll need partition discovery code twice. Once on
initrd, and once on my root, because if I insert PCMCIA harddrive on
runtime, I'll need same detection.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.

Re: initramfs programs (was [RFC] klibc requirements)

[Andreas Ferber]

On Wed, Jan 09, 2002 at 05:29:51PM +0100, Pavel Machek wrote:
> 
> But that means that I'll need partition discovery code twice. Once on
> initrd, and once on my root, because if I insert PCMCIA harddrive on
> runtime, I'll need same detection.

Where is the problem? Generally I think you will have /every/ piece of
code on the initramfs also somewhere on your normal fs, to build the
initramfs image from. Also, who says that you have to abandon your
initramfs after the switch to the real root fs?

Andreas
-- 
       Andreas Ferber - dev/consulting GmbH - Bielefeld, FRG
     ---------------------------------------------------------
         +49 521 1365800 - af@devcon.net - www.devcon.net

Re: initramfs programs (was [RFC] klibc requirements)

[Alex Bligh - linux-kernel]

>> Here's what I want to have in my initramfs:
>>         - /sbin/hotplug
>>         - /sbin/modprobe
>>         - modules.dep (needed for modprobe, but is a text file)
>>
>> What does everyone else need/want there?
>
> It is planned to move partition discovery to userspace which would then
> instruct the kernel of the positions of various partitions.
>
> The program(s) to do this will need to be in pretty much everyones
> initramfs...

What with mounting root via NFS, hence having to set up
IP et al, mounting various different
partition types, avoiding the kludge of fsck etc.,
being able to recover from a corrupted root, you
might as well just cpio up your /sbin and stick
that in, and be able to run single user mode without
a 'normal' root. <FX: ducks & runs>

seriously point: ls /sbin gives a /maximum/ range I'd
have thought.

--
Alex Bligh

Re: initramfs programs (was [RFC] klibc requirements)

[Anton Altaparmakov]

At 21:15 09/01/2002, Alex Bligh - linux-kernel wrote:
>>>Here's what I want to have in my initramfs:
>>>         - /sbin/hotplug
>>>         - /sbin/modprobe
>>>         - modules.dep (needed for modprobe, but is a text file)
>>>
>>>What does everyone else need/want there?
>>
>>It is planned to move partition discovery to userspace which would then
>>instruct the kernel of the positions of various partitions.
>>
>>The program(s) to do this will need to be in pretty much everyones
>>initramfs...
>
>What with mounting root via NFS, hence having to set up
>IP et al, mounting various different
>partition types, avoiding the kludge of fsck etc.,
>being able to recover from a corrupted root, you
>might as well just cpio up your /sbin and stick
>that in, and be able to run single user mode without
>a 'normal' root. <FX: ducks & runs>
>
>seriously point: ls /sbin gives a /maximum/ range I'd
>have thought.

Partition discovery is currently done within the kernel itself. The code 
will effectively 'just' move out into user space. As such it is not present 
in /sbin now but it will be in initramfs. The same is true for various 
other code I can imagine moving out of kernel mode into initramfs...

Best regards,

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: initramfs programs (was [RFC] klibc requirements)

[Greg KH]

On Wed, Jan 09, 2002 at 09:34:34PM +0000, Anton Altaparmakov wrote:
> Partition discovery is currently done within the kernel itself. The code 
> will effectively 'just' move out into user space. As such it is not present 
> in /sbin now but it will be in initramfs. The same is true for various 
> other code I can imagine moving out of kernel mode into initramfs...

For this code, I can see it staying in the kernel source tree, and being
built as part of the kernel build process, right?

greg k-h

Re: initramfs programs (was [RFC] klibc requirements)

[Anton Altaparmakov]

At 21:40 09/01/2002, Greg KH wrote:
>On Wed, Jan 09, 2002 at 09:34:34PM +0000, Anton Altaparmakov wrote:
> > Partition discovery is currently done within the kernel itself. The code
> > will effectively 'just' move out into user space. As such it is not 
> present
> > in /sbin now but it will be in initramfs. The same is true for various
> > other code I can imagine moving out of kernel mode into initramfs...
>
>For this code, I can see it staying in the kernel source tree, and being
>built as part of the kernel build process, right?

I would think that is a good idea but I am not sure that is what is planned 
/ will happen. Keeping it outside would have the advantage that a newer 
partition recognizer (or whatever other code) can be applied to any 
existing kernel version (that supports initramfs).

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: initramfs programs (was [RFC] klibc requirements)

[Andreas Ferber]

On Wed, Jan 09, 2002 at 09:55:34PM +0000, Anton Altaparmakov wrote:
> 
> I would think that is a good idea but I am not sure that is what is planned 
> / will happen. Keeping it outside would have the advantage that a newer 
> partition recognizer (or whatever other code) can be applied to any 
> existing kernel version (that supports initramfs).

This could be done anyway: just replace the initramfs image built by 
the kernel build with anotherone built from another source tree. It
would be helpful though if the tools were distributed both standalone
and included into the kernel tree.

Andreas
-- 
       Andreas Ferber - dev/consulting GmbH - Bielefeld, FRG
     ---------------------------------------------------------
         +49 521 1365800 - af@devcon.net - www.devcon.net

Re: initramfs programs (was [RFC] klibc requirements)

[Tom Rini]

On Wed, Jan 09, 2002 at 11:15:28PM +0100, Andreas Ferber wrote:
> On Wed, Jan 09, 2002 at 09:55:34PM +0000, Anton Altaparmakov wrote:
> > 
> > I would think that is a good idea but I am not sure that is what is planned 
> > / will happen. Keeping it outside would have the advantage that a newer 
> > partition recognizer (or whatever other code) can be applied to any 
> > existing kernel version (that supports initramfs).
> 
> This could be done anyway: just replace the initramfs image built by 
> the kernel build with anotherone built from another source tree. It
> would be helpful though if the tools were distributed both standalone
> and included into the kernel tree.

If the kernel is going to build an initramfs option, it also needs a way
to be given one.  The issue I'm thinking of is I know of a few platforms
where the initramfs archive will have to be part of the 'zImage' file
(much like they do for ramdisks now).

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: initramfs programs (was [RFC] klibc requirements)

[Andreas Ferber]

On Wed, Jan 09, 2002 at 05:25:07PM -0700, Tom Rini wrote:
> > 
> > This could be done anyway: just replace the initramfs image built by 
> > the kernel build with anotherone built from another source tree. It
> > would be helpful though if the tools were distributed both standalone
> > and included into the kernel tree.
> If the kernel is going to build an initramfs option, it also needs a way
> to be given one.  The issue I'm thinking of is I know of a few platforms
> where the initramfs archive will have to be part of the 'zImage' file
> (much like they do for ramdisks now).

Append it to the zImage and let the kernel look for it there. Plus add
a tool to util-linux (or maybe an option to rdev?) to let you replace
the embedded initramfs in a {,b}zImage with a customized one.

Andreas
-- 
       Andreas Ferber - dev/consulting GmbH - Bielefeld, FRG
     ---------------------------------------------------------
         +49 521 1365800 - af@devcon.net - www.devcon.net

Re: initramfs programs (was [RFC] klibc requirements)

[Tom Rini]

On Thu, Jan 10, 2002 at 01:38:56AM +0100, Andreas Ferber wrote:
> On Wed, Jan 09, 2002 at 05:25:07PM -0700, Tom Rini wrote:
> > > 
> > > This could be done anyway: just replace the initramfs image built by 
> > > the kernel build with anotherone built from another source tree. It
> > > would be helpful though if the tools were distributed both standalone
> > > and included into the kernel tree.
> > If the kernel is going to build an initramfs option, it also needs a way
> > to be given one.  The issue I'm thinking of is I know of a few platforms
> > where the initramfs archive will have to be part of the 'zImage' file
> > (much like they do for ramdisks now).
> 
> Append it to the zImage and let the kernel look for it there. Plus add
> a tool to util-linux (or maybe an option to rdev?) to let you replace
> the embedded initramfs in a {,b}zImage with a customized one.

Er, 'rdev' is an x86-only program, so lets not add common functionality
to that.  And I'd rather not throw something onto the end of the
'zImage' since I just got done removing annoying/broken things like
that.  Mind you I don't think x86 will take advantage (or have reason
to, really) embedded the initramfs image into the bzImage/zImage, unless
we're going to let them run w/o a bootloader still.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: initramfs programs (was [RFC] klibc requirements)

[Rob Landley]

On Wednesday 09 January 2002 09:42 pm, Tom Rini wrote:
> On Thu, Jan 10, 2002 at 01:38:56AM +0100, Andreas Ferber wrote:
> > On Wed, Jan 09, 2002 at 05:25:07PM -0700, Tom Rini wrote:
> > > > This could be done anyway: just replace the initramfs image built by
> > > > the kernel build with anotherone built from another source tree. It
> > > > would be helpful though if the tools were distributed both standalone
> > > > and included into the kernel tree.
> > >
> > > If the kernel is going to build an initramfs option, it also needs a
> > > way to be given one.  The issue I'm thinking of is I know of a few
> > > platforms where the initramfs archive will have to be part of the
> > > 'zImage' file (much like they do for ramdisks now).
> >
> > Append it to the zImage and let the kernel look for it there. Plus add
> > a tool to util-linux (or maybe an option to rdev?) to let you replace
> > the embedded initramfs in a {,b}zImage with a customized one.
>
> Er, 'rdev' is an x86-only program, so lets not add common functionality
> to that.  And I'd rather not throw something onto the end of the
> 'zImage' since I just got done removing annoying/broken things like
> that.

You want it to be an ELF section then?

Rob

Re: initramfs programs (was [RFC] klibc requirements)

[Tom Rini]

On Thu, Jan 10, 2002 at 09:09:01AM -0500, Rob Landley wrote:
> On Wednesday 09 January 2002 09:42 pm, Tom Rini wrote:
> > On Thu, Jan 10, 2002 at 01:38:56AM +0100, Andreas Ferber wrote:
> > > On Wed, Jan 09, 2002 at 05:25:07PM -0700, Tom Rini wrote:
> > > > > This could be done anyway: just replace the initramfs image built by
> > > > > the kernel build with anotherone built from another source tree. It
> > > > > would be helpful though if the tools were distributed both standalone
> > > > > and included into the kernel tree.
> > > >
> > > > If the kernel is going to build an initramfs option, it also needs a
> > > > way to be given one.  The issue I'm thinking of is I know of a few
> > > > platforms where the initramfs archive will have to be part of the
> > > > 'zImage' file (much like they do for ramdisks now).
> > >
> > > Append it to the zImage and let the kernel look for it there. Plus add
> > > a tool to util-linux (or maybe an option to rdev?) to let you replace
> > > the embedded initramfs in a {,b}zImage with a customized one.
> >
> > Er, 'rdev' is an x86-only program, so lets not add common functionality
> > to that.  And I'd rather not throw something onto the end of the
> > 'zImage' since I just got done removing annoying/broken things like
> > that.
> 
> You want it to be an ELF section then?

I want it to be implementation specific.  The running kernel shouldn't
care how it got the image, just that it did.

And yes, on PPC it will either be given one by a 'real' bootloader
(yaboot, ppcboot) or an ELF section like the kernel itself, ramdisk and
sometimes System.map are.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: initramfs programs (was [RFC] klibc requirements)

[H. Peter Anvin]

Followup to:  <200201102155.g0ALtkA22362@snark.thyrsus.com>
By author:    Rob Landley <landley@trommello.org>
In newsgroup: linux.dev.kernel
> >
> > Er, 'rdev' is an x86-only program, so lets not add common functionality
> > to that.  And I'd rather not throw something onto the end of the
> > 'zImage' since I just got done removing annoying/broken things like
> > that.
> 
> You want it to be an ELF section then?
> 

initrd.  It's already supported on all architectures; initramfs uses
the same buffer-transfer architecture, which means you can use
unmodified bootloaders.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: initramfs programs (was [RFC] klibc requirements)

[Alex Bligh - linux-kernel]

--On Wednesday, 09 January, 2002 9:34 PM +0000 Anton Altaparmakov 
<aia21@cam.ac.uk> wrote:

>> seriously point: ls /sbin gives a /maximum/ range I'd
>> have thought.
>
> Partition discovery is currently done within the kernel itself. The code
> will effectively 'just' move out into user space.

Apologies - of course; I meant ls /sbin union {anything moved
out of kernel} gives a maximum range. I could find rationales,
some more questionable than others, for about half the stuff
in /sbin (for instance, if you are mounting NFS over IP, you
/might/ want to have iptables support in there before you start
playing with ip operations which without them might cause
a comprimized root to be mounted - I said /might/).

--
Alex Bligh

Re: [RFC] klibc requirements

[Juan Quintela]

>>>>> "felix" == Felix von Leitner <felix-dietlibc@fefe.de> writes:

Hi

>> Comments from the various libc authors?  Comments from other kernel
>> developers about requirements and goals they would like to see from such
>> a libc?

felix> I think you need to ask initrd users.
felix> My understanding was that people want to use the IP autoconfiguration
felix> stuff from the kernel to initrd.  Is that still so?  What other programs
felix> are needed?

I can think of that in a fast thought:

- if fsck.* can be there, it will make fs nice, just now they have to
  be able to fsck a ro root fs.
- I suppose that you can put here also the raid autodetect code and
  things like that.
- you also need a very small minishell, something like nash (that
  cames with mkinitrd will suffice).
  Basic utility here is that you want to have several small programs
  and have a small shell script to configure them (a distribution
  dont' want to recompile all its boot programs for each target
  configuration).
- as I suppose that you have supposed insmod is essential :)
- you already put there IP autoconfiguration.

Later, Juan.



-- 
In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy

Re: [RFC] klibc requirements

[Erik Andersen]

On Wed Jan 09, 2002 at 02:23:46PM +0100, Juan Quintela wrote:
> I can think of that in a fast thought:
> 
> - if fsck.* can be there, it will make fs nice, just now they have to
>   be able to fsck a ro root fs.

That would be a nice addition.

> - I suppose that you can put here also the raid autodetect code and
>   things like that.
> - you also need a very small minishell, something like nash (that
>   cames with mkinitrd will suffice).

busybox lash
    http://busybox.net/cgi-bin/cvsweb/busybox/shell/lash.c?rev=1.142&content-type=text/vnd.viewcvs-markup
or busybox msh or busybox ash if you need something thats
all posix compliant. 

> - as I suppose that you have supposed insmod is essential :)

busybox insmod

> - you already put there IP autoconfiguration.

udhcp

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] klibc requirements

[Albert D. Cahalan]

Felix von Leitner writes:

> My understanding of what "initramfs programs" actually means is vague at
> best.  Are these just programs that are intended to work in an initial
> ram disk?  Or is it a special collection that is included in the kernel
> distribution?

1. special collection by default
2. user-specified as desired

I think the dietlibc idea has to be scrapped so we can run BSD apps.
(and others maybe, but I'm not looking to start a flame war)

uClibc is LGPL, so traditional 4-clause BSD licensed code can be
linked with it.

> there are no legacy requirements to cater to.  We can write code without
> printf and stdio, for example.  Also, we probably don't need regular
> expressions or DNS.  Those are the big space hogs when linking
> statically against a libc.  In the diet libc, all of the above are very
> small, but avoiding them in the first place is better then optimizing
> them for small size.

DNS is very good to have. There are many things one might want
to specify by name. NFS servers, NIS servers, SMB servers, and
even the machine itself to get an IP via DNS.

> This may look like a good idea, but dynamic linking should be avoided.
> Trust me on this.  While it is possible to squeeze the dynamic loader
> down to below 10k, 10k really is a lot of code.  And empty program with
> the diet libc is way below 1k on x86.  So to reap the benefit of dynamic
> linking, you would need a lot of programs.  Also please note that -fPIC
> makes code larger.  And we need to keep symbols around, which makes up a
> substantial part of the shared diet libc.

a.out

Even with ELF, you shouldn't need that 10 kB. Treat ELF like a.out,
getting rid of the -fPIC stuff in favor of offsets assigned when
you build the initramfs. Dynamic linking should be:

open
mmap
mmap
close

You know the file to open. You know what offset you need it at.
There isn't any need for symbols. OK, that's half-dynamic,
but it gets the job done.

Re: [RFC] klibc requirements

[Felix von Leitner]

Thus spake Albert D. Cahalan (acahalan@cs.uml.edu):
> I think the dietlibc idea has to be scrapped so we can run BSD apps.
> (and others maybe, but I'm not looking to start a flame war)

What apps are you talking about?

> DNS is very good to have. There are many things one might want
> to specify by name. NFS servers, NIS servers, SMB servers, and
> even the machine itself to get an IP via DNS.

You don't need NIS or SMB before mounting the root disk.
If you want NFS to mount your root file system, you get the IP via DHCP
normally, so you don't need DNS.  And you can't get your own IP via DNS
because you need to have an IP to use DNS.

> Even with ELF, you shouldn't need that 10 kB.

Please go ahead and implement it.
Fame and fortune await you! ;)

> Treat ELF like a.out, getting rid of the -fPIC stuff in favor of
> offsets assigned when you build the initramfs.

ELF is a standard.
You can't just go out and re-invent dynamic linking completely.
First and most importantly of all, the GNU toolchain support the ELF
standard, not my personal ELF dialect.  I have no desire to write a
linker.  I'm happy enough that I found Olaf who was willing to write an
ld.so (and I was close enough to hear all his ranting about how complex
ELF is).

> Dynamic linking should
> be:

> open
> mmap
> mmap
> close

> You know the file to open. You know what offset you need it at.
> There isn't any need for symbols. OK, that's half-dynamic,
> but it gets the job done.

Again, please do it.  I know that it would be possible.  But it would
not be standard and I don't think it's worth the effort.  The most
important reason is that static linking produces small binaries with the
diet libc and uclibc.  As long as your software does not destroy that by
having a few dozen mini binaries around, all linked against the same
monster do-it-all library, it works.

Felix

Re: [RFC] klibc requirements

[David Lang]

On Tue, 15 Jan 2002, Felix von Leitner wrote:

> Date: Tue, 15 Jan 2002 12:55:44 +0100
> From: Felix von Leitner <felix-dietlibc@fefe.de>
> To: Albert D. Cahalan <acahalan@cs.uml.edu>
> Cc: Greg KH <greg@kroah.com>, linux-kernel@vger.kernel.org,
>      andersen@codepoet.org
> Subject: Re: [RFC] klibc requirements
>
> Thus spake Albert D. Cahalan (acahalan@cs.uml.edu):
> > I think the dietlibc idea has to be scrapped so we can run BSD apps.
> > (and others maybe, but I'm not looking to start a flame war)
>
> What apps are you talking about?

he's talking about licencing issues.

LGPL libc replacements can be used with any program, GPL libc replacements
can only be used with programs licenced in a way that can be combined with
GPL (and the resulting program is GPL.

as an example (not for the boot process, but an example of a replacement
libc use) I use the firewall toolkit, it has been around for a _loooong_
time (in software terms anyway) and has a firly odd licence (free for you
to use, source available, cannot sell it) which is not compatable with the
GPL. with glibc staticly linked this makes huge binaries, with libc5 they
were a lot smaller. I would like to try to use this small libc for these
proxies, but if the library is GPL, not LGPL I'm not allowed to.

David Lang

Re: [RFC] klibc requirements

[Doug McNaught]

David Lang <david.lang@digitalinsight.com> writes:

> as an example (not for the boot process, but an example of a replacement
> libc use) I use the firewall toolkit, it has been around for a _loooong_
> time (in software terms anyway) and has a firly odd licence (free for you
> to use, source available, cannot sell it) which is not compatable with the
> GPL. with glibc staticly linked this makes huge binaries, with libc5 they
> were a lot smaller. I would like to try to use this small libc for these
> proxies, but if the library is GPL, not LGPL I'm not allowed to.

Hmm, I think you can; you just can't redistribute it.  Can you even
redistribute fwtk on non-commercial terms?

-Doug
-- 
Let us cross over the river, and rest under the shade of the trees.
   --T. J. Jackson, 1863

Re: [RFC] klibc requirements

[David Lang]

On 15 Jan 2002, Doug McNaught wrote:

>
> > as an example (not for the boot process, but an example of a replacement
> > libc use) I use the firewall toolkit, it has been around for a _loooong_
> > time (in software terms anyway) and has a firly odd licence (free for you
> > to use, source available, cannot sell it) which is not compatable with the
> > GPL. with glibc staticly linked this makes huge binaries, with libc5 they
> > were a lot smaller. I would like to try to use this small libc for these
> > proxies, but if the library is GPL, not LGPL I'm not allowed to.
>
> Hmm, I think you can; you just can't redistribute it.  Can you even
> redistribute fwtk on non-commercial terms?
>
nope, only allowed to get it from nai (and they sure don't make it easy to
find on their website)

David Lang

Re: [RFC] klibc requirements

[Doug McNaught]

David Lang <dlang@diginsite.com> writes:

> On 15 Jan 2002, Doug McNaught wrote:
> 
> >
> > > as an example (not for the boot process, but an example of a replacement
> > > libc use) I use the firewall toolkit, it has been around for a _loooong_
> > > time (in software terms anyway) and has a firly odd licence (free for you
> > > to use, source available, cannot sell it) which is not compatable with the
> > > GPL. with glibc staticly linked this makes huge binaries, with libc5 they
> > > were a lot smaller. I would like to try to use this small libc for these
> > > proxies, but if the library is GPL, not LGPL I'm not allowed to.
> >
> > Hmm, I think you can; you just can't redistribute it.  Can you even
> > redistribute fwtk on non-commercial terms?
> >
> nope, only allowed to get it from nai (and they sure don't make it easy to
> find on their website)

Problem solved, then; you can link fwtk with a GPL'd libc on your own
machines and use it all day.  You can't redistribute fwtk, so you
aren't even tempted to violate the GPL.

-Doug
-- 
Let us cross over the river, and rest under the shade of the trees.
   --T. J. Jackson, 1863

Re: [RFC] klibc requirements

[Erik Andersen]

On Tue Jan 15, 2002 at 06:54:46AM -0800, David Lang wrote:
> were a lot smaller. I would like to try to use this small libc for these
> proxies, but if the library is GPL, not LGPL I'm not allowed to.

So use uClibc, which is LGPL,

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] klibc requirements

[Albert D. Cahalan]

Felix von Leitner writes:
> Thus spake Albert D. Cahalan (acahalan@cs.uml.edu):

>> I think the dietlibc idea has to be scrapped so we can run BSD apps.
>> (and others maybe, but I'm not looking to start a flame war)
>
> What apps are you talking about?

I'm talking about all apps under the traditional 4-clause
BSD license. Every single one of them is incompatible with
a GPL libc. An LGPL library works fine.

BTW, the LGPL isn't just for libraries anymore. It's now the
called the "Lesser" GPL, perfectly suited for apps as well
as libraries. I know this must be what RMS had in mind, so I
released all my "ps" source code under the LGPL. :-)

> You don't need NIS or SMB before mounting the root disk.

NIS can be used to specify what filesystem to use.
SMB could be that filesystem.

>> Treat ELF like a.out, getting rid of the -fPIC stuff in favor of
>> offsets assigned when you build the initramfs.
>
> ELF is a standard.
> You can't just go out and re-invent dynamic linking completely.

Sure you can. As long as the kernel loader is happy, no problem.
People commit worse sins trying to squeeze stuff onto a floppy.

RE: initramfs programs (was [RFC] klibc requirements)

[Torrey Hoffman]

The interesting thing that I currently do with initrd support is a
custom network-booted Linux installer for an embedded system. 

I'd like to be able to do this with initramfs too.  It needs:

- busybox, of course
- sfdisk  (scripted fdisk)
- mkreiserfs
- lilo
- dhcpcd
- A utility to display images in the framebuffer, like "fbv".

Torrey Hoffman

Re: initramfs programs (was [RFC] klibc requirements)

[Tom Rini]

On Wed, Jan 09, 2002 at 09:54:28AM -0800, Torrey Hoffman wrote:

> The interesting thing that I currently do with initrd support is a
> custom network-booted Linux installer for an embedded system. 
> 
> I'd like to be able to do this with initramfs too.  It needs:
[snip]

Er, for this particular application, why would you use klibc, if
existant?  The initramfs stuff could work with glibc, if you didn't mind
a big enough image, it sounds like.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: initramfs programs (was [RFC] klibc requirements)

[Greg KH]

On Wed, Jan 09, 2002 at 09:54:28AM -0800, Torrey Hoffman wrote:
> - lilo

lilo?  Am I missing something, or why would you want to run lilo running
before the rest of your kernel has started up?

greg k-h

Re: initramfs programs (was [RFC] klibc requirements)

[Tom Rini]

On Wed, Jan 09, 2002 at 10:28:17AM -0800, Greg KH wrote:
> On Wed, Jan 09, 2002 at 09:54:28AM -0800, Torrey Hoffman wrote:
> > - lilo
> 
> lilo?  Am I missing something, or why would you want to run lilo running
> before the rest of your kernel has started up?

He's talking about an installer, for presumably x86, embedded hw.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

initramfs programs (was [RFC] klibc requirements)

[Eric S. Raymond]

greg k-h:
>What does everyone else need/want there?

dmidecode, so the init script can dump a DMI report in a known
location such as /var/run/dmi.  

I want this for autoconfiguration purposes.  If I can have it, I
won't need /proc/dmi.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

I cannot undertake to lay my finger on that article of the
Constitution which grant[s] a right to Congress of expending, on
objects of benevolence, the money of their constituents.
	-- James Madison, 1794

Re: initramfs programs (was [RFC] klibc requirements)

[Doug McNaught]

"Eric S. Raymond" <esr@snark.thyrsus.com> writes:

> greg k-h:
> >What does everyone else need/want there?
> 
> dmidecode, so the init script can dump a DMI report in a known
> location such as /var/run/dmi.  
> 
> I want this for autoconfiguration purposes.  If I can have it, I
> won't need /proc/dmi.

Why can't this happen inside the regular startup scripts?  They know
where to put such files; the kernel-level stuff doesn't--I can't think
of any current situation where the kernel writes to an arbitrary file
in the filesystem as it boots.  Sure, /var/run is in the FHS, but that
doesn't mean every system will have it.

IMHO, since /var/run/dmi is not needed by any stage of the kernel
boot, it should be created in the regular startup scripts (invoked by
init(8)). 

-Doug
-- 
Let us cross over the river, and rest under the shade of the trees.
   --T. J. Jackson, 1863

Re: initramfs programs (was [RFC] klibc requirements)

[Eric S. Raymond]

Doug McNaught <doug@wireboard.com>:
> "Eric S. Raymond" <esr@snark.thyrsus.com> writes:
> 
> > greg k-h:
> > >What does everyone else need/want there?
> > 
> > dmidecode, so the init script can dump a DMI report in a known
> > location such as /var/run/dmi.  
> > 
> > I want this for autoconfiguration purposes.  If I can have it, I
> > won't need /proc/dmi.
> 
> Why can't this happen inside the regular startup scripts?  They know
> where to put such files; the kernel-level stuff doesn't--I can't think
> of any current situation where the kernel writes to an arbitrary file
> in the filesystem as it boots.  Sure, /var/run is in the FHS, but that
> doesn't mean every system will have it.
> 
> IMHO, since /var/run/dmi is not needed by any stage of the kernel
> boot, it should be created in the regular startup scripts (invoked by
> init(8)). 

You're right, I don't need this to be done at kernel level.  I do need it to
be done *everywhere*.  I'm not sure how else to guarantee this will happen. 
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

The American Republic will endure, until politicians realize they can
bribe the people with their own money.
	-- Alexis de Tocqueville

Re: initramfs programs (was [RFC] klibc requirements)

[Doug McNaught]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> You're right, I don't need this to be done at kernel level.  I do need it to
> be done *everywhere*.  I'm not sure how else to guarantee this will happen. 

Time to call in your chits with Red Hat and the other distributors. ;)

-Doug
-- 
Let us cross over the river, and rest under the shade of the trees.
   --T. J. Jackson, 1863

Re: initramfs programs (was [RFC] klibc requirements)

[Andreas Ferber]

On Wed, Jan 09, 2002 at 03:44:25PM -0500, Eric S. Raymond wrote:
> 
> You're right, I don't need this to be done at kernel level.  I do need it to
> be done *everywhere*.  I'm not sure how else to guarantee this will happen. 

Then add an init script and include installation of it to the
installation steps of your autoconfigurator (it has to be installed
anyway). If a distributor packages your program, he will include the
init script into his package and enable it according to his init
policy, or write an own init script, if your provided one doesn't
fit.

That's the way it works for network daemons etc. for years.

Andreas
-- 
       Andreas Ferber - dev/consulting GmbH - Bielefeld, FRG
     ---------------------------------------------------------
         +49 521 1365800 - af@devcon.net - www.devcon.net

Re: initramfs programs (was [RFC] klibc requirements)

[Eric S. Raymond]

Andreas Ferber <aferber@techfak.uni-bielefeld.de>:
> Then add an init script and include installation of it to the
> installation steps of your autoconfigurator (it has to be installed
> anyway). If a distributor packages your program, he will include the
> init script into his package and enable it according to his init
> policy, or write an own init script, if your provided one doesn't
> fit.
> 
> That's the way it works for network daemons etc. for years.

This sounds like good advice.  The autoconfigurator is part of CML2,
which I expect to be distributed with the kernel.  Does that change 
your advice at all?
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

"Those who make peaceful revolution impossible 
will make violent revolution inevitable."
	-- John F. Kennedy

Re: initramfs programs (was [RFC] klibc requirements)

[H. Peter Anvin]

Followup to:  <20020109165658.A31246@thyrsus.com>
By author:    "Eric S. Raymond" <esr@thyrsus.com>
In newsgroup: linux.dev.kernel
>
> Andreas Ferber <aferber@techfak.uni-bielefeld.de>:
> > Then add an init script and include installation of it to the
> > installation steps of your autoconfigurator (it has to be installed
> > anyway). If a distributor packages your program, he will include the
> > init script into his package and enable it according to his init
> > policy, or write an own init script, if your provided one doesn't
> > fit.
> > 
> > That's the way it works for network daemons etc. for years.
> 
> This sounds like good advice.  The autoconfigurator is part of CML2,
> which I expect to be distributed with the kernel.  Does that change 
> your advice at all?
> 

The program at hand is actually dmiparse or whatever it's called.  If
dmiparse isn't available, or its input data, print an eror message.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: initramfs programs (was [RFC] klibc requirements)

[Andreas Ferber]

On Wed, Jan 09, 2002 at 04:56:58PM -0500, Eric S. Raymond wrote:

> > That's the way it works for network daemons etc. for years.
> This sounds like good advice.  The autoconfigurator is part of CML2,
> which I expect to be distributed with the kernel.  Does that change 
> your advice at all?

Yes, a little bit. Make dmidecode (or whatever you also need for
preparation steps that have to be done as root) a separate package,
which has to be installed before "make autoconfig" works, and write
that down in Documentation/Changes.

This gives you several benefits:

- you don't depend on the version of the /running/ kernel for "make
  autoconfig" to work (/dev/kmem is available for a /long/ time now).
- you can install and run dmidecode on one machine, copy the retrieved
  data to another machine and autoconfigure/build the kernel there.

Distributions can install the package by default, to make it work for
your grandmother as well, if that's what you want.

Andreas
-- 
       Andreas Ferber - dev/consulting GmbH - Bielefeld, FRG
     ---------------------------------------------------------
         +49 521 1365800 - af@devcon.net - www.devcon.net

Re: initramfs programs (was [RFC] klibc requirements)

[Pavel Machek]

Hi!

> You're right, I don't need this to be done at kernel level.  I do need it to
> be done *everywhere*.  I'm not sure how else to guarantee this will happen. 

Ehm? autoconfig is not good enough reason to force *everyone* to do dmi
scanning. I don't give a damn about autoconfig, and you are not going
to force it to my machines. Stop trying.

It may be reasonable if you wanted major distros to have that. Just don't
say *everyone*.
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: initramfs programs (was [RFC] klibc requirements)

[Patrick Mochel]

On Wed, 9 Jan 2002, Eric S. Raymond wrote:

> greg k-h:
> >What does everyone else need/want there?
>
> dmidecode, so the init script can dump a DMI report in a known
> location such as /var/run/dmi.

Why do you need that during that stage of the boot process? The DMI tables
won't go away.

Besides, is there a way to put them in such a place that when mounting the
real root partition, they won't go away? Or, is the initramfs partition
going to persist during runtime?

However, we may needs parsers to do IRQ routing. Yes, that means (dramatic
horror music) an ACPI interpreter and/or PIRQ table parser.

Which brings me to the question about initramfs configurability. Different
systems have different needs during that process. It seems that we need a
generic way to build applications for initramfs (and link them against the
libc that's used) and build initramfs images.

If building the image is part of the kernel build process, then it's
not an issue. But, it would be nice, IMHO, to separate the two; so you
could build a new initramfs image and tack it on to an already built
kernel...

	-pat

Re: initramfs programs (was [RFC] klibc requirements)

[Eric S. Raymond]

Patrick Mochel <mochel@osdl.org>:
> 
> On Wed, 9 Jan 2002, Eric S. Raymond wrote:
> 
> > greg k-h:
> > >What does everyone else need/want there?
> >
> > dmidecode, so the init script can dump a DMI report in a known
> > location such as /var/run/dmi.
> 
> Why do you need that during that stage of the boot process? The DMI tables
> won't go away.

I know.  My proposal to put dmidecode in initramfs is an alternative to
/proc/dmi, one which won't involve having dmidecode run in kernelspace.

The underlying problem is that dmidecode needs access to kmem, and I can't
assume that the person running my configurator will be root.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

The whole of the Bill [of Rights] is a declaration of the right of the
people at large or considered as individuals...  It establishes some
rights of the individual as unalienable and which consequently, no
majority has a right to deprive them of.
         -- Albert Gallatin, Oct 7 1789

Re: initramfs programs (was [RFC] klibc requirements)

[Matthew Kirkwood]

On Wed, 9 Jan 2002, Eric S. Raymond wrote:

> The underlying problem is that dmidecode needs access to kmem, and I
> can't assume that the person running my configurator will be root.

But you can "su -c" (also sudo, I suppose).  If that person
doesn't have root, then building a kernel isn't going to do
them much good.

Matthew.

Re: initramfs programs (was [RFC] klibc requirements)

[Eric S. Raymond]

Matthew Kirkwood <matthew@hairy.beasts.org>:
> > The underlying problem is that dmidecode needs access to kmem, and I
> > can't assume that the person running my configurator will be root.
> 
> But you can "su -c" (also sudo, I suppose).  If that person
> doesn't have root, then building a kernel isn't going to do
> them much good.

We've been over this already.  No, the configurator user should *not* have
to su at any point before actual kernel installation.  Bad practice, 
no doughnut.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

"The power to tax involves the power to destroy;...the power to
destroy may defeat and render useless the power to create...."
	-- Chief Justice John Marshall, 1819.

Re: initramfs programs (was [RFC] klibc requirements)

[H. Peter Anvin]

Followup to:  <20020109174637.A1742@thyrsus.com>
By author:    "Eric S. Raymond" <esr@thyrsus.com>
In newsgroup: linux.dev.kernel
>
> Matthew Kirkwood <matthew@hairy.beasts.org>:
> > > The underlying problem is that dmidecode needs access to kmem, and I
> > > can't assume that the person running my configurator will be root.
> > 
> > But you can "su -c" (also sudo, I suppose).  If that person
> > doesn't have root, then building a kernel isn't going to do
> > them much good.
> 
> We've been over this already.  No, the configurator user should *not* have
> to su at any point before actual kernel installation.  Bad practice, 
> no doughnut.
> 

We have also been over the fact that dmidecode, if written
appropriately, could be setuid, or call a "dmicat" setuid program.
This is a dmidecode implementation detail.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: initramfs programs (was [RFC] klibc requirements)

[Alan Cox]

> We have also been over the fact that dmidecode, if written
> appropriately, could be setuid, or call a "dmicat" setuid program.
> This is a dmidecode implementation detail.

We've also proved the DMI data is too unreliable to be used, so the entire
problem space is irrelevant

Re: initramfs programs (was [RFC] klibc requirements)

[Dave Jones]

On Thu, 10 Jan 2002, Alan Cox wrote:

> > We have also been over the fact that dmidecode, if written
> > appropriately, could be setuid, or call a "dmicat" setuid program.
> > This is a dmidecode implementation detail.
> We've also proved the DMI data is too unreliable to be used, so the entire
> problem space is irrelevant

That's not a problem, remember Eric volunteered to maintain the
enormous black list 8-)

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: initramfs programs (was [RFC] klibc requirements)

[Matthew Kirkwood]

On Wed, 9 Jan 2002, Eric S. Raymond wrote:

> > > The underlying problem is that dmidecode needs access to kmem, and I
> > > can't assume that the person running my configurator will be root.
> >
> > But you can "su -c" (also sudo, I suppose).  If that person
> > doesn't have root, then building a kernel isn't going to do
> > them much good.
>
> We've been over this already.  No, the configurator user should *not*
> have to su at any point before actual kernel installation.  Bad
> practice, no doughnut.

Why bad practice?  Anyway, you can:

	if [ /proc/ -nt /var/run/dmidecode ]; then
		echo We need to run some code as root.
		echo -n Enter root\'s\
		su -c 'dmidecode > /var/run/dmidecode'
	fi

Which provides at least a way to have your config tool
work without having to bloat the initramfs.

Matthew.

Re: initramfs programs (was [RFC] klibc requirements)

[Eric S. Raymond]

Matthew Kirkwood <matthew@hairy.beasts.org>:
> > We've been over this already.  No, the configurator user should *not*
> > have to su at any point before actual kernel installation.  Bad
> > practice, no doughnut.
> 
> Why bad practice?  Anyway, you can:
> 
> 	if [ /proc/ -nt /var/run/dmidecode ]; then
> 		echo We need to run some code as root.
> 		echo -n Enter root\'s\
> 		su -c 'dmidecode > /var/run/dmidecode'
> 	fi
> 
> Which provides at least a way to have your config tool
> work without having to bloat the initramfs.

OK.  One more time.

The autoconfigurator is *not* mean to be run at boot time, or as root.

It is intended to be run by ordinary users, after system boot time.
This is so they can configure and experimentally build kernels without
incurring the "oops..." risks of going root.

Therefore, the above 'solution' is not acceptable.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

A man who has nothing which he is willing to fight for, nothing 
which he cares about more than he does about his personal safety, 
is a miserable creature who has no chance of being free, unless made 
and kept so by the exertions of better men than himself. 
	-- John Stuart Mill, writing on the U.S. Civil War in 1862

Re: initramfs programs (was [RFC] klibc requirements)

[H. Peter Anvin]

Followup to:  <20020109182902.A2804@thyrsus.com>
By author:    "Eric S. Raymond" <esr@thyrsus.com>
In newsgroup: linux.dev.kernel
> 
> OK.  One more time.
> 
> The autoconfigurator is *not* mean to be run at boot time, or as root.
> 
> It is intended to be run by ordinary users, after system boot time.
> This is so they can configure and experimentally build kernels without
> incurring the "oops..." risks of going root.
> 
> Therefore, the above 'solution' is not acceptable.
> 

If /var/run/dmidata [or whatever you call it] isn't available, put an
error box on the screen and say "complain to your distribution
vendor."

End of story.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: initramfs programs (was [RFC] klibc requirements)

[Matthew Kirkwood]

On Wed, 9 Jan 2002, Eric S. Raymond wrote:

> > 	if [ /proc/ -nt /var/run/dmidecode ]; then
> > 		echo We need to run some code as root.
> > 		echo -n Enter root\'s\
> > 		su -c 'dmidecode > /var/run/dmidecode'
> > 	fi

> The autoconfigurator is *not* mean to be run at boot time, or as root.

Under normal circumstances.

> It is intended to be run by ordinary users, after system boot time.
> This is so they can configure and experimentally build kernels without
> incurring the "oops..." risks of going root.

Then ship it in a separate package with initscripts.  Either
CML2 is well enough designed that the autoconfigurator will
not need to change as the kernel does, or all your
overengineering was for nought.

> Therefore, the above 'solution' is not acceptable.

To who?  It provides as way for your configurator to function
out-of-the-box until distributions choose whether it's worth
their while to support your scheme.

Matthew.

Re: initramfs programs (was [RFC] klibc requirements)

[Pavel Machek]

Hi!

> > > We've been over this already.  No, the configurator user should *not*
> > > have to su at any point before actual kernel installation.  Bad
> > > practice, no doughnut.
> > 
> > Why bad practice?  Anyway, you can:
> > 
> > 	if [ /proc/ -nt /var/run/dmidecode ]; then
> > 		echo We need to run some code as root.
> > 		echo -n Enter root\'s> > 		su -c 'dmidecode > /var/run/dmidecode'
> > 	fi
> > 
> > Which provides at least a way to have your config tool
> > work without having to bloat the initramfs.
> 
> OK.  One more time.
> 
> The autoconfigurator is *not* mean to be run at boot time, or as root.
> 
> It is intended to be run by ordinary users, after system boot time.
> This is so they can configure and experimentally build kernels without
> incurring the "oops..." risks of going root.
> 
> Therefore, the above 'solution' is not acceptable.

Therefore, autoconfigurator is not acceptable. Tada.

You can't go around saying "my autoconfigurator needs this so put it into
kernel". autoconfigurator is not important enough to touch kernel/initrd.
									Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: initramfs programs (was [RFC] klibc requirements)

[Oliver Xymoron]

On Wed, 9 Jan 2002, Eric S. Raymond wrote:

> greg k-h:
> >What does everyone else need/want there?
>
> dmidecode, so the init script can dump a DMI report in a known
> location such as /var/run/dmi.

No, this belongs in a distribution's init scripts. Initramfs is stuff that
is needed before mounting real root, not everything that must be done
before we have a login prompt.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

RE: initramfs programs (was [RFC] klibc requirements)

[Torrey Hoffman]

Tom Rini wrote:
> On Wed, Jan 09, 2002 at 09:54:28AM -0800, Torrey Hoffman wrote:
> 
> > The interesting thing that I currently do with initrd support is a
> > custom network-booted Linux installer for an embedded system. 
> > 
> > I'd like to be able to do this with initramfs too.  It needs:
> [snip]
> 
> Er, for this particular application, why would you use klibc, if
> existant?  The initramfs stuff could work with glibc, if you 
> didn't mind
> a big enough image, it sounds like.

You are correct, the size of glibc is not a major problem for me right 
now and that's what I'm using.  However, it is the largest thing in my
initrd, which goes across the net using tftp, and glibc just keeps 
getting bigger...  I'm thinking of switching to ulibc when I get time.

One of the neat things about moving a lot of this formerly in-kernel
boot stuff to userspace is that it will be easier to do interesting
customization of the boot process, without having to hack the kernel.

I'm sure that this will be used in lots of innovative ways that people
haven't even thought of yet.  

So, I guess what I'd like to see with the initramfs build system is a 
easy way to build little apps like sfdisk and mkreiserfs against the 
libc it (normally) uses and add them to the ramdisk.

I'm not so worried for myself, but for an overworked sysadmin trying 
to customize the boot it could end up being very confusing... 

Torrey

Re: initramfs programs (was [RFC] klibc requirements)

[Tom Rini]

On Wed, Jan 09, 2002 at 12:25:37PM -0800, Torrey Hoffman wrote:
 
> One of the neat things about moving a lot of this formerly in-kernel
> boot stuff to userspace is that it will be easier to do interesting
> customization of the boot process, without having to hack the kernel.

Right.  But in doing this we also don't want to bloat the very small
programs/functionality we currently hack directly in

> I'm sure that this will be used in lots of innovative ways that people
> haven't even thought of yet.  

Yes, but how many of these require a 'klibc' ?  One of the other tiny
libcs is probably a better bet for any sort of 'large' project.  IMHO,
the klibc stuff should be just big enough to support the 'normal' cases,
ie stuff we do now and related.

> So, I guess what I'd like to see with the initramfs build system is a 
> easy way to build little apps like sfdisk and mkreiserfs against the 
> libc it (normally) uses and add them to the ramdisk.

Well, occording to the spec file hpa posted, it's just a cpio(1)
archive, so anything is possible.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: initramfs programs (was [RFC] klibc requirements)

[Giacomo Catenazzi]

Matthew Kirkwood wrote:

> On Wed, 9 Jan 2002, Eric S. Raymond wrote:
> 
>>The autoconfigurator is *not* mean to be run at boot time, or as root.
>>
> 
> Under normal circumstances.
> 


???? Could you tell me about an 'anormal' circumstance that
need autoconfigurator at boot time ?


> 
>>It is intended to be run by ordinary users, after system boot time.
>>This is so they can configure and experimentally build kernels without
>>incurring the "oops..." risks of going root.
>>
> 
> Then ship it in a separate package with initscripts.  Either
> CML2 is well enough designed that the autoconfigurator will
> not need to change as the kernel does, or all your
> overengineering was for nought.


No problem. Autoconfigurator can live without important files.
I.e. no /proc/bus/{pci,usb}, autoconfigurator will ignore such
detections. (it would not say: no PCI cards, but unfortunatelly
it will find only few PCI cards (via /proc/{devices,misc}, if you
have luke)).

BTW: IMHO I can complete the detection, also with ISA cards,
before Eric will start including dmi.
How to handle a driver db with dmi strings and kernel configurations?
It seems to me to complex to try it. We need every possible machine
to extract data.

	giacomo

Re: initramfs programs (was [RFC] klibc requirements)

[Giacomo Catenazzi]

Dave Jones wrote:

> On Thu, 10 Jan 2002, Alan Cox wrote:
> 
>>We've also proved the DMI data is too unreliable to be used, so the entire
>>problem space is irrelevant
>>
> 
> That's not a problem, remember Eric volunteered to maintain the
> enormous black list 8-)
> 


Surelly I will not maintain the DMI table!
It is already difficult to maintain the database of CPU.
The newer CPUs have name stored directly in CPU and no more
in kernel :-(

(
This is a call for help: how to write a table
CPU - CONFIG_SYMBOL ?
Now I use Vendor/Name/Family/Stepping/, but
maybe with Vendor + flags (CPUID flags) the result
will be more correct?

Other suggestions?

	giacomo

 

Re: initramfs programs (was [RFC] klibc requirements)

[Dave Jones]

On Thu, 10 Jan 2002, Giacomo Catenazzi wrote:

> It is already difficult to maintain the database of CPU.
> The newer CPUs have name stored directly in CPU and no more
> in kernel :-(

It's worse than you think.
Distinguishing between XP and MP athlon for example needs
capability bit testing.  (And some XP's _are_ now multiprocessor
capable, just to confuse the issue some more), so relying on
the cpuid identity string isn't foolproof.
(Also, some implementations allow for this string to be changed,
 some folks have it set to "PenguinPowered" and the likes 8-)

> This is a call for help: how to write a table
> CPU - CONFIG_SYMBOL ?
> Now I use Vendor/Name/Family/Stepping/, but
> maybe with Vendor + flags (CPUID flags) the result
> will be more correct?

Asides from the above issues, multiple CPUs have the same
cpuid sometimes, meaning you have to check things like
cache size as well (though most (all?) of these should
end up with the same CONFIG_ option iirc, so this shouldn't
be an issue -- you should check to be sure though)

x86info's identify.c files should give you a fairly
comprehensive guide to the various types.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: initramfs programs (was [RFC] klibc requirements)

[H. Peter Anvin]

Dave Jones wrote:

> 
> It's worse than you think.
> Distinguishing between XP and MP athlon for example needs
> capability bit testing.  (And some XP's _are_ now multiprocessor
> capable, just to confuse the issue some more), so relying on
> the cpuid identity string isn't foolproof.
> (Also, some implementations allow for this string to be changed,
>  some folks have it set to "PenguinPowered" and the likes 8-)
> 


Sure, but if you do that you're *asking*, in a very literal way, for 
your CPU to misidentified.  In fact, a major reason for making this 
string modifiable is due to certain vendors who hard-code CPUID strings 
in their code.

> 
> Asides from the above issues, multiple CPUs have the same
> cpuid sometimes, meaning you have to check things like
> cache size as well (though most (all?) of these should
> end up with the same CONFIG_ option iirc, so this shouldn't
> be an issue -- you should check to be sure though)
> 


Why -- if it doesn't change anything, all you're doing is making it 
confusing when the next derivative appears.  Remember that we *do* need, 
  as much as possible, to be forward compatible with future CPUs.


> x86info's identify.c files should give you a fairly
> comprehensive guide to the various types.


	-hpa

Re: initramfs programs (was [RFC] klibc requirements)

[Dave Jones]

On Thu, 10 Jan 2002, H. Peter Anvin wrote:

> > (Also, some implementations allow for this string to be changed,
> >  some folks have it set to "PenguinPowered" and the likes 8-)
> Sure, but if you do that you're *asking*, in a very literal way, for
> your CPU to misidentified.  In fact, a major reason for making this
> string modifiable is due to certain vendors who hard-code CPUID strings
> in their code.

Absolutely, no argument from me there.

> > Asides from the above issues, multiple CPUs have the same
> > cpuid sometimes, meaning you have to check things like
> > cache size as well (though most (all?) of these should
> > end up with the same CONFIG_ option iirc, so this shouldn't
> > be an issue -- you should check to be sure though)
> Why -- if it doesn't change anything, all you're doing is making it
> confusing when the next derivative appears.  Remember that we *do* need,
>   as much as possible, to be forward compatible with future CPUs.

If every combination for a given cpuid translates to the same CONFIG_
option, fine. I'm just not sure from memory if thats the case or not.
The various Celeron/PIII's for example, some have SSE, some don't.
Assuming Intel cpuid xxx translates to CONFIG_PENTIUMIII would break if
thats the case.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: initramfs programs (was [RFC] klibc requirements)

[H. Peter Anvin]

Dave Jones wrote:

> 
> If every combination for a given cpuid translates to the same CONFIG_
> option, fine. I'm just not sure from memory if thats the case or not.
> The various Celeron/PIII's for example, some have SSE, some don't.
> Assuming Intel cpuid xxx translates to CONFIG_PENTIUMIII would break if
> thats the case.
> 


Right, which is why you should look at the SSE feature flag and nothing 
else.

	-hpa

Re: initramfs programs (was [RFC] klibc requirements)

[Dave Jones]

On Thu, 10 Jan 2002, H. Peter Anvin wrote:

> Right, which is why you should look at the SSE feature flag and nothing
> else.

Agreed.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: initramfs programs (was [RFC] klibc requirements)

[Alan Cox]

> Surelly I will not maintain the DMI table!

Quite

> This is a call for help: how to write a table
> CPU - CONFIG_SYMBOL ?
> Now I use Vendor/Name/Family/Stepping/, but
> maybe with Vendor + flags (CPUID flags) the result
> will be more correct?

You need the family/model information to get the right optimisations. Its
often not that the instruction set is different but that the cpu
implementation is different that determines the choice. With a couple of 
exceptions cpu type is actually not too important and accidentally using
486 will make little difference
 

Re: initramfs programs (was [RFC] klibc requirements)

[H. Peter Anvin]

Alan Cox wrote:

>>Surelly I will not maintain the DMI table!
>>
> 
> Quite
> 
> 
>>This is a call for help: how to write a table
>>CPU - CONFIG_SYMBOL ?
>>Now I use Vendor/Name/Family/Stepping/, but
>>maybe with Vendor + flags (CPUID flags) the result
>>will be more correct?
>>
> 
> You need the family/model information to get the right optimisations. Its
> often not that the instruction set is different but that the cpu
> implementation is different that determines the choice. With a couple of 
> exceptions cpu type is actually not too important and accidentally using
> 486 will make little difference
>  


Something I'd really like to see would be to split "optimize for..." and 
"run on..." for CPU type; just like gcc has -mmach= and -march=.

	-hpa