[RFC] klibc requirements

[RFC] klibc requirements

[Greg KH]

Hi all,

First off, I do not want to fork off yet another tiny libc
implementation, unless it is determined that we really need to do it.  I
posted the klibc implementation because I have found that in the past,
people can talk all they want, but the only way to actually get people
all riled up and start paying attention is to post code :)

Now that people are riled up, and want to talk about it, let's try to
describe the problem and see if any of the existing libc implementations
help solve them.  Here's what I see as a list of requirements for a
klibc like library that can be used by initramfs programs (please,
everyone else jump in here with their requirements too):
	- portable, runs on all platforms that the kernel currently
	  works on, but doesn't have to run on any non-Linux based OS.
	- tiny.  If we link statically it should be _small_.  Both
	  dietLibc and uClibc are good examples of the size goal.  We do
	  not need to support all of POSIX here, only what we really
	  need.
	- If we end up having a lot of different programs in initramfs,
	  a dynamic version of the library should be available.  This
	  shared library should be _small_ and only contain the symbols
	  that are needed by the programs to run.  This should be able
	  to be determined at build time.
	- It has to "not suck" :)  This is a lovely relative feeling
	  about the quality of the code base, ease at building the
	  library, ease at understanding the code, and responsiveness of
	  the developers of the library.

I don't think either dietHotplug or uClibc or glibc or any other
existing libc implementation meets these goals right now, right?

I had asked the dietLibc authors about the ability of tweaking their
library into something that resembles the above, but didn't get a
response.  Hence my post.  I would love to work with the authors of an
existing libc to build such a library, as I have other things I would
rather work on than a libc :)

Comments from the various libc authors?  Comments from other kernel
developers about requirements and goals they would like to see from such
a libc?

thanks,

greg k-h

Re: [RFC] klibc requirements

[Erik Andersen]

On Tue Jan 08, 2002 at 11:24:50AM -0800, Greg KH wrote:
> 	- portable, runs on all platforms that the kernel currently
> 	  works on, but doesn't have to run on any non-Linux based OS.

uClibc currently only runs on arm, i386, m68k, mips, powerpc, sh,
and v850, with full shared lib support on arm, i386, and powerpc.
Porting to a new arch typically involves writing just a few asm
files.  I don't bother with anything non-Linux...

> 	- tiny.  If we link statically it should be _small_.  Both
> 	  dietLibc and uClibc are good examples of the size goal.  We do
> 	  not need to support all of POSIX here, only what we really
> 	  need.

uClibc does tiny static stuff just fine.  Though these days,
uClibc passes POSIX conformance tests (with some exceptions for
stupid things which have been omitted).

> 	- If we end up having a lot of different programs in initramfs,
> 	  a dynamic version of the library should be available.  This
> 	  shared library should be _small_ and only contain the symbols
> 	  that are needed by the programs to run.  This should be able
> 	  to be determined at build time.

uClibc currently has shared lib support only on arm, i386, and powerpc.
There is a library reducer script (to include only needed
symbols) in uClibc/extra/libstrip/libstrip which does a fine job.

I personally think busybox + uClibc are ideal for building
initramfs stuff, since you can build everything you need into 
a single multi-call binary (eliminating the need for shared libs
in most cases).

> 	- It has to "not suck" :)  This is a lovely relative feeling
> 	  about the quality of the code base, ease at building the
> 	  library, ease at understanding the code, and responsiveness of
> 	  the developers of the library.

Well, I do my best.  :)

As for ease of building the library, most people can just copy
the Config file into place and compile.  I build a fake
gcc-wrapper toolchain, so using the library is as simple as 
'make install' then running 'CC=/usr/i386-linux-uclibc/bin/gcc make'

I think one big advantage uClibc has in the "not suck"
department, is that it uses the header files from glibc 2.2.4
(with minor changes), meaning that for most apps, if it compiles
with glibc it will compile with uClibc.

> response.  Hence my post.  I would love to work with the authors of an
> existing libc to build such a library, as I have other things I would
> rather work on than a libc :)

Ok.  here I am.   Work with me.  :) 

> Comments from the various libc authors?  Comments from other kernel
> developers about requirements and goals they would like to see from such
> a libc?

If folks have requirements and goals, I'm very interested in
hearing them...

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] klibc requirements

[Felix von Leitner]

Thus spake Greg KH (greg@kroah.com):
> First off, I do not want to fork off yet another tiny libc
> implementation,

It's good to hear that.

> Now that people are riled up, and want to talk about it, let's try to
> describe the problem and see if any of the existing libc implementations
> help solve them.  Here's what I see as a list of requirements for a
> klibc like library that can be used by initramfs programs (please,
> everyone else jump in here with their requirements too):

My understanding of what "initramfs programs" actually means is vague at
best.  Are these just programs that are intended to work in an initial
ram disk?  Or is it a special collection that is included in the kernel
distribution?

I don't see why we would need to include those programs with the kernel.
The kernel tries hard to provide backwards compatibility to old versions
of syscalls that have changed over the years.  That's why we _have_ a
standard.  Also, we don't ship glibc with the kernel sources, and we
didn't when our libc was Linux specific.

If we follow that argumentation and are talking here about programs that
aren't included in the kernel, why demand a specific libc for them at
all?  Of course glibc is out of the question, but both the kernel API
_and_ the libc API are standardized.  It does not make sense to write
code that demands a specific libc if it can be avoided.  If you need
any special syscall supported that is not yet part of the diet libc or
uClibc, Eric and I will probably be glad to add support for it.

> 	- portable, runs on all platforms that the kernel currently
> 	  works on, but doesn't have to run on any non-Linux based OS.
> 	- tiny.  If we link statically it should be _small_.  Both
> 	  dietLibc and uClibc are good examples of the size goal.  We do
> 	  not need to support all of POSIX here, only what we really
> 	  need.

When you link statically, it does not matter whether the libc also
supports the rest of POSIX.  The size of libc.a does not matter.  It
only matters which parts are referenced.  Since we are talking about new
software specifically written for Linux and with the goal to be small,
there are no legacy requirements to cater to.  We can write code without
printf and stdio, for example.  Also, we probably don't need regular
expressions or DNS.  Those are the big space hogs when linking
statically against a libc.  In the diet libc, all of the above are very
small, but avoiding them in the first place is better then optimizing
them for small size.

And if we don't use all that bloaty code, it does not matter if we use
diet libc, uClibc or tomorrow's next great small libc.

> 	- If we end up having a lot of different programs in initramfs,
> 	  a dynamic version of the library should be available.  This
> 	  shared library should be _small_ and only contain the symbols
> 	  that are needed by the programs to run.  This should be able
> 	  to be determined at build time.

This may look like a good idea, but dynamic linking should be avoided.
Trust me on this.  While it is possible to squeeze the dynamic loader
down to below 10k, 10k really is a lot of code.  And empty program with
the diet libc is way below 1k on x86.  So to reap the benefit of dynamic
linking, you would need a lot of programs.  Also please note that -fPIC
makes code larger.  And we need to keep symbols around, which makes up a
substantial part of the shared diet libc.

How many programs are we talking about here?  And what should they do?

You can make a libc.so that only contains the superset of the symbols
used by your initramdisk programs.  But then you have to remake the
libc.so if you add another program, so that's not very flexible.

What I want to say here is: don't make rash decisions.  Profile, don't
speculate.  Also, don't underestimate the amount of time dynamic linking
takes.  It may seem insignificant on today's monster CPUs, but it isn't.
I optimized my boot sequence to below 3 seconds from the start of init
to the shell prompt on tty1 (the other ttys are spawned immediately).
All the dynamic linking and shell script overhead adds up quickly.

Please read http://www.fefe.de/dietlibc/talk.pdf for some more info
(slightly less than 100k).

> 	- It has to "not suck" :)  This is a lovely relative feeling
> 	  about the quality of the code base, ease at building the
> 	  library, ease at understanding the code, and responsiveness of
> 	  the developers of the library.

While I am of course sure to be the libc developer that sucks least ;),
I again would advise not to rush to any hasty decisions.  Don't bind
yourself to a libc unless you have to or there are any benefits to be
gained.  I fail to see the benefits.  Currently, the diet libc produces
the smallest binaries by a healthy margin, but binding yourself to the
diet libc does not gain you anything over using APIs that are portable
across all libcs.  In fact, I avoided offering any diet libc specific
APIs at all (with the exception of write12.h).

> I had asked the dietLibc authors about the ability of tweaking their
> library into something that resembles the above, but didn't get a
> response.

Huh?
What email are you talking about here?

Your initial email asked whether you could use parts of the diet libc
for diethotplug and for tweaking, and I said OK.

> Hence my post.  I would love to work with the authors of an
> existing libc to build such a library, as I have other things I would
> rather work on than a libc :)

Before doing any real work, we need to get the specification straight.
What exactly do we need?  The initrd stuff is too vague for my taste.
How many programs do we want to have?  What should those programs do?

> Comments from the various libc authors?  Comments from other kernel
> developers about requirements and goals they would like to see from such
> a libc?

I think you need to ask initrd users.
My understanding was that people want to use the IP autoconfiguration
stuff from the kernel to initrd.  Is that still so?  What other programs
are needed?

Felix

initramfs programs (was [RFC] klibc requirements)

[Greg KH]

On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> 
> How many programs are we talking about here?  And what should they do?

Very good question that we should probably answer first (I'll follow up
to your other points in a separate message).

Here's what I want to have in my initramfs:
	- /sbin/hotplug
	- /sbin/modprobe
	- modules.dep (needed for modprobe, but is a text file)

What does everyone else need/want there?

greg k-h

Re: initramfs programs (was [RFC] klibc requirements)

[Greg KH]

On Tue, Jan 08, 2002 at 08:34:47PM -0800, Greg KH wrote:
> 
> Here's what I want to have in my initramfs:
> 	- /sbin/hotplug
> 	- /sbin/modprobe
> 	- modules.dep (needed for modprobe, but is a text file)

Forgot the modules themselves.  That would be helpful...

greg k-h

Re: initramfs programs (was [RFC] klibc requirements)

[H. Peter Anvin]

Followup to:  <20020109061037.GB18024@kroah.com>
By author:    Greg KH <greg@kroah.com>
In newsgroup: linux.dev.kernel
>
> On Tue, Jan 08, 2002 at 08:34:47PM -0800, Greg KH wrote:
> > 
> > Here's what I want to have in my initramfs:
> > 	- /sbin/hotplug
> > 	- /sbin/modprobe
> > 	- modules.dep (needed for modprobe, but is a text file)
> 
> Forgot the modules themselves.  That would be helpful...
> 

Sure, but those are data files as far as the (k)libc is concerned.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: initramfs programs (was [RFC] klibc requirements)

[Kai Germaschewski]

On Tue, 8 Jan 2002, Greg KH wrote:

> On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> > 
> > How many programs are we talking about here?  And what should they do?
> 
> Very good question that we should probably answer first (I'll follow up
> to your other points in a separate message).
> 
> Here's what I want to have in my initramfs:
> 	- /sbin/hotplug
> 	- /sbin/modprobe
> 	- modules.dep (needed for modprobe, but is a text file)
> 
> What does everyone else need/want there?

Provision to mount the real root, i.e.
- mount

Mount the real root via NFS
- ifconfig/dhcpcd/pump
- portmap?

Or did I get that wrong?

--Kai

Re: initramfs programs (was [RFC] klibc requirements)

[Erik Andersen]

On Tue Jan 08, 2002 at 08:34:47PM -0800, Greg KH wrote:
> On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> > 
> > How many programs are we talking about here?  And what should they do?
> 
> Very good question that we should probably answer first (I'll follow up
> to your other points in a separate message).
> 
> Here's what I want to have in my initramfs:
> 	- /sbin/hotplug
> 	- /sbin/modprobe
> 	- modules.dep (needed for modprobe, but is a text file)
> 
> What does everyone else need/want there?

I think you want a staticly linked multi-call binary, a 
trivial shell, and some kernel modules...
    http://www.uwsg.indiana.edu/hypermail/linux/kernel/0112.2/0681.html

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] klibc requirements

[Greg KH]

On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> My understanding of what "initramfs programs" actually means is vague at
> best.  Are these just programs that are intended to work in an initial
> ram disk?  Or is it a special collection that is included in the kernel
> distribution?

I don't know if they will be included in the kernel distribution or not.
But they will be part of the kernel build process, if only to copy them
to the ramfs image which is then added to the kernel image.

The dietHotplug program will need to be built each time against the
kernel that it is going to be bundled up with (that's how it gets the
size improvements.)

Maybe in the end, specialized programs like dietHotplug will become part
of the kernel source tree, due to them being so tightly bound to it.
Does anyone else have any thoughts about this?

> I don't see why we would need to include those programs with the kernel.
> The kernel tries hard to provide backwards compatibility to old versions
> of syscalls that have changed over the years.  That's why we _have_ a
> standard.  Also, we don't ship glibc with the kernel sources, and we
> didn't when our libc was Linux specific.

These programs are basically the movement of a lot of existing in-kernel
code, to userspace.  And as they need to be bundled up within the kernel
image itself, they don't need to have to run on any kernel but that one.

> If we follow that argumentation and are talking here about programs that
> aren't included in the kernel, why demand a specific libc for them at
> all?  Of course glibc is out of the question, but both the kernel API
> _and_ the libc API are standardized.  It does not make sense to write
> code that demands a specific libc if it can be avoided.  If you need
> any special syscall supported that is not yet part of the diet libc or
> uClibc, Eric and I will probably be glad to add support for it.

Great!  We are needing a small libc for these userspace programs.  A
limited number of libc functions, and a clean syscall interface.

> > 	- portable, runs on all platforms that the kernel currently
> > 	  works on, but doesn't have to run on any non-Linux based OS.

You didn't address this.  What are the future plans of porting dietLibc
to the platforms that are not currently supported by it (but are by
Linux)?

> When you link statically, it does not matter whether the libc also
> supports the rest of POSIX.  The size of libc.a does not matter.  It
> only matters which parts are referenced.  Since we are talking about new
> software specifically written for Linux and with the goal to be small,
> there are no legacy requirements to cater to.  We can write code without
> printf and stdio, for example.  Also, we probably don't need regular
> expressions or DNS.  Those are the big space hogs when linking
> statically against a libc.  In the diet libc, all of the above are very
> small, but avoiding them in the first place is better then optimizing
> them for small size.
> 
> And if we don't use all that bloaty code, it does not matter if we use
> diet libc, uClibc or tomorrow's next great small libc.

Agreed.

<good argument against dynamic linking snipped>

You don't have to convince me about not really wanting dynamic linking.
As I do not know how many programs other people want in their initramfs,
or the size of these programs, I don't know if the size of a loader and
the symbol tables will outweigh the size of statically linking all of
the individual programs themselves.

It's just something to be aware of that we might have to do in the
future, that's all.  And it's nice to see that dietLibc supports this :)

> > I had asked the dietLibc authors about the ability of tweaking their
> > library into something that resembles the above, but didn't get a
> > response.
> 
> Huh?
> What email are you talking about here?

It was sent and received on the dietlibc mailing list on Jan 04, 2001.
It looks like the mailing list archive for the mailing list doesn't have
any messages for Jan, 2001, otherwise I would point you at it.  I can
forward it to you offline if you want me to.

> Before doing any real work, we need to get the specification straight.
> What exactly do we need?  The initrd stuff is too vague for my taste.
> How many programs do we want to have?  What should those programs do?
> 
> I think you need to ask initrd users.
> My understanding was that people want to use the IP autoconfiguration
> stuff from the kernel to initrd.  Is that still so?  What other programs
> are needed?

These are great questions that need to be answered.  I've started a
different topic just for it.

thanks,

greg k-h

Re: [RFC] klibc requirements

[H. Peter Anvin]

Followup to:  <20020109045109.GA17776@kroah.com>
By author:    Greg KH <greg@kroah.com>
In newsgroup: linux.dev.kernel
>
> On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> > My understanding of what "initramfs programs" actually means is vague at
> > best.  Are these just programs that are intended to work in an initial
> > ram disk?  Or is it a special collection that is included in the kernel
> > distribution?
> 
> I don't know if they will be included in the kernel distribution or not.
> But they will be part of the kernel build process, if only to copy them
> to the ramfs image which is then added to the kernel image.
> 

Why should it be included in the kernel image?  That's not the current
plan, as far as I know.  It should be a separate file or set of files
loaded by the bootloader (using an enhanced initrd protocol backward
compatible with old bootloaders.)  Of course, it might be convenient
to have them come out of the same source distribution, but that's
really an unrelated issue.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: [RFC] klibc requirements

[Greg KH]

On Tue, Jan 08, 2002 at 09:01:16PM -0800, H. Peter Anvin wrote:
> 
> Why should it be included in the kernel image?  That's not the current
> plan, as far as I know.  It should be a separate file or set of files
> loaded by the bootloader (using an enhanced initrd protocol backward
> compatible with old bootloaders.)

Hm, missed those messages.  I remember talk of adding the initramfs
image to the kernel image itself, which keeps from having to change any
bootloaders.  But if this has changed, that's ok with me.

Is any of this written down all in one place?

> Of course, it might be convenient
> to have them come out of the same source distribution, but that's
> really an unrelated issue.

If the files contain kernel modules, they need to come out of the same
source distribution :)

thanks,

greg k-h

Re: [RFC] klibc requirements

[H. Peter Anvin]

Followup to:  <20020109060951.GA18024@kroah.com>
By author:    Greg KH <greg@kroah.com>
In newsgroup: linux.dev.kernel
>
> On Tue, Jan 08, 2002 at 09:01:16PM -0800, H. Peter Anvin wrote:
> > 
> > Why should it be included in the kernel image?  That's not the current
> > plan, as far as I know.  It should be a separate file or set of files
> > loaded by the bootloader (using an enhanced initrd protocol backward
> > compatible with old bootloaders.)
> 
> Hm, missed those messages.  I remember talk of adding the initramfs
> image to the kernel image itself, which keeps from having to change any
> bootloaders.  But if this has changed, that's ok with me.
> 
> Is any of this written down all in one place?
> 

This is a *draft* of the specification; I was hoping to get Viro to
comment on it before releasing it publically -- there still may need
to be some tightening of the definition especially with respect to the
definition of alignment, but the "big picture" should be correct:


		       initramfs buffer format
		       -----------------------

		       Al Viro, H. Peter Anvin
		      Last revision: 2002-01-03

  ** DRAFT ** DRAFT ** DRAFT ** DRAFT ** DRAFT ** DRAFT ** DRAFT **

Starting with kernel 2.5.x, the old "initial ramdisk" protocol is
getting {replaced/complemented} with the new "initial ramfs"
(initramfs) protocol.  The initramfs contents is passed using the same
memory buffer protocol used by the initrd protocol, but the contents
is different.  The initramfs buffer contains an archive which is
expanded into a ramfs filesystem; this document details the format of
the initramfs buffer format.

The initramfs buffer format is based around the "newc" CPIO format,
and can be created with the cpio(1) utility.  The cpio archive can be
compressed using gzip(1).  The simplest form of the initramfs buffer
is thus a single .cpio.gz file.

The full format of the initramfs buffer is defined by the following
grammar, where:
	*	is used to indicate "0 or more occurrences of"
	(|)	indicates alternatives
	+	indicates concatenation
	GZIP()	indicates the gzip(1) of the operand
	PAD(n)	means padding with null bytes to an n-byte boundary
	[QUESTION: is the padding relative to the start of the
	previous header, or is it an absolute address?  Is it at all
	legal to have a header start on a non-multiple of 4?]

	initramfs  := ("\0" | cpio_archive | cpio_gzip_archive)*

	cpio_gzip_archive := GZIP(cpio_archive)

	cpio_archive := cpio_file* + (<nothing> | cpio_trailer)

	cpio_file := cpio_header + filename + "\0" + PAD(4) + data + PAD(4)

	cpio_trailer := cpio_header + "TRAILER!!!\0" + PAD(4)


In human terms, the initramfs buffer contains a collection of
compressed and/or uncompressed cpio archives (in the "newc" format);
arbitrary amounts zero bytes (for padding) can be added between
members.

The cpio "TRAILER!!!" entry (cpio end of file) is optional, but is not
ignored; see "handling of hard links" below.

The structure of the cpio_header is as follows (all 8-byte entries
contain 32-bit hexadecimal ASCII numbers):

Field name    Field size	 Meaning
c_magic	      6 bytes		 The string "070701" or "070702"
c_ino	      8 bytes		 File inode number
c_mode	      8 bytes		 File mode and permissions
c_uid	      8 bytes		 File uid
c_gid	      8 bytes		 File gid
c_nlink	      8 bytes		 Number of links
c_mtime	      8 bytes		 Modification time
c_filesize    8 bytes		 Size of data field
c_maj	      8 bytes		 Major part of file device number
c_min	      8 bytes		 Minor part of file device number
c_rmaj	      8 bytes		 Major part of device node reference
c_rmin	      8 bytes		 Minor part of device node reference
c_namesize    8 bytes		 Length of filename, including final \0
c_chksum      8 bytes		 CRC of data field if c_magic is 070702

The c_mode field matches the contents of st_mode returned by stat(2)
on Linux, and encodes the file type and file permissions.

The c_filesize should be zero for any non-regular file.

If the filename is "TRAILER!!!" this is actually an end-of-file
marker; the c_filesize for an end-of-file marker must be zero.


*** Handling of hard links

When a nondirectory with c_nlink > 1 is seen, the (c_maj,c_min,c_ino)
tuple is looked up in a tuple buffer.  If not found, it is entered in
the tuple buffer and the entry is created as usual; if found, a hard
link rather than a second copy of the file is created.  It is not
necessary (but permitted) to include a second copy of the file
contents; if the file contents is not included, the c_filesize field
should be set to zero to indicate no data section follows.

When a "TRAILER!!!" end-of-file marker is seen, the tuple buffer is
reset.  This permits archives which are generated independently to be
concatenated.

To combine file data from different sources (without having to
regenerate the (c_maj,c_min,c_ino) fields), therefore, either one of
the following techniques can be used:

a) Separate the different file data sources with a "TRAILER!!!"
   end-of-file marker, or

b) Make sure c_nlink == 1 for all nondirectory entries.

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: [RFC] klibc requirements

[Rusty Russell]

On 8 Jan 2002 23:26:46 -0800
"H. Peter Anvin" <hpa@zytor.com> wrote:
> *** Handling of hard links
> 
> When a nondirectory with c_nlink > 1 is seen, the (c_maj,c_min,c_ino)
> tuple is looked up in a tuple buffer.  If not found, it is entered in
> the tuple buffer and the entry is created as usual; if found, a hard
> link rather than a second copy of the file is created.  It is not

HPA,
	gnu cpio (v 2.4.2) actually puts the contents in the *last*
entry, for hardlinks in "newc" format.  This probably means you should
specify that if it's a found tuple, and c_filesize is non-zero,
overwrite the contents of the file.

Cheers!
Rusty.
-- 
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: [RFC] klibc requirements

[H. Peter Anvin]

Followup to:  <20020110173029.7616f752.rusty@rustcorp.com.au>
By author:    Rusty Russell <rusty@rustcorp.com.au>
In newsgroup: linux.dev.kernel
> 
> HPA,
> 	gnu cpio (v 2.4.2) actually puts the contents in the *last*
> entry, for hardlinks in "newc" format.  This probably means you should
> specify that if it's a found tuple, and c_filesize is non-zero,
> overwrite the contents of the file.
> 

Check.  I must admit this greatly surprises me (since it means that it
needs to do more than one pass), but it certainly isn't making the
spec any more difficult.  As I said, it's a draft.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: [RFC] klibc requirements

[Matthias Kilian]

On 8 Jan 2002, H. Peter Anvin wrote:

> 		       initramfs buffer format
[...]
> compressed using gzip(1).  The simplest form of the initramfs buffer
> is thus a single .cpio.gz file.

FYI: I've already implemented this *simplest* form for .tar.gz files. A
patch can be found here:

http://www.escape.de/users/outback/linux/patch-2.4.17-inittar.gz

(I could also prepare a patch against 2.5.x)

So, before starting to implement the initramfs stuff, may be you should
have a look on my version.

Changing it from .tar to .cpio (if required) shouldn't be too difficult.

Bye,
	Kili

Re: [RFC] klibc requirements

[H. Peter Anvin]

Matthias Kilian wrote:

> 
> FYI: I've already implemented this *simplest* form for .tar.gz files. A
> patch can be found here:
> 
> http://www.escape.de/users/outback/linux/patch-2.4.17-inittar.gz
> 
> (I could also prepare a patch against 2.5.x)
> 
> So, before starting to implement the initramfs stuff, may be you should
> have a look on my version.
> 


I think Al has already implemented it.

	-hpa

Re: [RFC] klibc requirements

[Felix von Leitner]

Thus spake Greg KH (greg@kroah.com):
> > > 	- portable, runs on all platforms that the kernel currently
> > > 	  works on, but doesn't have to run on any non-Linux based OS.
> You didn't address this.  What are the future plans of porting dietLibc
> to the platforms that are not currently supported by it (but are by
> Linux)?

We will attempt to port the diet libc to every Linux platform that we
have access to.  I won't go out and buy hardware to port my software to,
so if anyone needs the diet libc ported somewhere, I need an account on
a test box somewhere.

We are currently missing sh-linux, ia64-linux and m68k-linux and
probably a few architectures that I have never heard of ;)

> It was sent and received on the dietlibc mailing list on Jan 04, 2001.
> It looks like the mailing list archive for the mailing list doesn't have
> any messages for Jan, 2001, otherwise I would point you at it.  I can
> forward it to you offline if you want me to.

Please do so!

Felix

Re: [RFC] klibc requirements

[Lars Brinkhoff]

Felix von Leitner <felix-dietlibc@fefe.de> writes:
> Thus spake Greg KH (greg@kroah.com):
> > > > 	- portable, runs on all platforms that the kernel currently
> > > > 	  works on, but doesn't have to run on any non-Linux based OS.
> > What are the future plans of porting dietLibc to the platforms
> > that are not currently supported by it (but are by Linux)?
> We will attempt to port the diet libc to every Linux platform that
> we have access to.  We are currently missing sh-linux, ia64-linux
> and m68k-linux and probably a few architectures that I have never
> heard of ;)

For the record: cris, parisc, s390, vax, x86-64.

-- 
Lars Brinkhoff          http://lars.nocrew.org/     Linux, GCC, PDP-10
Brinkhoff Consulting    http://www.brinkhoff.se/    programming

Re: [RFC] klibc requirements

[Erik Andersen]

On Wed Jan 09, 2002 at 03:15:14PM +0100, Felix von Leitner wrote:
> Thus spake Greg KH (greg@kroah.com):
> > > > 	- portable, runs on all platforms that the kernel currently
> > > > 	  works on, but doesn't have to run on any non-Linux based OS.
> > You didn't address this.  What are the future plans of porting dietLibc
> > to the platforms that are not currently supported by it (but are by
> > Linux)?
> 
> We will attempt to port the diet libc to every Linux platform that we
> have access to.  I won't go out and buy hardware to port my software to,
> so if anyone needs the diet libc ported somewhere, I need an account on
> a test box somewhere.
> 
> We are currently missing sh-linux, ia64-linux and m68k-linux and
> probably a few architectures that I have never heard of ;)

uClibc runs on both sh and ,68k, so the only arch missing 
coverage with either dietlibc or uClibc seems to be ia64.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: initramfs programs (was [RFC] klibc requirements)

[Anton Altaparmakov]

At 04:34 09/01/02, Greg KH wrote:
>On Wed, Jan 09, 2002 at 05:23:31AM +0100, Felix von Leitner wrote:
> >
> > How many programs are we talking about here?  And what should they do?
>
>Very good question that we should probably answer first (I'll follow up
>to your other points in a separate message).
>
>Here's what I want to have in my initramfs:
>         - /sbin/hotplug
>         - /sbin/modprobe
>         - modules.dep (needed for modprobe, but is a text file)
>
>What does everyone else need/want there?

It is planned to move partition discovery to userspace which would then 
instruct the kernel of the positions of various partitions.

The program(s) to do this will need to be in pretty much everyones initramfs...

Best regards,

         Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: initramfs programs (was [RFC] klibc requirements)

[Pavel Machek]

Hi!

> >> How many programs are we talking about here?  And what should they do?
> >
> >Very good question that we should probably answer first (I'll follow up
> >to your other points in a separate message).
> >
> >Here's what I want to have in my initramfs:
> >        - /sbin/hotplug
> >        - /sbin/modprobe
> >        - modules.dep (needed for modprobe, but is a text file)
> >
> >What does everyone else need/want there?
> 
> It is planned to move partition discovery to userspace which would then 
> instruct the kernel of the positions of various partitions.

Hmm, and when you insert PCMCIA harddrive, what happens?

-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.

Re: initramfs programs (was [RFC] klibc requirements)

[Patrick Mochel]

On Wed, 9 Jan 2002, Pavel Machek wrote:

> Hi!
>
> > >> How many programs are we talking about here?  And what should they do?
> > >
> > >Very good question that we should probably answer first (I'll follow up
> > >to your other points in a separate message).
> > >
> > >Here's what I want to have in my initramfs:
> > >        - /sbin/hotplug
> > >        - /sbin/modprobe
> > >        - modules.dep (needed for modprobe, but is a text file)
> > >
> > >What does everyone else need/want there?
> >
> > It is planned to move partition discovery to userspace which would then
> > instruct the kernel of the positions of various partitions.
>
> Hmm, and when you insert PCMCIA harddrive, what happens?

If you're booting from that hard drive, you make sure you have the modules
to talk to the drive. In general, you only need to modules to mount the
real root partition, right?

Will we need cardmgr in the future, or will be able to get away with
/sbin/hotplug?

	-pat

Re: initramfs programs (was [RFC] klibc requirements)

[Greg KH]

On Wed, Jan 09, 2002 at 08:04:44AM -0800, Patrick Mochel wrote:
> 
> Will we need cardmgr in the future, or will be able to get away with
> /sbin/hotplug?

Hopefully only /sbin/hotplug if the pcmcia port to the kernel is ever
finished (some people were working on it.)  If not, then we will need
cardmgr for some cards.

greg k-h

Re: initramfs programs (was [RFC] klibc requirements)

[Pavel Machek]

Hi!

> > > >> How many programs are we talking about here?  And what should they do?
> > > >
> > > >Very good question that we should probably answer first (I'll follow up
> > > >to your other points in a separate message).
> > > >
> > > >Here's what I want to have in my initramfs:
> > > >        - /sbin/hotplug
> > > >        - /sbin/modprobe
> > > >        - modules.dep (needed for modprobe, but is a text file)
> > > >
> > > >What does everyone else need/want there?
> > >
> > > It is planned to move partition discovery to userspace which would then
> > > instruct the kernel of the positions of various partitions.
> >
> > Hmm, and when you insert PCMCIA harddrive, what happens?
> 
> If you're booting from that hard drive, you make sure you have the modules
> to talk to the drive. In general, you only need to modules to mount the
> real root partition, right?

But that means that I'll need partition discovery code twice. Once on
initrd, and once on my root, because if I insert PCMCIA harddrive on
runtime, I'll need same detection.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.

Re: initramfs programs (was [RFC] klibc requirements)

[Andreas Ferber]

On Wed, Jan 09, 2002 at 05:29:51PM +0100, Pavel Machek wrote:
> 
> But that means that I'll need partition discovery code twice. Once on
> initrd, and once on my root, because if I insert PCMCIA harddrive on
> runtime, I'll need same detection.

Where is the problem? Generally I think you will have /every/ piece of
code on the initramfs also somewhere on your normal fs, to build the
initramfs image from. Also, who says that you have to abandon your
initramfs after the switch to the real root fs?

Andreas
-- 
       Andreas Ferber - dev/consulting GmbH - Bielefeld, FRG
     ---------------------------------------------------------
         +49 521 1365800 - af@devcon.net - www.devcon.net

Re: initramfs programs (was [RFC] klibc requirements)

[Alex Bligh - linux-kernel]

>> Here's what I want to have in my initramfs:
>>         - /sbin/hotplug
>>         - /sbin/modprobe
>>         - modules.dep (needed for modprobe, but is a text file)
>>
>> What does everyone else need/want there?
>
> It is planned to move partition discovery to userspace which would then
> instruct the kernel of the positions of various partitions.
>
> The program(s) to do this will need to be in pretty much everyones
> initramfs...

What with mounting root via NFS, hence having to set up
IP et al, mounting various different
partition types, avoiding the kludge of fsck etc.,
being able to recover from a corrupted root, you
might as well just cpio up your /sbin and stick
that in, and be able to run single user mode without
a 'normal' root. <FX: ducks & runs>

seriously point: ls /sbin gives a /maximum/ range I'd
have thought.

--
Alex Bligh

Re: initramfs programs (was [RFC] klibc requirements)

[Anton Altaparmakov]

At 21:15 09/01/2002, Alex Bligh - linux-kernel wrote:
>>>Here's what I want to have in my initramfs:
>>>         - /sbin/hotplug
>>>         - /sbin/modprobe
>>>         - modules.dep (needed for modprobe, but is a text file)
>>>
>>>What does everyone else need/want there?
>>
>>It is planned to move partition discovery to userspace which would then
>>instruct the kernel of the positions of various partitions.
>>
>>The program(s) to do this will need to be in pretty much everyones
>>initramfs...
>
>What with mounting root via NFS, hence having to set up
>IP et al, mounting various different
>partition types, avoiding the kludge of fsck etc.,
>being able to recover from a corrupted root, you
>might as well just cpio up your /sbin and stick
>that in, and be able to run single user mode without
>a 'normal' root. <FX: ducks & runs>
>
>seriously point: ls /sbin gives a /maximum/ range I'd
>have thought.

Partition discovery is currently done within the kernel itself. The code 
will effectively 'just' move out into user space. As such it is not present 
in /sbin now but it will be in initramfs. The same is true for various 
other code I can imagine moving out of kernel mode into initramfs...

Best regards,

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: initramfs programs (was [RFC] klibc requirements)

[Greg KH]

On Wed, Jan 09, 2002 at 09:34:34PM +0000, Anton Altaparmakov wrote:
> Partition discovery is currently done within the kernel itself. The code 
> will effectively 'just' move out into user space. As such it is not present 
> in /sbin now but it will be in initramfs. The same is true for various 
> other code I can imagine moving out of kernel mode into initramfs...

For this code, I can see it staying in the kernel source tree, and being
built as part of the kernel build process, right?

greg k-h

Re: initramfs programs (was [RFC] klibc requirements)

[Anton Altaparmakov]

At 21:40 09/01/2002, Greg KH wrote:
>On Wed, Jan 09, 2002 at 09:34:34PM +0000, Anton Altaparmakov wrote:
> > Partition discovery is currently done within the kernel itself. The code
> > will effectively 'just' move out into user space. As such it is not 
> present
> > in /sbin now but it will be in initramfs. The same is true for various
> > other code I can imagine moving out of kernel mode into initramfs...
>
>For this code, I can see it staying in the kernel source tree, and being
>built as part of the kernel build process, right?

I would think that is a good idea but I am not sure that is what is planned 
/ will happen. Keeping it outside would have the advantage that a newer 
partition recognizer (or whatever other code) can be applied to any 
existing kernel version (that supports initramfs).

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: initramfs programs (was [RFC] klibc requirements)

[Andreas Ferber]

On Wed, Jan 09, 2002 at 09:55:34PM +0000, Anton Altaparmakov wrote:
> 
> I would think that is a good idea but I am not sure that is what is planned 
> / will happen. Keeping it outside would have the advantage that a newer 
> partition recognizer (or whatever other code) can be applied to any 
> existing kernel version (that supports initramfs).

This could be done anyway: just replace the initramfs image built by 
the kernel build with anotherone built from another source tree. It
would be helpful though if the tools were distributed both standalone
and included into the kernel tree.

Andreas
-- 
       Andreas Ferber - dev/consulting GmbH - Bielefeld, FRG
     ---------------------------------------------------------
         +49 521 1365800 - af@devcon.net - www.devcon.net

Re: initramfs programs (was [RFC] klibc requirements)

[Tom Rini]

On Wed, Jan 09, 2002 at 11:15:28PM +0100, Andreas Ferber wrote:
> On Wed, Jan 09, 2002 at 09:55:34PM +0000, Anton Altaparmakov wrote:
> > 
> > I would think that is a good idea but I am not sure that is what is planned 
> > / will happen. Keeping it outside would have the advantage that a newer 
> > partition recognizer (or whatever other code) can be applied to any 
> > existing kernel version (that supports initramfs).
> 
> This could be done anyway: just replace the initramfs image built by 
> the kernel build with anotherone built from another source tree. It
> would be helpful though if the tools were distributed both standalone
> and included into the kernel tree.

If the kernel is going to build an initramfs option, it also needs a way
to be given one.  The issue I'm thinking of is I know of a few platforms
where the initramfs archive will have to be part of the 'zImage' file
(much like they do for ramdisks now).

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: initramfs programs (was [RFC] klibc requirements)

[Andreas Ferber]

On Wed, Jan 09, 2002 at 05:25:07PM -0700, Tom Rini wrote:
> > 
> > This could be done anyway: just replace the initramfs image built by 
> > the kernel build with anotherone built from another source tree. It
> > would be helpful though if the tools were distributed both standalone
> > and included into the kernel tree.
> If the kernel is going to build an initramfs option, it also needs a way
> to be given one.  The issue I'm thinking of is I know of a few platforms
> where the initramfs archive will have to be part of the 'zImage' file
> (much like they do for ramdisks now).

Append it to the zImage and let the kernel look for it there. Plus add
a tool to util-linux (or maybe an option to rdev?) to let you replace
the embedded initramfs in a {,b}zImage with a customized one.

Andreas
-- 
       Andreas Ferber - dev/consulting GmbH - Bielefeld, FRG
     ---------------------------------------------------------
         +49 521 1365800 - af@devcon.net - www.devcon.net

Re: initramfs programs (was [RFC] klibc requirements)

[Tom Rini]

On Thu, Jan 10, 2002 at 01:38:56AM +0100, Andreas Ferber wrote:
> On Wed, Jan 09, 2002 at 05:25:07PM -0700, Tom Rini wrote:
> > > 
> > > This could be done anyway: just replace the initramfs image built by 
> > > the kernel build with anotherone built from another source tree. It
> > > would be helpful though if the tools were distributed both standalone
> > > and included into the kernel tree.
> > If the kernel is going to build an initramfs option, it also needs a way
> > to be given one.  The issue I'm thinking of is I know of a few platforms
> > where the initramfs archive will have to be part of the 'zImage' file
> > (much like they do for ramdisks now).
> 
> Append it to the zImage and let the kernel look for it there. Plus add
> a tool to util-linux (or maybe an option to rdev?) to let you replace
> the embedded initramfs in a {,b}zImage with a customized one.

Er, 'rdev' is an x86-only program, so lets not add common functionality
to that.  And I'd rather not throw something onto the end of the
'zImage' since I just got done removing annoying/broken things like
that.  Mind you I don't think x86 will take advantage (or have reason
to, really) embedded the initramfs image into the bzImage/zImage, unless
we're going to let them run w/o a bootloader still.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: initramfs programs (was [RFC] klibc requirements)

[Rob Landley]

On Wednesday 09 January 2002 09:42 pm, Tom Rini wrote:
> On Thu, Jan 10, 2002 at 01:38:56AM +0100, Andreas Ferber wrote:
> > On Wed, Jan 09, 2002 at 05:25:07PM -0700, Tom Rini wrote:
> > > > This could be done anyway: just replace the initramfs image built by
> > > > the kernel build with anotherone built from another source tree. It
> > > > would be helpful though if the tools were distributed both standalone
> > > > and included into the kernel tree.
> > >
> > > If the kernel is going to build an initramfs option, it also needs a
> > > way to be given one.  The issue I'm thinking of is I know of a few
> > > platforms where the initramfs archive will have to be part of the
> > > 'zImage' file (much like they do for ramdisks now).
> >
> > Append it to the zImage and let the kernel look for it there. Plus add
> > a tool to util-linux (or maybe an option to rdev?) to let you replace
> > the embedded initramfs in a {,b}zImage with a customized one.
>
> Er, 'rdev' is an x86-only program, so lets not add common functionality
> to that.  And I'd rather not throw something onto the end of the
> 'zImage' since I just got done removing annoying/broken things like
> that.

You want it to be an ELF section then?

Rob

Re: initramfs programs (was [RFC] klibc requirements)

[Tom Rini]

On Thu, Jan 10, 2002 at 09:09:01AM -0500, Rob Landley wrote:
> On Wednesday 09 January 2002 09:42 pm, Tom Rini wrote:
> > On Thu, Jan 10, 2002 at 01:38:56AM +0100, Andreas Ferber wrote:
> > > On Wed, Jan 09, 2002 at 05:25:07PM -0700, Tom Rini wrote:
> > > > > This could be done anyway: just replace the initramfs image built by
> > > > > the kernel build with anotherone built from another source tree. It
> > > > > would be helpful though if the tools were distributed both standalone
> > > > > and included into the kernel tree.
> > > >
> > > > If the kernel is going to build an initramfs option, it also needs a
> > > > way to be given one.  The issue I'm thinking of is I know of a few
> > > > platforms where the initramfs archive will have to be part of the
> > > > 'zImage' file (much like they do for ramdisks now).
> > >
> > > Append it to the zImage and let the kernel look for it there. Plus add
> > > a tool to util-linux (or maybe an option to rdev?) to let you replace
> > > the embedded initramfs in a {,b}zImage with a customized one.
> >
> > Er, 'rdev' is an x86-only program, so lets not add common functionality
> > to that.  And I'd rather not throw something onto the end of the
> > 'zImage' since I just got done removing annoying/broken things like
> > that.
> 
> You want it to be an ELF section then?

I want it to be implementation specific.  The running kernel shouldn't
care how it got the image, just that it did.

And yes, on PPC it will either be given one by a 'real' bootloader
(yaboot, ppcboot) or an ELF section like the kernel itself, ramdisk and
sometimes System.map are.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: initramfs programs (was [RFC] klibc requirements)

[H. Peter Anvin]

Followup to:  <200201102155.g0ALtkA22362@snark.thyrsus.com>
By author:    Rob Landley <landley@trommello.org>
In newsgroup: linux.dev.kernel
> >
> > Er, 'rdev' is an x86-only program, so lets not add common functionality
> > to that.  And I'd rather not throw something onto the end of the
> > 'zImage' since I just got done removing annoying/broken things like
> > that.
> 
> You want it to be an ELF section then?
> 

initrd.  It's already supported on all architectures; initramfs uses
the same buffer-transfer architecture, which means you can use
unmodified bootloaders.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: initramfs programs (was [RFC] klibc requirements)

[Alex Bligh - linux-kernel]

--On Wednesday, 09 January, 2002 9:34 PM +0000 Anton Altaparmakov 
<aia21@cam.ac.uk> wrote:

>> seriously point: ls /sbin gives a /maximum/ range I'd
>> have thought.
>
> Partition discovery is currently done within the kernel itself. The code
> will effectively 'just' move out into user space.

Apologies - of course; I meant ls /sbin union {anything moved
out of kernel} gives a maximum range. I could find rationales,
some more questionable than others, for about half the stuff
in /sbin (for instance, if you are mounting NFS over IP, you
/might/ want to have iptables support in there before you start
playing with ip operations which without them might cause
a comprimized root to be mounted - I said /might/).

--
Alex Bligh

Re: [RFC] klibc requirements

[Juan Quintela]

>>>>> "felix" == Felix von Leitner <felix-dietlibc@fefe.de> writes:

Hi

>> Comments from the various libc authors?  Comments from other kernel
>> developers about requirements and goals they would like to see from such
>> a libc?

felix> I think you need to ask initrd users.
felix> My understanding was that people want to use the IP autoconfiguration
felix> stuff from the kernel to initrd.  Is that still so?  What other programs
felix> are needed?

I can think of that in a fast thought:

- if fsck.* can be there, it will make fs nice, just now they have to
  be able to fsck a ro root fs.
- I suppose that you can put here also the raid autodetect code and
  things like that.
- you also need a very small minishell, something like nash (that
  cames with mkinitrd will suffice).
  Basic utility here is that you want to have several small programs
  and have a small shell script to configure them (a distribution
  dont' want to recompile all its boot programs for each target
  configuration).
- as I suppose that you have supposed insmod is essential :)
- you already put there IP autoconfiguration.

Later, Juan.



-- 
In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy

Re: [RFC] klibc requirements

[Erik Andersen]

On Wed Jan 09, 2002 at 02:23:46PM +0100, Juan Quintela wrote:
> I can think of that in a fast thought:
> 
> - if fsck.* can be there, it will make fs nice, just now they have to
>   be able to fsck a ro root fs.

That would be a nice addition.

> - I suppose that you can put here also the raid autodetect code and
>   things like that.
> - you also need a very small minishell, something like nash (that
>   cames with mkinitrd will suffice).

busybox lash
    http://busybox.net/cgi-bin/cvsweb/busybox/shell/lash.c?rev=1.142&content-type=text/vnd.viewcvs-markup
or busybox msh or busybox ash if you need something thats
all posix compliant. 

> - as I suppose that you have supposed insmod is essential :)

busybox insmod

> - you already put there IP autoconfiguration.

udhcp

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] klibc requirements

[Albert D. Cahalan]

Felix von Leitner writes:

> My understanding of what "initramfs programs" actually means is vague at
> best.  Are these just programs that are intended to work in an initial
> ram disk?  Or is it a special collection that is included in the kernel
> distribution?

1. special collection by default
2. user-specified as desired

I think the dietlibc idea has to be scrapped so we can run BSD apps.
(and others maybe, but I'm not looking to start a flame war)

uClibc is LGPL, so traditional 4-clause BSD licensed code can be
linked with it.

> there are no legacy requirements to cater to.  We can write code without
> printf and stdio, for example.  Also, we probably don't need regular
> expressions or DNS.  Those are the big space hogs when linking
> statically against a libc.  In the diet libc, all of the above are very
> small, but avoiding them in the first place is better then optimizing
> them for small size.

DNS is very good to have. There are many things one might want
to specify by name. NFS servers, NIS servers, SMB servers, and
even the machine itself to get an IP via DNS.

> This may look like a good idea, but dynamic linking should be avoided.
> Trust me on this.  While it is possible to squeeze the dynamic loader
> down to below 10k, 10k really is a lot of code.  And empty program with
> the diet libc is way below 1k on x86.  So to reap the benefit of dynamic
> linking, you would need a lot of programs.  Also please note that -fPIC
> makes code larger.  And we need to keep symbols around, which makes up a
> substantial part of the shared diet libc.

a.out

Even with ELF, you shouldn't need that 10 kB. Treat ELF like a.out,
getting rid of the -fPIC stuff in favor of offsets assigned when
you build the initramfs. Dynamic linking should be:

open
mmap
mmap
close

You know the file to open. You know what offset you need it at.
There isn't any need for symbols. OK, that's half-dynamic,
but it gets the job done.

Re: [RFC] klibc requirements

[Felix von Leitner]

Thus spake Albert D. Cahalan (acahalan@cs.uml.edu):
> I think the dietlibc idea has to be scrapped so we can run BSD apps.
> (and others maybe, but I'm not looking to start a flame war)

What apps are you talking about?

> DNS is very good to have. There are many things one might want
> to specify by name. NFS servers, NIS servers, SMB servers, and
> even the machine itself to get an IP via DNS.

You don't need NIS or SMB before mounting the root disk.
If you want NFS to mount your root file system, you get the IP via DHCP
normally, so you don't need DNS.  And you can't get your own IP via DNS
because you need to have an IP to use DNS.

> Even with ELF, you shouldn't need that 10 kB.

Please go ahead and implement it.
Fame and fortune await you! ;)

> Treat ELF like a.out, getting rid of the -fPIC stuff in favor of
> offsets assigned when you build the initramfs.

ELF is a standard.
You can't just go out and re-invent dynamic linking completely.
First and most importantly of all, the GNU toolchain support the ELF
standard, not my personal ELF dialect.  I have no desire to write a
linker.  I'm happy enough that I found Olaf who was willing to write an
ld.so (and I was close enough to hear all his ranting about how complex
ELF is).

> Dynamic linking should
> be:

> open
> mmap
> mmap
> close

> You know the file to open. You know what offset you need it at.
> There isn't any need for symbols. OK, that's half-dynamic,
> but it gets the job done.

Again, please do it.  I know that it would be possible.  But it would
not be standard and I don't think it's worth the effort.  The most
important reason is that static linking produces small binaries with the
diet libc and uclibc.  As long as your software does not destroy that by
having a few dozen mini binaries around, all linked against the same
monster do-it-all library, it works.

Felix

Re: [RFC] klibc requirements

[David Lang]

On Tue, 15 Jan 2002, Felix von Leitner wrote:

> Date: Tue, 15 Jan 2002 12:55:44 +0100
> From: Felix von Leitner <felix-dietlibc@fefe.de>
> To: Albert D. Cahalan <acahalan@cs.uml.edu>
> Cc: Greg KH <greg@kroah.com>, linux-kernel@vger.kernel.org,
>      andersen@codepoet.org
> Subject: Re: [RFC] klibc requirements
>
> Thus spake Albert D. Cahalan (acahalan@cs.uml.edu):
> > I think the dietlibc idea has to be scrapped so we can run BSD apps.
> > (and others maybe, but I'm not looking to start a flame war)
>
> What apps are you talking about?

he's talking about licencing issues.

LGPL libc replacements can be used with any program, GPL libc replacements
can only be used with programs licenced in a way that can be combined with
GPL (and the resulting program is GPL.

as an example (not for the boot process, but an example of a replacement
libc use) I use the firewall toolkit, it has been around for a _loooong_
time (in software terms anyway) and has a firly odd licence (free for you
to use, source available, cannot sell it) which is not compatable with the
GPL. with glibc staticly linked this makes huge binaries, with libc5 they
were a lot smaller. I would like to try to use this small libc for these
proxies, but if the library is GPL, not LGPL I'm not allowed to.

David Lang

Re: [RFC] klibc requirements

[Doug McNaught]

David Lang <david.lang@digitalinsight.com> writes:

> as an example (not for the boot process, but an example of a replacement
> libc use) I use the firewall toolkit, it has been around for a _loooong_
> time (in software terms anyway) and has a firly odd licence (free for you
> to use, source available, cannot sell it) which is not compatable with the
> GPL. with glibc staticly linked this makes huge binaries, with libc5 they
> were a lot smaller. I would like to try to use this small libc for these
> proxies, but if the library is GPL, not LGPL I'm not allowed to.

Hmm, I think you can; you just can't redistribute it.  Can you even
redistribute fwtk on non-commercial terms?

-Doug
-- 
Let us cross over the river, and rest under the shade of the trees.
   --T. J. Jackson, 1863

Re: [RFC] klibc requirements

[David Lang]

On 15 Jan 2002, Doug McNaught wrote:

>
> > as an example (not for the boot process, but an example of a replacement
> > libc use) I use the firewall toolkit, it has been around for a _loooong_
> > time (in software terms anyway) and has a firly odd licence (free for you
> > to use, source available, cannot sell it) which is not compatable with the
> > GPL. with glibc staticly linked this makes huge binaries, with libc5 they
> > were a lot smaller. I would like to try to use this small libc for these
> > proxies, but if the library is GPL, not LGPL I'm not allowed to.
>
> Hmm, I think you can; you just can't redistribute it.  Can you even
> redistribute fwtk on non-commercial terms?
>
nope, only allowed to get it from nai (and they sure don't make it easy to
find on their website)

David Lang

Re: [RFC] klibc requirements

[Doug McNaught]

David Lang <dlang@diginsite.com> writes:

> On 15 Jan 2002, Doug McNaught wrote:
> 
> >
> > > as an example (not for the boot process, but an example of a replacement
> > > libc use) I use the firewall toolkit, it has been around for a _loooong_
> > > time (in software terms anyway) and has a firly odd licence (free for you
> > > to use, source available, cannot sell it) which is not compatable with the
> > > GPL. with glibc staticly linked this makes huge binaries, with libc5 they
> > > were a lot smaller. I would like to try to use this small libc for these
> > > proxies, but if the library is GPL, not LGPL I'm not allowed to.
> >
> > Hmm, I think you can; you just can't redistribute it.  Can you even
> > redistribute fwtk on non-commercial terms?
> >
> nope, only allowed to get it from nai (and they sure don't make it easy to
> find on their website)

Problem solved, then; you can link fwtk with a GPL'd libc on your own
machines and use it all day.  You can't redistribute fwtk, so you
aren't even tempted to violate the GPL.

-Doug
-- 
Let us cross over the river, and rest under the shade of the trees.
   --T. J. Jackson, 1863

Re: [RFC] klibc requirements

[Erik Andersen]

On Tue Jan 15, 2002 at 06:54:46AM -0800, David Lang wrote:
> were a lot smaller. I would like to try to use this small libc for these
> proxies, but if the library is GPL, not LGPL I'm not allowed to.

So use uClibc, which is LGPL,

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] klibc requirements

[Albert D. Cahalan]

Felix von Leitner writes:
> Thus spake Albert D. Cahalan (acahalan@cs.uml.edu):

>> I think the dietlibc idea has to be scrapped so we can run BSD apps.
>> (and others maybe, but I'm not looking to start a flame war)
>
> What apps are you talking about?

I'm talking about all apps under the traditional 4-clause
BSD license. Every single one of them is incompatible with
a GPL libc. An LGPL library works fine.

BTW, the LGPL isn't just for libraries anymore. It's now the
called the "Lesser" GPL, perfectly suited for apps as well
as libraries. I know this must be what RMS had in mind, so I
released all my "ps" source code under the LGPL. :-)

> You don't need NIS or SMB before mounting the root disk.

NIS can be used to specify what filesystem to use.
SMB could be that filesystem.

>> Treat ELF like a.out, getting rid of the -fPIC stuff in favor of
>> offsets assigned when you build the initramfs.
>
> ELF is a standard.
> You can't just go out and re-invent dynamic linking completely.

Sure you can. As long as the kernel loader is happy, no problem.
People commit worse sins trying to squeeze stuff onto a floppy.