A random initramfs script

A random initramfs script

[Nix]

In the interests of pushing people away from in-kernel autodetection, I
thought I'd provide the initramfs script I just knocked up to boot my
RAID+LVM system. It's had a whole four days of testing so it must
work. :)

It's being used to boot a system that boots from RAID-1 and has almost
everything else on a pair of RAID-5 arrays (not quite everything, as the
disks are wildly different sizes, so I was left with a 20Gb slice at the
end of the largest disk that I'm swapping onto and using for data that I
don't care about).

It has a number of improvements over the initramfs embedded in the
script that comes with mdadm:

 - It handles LVM2 as well as md (obviously if you boot off RAID
   you still have to boot off RAID1, but /boot can be a RAID1
   filesystem of its own now, with / in LVM, on RAID, or both
   at once)
 - It fscks / before mounting it
 - If anything goes wrong, it drops you into an emergency shell
   in the rootfs, from where you have all the power of ash
   without hardly any builtin commands, lvm and mdadm to
   diagnose your problem :) you can't do *that* with in-
   kernel array autodetection!
 - it supports arguments `rescue', to drop into /bin/ash
   instead of init after mounting the real root filesystem,
   and `emergency', to drop into a shell on the initramfs
   before doing *anything*.
 - It supports root= and init= arguments, although for
   arcane reasons to do with LILO suckage you need to pass
   the root argument as `root=LABEL=/dev/some/device',
   or LILO will helpfully transform it into a device number,
   which is rarely useful if the device name is, say,
   /dev/emergency-volume-group/root ;) right now, if you
   don't pass root=, it tries to mount /dev/raid/root after
   initializing all the RAID arrays and LVM VGs it can.
 - it doesn't waste memory. initramfs isn't like initrd:
   if you just chroot into the new root filesystem, the
   data in the initramfs *stays around*, in *nonswappable*
   kernel memory. And it's not gzipped by that point, either!

The downsides:

 - it needs a very new busybox, from Subversion after the start of
   this year: I'm using svn://busybox.net/trunk/busybox revision 14406,
   and a 2.6.12+ kernel with sysfs and hotplug support; this is
   because it populates /dev with the `mdev' mini-udev tool inside
   busybox, and switches root filesystems with the `switch_root'
   tool, which chroots only after erasing the entire contents
   of the initramfs (taking *great* care not to recurse off that
   filesystem!)
 - if you link against uClibc (recommended), you need a CVS
   uClibc too (i.e., one newer than 0.9.27).
 - it doesn't try to e.g. set up the network, so it can't do really
   whizzy things like mount a root filesystem situated on a network
   block device on some other host: if you want to do something like
   that you've probably already written a script to do it long ago
 - the init script's got a few too many things hardwired still,
   like the type of the root filesystem. I expect it's short
   enough to easily hack up if you need to :)
 - you need an /etc/mdadm.conf and an /etc/lvm/lvm.conf, both taken
   by default from the system you built the kernel on: personally
   I'd recommend a really simple one with no device= lines, like

DEVICE partitions
ARRAY /dev/md0 UUID=some:long:uuid:here
ARRAY /dev/md1 UUID=another:long:uuid:here
ARRAY /dev/md2 UUID=yetanother:long:uuid:here
...

One oddity, also: after booting with this, I see some strange results
from --examine --scan with mdadm-2.3.1:

loki:/root# mdadm --examine --scan
ARRAY /dev/md0 level=raid1 num-devices=4 UUID=3a51b74f:8a759fe7:8520304c:3adbceb1
ARRAY /dev/?? level=raid5 metadata=1 num-devices=3 UUID=a5a6cad42c:7fdc0788:a409b919:2ed3bf name=large
ARRAY /dev/?? level=raid5 metadata=1 num-devices=3 UUID=fe44916da1:09857680:07fb812e:e33b5a
loki:/root# ls -l /dev/md*
brw-rw---- 1 root disk 9, 0 Mar 14 20:10 /dev/md0
brw-rw---- 1 root disk 9, 1 Mar 14 20:10 /dev/md1
brw-rw---- 1 root disk 9, 2 Mar 14 20:10 /dev/md2

This is decidedly peculiar because the kernel said it was using md1 and
md2 on the initramfs, and the device numbers are surely right:

raid5: device sda6 operational as raid disk 0
raid5: device hdc5 operational as raid disk 2
raid5: device sdb6 operational as raid disk 1
raid5: allocated 3155kB for md1
raid5: raid level 5 set md1 active with 3 out of 3 devices, algorithm 2
[...]
raid5: device sdb7 operational as raid disk 0
raid5: device hda5 operational as raid disk 2
raid5: device sda7 operational as raid disk 1
raid5: allocated 3155kB for md2
raid5: raid level 5 set md2 active with 3 out of 3 devices, algorithm 2


Anyway, without further ado, here's usr/init:

#!/bin/sh
#
# init --- locate and mount root filesystem
#          By Nix <nix@esperi.org.uk>.
#
#          Placed in the public domain.
#

export PATH=/sbin:/bin

/bin/mount -t proc proc /proc
/bin/mount -t sysfs sysfs /sys
CMDLINE=`cat /proc/cmdline`

# Populate /dev from /sys
/bin/mount -t tmpfs tmpfs /dev
/sbin/mdev -s

INIT_ARGS="$@"

# If there is a forced root filesystem or init, accept the forcing
for param in $CMDLINE; do
    case "$param" in
        init=*) eval "$param";;
        rescue) echo "Rescue boot mode: invoking ash.";
                init=/bin/ash;
                INIT_ARGS="-";;
        emergency) echo "Emergency boot mode. Dropping to a minimal shell.";
                   echo "Reboot with Ctrl-Alt-Delete.";
                   exec /bin/sh;;
        root=LABEL=*) root="`echo $param | cut -d= -f3-`";;
    esac
done

# Assemble the RAID arrays.
/sbin/mdadm --assemble --scan --auto=md --run

FAILED=

# Scan for volume groups.
/sbin/lvm vgscan --ignorelockingfailure --mknodes && /sbin/lvm vgchange -ay --ignorelockingfailure

[[ -z $root ]] && root=/dev/raid/root

fsck -a $root

if [[ $? -eq 4 ]]; then
    echo "Filesystem errors left uncorrected."
    echo
    echo "Dropping to a minimal shell.  Reboot with Ctrl-Alt-Delete."

    exec /bin/sh
fi

if [[ -n $root ]]; then 
    /bin/mount -o rw -t ext3 $root /new-root
fi

if /bin/mountpoint /new-root >/dev/null; then :; else
    echo "No root filesystem given to the kernel or found on the root RAID array."
    echo "Append the correct 'root=' boot option."
    echo
    echo "Dropping to a minimal shell.  Reboot with Ctrl-Alt-Delete."

    exec /bin/sh
fi

if [[ -z "$init" ]]; then
    init=/sbin/init
fi

# Unmount everything and switch root filesystems for good:
# exec the real init and begin the real boot process.
/bin/umount -l /proc
/bin/umount -l /sys
/bin/umount -l /dev

echo "Switching to /new-root and running '$init'"
exec switch_root /new-root $init $INIT_ARGS


And usr/initramfs (will need adjustment for your system):

#
# Files needed for early userspace.
# Placed in the public domain.
#

dir /bin 0755 0 0
file /bin/busybox /usr/i686-pc-linux-uclibc/bin/busybox 0755 0 0
slink /bin/sh /bin/busybox 0755 0 0
slink /bin/msh /bin/busybox 0755 0 0
slink /bin/[ /bin/busybox 0755 0 0
slink /bin/[[ /bin/busybox 0755 0 0
slink /bin/test /bin/busybox 0755 0 0
slink /bin/mount /bin/busybox 0755 0 0
slink /bin/umount /bin/busybox 0755 0 0
slink /bin/cat /bin/busybox 0755 0 0
slink /bin/ls /bin/busybox 0755 0 0
slink /bin/mountpoint /bin/busybox 0755 0 0
slink /bin/echo /bin/busybox 0755 0 0
slink /bin/false /bin/busybox 0755 0 0
slink /bin/true /bin/busybox 0755 0 0
slink /bin/mkdir /bin/busybox 0755 0 0
dir /sbin 0755 0 0
slink /sbin/mdev /bin/busybox 0755 0 0
slink /sbin/fsck /bin/busybox 0755 0 0
slink /sbin/e2fsck /bin/busybox 0755 0 0
slink /sbin/fsck.ext2 /bin/busybox 0755 0 0
slink /sbin/fsck.ext3 /bin/busybox 0755 0 0
slink /sbin/switch_root /bin/busybox 0755 0 0
file /sbin/mdadm /usr/i686-pc-linux-uclibc/sbin/mdadm 0755 0 0
file /sbin/lvm /usr/i686-pc-linux-uclibc/sbin/lvm 0755 0 0
file /init usr/init 0755 0 0

# supporting directories
dir /proc 0755 0 0
dir /sys 0755 0 0
dir /new-root 0755 0 0
dir /etc 0755 0 0
dir /etc/lvm 0755 0 0
file /etc/lvm/lvm.conf /etc/lvm/lvm.conf 0644 0 0
file /etc/mdadm.conf /etc/mdadm.conf 0644 0 0

# initial device files required (mdev creates the rest)
dir /dev 0755 0 0
nod /dev/console 0600 0 0 c 5 1
nod /dev/null 0666 0 0 c 1 3


And the busybox config file I used for all this --- you *will* need to
change the CROSS_COMPILER_PREFIX and the EXTRA_CFLAGS_OPTIONS, and you
might want to build in more tools as well for use when things go wrong,
in emergency mode:

HAVE_DOT_CONFIG=y
CONFIG_FEATURE_BUFFERS_USE_MALLOC=y
CONFIG_FEATURE_DEVPTS=y
CONFIG_STATIC=y
CONFIG_LFS=y
USING_CROSS_COMPILER=y
CROSS_COMPILER_PREFIX="/usr/bin/i686-pc-linux-uclibc-"
EXTRA_CFLAGS_OPTIONS="-march=pentium3 -fomit-frame-pointer"
CONFIG_INSTALL_NO_USR=y
CONFIG_INSTALL_APPLET_SYMLINKS=y
PREFIX="./_install"
CONFIG_CAT=y
CONFIG_CUT=y
CONFIG_ECHO=y
CONFIG_FALSE=y
CONFIG_LS=y
CONFIG_MKDIR=y
CONFIG_TEST=y
CONFIG_TRUE=y
CONFIG_FEATURE_AUTOWIDTH=y
CONFIG_E2FSCK=y
CONFIG_FSCK=y
FDISK_SUPPORT_LARGE_DISKS=y
CONFIG_MDEV=y
CONFIG_MOUNT=y
CONFIG_SWITCH_ROOT=y
CONFIG_UMOUNT=y
CONFIG_MOUNTPOINT=y
CONFIG_FEATURE_SH_IS_MSH=y
CONFIG_MSH=y
CONFIG_FEATURE_SH_EXTRA_QUIET=y
CONFIG_FEATURE_SH_STANDALONE_SHELL=y
CONFIG_FEATURE_COMMAND_EDITING=y
CONFIG_FEATURE_COMMAND_EDITING_VI=y
CONFIG_FEATURE_COMMAND_HISTORY=15
CONFIG_FEATURE_COMMAND_TAB_COMPLETION=y
CONFIG_FEATURE_IPC_SYSLOG_BUFFER_SIZE=0
CONFIG_MD5_SIZE_VS_SPEED=2

-- 
`Come now, you should know that whenever you plan the duration of your
 unplanned downtime, you should add in padding for random management
 freakouts.'

Re: A random initramfs script

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 3199 bytes --]

On 21:37, Nix wrote:

> In the interests of pushing people away from in-kernel autodetection,
> I thought I'd provide the initramfs script I just knocked up to boot
> my RAID+LVM system.  It's had a whole four days of testing so it must
> work. :)

I'm using a similar setup since December or so with no problems so
far. However, all my systems have initramfs as their rootfs.

>  - it doesn't waste memory. initramfs isn't like initrd:
>    if you just chroot into the new root filesystem, the
>    data in the initramfs *stays around*, in *nonswappable*
>    kernel memory. And it's not gzipped by that point, either!

If you really care, you can remove almost everything from the initramfs
just before mounting root. But that's not really necessary unless you
are very short on memory. BTW: I'm also using a complete rescue system
(45 MB unpacked) which has everything on initramfs. lilo boots this
25 MB kernel just fine, and so does etherboot.

> The downsides:
> 
>  - if you link against uClibc (recommended), you need a CVS
>    uClibc too (i.e., one newer than 0.9.27).

glibc works also fine. For the records: You'll need these:

/lib/ld-linux.so.2
/lib/libc.so.6
/lib/libwrap.so.2
/lib/libnsl.so.1
/lib/libnss_nis.so.2
/lib/libnss_files.so.2
/lib/libnss_compat.so.2

>  - it doesn't try to e.g. set up the network, so it can't do really
>    whizzy things like mount a root filesystem situated on a network
>    block device on some other host: if you want to do something like
>    that you've probably already written a script to do it long ago

Yep. Here's what I do for my discless clients (no more nfsroot needed):

ifconfig eth0
route del default
route add default gw $NET.$gw_ip eth0
portmap
ifconfig lo 127.0.0.1
route add -net 127.0.0.0 netmask 255.0.0.0 dev lo
# mount nfs filesystems

>  - you need an /etc/mdadm.conf and an /etc/lvm/lvm.conf, both taken
>    by default from the system you built the kernel on: personally
>    I'd recommend a really simple one with no device= lines, like
> 
> DEVICE partitions
> ARRAY /dev/md0 UUID=some:long:uuid:here
> ARRAY /dev/md1 UUID=another:long:uuid:here
> ARRAY /dev/md2 UUID=yetanother:long:uuid:here

The following works pretty well for me:

	echo "DEVICE /dev/hd*[0-9] /dev/sd*[0-9] /dev/md[0-9]" > /etc/mdadm.conf
	mdadm --examine --scan --config=/etc/mdadm.conf  >> /etc/mdadm.conf
	mdadm --assemble --scan

> Anyway, without further ado, here's usr/init:
> 

> /sbin/mdev -s

What's mdev? udevstart works too, but it seems to be depreciated now.

> # Assemble the RAID arrays.
> /sbin/mdadm --assemble --scan --auto=md --run

Minor suggestion:

	if test -e /proc/mdstat; then /sbin/mdadm ...

> # Scan for volume groups.
> /sbin/lvm vgscan --ignorelockingfailure --mknodes && /sbin/lvm vgchange -ay --ignorelockingfailure

Similarly,

	if test -c /dev/mapper/control; then ...

> And usr/initramfs (will need adjustment for your system):

> file /sbin/lvm /usr/i686-pc-linux-uclibc/sbin/lvm 0755 0 0

Isn't libdevmapper also needed?

Regards,
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: A random initramfs script

[Nix]

On Wed, 15 Mar 2006, Andre Noll gibbered uncontrollably:
> On 21:37, Nix wrote:
> 
>> In the interests of pushing people away from in-kernel autodetection,
>> I thought I'd provide the initramfs script I just knocked up to boot
>> my RAID+LVM system.  It's had a whole four days of testing so it must
>> work. :)
> 
> I'm using a similar setup since December or so with no problems so
> far. However, all my systems have initramfs as their rootfs.

(well, technically, *every* 2.6 system has an initramfs unpacked into
its rootfs: it's just that for most people it's empty.)

>>  - it doesn't waste memory. initramfs isn't like initrd:
>>    if you just chroot into the new root filesystem, the
>>    data in the initramfs *stays around*, in *nonswappable*
>>    kernel memory. And it's not gzipped by that point, either!
> 
> If you really care, you can remove almost everything from the initramfs
> just before mounting root. But that's not really necessary unless you
> are very short on memory.

The kernel hackers are going through conniptions right now trimming
hundreds of *bytes* off the kernel image. Given that the people who know
how the Linux memory manager works are treating adding to nonswappable
memory that seriously, adding hundreds of Kb to nonswappable memory load
for lack of a 100-line C program to do a single-file-system rm -rf /-
and-chroot() seems unwise.

>                           BTW: I'm also using a complete rescue system
> (45 MB unpacked) which has everything on initramfs. lilo boots this
> 25 MB kernel just fine, and so does etherboot.

There *is* a limit (depending on the size of the statically-populated
page tables); right now I think it's 32Mb on i386/x86_64. Look out.

>>  - if you link against uClibc (recommended), you need a CVS
>>    uClibc too (i.e., one newer than 0.9.27).
> 
> glibc works also fine. For the records: You'll need these:

glibc works fine if you don't care about initramfs bloat. My initramfs
is ~600Kb unpacked, 270Kb when gzipped onto the end of the kernel,
and nearly half of that is the busybox fsck.

I tend to treat initramfs like an embedded system: every byte counts,
because if switch_root were to fail to delete everything (which *can*
happen under certain obscure circumstances), I'd be paying for every
byte for as long as the system runs.

>>  - it doesn't try to e.g. set up the network, so it can't do really
>>    whizzy things like mount a root filesystem situated on a network
>>    block device on some other host: if you want to do something like
>>    that you've probably already written a script to do it long ago
> 
> Yep. Here's what I do for my discless clients (no more nfsroot needed):
> 
> ifconfig eth0
> route del default
> route add default gw $NET.$gw_ip eth0

ifconfig? route? ick. ip(8) is the wave of the future :) far more flexible
and actually has a comprehensible command line as well.

I try to avoid running daemons out of initramfs, because all those daemons
share *no* inodes with anything else you'll ever run: more permanent memory
load for as long as those daemons are running, although at least it's
swappable load.

>> DEVICE partitions
>> ARRAY /dev/md0 UUID=some:long:uuid:here
>> ARRAY /dev/md1 UUID=another:long:uuid:here
>> ARRAY /dev/md2 UUID=yetanother:long:uuid:here
> 
> The following works pretty well for me:
> 
> 	echo "DEVICE /dev/hd*[0-9] /dev/sd*[0-9] /dev/md[0-9]" > /etc/mdadm.conf
> 	mdadm --examine --scan --config=/etc/mdadm.conf  >> /etc/mdadm.conf
> 	mdadm --assemble --scan

Yeah, that would work. Neil's very *emphatic* about hardwiring the UUIDs of
your arrays, though I'll admit that given the existence of --examine --scan,
I don't really see why. :)

>> /sbin/mdev -s
> 
> What's mdev? udevstart works too, but it seems to be depreciated now.

mdev is `micro-udev', a 255-line tiny replacement for udev. It's part of
busybox.

It's not really a full-blown udev replacement: it doesn't do all the
cool configurability stuff that udev can do; it just scans /sys and
populates /dev with all the appropriate kernel names. (`-s' means `do
this for the first time, don't become a daemon because I don't care
about hotplugging').

`switch_root', at the end of the script, is also a recent busybox
addition, which does the delete-everything-and-chroot dance you need to
do to efficiently switch away from the rootfs when there's an initramfs
in it.

>> # Assemble the RAID arrays.
>> /sbin/mdadm --assemble --scan --auto=md --run
> 
> Minor suggestion:
> 
> 	if test -e /proc/mdstat; then /sbin/mdadm ...

Pointless, really: if /proc fails to mount we're dead anyway (because /sys
will almost certainly also misbehave and now we don't have any block
devices because /dev didn't get populated); and the nice thing about
initramfs is that I can be *sure* that RAID was built into this kernel
because the initramfs is linked into the kernel image :)

I suppose for general consumption, that might be a good move: I can't
guarantee that other people will have RAID and LVM built in.

>> # Scan for volume groups.
>> /sbin/lvm vgscan --ignorelockingfailure --mknodes && /sbin/lvm vgchange -ay --ignorelockingfailure
> 
> Similarly,
> 
> 	if test -c /dev/mapper/control; then ...

Agreed.

>> And usr/initramfs (will need adjustment for your system):
> 
>> file /sbin/lvm /usr/i686-pc-linux-uclibc/sbin/lvm 0755 0 0
> 
> Isn't libdevmapper also needed?

No, I linked everything statically because there are only three
independent binaries on that disk, so the space saving from not
including stuff in the libc that is unused exceeds that from not
duplicating stuff. I didn't provide shared library support in
that uClibc at all.

Binary sizes:

-rwxr-xr-x 1 root root 248580 Mar 12 17:15 bin/busybox
-r-xr-xr-x 1 root root 512693 Mar  5 17:53 sbin/lvm
-rwxr-xr-x 1 root root 173349 Mar  5 16:50 sbin/mdadm

lvm's a bit big, but I have no real choice there. I chose to build
in mdadm and not mdassemble for the sake of disaster recovery; it
seems you can fix just about anything you can imagine going wrong
if you have a copy of mdadm to hand. :)

-- 
`Come now, you should know that whenever you plan the duration of your
 unplanned downtime, you should add in padding for random management
 freakouts.'

Re: A random initramfs script

[Andre Noll]

On 08:29, Nix wrote:

> > I'm using a similar setup since December or so with no problems so
> > far. However, all my systems have initramfs as their rootfs.
> 
> (well, technically, *every* 2.6 system has an initramfs unpacked into
> its rootfs: it's just that for most people it's empty.)

What I meant is that it never gets overmounted by a real rootfs.

> There *is* a limit (depending on the size of the statically-populated
> page tables); right now I think it's 32Mb on i386/x86_64. Look out.

Care to explain a bit further? Are these statically-populated page
tables only used at boot time? I always thought an N MB initramfs
cuts about N MB of the main memory, plus some housekeeping data,
but that's all. This view is probably a little bit too naive ;)

> ifconfig? route? ick. ip(8) is the wave of the future :) far more flexible
> and actually has a comprehensible command line as well.

Thanks for the tip. I'll give it a try.

> I try to avoid running daemons out of initramfs, because all those daemons
> share *no* inodes with anything else you'll ever run: more permanent memory
> load for as long as those daemons are running, although at least it's
> swappable load.

Fair enough, but portmap is needed to mount NFS filesystems. How much
memory would be saved if I'd kill portmap after the mount and restart
it afterwards from a mounted filesystem?

> > The following works pretty well for me:
> > 
> > 	echo "DEVICE /dev/hd*[0-9] /dev/sd*[0-9] /dev/md[0-9]" > /etc/mdadm.conf
> > 	mdadm --examine --scan --config=/etc/mdadm.conf  >> /etc/mdadm.conf
> > 	mdadm --assemble --scan
> 
> Yeah, that would work. Neil's very *emphatic* about hardwiring the UUIDs of
> your arrays, though I'll admit that given the existence of --examine --scan,
> I don't really see why. :)

He likes to compare the situation with /etc/fstab. Nobody complains
about having to edit /etc/fstab, so why keep people complaining about
having to edit /etc/mdadm.conf?

> >> /sbin/mdev -s
> > 
> > What's mdev? udevstart works too, but it seems to be depreciated now.
> 
> mdev is `micro-udev', a 255-line tiny replacement for udev. It's part of
> busybox.

Cool. Guess I'll have to update busybox..

> > Minor suggestion:
> > 
> > 	if test -e /proc/mdstat; then /sbin/mdadm ...
> 
> Pointless, really: if /proc fails to mount we're dead anyway 

It's not about /proc, it's about /proc/mdstat. If the machine does
not have raid, there won't be a /proc/mdstat.

> I can be *sure* that RAID was built into this kernel
> because the initramfs is linked into the kernel image :)

OK. But then you'll need different init scripts for different setups.

> Binary sizes:
> 
> -rwxr-xr-x 1 root root 248580 Mar 12 17:15 bin/busybox
> -r-xr-xr-x 1 root root 512693 Mar  5 17:53 sbin/lvm
> -rwxr-xr-x 1 root root 173349 Mar  5 16:50 sbin/mdadm

That's indeed much smaller than what I have (6 MB). However, what
I like about using glibc is that there is no need to have different
versions of these tools. If I install a new mdadm, the next kernel's
initramfs will automatically contain the new version. Moreover, one
can add another binary to the initramfs simply by putting one more
line to a text file.

Thanks
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

Re: A random initramfs script

[Andre Noll]

On 20:56, Nix wrote:

> > What I meant is that it never gets overmounted by a real rootfs.
> 
> The rootfs never overmounts anything. Do you mean `it never gets
> overmounted by a real root filesystem'?

Yes, that's what I tried to say. Sorry about the confusion. Basically
I have three similar setups that work with the same init script:

	The main server which has all its filesystems on lvm.
	There is only one vg and only one pv, a raid 10 array. /bin,
	/etc, /home, /lib /root, /sbin, /tmp, /usr and /var are
	all lvs. They are mounted by the the init script of the
	initramfs. That script does the full boot and finally execs
	busybox init just do start the getty processes.

	A bunch of NFS clients. They boot similarly, but mount
	everything (except /etc) via NFS.

	The rescue system. Still the same, but does not mount anything
	as everything is contained in the initramfs.

None of these have a real root filesystem. nfsroot support isn't even
compiled in for the NFS clients.

> If so, that's a... most peculiar setup: I don't think I've ever heard of
> anything but systems meant to run on diskless kiosks and the occasional
> live cd running like that.

It works pretty well. My diskless desktop is up for more than 100
days and the main server had a similar uptime before I rebooted some
days ago (to try out dccp).

> ... but looking at init/initramfs.c, you should be safe as long as the
> initramfs can be loaded at all (on i386 I think there's a 16Mb limit on
> that but it depends on the arch and the bootloader and all sorts of
> horrid arcana: most arches have a much higher limit).

I just booted a rescue system on i386 which is definitely larger
than 16MB compressed.

> >                                I always thought an N MB initramfs
> > cuts about N MB of the main memory, plus some housekeeping data,
> > but that's all. This view is probably a little bit too naive ;)
> 
> It does, *unless* you switch root filesystems appropriately, that is,
> you unlink everything in the initramfs before `exec chroot'ing into the
> real root filesystem. In practice this has to be done by a dedicated C
> program in order to get away with deleting /dev and the very chroot
> binary itself :) and that's what busybox's switch_root does.

In view of this chicken-egg problem it looks much cleaner to me to avoid
the chroot :)

> >> ifconfig? route? ick. ip(8) is the wave of the future :) far more flexible
> >> and actually has a comprehensible command line as well.
> > 
> > Thanks for the tip. I'll give it a try.
> 
> <http://lartc.org/> describes many of its myriad extra features in more
> detail.

All that stuff seems to be fairly old, linux-2.6. isn't mentioned at
all and the cvs server doesn't work. Is it still up do date?

> >> I try to avoid running daemons out of initramfs, because all those daemons
> >> share *no* inodes with anything else you'll ever run: more permanent memory
> >> load for as long as those daemons are running, although at least it's
> >> swappable load.
> > 
> > Fair enough, but portmap is needed to mount NFS filesystems. How much
> > memory would be saved if I'd kill portmap after the mount and restart
> > it afterwards from a mounted filesystem?
> 
> All of it. The only reason running daemons from initramfs wastes memory
> is that if you've got the binaries open/running and you delete them, in
> standard Unix fashion the space used by them won't be reclaimed until
> they're closed/stopped --- and the standard procedure for switching from
> an initramfs involves unlinking everything on the filesystem.
> 
> There's *nothing* special about the rootfs filesystem on which initramfs
> runs except that it's a ramfs filesystem, and as such nonswappable, and
> that it's the start and end of the kernel's list of mount points (which
> means that unmounting it, mount --moving it, and so on, is impossible).
> It behaves just like a POSIX filesystem because it is one. In fact it's
> almost exactly the same as a tmpfs except that it's not swap-backed, and
> its maximum size can't be limited in the way tmpfs can. (tmpfs is a small
> hack atop ramfs.)

Thanks for the explanation. I really appreciate it!

> so why is dynamic scanning the preferred method in LVM,
> yet discouraged in the md world? I see some conflicted messages here ;)

I'm afraid, only Neil will be able to answer this. BTW: I agree with
you and do not see the point in hardwiring the UUIDs either.

> >> mdev is `micro-udev', a 255-line tiny replacement for udev. It's part of
> >> busybox.
> > 
> > Cool. Guess I'll have to update busybox..

done. The new busybox (from SVN) seems to work fine, just like the
old one did. The init script doesn't use mdev yet, but from a first
reading this is just a matter of translating /etc/udev/udev.conf
to the mdev syntax.

> You need SVN uClibc too (if you're using uClibc rather than glibc);
> older versions don't maintain the d_type field in the struct dirent, so
> mdev's scanning of /sys gets very confused and you end up with an empty
> /dev.

Damn. I just compiled 0.9.28. Guess this one is too old.

> > That's indeed much smaller than what I have (6 MB). However, what
> 
> 6Mb!? Ye *gods*.

Hehe, didn't god create his first initramfs within 6 MB, but was
arrested on Sunday, the seventh? :)

Seriously, what's 6 MB if the onboard graphic chip easily cuts away
another 64 MB?

> (Also, you do know about the `mdadm.uclibc' target in the mdadm
> makefile, right?

No, I didn't. Thanks for enlightening me.
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

Re: A random initramfs script

[Neil Brown]

On Wednesday March 15, maan@systemlinux.org wrote:
> On 08:29, Nix wrote:
> > I try to avoid running daemons out of initramfs, because all those daemons
> > share *no* inodes with anything else you'll ever run: more permanent memory
> > load for as long as those daemons are running, although at least it's
> > swappable load.
> 
> Fair enough, but portmap is needed to mount NFS filesystems. How much
> memory would be saved if I'd kill portmap after the mount and restart
> it afterwards from a mounted filesystem?
> 

You shouldn't need portmap to mount an NFS filesystem unless you
enable locking, and you shouldn't enable nfs locking on the root
filesystem, so you shouldn't need portmap in the initramfs.


> > > The following works pretty well for me:
> > > 
> > > 	echo "DEVICE /dev/hd*[0-9] /dev/sd*[0-9] /dev/md[0-9]" > /etc/mdadm.conf
> > > 	mdadm --examine --scan --config=/etc/mdadm.conf  >> /etc/mdadm.conf
> > > 	mdadm --assemble --scan
> > 
> > Yeah, that would work. Neil's very *emphatic* about hardwiring the UUIDs of
> > your arrays, though I'll admit that given the existence of --examine --scan,
> > I don't really see why. :)
> 
> He likes to compare the situation with /etc/fstab. Nobody complains
> about having to edit /etc/fstab, so why keep people complaining about
> having to edit /etc/mdadm.conf?

Indeed!  And if you plug in some devices off another machine for
disaster recovery, you don't want another disaster because you
assembled the wrong arrays.

I would like an md superblock to be able to contain some indication of
the 'name' of the machine which is meant to host the array, so that
once a machine knows its own name, it can automatically find and mount
its own arrays, but that isn't near the top of my list of priorities
yet.


NeilBrown

Re: A random initramfs script

[Nix]

On Thu, 16 Mar 2006, Neil Brown wrote:
> On Wednesday March 15, maan@systemlinux.org wrote:
>> On 08:29, Nix wrote:
>> > Yeah, that would work. Neil's very *emphatic* about hardwiring the UUIDs of
>> > your arrays, though I'll admit that given the existence of --examine --scan,
>> > I don't really see why. :)
>> 
>> He likes to compare the situation with /etc/fstab. Nobody complains
>> about having to edit /etc/fstab, so why keep people complaining about
>> having to edit /etc/mdadm.conf?
> 
> Indeed!  And if you plug in some devices off another machine for
> disaster recovery, you don't want another disaster because you
> assembled the wrong arrays.

Well, I can't have that go wrong because I assemble all of them :)

One thing I would like to know, though: I screwed up construction
of one of my arrays and forgot to give it a name, so every array
with a V1 superblock has a name *except one*.

Is there a way to change the name after array creation? (Another
overloading of --grow, perhaps?)

(I'm still quite new to md so rather queasy about `just trying'
things like this with active arrays containing critical data.
I suppose I should build a test array on a sparse loopback mount
or something...)

> I would like an md superblock to be able to contain some indication of
> the 'name' of the machine which is meant to host the array, so that
> once a machine knows its own name, it can automatically find and mount
> its own arrays,

... and of course you could request the mounting of some other machine's
arrays if you cart disks around between machines :)

Seems like a good idea.

-- 
`Come now, you should know that whenever you plan the duration of your
 unplanned downtime, you should add in padding for random management
 freakouts.'

Re: A random initramfs script

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 4648 bytes --]

On 07:50, Nix wrote:

> I suppose that if *every* filesystem hanging off / is its own
> fs, using rootfs as your / is not inefficient because there's
> still nothing in it.
> 
> But it still makes me worry: what if some mad script makes a
> huge file in /? It's happened to me a couple of times, and
> because /var was on a different fs, all that happened was that
> / filled up and nothing bad resulted.

> If / was a ramfs (as rootfs is), you'd run out of memory...

Yes, it's an additional piece of rope, and I already used it to shoot
myself in the foot by doing a backup with "rsync -a /home /mnt" without
mounting /mnt. First the machine went slow, then the OOM killer kicked
in and killed everything. Finally the system was totally unresponsible
and I had to use the "so everything is unusual - boot" thing.

But only root can write to /mnt, and there are much simpler ways for
root to halt the system..

> [ip(8)]
> >> <http://lartc.org/> describes many of its myriad extra features in more
> >> detail.
> > 
> > All that stuff seems to be fairly old, linux-2.6. isn't mentioned at
> > all and the cvs server doesn't work. Is it still up do date?
> 
> Well, there's some extra stuff, but it's mostly on the iptables side:
> the advanced routing has mostly been stable since not just 2.4 but 2.2!

So I downloaded iproute2-2.4.7-now-ss020116-try.tar.gz, but there
seems to be a problem with errno.h:

make[1]: Entering directory `/home/work/install/src/iproute2/lib'
gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -g -I../include-glibc -I/usr/include/db3 -include ../include-glibc/glibc-bugs.h -I/home/install/w/linux/stable/include -I../include -DRESOLVE_HOSTNAMES -c -o libnetlink.o libnetlink.c
distcc[13445] ERROR: compile
/home/install/w/var/ccache/libnetlink.tmp.p133.13441.i on p133 failed
libnetlink.c: In function `rtnl_dump_filter':
libnetlink.c:149: error: `EINTR' undeclared (first use in this function)
libnetlink.c:149: error: (Each undeclared identifier is reported only once
libnetlink.c:149: error: for each function it appears in.)
libnetlink.c: In function `rtnl_talk':
libnetlink.c:248: error: `EINTR' undeclared (first use in this function)
libnetlink.c: In function `rtnl_listen':
libnetlink.c:350: error: `EINTR' undeclared (first use in this function)
libnetlink.c: In function `rtnl_from_file':
libnetlink.c:416: error: `EINTR' undeclared (first use in this function)
make[1]: *** [libnetlink.o] Error 1
make[1]: Leaving directory `/home/work/install/src/iproute2/lib'
make: *** [all] Error 2

> >> >> mdev is `micro-udev', a 255-line tiny replacement for udev. It's part of
> >> >> busybox.
> >> > 
> >> > Cool. Guess I'll have to update busybox..
> > 
> > done. The new busybox (from SVN) seems to work fine, just like the
> > old one did. The init script doesn't use mdev yet, but from a first
> > reading this is just a matter of translating /etc/udev/udev.conf
> > to the mdev syntax.
> 
> You don't need an mdev.conf at all; by default mdev creates a /dev with
> the KERNEL= names. All it's needed for is putting things in strange
> places or fiddling permissions, and that's not necessary for a boot
> initramfs :)

Nice, and works like a charm. I just removed udev* from the initramfs.

> (I'd recommend managing the *real* /dev with udev, still; it's vastly
> more flexible... 

Yes, and it's needed for hotplugable devices anyway.

> but of course it's also about fifty times larger at a
> horrifying 50K plus 70K of rules...
 
No need for such a huge rules file:

# find  /etc/udev/ -type f -printf '%f %s\n'
udev.conf 768
udev.rules 5200
scsi-model.sh 1326
ide-model.sh 1201

> >> You need SVN uClibc too (if you're using uClibc rather than glibc);
> >> older versions don't maintain the d_type field in the struct dirent, so
> >> mdev's scanning of /sys gets very confused and you end up with an empty
> >> /dev.
> > 
> > Damn. I just compiled 0.9.28. Guess this one is too old.
> 
> Yep. Of course the SVN release has broken binary compatibility, so you
> need to rebuild everything that depends on it (probably the
> cross-toolchain too, for safety). I scripted this long ago, of course,
> because it's a bit annoying otherwise...

I tried to built the cross-compilation toolchain with Buildroot,
but it didn't even start building because it couldn't download gcc
from mirrors.rcn.net which appears to be down ATM. Isn't it possible
to change the gcc mirror? I did not find a config option for that.

Thanks
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: A random initramfs script

[Nix]

On Fri, 17 Mar 2006, Andre Noll stated:
> On 07:50, Nix wrote:
>> If / was a ramfs (as rootfs is), you'd run out of memory...
> 
> Yes, it's an additional piece of rope, and I already used it to shoot
> myself in the foot by doing a backup with "rsync -a /home /mnt" without
> mounting /mnt. First the machine went slow, then the OOM killer kicked
> in and killed everything. Finally the system was totally unresponsible
> and I had to use the "so everything is unusual - boot" thing.

Oops!

> But only root can write to /mnt, and there are much simpler ways for
> root to halt the system..

True. I guess I'm a traditionalist: I'd like / to be a real filesystem
if possible. Both ways work: TMTOWTDI :)

>> Well, there's some extra stuff, but it's mostly on the iptables side:
>> the advanced routing has mostly been stable since not just 2.4 but 2.2!
> 
> So I downloaded iproute2-2.4.7-now-ss020116-try.tar.gz, but there
> seems to be a problem with errno.h:

Holy meatballs that's ancient.

Try <http://developer.osdl.org/dev/iproute2/download/iproute2-2.6.15-060110.tar.gz>
for a rather newer and more capable copy. :)

>> You don't need an mdev.conf at all; by default mdev creates a /dev with
>> the KERNEL= names. All it's needed for is putting things in strange
>> places or fiddling permissions, and that's not necessary for a boot
>> initramfs :)
> 
> Nice, and works like a charm. I just removed udev* from the initramfs.

For those places that are still using udev, if you're running 2.6.15+,
you can soon ditch udevstart, as distros are doing the moral equivalent
of this nowadays:

,----
| udevd --daemon
|
| list=$(echo /sys/bus/*/devices/*/uevent /sys/class/*/*/uevent)
| list=$(echo $list /sys/block/*/uevent /sys/block/*/*/uevent)
| 
| for file in $list; do
|   case "$file" in
|     */device/uevent) ;;        # skip followed device symlinks
|     */\*/*) ;;
|     *" "*) ;;
| 
|     */class/mem/*)             # for /dev/null
|       first="$first $file" ;;
| 
|     */block/md[0-9]*)
|       last="$last $file" ;;
| 
|     *)
|       default="$default $file" ;;
|   esac
| done
| 
| for file in $first $default $last; do
|   echo 'add' > $file
| done
`----

Yes, the initial population of /dev is done by firing messages at udevd
*from a shell script*. It's gone all the way from devfs's kernel-space
hardwiring to something sufficiently extensible that a shell script can
do all the neessary stuff to populate /dev :)

(This is also much faster than udevstart; <1s to populate a crowded /dev
on my P233...)

>> (I'd recommend managing the *real* /dev with udev, still; it's vastly
>> more flexible... 
> 
> Yes, and it's needed for hotplugable devices anyway.

mdev can handle hotplugging: leave the -s off. But yes, udev is probably
preferable on non-space-constrained systems, for much the same reason
that glibc is preferable to uClibc in those situations.

>> but of course it's also about fifty times larger at a
>> horrifying 50K plus 70K of rules...
>  
> No need for such a huge rules file:
> 
> # find  /etc/udev/ -type f -printf '%f %s\n'
> udev.conf 768
> udev.rules 5200
> scsi-model.sh 1326
> ide-model.sh 1201

Er, true. Actually I goofed: I du -s'd the directory and forgot that I
was keeping it in subversion... I'm actually using 10K of rules. Not
a very accurate estimate on my part, that.

[uClibc]
>> Yep. Of course the SVN release has broken binary compatibility, so you
>> need to rebuild everything that depends on it (probably the
>> cross-toolchain too, for safety). I scripted this long ago, of course,
>> because it's a bit annoying otherwise...
> 
> I tried to built the cross-compilation toolchain with Buildroot,
> but it didn't even start building because it couldn't download gcc
> from mirrors.rcn.net which appears to be down ATM. Isn't it possible
> to change the gcc mirror? I did not find a config option for that.

I wouldn't know: I don't use buildroot. Bugging the buildroot author
(who also happens to be the uClibc author) might be a good idea.

Alternatively, just suck down GCC from, say, svn://gcc.gnu.org/svn/gcc/tags/gcc_3_4_5_release,
or ftp.gnu.org, or somewhere, and point buildroot at that. (I hope
you can do that: like I say, I've never done more than suck buildroot
down to pull the uClibc patches out of it and apply them to the
dev trees of binutils and GCC that I already had...)

-- 
`Come now, you should know that whenever you plan the duration of your
 unplanned downtime, you should add in padding for random management
 freakouts.'

Re: A random initramfs script

[Andre Noll]

On 15:23, Neil Brown wrote:

> You shouldn't need portmap to mount an NFS filesystem unless you
> enable locking,

That's news to me, thanks for pointing it out. But I do need portmap
for mounting a NFS filesystem read-only (/usr, which contains
portmap). Is that correct?

> > He likes to compare the situation with /etc/fstab. Nobody complains
> > about having to edit /etc/fstab, so why keep people complaining about
> > having to edit /etc/mdadm.conf?
> 
> Indeed!  And if you plug in some devices off another machine for
> disaster recovery, you don't want another disaster because you
> assembled the wrong arrays.

How is such a disaster possible, given each md device contains an
ID for the array it belongs to? But yes, it is certainly a good
idea to doublecheck everything before assembling the array in such
a recovery situation.

> I would like an md superblock to be able to contain some indication of
> the 'name' of the machine which is meant to host the array, so that
> once a machine knows its own name, it can automatically find and mount
> its own arrays, but that isn't near the top of my list of priorities
> yet.

How about a user-defined name?

	mdadm --create --name the_extra_noisy_array /dev/md0 --level...

would use some fixed algorithm to compute a usual UUID for the new
array from the string "the_extra_noisy_array", and

	mdadm --assemble /dev/md0 --name the_extra_noisy_array

could use the same algorithm and take into account only those devices
which have a UUID equal to the computed one.

Just a thought.
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

Re: A random initramfs script

[Andre Noll]

On 00:41, Nix wrote:

> > So I downloaded iproute2-2.4.7-now-ss020116-try.tar.gz, but there
> > seems to be a problem with errno.h:
> 
> Holy meatballs that's ancient.

It is the most recent version on the ftp server mentioned in the HOWTO.

> Try <http://developer.osdl.org/dev/iproute2/download/iproute2-2.6.15-060110.tar.gz>
> for a rather newer and more capable copy. :)

Much better. Thanks. This version works fine for me, just like busybox
ip does.

> Yes, the initial population of /dev is done by firing messages at udevd
> *from a shell script*. It's gone all the way from devfs's kernel-space
> hardwiring to something sufficiently extensible that a shell script can
> do all the neessary stuff to populate /dev :)

Yeah, Linux rulez :)

> [uClibc]
> 
> Alternatively, just suck down GCC from, say, svn://gcc.gnu.org/svn/gcc/tags/gcc_3_4_5_release,
> or ftp.gnu.org, or somewhere, and point buildroot at that.

Yep, there's a 'dl' directory which contains all downloads. One can
download the tarballs from anywhere else to that directory. Seems to
work now.

Thanks
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

Re: A random initramfs script

[Nix]

On Fri, 17 Mar 2006, Andre Noll murmured woefully:
> On 00:41, Nix wrote:
> 
>> > So I downloaded iproute2-2.4.7-now-ss020116-try.tar.gz, but there
>> > seems to be a problem with errno.h:
>> 
>> Holy meatballs that's ancient.
> 
> It is the most recent version on the ftp server mentioned in the HOWTO.

OK, so I guess the howto is a bit out of date :/

>> [uClibc]
>> 
>> Alternatively, just suck down GCC from, say, svn://gcc.gnu.org/svn/gcc/tags/gcc_3_4_5_release,
>> or ftp.gnu.org, or somewhere, and point buildroot at that.
> 
> Yep, there's a 'dl' directory which contains all downloads. One can
> download the tarballs from anywhere else to that directory. Seems to
> work now.

Ah, I just work from svn checkouts :) but they *are* twice the size of
a normal tarball, so I guess I understand using the tarballs instead :)

-- 
`Come now, you should know that whenever you plan the duration of your
 unplanned downtime, you should add in padding for random management
 freakouts.'