/linuxrc query

/linuxrc query

[Amit D Chaudhary]

Hi,

I have a initrd working, a /linuxrc on it that runs and executes. My question 
for the commands after pivot_root which works like a charm, thanks to initrd.txt,

what does redirecting stdin\stdout\stderr to dev/console achieve? I thought 
since the root is now the "new" root, dev/console will be used automatically? 
Also, why chroot, why not call init directly?
#exec chroot . sbin/init 3 <dev/console >dev/console 2>&1

Since the above never returns, what follows in not freed. Does this mean I have 
around 4-6 mb of ram being used up unnecessarily? Any solution?

#umount /initrd
#blockdev --flushbufs /dev/ram0    # /dev/rd/0 if using devfs


Thanks and Regards
Amit

Re: /linuxrc query

[Werner Almesberger]

Amit D Chaudhary wrote:
> what does redirecting stdin\stdout\stderr to dev/console achieve? I thought 
> since the root is now the "new" root, dev/console will be used automatically?

No, you would continue using the file descriptors which are already
open, i.e. on /dev/console on the old root.

> Also, why chroot, why not call init directly?

To make sure the root of the current process is indeed changed.
pivot_root currently forces a chroot on all processes (except the
ones that have explicitly moved out of /) in order to move all the
kernel threads too, but this is not a nice solution. Once a better
solution is implemented for the kernel threads, we might drop the
forced chroot, and then the explicit chroot here becomes important.

> Since the above never returns, what follows in not freed.

You can run them later, e.g. /etc/rc.d/rc.local
Or, if you needs the space immediately,  make "what-follows" a
script than first frees them, and then exec's init.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, ICA, EPFL, CH           Werner.Almesberger@epfl.ch /
/_IN_N_032__Tel_+41_21_693_6621__Fax_+41_21_693_6610_____________________/

Re: /linuxrc query

[Amit D Chaudhary]

Hi,

Thanks for the response. PSB,

Werner Almesberger wrote:

> Amit D Chaudhary wrote:
> 
> No, you would continue using the file descriptors which are already
> open, i.e. on /dev/console on the old root.
So, makes sense. And the child process that follow will use now the new fd's.

>> Also, why chroot, why not call init directly?
> 
> 
> To make sure the root of the current process is indeed changed.
> pivot_root currently forces a chroot on all processes (except the
> ones that have explicitly moved out of /) in order to move all the
> kernel threads too, but this is not a nice solution. Once a better
> solution is implemented for the kernel threads, we might drop the
> forced chroot, and then the explicit chroot here becomes important.
So, it is not a requirement currently but it is useful to have the script not 
dependent on the current pivot_root implementation.


> You can run them later, e.g. /etc/rc.d/rc.local
> Or, if you needs the space immediately,  make "what-follows" a
> script than first frees them, and then exec's init.
Sure will put in a script that does it. I had left it in /linuxrc as I thought 
that's what initrd.txt suggested one to do. But other information in the 
initrd.txt mentions otherwise, hence the query here.

I am assuming umount and thereby blockdev after pivot_script and before "chroot 
. init ..." don't make sense as files(dev/console among others) are\might still 
be in use.

Best Regards
Amit

Re: /linuxrc query

[Werner Almesberger]

Amit D Chaudhary wrote:
> So, it is not a requirement currently but it is useful to have the script not 
> dependent on the current pivot_root implementation.

Yes. Also note that the relative path for  dev/console  works in
either case, while /dev/console would fail without the implied
chroot in pivot_root.

> But other information in the 
> initrd.txt mentions otherwise, hence the query here.

Hmm, sounds like a bug. Where did you find this ?

> I am assuming umount and thereby blockdev after pivot_script and before
> "chroot . init ..." don't make sense as files(dev/console among others)
> are\might still be in use.

Exactly. They's in use in any case until you close and re-open the
console.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, ICA, EPFL, CH           Werner.Almesberger@epfl.ch /
/_IN_N_032__Tel_+41_21_693_6621__Fax_+41_21_693_6610_____________________/

Re: /linuxrc query

[Amit D Chaudhary]

Werner Almesberger wrote:

> Amit D Chaudhary wrote:
>> But other information in the 
>> initrd.txt mentions otherwise, hence the query here.
> 
> 
> Hmm, sounds like a bug. Where did you find this ?
I quote from the version in linux-2.4.2-ac22
"
Now, the initrd can be unmounted and the memory allocated by the RAM
disk can be freed:

# umount /initrd
# blockdev --flushbufs /dev/ram0    # /dev/rd/0 if using devfs
"

I guess I assumed the above has to be put in /linuxrc as it follows other 
commands which do go in the /linuxrc. The assumption is not quite correct as I 
have since found out.

Regards
Amit

Re: /linuxrc query

[Amit D Chaudhary]

Hi,

Also as a note, what we are doing is keeping our rootfs on flash as a tar.gz and 
reading it and mounting it on a ramfs in the /linuxrc before doing a pivot_root. 
To summarize, pivot_root has been a life saver as the earlier real_root_dev 
might not have been useful in this case.
Not using the ramfs limits for now, will do soon.

Thanks
Amit

Werner Almesberger wrote:

> Amit D Chaudhary wrote:
> 
>> what does redirecting stdin\stdout\stderr to dev/console achieve? I thought 
>> since the root is now the "new" root, dev/console will be used automatically?
> 
> 
> No, you would continue using the file descriptors which are already
> open, i.e. on /dev/console on the old root.
> 
> 
>> Also, why chroot, why not call init directly?
> 
> 
> To make sure the root of the current process is indeed changed.
> pivot_root currently forces a chroot on all processes (except the
> ones that have explicitly moved out of /) in order to move all the
> kernel threads too, but this is not a nice solution. Once a better
> solution is implemented for the kernel threads, we might drop the
> forced chroot, and then the explicit chroot here becomes important.
> 
> 
>> Since the above never returns, what follows in not freed.
> 
> 
> You can run them later, e.g. /etc/rc.d/rc.local
> Or, if you needs the space immediately,  make "what-follows" a
> script than first frees them, and then exec's init.
> 
> - Werner

Re: /linuxrc query

[Werner Almesberger]

Amit D Chaudhary wrote:
> To summarize, pivot_root has been a life saver as the earlier real_root_dev 
> might not have been useful in this case.

The whole old change_root mechanism with real_root_dev is best forgotten
quickly ;-) It's also completely helpless as soon as you fire off some
kernel threads that don't call exit_fs.

> Not using the ramfs limits for now, will do soon.

BTW, if you can't free the RAM disk, you may have to apply
http://icawww1.epfl.ch/~almesber/patches/rdfree

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, ICA, EPFL, CH           Werner.Almesberger@epfl.ch /
/_IN_N_032__Tel_+41_21_693_6621__Fax_+41_21_693_6610_____________________/

Re: /linuxrc query

[David Woodhouse]

amit@muppetlabs.com said:
>  Also as a note, what we are doing is keeping our rootfs on flash as a
> tar.gz and  reading it and mounting it on a ramfs in the /linuxrc
> before doing a pivot_root.  To summarize, pivot_root has been a life
> saver as the earlier real_root_dev  might not have been useful in this
> case. Not using the ramfs limits for now, will do soon.

If you're concerned about memory usage - why untar the whole of your root
filesystem into a ramfs? My preferred solution is to just mount the root
filesystem directly from the flash as cramfs (or JFFS2), with symlinks into a
ramfs for appropriate parts like /tmp and /var.

I suppose the best option is actually to union-mount the ramfs over the 
root, rather than mucking about with symlinks. I just haven't got round to 
doing that yet.

--
dwmw2

RAMFS, CRAMFS and JFFS2(was Re: /linuxrc query)

[Amit D Chaudhary]

Hi David,

I did consider CRAMFS and JFFS2 when it was announced on the mtd list. 
Conserving flash over system ram is more relevant. Our reasons are below:

RAMFS v/s CRAMFS
1. RAMFS is just more stable in terms of less complexity, less bugs reported 
over the time, etc.
2. RAMFS is a fairly robust filesystem and all features required as far as I can 
tell.

RAMFS v/s ext2 on a ramdisk
1. Fixed size. Results in overallocation or more dangerously overshooting the 
size decided earlier during mke2fs.

RAMFS v/s JFFS2
1. JFFS2 was announced around a month ago and few utils were still in 
development, hence it was not there for the first development cycle.
2. Just offhand, joining 100 small files (say in /etc) and then compressing into 
a one file over these 100 files in a JFFS2 fs seems to give better overall space 
usage and probably less flash wear during modifications.
3. JFFS2 does has some compelling reasons, more robustness ofcourse with JFFS2 
over above approach, flash wearing(though would require sufficient free space on 
that mtd partition), etc. Plan to compare these later.

I might be wrong and hence would welcome any suggestions.

Regards
Amit

David Woodhouse wrote:

> amit@muppetlabs.com said:
> 
>>  Also as a note, what we are doing is keeping our rootfs on flash as a
>> tar.gz and  reading it and mounting it on a ramfs in the /linuxrc
>> before doing a pivot_root.  To summarize, pivot_root has been a life
>> saver as the earlier real_root_dev  might not have been useful in this
>> case. Not using the ramfs limits for now, will do soon.
> 
> 
> If you're concerned about memory usage - why untar the whole of your root
> filesystem into a ramfs? My preferred solution is to just mount the root
> filesystem directly from the flash as cramfs (or JFFS2), with symlinks into a
> ramfs for appropriate parts like /tmp and /var.
> 
> I suppose the best option is actually to union-mount the ramfs over the 
> root, rather than mucking about with symlinks. I just haven't got round to 
> doing that yet.
> 
> --
> dwmw2

Re: RAMFS, CRAMFS and JFFS2(was Re: /linuxrc query)

[David Woodhouse]

On Fri, 23 Mar 2001, Amit D Chaudhary wrote:

> Hi David,
> 
> I did consider CRAMFS and JFFS2 when it was announced on the mtd list. 
> Conserving flash over system ram is more relevant. Our reasons are below:
> 
> RAMFS v/s CRAMFS
> 1. RAMFS is just more stable in terms of less complexity, less bugs reported 
> over the time, etc.
> 2. RAMFS is a fairly robust filesystem and all features required as far as I can 
> tell.

I'm not aware of any bugs being found in cramfs recently - unless you 
wanted to use it on Alpha (or anything else where PAGE_SIZE != the 
hard-coded 4096 in mkcramfs.c).

I wouldn't avoid it for those reasons - although if you're _really_ short 
of flash space, the same argument applies as for JFFS2 - a single 
compression stream (tar.gz) will be smaller than compressing individual 
pages like JFFS2 and cramfs do.


> I might be wrong and hence would welcome any suggestions.

Given your stated constraints - you're very short of flash and don't care
too much about the RAM you use, you've may have made the same choice I
would have done.

Bearing in mind that you have to take into account the overhead of the 
initrd which does the untarring - what's the total size of the initrd + 
tarball on the flash, and what size would the corresponding cramfs be?

If you could fit your root filesystem into a cramfs on the flash, I'd do
that instead and use ramfs for the parts which need to be writeable.


-- 
dwmw2

Re: CRAMFS

[Bjorn Wesen]

On Fri, 23 Mar 2001, David Woodhouse wrote:
> > 1. RAMFS is just more stable in terms of less complexity, less bugs reported 
> > over the time, etc.
> > 2. RAMFS is a fairly robust filesystem and all features required as far as I can 
> > tell.

Ok, ramfs is really simple, but heck, cramfs is not much more complex :)
It's as simple a flash-filesystem as you can get.

I don't know why the comparision is made though, they are used for two
completely different things... ramfs is for temporary file storage, cramfs
is for immutable files stored on flash. Each by itself is quite optimal
for what it's designed for, isn't it ?

> I'm not aware of any bugs being found in cramfs recently - unless you 
> wanted to use it on Alpha (or anything else where PAGE_SIZE != the 
> hard-coded 4096 in mkcramfs.c).

I committed a patch that disappeared that added the choice of page size
(trivial yes :), we have PAGE_SIZE == 8192 on our systems. Works fine.

> I wouldn't avoid it for those reasons - although if you're _really_ short 
> of flash space, the same argument applies as for JFFS2 - a single 
> compression stream (tar.gz) will be smaller than compressing individual 
> pages like JFFS2 and cramfs do.

Here are some results from a quite mixed filesystem:

[bjornw@godzilla linux]$ ls -l cram*
-rw-r--r--   1 bjornw   users     1179648 Mar 23 22:38 cram32768
-rw-r--r--   1 bjornw   users     1282048 Mar 23 22:38 cram4096
-rw-r--r--   1 bjornw   users     1220608 Mar 23 22:38 cram8192

(the numbers correspond to blocksize)

There's not any big difference here. 

With bigger files though, the difference get larger. YMMV.

Notice that you can change cramfs so it uses a blocksize that is bigger
than PAGE_SIZE, of course, if it really is necessary. You'll get worse
performance at run-time though since you need to cache the page and hope
for read-ahead or similar (you can stuff the pages in the page-cache even
if they are not requested for example).

-BW

Re: CRAMFS

[Amit D Chaudhary]

> I don't know why the comparision is made though, they are used for two
> completely different things... ramfs is for temporary file storage, cramfs
> is for immutable files stored on flash. Each by itself is quite optimal
> for what it's designed for, isn't it ?

Exactly. My mistake earlier to assume cramfs was "compressed ramfs"! ;-)
I should compare it to the tar.gz option and JFFS2. Will do in the next 
evaluation.
This will be more of a replace initrd+custom /linuxrc with a 
CRAMFS-based rootfs on a flash device assuming CRAMFS can be directly 
read by kernel\init for getting the rootfs. Ditto for JFFS2

Also, the platform is PPC, IBM 405GP to be precise.

Regards
Amit

cramfs

[Trever L. Adams]

I hate to ask this, however here goes.  I am doing some remote upgrading 
and some other really funky stuff to some boxes I keep up.

Part of these are total system upgrades and I need to move data out of 
the way while still having a working box.  I decided that cramfs may be 
the way to do this.  If you can tell me no and point me to a resource on 
how to do this, I would LOVE to hear about it.

However, the question is, how can I tell lilo to tell the kernel too 
boot off a cramfs file system?  I have already created the file with 
/etc /bin /sbin /dev and /lib from a working system, doing the correct 
deletions and other such changes.  I have a 15 meg cramfs that should do 
the trick.

Thank you for any help you can offer.

Trever Adams

Re: cramfs

[Daniel Quinlan]

"Trever L. Adams" <vichu@digitalme.com> writes:

> I hate to ask this, however here goes.  I am doing some remote upgrading 
> and some other really funky stuff to some boxes I keep up.
> 
> Part of these are total system upgrades and I need to move data out of 
> the way while still having a working box.  I decided that cramfs may be 
> the way to do this.  If you can tell me no and point me to a resource on 
> how to do this, I would LOVE to hear about it.
> 
> However, the question is, how can I tell lilo to tell the kernel too 
> boot off a cramfs file system?  I have already created the file with 
> /etc /bin /sbin /dev and /lib from a working system, doing the correct 
> deletions and other such changes.  I have a 15 meg cramfs that should do 
> the trick.
> 
> Thank you for any help you can offer.

You want "cramfsboot" to boot cramfs root partitions directly.
(You'll also need the cramfs patch which is part of the Midori Linux
kernel package.)

ftp://midori.transmeta.com/midori-1.0.0-beta2/apps/cramfsboot-0.2_ML1.0.0-beta2-5.mlz

(It's a tarball.)

There are also several packages that handle upgrades of cramfs
partitions, etc.

- Dan

CRAMFS

[Bradley Bozarth]

Daniel Quinlan originally maintained this, now it is orphaned.  His endian
patch, which implemented the correct behavior according to the docs (which
basically listed always do little endian as a todo), was dropped.

We have been maintaining this patch on our kernel, but it really should go
in - I don't want to spend a ton of time like I did last time to have it
dropped again, however - is anyone thinking of maintaining cramfs?  What
are the chances of the endian fix going in if I submit it again?  (if even
the former maintainer had no success).  I would maintain cramfs if desired
- it hasn't really changed in a long time except in regards to higher
level fs changes.

Thanks,
Brad

Re: CRAMFS

[H. Peter Anvin]

Followup to:  <35438.128.107.165.13.1068235628.squirrel@mail.yumbrad.com>
By author:    "Bradley Bozarth" <prettygood@cs.stanford.edu>
In newsgroup: linux.dev.kernel
>
> Daniel Quinlan originally maintained this, now it is orphaned.  His endian
> patch, which implemented the correct behavior according to the docs (which
> basically listed always do little endian as a todo), was dropped.
> 
> We have been maintaining this patch on our kernel, but it really should go
> in - I don't want to spend a ton of time like I did last time to have it
> dropped again, however - is anyone thinking of maintaining cramfs?  What
> are the chances of the endian fix going in if I submit it again?  (if even
> the former maintainer had no success).  I would maintain cramfs if desired
> - it hasn't really changed in a long time except in regards to higher
> level fs changes.
> 

I think Al Viro has been doing a rewrite.  You may want to check with him.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
If you send me mail in HTML format I will assume it's spam.
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64