Busy on BLKFLSBUF w/initrd

Busy on BLKFLSBUF w/initrd

[Maciek Nowacki]

Hi,

I have continuing problems with getting the initrd ramdisk out of memory once
bootup is complete.

This is with recent -ac kernels which have the fix-up posted a few months ago
applied.

The sequence is roughly:

- boot via pxelinux, loads up bzImage <1MB and root.romfs.gz ~7MB, expands to
  about 30.
- mount -n -t tmpfs none tmp
- cp root -> /tmp
- pivot_root -> /tmp
- umount old_root/dev (devfs kernel)
- umount old_root

Then another pivot_root a bit later on before /sbin/init is invoked. The
point is to move the initrd to a tmpfs which will reclaim memory in the event
that the filesystem cannot be unmounted, due to having mountpoints for
loopback host filesystems in this case.

It seems to work fine, and the initrd _does_ unmount with no problems
(unmounting old_root/dev does complain about mtab, but works, ls old_root/dev
shows empty). However, blockdev --flushbufs /dev/rd/0 fails with busy on
BLKFLSBUF once the system is running.

About the only odd thing besides the message about mtab from umount is the
kernel notice right before entering /bin/sh on the romfs initrd - it prints
out two messages about mounting the ramdisk:

May 22 09:14:30 wintermute kernel: Linux version 2.4.4-ac12 (maciek@wintermute) (gcc version 2.95.4 20010506 (Debian prerelease)) #1 Mon May 21 20:08:36 MDT 2001
[...]
May 22 09:14:31 wintermute kernel: RAMDISK: romfs filesystem found at block 0
May 22 09:14:31 wintermute kernel: RAMDISK: Loading 28216 blocks [1 disk] into ram disk... done.
May 22 09:14:31 wintermute kernel: Freeing initrd memory: 28216k freed
May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
May 22 09:14:31 wintermute kernel: change_root: old root has d_count=7
May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
May 22 09:14:31 wintermute kernel: Trying to unmount old root ... okay

Perhaps they're bumping up the reference count so that it is impossible to
free the ramdisk later?

Thanks very much for any help!

Maciek

Re: Busy on BLKFLSBUF w/initrd

[Alexander Viro]

On Wed, 23 May 2001, Maciek Nowacki wrote:

> May 22 09:14:31 wintermute kernel: RAMDISK: romfs filesystem found at block 0
> May 22 09:14:31 wintermute kernel: RAMDISK: Loading 28216 blocks [1 disk] into ram disk... done.
> May 22 09:14:31 wintermute kernel: Freeing initrd memory: 28216k freed
> May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
> May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
> May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
> May 22 09:14:31 wintermute kernel: change_root: old root has d_count=7
> May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
> May 22 09:14:31 wintermute kernel: Trying to unmount old root ... okay

At that point /dev/ram0 _is_ killed.

> Perhaps they're bumping up the reference count so that it is impossible to
> free the ramdisk later?

If you want to keep it until later (i.e. want to destiry it by hands)
mkdir /initrd on your final root and old one will be remounted there.
Again, "Trying to unmount old root ... okay" means that it already got
an equivalent of BKLFLSBUF

I wonder why does it get -EBUSY if you repeat it... Uh-oh...

Folks, who the hell is responsible for rd_inodes[] idiocy?

Re: Busy on BLKFLSBUF w/initrd

[Maciek Nowacki]

On Wed, May 23, 2001 at 06:05:52PM -0400, Alexander Viro wrote:
> 
> 
> On Wed, 23 May 2001, Maciek Nowacki wrote:
> 
> > May 22 09:14:31 wintermute kernel: RAMDISK: romfs filesystem found at block 0
> > May 22 09:14:31 wintermute kernel: RAMDISK: Loading 28216 blocks [1 disk] into ram disk... done.
> > May 22 09:14:31 wintermute kernel: Freeing initrd memory: 28216k freed
> > May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
> > May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
> > May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
> > May 22 09:14:31 wintermute kernel: change_root: old root has d_count=7
> > May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
> > May 22 09:14:31 wintermute kernel: Trying to unmount old root ... okay
> 
> At that point /dev/ram0 _is_ killed.

Really? Because I can still mount it later on without any trouble at all..
after init has started. Hmm, I will try later after running the system for a
while to see if the data is still comprehensible.

> > Perhaps they're bumping up the reference count so that it is impossible to
> > free the ramdisk later?
> 
> If you want to keep it until later (i.e. want to destiry it by hands)
> mkdir /initrd on your final root and old one will be remounted there.
> Again, "Trying to unmount old root ... okay" means that it already got
> an equivalent of BKLFLSBUF

Ah, okay.. I assumed this behavior had been removed. I will try this as well.

Maciek

Re: Busy on BLKFLSBUF w/initrd

[Alexander Viro]

On Wed, 23 May 2001, Maciek Nowacki wrote:

> > If you want to keep it until later (i.e. want to destiry it by hands)
> > mkdir /initrd on your final root and old one will be remounted there.
> > Again, "Trying to unmount old root ... okay" means that it already got
> > an equivalent of BKLFLSBUF
> 
> Ah, okay.. I assumed this behavior had been removed. I will try this as well.

change_root() in 2.4.4 gives you explicit destroy_buffers(). In 2.4.5-pre5
it simply does BLKFLSBUF - calls ioctl_by_bdev(). And BLKFLSBUF boils
down to destroy_buffers().

I would really like to hear details re survival of the initrd contents.
I've looked at the way rd.c "protects" the data and it seems to be
b0rken - playing games with igrab() is not a good idea for driver...

Re: Busy on BLKFLSBUF w/initrd

[Maciek Nowacki]

On Wed, May 23, 2001 at 06:21:23PM -0400, Alexander Viro wrote:
> 
> 
> On Wed, 23 May 2001, Maciek Nowacki wrote:
> 
> > > If you want to keep it until later (i.e. want to destiry it by hands)
> > > mkdir /initrd on your final root and old one will be remounted there.
> > > Again, "Trying to unmount old root ... okay" means that it already got
> > > an equivalent of BKLFLSBUF
> > 
> > Ah, okay.. I assumed this behavior had been removed. I will try this as well.
> 
> change_root() in 2.4.4 gives you explicit destroy_buffers(). In 2.4.5-pre5
> it simply does BLKFLSBUF - calls ioctl_by_bdev(). And BLKFLSBUF boils
> down to destroy_buffers().
> 
> I would really like to hear details re survival of the initrd contents.
> I've looked at the way rd.c "protects" the data and it seems to be
> b0rken - playing games with igrab() is not a good idea for driver...

I wrote out the contents of /dev/rd/0 a few times and diff'ed with the
uncompressed image of the initrd on the server. No difference each time. The
same after digging into swap, turning off swap, running blockdev --flushbufs
several times (always with BLKFLSBUF: Device or resource busy).

The next test will be to create an initrd that has the 'initrd' directory..

(this is all with 2.4.4-ac13 +crypto-2.4.3.1 +alsa-cvs)

Maciek

Re: Busy on BLKFLSBUF w/initrd

[Alexander Viro]

On Wed, 23 May 2001, Maciek Nowacki wrote:

> On Wed, May 23, 2001 at 06:21:23PM -0400, Alexander Viro wrote:
> I wrote out the contents of /dev/rd/0 a few times and diff'ed with the
> uncompressed image of the initrd on the server. No difference each time. The
> same after digging into swap, turning off swap, running blockdev --flushbufs
> several times (always with BLKFLSBUF: Device or resource busy).
> 
> The next test will be to create an initrd that has the 'initrd' directory..

Not initrd with /initrd in it, final root with /initrd, so that change_root()
had a place to remount the thing on.

Re: Busy on BLKFLSBUF w/initrd

[Maciek Nowacki]

On Wed, May 23, 2001 at 07:28:14PM -0400, Alexander Viro wrote:
> 
> 
> On Wed, 23 May 2001, Maciek Nowacki wrote:
> 
> > On Wed, May 23, 2001 at 06:21:23PM -0400, Alexander Viro wrote:
> > I wrote out the contents of /dev/rd/0 a few times and diff'ed with the
> > uncompressed image of the initrd on the server. No difference each time. The
> > same after digging into swap, turning off swap, running blockdev --flushbufs
> > several times (always with BLKFLSBUF: Device or resource busy).
> > 
> > The next test will be to create an initrd that has the 'initrd' directory..
> 
> Not initrd with /initrd in it, final root with /initrd, so that change_root()
> had a place to remount the thing on.

Yeah, but.. I get the message about change_root() before I have a chance to
do anything (there is no /linuxrc on my initrd). I could try with a linuxrc,
but I wanted to avoid that since since it seemed to be becoming obsolete.

Maciek

Re: Busy on BLKFLSBUF w/initrd

[Mike Galbraith]

On Wed, 23 May 2001, Alexander Viro wrote:

> Folks, who the hell is responsible for rd_inodes[] idiocy?

That would have been me.  It was simple and needed at the time..
feel free to rip it up :)

	-Mike

Re: Busy on BLKFLSBUF w/initrd

[Alexander Viro]

On Thu, 24 May 2001, Mike Galbraith wrote:

> On Wed, 23 May 2001, Alexander Viro wrote:
> 
> > Folks, who the hell is responsible for rd_inodes[] idiocy?
> 
> That would have been me.  It was simple and needed at the time..
> feel free to rip it up :)

Mike, I see what you are using it for, but you do realize that it
means that creating /tmp/ram0 and opening it once will make /tmp
impossible to unmount?

Re: Busy on BLKFLSBUF w/initrd

[Mike Galbraith]

On Thu, 24 May 2001, Alexander Viro wrote:

> On Thu, 24 May 2001, Mike Galbraith wrote:
>
> > On Wed, 23 May 2001, Alexander Viro wrote:
> >
> > > Folks, who the hell is responsible for rd_inodes[] idiocy?
> >
> > That would have been me.  It was simple and needed at the time..
> > feel free to rip it up :)
>
> Mike, I see what you are using it for, but you do realize that it
> means that creating /tmp/ram0 and opening it once will make /tmp
> impossible to unmount?

I don't _think_ that was the case at the time I did it.  I tested
the idio^Wbandaid before submission.. mighta fscked up though :)

	-Mike

Re: Busy on BLKFLSBUF w/initrd

[Alexander Viro]

On Thu, 24 May 2001, Mike Galbraith wrote:

> On Thu, 24 May 2001, Alexander Viro wrote:
> 
> > On Thu, 24 May 2001, Mike Galbraith wrote:
> >
> > > On Wed, 23 May 2001, Alexander Viro wrote:
> > >
> > > > Folks, who the hell is responsible for rd_inodes[] idiocy?
> > >
> > > That would have been me.  It was simple and needed at the time..
> > > feel free to rip it up :)
> >
> > Mike, I see what you are using it for, but you do realize that it
> > means that creating /tmp/ram0 and opening it once will make /tmp
> > impossible to unmount?
> 
> I don't _think_ that was the case at the time I did it.  I tested
> the idio^Wbandaid before submission.. mighta fscked up though :)

Erm... You pin the inode down. That makes filesystem busy by any
definition I can think of...

Re: Busy on BLKFLSBUF w/initrd

[Herbert Xu]

Alexander Viro <viro@math.psu.edu> wrote:

> Erm... You pin the inode down. That makes filesystem busy by any
> definition I can think of...

That's just because the 2.4.4 code doesn't release it with a blkdev_put.
The fix is in the ac patches.
-- 
Debian GNU/Linux 2.2 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Busy on BLKFLSBUF w/initrd

[Mike Galbraith]

On Thu, 24 May 2001, Alexander Viro wrote:

> On Thu, 24 May 2001, Mike Galbraith wrote:
>
> > On Thu, 24 May 2001, Alexander Viro wrote:
> >
> > > On Thu, 24 May 2001, Mike Galbraith wrote:
> > >
> > > > On Wed, 23 May 2001, Alexander Viro wrote:
> > > >
> > > > > Folks, who the hell is responsible for rd_inodes[] idiocy?
> > > >
> > > > That would have been me.  It was simple and needed at the time..
> > > > feel free to rip it up :)
> > >
> > > Mike, I see what you are using it for, but you do realize that it
> > > means that creating /tmp/ram0 and opening it once will make /tmp
> > > impossible to unmount?
> >
> > I don't _think_ that was the case at the time I did it.  I tested
> > the idio^Wbandaid before submission.. mighta fscked up though :)
>
> Erm... You pin the inode down. That makes filesystem busy by any
> definition I can think of...

Yes. I pulled the pins when I was done with them though.
(at least I think I did.. been a long time)

	-Mike

Re: Busy on BLKFLSBUF w/initrd

[Alexander Viro]

On Thu, 24 May 2001, Mike Galbraith wrote:

> > Erm... You pin the inode down. That makes filesystem busy by any
> > definition I can think of...
> 
> Yes. I pulled the pins when I was done with them though.
> (at least I think I did.. been a long time)

Only when you do rd_cleanup().

Re: Busy on BLKFLSBUF w/initrd

[Maciek Nowacki]

Problem seems to be solved. Here is what I did, for anyone who is interested
in using a loopback file on a local disk as root:

- create a romfs initrd with /linuxrc -> /bin/sh and an empty /initrd dir

- boot, with root=/dev/rd/1 init=/bin/sh

- when /linuxrc comes up, cat /dev/rd/0 > /dev/rd/1

- CTRL-D, exit, etc. to invoke change_root -> /dev/rd/1

- umount /initrd/dev and /initrd

- load modules needed for boot.. ext2, ide, loopback, crypto, and sticky
order-dependent things (in my case, uhci MUST precede eepro100)

- mount -n -t tmpfs none /tmp

- copy needed things to /tmp like /bin, /sbin, /lib (not modules, unless
memory isn't a problem)

- mount -n -t devfs none /tmp/dev

- cd /tmp ; pivot_root . old_root ; chroot bin/sh ...

- now we are chrooted into bin/sh on a tmpfs root

- umount old_root/dev ; umount old_root

- mount real filesystem - mount host on /mnt, loopback on /real_root

- finally chroot into real_root/sbin/init

- once system is running, clear out unneeded files in tmpfs, unmount the
  devfs - memory usage goes to almost nothing (tmpfs is still needed since
  the host for the loopback lives there)

- blockdev --flushbufs /dev/rd/*

(note, I have glossed over some details. /proc is needed in several steps,
and must be unmounted and remounted and so on..)

After running for a while and filling up memory, buffers has gone down far
below the size of the ramdisk, and trying to mount /dev/rd/0 and /dev/rd/1
shows that they are no longer valid filesystems.

This method depends on the change_root() mechanism which I had assumed is
becoming obsolete. It works, and there is no need to mess with
/proc/sys/kernel/real_root_dev if the root is specified on the command line.
Trying to use only pivot_root did not work as /dev/rd/0 could never be
flushed (see previous messages in this thread).

Thanks very much to everyone who helped!

Maciek

Re: Busy on BLKFLSBUF w/initrd

[Scott Murray]

On Thu, 24 May 2001, Maciek Nowacki wrote:

> Problem seems to be solved. Here is what I did, for anyone who is interested
> in using a loopback file on a local disk as root:
[snip recipe]
> This method depends on the change_root() mechanism which I had assumed is
> becoming obsolete. It works, and there is no need to mess with
> /proc/sys/kernel/real_root_dev if the root is specified on the command line.
> Trying to use only pivot_root did not work as /dev/rd/0 could never be
> flushed (see previous messages in this thread).

I was having similiar problems a few months back.  I was also trying
to pivot_root out of an initial ramdisk and then unmount it.  I got it
working, but kept forgetting to post one of the fixes that I found
necessary to make it work when using auto-mounted devfs.

Here it is (exported from BitKeeper, hence the a and b):

diff -Nru a/init/main.c b/init/main.c
--- a/init/main.c	Fri May 25 00:31:25 2001
+++ b/init/main.c	Fri May 25 00:31:25 2001
@@ -273,6 +273,9 @@
 #ifdef CONFIG_NFTL
 	{ "nftla", 0x5d00 },
 #endif
+#ifdef CONFIG_DEVFS_MOUNT
+	{ "rd/",     0x0100 },
+#endif
 	{ NULL, 0 }
 };

What this does is make /dev/rd/* parseable by name_to_kdev_t, with
the result that ROOT_DEV gets set correctly in root_dev_setup and
the initrd logic at the bottom of do_basic_setup doesn't automatically
trigger the change_root.  The same effect can be achieved by setting
real_root_dev in proc in your ramdisk, but since that's not required
for non-devfs ramdisks, it takes some digging for the uninitiated to
figure out why things don't work.

With this fix and the fix for the missing bldev_put in drivers/block/rd.c
(which has been in Alan's tree for ages), the pivot_root instructions
given in the file Documentation/initrd.txt have been working fine for me.

Scott


-- 
=============================================================================
Scott Murray                                        email: scott@spiteful.org
http://www.spiteful.org (coming soon)                 ICQ: 10602428
-----------------------------------------------------------------------------
     "Good, bad ... I'm the guy with the gun." - Ash, "Army of Darkness"

Re: Busy on BLKFLSBUF w/initrd

[Maciek Nowacki]

On Fri, May 25, 2001 at 01:20:00AM -0400, Scott Murray wrote:
> On Thu, 24 May 2001, Maciek Nowacki wrote:
> 
> > This method depends on the change_root() mechanism which I had assumed is
> > becoming obsolete. It works, and there is no need to mess with
> > /proc/sys/kernel/real_root_dev if the root is specified on the command line.
> > Trying to use only pivot_root did not work as /dev/rd/0 could never be
> > flushed (see previous messages in this thread).
> 
> I was having similiar problems a few months back.  I was also trying
> to pivot_root out of an initial ramdisk and then unmount it.  I got it
> working, but kept forgetting to post one of the fixes that I found
> necessary to make it work when using auto-mounted devfs.
>
> [patch to init/main.c]

Cool, that did it. change_root doesn't occur anymore and BLKFLSBUF clears
things out of memory without any muckage. I will add this one to my patch
directory. Thanks!

Maciek