RE: Unable to mount compressed root f/s because init_mtd() does n ot r egister block device!

RE: Unable to mount compressed root f/s because init_mtd() does n ot r egister block device!

[Vipin Malik]

Ok, looking a bit more at it, it seems that all the init for 
the mtd stuff is happening when do_inicalls() gets called as
they are all declared as initcalls.

But looking at the other block device stuff (hd, floppy etc.)
they all get init'ed by calling blk_dev_init() from genhd.c which
gets called by device_setup() called from init/main.c
 

This guarantees that the block devices are registered before
the ramdisk load stuff gets called, hence allowing them to be used
to load compressed filesystems.

This should be quite important to all as flash memory is considerably
more expensive than DRAM, hence the need to load compressed root
file systems from flash into RAMDISK.

How to we fix this (as I'm not too familiar with the init stuff and 
how to change that)?

Thanks,

Vipin

-----Original Message-----
From: Vipin Malik
To: 'mtd@infradead.org'
Sent: 2/21/01 10:19 PM
Subject: Unable to mount compressed root f/s because init_mtd() does not r
egister block device!

In 2.2.18 patched with the 2.2.18 patch, I'm trying to boot
a kernel from raw (CFI) flash and decompress an ext2 root
f/s image (from /dev/mtdblock0) into ramdisk and mount it as root.

Everything goes fine, (kernel boots etc.) but when the load ramdisk
code runs, it fails with a -ENODEV when it tries to open
the root device (to load the compressed root file system, present
on /dev/mtdblock0- right after the kernel).

It fails because, even though init_mtd() has been already
called before the rd_load() call in init/main.c, init_mtd()
does NOT call init_mtdblock() which actually registers the
mtdblock device.
This is done much later when the file system is actually installed.

Why? Any particular reason or no one tried this before.

Thanks

Vipin


To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org


To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

Re: Unable to mount compressed root f/s because init_mtd() does n ot r egister block device!

[David Woodhouse]

Vipin.Malik@daniel.com said:
> Ok, looking a bit more at it, it seems that all the init for  the mtd
> stuff is happening when do_inicalls() gets called as they are all
> declared as initcalls.

> But looking at the other block device stuff (hd, floppy etc.) they all
> get init'ed by calling blk_dev_init() from genhd.c which gets called
> by device_setup() called from init/main.c
>  

> This guarantees that the block devices are registered before the
> ramdisk load stuff gets called, hence allowing them to be used to load
> compressed filesystems.

> How to we fix this (as I'm not too familiar with the init stuff and
> how to change that)? 

Either we stop using initcalls for the MTD stuff in 2.2.18 - just revert to 
the old method as we did in earlier 2.2, or we fix the remainder of 2.2.19 
to use initcalls properly. Probably the former.

--
dwmw2




To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

RE: Unable to mount compressed root f/s because init_mtd() does n ot r egister block device!

[Vipin Malik]

David,
But isn't the same init calls method used to init the stuff in the
2.4.x series of kernels. So it's not just a 2.2.18/19 issue.

Thanks

Vipin
P.S. Did you look at the 2nd patch to inode-v23.c that I sent yesterday
(to fix the partial page write stuff)?


-----Original Message-----
From: David Woodhouse
To: Vipin Malik
Cc: ''mtd@infradead.org' '
Sent: 2/22/01 2:08 AM
Subject: Re: Unable to mount compressed root f/s because init_mtd() does n
ot r egister block device! 


Vipin.Malik@daniel.com said:
> Ok, looking a bit more at it, it seems that all the init for  the mtd
> stuff is happening when do_inicalls() gets called as they are all
> declared as initcalls.

> But looking at the other block device stuff (hd, floppy etc.) they all
> get init'ed by calling blk_dev_init() from genhd.c which gets called
> by device_setup() called from init/main.c
>  

> This guarantees that the block devices are registered before the
> ramdisk load stuff gets called, hence allowing them to be used to load
> compressed filesystems.

> How to we fix this (as I'm not too familiar with the init stuff and
> how to change that)? 

Either we stop using initcalls for the MTD stuff in 2.2.18 - just revert
to 
the old method as we did in earlier 2.2, or we fix the remainder of
2.2.19 
to use initcalls properly. Probably the former.

--
dwmw2




To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org


To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

Re: Unable to mount compressed root f/s because init_mtd() does n ot r egister block device!

[David Woodhouse]

Vipin.Malik@daniel.com said:
> But isn't the same init calls method used to init the stuff in the
> 2.4.x series of kernels. So it's not just a 2.2.18/19 issue.

Yes, but it works in 2.4, and it doesn't in 2.2.18. So we might as well just
use the 2.2.17 method in 2.2.18 instead of a hacked-up version of the 2.4
method.

> Vipin P.S. Did you look at the 2nd patch to inode-v23.c that I sent
> yesterday (to fix the partial page write stuff)?

Yeah - looks good, thanks. Can you commit it?

--
dwmw2




To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

Re: Unable to mount compressed root f/s because init_mtd() does n ot r egister block device!

[David Woodhouse]

vmalik@danielind.com said:
>  the ramdisk driver cannot find the mtdblock device registered when it
> tries to load the compressed f/s image from flash.

Hmmm. drivers/mtd is later than drivers/block in the link order. Try 
changing that in the toplevel Makefile.

--
dwmw2




To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

Re: Unable to mount compressed root f/s because init_mtd() does n ot r egister block device!

[Vipin Malik]

David Woodhouse wrote:

> Vipin.Malik@daniel.com said:
> > But isn't the same init calls method used to init the stuff in the
> > 2.4.x series of kernels. So it's not just a 2.2.18/19 issue.
>
> Yes, but it works in 2.4, and it doesn't in 2.2.18. So we might as well just
> use the 2.2.17 method in 2.2.18 instead of a hacked-up version of the 2.4
> method.

Has anyone really tried this in 2.4.x ?
I've been playing all morning with the 2.4.1 kernel trying to get the physmap.o
__init stuff to run before the rd_load() and
just can't seem to get it done.

the ramdisk driver cannot find the mtdblock device registered when it tries to
load the compressed f/s image from flash.

Help!

Vipin


>
>
> > Vipin P.S. Did you look at the 2nd patch to inode-v23.c that I sent
> > yesterday (to fix the partial page write stuff)?
>
> Yeah - looks good, thanks. Can you commit it?
>
> --
> dwmw2



To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

Re: Unable to mount compressed root f/s because init_mtd() does n ot r egister block device!

[Vipin Malik]

Nope, doesn't work. Already tried that this morning (along with declaring
physmap.o as an OX_OBJS).

Vipin

David Woodhouse wrote:

> vmalik@danielind.com said:
> >  the ramdisk driver cannot find the mtdblock device registered when it
> > tries to load the compressed f/s image from flash.
>
> Hmmm. drivers/mtd is later than drivers/block in the link order. Try
> changing that in the toplevel Makefile.
>
> --
> dwmw2
>
> To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org



To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

Re: Unable to mount compressed root f/s because init_mtd() does n ot r egister block device!

[Vipin Malik]

Success!! Well sort of ;)

After a terrible hack of explicitly calling rd_load() again at the end of
init_mtdblock(), forces the ramdisk stuff to look for the compressed root
file system again. This time it
finds it (now that mtdblock has registered the block device) enabling the
root file system to be loaded and be mounted!

This proves for sure that that is the only issue left to boot compressed root
file systems off raw flash. Once we figure out how to force init_mtdblock()
to be linked in
before the ram disk stuff, we would be set.

David? :_)

Vipin





Vipin Malik wrote:

> Nope, doesn't work. Already tried that this morning (along with declaring
> physmap.o as an OX_OBJS).
>
> Vipin
>
> David Woodhouse wrote:
>
> > vmalik@danielind.com said:
> > >  the ramdisk driver cannot find the mtdblock device registered when it
> > > tries to load the compressed f/s image from flash.
> >
> > Hmmm. drivers/mtd is later than drivers/block in the link order. Try
> > changing that in the toplevel Makefile.
> >
> > --
> > dwmw2
> >
> > To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org
>
> To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org



To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

Re: Unable to mount compressed root f/s because init_mtd() does n ot r egister block device!

[David Woodhouse]

On Thu, 22 Feb 2001, Vipin Malik wrote:

> Success!! Well sort of ;)
> 
> After a terrible hack of explicitly calling rd_load() again at the end
> of init_mtdblock(), forces the ramdisk stuff to look for the
> compressed root file system again. This time it finds it (now that
> mtdblock has registered the block device) enabling the root file
> system to be loaded and be mounted!

Heh. Well done.

> This proves for sure that that is the only issue left to boot
> compressed root file systems off raw flash. Once we figure out how to
> force init_mtdblock() to be linked in before the ram disk stuff, we
> would be set.

Cool. You'll have to hurry, though - We'll have JFFS2 (with compression) 
ready to roll out quite soon :)

-- 
dwmw2




To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

Re: Unable to mount compressed root f/s because init_mtd() does not r egister block device!

[Vipin Malik]

David Woodhouse wrote:

> On Thu, 22 Feb 2001, Vipin Malik wrote:
>
> > Success!! Well sort of ;)
> >
> > After a terrible hack of explicitly calling rd_load() again at the end
> > of init_mtdblock(), forces the ramdisk stuff to look for the
> > compressed root file system again. This time it finds it (now that
> > mtdblock has registered the block device) enabling the root file
> > system to be loaded and be mounted!
>
> Heh. Well done.

Thanks.

>
>
> > This proves for sure that that is the only issue left to boot
> > compressed root file systems off raw flash. Once we figure out how to
> > force init_mtdblock() to be linked in before the ram disk stuff, we
> > would be set.
>
> Cool. You'll have to hurry, though -

Are you saying that you'll not fix this "bug" for me ;)
Would saying please help :)

> We'll have JFFS2 (with compression)
>ready to roll out quite soon :)

Well, there still would be reasons to decompress a root file system into
ramdisk and run from there, namely:

1. Processor cache: Most likely you would have turned off the processor
cache on the flash window, to enable writing.
A cached (or possibly even uncached) DRAM would be faster than
FLASH (specially if one saved some money and
bought those 120ns ones :) Additionally many high end embedded processors
have (or will have in the future), write merging and
read ahead that you want turned off again so as not to screw up write and
erase cycles.

2. Reliability: A wayward program is less likely to completely screw up the
system. A reboot would rebuild the root file system
to a pristine state.

3. Compressibility: I would hazard a guess that the compressibility
obtained with compressing an entire file system image would
be better than that obtained with just compressing individual inode data.

4. For faster access times, you want the flash connected across the entire
bus data width. Say using x8 chips with 64KB sectors,
and 2 sectors min erase area, you are looking at 512KBytes of non usable
flash space. This is less of an issue with compressed fs!
One can connect just a x8 chip and have the performance of running out of
dram and the small overhead of 2*64=128KBytes erase
buffer available.

5. Compressed root file systems will cook and do your laundry for you, will
jffs2 do that?

:)

Vipin

>
>
> --
> dwmw2



To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

Re: Unable to mount compressed root f/s because init_mtd() does not r egister block device!

[David Woodhouse]

vmalik@danielind.com said:
>  Are you saying that you'll not fix this "bug" for me ;) Would saying
> please help :) 

Heh. Of course I'll help. Not sure how best to go about it, though. Init 
order dependencies are ugly. rd_load() should probably be one of the last 
things in the boot sequence, rather than an initcall. Can you add it to 
init/main.c just before mount_root()?



> > We'll have JFFS2 (with compression) ready to roll out quite soon :)

> Well, there still would be reasons to decompress a root file system
> into ramdisk and run from there, namely: 

Definitely. Just pushing my new baby, that's all :)


>  3. Compressibility: I would hazard a guess that the compressibility
> obtained with compressing an entire file system image would be better
> than that obtained with just compressing individual inode data. 

Yep. Actually we only compress a single page at a time 
in jffs2, not even the whole inode.


--
dwmw2




To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

Re: Unable to mount compressed root f/s because init_mtd() does not r egister block device!

[David Woodhouse]

vmalik@danielind.com said:
>  Do you have any figures as to how well it compresses vs say gzip -9?

Not wonderfully, especially if you're compressing something other than 
MIPS32 executable code. But it's easy enough to plug in new compression 
types. gzip would be welcome.

> Do you know why nobody added bzip2 support for compressed root file
> systems? bzip2 does a better job at compression. Size of the code? 

scratch space required to decompress

--
dwmw2




To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org

Re: Unable to mount compressed root f/s because init_mtd() does not r egister block device!

[Vipin Malik]

David Woodhouse wrote:

> vmalik@danielind.com said:
> >  Are you saying that you'll not fix this "bug" for me ;) Would saying
> > please help :)
>
> Heh. Of course I'll help. Not sure how best to go about it, though. Init
> order dependencies are ugly. rd_load() should probably be one of the last
> things in the boot sequence, rather than an initcall. Can you add it to
> init/main.c just before mount_root()?

Ok, I did it in my kernel and that worked too, but that's not the permanent
solution is it?


>
>
> > > We'll have JFFS2 (with compression) ready to roll out quite soon :)
>
> > Well, there still would be reasons to decompress a root file system
> > into ramdisk and run from there, namely:
>
> Definitely. Just pushing my new baby, that's all :)
>
> >  3. Compressibility: I would hazard a guess that the compressibility
> > obtained with compressing an entire file system image would be better
> > than that obtained with just compressing individual inode data.
>
> Yep. Actually we only compress a single page at a time
> in jffs2, not even the whole inode.

Do you have any figures as to how well it compresses vs say gzip -9?

Do you know why nobody added bzip2 support for compressed root file systems?
bzip2
does a better job at compression. Size of the code?

Vipin

>
>
> --
> dwmw2
>
> To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org



To unsubscribe, send "unsubscribe mtd" to majordomo@infradead.org