cramfs in big endian

cramfs in big endian

[Andi Drebes]

Hi!
I'm currently trying to enable the cramfs to mount filesystems with a 
different endianness. All I have is an intel compatible machine that produces 
cramfs images in little endian format. I found a modified version of mkcramfs 
that is able to produce images in big endian mode. Unfortunately it seems to 
use a different compression algorithm (LZO) than the standard implementation 
(zlib).

I would be very glad if somebody could send me an image in big endian mode 
using the zlib deflate compression algorithm. I have uploaded a bz2 archive 
containing some testfiles. You can find it at [1].


Regards,
	Andi

[1] http://drebesium.org/~hackbert/cramfs-testfiles.tar.bz2

Re: cramfs in big endian

[Tomas M]

> I'm currently trying to enable the cramfs to mount filesystems with a 
> different endianness. 

I would suggest you to use squashfs instead of cramfs.

First, it's newer, it's better, it's actively developed, it doesn't have any limits like the bad cramfs.
Moreover, it currently supports both endians. 

(hurry up, as kernel people said in the past that squashfs should NEVER EVER support multiple endians, so the feature will be dropped from squashfs, in order to get it into mainline kernel more easily; if my informations are correct).



Tomas M

Re: cramfs in big endian

[Andi Drebes]

Hi!

> I would suggest you to use squashfs instead of cramfs.
> First, it's newer, it's better, it's actively developed, it doesn't have any
> limits like the bad cramfs. 
I'm developing a new linux based firmware for my router which uses cramfs. 
Switching to squashfs still needs some time. Meanwhile, I have to work with 
cramfs. As the router uses the big endian format and as my machine works with 
the little endian format, I'm unable to mount the router's filesystem images.


	Andi

Re: cramfs in big endian

[Christoph Hellwig]

On Wed, Nov 07, 2007 at 09:51:48PM +0100, Andi Drebes wrote:
> Hi!
> 
> > I would suggest you to use squashfs instead of cramfs.
> > First, it's newer, it's better, it's actively developed, it doesn't have any
> > limits like the bad cramfs. 
> I'm developing a new linux based firmware for my router which uses cramfs. 
> Switching to squashfs still needs some time. Meanwhile, I have to work with 
> cramfs. As the router uses the big endian format and as my machine works with 
> the little endian format, I'm unable to mount the router's filesystem images.

Making cramfs endianess-independent shouldn't be much work.  Take a look
at the helpers in fs/ufs/swab.h and use them for every ondisk access in
cramfs.  Drop me a not if you need some help.

Re: cramfs in big endian

[Andi Drebes]

Hi!

> Making cramfs endianess-independent shouldn't be much work.  Take a look
> at the helpers in fs/ufs/swab.h and use them for every ondisk access in
> cramfs.  Drop me a not if you need some help.
I already started working on this feature some time ago. I think I am able to 
mount cramfs images with another endianness. But all I have to test this are 
images that use LZO compression istead of deflate.

In the original post I was just asking somebody possessing a big endian 
machine to create an image in big endian / deflate format. This would help me 
a lot. Later, I want to add support for multiple compression algorithms.

Thanks so far,
	Andi

Re: cramfs in big endian

[H. Peter Anvin]

Christoph Hellwig wrote:
> On Wed, Nov 07, 2007 at 09:51:48PM +0100, Andi Drebes wrote:
>> Hi!
>>
>>> I would suggest you to use squashfs instead of cramfs.
>>> First, it's newer, it's better, it's actively developed, it doesn't have any
>>> limits like the bad cramfs. 
>> I'm developing a new linux based firmware for my router which uses cramfs. 
>> Switching to squashfs still needs some time. Meanwhile, I have to work with 
>> cramfs. As the router uses the big endian format and as my machine works with 
>> the little endian format, I'm unable to mount the router's filesystem images.
> 
> Making cramfs endianess-independent shouldn't be much work.  Take a look
> at the helpers in fs/ufs/swab.h and use them for every ondisk access in
> cramfs.  Drop me a not if you need some help.

And it would suck.  Hard.

Endian-independent code is slower than wrong-endian code, because of the 
necessary conditionals.  Thus, you DO NOT WANT this(*).

The only way to not make it totally suck is to compile the code twice 
and do switching at a VERY high level -- in effect, creating two 
separate filesystem drivers.

	-hpa

(*) Readonly filesystems like iso9660 have attempted to be 
*both*-endian, by having its datastructures in both forms at all times. 
    That is inefficient with space, but would be CPU-efficient in 
theory.  Unfortunately it causes output validation problems; apparently 
many iso9660 mastering programs output invalid bigendian information as 
it is never tested; consequently, many bigendian platforms read the 
littleendian information anyway.

Re: cramfs in big endian

[Christoph Hellwig]

On Fri, Nov 09, 2007 at 05:03:01PM -0800, H. Peter Anvin wrote:
> Endian-independent code is slower than wrong-endian code, because of the 
> necessary conditionals.  Thus, you DO NOT WANT this(*).

I'd prefer not to have it either.  But a someone (pinhead) was smart
enough not to define an endianess for cramfs from the beginning we're
stuck with it.

Re: cramfs in big endian

[H. Peter Anvin]

Christoph Hellwig wrote:
> On Fri, Nov 09, 2007 at 05:03:01PM -0800, H. Peter Anvin wrote:
>> Endian-independent code is slower than wrong-endian code, because of the 
>> necessary conditionals.  Thus, you DO NOT WANT this(*).
> 
> I'd prefer not to have it either.  But a someone (pinhead) was smart
> enough not to define an endianess for cramfs from the beginning we're
> stuck with it.
> 

I thought cramfs was "always" defined as littleendian?

Either way... I thought the primary discussion was about squashfs.

	-hpa

Re: cramfs in big endian

[Andi Drebes]

> > Endian-independent code is slower than wrong-endian code, because of the 
> > necessary conditionals.  Thus, you DO NOT WANT this(*).
> 
> I'd prefer not to have it either.  But a someone (pinhead) was smart
> enough not to define an endianess for cramfs from the beginning we're
> stuck with it.

Indeed, this is the problem. The readme file fs/cramfs/README states:

"One part of that is addressing endianness.  The two options here are
`always use little-endian' (like ext2fs) or `writer chooses
endianness; kernel adapts at runtime'.  Little-endian wins because of
code simplicity and little CPU overhead even on big-endian machines."

Unfortunately, the better idea was never implemented. Further, there's no 
information about the endianness stored in the filesystem. Guessing it and 
mounting the filesystem isn't a clean solution. Even worse, there's no 
information about which compression algorithm was used to create the 
filesystem. Guessing the compression method may lead to serious problems.

So here is my proposal for future development of cramfs:
The should tell cramfs how to mount a filesystem. Therefore, the endianness 
and the compression method both have to be specified manually. If none is 
specified, cramfs will assume host endianness and that deflate can be used in 
order to decompress the contents. If something seems to be wrong with the 
filesystem (e.g. wrong magic), cramfs will guess the endianness and inform 
the user about the guess, but it won't mount the filesystem if it doesn't 
match the endianness specified or the host's one.

	Andi

Re: cramfs in big endian

[H. Peter Anvin]

Andi Drebes wrote:
> 
> Indeed, this is the problem. The readme file fs/cramfs/README states:
> 
> "One part of that is addressing endianness.  The two options here are
> `always use little-endian' (like ext2fs) or `writer chooses
> endianness; kernel adapts at runtime'.  Little-endian wins because of
> code simplicity and little CPU overhead even on big-endian machines."
> 
> Unfortunately, the better idea was never implemented. Further, there's no 
> information about the endianness stored in the filesystem. Guessing it and 
> mounting the filesystem isn't a clean solution. Even worse, there's no 
> information about which compression algorithm was used to create the 
> filesystem. Guessing the compression method may lead to serious problems.
> 
> So here is my proposal for future development of cramfs:
> The should tell cramfs how to mount a filesystem. Therefore, the endianness 
> and the compression method both have to be specified manually. If none is 
> specified, cramfs will assume host endianness and that deflate can be used in 
> order to decompress the contents. If something seems to be wrong with the 
> filesystem (e.g. wrong magic), cramfs will guess the endianness and inform 
> the user about the guess, but it won't mount the filesystem if it doesn't 
> match the endianness specified or the host's one.
> 

What about simply deep-sixing cramfs and replacing it with squashfs or 
something else?

	-hpa

Re: cramfs in big endian

[Andi Drebes]

> What about simply deep-sixing cramfs and replacing it with squashfs or 
> something else?
I think this is the long term solution. Cramfs isn't a very beautiful 
filesystem. It's a good candidate for removal. However, there are still some 
distributions that use cramfs for initrds. So removing it immediately isn't a 
good idea.

	Andi