Re: Bug#389772: e2fsprogs: e2fsck produces broken htree on ppc

Re: Bug#389772: e2fsprogs: e2fsck produces broken htree on ppc

[Theodore Tso]

On Wed, Sep 27, 2006 at 02:22:51PM +0200, Jörg Sommer wrote:
> Package: e2fsprogs
> Version: 1.39-1
> Severity: important
> 
> you set the compiler option -fsigned-char, but on PowerPC the default is
> unsigned char. This makes the kernel uses unsigned and e2fsck uses signed
> chars.
> 
> In the case of an 8 bit character, str2hashbuf() in lib/ext2fs/dirhash.c
> produces a different buf than the kernel, which leads to the problem that
> the hash calculated by TEA_transform() is a different one.

Oh, dear.  This is actually a kernel bug, because the on all other
platforms, the TEA hash will be using a signed char --- and if you
want filesystems to be portable between different systems (hint: we
do), then all architectures should be using the same algorithm.  And
the vast majority of the systems out there are using signed chars.
Unfortunately PowerPC decided to be different.  :-(

Fixing this is going to be painful, because if we fix the kernel to
suddenly use signed chars in the TEA algorithm, like all other right
thinking architectures (joke! joke!), then existing ext3 filesystems
with directory hashing enabled will break.

But given that e2fsck is doing it right, and the kernel is doing it
wrong, we have the mismatch already.  Sigh, this is going to be
especially painful, given that all the major distributions (SLES,
RHEL, Debian, Ubuntu, etc.) are now shipping with directory hashing
enabled by default, and so this is going to impact a huge number of
PowerPC Linux users and customers.

OK, I think how are going to have to fix this is to define two new
flags in the superblock, which indicates whether str2hashbuf is using
signed or unsigned chars.  If neither flag is set, then new kernels
and new e2fsprogs will set the flag for the "native" signed/unsigned
char convention, and assume that it is correct.  I think this is only
a PowerPC problem, but we'll probably need to do a quick check of all
of the other architectures.

Paul, since you're the PowerPC maintainer, I figured I'd better cc
you.  Does this sound sane to you?

							- Ted
-
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bug#389772: e2fsprogs: e2fsck produces broken htree on ppc

[Andreas Dilger]

On Sep 28, 2006  21:08 -0400, Theodore Tso wrote:
> On Wed, Sep 27, 2006 at 02:22:51PM +0200, Jörg Sommer wrote:
> > Package: e2fsprogs
> > Version: 1.39-1
> > Severity: important
> > 
> > you set the compiler option -fsigned-char, but on PowerPC the default is
> > unsigned char. This makes the kernel uses unsigned and e2fsck uses signed
> > chars.
> > 
> > In the case of an 8 bit character, str2hashbuf() in lib/ext2fs/dirhash.c
> > produces a different buf than the kernel, which leads to the problem that
> > the hash calculated by TEA_transform() is a different one.
> 
> Oh, dear.  This is actually a kernel bug, because the on all other
> platforms, the TEA hash will be using a signed char --- and if you
> want filesystems to be portable between different systems (hint: we
> do), then all architectures should be using the same algorithm.  And
> the vast majority of the systems out there are using signed chars.
> Unfortunately PowerPC decided to be different.  :-(
> 
> But given that e2fsck is doing it right, and the kernel is doing it
> wrong, we have the mismatch already.  Sigh, this is going to be
> especially painful, given that all the major distributions (SLES,
> RHEL, Debian, Ubuntu, etc.) are now shipping with directory hashing
> enabled by default, and so this is going to impact a huge number of
> PowerPC Linux users and customers.

Hmm, except isn't the problem ALREADY that PPC is broken with 8-bit
chars and htree?  That's what started this problem in the first place.
Running e2fsck allowed the kernel htree code to find the file, when
it could not otherwise be looked up...  Need to verify that (my mental
stack is overflowing).

IIRC this problem was also reported in the past but no solution was found.
I think fixing the kernel to specify signed chars for the hash will FIX
the PPC kernel code.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

-
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bug#389772: e2fsprogs: e2fsck produces broken htree on ppc

[Theodore Tso]

On Fri, Sep 29, 2006 at 11:15:47AM -0600, Andreas Dilger wrote:
> Hmm, except isn't the problem ALREADY that PPC is broken with 8-bit
> chars and htree?  That's what started this problem in the first place.
> Running e2fsck allowed the kernel htree code to find the file, when
> it could not otherwise be looked up...  Need to verify that (my mental
> stack is overflowing).
> 
> IIRC this problem was also reported in the past but no solution was found.
> I think fixing the kernel to specify signed chars for the hash will FIX
> the PPC kernel code.

If the filesystem is empty (or at least no no hashtree directories),
then when the kernel creates new directories and expands to the point
where they become indexed, they will be indexed with the PPC variant
of the hash algorithm.  This will be self consistent, and everything
will work fine --- until the filesystem gets corrupted to the point
where e2fsprogs needs to rebuild one or more hashed directories.  At
that point the directories will be rebuilt using the same conventions
used by all other conventions, but the directories will no longer be
useful on the PPC kernel.

Joerg, can you confirm this?  On a PPC machine, can you create a
smallish ext3 filesystem (say, 4-8 megabytes), create a directory with
enough files in it that it becomes indexed (verify using lsattr), and
show that it works just fine on a PPC.  Now take that image, and
transfer it to an x86 machine; you should find that the kernel can't
look up any of the directories on the x86 machine.  If you then run
e2fsck -fD on that filesystem (running the e2fsck on either x86 or
PPC; it shouldn't make a difference), then the resulting filesystem
should work just fine on the x86, and fail on the PPC.

If I understand this problem correctly, the above experiment should
confirm what I suspect is going on.

						- Ted

Re: Bug#389772: e2fsprogs: e2fsck produces broken htree on ppc

[Jörg Sommer]

[-- Attachment #1: Type: text/plain, Size: 1873 bytes --]

Hello Theodore,

Theodore Tso schrieb am Fri 29. Sep, 17:15 (-0400):
> If the filesystem is empty (or at least no no hashtree directories),
> then when the kernel creates new directories and expands to the point
> where they become indexed, they will be indexed with the PPC variant
> of the hash algorithm.  This will be self consistent, and everything
> will work fine --- until the filesystem gets corrupted to the point
> where e2fsprogs needs to rebuild one or more hashed directories.  At
> that point the directories will be rebuilt using the same conventions
> used by all other conventions, but the directories will no longer be
> useful on the PPC kernel.
> 
> Joerg, can you confirm this?

Yes.

> On a PPC machine, can you create a smallish ext3 filesystem (say, 4-8
> megabytes), create a directory with enough files in it that it becomes
> indexed (verify using lsattr), and show that it works just fine on a
> PPC.  Now take that image, and transfer it to an x86 machine;

I don't have a x86 machine.

> you should find that the kernel can't look up any of the directories on
> the x86 machine.  If you then run e2fsck -fD on that filesystem
> (running the e2fsck on either x86 or PPC; it shouldn't make a
> difference), then the resulting filesystem should work just fine on the
> x86, and fail on the PPC.

I put two files at http://www.minet.uni-jena.de/~joergs/ img-broken and
img-working. In the last image, img-working, I can access all files,
especially "test/broken fürß". The first image, img-broken, is the
image after running e2fsck -Df on img-working. So you can test if one
of these images work on x86, especially if the file "test/broken fürß"
is accessable.

Bye, Jörg.
-- 
Wer A sagt, muß nicht B sagen. Er kann auch erkennen, daß A falsch war.
      	    	      	       	       	    	(Erich Kästner)

[-- Attachment #2: Type: application/pgp-signature, Size: 481 bytes --]