ext3 duplicate entries with readdir

ext3 duplicate entries with readdir

[Andries E. Brouwer]

In a largish directory (21292 files) a scan using ext3 readdir
found 21293 files, returning one entry twice.

Google "ext3 readdir twice" gives me a handful of identical complaints.
I see reports from 2002-2008.
So, this is an old and well-known symptom but seemingly still not fixed,
at least not in the kernel on this machine here.
This report is for Fedora release 8 (Werewolf) with kernel 2.6.24.5.

A 2002 post says "This is unfortunate, but the chances of our hitting
this failure are relatively small".

In http://www.redhat.com/archives/ext3-users/2008-August/msg00009.html
tytso seems to understand what happens: a hash collision.
But has this been fixed in the current tree?


Andries


Thomas Trauner gave a test program. It prints
expected 25989 files, but readdir reports 25990

So, yes, this is reproducible.

Re: ext3 duplicate entries with readdir

[Mike Snitzer]

On Tue, Mar 3, 2009 at 3:05 PM, Andries E. Brouwer
<Andries.Brouwer@cwi.nl> wrote:
> In a largish directory (21292 files) a scan using ext3 readdir
> found 21293 files, returning one entry twice.
>
> Google "ext3 readdir twice" gives me a handful of identical complaints.
> I see reports from 2002-2008.
> So, this is an old and well-known symptom but seemingly still not fixed,
> at least not in the kernel on this machine here.
> This report is for Fedora release 8 (Werewolf) with kernel 2.6.24.5.
>
> A 2002 post says "This is unfortunate, but the chances of our hitting
> this failure are relatively small".
>
> In http://www.redhat.com/archives/ext3-users/2008-August/msg00009.html
> tytso seems to understand what happens: a hash collision.
> But has this been fixed in the current tree?

Yes, it has been fixed, please see the following git commits:
6a897cf447a83c9c3fd1b85a1e525c02d6eada7d
8c9fa93d51123c5540762b1a9e1919d6f9c4af7c

Mike

Re: ext3 duplicate entries with readdir

[Andries E. Brouwer]

On Tue, Mar 03, 2009 at 03:54:02PM -0500, Mike Snitzer wrote:
> On Tue, Mar 3, 2009 at 3:05 PM, Andries E. Brouwer wrote:
>
>> In a largish directory (21292 files) a scan using ext3 readdir
>> found 21293 files, returning one entry twice.
>>
>> Has this been fixed in the current tree?
> 
> Yes, it has been fixed, please see the following git commits:
>
> 6a897cf447a83c9c3fd1b85a1e525c02d6eada7d
> 8c9fa93d51123c5540762b1a9e1919d6f9c4af7c

Good, thanks!

Andries


[On another machine, Ubuntu with 2.6.27-7-server still fails:
% ../readdirtest
expected 61005 files, but readdir reports 61006
% rm *
rm: cannot remove `59992': No such file or directory
(where the same name occurs twice in the expansion of *).]

Re: ext3 duplicate entries with readdir

[Theodore Tso]

On Tue, Mar 03, 2009 at 10:51:26PM +0100, Andries E. Brouwer wrote:
> > Yes, it has been fixed, please see the following git commits:
> >
> > 6a897cf447a83c9c3fd1b85a1e525c02d6eada7d
> > 8c9fa93d51123c5540762b1a9e1919d6f9c4af7c
> 
> [On another machine, Ubuntu with 2.6.27-7-server still fails:
> % ../readdirtest
> expected 61005 files, but readdir reports 61006
> % rm *
> rm: cannot remove `59992': No such file or directory
> (where the same name occurs twice in the expansion of *).]

The patches were backported to 2.6.27 stable in 2.6.27.8.  So the
problem would expected to be still present in an 2.6.27.7 kernel.

	      	       	     	   	   - Ted

Re: ext3 duplicate entries with readdir

[Goswin von Brederlow]

Theodore Tso <tytso@mit.edu> writes:

> On Tue, Mar 03, 2009 at 10:51:26PM +0100, Andries E. Brouwer wrote:
>> > Yes, it has been fixed, please see the following git commits:
>> >
>> > 6a897cf447a83c9c3fd1b85a1e525c02d6eada7d
>> > 8c9fa93d51123c5540762b1a9e1919d6f9c4af7c
>> 
>> [On another machine, Ubuntu with 2.6.27-7-server still fails:
>> % ../readdirtest
>> expected 61005 files, but readdir reports 61006
>> % rm *
>> rm: cannot remove `59992': No such file or directory
>> (where the same name occurs twice in the expansion of *).]
>
> The patches were backported to 2.6.27 stable in 2.6.27.8.  So the
> problem would expected to be still present in an 2.6.27.7 kernel.
>
> 	      	       	     	   	   - Ted

Does the bug also affect xfs? Because I see the problem there.

MfG
        Goswin

Re: ext3 duplicate entries with readdir

[Eric Sandeen]

Goswin von Brederlow wrote:
> Theodore Tso <tytso@mit.edu> writes:
> 
>> On Tue, Mar 03, 2009 at 10:51:26PM +0100, Andries E. Brouwer wrote:
>>>> Yes, it has been fixed, please see the following git commits:
>>>>
>>>> 6a897cf447a83c9c3fd1b85a1e525c02d6eada7d
>>>> 8c9fa93d51123c5540762b1a9e1919d6f9c4af7c
>>> [On another machine, Ubuntu with 2.6.27-7-server still fails:
>>> % ../readdirtest
>>> expected 61005 files, but readdir reports 61006
>>> % rm *
>>> rm: cannot remove `59992': No such file or directory
>>> (where the same name occurs twice in the expansion of *).]
>> The patches were backported to 2.6.27 stable in 2.6.27.8.  So the
>> problem would expected to be still present in an 2.6.27.7 kernel.
>>
>> 	      	       	     	   	   - Ted
> 
> Does the bug also affect xfs? Because I see the problem there.

Completely different, and the wrong list for the question, really.

Please point me to the bug you filed for this problem on xfs, I'll look
into it (and when you do so please cc: the xfs list, and trim off the
current ext4-related recipients)

Thanks,

-Eric

Re: ext3 duplicate entries with readdir

[Goswin von Brederlow]

Eric Sandeen <sandeen@redhat.com> writes:

> Goswin von Brederlow wrote:
>> Theodore Tso <tytso@mit.edu> writes:
>> 
>>> On Tue, Mar 03, 2009 at 10:51:26PM +0100, Andries E. Brouwer wrote:
>>>>> Yes, it has been fixed, please see the following git commits:
>>>>>
>>>>> 6a897cf447a83c9c3fd1b85a1e525c02d6eada7d
>>>>> 8c9fa93d51123c5540762b1a9e1919d6f9c4af7c
>>>> [On another machine, Ubuntu with 2.6.27-7-server still fails:
>>>> % ../readdirtest
>>>> expected 61005 files, but readdir reports 61006
>>>> % rm *
>>>> rm: cannot remove `59992': No such file or directory
>>>> (where the same name occurs twice in the expansion of *).]
>>> The patches were backported to 2.6.27 stable in 2.6.27.8.  So the
>>> problem would expected to be still present in an 2.6.27.7 kernel.
>>>
>>> 	      	       	     	   	   - Ted
>> 
>> Does the bug also affect xfs? Because I see the problem there.
>
> Completely different, and the wrong list for the question, really.
>
> Please point me to the bug you filed for this problem on xfs, I'll look
> into it (and when you do so please cc: the xfs list, and trim off the
> current ext4-related recipients)
>
> Thanks,
>
> -Eric

Haven't filed a bug and can't reproduce the problem but I had 2 cases
that behaved the same way so I wondered if it was the same
issue. Would have ment I could stop worrying about it and wouldn't
have to try to reproduce it.

MfG
        Goswin