readdir & bonnie++

readdir & bonnie++

[Tomas Hruby]

Hi all,

I was testing our fs project using the bonnie++ tool and the following issue came
up. 

Bonnie++ creates a directory with 16384 files. Then it calls readdir in a loop
to read all the entries and calls stat for each. The test fails since only 16383
entries were returned (one is missing). According to my logs our readdir calls
filldir on the last item as well. Also strace says that all entries were
returned. But the last one is missing. Moreover if I use ls -l on the same
directory, all 16384 entries are returned.  I don't know what is the difference
between the readdir loop in bonnie++ and in ls, but I expect a similar loop.
Perhaps using getdents direcly?

Thanks for any suggestions,

cheers,

		Tomas

Re: readdir & bonnie++

[Bob Copeland]

On 6/1/06, Tomas Hruby <thruby@few.vu.nl> wrote:
> Bonnie++ creates a directory with 16384 files. Then it calls readdir in a loop
> to read all the entries and calls stat for each. The test fails since only 16383
> entries were returned (one is missing). According to my logs our readdir calls
> filldir on the last item as well. Also strace says that all entries were
> returned. But the last one is missing.

Possibly unrelated, but I saw something similar in my filesystem.  A
simple loop calling readdir would fail, but explicit calls to getdents
worked properly (and whatever ls did also worked).  In my case, I was
playing games with d_off (filp->fpos) that caused d_off to overflow 32
bits.  Then glibc would detect this and be "smart" -- seek back to
some point and start over to read up to the overflow.  Eliminating the
overflows fixed it.

Re: readdir & bonnie++

[Tomas Hruby]

>> Bonnie++ creates a directory with 16384 files. Then it calls readdir in a loop
>> to read all the entries and calls stat for each. The test fails since only 16383
>> entries were returned (one is missing). According to my logs our readdir calls
>> filldir on the last item as well. Also strace says that all entries were
>> returned. But the last one is missing.
>
>Possibly unrelated, but I saw something similar in my filesystem.  A
>simple loop calling readdir would fail, but explicit calls to getdents
>worked properly (and whatever ls did also worked).  In my case, I was
>playing games with d_off (filp->fpos) that caused d_off to overflow 32
>bits.  Then glibc would detect this and be "smart" -- seek back to
>some point and start over to read up to the overflow.  Eliminating the

Thanks for a good hint, I checked that but there is no overflow. 

> Are you getting this problem *after* bonnie has deleted some entries
> from the directory?  I had some problems with bonnie because it tries
> to delete entries from a directory in a while(readdir) loop, which
> happens to test a somewhat obscure and underspecified behaviour of
> directories.  I discussed this on the ML a while back.  Maybe this
> thread might help you:

I check the dicsussion on the suggested thread. It describes all issues I was
already facing :-) since we use similar tree structure as ext3.
I return hash (32bit) instead of a real position, but it works fine with ls 
rm -r so far.

My problem with bonnie happens in a test which counts how many files where
created. It just call readdir in loop and counts how many entries were
returned. As far as I understand the bonnie code, there were no files deleted
yet.

I checked the glibc code (i hope that the correct one) and it seems that the
loop inside readdir is returned only if there are no more data in the buffer
and so the getdents is called again. My problem is that there SHOULD be the
last entry still in the buffer.

		Tomas

Re: readdir & bonnie++

[Akshat Aranya]

On 6/4/06, Tomas Hruby <thruby@few.vu.nl> wrote:
> My problem with bonnie happens in a test which counts how many files where
> created. It just call readdir in loop and counts how many entries were
> returned. As far as I understand the bonnie code, there were no files deleted
> yet.
>
> I checked the glibc code (i hope that the correct one) and it seems that the
> loop inside readdir is returned only if there are no more data in the buffer
> and so the getdents is called again. My problem is that there SHOULD be the
> last entry still in the buffer.
>

So, what you're saying is that the getdents() call actually returns
the last entry, but readdir() somehow ignores that last entry?  If
that is the case, I would look at the d_off value for that last entry
for weirdness.


>                 Tomas

-Akshat

Re: readdir & bonnie++

[Tomas Hruby]

> >My problem with bonnie happens in a test which counts how many files where
> >created. It just call readdir in loop and counts how many entries were
> >returned. As far as I understand the bonnie code, there were no files 
> >deleted
> >yet.
> >
> >I checked the glibc code (i hope that the correct one) and it seems that 
> >the
> >loop inside readdir is returned only if there are no more data in the 
> >buffer
> >and so the getdents is called again. My problem is that there SHOULD be the
> >last entry still in the buffer.
> >
> 
> So, what you're saying is that the getdents() call actually returns
> the last entry, but readdir() somehow ignores that last entry?  If
> that is the case, I would look at the d_off value for that last entry
> for weirdness.

Finally, we found that bug. Thank you all. The problem was that filp->f_pos was
set to a wrong value when EOF was detected. And this value is assigned to
->d_off of the last returned entry after returning from vfs_readdir().  This
value is checked by getdents() in glibc when checking if the kernel structures
are the same as in userspace. The wrong value was 64bit (I forgot to truncate it
to 32bits) whereas the userspace value is 32bit only :(

		Tomas