Bonnie++ Burps on XFS

Bonnie++ Burps on XFS

[Kurt Wall]

I've been using bonnie++ off and on for a long time. Suddenly, it has
started failing when run against an XFS filesystem situated on a SATA
drive. Here's the output of a run:

$ bench/sbin/bonnie++ -m "Luther 2.6.17-rc1 XFS SATA" \
> -d /archive/bench \
> -s 3008 \
> -n 1024 >> bonnie.out 2>> bonnie.err
Version: 1.03
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...Bonnie: drastic I/O error (rmdir):
Directory not empty
Cleaning up test directory after error.

This is on a 2.6.17-rc1 Linux kernel. The interesting thing
is that not even root cannot delete the directory in which
bonnie++ is working:

[~]$ ls -ld /archive/bench/Bonnie.8426/
drwx------  2 kwall users 4096 2006-04-05 09:06 /archive/bench/Bonnie.8426/
[~]$ ls -al /archive/bench/Bonnie.8426/
total 24
drwx------  2 kwall users 4096 2006-04-05 09:06 ./
drwxr-xr-x  4 kwall users   42 2006-04-05 06:24 ../
[~]$ sudo rm -rf /archive/bench/Bonnie.8426
rm: cannot remove directory `/archive/bench/Bonnie.8426': Directory not
empty

As you can see, the directory is empty, but a "rm" fails (as does
"rmdir"). I'm retesting with a 2.6.16 kernel as I type. It isn't
clear if the problems is XFS, some other piece of the kernel, or
bonnie++. Neither dmesg nor syslog shows anything amiss. I suppose 
I could strace the run, but I'm not especially eager to deal with 
a multi-gigabyte output file if there is a more focused method to
isolate the problem.

I've also written to bonnie++'s author.

Thanks for the help.

Best regards,

Kurt
-- 
The full impact of parenthood doesn't hit you until you multiply the
number of your kids by 32 teeth.

Re: Bonnie++ Burps on XFS

[Nathan Scott]

On Wed, Apr 05, 2006 at 10:34:45PM -0400, Kurt Wall wrote:
> I've been using bonnie++ off and on for a long time. Suddenly, it has
> started failing when run against an XFS filesystem situated on a SATA
> drive. Here's the output of a run:

[ Please report these things to linux-xfs@oss.sgi.com... ]

> Delete files in sequential order...Bonnie: drastic I/O error (rmdir):

Anything in your system log?

> clear if the problems is XFS, some other piece of the kernel, or
> bonnie++. Neither dmesg nor syslog shows anything amiss. I suppose 
> I could strace the run, but I'm not especially eager to deal with 
> a multi-gigabyte output file if there is a more focused method to
> isolate the problem.

See if 2.6.16.1 fails too for starters then we'll take it from there.

cheers.

-- 
Nathan

Re: Bonnie++ Burps on XFS

[Nathan Scott]

On Thu, Apr 06, 2006 at 12:57:56PM +1000, Nathan Scott wrote:
> On Wed, Apr 05, 2006 at 10:34:45PM -0400, Kurt Wall wrote:
> > I've been using bonnie++ off and on for a long time. Suddenly, it has
> > started failing when run against an XFS filesystem situated on a SATA
> > drive. Here's the output of a run:
> 
> [ Please report these things to linux-xfs@oss.sgi.com... ]
> 
> > Delete files in sequential order...Bonnie: drastic I/O error (rmdir):
> 
> Anything in your system log?

Lemme answer that for you - "no".  I've reproduced the problem,
I'll get back to you once I've nutted out whats gone wrong.

Thanks for reporting it.

cheers.

-- 
Nathan

Re: Bonnie++ Burps on XFS

[David Greaves]

Nathan Scott wrote:

>On Thu, Apr 06, 2006 at 12:57:56PM +1000, Nathan Scott wrote:
>  
>
>>On Wed, Apr 05, 2006 at 10:34:45PM -0400, Kurt Wall wrote:
>>    
>>
>>>I've been using bonnie++ off and on for a long time. Suddenly, it has
>>>started failing when run against an XFS filesystem situated on a SATA
>>>drive. Here's the output of a run:
>>>      
>>>
>>[ Please report these things to linux-xfs@oss.sgi.com... ]
>>
>>    
>>
>>>Delete files in sequential order...Bonnie: drastic I/O error (rmdir):
>>>      
>>>
>>Anything in your system log?
>>    
>>
>
>Lemme answer that for you - "no".  I've reproduced the problem,
>I'll get back to you once I've nutted out whats gone wrong.
>
>Thanks for reporting it.
>
>cheers.
>
>  
>
Me too.

2.6.16

I had filesystem corruption and needed ran xfs_repair.

After I finished I removed most of the lost and found bits but was left
with:

haze:/scratch/lost+found# ll
total 28
drwxr-xr-x   4 root root   38 2006-04-02 08:29 ./
drwxrwxrwx  26 root root 4096 2006-04-04 18:25 ../
drwxrwxr-x   2 1046 1046 8192 2006-04-02 08:30 658616481/
drwxrwxr-x   2 1046 1046 8192 2006-04-02 08:22 823441168/
haze:/scratch/lost+found# rmdir *
rmdir: `658616481': Directory not empty
rmdir: `823441168': Directory not empty
haze:/scratch/lost+found# ll *
658616481:
total 12
drwxrwxr-x  2 1046 1046 8192 2006-04-02 08:30 ./
drwxr-xr-x  4 root root   38 2006-04-02 08:29 ../

823441168:
total 12
drwxrwxr-x  2 1046 1046 8192 2006-04-02 08:22 ./
drwxr-xr-x  4 root root   38 2006-04-02 08:29 ../

Obviously I had an fs corruption (due to raid failure due to some bogus
libata errors) so this may not be the same thing.

David

--