performance problems with XFS and Linux kernel 2.6.24.4

performance problems with XFS and Linux kernel 2.6.24.4

[Bond Masuda]

Hi,

I am aware of the change regarding barrier/nobarrier in 2.6.17 and above.
However, the performance hit I'm seeing is very severe and much worse than
what I was seeing in 2.6.17 (bundled with latest updates in Fedora Core 4).

I'm currently running Fedora 8 with 2.6.24.4-64.fc8. If I remove nobarrier
mount option, it takes seconds just to write few megabytes and the system
becomes very very sluggish. On 2.6.17 (Fedora 4) without the nobarrier
option it was still usable.

The XFS file system is about 900GB, if that matters.

Is this a known problem? How can I gather more information to figure out
exactly what is going on?

Thanks,
-Bond

Re: performance problems with XFS and Linux kernel 2.6.24.4

[David Chinner]

On Mon, Apr 07, 2008 at 02:24:41PM -0700, Bond Masuda wrote:
> Hi,
> 
> I am aware of the change regarding barrier/nobarrier in 2.6.17 and above.
> However, the performance hit I'm seeing is very severe and much worse than
> what I was seeing in 2.6.17 (bundled with latest updates in Fedora Core 4).
> 
> I'm currently running Fedora 8 with 2.6.24.4-64.fc8. If I remove nobarrier
> mount option, it takes seconds just to write few megabytes and the system
> becomes very very sluggish. On 2.6.17 (Fedora 4) without the nobarrier
> option it was still usable.

Doesn't sound like a barrier problem.

> The XFS file system is about 900GB, if that matters.
> 
> Is this a known problem?

<shrug>

> How can I gather more information to figure out
> exactly what is going on?

Start with what your hardware is, what your storage config is, output
of 'xfs_info <mtpt>', the log of your test (complete output) on working
and non-working kernels, etc.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Is this XFS problem or something else? : xfs_trans_read_buf error 5 buf count 4096

[Bond Masuda]

Not sure if this is XFS related or not, but this seems to be reoccurring. 

The hardware is:

Dell Inspiron 530, Q6600 quad core, 4GB Ram, 2x750GB SATA HDD

I have Fedora 8/64bit installed, using XFS for "/", ext3 for "/boot". The
two identical 750GB drives are striped to create a 1.4TB "md1" device, which
is mounted as "/". Kernel is from Fedora8, 2.6.24.4-64.fc8.

During the install, everything seemed to work fine. I was able to use the
system without any problems for about 2 days. Then I reboot, and the md1
device running XFS shows the following error messages:

I/O error in filesystem ("md1") meta-data dev md1 block 0x15d048574120008
("xfs_trans_read_buf") error 5 buf count 4096
attempt to access beyond end of device
md1: rw=0, want=98239738096517136, limit=2927780864

The above message repeats several times, but the system manages to boot to
runlevel 3. I can login, but almost any command triggers that message which
has flooded the logs. here is xfs_info on "/":

meta-data=/dev/root			isize=256	agcount=32,
agsize=11436608 blks
	   =					sectsz=512	attr=1
data	   =					bsize=4096
blocks=365971456, imaxpct=25
	   =					sunit=64	swidth=128
blks, unwritten=1
naming   =version2			bsize=4096
log	   =internal			bsize=4096	blocks=32768,
version=1
	   =					sectsz=512	sunit=0
blks, lazy-count=0
realtime =none				extsz=524288 blocks=0, rtextents=0

At first, I thought perhaps something wrong with the disks so I had them
replaced. Reinstalled Fedora8 as before, then again, 2 days later (I don't
know if the 2 day time period is coincidental or not), I reboot, and the
exact same problem occurred. Since I've gone through 4 brand new 750GB HDD
now, (Seagate diagnostics show all the disks to be fine in both occurances,
but I replaced them anyway) I'm feeling that this is not a hardware issue,
but software related.

A search on google seems to show others with possibly similar issues:

http://ubuntuforums.org/archive/index.php/t-568309.html

Is that related? Other google searches show a pattern of this problem only
with >1TB XFS partitions, but that's just what I found googling.

I'm considering re-installing again, but using ext3 for "/" to see if the
problem reoccurs, in hopes of "isolating" the issue. Not sure if it is
related to XFS, the software stripe md1, or something else (but I don't
think it's HDD related now).

Thanks for any info or assistance. Though the system isn't really usable,
(many commands fail, but does boot to runlevel 3 and permits me to login)
I'll leave it as is for today so let me know if there's other info I can try
to gather that would be useful.

Any help would be appreciated.
-Bond

Re: Is this XFS problem or something else? : xfs_trans_read_buf error 5 buf count 4096

[Eric Sandeen]

Bond Masuda wrote:
> Not sure if this is XFS related or not, but this seems to be reoccurring. 

You are probably hitting:

https://bugzilla.redhat.com/show_bug.cgi?id=437968
Bugzilla Bug 437968: Corrupt xfs root filesystem with kernel
kernel-2.6.24.3-xx

Fix is in 2.6.25-rc9 upstream and kernel-2.6.24.5-85.fc8 has been
submitted as an update for Fedora 8 and is now in stable.

If you need to keep the system as-is I'd upgrade the kernel immediately,
then try xfs_repair.

A fresh install might be best; do an install, upgrade to that kernel,
reboot into it, *then* yum update the rest.

-Eric

> The hardware is:
> 
> Dell Inspiron 530, Q6600 quad core, 4GB Ram, 2x750GB SATA HDD
> 
> I have Fedora 8/64bit installed, using XFS for "/", ext3 for "/boot". The
> two identical 750GB drives are striped to create a 1.4TB "md1" device, which
> is mounted as "/". Kernel is from Fedora8, 2.6.24.4-64.fc8.
> 
> During the install, everything seemed to work fine. I was able to use the
> system without any problems for about 2 days. Then I reboot, and the md1
> device running XFS shows the following error messages:
> 
> I/O error in filesystem ("md1") meta-data dev md1 block 0x15d048574120008
> ("xfs_trans_read_buf") error 5 buf count 4096
> attempt to access beyond end of device
> md1: rw=0, want=98239738096517136, limit=2927780864
> 
> The above message repeats several times, but the system manages to boot to
> runlevel 3. I can login, but almost any command triggers that message which
> has flooded the logs. here is xfs_info on "/":
> 
> meta-data=/dev/root			isize=256	agcount=32,
> agsize=11436608 blks
> 	   =					sectsz=512	attr=1
> data	   =					bsize=4096
> blocks=365971456, imaxpct=25
> 	   =					sunit=64	swidth=128
> blks, unwritten=1
> naming   =version2			bsize=4096
> log	   =internal			bsize=4096	blocks=32768,
> version=1
> 	   =					sectsz=512	sunit=0
> blks, lazy-count=0
> realtime =none				extsz=524288 blocks=0, rtextents=0
> 
> At first, I thought perhaps something wrong with the disks so I had them
> replaced. Reinstalled Fedora8 as before, then again, 2 days later (I don't
> know if the 2 day time period is coincidental or not), I reboot, and the
> exact same problem occurred. Since I've gone through 4 brand new 750GB HDD
> now, (Seagate diagnostics show all the disks to be fine in both occurances,
> but I replaced them anyway) I'm feeling that this is not a hardware issue,
> but software related.
> 
> A search on google seems to show others with possibly similar issues:
> 
> http://ubuntuforums.org/archive/index.php/t-568309.html
> 
> Is that related? Other google searches show a pattern of this problem only
> with >1TB XFS partitions, but that's just what I found googling.
> 
> I'm considering re-installing again, but using ext3 for "/" to see if the
> problem reoccurs, in hopes of "isolating" the issue. Not sure if it is
> related to XFS, the software stripe md1, or something else (but I don't
> think it's HDD related now).
> 
> Thanks for any info or assistance. Though the system isn't really usable,
> (many commands fail, but does boot to runlevel 3 and permits me to login)
> I'll leave it as is for today so let me know if there's other info I can try
> to gather that would be useful.
> 
> Any help would be appreciated.
> -Bond
> 
> 
>