Re: suddenly slow writes on XFS Filesystem

Dave Chinner
2012-05-07T07:17:26Z

On Mon, May 07, 2012 at 08:39:13AM +0200, Stefan Priebe - Profihost AG wrote:
> Hi,
>
> after deleting 400GB it was faster. Now there are still 300GB free but
> it is slow as hell again ;-(
>
> Am 07.05.2012 03:34, schrieb Dave Chinner:
> > On Sun, May 06, 2012 at 11:01:14AM +0200, Stefan Priebe wrote:
> >> Hi,
> >>
> >> since a few days i've experienced a really slow fs on one of our
> >> backup systems.
> >>
> >> I'm not sure whether this is XFS related or related to the
> >> Controller / Disks.
> >>
> >> It is a raid 10 of 20 SATA Disks and i can only write to them with
> >> about 700kb/s while doing random i/o.
> >
> > What sort of random IO? size, read, write, direct or buffered, data
> > or metadata, etc?
> There are 4 rsync processes running and doing backups of other severs.
>
> > iostat -x -d -m 5 and vmstat 5 traces would be
> > useful to see if it is your array that is slow.....
>
> ~ # iostat -x -d -m 5
> Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_
> (8 CPU)
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 254,80 25,40 1,72 0,16 13,71 0,86 3,08 2,39 67,06
> sda 0,00 0,20 0,00 1,20 0,00 0,00 6,50 0,00 0,00 0,00 0,00
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 187,40 24,20 1,26 0,19 14,05 0,75 3,56 3,33 70,50
> sda 0,00 0,00 0,00 0,40 0,00 0,00 4,50 0,00 0,00 0,00 0,00
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 11,20 242,40 92,00 1,56 0,89 15,00 4,70 14,06 1,58 52,68
> sda 0,00 0,20 0,00 2,60 0,00 0,02 12,00 0,00 0,00 0,00 0,00
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 166,20 24,00 0,99 0,17 12,51 0,57 3,02 2,40 45,56
> sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 188,00 25,40 1,22 0,16 13,23 0,44 2,04 1,78 38,02
> sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00

> # vmstat

"vmstat 5", not vmstat 5 times.... :/

> >> I tried vanilla Kernel 3.0.30
> >> and 3.3.4 - no difference. Writing to another partition on another
> >> xfs array works fine.
> >>
> >> Details:
> >> #~ df -h
> >> /dev/sdb1 4,6T 4,4T 207G 96% /mnt
> >
> > Your filesystem is near full - the allocation algorithms definitely
> > slow down as you approach ENOSPC, and IO efficiency goes to hell
> > because of a lack of contiguous free space to allocate from.
> I've now 94% used but it is still slow. It seems it was just getting
> fast with more than 450GB free space.
>
> /dev/sdb1 4,6T 4,3T 310G 94% /mnt

Well, you've probably badly fragmented the free space you have. what
does the 'xfs_db -r -c freesp <dev>' command tell you?

> >> #~ df -i
> >> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt
> > You have 4.6 *billion* inodes in your filesystem?
> Yes - it backups around 100 servers with a lot of files.

So you have what - lots of symlinks? I mean, 4.6 billion inodes
alone requires 1.2TB of space, but if I read the fragmentation
you only have 82 million files with data extents. The only thing
that would other wise use inodes are directories and symlinks....

Still, I can't see how you'd only have 82 million data inodes and 4.5
billion directory inodes - where are all the inodes being consumed?
A massive symlink farm?

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com