Re: suddenly slow writes on XFS Filesystem

Stefan Priebe - Profihost AG
2012-05-07T06:39:18Z

Hi,

after deleting 400GB it was faster. Now there are still 300GB free but
it is slow as hell again ;-(

Am 07.05.2012 03:34, schrieb Dave Chinner:
> On Sun, May 06, 2012 at 11:01:14AM +0200, Stefan Priebe wrote:
>> Hi,
>>
>> since a few days i've experienced a really slow fs on one of our
>> backup systems.
>>
>> I'm not sure whether this is XFS related or related to the
>> Controller / Disks.
>>
>> It is a raid 10 of 20 SATA Disks and i can only write to them with
>> about 700kb/s while doing random i/o.
>
> What sort of random IO? size, read, write, direct or buffered, data
> or metadata, etc?
There are 4 rsync processes running and doing backups of other severs.

> iostat -x -d -m 5 and vmstat 5 traces would be
> useful to see if it is your array that is slow.....

~ # iostat -x -d -m 5
Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_
(8 CPU)

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 254,80 25,40 1,72 0,16
13,71 0,86 3,08 2,39 67,06
sda 0,00 0,20 0,00 1,20 0,00 0,00
6,50 0,00 0,00 0,00 0,00

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 187,40 24,20 1,26 0,19
14,05 0,75 3,56 3,33 70,50
sda 0,00 0,00 0,00 0,40 0,00 0,00
4,50 0,00 0,00 0,00 0,00

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 11,20 242,40 92,00 1,56 0,89
15,00 4,70 14,06 1,58 52,68
sda 0,00 0,20 0,00 2,60 0,00 0,02
12,00 0,00 0,00 0,00 0,00

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 166,20 24,00 0,99 0,17
12,51 0,57 3,02 2,40 45,56
sda 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00

qDevice: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 188,00 25,40 1,22 0,16
13,23 0,44 2,04 1,78 38,02
sda 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00

# vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
7 0 0 788632 48 12189652 0 0 173 395 13 45 1
16 82 1
[root@server844-han /serverbackup (master)]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
4 0 0 778148 48 12189776 0 0 173 395 13 45 1
16 82 1
[root@server844-han /serverbackup (master)]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
2 0 0 774372 48 12189876 0 0 173 395 13 45 1
16 82 1
[root@server844-han /serverbackup (master)]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
5 0 0 771240 48 12189936 0 0 173 395 13 45 1
16 82 1
[root@server844-han /serverbackup (master)]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
6 0 0 768636 48 12190000 0 0 173 395 13 45 1
16 82 1

>
>> I tried vanilla Kernel 3.0.30
>> and 3.3.4 - no difference. Writing to another partition on another
>> xfs array works fine.
>>
>> Details:
>> #~ df -h
>> /dev/sdb1 4,6T 4,4T 207G 96% /mnt
>
> Your filesystem is near full - the allocation algorithms definitely
> slow down as you approach ENOSPC, and IO efficiency goes to hell
> because of a lack of contiguous free space to allocate from.
I've now 94% used but it is still slow. It seems it was just getting
fast with more than 450GB free space.

/dev/sdb1 4,6T 4,3T 310G 94% /mnt

>> #~ df -i
>> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt
> You have 4.6 *billion* inodes in your filesystem?
Yes - it backups around 100 servers with a lot of files.

Greet Stefan