Re: suddenly slow writes on XFS Filesystem

[Dave Chinner <david@fromorbit.com>]

On Mon, May 07, 2012 at 09:22:42AM +0200, Stefan Priebe - Profihost AG wrote:
> 
> >> # vmstat
> > "vmstat 5", not vmstat 5 times....  :/
> oh sorry. Sadly the rsync processes do not run right know i've to kill
> them. Is the output still usable?
> # vmstat 5
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
> id wa
>  0  1      0 5582136     48 5849956    0    0   176   394   34   54  1
> 16 82  1
>  0  1      0 5552180     48 5854280    0    0  2493  2496 3079 2172  1
> 4 86  9
>  3  2      0 5601308     48 5857672    0    0  1098 28043 5150 1913  0
> 10 73 17
>  0  2      0 5595360     48 5863180    0    0  1098 14336 3945 1897  0
> 8 69 22
>  3  2      0 5594088     48 5865280    0    0   432 15897 4209 2366  0
> 8 71 21
>  0  2      0 5591068     48 5868940    0    0   854 10989 3519 2107  0
> 7 70 23
>  1  1      0 5592004     48 5869872    0    0   180  7886 3605 2436  0
> 3 76 22

It tells me that there is still quite an IO load on the system even
when the rsyncs are not running...

> >> /dev/sdb1             4,6T  4,3T  310G  94% /mnt
> > Well, you've probably badly fragmented the free space you have. what
> > does the 'xfs_db -r -c freesp <dev>' command tell you?
> 
>    from      to extents  blocks    pct
>       1       1  942737  942737   0,87
>       2       3  671860 1590480   1,47
>       4       7  461268 2416025   2,23
>       8      15 1350517 18043063  16,67
>      16      31  111254 2547581   2,35
>      32      63  192032 9039799   8,35

So that's roughly 3.7 million free space extents of 256kB or less
totalling about 32% of the freespace (~100GB).  That's pretty badly
fragmented, and given the workload, probably unrecoverable. Dump,
mkfs and restore is probably the only way to unfragment the free
space now, but that would only be a temporary solution if you
continue to run at >90% full. Even if you do keep it at below 90%
full, such a workload will age the filesystem and slowly fragment
free space, but it should take a lot longer to get to this state...

> >>>> /dev/sdb1            4875737052 4659318044 216419008  96% /mnt
> >>> You have 4.6 *billion* inodes in your filesystem?
> >> Yes - it backups around 100 servers with a lot of files.
> i rechecked this and it seems i sadly copied the wrong output ;-( sorry
> for that.
> 
> Here is the correct one:
> #~ df -i
> /dev/sdb1            975173568 95212355 879961213   10% /mnt

That makes more sense. :)

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs