Re: suddenly slow writes on XFS Filesystem

Martin Steigerwald
2012-05-07T14:32:09Z

Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
> Am 07.05.2012 10:31, schrieb Martin Steigerwald:
> > Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
> > Did you verify that at the time you perceive slowness the servers you
> > backup can deliver data fast enough?
>
> Yes. It works fine to another partition.

With more free space I suppose?

>
> > I would like to now, whether there are really processes waiting for
> > I/O during rsync workload.
> >
> > Can you try vmstat 5 and
> > while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
>
> Here it is:
> # vmstat 5
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy
> id wa
> 0 1 0 396688 48 7728136 0 0 229 374 22 32 1
> 12 85 2
> 2 0 0 421744 48 7717348 0 0 3722 1169 9015 8389 1
> 2 95 2
> 0 1 0 405780 48 7751904 0 0 8937 207 10780 9290 2
> 2 88 7
> 0 0 0 486928 48 7733300 0 0 2526 1277 8275 7791 1
> 2 96 1
> 0 0 0 416692 48 7778164 0 0 5046 43750 8548 8141 1
> 2 95 2
> 0 0 0 444968 48 7777792 0 0 8021 1709 9315 8573 2
> 2 94 2
> 2 0 0 357924 48 7946532 0 0 48181 1031 17646 12684 10
> 4 87 0
> 1 0 0 348696 48 8137200 0 0 74366 1056 24362 16775 15
> 5 81 0
> 1 0 0 391552 48 8279000 0 0 54693 1242 19224 13957 11
> 4 85 0

Hmm, there are some I/O wait times, but they are lower than I expected.

In the last lines the throughput is higher than in the first lines. Does
that correlate to higher speed in rsync? It may also be excessive writes
due to free space fragmentation, but Dave or someone else might be able to
say whether such a huge difference can be blamed to free space
fragmentation alone.

> # while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
> root 12493 2.0 0.2 101780 48392 ? D 15:35 0:24 rsync
> --daemon
> root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync
> --daemon
> root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync
> --daemon
> root 12494 3.9 0.2 128220 44328 ? D 15:35 0:47 rsync
> --daemon
> root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync
> --daemon

Still these rsyncs appear to be in uninteruptible sleep quite
consistently. They shouldn´t be in that state when waiting on network
socket readyness.

So this workloads seems to be I/O bound to me.

I suggest trying to keep the volume above 500 GB free and see whether that
works consistently.

Thanks,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7