Re: suddenly slow writes on XFS Filesystem

Stefan Priebe - Profihost AG
2012-05-07T13:57:43Z

Am 07.05.2012 10:31, schrieb Martin Steigerwald:
> Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
> Did you verify that at the time you perceive slowness the servers you
> backup can deliver data fast enough?
Yes. It works fine to another partition.

> I would like to now, whether there are really processes waiting for I/O
> during rsync workload.
>
> Can you try vmstat 5 and
> while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done

Here it is:
# vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
0 1 0 396688 48 7728136 0 0 229 374 22 32 1 12
85 2
2 0 0 421744 48 7717348 0 0 3722 1169 9015 8389 1 2
95 2
0 1 0 405780 48 7751904 0 0 8937 207 10780 9290 2
2 88 7
0 0 0 486928 48 7733300 0 0 2526 1277 8275 7791 1 2
96 1
0 0 0 416692 48 7778164 0 0 5046 43750 8548 8141 1 2
95 2
0 0 0 444968 48 7777792 0 0 8021 1709 9315 8573 2 2
94 2
2 0 0 357924 48 7946532 0 0 48181 1031 17646 12684 10
4 87 0
1 0 0 348696 48 8137200 0 0 74366 1056 24362 16775 15
5 81 0
1 0 0 391552 48 8279000 0 0 54693 1242 19224 13957 11
4 85 0

# while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
root 12493 2.0 0.2 101780 48392 ? D 15:35 0:24 rsync
--daemon
root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync
--daemon
root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync
--daemon
root 12494 3.9 0.2 128220 44328 ? D 15:35 0:47 rsync
--daemon
root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync
--daemon
root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync
--daemon
root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync
--daemon
root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync
--daemon
root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync
--daemon
root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 21720 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync
--daemon
root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 21720 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 21980 ? D 15:50 0:11 rsync
--daemon
root 13582 1.7 0.0 74308 13280 ? D 15:50 0:05 rsync
--daemon
root 12493 2.0 0.2 87956 48212 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 21980 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86420 48156 ? D 15:35 0:25 rsync
--daemon
root 12493 2.0 0.2 86420 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync
--daemon
root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22772 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22772 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 3378 0.1 0.0 0 0 ? D 13:59 0:09
[flush-8:16]
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon

Stefan