From: Dave Chinner <david@fromorbit.com>
To: Linas Jankauskas <linas.j@iv.lt>
Cc: xfs@oss.sgi.com
Subject: Re: Slow performance after ~4.5TB
Date: Mon, 12 Nov 2012 20:04:48 +1100 [thread overview]
Message-ID: <20121112090448.GS24575@dastard> (raw)
In-Reply-To: <50A0AFD5.2020607@iv.lt>
On Mon, Nov 12, 2012 at 10:14:13AM +0200, Linas Jankauskas wrote:
> Hello,
>
> we have 30 backup servers with 20TB backup partition each.
> While server is new and empty rsync is compying data prety fast, but
> when it reaches about 4.5TB write operation become very slow (about 10
> times slower).
>
> I have attached cpu and disk graphs.
>
> As you can see first week, while server was empty, rsync was using "user"
> cpu and data copying was fast. Later rsync started to use "system" cpu
> and data copying became much slower. Same situation is on all our backup
> servers. Before we had used smaller partition with ext4 and we had no
> problems.
>
> Most time rsync is spending on ftruncate:
>
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 99.99 18.362863 165431 111 ftruncate
> 0.00 0.000712 3 224 112 open
> 0.00 0.000195 1 257 write
> 0.00 0.000171 1 250 read
> 0.00 0.000075 1 112 lchown
> 0.00 0.000039 0 112 lstat
> 0.00 0.000028 0 112 close
> 0.00 0.000021 0 112 chmod
> 0.00 0.000011 0 396 select
> 0.00 0.000000 0 112 utimes
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 18.364115 1798 112 total
Never seen that before. More info needed. Start here:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
And we can go from there.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2012-11-12 9:02 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-11-12 8:14 Slow performance after ~4.5TB Linas Jankauskas
2012-11-12 9:04 ` Dave Chinner [this message]
2012-11-12 9:46 ` Linas Jankauskas
2012-11-12 12:32 ` Dave Chinner
2012-11-12 13:58 ` Linas Jankauskas
2012-11-12 22:36 ` Dave Chinner
2012-11-13 9:13 ` Linas Jankauskas
2012-11-13 19:50 ` Dave Chinner
2012-11-14 9:01 ` Linas Jankauskas
2012-11-14 21:13 ` Dave Chinner
2012-11-15 8:34 ` Linas Jankauskas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20121112090448.GS24575@dastard \
--to=david@fromorbit.com \
--cc=linas.j@iv.lt \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox