From: Dave Chinner <david@fromorbit.com>
To: Marcin Deranek <marcin.deranek@booking.com>
Cc: xfs@oss.sgi.com
Subject: Re: Performance degradation over time
Date: Thu, 11 Oct 2012 10:37:04 +1100 [thread overview]
Message-ID: <20121010233704.GC23644@dastard> (raw)
In-Reply-To: <20121010105142.148519ca@booking.com>
On Wed, Oct 10, 2012 at 10:51:42AM +0200, Marcin Deranek wrote:
> Hi,
>
> We are running XFS filesystem on one of out machines which is a big
> store (~3TB) of different data files (mostly images). Quite recently we
> experienced some performance problems - machine wasn't able to keep up
> with updates. After some investigation it turned out that open()
> syscalls (open for writing) were taking significantly more time than
> they should eg. 15-20ms vs 100-150us.
Which is clearly an IO latency vs cache hit latency.
> Some more info about our workload as I think it's important here:
> our XFS filesystem is exclusively used as data store, so we only
> read and write our data (we mostly write). When new update comes it's
> written to a temporary file eg.
>
> /mountpoint/some/path/.tmp/file
>
> When file is completely stored we move it to final location eg.
>
> /mountpoint/some/path/different/subdir/newname
>
> That means that we create lots of files in /mountpoint/some/path/.tmp
> directory, but directory is empty as they are moved (rename() syscall)
> shortly after file creation to a different directory on the same
> filesystem.
> The workaround which I found so far is to remove that directory
> (/mountpoint/some/path/.tmp in our case) with its content and re-create
> it. After this operation open() syscall goes down to 100-150us again.
> Is this a known problem ?
By emptying the directory, you are making it smaller and likely
causing it to be cached in memory again as new files are added to
it. Over time, blocks will be removed from the cache due to memory
pressure, and latencies will be seen again.
> Information regarding our system:
> CentOS 5.8 / kernel 2.6.18-308.el5 / kmod-xfs-0.4-2
Use a more recent distro. I reworked the metadata caching algorithms
a couple of years ago to avoid these sorts of problems with memory
reclaim.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2012-10-10 23:35 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-10-10 8:51 Performance degradation over time Marcin Deranek
2012-10-10 13:17 ` Stan Hoeppner
2012-10-10 14:31 ` Eric Sandeen
2012-10-11 8:33 ` Marcin Deranek
2012-10-11 9:15 ` Marcin Deranek
2012-10-14 19:31 ` Peter Grandi
2012-10-10 23:37 ` Dave Chinner [this message]
2012-10-11 8:42 ` Marcin Deranek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20121010233704.GC23644@dastard \
--to=david@fromorbit.com \
--cc=marcin.deranek@booking.com \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox