From: Bernd Schubert <bschubert@ddn.com>
To: Mark Seger <mjseger@gmail.com>, Linux fs XFS <xfs@oss.sgi.com>
Cc: Laurence Oberman <loberman@redhat.com>
Subject: Re: xfs and swift
Date: Mon, 25 Jan 2016 18:24:42 +0000 [thread overview]
Message-ID: <56A66869.3080506@ddn.com> (raw)
In-Reply-To: <CAC2B=ZGX2bkEhdgCrpS2X5v+SpAg0jtxZ19vk_9+O9aHME-FSA@mail.gmail.com>
Hi Mark!
On 01/06/2016 04:15 PM, Mark Seger wrote:
> I've recently found the performance our development swift system is
> degrading over time as the number of objects/files increases. This is a
> relatively small system, each server has 3 400GB disks. The system I'm
> currently looking at has about 70GB tied up in slabs alone, close to 55GB
> in xfs inodes and ili, and about 2GB free. The kernel
> is 3.14.57-1-amd64-hlinux.
>
> Here's the way the filesystems are mounted:
>
> /dev/sdb1 on /srv/node/disk0 type xfs
> (rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=1536,noquota)
>
> I can do about 2000 1K file creates/sec when running 2 minute PUT tests at
> 100 threads. If I repeat that tests for multiple hours, I see the number
> of IOPS steadily decreasing to about 770 and the very next run it drops to
> 260 and continues to fall from there. This happens at about 12M files.
>
> The directory structure is 2 tiered, with 1000 directories per tier so we
> can have about 1M of them, though they don't currently all exist.
This sounds pretty much like hash directories as used by some parallel
file systems (Lustre and in the past BeeGFS). For us the file create
slow down was due to lookup in directories if a file with the same name
already exists. At least for ext4 it was rather easy to demonstrate that
simply caching directory blocks would eliminate that issue.
We then considered working on a better kernel cache, but in the end
simply found a way to get rid of such a simple directory structure in
BeeGFS and changed it to a more complex layout, but with less random
access and so we could eliminate the main reason for the slow down.
Now I have no idea what a "swift system" is and in which order it
creates and accesses those files and if it would be possible to change
the access pattern. One thing you might try and which should work much
better since 3.11 is the vfs_cache_pressure setting. The lower it is the
less dentries/inodes are dropped from cache when pages are needed for
file data.
Cheers,
Bernd
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2016-01-25 18:24 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-01-06 15:15 xfs and swift Mark Seger
2016-01-06 22:04 ` Dave Chinner
2016-01-06 22:10 ` Dave Chinner
2016-01-06 22:46 ` Mark Seger
2016-01-06 23:49 ` Dave Chinner
2016-01-25 16:38 ` Mark Seger
2016-02-01 5:27 ` Dave Chinner
2016-01-25 18:24 ` Bernd Schubert [this message]
2016-01-25 19:00 ` Mark Seger
2016-01-25 19:33 ` Bernd Schubert
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56A66869.3080506@ddn.com \
--to=bschubert@ddn.com \
--cc=loberman@redhat.com \
--cc=mjseger@gmail.com \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox