Re: Crash and strange things on MDS

[Kevin Decherf <kevin@kdecherf.com>]

On Tue, Feb 19, 2013 at 05:09:30PM -0800, Gregory Farnum wrote:
> On Tue, Feb 19, 2013 at 5:00 PM, Kevin Decherf <kevin@kdecherf.com> wrote:
> > On Tue, Feb 19, 2013 at 10:15:48AM -0800, Gregory Farnum wrote:
> >> Looks like you've got ~424k dentries pinned, and it's trying to keep
> >> 400k inodes in cache. So you're still a bit oversubscribed, yes. This
> >> might just be the issue where your clients are keeping a bunch of
> >> inodes cached for the VFS (http://tracker.ceph.com/issues/3289).
> >
> > Thanks for the analyze. We use only one ceph-fuse client at this time
> > which makes all "high-load" commands like rsync, tar and cp on a huge
> > amount of files. Well, I will replace it by the kernel client.
> 
> Oh, that bug is just an explanation of what's happening; I believe it
> exists in the kernel client as well.

After setting the mds cache size to 900k, storms are gone.
However we continue to observe high latency on some clients (always the
same clients): each IO takes between 40 and 90ms (for example with
Wordpress, it takes ~20 seconds to load all needed files...).
With a non-laggy client, IO requests take less than 1ms.

-- 
Kevin Decherf - @Kdecherf
GPG C610 FE73 E706 F968 612B E4B2 108A BD75 A81E 6E2F
http://kdecherf.com