Re: Crash and strange things on MDS

[Kevin Decherf <kevin@kdecherf.com>]

On Tue, Feb 26, 2013 at 12:26:17PM -0800, Gregory Farnum wrote:
> On Tue, Feb 26, 2013 at 11:58 AM, Kevin Decherf <kevin@kdecherf.com> wrote:
> > We have one folder per application (php, java, ruby). Every application has
> > small (<1M) files. The folder is mounted by only one client by default.
> >
> > In case of overload, another clients spawn to mount the same folder and
> > access the same files.
> >
> > In the following test, only one client was used to serve the
> > application (a website using wordpress).
> >
> > I made the test with strace to see the time of each IO request (strace -T
> > -e trace=file) and I noticed the same pattern:
> >
> > ...
> > [pid  4378] stat("/data/wp-includes/user.php", {st_mode=S_IFREG|0750, st_size=28622, ...}) = 0 <0.033409>
> > [pid  4378] lstat("/data/wp-includes/user.php", {st_mode=S_IFREG|0750, st_size=28622, ...}) = 0 <0.081642>
> > [pid  4378] open("/data/wp-includes/user.php", O_RDONLY) = 5 <0.041138>
> > [pid  4378] stat("/data/wp-includes/meta.php", {st_mode=S_IFREG|0750, st_size=10896, ...}) = 0 <0.082303>
> > [pid  4378] lstat("/data/wp-includes/meta.php", {st_mode=S_IFREG|0750, st_size=10896, ...}) = 0 <0.004090>
> > [pid  4378] open("/data/wp-includes/meta.php", O_RDONLY) = 5 <0.081929>
> > ...
> >
> > ~250 files were accessed for only one request (thanks Wordpress.).
> 
> Okay, that is slower than I'd expect, even for an across-the-wire request...
> 
> > The fs is mounted with these options: rw,noatime,name=<hidden>,secret=<hidden>,nodcache.
> 
> What kernel and why are you using nodcache?

We use kernel 3.7.0. nodcache is enabled by default (we only specify user and
secretfile as mount options) and I didn't find it in the documentation of
mount.ceph.

> Did you have problems
> without that mount option? That's forcing an MDS access for most
> operations, rather than using local data.

Good question, I will try it (-o dcache?).

> > I have a debug (debug_mds=20) log of the active mds during this test if you want.
> 
> Yeah, can you post it somewhere?

Upload in progress :-)

-- 
Kevin Decherf - @Kdecherf
GPG C610 FE73 E706 F968 612B E4B2 108A BD75 A81E 6E2F
http://kdecherf.com