Re: Slow ceph fs performance

[Mark Nelson <mark.nelson@inktank.com>]

On 10/01/2012 11:47 AM, Tommi Virtanen wrote:
> On Thu, Sep 27, 2012 at 11:04 AM, Gregory Farnum<greg@inktank.com>  wrote:
>> However, my suspicion is that you're limited by metadata throughput
>> here. How large are your files? There might be some MDS or client
>> tunables we can adjust, but rsync's workload is a known weak spot for
>> CephFS.
>
> I feel like people are missing this part of Greg's message. Everyone
> is so busy benchmarking RADOS small I/O, but what if it's currently
> bottlenecked by all the file-level access operations that interact
> with the MDS? Rsync causes a ton of those.
>
> If you want to benchmark just the small IO, you can't compare rsync to rsync.
>
> If you want to benchmark just the metadata part, rsync with 0-size
> files might actually be an interesting workload.

I guess most of the small IO testing we've seen/done has been without 
CephFS at all.  It's entirely possible that the MDS is slowing things 
down with an rsync workload like this on a fresh filesystem though. 
Having said that, I don't like the way that our small IO performance 
behaves (especially over time) when doing something like RADOS Bench. 
It definitely seems like there is some pretty nasty underlying 
filesystem metadata fragmentation or something going on after a while.

Mark