Re: Poor read performance in KVM

[Mark Nelson <mark.nelson@inktank.com>]

On 7/29/12 10:31 AM, Vladimir Bashkirtsev wrote:
> On 21/07/12 02:12, Tommi Virtanen wrote:
>> But it leaves me with very final question: should we rely on btrfs at
>> this
>> point given it is having such major faults? What if I will use well
>> tested
>> by time ext4?
>> You might want to try xfs. We hear/see problems with all three, but
>> xfs currently seems to have the best long-term performance and
>> reliability.
>>
>> I'm not sure if anyone's run detailed tests with ext4 after the
>> xattrs-in-leveldb feature; before that, we ran into fs limitations.
> Just reporting back what was going on for last week. I have rebuilt all
> OSDs with fresh btrfs and leaf size of 64K. Straight after rebuild
> everything was flying! But mysql processing I wrote about continued and
> whole cluster was brought again to a stand still in a week. I have done
> some investigation as to causes and it appears that fragmentation went
> ballistic. Reading somewhere on the net I have seen suggestion that if
> cow is not really needed then btrfs mounted with nocow option less
> likely to get overly fragmented. Haven't tried it actually but wondering
> will ceph cope well with nocow? ie does it rely on cow feature?
> Something tells me that as ceph can run on FS which does not have cow we
> actually can mount nocow. Just need some confirmation from devs.
>

Hi Vladimir,

I haven't tried nocow, but we did try with autodefrag which didn't do 
much to improve the situation.  So far most of the degradation I've seen 
was also with small writes.

> In the mean time I opted to convert all OSDs to xfs. Even after
> rebuilding only two OSDs performance boost is apparent again. So it
> appears that btrfs as it currently is in 3.4.6 is not up to prime time
> and good number of random writes consistently bring it to a halt.
>
> As xfs apparently have its own share of problems when ageing I think
> that periodic online defragmentation may bring xfs back to reasonable
> condition. Have anyone tried xfs defragmentation while ceph uses it?

I haven't tried doing xfs defragmentation while ceph is running, though 
we did test performance degradation on XFS.  XFS started out slower than 
btrfs but degraded more slowly than btrfs, so overall ended up faster by 
the end of the test. It would be interesting to try doing periodic 
defragmentation and see if that brings the performance back up.

Mark