Re: [PATCH 02/20] Btrfs: initialize new bitmaps' list

[Christian Brunner <chb@muc.de>]

2011/12/1 Christian Brunner <chb@muc.de>:
> 2011/12/1 Alexandre Oliva <oliva@lsd.ic.unicamp.br>:
>> On Nov 29, 2011, Christian Brunner <chb@muc.de> wrote:
>>
>>> When I'm doing havy reading in our ceph cluster. The load and wait-io
>>> on the patched servers is higher than on the unpatched ones.
>>
>> That's unexpected.

In the mean time I know, that it's not related to the reads.

>> I suppose I could wave my hands while explaining that you're getting
>> higher data throughput, so it's natural that it would take up more
>> resources, but that explanation doesn't satisfy me.  I suppose
>> allocation might have got slightly more CPU intensive in some cases, as
>> we now use bitmaps where before we'd only use the cheaper-to-allocate
>> extents.  But that's unsafisfying as well.
>
> I must admit, that I do not completely understand the difference
> between bitmaps and extents.
>
> From what I see on my servers, I can tell, that the degradation over
> time is gone. (Rebooting the servers every day is no longer needed.
> This is a real plus.) But the performance compared to a freshly
> booted, unpatched server is much slower with my ceph workload.
>
> I wonder if it would make sense to initialize the list field only,
> when the cluster setup fails? This would avoid the fallback to the
> much unclustered allocation and would give us the cheaper-to-allocate
> extents.

I've now tried various combinations of you patches and I can really
nail it down to this one line.

With this patch applied I get much higher write-io values than without
it. Some of the other patches help to reduce the effect, but it's
still significant.

iostat on an unpatched node is giving me:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
avgrq-sz avgqu-sz   await  svctm  %util
sda             105.90     0.37   15.42   14.48  2657.33   560.13
107.61     1.89   62.75   6.26  18.71

while on a node with this patch it's
sda             128.20     0.97   11.10   57.15  3376.80   552.80
57.58    20.58  296.33   4.16  28.36


Also interesting, is the fact that the average request size on the
patched node is much smaller.

Josef was telling me, that this could be related to the number of
bitmaps we write out, but I've no idea how to trace this.

I would be very happy if someone could give me a hint on what to do
next, as this is one of the last remaining issues with our ceph
cluster.

Thanks,
Christian