Newstore Update Part 2

Newstore Update Part 2

[Mark Nelson]

Hi Guys,

At the performance meeting this week we discussed the possibility of 
changing how newstore handles large partial overwrites.  To determine if 
this is useful, we look at rados bench and rbd performance of newstore 
at various IO sizes.  I've also included some rados bench read results 
as folks were interested in those.  On the graphs I've tried to mark 
interesting improvements/regressions and also some questions/comments. 
These tests were run without overlay which may help small IO performance 
in some cases.

Graphs are here:

http://nhm.ceph.com/newstore/8c8c5903_rbd_rados_tests.pdf

Thanks,
Mark

Re: Newstore Update Part 2

[Gregory Farnum]

On Fri, May 8, 2015 at 4:21 PM, Mark Nelson <mnelson@redhat.com> wrote:
> Hi Guys,
>
> At the performance meeting this week we discussed the possibility of
> changing how newstore handles large partial overwrites.

Er, what was the potential change?

> To determine if
> this is useful, we look at rados bench and rbd performance of newstore at
> various IO sizes.  I've also included some rados bench read results as folks
> were interested in those.  On the graphs I've tried to mark interesting
> improvements/regressions and also some questions/comments. These tests were
> run without overlay which may help small IO performance in some cases.
>
> Graphs are here:
>
> http://nhm.ceph.com/newstore/8c8c5903_rbd_rados_tests.pdf
>
> Thanks,
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Newstore Update Part 2

[Mark Nelson]

On 05/08/2015 07:19 PM, Gregory Farnum wrote:
> On Fri, May 8, 2015 at 4:21 PM, Mark Nelson <mnelson@redhat.com> wrote:
>> Hi Guys,
>>
>> At the performance meeting this week we discussed the possibility of
>> changing how newstore handles large partial overwrites.
>
> Er, what was the potential change?

Basically breaking the object up into fragments on overwrite so that we 
don't do big updates to 4MB files behind the scenes.  Instead for 
overwrites of a certain size (say 1MB+) create a new fragment.

>
>> To determine if
>> this is useful, we look at rados bench and rbd performance of newstore at
>> various IO sizes.  I've also included some rados bench read results as folks
>> were interested in those.  On the graphs I've tried to mark interesting
>> improvements/regressions and also some questions/comments. These tests were
>> run without overlay which may help small IO performance in some cases.
>>
>> Graphs are here:
>>
>> http://nhm.ceph.com/newstore/8c8c5903_rbd_rados_tests.pdf
>>
>> Thanks,
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Newstore Update Part 2

[Milosz Tanski]

On Fri, May 8, 2015 at 7:21 PM, Mark Nelson <mnelson@redhat.com> wrote:
>
> Hi Guys,
>
> At the performance meeting this week we discussed the possibility of changing how newstore handles large partial overwrites.  To determine if this is useful, we look at rados bench and rbd performance of newstore at various IO sizes.  I've also included some rados bench read results as folks were interested in those.  On the graphs I've tried to mark interesting improvements/regressions and also some questions/comments. These tests were run without overlay which may help small IO performance in some cases.
>
> Graphs are here:
>
> http://nhm.ceph.com/newstore/8c8c5903_rbd_rados_tests.pdf
>
> Thanks,
> Mark


May I suggest that we try to test new store aging (no pun intended).
I'm worried that inevitably we'll have cases crop up where newstore
degrades after prolonged usage (real-world). Case like:

1) Behavior after a few TB has been written
2) Behavior of OSDs with newstore after it's 50% full.
3) Behavior after many write/del operations.

I'm sure I'm missing many scenarios. We're dealing with quintessential
filesystem design problem, how do deliver predictable performance
after it's been used for a long time. Design maturing pains.

-- 
Milosz Tanski
CTO
16 East 34th Street, 15th floor
New York, NY 10016

p: 646-253-9055
e: milosz@adfin.com

Re: Newstore Update Part 2

[Mark Nelson]

On 05/11/2015 10:21 AM, Milosz Tanski wrote:
> On Fri, May 8, 2015 at 7:21 PM, Mark Nelson <mnelson@redhat.com> wrote:
>>
>> Hi Guys,
>>
>> At the performance meeting this week we discussed the possibility of changing how newstore handles large partial overwrites.  To determine if this is useful, we look at rados bench and rbd performance of newstore at various IO sizes.  I've also included some rados bench read results as folks were interested in those.  On the graphs I've tried to mark interesting improvements/regressions and also some questions/comments. These tests were run without overlay which may help small IO performance in some cases.
>>
>> Graphs are here:
>>
>> http://nhm.ceph.com/newstore/8c8c5903_rbd_rados_tests.pdf
>>
>> Thanks,
>> Mark
>
>
> May I suggest that we try to test new store aging (no pun intended).
> I'm worried that inevitably we'll have cases crop up where newstore
> degrades after prolonged usage (real-world). Case like:
>
> 1) Behavior after a few TB has been written
> 2) Behavior of OSDs with newstore after it's 50% full.
> 3) Behavior after many write/del operations.
>
> I'm sure I'm missing many scenarios. We're dealing with quintessential
> filesystem design problem, how do deliver predictable performance
> after it's been used for a long time. Design maturing pains.


We do definitely need to do this, though I was hoping to wait until we 
have things a bit more stabilized since aging tests take so long to run 
and the results could change depending on if/how we handle object 
fragments for overwrites.

Mark