bcache vs flashcache vs cache tiering

bcache vs flashcache vs cache tiering

[Dongsheng Yang]

Hi Sage and all,
     We are going to use SSDs for cache in ceph. But I am not sure which 
one is the best solution, bcache? flashcache? or cache tier?

I found there are some CAUTION in ceph.com about cache tiering. Is cache 
tiering is already production ready? especially for rbd.

thanx in advance.
Yang

Re: bcache vs flashcache vs cache tiering

[Dongsheng Yang]

thanx Nick, Gregory and Wido,
     So at least, we can say the cache tiering in Jewel is stable enough 
I think.
I like cache tiering more than the others, but yes, there is a problem 
about cache tiering in
flushing data between different nodes, which are not a problem in local 
caching solution.

guys:
     Is there any plan to enhance cache tiering to solve such problem? 
Or as Nick asked, is
that cache tiering fading away?

Yang

On 15/02/2017, 06:42, Nick Fisk wrote:
>> -----Original Message-----
>> From: Gregory Farnum [mailto:gfarnum-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
>> Sent: 14 February 2017 21:05
>> To: Wido den Hollander <wido-fspyXLx8qC4@public.gmane.org>
>> Cc: Dongsheng Yang <dongsheng.yang-zgCPCz31fAiwd6j6Zg/5ug@public.gmane.org>; Nick Fisk
>> <nick-ksME7r3P/wO1Qrn1Bg8BZw@public.gmane.org>; Ceph Users <ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org>
>> Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering
>>
>> On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander <wido-fspyXLx8qC4@public.gmane.org>
>> wrote:
>>>> Op 14 februari 2017 om 11:14 schreef Nick Fisk <nick-ksME7r3P/wO1Qrn1Bg8BZw@public.gmane.org>:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: ceph-users [mailto:ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org] On
>>>>> Behalf Of Dongsheng Yang
>>>>> Sent: 14 February 2017 09:01
>>>>> To: Sage Weil <notifications-9UaJU3cA/F/QT0dZR+AlfA@public.gmane.org>
>>>>> Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>>>>> Subject: [ceph-users] bcache vs flashcache vs cache tiering
>>>>>
>>>>> Hi Sage and all,
>>>>>       We are going to use SSDs for cache in ceph. But I am not sure
>>>>> which one is the best solution, bcache? flashcache? or cache
>>>> tier?
>>>>
>>>> I would vote for cache tier. Being able to manage it from within
>>>> Ceph, instead of having to manage X number of bcache/flashcache
>>>> instances, appeals to me more. Also last time I looked Flashcache
>>>> seems unmaintained and bcache might be going that way with talk of
>>>> this new bcachefs. Another point to consider is that Ceph has had a lot of
>> work done on it to ensure data consistency; I don't ever want to be in a
>> position where I'm trying to diagnose problems that might be being caused
>> by another layer sitting in-between Ceph and the Disk.
>>>> However, I know several people on here are using bcache and
>>>> potentially getting better performance than with cache tiering, so
>> hopefully someone will give their views.
>>> I am using Bcache on various systems and it performs really well. The
>> caching layer in Ceph is slow. Promoting Objects is slow and it also involves
>> additional RADOS lookups.
>>
>> Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use cases
>> where they're effective are still pretty limited and I think in-node caching has
>> a brighter future. We just don't like to maintain the global state that makes
>> separate caching locations viable and unless you're doing something
>> analogous to the supercomputing "burst buffers" (which some people are!),
>> it's going to be hard to beat something that doesn't have to pay the cost of
>> extra network hops/bandwidth.
>> Cache tiers are also not a feature that all the vendors support in their
>> downstream products, so it will probably see less ongoing investment than
>> you'd expect from such a system.
> Should that be taken as an unofficial sign that the tiering support is likely to fade away?
>
> I think both approaches have different strengths and probably the difference between a tiering system and a caching one is what causes some of the problems.
>
> If something like bcache is going to be the preferred approach, then I think more work needs to be done around certifying it for use with Ceph and allowing its behavior to be more controlled by Ceph as well. I assume there are issues around backfilling and scrubbing polluting the cache? Maybe you would want to be able to pass hints down from Ceph, which could also allow per pool cache behavior??
>
>> -Greg
>

Re: bcache vs flashcache vs cache tiering

[Dongsheng Yang]

BTW, is there any body using EnhanceIO?

On 02/15/2017 05:51 PM, Dongsheng Yang wrote:
> thanx Nick, Gregory and Wido,
>     So at least, we can say the cache tiering in Jewel is stable 
> enough I think.
> I like cache tiering more than the others, but yes, there is a problem 
> about cache tiering in
> flushing data between different nodes, which are not a problem in 
> local caching solution.
>
> guys:
>     Is there any plan to enhance cache tiering to solve such problem? 
> Or as Nick asked, is
> that cache tiering fading away?
>
> Yang
>
> On 15/02/2017, 06:42, Nick Fisk wrote:
>>> -----Original Message-----
>>> From: Gregory Farnum [mailto:gfarnum-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
>>> Sent: 14 February 2017 21:05
>>> To: Wido den Hollander <wido-fspyXLx8qC4@public.gmane.org>
>>> Cc: Dongsheng Yang <dongsheng.yang-zgCPCz31fAiwd6j6Zg/5ug@public.gmane.org>; Nick Fisk
>>> <nick-ksME7r3P/wO1Qrn1Bg8BZw@public.gmane.org>; Ceph Users <ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org>
>>> Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering
>>>
>>> On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander <wido-fspyXLx8qC4@public.gmane.org>
>>> wrote:
>>>>> Op 14 februari 2017 om 11:14 schreef Nick Fisk <nick-ksME7r3P/wO1Qrn1Bg8BZw@public.gmane.org>:
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: ceph-users [mailto:ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org] On
>>>>>> Behalf Of Dongsheng Yang
>>>>>> Sent: 14 February 2017 09:01
>>>>>> To: Sage Weil <notifications-9UaJU3cA/F/QT0dZR+AlfA@public.gmane.org>
>>>>>> Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>>>>>> Subject: [ceph-users] bcache vs flashcache vs cache tiering
>>>>>>
>>>>>> Hi Sage and all,
>>>>>>       We are going to use SSDs for cache in ceph. But I am not sure
>>>>>> which one is the best solution, bcache? flashcache? or cache
>>>>> tier?
>>>>>
>>>>> I would vote for cache tier. Being able to manage it from within
>>>>> Ceph, instead of having to manage X number of bcache/flashcache
>>>>> instances, appeals to me more. Also last time I looked Flashcache
>>>>> seems unmaintained and bcache might be going that way with talk of
>>>>> this new bcachefs. Another point to consider is that Ceph has had 
>>>>> a lot of
>>> work done on it to ensure data consistency; I don't ever want to be 
>>> in a
>>> position where I'm trying to diagnose problems that might be being 
>>> caused
>>> by another layer sitting in-between Ceph and the Disk.
>>>>> However, I know several people on here are using bcache and
>>>>> potentially getting better performance than with cache tiering, so
>>> hopefully someone will give their views.
>>>> I am using Bcache on various systems and it performs really well. The
>>> caching layer in Ceph is slow. Promoting Objects is slow and it also 
>>> involves
>>> additional RADOS lookups.
>>>
>>> Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use 
>>> cases
>>> where they're effective are still pretty limited and I think in-node 
>>> caching has
>>> a brighter future. We just don't like to maintain the global state 
>>> that makes
>>> separate caching locations viable and unless you're doing something
>>> analogous to the supercomputing "burst buffers" (which some people 
>>> are!),
>>> it's going to be hard to beat something that doesn't have to pay the 
>>> cost of
>>> extra network hops/bandwidth.
>>> Cache tiers are also not a feature that all the vendors support in 
>>> their
>>> downstream products, so it will probably see less ongoing investment 
>>> than
>>> you'd expect from such a system.
>> Should that be taken as an unofficial sign that the tiering support 
>> is likely to fade away?
>>
>> I think both approaches have different strengths and probably the 
>> difference between a tiering system and a caching one is what causes 
>> some of the problems.
>>
>> If something like bcache is going to be the preferred approach, then 
>> I think more work needs to be done around certifying it for use with 
>> Ceph and allowing its behavior to be more controlled by Ceph as well. 
>> I assume there are issues around backfilling and scrubbing polluting 
>> the cache? Maybe you would want to be able to pass hints down from 
>> Ceph, which could also allow per pool cache behavior??
>>
>>> -Greg
>>
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>