dm-cache module

dm-cache module

[Olivier B.]

Hi,

I'm looking at the dm-cache module from Ming Zhao ( 
http://users.cis.fiu.edu/~zhaom/dmcache/index.html ), and would like to 
know if there is a problem to include it upstream.

It looks stable for a while, and some people seem to use it in production.

Olivier

Re: dm-cache module

[Mike Snitzer]

On Sat, Mar 20 2010 at  8:03pm -0400,
Olivier B. <dm.list@daevel.fr> wrote:

> Hi,
> 
> I'm looking at the dm-cache module from Ming Zhao (
> http://users.cis.fiu.edu/~zhaom/dmcache/index.html ), and would like
> to know if there is a problem to include it upstream.
> 
> It looks stable for a while, and some people seem to use it in production.

Such a DM target would be quite useful to have upstream given how
prevalent SSDs have become.

Other work has been a priority but a caching DM target is definitely on
the TODO.  If others can help take steps to make a robust caching DM
target a reality then we should be able to get an implementation
upstream sooner rather than later.

Just curious: where have you gotten this insight about dm-cache being
stable and used in production?  That should help dm-cache's cause if it
is submitted for review.

Seems Ming posted dm-cache to dm-devel (as a large monolithic patch) on
8/24/07.

dm-cache would need to be rebased to the latest upstream kernel.org
kernel (e.g. Linux >= 2.6.34-rc1).

Ideally dm-cache would be resubmitted to dm-devel incrementally such
that basic infrastructure is introduced in early patches and more
advanced capabilities (e.g. writeback) follow in later patches.  Each
patch should have a corresponding patch header the documents what the
patch provides.

Writeback support is arguably the most useful aspect of a caching DM
target so emphasis must be placed on its review/implementation.  Using
SSD as a persistent writeback cache should afford us much more fault
tolerance in the face of system crashes during writeback to the slower
media.

I know Heinz has taken steps to implement a caching DM target.  So
reconciling dm-cache's design and implementation with what Heinz has
would be a worthwhile component of the dm-cache review (assuming Heinz's
implementation is also posted for review).

Mike

Re: dm-cache module

[Olivier B.]

On 21/03/2010 04:37, Mike Snitzer wrote:
> Such a DM target would be quite useful to have upstream given how
> prevalent SSDs have become.
>    

I'm totally agree with that.


> Just curious: where have you gotten this insight about dm-cache being
> stable and used in production?  That should help dm-cache's cause if it
> is submitted for review.
>    
Well I wrote to Ming before post here, and found comments about the 
module in the project OpenFiler which seem to integrate it : 
https://forums.openfiler.com/viewtopic.php?id=3595


> Writeback support is arguably the most useful aspect of a caching DM
> target so emphasis must be placed on its review/implementation.  Using
> SSD as a persistent writeback cache should afford us much more fault
> tolerance in the face of system crashes during writeback to the slower
> media.
>    

Yes it's also for the writeback feature I'm interested.

I don't think be enough skilled to help for update the code, but maybe I 
can do some tests.

Thanks,
Olivier

Re: dm-cache module

[Heinz Mauelshagen]

On Sun, 2010-03-21 at 00:56 -0400, Ming Zhao wrote:
> Hi Mike,
> 
> Thank you very much for your advice!
> 
> I can revise dm-cache code and resubmit it as you suggested. I would
> also love to know Heinz's progress on his implementation and work with
> him if there anything I could contribute.

Hi all,

this is a list of the functions of my dm-hstore
device-mapper target implementation:

o caches reads and writes keeping persistent state metadata.
o writes back in order to enhance streaming performance
  on fragmented access pattern.
o can run on top of readonly original device
o if so, writes back any dirty areas when set readwrite
  (useful for tests)
o readonly <-> readwrite access changes supported via message interface
o initializes metadata for extents in cache in the background
  in order to fasten cache construction
o supports cache resizing via message interface or constructor
o keeps metadata persistent by default
o stores CRCs with metadata for integrity checks
o stores versions with metadata to support future metadata migration

Test features only:
o transient cache
o cache write through


Provides very good performance on SSD cache backing stores.

Has been shelved for a while because of other priorities so I need to
rebase it to the actual kernel.

Regards,
Heinz

> 
> Best,
> - Ming
> 
> 
> 
> On Sat, Mar 20, 2010 at 11:37 PM, Mike Snitzer <snitzer@redhat.com> wrote:
> > On Sat, Mar 20 2010 at  8:03pm -0400,
> > Olivier B. <dm.list@daevel.fr> wrote:
> >
> >> Hi,
> >>
> >> I'm looking at the dm-cache module from Ming Zhao (
> >> http://users.cis.fiu.edu/~zhaom/dmcache/index.html ), and would like
> >> to know if there is a problem to include it upstream.
> >>
> >> It looks stable for a while, and some people seem to use it in production.
> >
> > Such a DM target would be quite useful to have upstream given how
> > prevalent SSDs have become.
> >
> > Other work has been a priority but a caching DM target is definitely on
> > the TODO.  If others can help take steps to make a robust caching DM
> > target a reality then we should be able to get an implementation
> > upstream sooner rather than later.
> >
> > Just curious: where have you gotten this insight about dm-cache being
> > stable and used in production?  That should help dm-cache's cause if it
> > is submitted for review.
> >
> > Seems Ming posted dm-cache to dm-devel (as a large monolithic patch) on
> > 8/24/07.
> >
> > dm-cache would need to be rebased to the latest upstream kernel.org
> > kernel (e.g. Linux >= 2.6.34-rc1).
> >
> > Ideally dm-cache would be resubmitted to dm-devel incrementally such
> > that basic infrastructure is introduced in early patches and more
> > advanced capabilities (e.g. writeback) follow in later patches.  Each
> > patch should have a corresponding patch header the documents what the
> > patch provides.
> >
> > Writeback support is arguably the most useful aspect of a caching DM
> > target so emphasis must be placed on its review/implementation.  Using
> > SSD as a persistent writeback cache should afford us much more fault
> > tolerance in the face of system crashes during writeback to the slower
> > media.
> >
> > I know Heinz has taken steps to implement a caching DM target.  So
> > reconciling dm-cache's design and implementation with what Heinz has
> > would be a worthwhile component of the dm-cache review (assuming Heinz's
> > implementation is also posted for review).
> >
> > Mike
> >

Re: dm-cache module

[Olivier B.]

On 22/03/2010 16:05, Heinz Mauelshagen wrote:
> On Sun, 2010-03-21 at 00:56 -0400, Ming Zhao wrote:
>    
>> Hi Mike,
>>
>> Thank you very much for your advice!
>>
>> I can revise dm-cache code and resubmit it as you suggested. I would
>> also love to know Heinz's progress on his implementation and work with
>> him if there anything I could contribute.
>>      
> Hi all,
>
> this is a list of the functions of my dm-hstore
> device-mapper target implementation:
>
> o caches reads and writes keeping persistent state metadata.
> o writes back in order to enhance streaming performance
>    on fragmented access pattern.
> o can run on top of readonly original device
> o if so, writes back any dirty areas when set readwrite
>    (useful for tests)
> o readonly<->  readwrite access changes supported via message interface
> o initializes metadata for extents in cache in the background
>    in order to fasten cache construction
> o supports cache resizing via message interface or constructor
> o keeps metadata persistent by default
> o stores CRCs with metadata for integrity checks
> o stores versions with metadata to support future metadata migration
>
> Test features only:
> o transient cache
> o cache write through
>
>
> Provides very good performance on SSD cache backing stores.
>
> Has been shelved for a while because of other priorities so I need to
> rebase it to the actual kernel.
>
> Regards,
> Heinz
>
>    
Hi,

Facebook have just released "FlashCache" : 
http://github.com/facebook/flashcache
in the documentation we can read :

Flashcache is built using the Linux Device Mapper (DM), part of the 
Linux Storage Stack infrastructure that facilitates building SW-RAID and 
other components.

So it's an other implementation of the same concept, no ?

Re: dm-cache module

[Heinz Mauelshagen]

On Wed, 2010-04-28 at 03:33 +0200, Olivier B. wrote:
> On 22/03/2010 16:05, Heinz Mauelshagen wrote:
> > On Sun, 2010-03-21 at 00:56 -0400, Ming Zhao wrote:
> >    
> >> Hi Mike,
> >>
> >> Thank you very much for your advice!
> >>
> >> I can revise dm-cache code and resubmit it as you suggested. I would
> >> also love to know Heinz's progress on his implementation and work with
> >> him if there anything I could contribute.
> >>      
> > Hi all,
> >
> > this is a list of the functions of my dm-hstore
> > device-mapper target implementation:
> >
> > o caches reads and writes keeping persistent state metadata.
> > o writes back in order to enhance streaming performance
> >    on fragmented access pattern.
> > o can run on top of readonly original device
> > o if so, writes back any dirty areas when set readwrite
> >    (useful for tests)
> > o readonly<->  readwrite access changes supported via message interface
> > o initializes metadata for extents in cache in the background
> >    in order to fasten cache construction
> > o supports cache resizing via message interface or constructor
> > o keeps metadata persistent by default
> > o stores CRCs with metadata for integrity checks
> > o stores versions with metadata to support future metadata migration
> >
> > Test features only:
> > o transient cache
> > o cache write through
> >
> >
> > Provides very good performance on SSD cache backing stores.
> >
> > Has been shelved for a while because of other priorities so I need to
> > rebase it to the actual kernel.
> >
> > Regards,
> > Heinz
> >
> >    
> Hi,
> 
> Facebook have just released "FlashCache" : 
> http://github.com/facebook/flashcache
> in the documentation we can read :

> Flashcache is built using the Linux Device Mapper (DM), part of the 
> Linux Storage Stack infrastructure that facilitates building SW-RAID and 
> other components.
> 
> So it's an other implementation of the same concept, no ?

Yes, it looks like it. I'll have a closer look how it differs form mine.

Heinz

> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: dm-cache module

[Heinz Mauelshagen]

Olivier,

some initial impressions:

I've spent an hour porting the source code to 2.6.34-rc4 and doing first
tests.

Those unveil, that it only works with cache block size equal to page
size (4KB here). With large cache block sizes it doesn't cache at all.

That may well be the result of a porting flaw but I have to have a
closer look to tell. It works with quite decent performance with that
block size. Code looks like heavily derived from the dm-cache source and
overly complicated.

Destroying a flashcache mapping with a large dirty cache flushs the
whole dirty content out which is not suitable to practical dm needs
requiring a mapping to be suspendable quickly during reconfiguration.

Regards,
Heinz

On Wed, 2010-04-28 at 10:21 +0200, Heinz Mauelshagen wrote:
> On Wed, 2010-04-28 at 03:33 +0200, Olivier B. wrote:
> > On 22/03/2010 16:05, Heinz Mauelshagen wrote:
> > > On Sun, 2010-03-21 at 00:56 -0400, Ming Zhao wrote:
> > >    
> > >> Hi Mike,
> > >>
> > >> Thank you very much for your advice!
> > >>
> > >> I can revise dm-cache code and resubmit it as you suggested. I would
> > >> also love to know Heinz's progress on his implementation and work with
> > >> him if there anything I could contribute.
> > >>      
> > > Hi all,
> > >
> > > this is a list of the functions of my dm-hstore
> > > device-mapper target implementation:
> > >
> > > o caches reads and writes keeping persistent state metadata.
> > > o writes back in order to enhance streaming performance
> > >    on fragmented access pattern.
> > > o can run on top of readonly original device
> > > o if so, writes back any dirty areas when set readwrite
> > >    (useful for tests)
> > > o readonly<->  readwrite access changes supported via message interface
> > > o initializes metadata for extents in cache in the background
> > >    in order to fasten cache construction
> > > o supports cache resizing via message interface or constructor
> > > o keeps metadata persistent by default
> > > o stores CRCs with metadata for integrity checks
> > > o stores versions with metadata to support future metadata migration
> > >
> > > Test features only:
> > > o transient cache
> > > o cache write through
> > >
> > >
> > > Provides very good performance on SSD cache backing stores.
> > >
> > > Has been shelved for a while because of other priorities so I need to
> > > rebase it to the actual kernel.
> > >
> > > Regards,
> > > Heinz
> > >
> > >    
> > Hi,
> > 
> > Facebook have just released "FlashCache" : 
> > http://github.com/facebook/flashcache
> > in the documentation we can read :
> 
> > Flashcache is built using the Linux Device Mapper (DM), part of the 
> > Linux Storage Stack infrastructure that facilitates building SW-RAID and 
> > other components.
> > 
> > So it's an other implementation of the same concept, no ?
> 
> Yes, it looks like it. I'll have a closer look how it differs form mine.
> 
> Heinz
> 
> > 
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: dm-cache module

[Phillip Susi]

What ever came of this?  It doesn't look like either competing cache 
target has made it into mainline yet.

It looks like two caching systems have made it into mainline, but 
neither are usable for having an ssd cache a large hard disk. 
CleanCache isn't usable because it does not have a backend that can use 
a block device to hold the cache, and FsCache is only supported by nfs 
and cifs.

Apparently Intel has added the ability to use an SSD to cache a big hard 
disk to their matrix storage fakeraid in the new Z68 chipset and it is 
generating a good deal of interest.  It would be nice to be able to do 
something similar in Linux.

On 4/28/2010 10:02 AM, Heinz Mauelshagen wrote:
> Olivier,
>
> some initial impressions:
>
> I've spent an hour porting the source code to 2.6.34-rc4 and doing first
> tests.
>
> Those unveil, that it only works with cache block size equal to page
> size (4KB here). With large cache block sizes it doesn't cache at all.
>
> That may well be the result of a porting flaw but I have to have a
> closer look to tell. It works with quite decent performance with that
> block size. Code looks like heavily derived from the dm-cache source and
> overly complicated.
>
> Destroying a flashcache mapping with a large dirty cache flushs the
> whole dirty content out which is not suitable to practical dm needs
> requiring a mapping to be suspendable quickly during reconfiguration.
>
> Regards,
> Heinz
>
> On Wed, 2010-04-28 at 10:21 +0200, Heinz Mauelshagen wrote:
>> On Wed, 2010-04-28 at 03:33 +0200, Olivier B. wrote:
>>> On 22/03/2010 16:05, Heinz Mauelshagen wrote:
>>>> On Sun, 2010-03-21 at 00:56 -0400, Ming Zhao wrote:
>>>>
>>>>> Hi Mike,
>>>>>
>>>>> Thank you very much for your advice!
>>>>>
>>>>> I can revise dm-cache code and resubmit it as you suggested. I would
>>>>> also love to know Heinz's progress on his implementation and work with
>>>>> him if there anything I could contribute.
>>>>>
>>>> Hi all,
>>>>
>>>> this is a list of the functions of my dm-hstore
>>>> device-mapper target implementation:
>>>>
>>>> o caches reads and writes keeping persistent state metadata.
>>>> o writes back in order to enhance streaming performance
>>>>     on fragmented access pattern.
>>>> o can run on top of readonly original device
>>>> o if so, writes back any dirty areas when set readwrite
>>>>     (useful for tests)
>>>> o readonly<->   readwrite access changes supported via message interface
>>>> o initializes metadata for extents in cache in the background
>>>>     in order to fasten cache construction
>>>> o supports cache resizing via message interface or constructor
>>>> o keeps metadata persistent by default
>>>> o stores CRCs with metadata for integrity checks
>>>> o stores versions with metadata to support future metadata migration
>>>>
>>>> Test features only:
>>>> o transient cache
>>>> o cache write through
>>>>
>>>>
>>>> Provides very good performance on SSD cache backing stores.
>>>>
>>>> Has been shelved for a while because of other priorities so I need to
>>>> rebase it to the actual kernel.
>>>>
>>>> Regards,
>>>> Heinz
>>>>
>>>>
>>> Hi,
>>>
>>> Facebook have just released "FlashCache" :
>>> http://github.com/facebook/flashcache
>>> in the documentation we can read :
>>
>>> Flashcache is built using the Linux Device Mapper (DM), part of the
>>> Linux Storage Stack infrastructure that facilitates building SW-RAID and
>>> other components.
>>>
>>> So it's an other implementation of the same concept, no ?
>>
>> Yes, it looks like it. I'll have a closer look how it differs form mine.
>>
>> Heinz

Re: dm-cache module

[Heinz Mauelshagen]

On Fri, 2011-06-03 at 15:16 -0400, Phillip Susi wrote:
> What ever came of this?  It doesn't look like either competing cache 
> target has made it into mainline yet.

Right.

We are working on a device-mapper caching target which shares code for
block, btree and transaction management with thin provisioning and
multiple snapshot targets which are aiming to go upstream shortly. This
is aiming at code sharing rather than distinct implementations of the
aforementioned functionalities.
All of those will receive LVM2 support as we go.

Heinz

> 
> It looks like two caching systems have made it into mainline, but 
> neither are usable for having an ssd cache a large hard disk. 
> CleanCache isn't usable because it does not have a backend that can use 
> a block device to hold the cache, and FsCache is only supported by nfs 
> and cifs.
> 
> Apparently Intel has added the ability to use an SSD to cache a big hard 
> disk to their matrix storage fakeraid in the new Z68 chipset and it is 
> generating a good deal of interest.  It would be nice to be able to do 
> something similar in Linux.
> 
> On 4/28/2010 10:02 AM, Heinz Mauelshagen wrote:
> > Olivier,
> >
> > some initial impressions:
> >
> > I've spent an hour porting the source code to 2.6.34-rc4 and doing first
> > tests.
> >
> > Those unveil, that it only works with cache block size equal to page
> > size (4KB here). With large cache block sizes it doesn't cache at all.
> >
> > That may well be the result of a porting flaw but I have to have a
> > closer look to tell. It works with quite decent performance with that
> > block size. Code looks like heavily derived from the dm-cache source and
> > overly complicated.
> >
> > Destroying a flashcache mapping with a large dirty cache flushs the
> > whole dirty content out which is not suitable to practical dm needs
> > requiring a mapping to be suspendable quickly during reconfiguration.
> >
> > Regards,
> > Heinz
> >
> > On Wed, 2010-04-28 at 10:21 +0200, Heinz Mauelshagen wrote:
> >> On Wed, 2010-04-28 at 03:33 +0200, Olivier B. wrote:
> >>> On 22/03/2010 16:05, Heinz Mauelshagen wrote:
> >>>> On Sun, 2010-03-21 at 00:56 -0400, Ming Zhao wrote:
> >>>>
> >>>>> Hi Mike,
> >>>>>
> >>>>> Thank you very much for your advice!
> >>>>>
> >>>>> I can revise dm-cache code and resubmit it as you suggested. I would
> >>>>> also love to know Heinz's progress on his implementation and work with
> >>>>> him if there anything I could contribute.
> >>>>>
> >>>> Hi all,
> >>>>
> >>>> this is a list of the functions of my dm-hstore
> >>>> device-mapper target implementation:
> >>>>
> >>>> o caches reads and writes keeping persistent state metadata.
> >>>> o writes back in order to enhance streaming performance
> >>>>     on fragmented access pattern.
> >>>> o can run on top of readonly original device
> >>>> o if so, writes back any dirty areas when set readwrite
> >>>>     (useful for tests)
> >>>> o readonly<->   readwrite access changes supported via message interface
> >>>> o initializes metadata for extents in cache in the background
> >>>>     in order to fasten cache construction
> >>>> o supports cache resizing via message interface or constructor
> >>>> o keeps metadata persistent by default
> >>>> o stores CRCs with metadata for integrity checks
> >>>> o stores versions with metadata to support future metadata migration
> >>>>
> >>>> Test features only:
> >>>> o transient cache
> >>>> o cache write through
> >>>>
> >>>>
> >>>> Provides very good performance on SSD cache backing stores.
> >>>>
> >>>> Has been shelved for a while because of other priorities so I need to
> >>>> rebase it to the actual kernel.
> >>>>
> >>>> Regards,
> >>>> Heinz
> >>>>
> >>>>
> >>> Hi,
> >>>
> >>> Facebook have just released "FlashCache" :
> >>> http://github.com/facebook/flashcache
> >>> in the documentation we can read :
> >>
> >>> Flashcache is built using the Linux Device Mapper (DM), part of the
> >>> Linux Storage Stack infrastructure that facilitates building SW-RAID and
> >>> other components.
> >>>
> >>> So it's an other implementation of the same concept, no ?
> >>
> >> Yes, it looks like it. I'll have a closer look how it differs form mine.
> >>
> >> Heinz

Re: dm-cache module

[Phillip Susi]

Well here we are a year and a half later, and I still don't see this.
Am I missing it or has it not materialized?

On 6/6/2011 6:06 AM, Heinz Mauelshagen wrote:
> On Fri, 2011-06-03 at 15:16 -0400, Phillip Susi wrote:
>> What ever came of this?  It doesn't look like either competing cache 
>> target has made it into mainline yet.
> 
> Right.
> 
> We are working on a device-mapper caching target which shares code for
> block, btree and transaction management with thin provisioning and
> multiple snapshot targets which are aiming to go upstream shortly. This
> is aiming at code sharing rather than distinct implementations of the
> aforementioned functionalities.
> All of those will receive LVM2 support as we go.
> 
> Heinz

Re: dm-cache module

[Alasdair G Kergon]

On Tue, Dec 11, 2012 at 04:49:15PM -0500, Phillip Susi wrote:
> Well here we are a year and a half later, and I still don't see this.
> Am I missing it or has it not materialized?
 
github is being used for development
  https://github.com/jthornber/

Joe/Heinz: Please would you post links to the best branches for people who are
interested in this to track?

Alasdair
-- 
agk@redhat.com