dm-cache: can the same cache be used with multiple origin devices?

dm-cache: can the same cache be used with multiple origin devices?

[Mears, Morgan]

Hi,

In reference to dm-cache: can the same cache and metadata devices be used with multiple origin devices?  This can be configured, and we've done some tests that appear to show that it works - we're looking for confirmation (or otherwise).

Here's an example test setup to clarify -- ssd_metadata and ssd_blocks are being used to cache sdc and sdd.  In testing, different patterns were written to areas of sdc_cached and  sdd_cached; afterwards, the contents of sdc and sdd were as expected.

dmsetup create sdc_cached --table '0 4194304 cache /dev/mapper/ssd_metadata /dev/mapper/ssd_blocks /dev/sdc 512 1 writethrough default 0'
dmsetup create sdd_cached --table '0 4194304 cache /dev/mapper/ssd_metadata /dev/mapper/ssd_blocks /dev/sdd 512 1 writethrough default 0'

Our thinking is that using one large cache for multiple origin devices will result in a more efficient use of flash resources than statically partitioning the flash amongst the origins.  Also, there's no need to repartition or buy another SSD to start caching a newly-added origin.

Thanks,
Morgan Mears

Re: dm-cache: can the same cache be used with multiple origin devices?

[Mike Snitzer]

On Mon, Jul 15 2013 at  3:01pm -0400,
Mears, Morgan <Morgan.Mears@netapp.com> wrote:

> Hi,
> 
> In reference to dm-cache: can the same cache and metadata devices be
> used with multiple origin devices?  This can be configured, and we've
> done some tests that appear to show that it works - we're looking for
> confirmation (or otherwise).

It is _not_ supported.
 
> Here's an example test setup to clarify -- ssd_metadata and ssd_blocks
> are being used to cache sdc and sdd.  In testing, different patterns
> were written to areas of sdc_cached and  sdd_cached; afterwards, the
> contents of sdc and sdd were as expected.
> 
> dmsetup create sdc_cached --table '0 4194304 cache /dev/mapper/ssd_metadata /dev/mapper/ssd_blocks /dev/sdc 512 1 writethrough default 0'
> dmsetup create sdd_cached --table '0 4194304 cache /dev/mapper/ssd_metadata /dev/mapper/ssd_blocks /dev/sdd 512 1 writethrough default 0'

Interesting.

The current cache target obviously fails to detect that the metadata or
data devices are already in use.  But that doesn't mean it is safe to
utilize the cache in this mode (I'll have a think about where the code
will break down).  But the cache is managed/written in a manner that
only assumes a single backing origin for each cache.

> Our thinking is that using one large cache for multiple origin devices
> will result in a more efficient use of flash resources than statically
> partitioning the flash amongst the origins.  Also, there's no need to
> repartition or buy another SSD to start caching a newly-added origin.

We originally thought about allowing a shared cache (metadata+data) for
N origin devices but decided against it to reduce the complexity of the
initial target code.

Re: dm-cache: can the same cache be used with multiple origin devices?

[Mike Snitzer]

On Mon, Jul 15 2013 at  3:59pm -0400,
Mike Snitzer <snitzer@redhat.com> wrote:

> On Mon, Jul 15 2013 at  3:01pm -0400,
> Mears, Morgan <Morgan.Mears@netapp.com> wrote:
> 
> > Hi,
> > 
> > In reference to dm-cache: can the same cache and metadata devices be
> > used with multiple origin devices?  This can be configured, and we've
> > done some tests that appear to show that it works - we're looking for
> > confirmation (or otherwise).
> 
> It is _not_ supported.
>  
> > Here's an example test setup to clarify -- ssd_metadata and ssd_blocks
> > are being used to cache sdc and sdd.  In testing, different patterns
> > were written to areas of sdc_cached and  sdd_cached; afterwards, the
> > contents of sdc and sdd were as expected.
> > 
> > dmsetup create sdc_cached --table '0 4194304 cache /dev/mapper/ssd_metadata /dev/mapper/ssd_blocks /dev/sdc 512 1 writethrough default 0'
> > dmsetup create sdd_cached --table '0 4194304 cache /dev/mapper/ssd_metadata /dev/mapper/ssd_blocks /dev/sdd 512 1 writethrough default 0'
> 
> Interesting.
> 
> The current cache target obviously fails to detect that the metadata or
> data devices are already in use.  But that doesn't mean it is safe to
> utilize the cache in this mode (I'll have a think about where the code
> will break down).  But the cache is managed/written in a manner that
> only assumes a single backing origin for each cache.

The dm-cache code is inherently unsafe to use in the above configuration
because it results in disjoint tasks accessing the same metadata
device.  This is inherently racey because cmd->root_lock (metadata) and
cache->lock (cache device) are completely independent.

So things like tearing down one cache device (sdc_cached) while
issuing discards to the other (sdd_cached) would be one example of
competing tasks stepping all over one another.  More fundamental actions
like allocating a new cache block is racey.  Etc.

There are many other potential problems -- you apparently have just been
"lucky" not to hit them yet.  These races would likely become much more
transparent if you created 10 sdX_cached devices that share the same
metadata device and you then data integrity sensitive workloads against
each sdX_cached device (like repeat linux.git checkouts or even netapp's
dt).

Mike