Re: dm-cache: can the same cache be used with multiple origin devices?

[Mike Snitzer <snitzer@redhat.com>]

On Mon, Jul 15 2013 at  3:59pm -0400,
Mike Snitzer <snitzer@redhat.com> wrote:

> On Mon, Jul 15 2013 at  3:01pm -0400,
> Mears, Morgan <Morgan.Mears@netapp.com> wrote:
> 
> > Hi,
> > 
> > In reference to dm-cache: can the same cache and metadata devices be
> > used with multiple origin devices?  This can be configured, and we've
> > done some tests that appear to show that it works - we're looking for
> > confirmation (or otherwise).
> 
> It is _not_ supported.
>  
> > Here's an example test setup to clarify -- ssd_metadata and ssd_blocks
> > are being used to cache sdc and sdd.  In testing, different patterns
> > were written to areas of sdc_cached and  sdd_cached; afterwards, the
> > contents of sdc and sdd were as expected.
> > 
> > dmsetup create sdc_cached --table '0 4194304 cache /dev/mapper/ssd_metadata /dev/mapper/ssd_blocks /dev/sdc 512 1 writethrough default 0'
> > dmsetup create sdd_cached --table '0 4194304 cache /dev/mapper/ssd_metadata /dev/mapper/ssd_blocks /dev/sdd 512 1 writethrough default 0'
> 
> Interesting.
> 
> The current cache target obviously fails to detect that the metadata or
> data devices are already in use.  But that doesn't mean it is safe to
> utilize the cache in this mode (I'll have a think about where the code
> will break down).  But the cache is managed/written in a manner that
> only assumes a single backing origin for each cache.

The dm-cache code is inherently unsafe to use in the above configuration
because it results in disjoint tasks accessing the same metadata
device.  This is inherently racey because cmd->root_lock (metadata) and
cache->lock (cache device) are completely independent.

So things like tearing down one cache device (sdc_cached) while
issuing discards to the other (sdd_cached) would be one example of
competing tasks stepping all over one another.  More fundamental actions
like allocating a new cache block is racey.  Etc.

There are many other potential problems -- you apparently have just been
"lucky" not to hit them yet.  These races would likely become much more
transparent if you created 10 sdX_cached devices that share the same
metadata device and you then data integrity sensitive workloads against
each sdX_cached device (like repeat linux.git checkouts or even netapp's
dt).

Mike