From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mike Snitzer Subject: Re: dm-cache: can the same cache be used with multiple origin devices? Date: Mon, 22 Jul 2013 10:35:37 -0400 Message-ID: <20130722143537.GA13965@redhat.com> References: <33A0129EBFD46748804DE81B354CA1B2165B1533@SACEXCMBX06-PRD.hq.netapp.com> <20130715195911.GC15172@redhat.com> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <20130715195911.GC15172@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: "Mears, Morgan" Cc: "dm-devel@redhat.com" List-Id: dm-devel.ids On Mon, Jul 15 2013 at 3:59pm -0400, Mike Snitzer wrote: > On Mon, Jul 15 2013 at 3:01pm -0400, > Mears, Morgan wrote: > > > Hi, > > > > In reference to dm-cache: can the same cache and metadata devices be > > used with multiple origin devices? This can be configured, and we've > > done some tests that appear to show that it works - we're looking for > > confirmation (or otherwise). > > It is _not_ supported. > > > Here's an example test setup to clarify -- ssd_metadata and ssd_blocks > > are being used to cache sdc and sdd. In testing, different patterns > > were written to areas of sdc_cached and sdd_cached; afterwards, the > > contents of sdc and sdd were as expected. > > > > dmsetup create sdc_cached --table '0 4194304 cache /dev/mapper/ssd_metadata /dev/mapper/ssd_blocks /dev/sdc 512 1 writethrough default 0' > > dmsetup create sdd_cached --table '0 4194304 cache /dev/mapper/ssd_metadata /dev/mapper/ssd_blocks /dev/sdd 512 1 writethrough default 0' > > Interesting. > > The current cache target obviously fails to detect that the metadata or > data devices are already in use. But that doesn't mean it is safe to > utilize the cache in this mode (I'll have a think about where the code > will break down). But the cache is managed/written in a manner that > only assumes a single backing origin for each cache. The dm-cache code is inherently unsafe to use in the above configuration because it results in disjoint tasks accessing the same metadata device. This is inherently racey because cmd->root_lock (metadata) and cache->lock (cache device) are completely independent. So things like tearing down one cache device (sdc_cached) while issuing discards to the other (sdd_cached) would be one example of competing tasks stepping all over one another. More fundamental actions like allocating a new cache block is racey. Etc. There are many other potential problems -- you apparently have just been "lucky" not to hit them yet. These races would likely become much more transparent if you created 10 sdX_cached devices that share the same metadata device and you then data integrity sensitive workloads against each sdX_cached device (like repeat linux.git checkouts or even netapp's dt). Mike