dm-cache race on nr_dirty in set_dirty/clear_dirty?

dm-cache race on nr_dirty in set_dirty/clear_dirty?

[Anssi Hannula]

Hi,

I'm seeing the following "dmsetup status" on one volume:
> 0 5368709120 cache 8 3926/32768 128 335265/1978880 5589425 8052258 2254781 3910141 0 335265 4294967293 1 writeback 2 migration_threshold 2048 mq 10 random_threshold 4 sequential_threshold 512 discard_promote_adjustment 1 read_promote_adjustment 4 write_promote_adjustment 8

Note the clearly wrong 4294967293 in the nr_dirty field.

Looking at the code, I see nr_dirty is set in the following functions in
dm-cache-target.c:

> static void set_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cblock)
> {
>         if (!test_and_set_bit(from_cblock(cblock), cache->dirty_bitset)) {
>                 cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) + 1);
>                 policy_set_dirty(cache->policy, oblock);
>         }
> }
> 
> static void clear_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cblock)
> {
>         if (test_and_clear_bit(from_cblock(cblock), cache->dirty_bitset)) {
>                 policy_clear_dirty(cache->policy, oblock);
>                 cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) - 1);
>                 if (!from_cblock(cache->nr_dirty))
>                         dm_table_event(cache->ti->table);
>         }
> }

That looks like a race to me? As nothing is protecting cache->nr_dirty
from multiple access (unlike cache->dirty_bitset).

Unless I'm missing something, as I'm not familiar with this code...

-- 
Anssi Hannula

[PATCH] dm cache: fix race affecting dirty block count

[Anssi Hannula]

nr_dirty is updated without locking, causing it to drift so that it is
non-zero (either a small positive integer, or a very large one when an
underflow occurs) even when there are no actual dirty blocks.

Fix that by using an atomic type for nr_dirty.

Signed-off-by: Anssi Hannula <anssi.hannula@iki.fi>
Cc: Joe Thornber <ejt@redhat.com>
Cc: stable@vger.kernel.org

---

So here's a patch for the issue I reported a few days ago.

I'm not sure what the actual effects of having wrong nr_dirty are
beyond the wrong value reported to userspace. Having wrong nr_dirty
causes dm_table_event(cache->ti->table) to be called at incorrect times,
but I'm not sure what the implications of that are.

So someone more knowledged should really amend my commit message
accordingly so that it explains the user-visible issues :)

I did notice that after a reboot (after having incorrect nr_dirty) the
entire cache was considered to be dirty and a full writeback occurred,
but I don't know if that was related at all.


 drivers/md/dm-cache-target.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
index 0df3ec085ebb..83a7eb5ef113 100644
--- a/drivers/md/dm-cache-target.c
+++ b/drivers/md/dm-cache-target.c
@@ -154,7 +154,7 @@ struct cache {
 	/*
 	 * cache_size entries, dirty if set
 	 */
-	dm_cblock_t nr_dirty;
+	atomic_t nr_dirty;
 	unsigned long *dirty_bitset;
 
 	/*
@@ -403,7 +403,7 @@ static bool is_dirty(struct cache *cache, dm_cblock_t b)
 static void set_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cblock)
 {
 	if (!test_and_set_bit(from_cblock(cblock), cache->dirty_bitset)) {
-		cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) + 1);
+		atomic_inc(&cache->nr_dirty);
 		policy_set_dirty(cache->policy, oblock);
 	}
 }
@@ -412,8 +412,7 @@ static void clear_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cbl
 {
 	if (test_and_clear_bit(from_cblock(cblock), cache->dirty_bitset)) {
 		policy_clear_dirty(cache->policy, oblock);
-		cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) - 1);
-		if (!from_cblock(cache->nr_dirty))
+		if (atomic_dec_return(&cache->nr_dirty) == 0)
 			dm_table_event(cache->ti->table);
 	}
 }
@@ -2003,7 +2002,7 @@ static int cache_create(struct cache_args *ca, struct cache **result)
 	init_waitqueue_head(&cache->migration_wait);
 
 	r = -ENOMEM;
-	cache->nr_dirty = 0;
+	atomic_set(&cache->nr_dirty, 0);
 	cache->dirty_bitset = alloc_bitset(from_cblock(cache->cache_size));
 	if (!cache->dirty_bitset) {
 		*error = "could not allocate dirty bitset";
@@ -2513,7 +2512,7 @@ static void cache_status(struct dm_target *ti, status_type_t type,
 		       (unsigned) atomic_read(&cache->stats.demotion),
 		       (unsigned) atomic_read(&cache->stats.promotion),
 		       (unsigned long long) from_cblock(residency),
-		       cache->nr_dirty);
+		       (unsigned) atomic_read(&cache->nr_dirty));
 
 		if (cache->features.write_through)
 			DMEMIT("1 writethrough ");
-- 
1.8.4.5

Re: [PATCH] dm cache: fix race affecting dirty block count

[Joe Thornber]

On Thu, Jul 31, 2014 at 09:31:27PM +0300, Anssi Hannula wrote:
> I did notice that after a reboot (after having incorrect nr_dirty) the
> entire cache was considered to be dirty and a full writeback occurred,
> but I don't know if that was related at all.

Did you take the cache down cleanly?  Normally this happens when you
shutdown without destroying the cache device first.

Patch looks good.

- Joe

Re: [PATCH] dm cache: fix race affecting dirty block count

[Anssi Hannula]

01.08.2014, 18:17, Joe Thornber kirjoitti:
> On Thu, Jul 31, 2014 at 09:31:27PM +0300, Anssi Hannula wrote:
>> I did notice that after a reboot (after having incorrect nr_dirty) the
>> entire cache was considered to be dirty and a full writeback occurred,
>> but I don't know if that was related at all.
> 
> Did you take the cache down cleanly?  Normally this happens when you
> shutdown without destroying the cache device first.

Apparently not. Looking at the logs systemd-cryptsetup failed to
deactivate the dm-crypt volume on top of the LV, as it was still mounted
as it seems systemd (v208) tried to unmount them way too early, while
they were still in use. Or maybe it should have retried later but didn't
for some reason... will have to debug further.

In any case, not a dm-cache issue it seems.

> Patch looks good.
> 
> - Joe
> 


-- 
Anssi Hannula

Re: [PATCH] dm cache: fix race affecting dirty block count

[Anssi Hannula]

01.08.2014, 18:17, Joe Thornber kirjoitti:
> On Thu, Jul 31, 2014 at 09:31:27PM +0300, Anssi Hannula wrote:
>> nr_dirty is updated without locking, causing it to drift so that it is
>> non-zero (either a small positive integer, or a very large one when an
>> underflow occurs) even when there are no actual dirty blocks.
>> 
>> Fix that by using an atomic type for nr_dirty.
[...]
> Patch looks good.

Unfortunately it seems there is some other potentially more serious bug
still in there...

I'm again seeing spurious dirty counts on two volumes [1]. As expected
with the previous fix, they are now both positive (as nr_dirty should
now be perfectly in sync with dirty_bitset and the bitset can't have a
negative count).


My first suspect was this kind of scenario:

Thread 1                      Thread 2
enter clear_dirty(61)         .
clear dirty bit               .
.                             enter set_dirty(61)
.                             set dirty bit
.                             policy_set_dirty()
.                             => mq adds to dirty queue
policy_clear_dirty()          .
=> mq drops from dirty queue  .


So the block remains dirty on target side but clean on mq side, and it
never gets written back since it is not on the mq dirty queue.

However, if the above is really what happens, i.e. concurrent calls to
set_dirty()/clear_dirty() for the same block, is that something that is
supposed to happen? and is the block then really dirty or not?

Or maybe something else is going on...?

This is on 3.15.7 + previous patch.

[1]

0 101876654080 cache 8 30512/32768 128 936412/1978944 8028417 56720875
10942325 11368177 0 744175 13 1 writeback 2 migration_threshold 2048 mq
10 random_threshold 4 sequential_threshold 512
discard_promote_adjustment 1 read_promote_adjustment 4
write_promote_adjustment 8

0 5368709120 cache 8 7184/32768 128 366217/1978880 39346555 8471758
13307043 4709237 0 7055 1 1 writeback 2 migration_threshold 2048 mq 10
random_threshold 4 sequential_threshold 512 discard_promote_adjustment 1
read_promote_adjustment 4 write_promote_adjustment 8

-- 
Anssi Hannula

[PATCH] dm cache: fix race causing dirty blocks to be marked as clean

[Anssi Hannula]

When a writeback or a promotion of a block is completed, the cell of
that block is removed from the prison, the block is marked as clean, and
the clear_dirty() callback of the cache policy is called.

Unfortunately, performing those actions in this order allows an incoming
new write bio for that block to come in before clearing the dirty status
is completed and therefore possibly causing one of these two scenarios:

Scenario A:

Thread 1                      Thread 2
cell_defer()                  .
- cell removed from prison    .
- detained bios queued        .
.                             incoming write bio
.                             remapped to cache
.                             set_dirty() called,
.                               but block already dirty
.                               => it does nothing
clear_dirty()                 .
- block marked clean          .
- policy clear_dirty() called .

Result: Block is marked clean even though it is actually dirty. No
writeback will occur.

Scenario B:

Thread 1                      Thread 2
cell_defer()                  .
- cell removed from prison    .
- detained bios queued        .
clear_dirty()                 .
- block marked clean          .
.                             incoming write bio
.                             remapped to cache
.                             set_dirty() called
.                             - block marked dirty
.                             - policy set_dirty() called
- policy clear_dirty() called .

Result: Block is properly marked as dirty, but policy thinks it is clean
and therefore never asks us to writeback it.
This case is visible in "dmsetup status" dirty block count (which
normally decreases to 0 on a quiet device).

Fix these issues by calling clear_dirty() before calling cell_defer().
Incoming bios for that block will then be detained in the cell and
released only after clear_dirty() has completed, so the race will not
occur.

Found by inspecting the code after noticing spurious dirty counts
(scenario B).

Signed-off-by: Anssi Hannula <anssi.hannula@iki.fi>
Cc: Joe Thornber <ejt@redhat.com>
Cc: stable@vger.kernel.org
---

> Unfortunately it seems there is some other potentially more serious bug
> still in there...

After looking through the code that indeed seems to be the case, as
explained above.

Unless I'm missing something?

I can't say with 100% certainty if this fixes the spurious counts I saw
since those took quite a long time (1-2 weeks?) to appear and the load
of that system is somewhat irregular.


 drivers/md/dm-cache-target.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
index 1af40ee209e2..7130505c2425 100644
--- a/drivers/md/dm-cache-target.c
+++ b/drivers/md/dm-cache-target.c
@@ -895,8 +895,8 @@ static void migration_success_pre_commit(struct dm_cache_migration *mg)
 	struct cache *cache = mg->cache;
 
 	if (mg->writeback) {
-		cell_defer(cache, mg->old_ocell, false);
 		clear_dirty(cache, mg->old_oblock, mg->cblock);
+		cell_defer(cache, mg->old_ocell, false);
 		cleanup_migration(mg);
 		return;
 
@@ -951,13 +951,13 @@ static void migration_success_post_commit(struct dm_cache_migration *mg)
 		}
 
 	} else {
+		clear_dirty(cache, mg->new_oblock, mg->cblock);
 		if (mg->requeue_holder)
 			cell_defer(cache, mg->new_ocell, true);
 		else {
 			bio_endio(mg->new_ocell->holder, 0);
 			cell_defer(cache, mg->new_ocell, false);
 		}
-		clear_dirty(cache, mg->new_oblock, mg->cblock);
 		cleanup_migration(mg);
 	}
 }
-- 
1.8.4.5

Re: [PATCH] dm cache: fix race causing dirty blocks to be marked as clean

[Joe Thornber]

Ack.  Thanks.

On Fri, Sep 05, 2014 at 03:11:28AM +0300, Anssi Hannula wrote:
> When a writeback or a promotion of a block is completed, the cell of
> that block is removed from the prison, the block is marked as clean, and
> the clear_dirty() callback of the cache policy is called.
> 
> Unfortunately, performing those actions in this order allows an incoming
> new write bio for that block to come in before clearing the dirty status
> is completed and therefore possibly causing one of these two scenarios:
> 
> Scenario A:
> 
> Thread 1                      Thread 2
> cell_defer()                  .
> - cell removed from prison    .
> - detained bios queued        .
> .                             incoming write bio
> .                             remapped to cache
> .                             set_dirty() called,
> .                               but block already dirty
> .                               => it does nothing
> clear_dirty()                 .
> - block marked clean          .
> - policy clear_dirty() called .
> 
> Result: Block is marked clean even though it is actually dirty. No
> writeback will occur.
> 
> Scenario B:
> 
> Thread 1                      Thread 2
> cell_defer()                  .
> - cell removed from prison    .
> - detained bios queued        .
> clear_dirty()                 .
> - block marked clean          .
> .                             incoming write bio
> .                             remapped to cache
> .                             set_dirty() called
> .                             - block marked dirty
> .                             - policy set_dirty() called
> - policy clear_dirty() called .
> 
> Result: Block is properly marked as dirty, but policy thinks it is clean
> and therefore never asks us to writeback it.
> This case is visible in "dmsetup status" dirty block count (which
> normally decreases to 0 on a quiet device).
> 
> Fix these issues by calling clear_dirty() before calling cell_defer().
> Incoming bios for that block will then be detained in the cell and
> released only after clear_dirty() has completed, so the race will not
> occur.
> 
> Found by inspecting the code after noticing spurious dirty counts
> (scenario B).
> 
> Signed-off-by: Anssi Hannula <anssi.hannula@iki.fi>
> Cc: Joe Thornber <ejt@redhat.com>
> Cc: stable@vger.kernel.org
> ---
> 
> > Unfortunately it seems there is some other potentially more serious bug
> > still in there...
> 
> After looking through the code that indeed seems to be the case, as
> explained above.
> 
> Unless I'm missing something?
> 
> I can't say with 100% certainty if this fixes the spurious counts I saw
> since those took quite a long time (1-2 weeks?) to appear and the load
> of that system is somewhat irregular.
> 
> 
>  drivers/md/dm-cache-target.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
> index 1af40ee209e2..7130505c2425 100644
> --- a/drivers/md/dm-cache-target.c
> +++ b/drivers/md/dm-cache-target.c
> @@ -895,8 +895,8 @@ static void migration_success_pre_commit(struct dm_cache_migration *mg)
>  	struct cache *cache = mg->cache;
>  
>  	if (mg->writeback) {
> -		cell_defer(cache, mg->old_ocell, false);
>  		clear_dirty(cache, mg->old_oblock, mg->cblock);
> +		cell_defer(cache, mg->old_ocell, false);
>  		cleanup_migration(mg);
>  		return;
>  
> @@ -951,13 +951,13 @@ static void migration_success_post_commit(struct dm_cache_migration *mg)
>  		}
>  
>  	} else {
> +		clear_dirty(cache, mg->new_oblock, mg->cblock);
>  		if (mg->requeue_holder)
>  			cell_defer(cache, mg->new_ocell, true);
>  		else {
>  			bio_endio(mg->new_ocell->holder, 0);
>  			cell_defer(cache, mg->new_ocell, false);
>  		}
> -		clear_dirty(cache, mg->new_oblock, mg->cblock);
>  		cleanup_migration(mg);
>  	}
>  }
> -- 
> 1.8.4.5
> 

Re: [PATCH] dm cache: fix race affecting dirty block count

[Pranith Kumar]

Hello Anssi, Joe, Mike,

I just found your patch in the latest rc and wanted to ask a few
questions. I am not sure how this patch helps at all other than luck in
that dm_cblock_t type is of type int32_t, which should guarantee that it
is atomic on most platforms. Which begs the question, what platform did
you encounter this problem?

The patch purports to solve a race condition by making use of atomic_t.
I am not sure that is enough. If indeed there is a race you need to use
smp_mb__{before/after}_atomic() for both your uses of atomic_inc() and
atomic_set(). 

Also I have a concern about why this mail was not CC'ed on LKML. I had
to go to some difficulty in finding this patch. So please CC LKML for
such patches.

Thanks,
-- 
Pranith

-- Begin forwarded Message --


nr_dirty is updated without locking, causing it to drift so that it is
non-zero (either a small positive integer, or a very large one when an
underflow occurs) even when there are no actual dirty blocks.

Fix that by using an atomic type for nr_dirty.

Signed-off-by: Anssi Hannula <anssi hannula iki fi>
Cc: Joe Thornber <ejt redhat com>
Cc: stable vger kernel org

---

So here's a patch for the issue I reported a few days ago.

I'm not sure what the actual effects of having wrong nr_dirty are
beyond the wrong value reported to userspace. Having wrong nr_dirty
causes dm_table_event(cache->ti->table) to be called at incorrect times,
but I'm not sure what the implications of that are.

So someone more knowledged should really amend my commit message
accordingly so that it explains the user-visible issues :)

I did notice that after a reboot (after having incorrect nr_dirty) the
entire cache was considered to be dirty and a full writeback occurred,
but I don't know if that was related at all.


 drivers/md/dm-cache-target.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
index 0df3ec085ebb..83a7eb5ef113 100644
--- a/drivers/md/dm-cache-target.c
+++ b/drivers/md/dm-cache-target.c
@@ -154,7 +154,7 @@ struct cache {
 	/*
 	 * cache_size entries, dirty if set
 	 */
-	dm_cblock_t nr_dirty;
+	atomic_t nr_dirty;
 	unsigned long *dirty_bitset;
 
 	/*
@@ -403,7 +403,7 @@ static bool is_dirty(struct cache *cache, dm_cblock_t b)
 static void set_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cblock)
 {
 	if (!test_and_set_bit(from_cblock(cblock), cache->dirty_bitset)) {
-		cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) + 1);
+		atomic_inc(&cache->nr_dirty);
 		policy_set_dirty(cache->policy, oblock);
 	}
 }
@@ -412,8 +412,7 @@ static void clear_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cbl
 {
 	if (test_and_clear_bit(from_cblock(cblock), cache->dirty_bitset)) {
 		policy_clear_dirty(cache->policy, oblock);
-		cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) - 1);
-		if (!from_cblock(cache->nr_dirty))
+		if (atomic_dec_return(&cache->nr_dirty) == 0)
 			dm_table_event(cache->ti->table);
 	}
 }
@@ -2003,7 +2002,7 @@ static int cache_create(struct cache_args *ca, struct cache **result)
 	init_waitqueue_head(&cache->migration_wait);
 
 	r = -ENOMEM;
-	cache->nr_dirty = 0;
+	atomic_set(&cache->nr_dirty, 0);
 	cache->dirty_bitset = alloc_bitset(from_cblock(cache->cache_size));
 	if (!cache->dirty_bitset) {
 		*error = "could not allocate dirty bitset";
@@ -2513,7 +2512,7 @@ static void cache_status(struct dm_target *ti, status_type_t type,
 		       (unsigned) atomic_read(&cache->stats.demotion),
 		       (unsigned) atomic_read(&cache->stats.promotion),
 		       (unsigned long long) from_cblock(residency),
-		       cache->nr_dirty);
+		       (unsigned) atomic_read(&cache->nr_dirty));
 
 		if (cache->features.write_through)
 			DMEMIT("1 writethrough ");
-- 
1.8.4.5

-- End forwarded Message --

Re: [PATCH] dm cache: fix race affecting dirty block count

[Pranith Kumar]

Correctly adding Anssi this time.

On Sat, Aug 2, 2014 at 10:00 PM, Pranith Kumar <bobby.prani@gmail.com> wrote:
> Hello Anssi, Joe, Mike,
>
> I just found your patch in the latest rc and wanted to ask a few
> questions. I am not sure how this patch helps at all other than luck in
> that dm_cblock_t type is of type int32_t, which should guarantee that it
> is atomic on most platforms. Which begs the question, what platform did
> you encounter this problem?
>
> The patch purports to solve a race condition by making use of atomic_t.
> I am not sure that is enough. If indeed there is a race you need to use
> smp_mb__{before/after}_atomic() for both your uses of atomic_inc() and
> atomic_set().
>
> Also I have a concern about why this mail was not CC'ed on LKML. I had
> to go to some difficulty in finding this patch. So please CC LKML for
> such patches.
>
> Thanks,
> --
> Pranith
>
> -- Begin forwarded Message --
>
>
> nr_dirty is updated without locking, causing it to drift so that it is
> non-zero (either a small positive integer, or a very large one when an
> underflow occurs) even when there are no actual dirty blocks.
>
> Fix that by using an atomic type for nr_dirty.
>
> Signed-off-by: Anssi Hannula <anssi hannula iki fi>
> Cc: Joe Thornber <ejt redhat com>
> Cc: stable vger kernel org
>
> ---
>
> So here's a patch for the issue I reported a few days ago.
>
> I'm not sure what the actual effects of having wrong nr_dirty are
> beyond the wrong value reported to userspace. Having wrong nr_dirty
> causes dm_table_event(cache->ti->table) to be called at incorrect times,
> but I'm not sure what the implications of that are.
>
> So someone more knowledged should really amend my commit message
> accordingly so that it explains the user-visible issues :)
>
> I did notice that after a reboot (after having incorrect nr_dirty) the
> entire cache was considered to be dirty and a full writeback occurred,
> but I don't know if that was related at all.
>
>
>  drivers/md/dm-cache-target.c | 11 +++++------
>  1 file changed, 5 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
> index 0df3ec085ebb..83a7eb5ef113 100644
> --- a/drivers/md/dm-cache-target.c
> +++ b/drivers/md/dm-cache-target.c
> @@ -154,7 +154,7 @@ struct cache {
>         /*
>          * cache_size entries, dirty if set
>          */
> -       dm_cblock_t nr_dirty;
> +       atomic_t nr_dirty;
>         unsigned long *dirty_bitset;
>
>         /*
> @@ -403,7 +403,7 @@ static bool is_dirty(struct cache *cache, dm_cblock_t b)
>  static void set_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cblock)
>  {
>         if (!test_and_set_bit(from_cblock(cblock), cache->dirty_bitset)) {
> -               cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) + 1);
> +               atomic_inc(&cache->nr_dirty);
>                 policy_set_dirty(cache->policy, oblock);
>         }
>  }
> @@ -412,8 +412,7 @@ static void clear_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cbl
>  {
>         if (test_and_clear_bit(from_cblock(cblock), cache->dirty_bitset)) {
>                 policy_clear_dirty(cache->policy, oblock);
> -               cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) - 1);
> -               if (!from_cblock(cache->nr_dirty))
> +               if (atomic_dec_return(&cache->nr_dirty) == 0)
>                         dm_table_event(cache->ti->table);
>         }
>  }
> @@ -2003,7 +2002,7 @@ static int cache_create(struct cache_args *ca, struct cache **result)
>         init_waitqueue_head(&cache->migration_wait);
>
>         r = -ENOMEM;
> -       cache->nr_dirty = 0;
> +       atomic_set(&cache->nr_dirty, 0);
>         cache->dirty_bitset = alloc_bitset(from_cblock(cache->cache_size));
>         if (!cache->dirty_bitset) {
>                 *error = "could not allocate dirty bitset";
> @@ -2513,7 +2512,7 @@ static void cache_status(struct dm_target *ti, status_type_t type,
>                        (unsigned) atomic_read(&cache->stats.demotion),
>                        (unsigned) atomic_read(&cache->stats.promotion),
>                        (unsigned long long) from_cblock(residency),
> -                      cache->nr_dirty);
> +                      (unsigned) atomic_read(&cache->nr_dirty));
>
>                 if (cache->features.write_through)
>                         DMEMIT("1 writethrough ");
> --
> 1.8.4.5
>
> -- End forwarded Message --
>



-- 
Pranith

Re: [PATCH] dm cache: fix race affecting dirty block count

[Pranith Kumar]

Corrently adding Anssi this time.

On Sat, Aug 2, 2014 at 10:00 PM, Pranith Kumar <bobby.prani@gmail.com> wrote:
> Hello Anssi, Joe, Mike,
>
> I just found your patch in the latest rc and wanted to ask a few
> questions. I am not sure how this patch helps at all other than luck in
> that dm_cblock_t type is of type int32_t, which should guarantee that it
> is atomic on most platforms. Which begs the question, what platform did
> you encounter this problem?
>
> The patch purports to solve a race condition by making use of atomic_t.
> I am not sure that is enough. If indeed there is a race you need to use
> smp_mb__{before/after}_atomic() for both your uses of atomic_inc() and
> atomic_set().
>
> Also I have a concern about why this mail was not CC'ed on LKML. I had
> to go to some difficulty in finding this patch. So please CC LKML for
> such patches.
>
> Thanks,
> --
> Pranith
>
> -- Begin forwarded Message --
>
>
> nr_dirty is updated without locking, causing it to drift so that it is
> non-zero (either a small positive integer, or a very large one when an
> underflow occurs) even when there are no actual dirty blocks.
>
> Fix that by using an atomic type for nr_dirty.
>
> Signed-off-by: Anssi Hannula <anssi hannula iki fi>
> Cc: Joe Thornber <ejt redhat com>
> Cc: stable vger kernel org
>
> ---
>
> So here's a patch for the issue I reported a few days ago.
>
> I'm not sure what the actual effects of having wrong nr_dirty are
> beyond the wrong value reported to userspace. Having wrong nr_dirty
> causes dm_table_event(cache->ti->table) to be called at incorrect times,
> but I'm not sure what the implications of that are.
>
> So someone more knowledged should really amend my commit message
> accordingly so that it explains the user-visible issues :)
>
> I did notice that after a reboot (after having incorrect nr_dirty) the
> entire cache was considered to be dirty and a full writeback occurred,
> but I don't know if that was related at all.
>
>
>  drivers/md/dm-cache-target.c | 11 +++++------
>  1 file changed, 5 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
> index 0df3ec085ebb..83a7eb5ef113 100644
> --- a/drivers/md/dm-cache-target.c
> +++ b/drivers/md/dm-cache-target.c
> @@ -154,7 +154,7 @@ struct cache {
>         /*
>          * cache_size entries, dirty if set
>          */
> -       dm_cblock_t nr_dirty;
> +       atomic_t nr_dirty;
>         unsigned long *dirty_bitset;
>
>         /*
> @@ -403,7 +403,7 @@ static bool is_dirty(struct cache *cache, dm_cblock_t b)
>  static void set_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cblock)
>  {
>         if (!test_and_set_bit(from_cblock(cblock), cache->dirty_bitset)) {
> -               cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) + 1);
> +               atomic_inc(&cache->nr_dirty);
>                 policy_set_dirty(cache->policy, oblock);
>         }
>  }
> @@ -412,8 +412,7 @@ static void clear_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cbl
>  {
>         if (test_and_clear_bit(from_cblock(cblock), cache->dirty_bitset)) {
>                 policy_clear_dirty(cache->policy, oblock);
> -               cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) - 1);
> -               if (!from_cblock(cache->nr_dirty))
> +               if (atomic_dec_return(&cache->nr_dirty) == 0)
>                         dm_table_event(cache->ti->table);
>         }
>  }
> @@ -2003,7 +2002,7 @@ static int cache_create(struct cache_args *ca, struct cache **result)
>         init_waitqueue_head(&cache->migration_wait);
>
>         r = -ENOMEM;
> -       cache->nr_dirty = 0;
> +       atomic_set(&cache->nr_dirty, 0);
>         cache->dirty_bitset = alloc_bitset(from_cblock(cache->cache_size));
>         if (!cache->dirty_bitset) {
>                 *error = "could not allocate dirty bitset";
> @@ -2513,7 +2512,7 @@ static void cache_status(struct dm_target *ti, status_type_t type,
>                        (unsigned) atomic_read(&cache->stats.demotion),
>                        (unsigned) atomic_read(&cache->stats.promotion),
>                        (unsigned long long) from_cblock(residency),
> -                      cache->nr_dirty);
> +                      (unsigned) atomic_read(&cache->nr_dirty));
>
>                 if (cache->features.write_through)
>                         DMEMIT("1 writethrough ");
> --
> 1.8.4.5
>
> -- End forwarded Message --
>



-- 
Pranith

Re: [PATCH] dm cache: fix race affecting dirty block count

[Anssi Hannula]

03.08.2014, 05:10, Pranith Kumar kirjoitti:
> Corrently adding Anssi this time.
> 
> On Sat, Aug 2, 2014 at 10:00 PM, Pranith Kumar <bobby.prani@gmail.com> wrote:
>> Hello Anssi, Joe, Mike,

Hi,

>> I just found your patch in the latest rc and wanted to ask a few
>> questions. I am not sure how this patch helps at all other than luck in
>> that dm_cblock_t type is of type int32_t, which should guarantee that it
>> is atomic on most platforms. Which begs the question, what platform did
>> you encounter this problem?

On x86_64. Regular integer increment/decrement/addition/subtraction are
not atomic on x86(_64).

Assembly for increment without patch is
	addl   $0x1,0x108(%rdi)
and for decrement
	subl   $0x1,0x108(%rbx)

while with patch:
	lock incl 0x108(%rdi)
and
	mov    $0xffffffff,%eax
	lock xadd %eax,0x108(%rbx)
.


>> The patch purports to solve a race condition by making use of atomic_t.
>> I am not sure that is enough. If indeed there is a race you need to use
>> smp_mb__{before/after}_atomic() for both your uses of atomic_inc() and
>> atomic_set().

The issue was only about concurrent increments/decrements getting lost
completely from time to time, which atomic_inc/dec will prevent without
any explicit barriers.

>> Also I have a concern about why this mail was not CC'ed on LKML. I had
>> to go to some difficulty in finding this patch. So please CC LKML for
>> such patches.

I don't usually CC LKML if there is a subsystem-specific mailing list
unless there is a specific reason to CC LKML as well. I thought this was
standard practice, but I'm happy to spam LKML as well from now on if
that is preferred by others as well.


>> Thanks,
>> --
>> Pranith
>>
>> -- Begin forwarded Message --
>>
>>
>> nr_dirty is updated without locking, causing it to drift so that it is
>> non-zero (either a small positive integer, or a very large one when an
>> underflow occurs) even when there are no actual dirty blocks.
>>
>> Fix that by using an atomic type for nr_dirty.
>>
>> Signed-off-by: Anssi Hannula <anssi hannula iki fi>
>> Cc: Joe Thornber <ejt redhat com>
>> Cc: stable vger kernel org
>>
>> ---
>>
>> So here's a patch for the issue I reported a few days ago.
>>
>> I'm not sure what the actual effects of having wrong nr_dirty are
>> beyond the wrong value reported to userspace. Having wrong nr_dirty
>> causes dm_table_event(cache->ti->table) to be called at incorrect times,
>> but I'm not sure what the implications of that are.
>>
>> So someone more knowledged should really amend my commit message
>> accordingly so that it explains the user-visible issues :)
>>
>> I did notice that after a reboot (after having incorrect nr_dirty) the
>> entire cache was considered to be dirty and a full writeback occurred,
>> but I don't know if that was related at all.
>>
>>
>>  drivers/md/dm-cache-target.c | 11 +++++------
>>  1 file changed, 5 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
>> index 0df3ec085ebb..83a7eb5ef113 100644
>> --- a/drivers/md/dm-cache-target.c
>> +++ b/drivers/md/dm-cache-target.c
>> @@ -154,7 +154,7 @@ struct cache {
>>         /*
>>          * cache_size entries, dirty if set
>>          */
>> -       dm_cblock_t nr_dirty;
>> +       atomic_t nr_dirty;
>>         unsigned long *dirty_bitset;
>>
>>         /*
>> @@ -403,7 +403,7 @@ static bool is_dirty(struct cache *cache, dm_cblock_t b)
>>  static void set_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cblock)
>>  {
>>         if (!test_and_set_bit(from_cblock(cblock), cache->dirty_bitset)) {
>> -               cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) + 1);
>> +               atomic_inc(&cache->nr_dirty);
>>                 policy_set_dirty(cache->policy, oblock);
>>         }
>>  }
>> @@ -412,8 +412,7 @@ static void clear_dirty(struct cache *cache, dm_oblock_t oblock, dm_cblock_t cbl
>>  {
>>         if (test_and_clear_bit(from_cblock(cblock), cache->dirty_bitset)) {
>>                 policy_clear_dirty(cache->policy, oblock);
>> -               cache->nr_dirty = to_cblock(from_cblock(cache->nr_dirty) - 1);
>> -               if (!from_cblock(cache->nr_dirty))
>> +               if (atomic_dec_return(&cache->nr_dirty) == 0)
>>                         dm_table_event(cache->ti->table);
>>         }
>>  }
>> @@ -2003,7 +2002,7 @@ static int cache_create(struct cache_args *ca, struct cache **result)
>>         init_waitqueue_head(&cache->migration_wait);
>>
>>         r = -ENOMEM;
>> -       cache->nr_dirty = 0;
>> +       atomic_set(&cache->nr_dirty, 0);
>>         cache->dirty_bitset = alloc_bitset(from_cblock(cache->cache_size));
>>         if (!cache->dirty_bitset) {
>>                 *error = "could not allocate dirty bitset";
>> @@ -2513,7 +2512,7 @@ static void cache_status(struct dm_target *ti, status_type_t type,
>>                        (unsigned) atomic_read(&cache->stats.demotion),
>>                        (unsigned) atomic_read(&cache->stats.promotion),
>>                        (unsigned long long) from_cblock(residency),
>> -                      cache->nr_dirty);
>> +                      (unsigned) atomic_read(&cache->nr_dirty));
>>
>>                 if (cache->features.write_through)
>>                         DMEMIT("1 writethrough ");
>> --
>> 1.8.4.5
>>
>> -- End forwarded Message --
>>
> 
> 
> 


-- 
Anssi Hannula

Re: [PATCH] dm cache: fix race affecting dirty block count

[Pranith Kumar]

On Sat, Aug 2, 2014 at 11:33 PM, Anssi Hannula <anssi.hannula@iki.fi> wrote:
> 03.08.2014, 05:10, Pranith Kumar kirjoitti:
>> Corrently adding Anssi this time.
>>
>> On Sat, Aug 2, 2014 at 10:00 PM, Pranith Kumar <bobby.prani@gmail.com> wrote:
>>> Hello Anssi, Joe, Mike,
>
> Hi,
>
>>> I just found your patch in the latest rc and wanted to ask a few
>>> questions. I am not sure how this patch helps at all other than luck in
>>> that dm_cblock_t type is of type int32_t, which should guarantee that it
>>> is atomic on most platforms. Which begs the question, what platform did
>>> you encounter this problem?
>
> On x86_64. Regular integer increment/decrement/addition/subtraction are
> not atomic on x86(_64).
>

You are right. Only loads/stores are atomic for an integer. My point
was more about barriers after atomic_inc/dec.

Also dm_cblock_t is uint32_t, but atomic_t changes that to int. You
should correct that to atomic64_t to preserve original semantics.

>>> The patch purports to solve a race condition by making use of atomic_t.
>>> I am not sure that is enough. If indeed there is a race you need to use
>>> smp_mb__{before/after}_atomic() for both your uses of atomic_inc() and
>>> atomic_set().
>
> The issue was only about concurrent increments/decrements getting lost
> completely from time to time, which atomic_inc/dec will prevent without
> any explicit barriers.
>

These increments and decrements will still be lost if you do not use
barriers in presence of concurrent accesses. Please see
Documentation/memory-barriers.txt.

>>> Also I have a concern about why this mail was not CC'ed on LKML. I had
>>> to go to some difficulty in finding this patch. So please CC LKML for
>>> such patches.
>
> I don't usually CC LKML if there is a subsystem-specific mailing list
> unless there is a specific reason to CC LKML as well. I thought this was
> standard practice, but I'm happy to spam LKML as well from now on if
> that is preferred by others as well.
>

I guess it is always a good idea to CC LKML in case of patches in
addition to sub-system mailing lists. For one, it is easy to forward a
message from lkml and then reply to that in case there is something to
discuss.


-- 
Pranith

Re: [PATCH] dm cache: fix race affecting dirty block count

[Joe Thornber]

On Sun, Aug 03, 2014 at 12:01:17AM -0400, Pranith Kumar wrote:
> Also dm_cblock_t is uint32_t, but atomic_t changes that to int. You
> should correct that to atomic64_t to preserve original semantics.

atomic_t used to have only 24 bits of range due to the Sparc
implementation holding a lock in one of the bytes.  I understand this
limitation was removed during 2.6 and the full 32 bits are now
available.

eg,

    https://github.com/jthornber/linux-2.6/commit/37682177af68478fa83429b735fa16913c2fbb2b

> These increments and decrements will still be lost if you do not use
> barriers in presence of concurrent accesses. Please see
> Documentation/memory-barriers.txt.

You do not need to use barriers for plain atomic_inc/dec().

    https://github.com/jthornber/linux-2.6/blob/thin-dev/Documentation/atomic_ops.txt#L187

You _do_ need to use a memory barrier for the ops that return a value
(such as atomic_dec_and_test()), But only if there's some other state
that needs synchronising.  See the nice example in atomic_ops.txt:

    https://github.com/jthornber/linux-2.6/blob/thin-dev/Documentation/atomic_ops.txt#L321

We just trigger a stateless event when the counter hits zero, so the
patch is fine.

- Joe

Re: [PATCH] dm cache: fix race affecting dirty block count

[Pranith Kumar]

On Mon, Aug 4, 2014 at 6:48 AM, Joe Thornber <thornber@redhat.com> wrote:
> On Sun, Aug 03, 2014 at 12:01:17AM -0400, Pranith Kumar wrote:
>> Also dm_cblock_t is uint32_t, but atomic_t changes that to int. You
>> should correct that to atomic64_t to preserve original semantics.
>
> atomic_t used to have only 24 bits of range due to the Sparc
> implementation holding a lock in one of the bytes.  I understand this
> limitation was removed during 2.6 and the full 32 bits are now
> available.
>

I meant to point out that atomic_t is a signed integer (int) type
using the full 32 bits with signed operations. dm_cblock_t is unsgined
int.

>
>> These increments and decrements will still be lost if you do not use
>> barriers in presence of concurrent accesses. Please see
>> Documentation/memory-barriers.txt.
>
> You do not need to use barriers for plain atomic_inc/dec().
>
>     https://github.com/jthornber/linux-2.6/blob/thin-dev/Documentation/atomic_ops.txt#L187

That talks about implementation of atomic_inc/dec() for arch porters.
Users of atomic_inc/dec() should use memory barriers.

>
> You _do_ need to use a memory barrier for the ops that return a value
> (such as atomic_dec_and_test()), But only if there's some other state
> that needs synchronising.  See the nice example in atomic_ops.txt:
>
>     https://github.com/jthornber/linux-2.6/blob/thin-dev/Documentation/atomic_ops.txt#L321

Again when it says it needs explicit memory barriers, it is for the
arch porters. So atomic_dec_and_test(), atomic_dec_return() etc., have
implicit memory barriers.

>
> We just trigger a stateless event when the counter hits zero, so the
> patch is fine.
>

Your use of atomic_dec_return() is what is fixing the race issue here
I guess , as it has implicit memory barriers. But I suggest checking
out if you need barriers for atomic_inc() and atomic_set() too.

-- 
Pranith

Re: [PATCH] dm cache: fix race affecting dirty block count

[Pranith Kumar]

On 08/02/2014 10:10 PM, Pranith Kumar wrote:
> Corrently adding Anssi this time.
> 
> On Sat, Aug 2, 2014 at 10:00 PM, Pranith Kumar <bobby.prani@gmail.com> wrote:
>> Hello Anssi, Joe, Mike,
>>
>> I just found your patch in the latest rc and wanted to ask a few
>> questions. I am not sure how this patch helps at all other than luck in
>> that dm_cblock_t type is of type int32_t, which should guarantee that it
>> is atomic on most platforms. Which begs the question, what platform did
>> you encounter this problem?
>>
>> The patch purports to solve a race condition by making use of atomic_t.
>> I am not sure that is enough. If indeed there is a race you need to use
>> smp_mb__{before/after}_atomic() for both your uses of atomic_inc() and
>> atomic_set().
>>
>> Also I have a concern about why this mail was not CC'ed on LKML. I had
>> to go to some difficulty in finding this patch. So please CC LKML for
>> such patches.
>>
>> Thanks,
>> --
>> Pranith
>>
>> -- Begin forwarded Message --
>>
>>
>> nr_dirty is updated without locking, causing it to drift so that it is
>> non-zero (either a small positive integer, or a very large one when an
>> underflow occurs) even when there are no actual dirty blocks.
>>
>> Fix that by using an atomic type for nr_dirty.
>>
>> Signed-off-by: Anssi Hannula <anssi hannula iki fi>
>> Cc: Joe Thornber <ejt redhat com>
>> Cc: stable vger kernel org

Well, someone got their pointers mixed up. The following should help, but this is not enough if there are indeed concurrent accesses. Are you sure there are concurrent accesses?

diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
index 2c63326..d49ce45 100644
--- a/drivers/md/dm-cache-target.c
+++ b/drivers/md/dm-cache-target.c
@@ -1765,11 +1765,11 @@ static void destroy(struct cache *cache)
        if (cache->wq)
                destroy_workqueue(cache->wq);
 
-       if (cache->dirty_bitset)
-               free_bitset(cache->dirty_bitset);
+       if (cache.dirty_bitset)
+               free_bitset(cache.dirty_bitset);
 
-       if (cache->discard_bitset)
-               free_bitset(cache->discard_bitset);
+       if (cache.discard_bitset)
+               free_bitset(cache.discard_bitset);
 
        if (cache->copier)
                dm_kcopyd_client_destroy(cache->copier);
@@ -2269,15 +2269,15 @@ static int cache_create(struct cache_args *ca, struct cache **result)
 
        r = -ENOMEM;
        atomic_set(&cache->nr_dirty, 0);
-       cache->dirty_bitset = alloc_bitset(from_cblock(cache->cache_size));
-       if (!cache->dirty_bitset) {
+       cache.dirty_bitset = alloc_bitset(from_cblock(cache->cache_size));
+       if (!cache.dirty_bitset) {
                *error = "could not allocate dirty bitset";
                goto bad;
        }
        clear_bitset(cache->dirty_bitset, from_cblock(cache->cache_size));
 
        cache->discard_nr_blocks = cache->origin_blocks;
-       cache->discard_bitset = alloc_bitset(from_oblock(cache->discard_nr_blocks));
+       cache.discard_bitset = alloc_bitset(from_oblock(cache->discard_nr_blocks));
        if (!cache->discard_bitset) {
                *error = "could not allocate discard bitset";
                goto bad;

Re: [PATCH] dm cache: fix race affecting dirty block count

[Pranith Kumar]

On 08/03/2014 12:46 AM, Pranith Kumar wrote:
> On 08/02/2014 10:10 PM, Pranith Kumar wrote:
>> Corrently adding Anssi this time.
>>
>> On Sat, Aug 2, 2014 at 10:00 PM, Pranith Kumar <bobby.prani@gmail.com> wrote:
>>> Hello Anssi, Joe, Mike,
>>>
>>> I just found your patch in the latest rc and wanted to ask a few
>>> questions. I am not sure how this patch helps at all other than luck in
>>> that dm_cblock_t type is of type int32_t, which should guarantee that it
>>> is atomic on most platforms. Which begs the question, what platform did
>>> you encounter this problem?
>>>
>>> The patch purports to solve a race condition by making use of atomic_t.
>>> I am not sure that is enough. If indeed there is a race you need to use
>>> smp_mb__{before/after}_atomic() for both your uses of atomic_inc() and
>>> atomic_set().
>>>
>>> Also I have a concern about why this mail was not CC'ed on LKML. I had
>>> to go to some difficulty in finding this patch. So please CC LKML for
>>> such patches.
>>>
>>> Thanks,
>>> --
>>> Pranith
>>>
>>> -- Begin forwarded Message --
>>>
>>>
>>> nr_dirty is updated without locking, causing it to drift so that it is
>>> non-zero (either a small positive integer, or a very large one when an
>>> underflow occurs) even when there are no actual dirty blocks.
>>>
>>> Fix that by using an atomic type for nr_dirty.
>>>
>>> Signed-off-by: Anssi Hannula <anssi hannula iki fi>
>>> Cc: Joe Thornber <ejt redhat com>
>>> Cc: stable vger kernel org

I found one more location being incorrectly referenced. Please find a fixed
patch below. Also I think we will need to revert the previous patch since it
clearly does not fix anything if there were races.


From: Pranith Kumar <bobby.prani@gmail.com>
Date: Sun, 3 Aug 2014 00:53:20 -0400
Subject: [PATCH 1/1] dm cache: Fix incorrect assignment to pointer

Signed-off-by: Pranith Kumar <bobby.prani@gmail.com>
---
 drivers/md/dm-cache-target.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
index 2c63326..49e47e7 100644
--- a/drivers/md/dm-cache-target.c
+++ b/drivers/md/dm-cache-target.c
@@ -1765,11 +1765,11 @@ static void destroy(struct cache *cache)
 	if (cache->wq)
 		destroy_workqueue(cache->wq);
 
-	if (cache->dirty_bitset)
-		free_bitset(cache->dirty_bitset);
+	if (cache.dirty_bitset)
+		free_bitset(cache.dirty_bitset);
 
-	if (cache->discard_bitset)
-		free_bitset(cache->discard_bitset);
+	if (cache.discard_bitset)
+		free_bitset(cache.discard_bitset);
 
 	if (cache->copier)
 		dm_kcopyd_client_destroy(cache->copier);
@@ -2269,16 +2269,16 @@ static int cache_create(struct cache_args *ca, struct cache **result)
 
 	r = -ENOMEM;
 	atomic_set(&cache->nr_dirty, 0);
-	cache->dirty_bitset = alloc_bitset(from_cblock(cache->cache_size));
-	if (!cache->dirty_bitset) {
+	cache.dirty_bitset = alloc_bitset(from_cblock(cache->cache_size));
+	if (!cache.dirty_bitset) {
 		*error = "could not allocate dirty bitset";
 		goto bad;
 	}
 	clear_bitset(cache->dirty_bitset, from_cblock(cache->cache_size));
 
 	cache->discard_nr_blocks = cache->origin_blocks;
-	cache->discard_bitset = alloc_bitset(from_oblock(cache->discard_nr_blocks));
-	if (!cache->discard_bitset) {
+	cache.discard_bitset = alloc_bitset(from_oblock(cache->discard_nr_blocks));
+	if (!cache.discard_bitset) {
 		*error = "could not allocate discard bitset";
 		goto bad;
 	}
-- 
1.9.1

Re: [PATCH] dm cache: fix race affecting dirty block count

[Pranith Kumar]

On 08/03/2014 12:57 AM, Pranith Kumar wrote:
> On 08/03/2014 12:46 AM, Pranith Kumar wrote:
>> On 08/02/2014 10:10 PM, Pranith Kumar wrote:
>>> Corrently adding Anssi this time.
>>>
>>> On Sat, Aug 2, 2014 at 10:00 PM, Pranith Kumar <bobby.prani@gmail.com> wrote:
>>>> Hello Anssi, Joe, Mike,
>>>>
>>>> I just found your patch in the latest rc and wanted to ask a few
>>>> questions. I am not sure how this patch helps at all other than luck in
>>>> that dm_cblock_t type is of type int32_t, which should guarantee that it
>>>> is atomic on most platforms. Which begs the question, what platform did
>>>> you encounter this problem?
>>>>
>>>> The patch purports to solve a race condition by making use of atomic_t.
>>>> I am not sure that is enough. If indeed there is a race you need to use
>>>> smp_mb__{before/after}_atomic() for both your uses of atomic_inc() and
>>>> atomic_set().
>>>>
>>>> Also I have a concern about why this mail was not CC'ed on LKML. I had
>>>> to go to some difficulty in finding this patch. So please CC LKML for
>>>> such patches.
>>>>
>>>> Thanks,
>>>> --
>>>> Pranith
>>>>
>>>> -- Begin forwarded Message --
>>>>
>>>>
>>>> nr_dirty is updated without locking, causing it to drift so that it is
>>>> non-zero (either a small positive integer, or a very large one when an
>>>> underflow occurs) even when there are no actual dirty blocks.
>>>>
>>>> Fix that by using an atomic type for nr_dirty.
>>>>
>>>> Signed-off-by: Anssi Hannula <anssi hannula iki fi>
>>>> Cc: Joe Thornber <ejt redhat com>
>>>> Cc: stable vger kernel org
> 

There are more in the following patch. I think you need to really check what
else I might be missing.

From: Pranith Kumar <bobby.prani@gmail.com>
Date: Sun, 3 Aug 2014 01:15:10 -0400
Subject: [PATCH 1/1] dm cache: Fix more incorrect pointer assignments

Fix incorrect pointer uses and assignments.

Signed-off-by: Pranith Kumar <bobby.prani@gmail.com>
---
 drivers/md/dm-cache-target.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
index 49e47e7..1627035 100644
--- a/drivers/md/dm-cache-target.c
+++ b/drivers/md/dm-cache-target.c
@@ -1777,13 +1777,13 @@ static void destroy(struct cache *cache)
 	if (cache->cmd)
 		dm_cache_metadata_close(cache->cmd);
 
-	if (cache->metadata_dev)
+	if (cache.metadata_dev)
 		dm_put_device(cache->ti, cache->metadata_dev);
 
-	if (cache->origin_dev)
+	if (cache.origin_dev)
 		dm_put_device(cache->ti, cache->origin_dev);
 
-	if (cache->cache_dev)
+	if (cache.cache_dev)
 		dm_put_device(cache->ti, cache->cache_dev);
 
 	if (cache->policy)
@@ -1861,13 +1861,13 @@ struct cache_args {
 
 static void destroy_cache_args(struct cache_args *ca)
 {
-	if (ca->metadata_dev)
+	if (ca.metadata_dev)
 		dm_put_device(ca->ti, ca->metadata_dev);
 
-	if (ca->cache_dev)
+	if (ca.cache_dev)
 		dm_put_device(ca->ti, ca->cache_dev);
 
-	if (ca->origin_dev)
+	if (ca.origin_dev)
 		dm_put_device(ca->ti, ca->origin_dev);
 
 	kfree(ca);
@@ -2190,7 +2190,7 @@ static int cache_create(struct cache_args *ca, struct cache **result)
 	cache->origin_dev = ca->origin_dev;
 	cache->cache_dev = ca->cache_dev;
 
-	ca->metadata_dev = ca->origin_dev = ca->cache_dev = NULL;
+	ca.metadata_dev = ca.origin_dev = ca.cache_dev = NULL;
 
 	/* FIXME: factor out this whole section */
 	origin_blocks = cache->origin_sectors = ca->origin_sectors;
-- 
1.9.1

Re: [PATCH] dm cache: fix race affecting dirty block count

[Pranith Kumar]

On Sun, Aug 3, 2014 at 1:17 AM, Pranith Kumar <bobby.prani@gmail.com> wrote:
> On 08/03/2014 12:57 AM, Pranith Kumar wrote:
>> On 08/03/2014 12:46 AM, Pranith Kumar wrote:
>>> On 08/02/2014 10:10 PM, Pranith Kumar wrote:
>>>> Corrently adding Anssi this time.
>>>>
>>>> On Sat, Aug 2, 2014 at 10:00 PM, Pranith Kumar <bobby.prani@gmail.com> wrote:
>>>>> Hello Anssi, Joe, Mike,
>>>>>
>>>>> I just found your patch in the latest rc and wanted to ask a few
>>>>> questions. I am not sure how this patch helps at all other than luck in
>>>>> that dm_cblock_t type is of type int32_t, which should guarantee that it
>>>>> is atomic on most platforms. Which begs the question, what platform did
>>>>> you encounter this problem?
>>>>>
>>>>> The patch purports to solve a race condition by making use of atomic_t.
>>>>> I am not sure that is enough. If indeed there is a race you need to use
>>>>> smp_mb__{before/after}_atomic() for both your uses of atomic_inc() and
>>>>> atomic_set().
>>>>>
>>>>> Also I have a concern about why this mail was not CC'ed on LKML. I had
>>>>> to go to some difficulty in finding this patch. So please CC LKML for
>>>>> such patches.
>>>>>
>>>>> Thanks,
>>>>> --
>>>>> Pranith
>>>>>
>>>>> -- Begin forwarded Message --
>>>>>
>>>>>
>>>>> nr_dirty is updated without locking, causing it to drift so that it is
>>>>> non-zero (either a small positive integer, or a very large one when an
>>>>> underflow occurs) even when there are no actual dirty blocks.
>>>>>
>>>>> Fix that by using an atomic type for nr_dirty.
>>>>>
>>>>> Signed-off-by: Anssi Hannula <anssi hannula iki fi>
>>>>> Cc: Joe Thornber <ejt redhat com>
>>>>> Cc: stable vger kernel org
>>
>

Bah, please ignore all the previous patches about pointers. It is me
who is drunk and got my pointers mixed up. :(

-- 
Pranith