All of lore.kernel.org
 help / color / mirror / Atom feed
From: Benjamin Marzinski <bmarzins@redhat.com>
To: Qiliang Yuan <qiliang.yuan@linux.dev>
Cc: Alasdair Kergon <agk@redhat.com>,
	Mike Snitzer <snitzer@kernel.org>,
	Mikulas Patocka <mpatocka@redhat.com>,
	dm-devel@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [PATCH 2/2] dm thin: free thins with kfree_rcu() instead of waiting in thin_dtr()
Date: Tue, 17 Nov 2026 20:13:12 -0500	[thread overview]
Message-ID: <avz7qDWd2g-uubmL@redhat.com> (raw)
In-Reply-To: <20261116-bug-dm-thin-rcu-gp-v1-2-14ed3567da3e@linux.dev>

On Mon, Nov 16, 2026 at 12:12:04AM -0500, Qiliang Yuan wrote:
> get_first_thin() and get_next_thin() find a thin on pool->active_thins
> under rcu_read_lock() and take a reference on it. thin_dtr() removes the
> thin from the list and waits for a grace period before dropping its own
> reference, so that no walker takes one after the last is gone.
> 
> Removing a thin device thus waits for a full grace period in thin_dtr(),
> before it waits for the references that are still held.
> 
> Take the references with refcount_inc_not_zero() and skip the thins
> whose last reference is gone, so that thin_dtr() only waits for the
> references already taken. Free the thin with kfree_rcu(), as
> requeue_bios() and error_retry_list_with_code() may still look at it
> under rcu_read_lock().
> 
> Removing 100 thin devices of an idle pool with dmsetup remove takes per
> device (mean of five rounds, min-max of the rounds in parentheses). The
> queue freeze in del_gendisk() waits for a grace period of its own, which
> the one in thin_dtr() used to overlap, so only part of it is saved:
> 
>   before (ms)        after (ms)         change
>   16.9 (15.8-17.3)   14.4 (13.8-15.3)     -15%
> 
> Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>

So, this isn't directly a problem with your patch, but in reviewing it,
I noticed a bug in the interaction between thin_dtr() and
get_next_thin(). thin_dtr() can take tc off the active_thins list, while
it's being held by a worker. The refcount keeps it from being deleted.
But once tc is removed from the list, there's nothing that guarantees
that tc->list.next is still meaningful. If "dmsetup remove" is called on
the thin device that is next in the active_thins list, that thin_dtr()
call can free the thin pointed to by the original tc->list.next pointer.
So when the original thin_dtr() calls get_next_thin(), it will access
freed memory.

I think the solution is for thin_dtr to mark tc as dead somehow, instead
of removing it from active_thins where we currently do. Once the
refcount drops to 0, we can remove it, wait for a grace period, and free
it.

Or am I missing something subtle that keeps this issue from happening?

-Ben

> ---
>  drivers/md/dm-thin.c | 30 ++++++++++++++++++------------
>  1 file changed, 18 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c
> index d5ff937eeacba..8386215101eb2 100644
> --- a/drivers/md/dm-thin.c
> +++ b/drivers/md/dm-thin.c
> @@ -360,6 +360,7 @@ struct thin_c {
>  	 */
>  	refcount_t refcount;
>  	struct completion can_destroy;
> +	struct rcu_head rcu;
>  };
>  
>  /*----------------------------------------------------------------*/
> @@ -2317,25 +2318,29 @@ static void process_thin_deferred_cells(struct thin_c *tc)
>  	} while (!list_empty(&cells));
>  }
>  
> -static void thin_get(struct thin_c *tc);
> +static bool thin_tryget(struct thin_c *tc);
>  static void thin_put(struct thin_c *tc);
>  
>  /*
>   * We can't hold rcu_read_lock() around code that can block.  So we
>   * find a thin with the rcu lock held; bump a refcount; then drop
> - * the lock.
> + * the lock.  Skip the thins whose refcount thin_dtr() already dropped
> + * to zero.
>   */
>  static struct thin_c *get_first_thin(struct pool *pool)
>  {
> -	struct thin_c *tc = NULL;
> +	struct thin_c *tc;
>  
>  	rcu_read_lock();
> -	tc = list_first_or_null_rcu(&pool->active_thins, struct thin_c, list);
> -	if (tc)
> -		thin_get(tc);
> +	list_for_each_entry_rcu(tc, &pool->active_thins, list) {
> +		if (thin_tryget(tc)) {
> +			rcu_read_unlock();
> +			return tc;
> +		}
> +	}
>  	rcu_read_unlock();
>  
> -	return tc;
> +	return NULL;
>  }
>  
>  static struct thin_c *get_next_thin(struct pool *pool, struct thin_c *tc)
> @@ -2344,7 +2349,8 @@ static struct thin_c *get_next_thin(struct pool *pool, struct thin_c *tc)
>  
>  	rcu_read_lock();
>  	list_for_each_entry_continue_rcu(tc, &pool->active_thins, list) {
> -		thin_get(tc);
> +		if (!thin_tryget(tc))
> +			continue;
>  		thin_put(old_tc);
>  		rcu_read_unlock();
>  		return tc;
> @@ -4128,9 +4134,9 @@ static struct target_type pool_target = {
>   * Thin target methods
>   *--------------------------------------------------------------
>   */
> -static void thin_get(struct thin_c *tc)
> +static bool thin_tryget(struct thin_c *tc)
>  {
> -	refcount_inc(&tc->refcount);
> +	return refcount_inc_not_zero(&tc->refcount);
>  }
>  
>  static void thin_put(struct thin_c *tc)
> @@ -4146,7 +4152,6 @@ static void thin_dtr(struct dm_target *ti)
>  	spin_lock_irq(&tc->pool->lock);
>  	list_del_rcu(&tc->list);
>  	spin_unlock_irq(&tc->pool->lock);
> -	synchronize_rcu();
>  
>  	thin_put(tc);
>  	wait_for_completion(&tc->can_destroy);
> @@ -4158,7 +4163,8 @@ static void thin_dtr(struct dm_target *ti)
>  	dm_put_device(ti, tc->pool_dev);
>  	if (tc->origin_dev)
>  		dm_put_device(ti, tc->origin_dev);
> -	kfree(tc);
> +	/* Let RCU readers of active_thins finish with tc before freeing it */
> +	kfree_rcu(tc, rcu);
>  
>  	mutex_unlock(&dm_thin_pool_table.mutex);
>  }
> 
> -- 
> 2.43.0


      reply	other threads:[~2026-11-18  1:13 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16  5:12 [PATCH 0/2] dm thin: stop waiting for grace periods when activating and removing thins Qiliang Yuan
2026-11-16  5:12 ` [PATCH 1/2] dm thin: drop the grace period from thin_ctr() Qiliang Yuan
2026-11-18  0:51   ` Benjamin Marzinski
2026-11-16  5:12 ` [PATCH 2/2] dm thin: free thins with kfree_rcu() instead of waiting in thin_dtr() Qiliang Yuan
2026-11-18  1:13   ` Benjamin Marzinski [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=avz7qDWd2g-uubmL@redhat.com \
    --to=bmarzins@redhat.com \
    --cc=agk@redhat.com \
    --cc=dm-devel@lists.linux.dev \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mpatocka@redhat.com \
    --cc=qiliang.yuan@linux.dev \
    --cc=snitzer@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.