All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jens Axboe <axboe@kernel.dk>
To: Caleb Sander Mateos <csander@purestorage.com>
Cc: Ming Lei <ming.lei@redhat.com>,
	linux-block@vger.kernel.org, Nilay Shroff <nilay@linux.ibm.com>,
	Yu Kuai <yukuai@fnnas.com>, Guangwu Zhang <guazhang@redhat.com>
Subject: Re: [PATCH V2] block: fix race between wbt_enable_default and IO submission
Date: Thu, 18 Dec 2025 08:11:26 -0700	[thread overview]
Message-ID: <54f07e42-0484-4d69-be8c-aee7182d97a6@kernel.dk> (raw)
In-Reply-To: <CADUfDZq8F2YL-vWcbop4KDrZ1fz5nBP6dbXVB6765kpFgxuy1g@mail.gmail.com>

On 12/17/25 9:41 PM, Caleb Sander Mateos wrote:
> On Wed, Dec 17, 2025 at 7:20 PM Jens Axboe <axboe@kernel.dk> wrote:
>>
>> On 12/17/25 8:18 PM, Ming Lei wrote:
>>> On Fri, Dec 12, 2025 at 10:35?PM Ming Lei <ming.lei@redhat.com> wrote:
>>>>
>>>> When wbt_enable_default() is moved out of queue freezing in elevator_change(),
>>>> it can cause the wbt inflight counter to become negative (-1), leading to hung
>>>> tasks in the writeback path. Tasks get stuck in wbt_wait() because the counter
>>>> is in an inconsistent state.
>>>>
>>>> The issue occurs because wbt_enable_default() could race with IO submission,
>>>> allowing the counter to be decremented before proper initialization. This manifests
>>>> as:
>>>>
>>>>   rq_wait[0]:
>>>>     inflight:             -1
>>>>     has_waiters:        True
>>>>
>>>> rwb_enabled() checks the state, which can be updated exactly between wbt_wait()
>>>> (rq_qos_throttle()) and wbt_track()(rq_qos_track()), then the inflight counter
>>>> will become negative.
>>>>
>>>> And results in hung task warnings like:
>>>>   task:kworker/u24:39 state:D stack:0 pid:14767
>>>>   Call Trace:
>>>>     rq_qos_wait+0xb4/0x150
>>>>     wbt_wait+0xa9/0x100
>>>>     __rq_qos_throttle+0x24/0x40
>>>>     blk_mq_submit_bio+0x672/0x7b0
>>>>     ...
>>>>
>>>> Fix this by:
>>>>
>>>> 1. Splitting wbt_enable_default() into:
>>>>    - __wbt_enable_default(): Returns true if wbt_init() should be called
>>>>    - wbt_enable_default(): Wrapper for existing callers (no init)
>>>>    - wbt_init_enable_default(): New function that checks and inits WBT
>>>>
>>>> 2. Using wbt_init_enable_default() in blk_register_queue() to ensure
>>>>    proper initialization during queue registration
>>>>
>>>> 3. Move wbt_init() out of wbt_enable_default() which is only for enabling
>>>>    disabled wbt from bfq and iocost, and wbt_init() isn't needed. Then the
>>>>    original lock warning can be avoided.
>>>>
>>>> 4. Removing the ELEVATOR_FLAG_ENABLE_WBT_ON_EXIT flag and its handling
>>>>    code since it's no longer needed
>>>>
>>>> This ensures WBT is properly initialized before any IO can be submitted,
>>>> preventing the counter from going negative.
>>>>
>>>> Cc: Nilay Shroff <nilay@linux.ibm.com>
>>>> Cc: Yu Kuai <yukuai@fnnas.com>
>>>> Cc: Guangwu Zhang <guazhang@redhat.com>
>>>> Fixes: 78c271344b6f ("block: move wbt_enable_default() out of queue freezing from sched ->exit()")
>>>> Signed-off-by: Ming Lei <ming.lei@redhat.com>
>>>> ---
>>>> V2:
>>>>         - explain the race in commit log(Nilay, YuKuai)
>>>
>>> Hi Jens,
>>>
>>> Can you consider this fix for v6.19 if you are fine? Yu Kuai has one
>>> patchset which depends
>>> on this fix.
>>
>> It was queued up on the 12/12, now I'm pondering if the "thanks applied"
>> email never got sent out? In any case, it is in block-6.19.
> 
> I also don't see an email about this series being applied, though it
> appears to be queued in block-6.19:
> https://lore.kernel.org/linux-block/20251212171707.1876250-1-csander@purestorage.com/

Funky! Applied on the same day, so quite possible I messed it up
by flushing the b4 queue or something.

-- 
Jens Axboe


      reply	other threads:[~2025-12-18 15:11 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-12-12 14:35 [PATCH V2] block: fix race between wbt_enable_default and IO submission Ming Lei
2025-12-12 15:57 ` Nilay Shroff
2025-12-14  9:13 ` Yu Kuai
2025-12-18  3:18 ` Ming Lei
2025-12-18  3:20   ` Jens Axboe
2025-12-18  4:41     ` Caleb Sander Mateos
2025-12-18 15:11       ` Jens Axboe [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=54f07e42-0484-4d69-be8c-aee7182d97a6@kernel.dk \
    --to=axboe@kernel.dk \
    --cc=csander@purestorage.com \
    --cc=guazhang@redhat.com \
    --cc=linux-block@vger.kernel.org \
    --cc=ming.lei@redhat.com \
    --cc=nilay@linux.ibm.com \
    --cc=yukuai@fnnas.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.