CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown

CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown

[Greg Kroah-Hartman]

From: Greg Kroah-Hartman <gregkh@kernel.org>

Description
===========

In the Linux kernel, the following vulnerability has been resolved:

blk-mq: fix potential deadlock while nr_requests grown

Allocate and free sched_tags while queue is freezed can deadlock[1],
this is a long term problem, hence allocate memory before freezing
queue and free memory after queue is unfreezed.

[1] https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/

The Linux kernel CVE team has assigned CVE-2025-40146 to this issue.


Affected and fixed versions
===========================

	Issue introduced in 3.16 with commit e3a2b3f931f59d5284abd13faf8bded726884ffd and fixed in 6.17.3 with commit 8d26acf8477174d8ef690eb6affe13a630f586ae
	Issue introduced in 3.16 with commit e3a2b3f931f59d5284abd13faf8bded726884ffd and fixed in 6.18-rc1 with commit b86433721f46d934940528f28d49c1dedb690df1

Please see https://www.kernel.org for a full list of currently supported
kernel versions by the kernel community.

Unaffected versions might change over time as fixes are backported to
older supported kernel versions.  The official CVE entry at
	https://cve.org/CVERecord/?id=CVE-2025-40146
will be updated if fixes are backported, please check that for the most
up to date information about this issue.


Affected files
==============

The file(s) affected by this issue are:
	block/blk-mq.c
	block/blk-mq.h
	block/blk-sysfs.c


Mitigation
==========

The Linux kernel CVE team recommends that you update to the latest
stable kernel version for this, and many other bugfixes.  Individual
changes are never tested alone, but rather are part of a larger kernel
release.  Cherry-picking individual commits is not recommended or
supported by the Linux kernel community at all.  If however, updating to
the latest release is impossible, the individual changes to resolve this
issue can be found at these commits:
	https://git.kernel.org/stable/c/8d26acf8477174d8ef690eb6affe13a630f586ae
	https://git.kernel.org/stable/c/b86433721f46d934940528f28d49c1dedb690df1

Re: CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown

[Zheng Qixing]

Hi,

Commit b86433721f46 ("blk-mq: fix potential deadlock while nr_requests 
grown")  aims to avoid a deadlock issue when the queue is frozen and 
memory reclaim is triggered.

However, the sysfs nr_requests update path is already under a 
memalloc_noio_save() region while the queue is frozen (via 
blk_mq_freeze_queue()).

Would it be possible to reject this CVE, or clarify why it needs CVE 
assignment in this case?

Any feedback or further explanation would be appreciated.


Best regards,

Qixing

Re: CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown

[Greg KH]

On Thu, Nov 27, 2025 at 09:22:42PM +0800, Zheng Qixing wrote:
> Hi,
> 
> Commit b86433721f46 ("blk-mq: fix potential deadlock while nr_requests
> grown")  aims to avoid a deadlock issue when the queue is frozen and memory
> reclaim is triggered.
> 
> However, the sysfs nr_requests update path is already under a
> memalloc_noio_save() region while the queue is frozen (via
> blk_mq_freeze_queue()).

Did the lockdep splat in
https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/
not describe the issue here that the commit is attempting to solve?

thanks,

greg k-h

Re: CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown

[Zheng Qixing]

在 2025/11/27 21:39, Greg KH 写道:
> On Thu, Nov 27, 2025 at 09:22:42PM +0800, Zheng Qixing wrote:
>> Hi,
>>
>> Commit b86433721f46 ("blk-mq: fix potential deadlock while nr_requests
>> grown")  aims to avoid a deadlock issue when the queue is frozen and memory
>> reclaim is triggered.
>>
>> However, the sysfs nr_requests update path is already under a
>> memalloc_noio_save() region while the queue is frozen (via
>> blk_mq_freeze_queue()).
> Did the lockdep splat in
> https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/
> not describe the issue here that the commit is attempting to solve?
>
> thanks,
>
> greg k-h


The deadlock issue described in this link is about elevator switch path, 
but the patch modifies sysfs nr_requests update path.

I didn't identify any potential deadlock issues on this path. If I 
misunderstood something, could someone help clarify?


Thanks,

Qixing

Re: CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown

[Nilay Shroff]

On 11/28/25 6:45 AM, Zheng Qixing wrote:
> 
> 在 2025/11/27 21:39, Greg KH 写道:
>> On Thu, Nov 27, 2025 at 09:22:42PM +0800, Zheng Qixing wrote:
>>> Hi,
>>>
>>> Commit b86433721f46 ("blk-mq: fix potential deadlock while nr_requests
>>> grown")  aims to avoid a deadlock issue when the queue is frozen and memory
>>> reclaim is triggered.
>>>
>>> However, the sysfs nr_requests update path is already under a
>>> memalloc_noio_save() region while the queue is frozen (via
>>> blk_mq_freeze_queue()).
>> Did the lockdep splat in
>> https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/
>> not describe the issue here that the commit is attempting to solve?
>>
>> thanks,
>>
>> greg k-h
> 
> 
> The deadlock issue described in this link is about elevator switch path, but the patch modifies sysfs nr_requests update path.
> 
> I didn't identify any potential deadlock issues on this path. If I misunderstood something, could someone help clarify?
> 
> 
Let me clarify the confusion here.

The deadlock reported in [1] requires updates across multiple code paths. There are
three distinct paths that need to be fixed to avoid the deadlock. While the report
in [1] only exposed the issue in one of these paths, we already knew that all three
paths needed changes to fully resolve the problem:

1. Elevator change path (via sysfs attribute) that triggers a scheduler tags update
2. Elevator change path triggered by an nr_hw_queues update which also triggers a
   scheduler tags update
3. Scheduler tags update triggered through the nr_requests sysfs attribute (please 
   note when nr_requests grows beyond current queue depth it triggers scheduler
   tags update)

The first two code paths were addressed by:
commit f5a6604f7a44 (“block: fix lockdep warning caused by lock dependency in 
elv_iosched_store”), and commit 04225d13aef1 (“block: fix potential deadlock while
running nr_hw_queue update”) respectively.

The third code path was fixed by:
commit b86433721f46 (“blk-mq: fix potential deadlock while nr_requests grown”).

Ideally, all of these commits should be referenced as collectively fixing the lockdep
splat reported in [1]. I hope this clarifies the situation. Please let me know if you
have any further questions.

[1] https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/

Thanks,
--Nilay

Re: CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown

[yangerkun]

在 2025/11/28 13:20, Nilay Shroff 写道:
> 
> 
> On 11/28/25 6:45 AM, Zheng Qixing wrote:
>>
>> 在 2025/11/27 21:39, Greg KH 写道:
>>> On Thu, Nov 27, 2025 at 09:22:42PM +0800, Zheng Qixing wrote:
>>>> Hi,
>>>>
>>>> Commit b86433721f46 ("blk-mq: fix potential deadlock while nr_requests
>>>> grown")  aims to avoid a deadlock issue when the queue is frozen and memory
>>>> reclaim is triggered.
>>>>
>>>> However, the sysfs nr_requests update path is already under a
>>>> memalloc_noio_save() region while the queue is frozen (via
>>>> blk_mq_freeze_queue()).
>>> Did the lockdep splat in
>>> https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/
>>> not describe the issue here that the commit is attempting to solve?
>>>
>>> thanks,
>>>
>>> greg k-h
>>
>>
>> The deadlock issue described in this link is about elevator switch path, but the patch modifies sysfs nr_requests update path.
>>
>> I didn't identify any potential deadlock issues on this path. If I misunderstood something, could someone help clarify?
>>
>>
> Let me clarify the confusion here.
> 
> The deadlock reported in [1] requires updates across multiple code paths. There are
> three distinct paths that need to be fixed to avoid the deadlock. While the report
> in [1] only exposed the issue in one of these paths, we already knew that all three
> paths needed changes to fully resolve the problem:
> 
> 1. Elevator change path (via sysfs attribute) that triggers a scheduler tags update
> 2. Elevator change path triggered by an nr_hw_queues update which also triggers a
>     scheduler tags update
> 3. Scheduler tags update triggered through the nr_requests sysfs attribute (please
>     note when nr_requests grows beyond current queue depth it triggers scheduler
>     tags update)

commit b86433721f46d934940528f28d49c1dedb690df1 (HEAD -> master)
Author: Yu Kuai <yukuai3@huawei.com>
Date:   Wed Sep 10 16:04:43 2025 +0800

     blk-mq: fix potential deadlock while nr_requests grown

     Allocate and free sched_tags while queue is freezed can deadlock[1],
     this is a long term problem, hence allocate memory before freezing
     queue and free memory after queue is unfreezed.

     [1] 
https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/
     Fixes: e3a2b3f931f5 ("blk-mq: allow changing of queue depth through 
sysfs")

     Signed-off-by: Yu Kuai <yukuai3@huawei.com>
     Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
     Signed-off-by: Jens Axboe <axboe@kernel.dk>

We are assume that what's the problem Yu describe is when we update
nr_request, we may need some memory allocation(nr_requests grows). And
the memory allocation may trigger some memory reclaim, and fall into
another I/O process, and since the request_queue has been freezen, there
exist deadlock.

But after checking the source code, there exist
queue_requests_store->blk_mq_freeze_queue->memalloc_noio_save, the
whole process which may trigger memory allocation won't trigger I/O
process. So deadlock can not happened... And if that's true, this patch
does not fix any problem.

Thanks,
Erkun.

> 
> The first two code paths were addressed by:
> commit f5a6604f7a44 (“block: fix lockdep warning caused by lock dependency in
> elv_iosched_store”), and commit 04225d13aef1 (“block: fix potential deadlock while
> running nr_hw_queue update”) respectively.
> 
> The third code path was fixed by:
> commit b86433721f46 (“blk-mq: fix potential deadlock while nr_requests grown”).
> 
> Ideally, all of these commits should be referenced as collectively fixing the lockdep
> splat reported in [1]. I hope this clarifies the situation. Please let me know if you
> have any further questions.
> 
> [1] https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/
> 
> Thanks,
> --Nilay
> 

Re: CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown

[Nilay Shroff]

> 
> commit b86433721f46d934940528f28d49c1dedb690df1 (HEAD -> master)
> Author: Yu Kuai <yukuai3@huawei.com>
> Date:   Wed Sep 10 16:04:43 2025 +0800
> 
>     blk-mq: fix potential deadlock while nr_requests grown
> 
>     Allocate and free sched_tags while queue is freezed can deadlock[1],
>     this is a long term problem, hence allocate memory before freezing
>     queue and free memory after queue is unfreezed.
> 
>     [1] https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/
>     Fixes: e3a2b3f931f5 ("blk-mq: allow changing of queue depth through sysfs")
> 
>     Signed-off-by: Yu Kuai <yukuai3@huawei.com>
>     Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
>     Signed-off-by: Jens Axboe <axboe@kernel.dk>
> 
> We are assume that what's the problem Yu describe is when we update
> nr_request, we may need some memory allocation(nr_requests grows). And
> the memory allocation may trigger some memory reclaim, and fall into
> another I/O process, and since the request_queue has been freezen, there
> exist deadlock.
> 
> But after checking the source code, there exist
> queue_requests_store->blk_mq_freeze_queue->memalloc_noio_save, the
> whole process which may trigger memory allocation won't trigger I/O
> process. So deadlock can not happened... And if that's true, this patch
> does not fix any problem.
> 

Yes, memalloc_noio_save() is invoked before we freeze the queue (e.g., in
elv_iosched_store()), but that does not prevent the deadlock scenario described
in the lockdep splat.

If you look closely at the splat, the problematic lock is not fs_reclaim (which
may be the first impression), but rather ->pcpu_alloc_mutex. From the splat, the
chain of dependencies looks like this:

thread #0: blocked on q->elevator_lock  
  thread #1: blocked on ->pcpu_alloc_mutex
    thread #2: blocked on fs-reclaim
 
Here is the key detail:

Thread #0 is running under GFP_NOIO scope (due to memalloc_noio_save()). 
However, it is not blocked on fs_reclaim. Instead, it is blocked 
on ->elevator_lock.

Thread #1 is also running with GFP_NOIO and holds ->elevator_lock
while the queue is frozen. It is blocked on ->pcpu_alloc_mutex,
which is already held by Thread #2 (the thread that is stuck in
fs_reclaim). Thread #2 is running without GFP_NOIO scope.

In other words:
- GFP_NOIO prevents a thread from entering fs_reclaim, but it does
  not prevent triggering per-CPU memory allocations, which require
  taking ->pcpu_alloc_mutex.
- This ->pcpu_alloc_mutex is the actual source of contention in the
  splat, and it sits outside the protections offered by GFP_NOIO. 

That means:
- Even though memalloc_noio_save() avoids fs reclaim recursion, 
  it does not prevent per-CPU allocations from blocking, and thus
  it cannot prevent the deadlock involving ->pcpu_alloc_mutex.

So the reasoning that “memalloc_noio_save() prevents deadlock” is
incomplete — GFP_NOIO only handles reclaim, not per-CPU allocations.

This is why the patch is still needed: GFP_NOIO scope with freezing
the queue and modifying scheduler tags can still lead to a circular
dependency involving ->pcpu_alloc_mutex, which the splat clearly shows. 

If you look at the reasoning described in commit f5a6604f7a44 (“block: 
fix lockdep warning caused by lock dependency in elv_iosched_store”) and
commit 04225d13aef1 (“block: fix potential deadlock while running 
nr_hw_queue update”), both explicitly explain that the goal is to break
the dependency chain between the percpu allocator lock and the elevator lock.

The third commit, b86433721f46 (“blk-mq: fix potential deadlock while
nr_requests grown”), is less explicit in its explanation, but it addresses
the same underlying issue.

Taken together, all these commits resolve the deadlock described in the
lockdep splat.

Thanks,
--Nilay

Re: CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown

[Zheng Qixing]

在 2025/11/28 15:15, Nilay Shroff 写道:
>> commit b86433721f46d934940528f28d49c1dedb690df1 (HEAD -> master)
>> Author: Yu Kuai <yukuai3@huawei.com>
>> Date:   Wed Sep 10 16:04:43 2025 +0800
>>
>>      blk-mq: fix potential deadlock while nr_requests grown
>>
>>      Allocate and free sched_tags while queue is freezed can deadlock[1],
>>      this is a long term problem, hence allocate memory before freezing
>>      queue and free memory after queue is unfreezed.
>>
>>      [1] https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/
>>      Fixes: e3a2b3f931f5 ("blk-mq: allow changing of queue depth through sysfs")
>>
>>      Signed-off-by: Yu Kuai <yukuai3@huawei.com>
>>      Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
>>      Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>
>> We are assume that what's the problem Yu describe is when we update
>> nr_request, we may need some memory allocation(nr_requests grows). And
>> the memory allocation may trigger some memory reclaim, and fall into
>> another I/O process, and since the request_queue has been freezen, there
>> exist deadlock.
>>
>> But after checking the source code, there exist
>> queue_requests_store->blk_mq_freeze_queue->memalloc_noio_save, the
>> whole process which may trigger memory allocation won't trigger I/O
>> process. So deadlock can not happened... And if that's true, this patch
>> does not fix any problem.
>>
> Yes, memalloc_noio_save() is invoked before we freeze the queue (e.g., in
> elv_iosched_store()), but that does not prevent the deadlock scenario described
> in the lockdep splat.
>
> If you look closely at the splat, the problematic lock is not fs_reclaim (which
> may be the first impression), but rather ->pcpu_alloc_mutex. From the splat, the
> chain of dependencies looks like this:
>
> thread #0: blocked on q->elevator_lock
>    thread #1: blocked on ->pcpu_alloc_mutex
>      thread #2: blocked on fs-reclaim
>   
> Here is the key detail:
>
> Thread #0 is running under GFP_NOIO scope (due to memalloc_noio_save()).
> However, it is not blocked on fs_reclaim. Instead, it is blocked
> on ->elevator_lock.
>
> Thread #1 is also running with GFP_NOIO and holds ->elevator_lock
> while the queue is frozen. It is blocked on ->pcpu_alloc_mutex,
> which is already held by Thread #2 (the thread that is stuck in
> fs_reclaim). Thread #2 is running without GFP_NOIO scope.
>
> In other words:
> - GFP_NOIO prevents a thread from entering fs_reclaim, but it does
>    not prevent triggering per-CPU memory allocations, which require
>    taking ->pcpu_alloc_mutex.
> - This ->pcpu_alloc_mutex is the actual source of contention in the
>    splat, and it sits outside the protections offered by GFP_NOIO.
>
> That means:
> - Even though memalloc_noio_save() avoids fs reclaim recursion,
>    it does not prevent per-CPU allocations from blocking, and thus
>    it cannot prevent the deadlock involving ->pcpu_alloc_mutex.
>

Thank you for the detailed explanation.

Now I understand that there could indeed be a deadlock issue here :)


Thanks,

Qixing

Re: CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown

[Zheng Qixing]

Commit 28307d938fb2 ("percpu: make pcpu_alloc() aware of current gfp 
context") has fixed a reclaim recursion for scoped GFP_NOFS context by 
avoiding taking pcpu_alloc_mutex.

@@ -1569,6 +1569,12 @@ static void __percpu *pcpu_alloc(size_t size, 
size_t align, bool reserved, void __percpu *ptr; size_t bits, bit_align; 
+ gfp = current_gfp_context(gfp); + /* whitelisted flags that can be 
passed to the backing allocators */ + pcpu_gfp = gfp & (GFP_KERNEL | 
__GFP_NORETRY | __GFP_NOWARN); + is_atomic = (gfp & GFP_KERNEL) != 
GFP_KERNEL; + do_warn = !(gfp & __GFP_NOWARN); Commit 9a5b183941b5 ("mm, 
percpu: do not consider sleepable allocations atomic") fixes premature 
allocation failures in certain scenarios. However, this change made it 
possible to acquire the pcpu_alloc_mutex under GFP_NOIO scope.

@@ -1745,7 +1745,7 @@ void __percpu *pcpu_alloc_noprof(size_t size, 
size_t align, bool reserved, gfp = current_gfp_context(gfp); /* 
whitelisted flags that can be passed to the backing allocators */ 
pcpu_gfp = gfp & (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN); - 
is_atomic = (gfp & GFP_KERNEL) != GFP_KERNEL; + is_atomic = 
!gfpflags_allow_blocking(gfp); do_warn = !(gfp & __GFP_NOWARN); Here's 
the relevant commit timeline:

e3a2b3f931f5 ("blk-mq: allow changing of queue depth through sysfs") 
v3.16-rc1 28307d938fb2 ("percpu: make pcpu_alloc() aware of current gfp 
context") v5.7-rc5 9a5b183941b5 ("mm, percpu: do not consider sleepable 
allocations atomic") v6.15-rc1 b86433721f46 ("blk-mq: fix potential 
deadlock while nr_requests grown") v6.18-rc1

This means that in the Linux master branch, this deadlock issue *did not 
exist* during the version window from v5.7-rc5 to v6.15-rc1. After 
analyzing the LTS versions, I found that linux-5.7.y through 
linux-6.13.y should also not have this deadlock issue.

If you have any questions or concerns, please feel free to discuss further.


Best regards,

Qixing

Re: CVE-2025-40146: blk-mq: fix potential deadlock while nr_requests grown

[Zheng Qixing]

Sorry for resending this message.


Commit 28307d938fb2 ("percpu: make pcpu_alloc() aware of current gfp 
context") has fixed a reclaim recursion for scoped GFP_NOFS context by 
avoiding taking pcpu_alloc_mutex.

@@ -1569,6 +1569,12 @@ static void __percpu *pcpu_alloc(size_t size, 
size_t align, bool reserved,
      void __percpu *ptr;
      size_t bits, bit_align;

+    gfp = current_gfp_context(gfp);
+    /* whitelisted flags that can be passed to the backing allocators */
+    pcpu_gfp = gfp & (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN);
+    is_atomic = (gfp & GFP_KERNEL) != GFP_KERNEL;
+    do_warn = !(gfp & __GFP_NOWARN);


Commit 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations 
atomic") fixes premature allocation failures in certain scenarios. 
However, this change made it possible to acquire the pcpu_alloc_mutex 
under GFP_NOIO scope.

@@ -1745,7 +1745,7 @@ void __percpu *pcpu_alloc_noprof(size_t size, 
size_t align, bool reserved,
      gfp = current_gfp_context(gfp);
      /* whitelisted flags that can be passed to the backing allocators */
      pcpu_gfp = gfp & (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN);
-    is_atomic = (gfp & GFP_KERNEL) != GFP_KERNEL;
+    is_atomic = !gfpflags_allow_blocking(gfp);
      do_warn = !(gfp & __GFP_NOWARN);


Here's the relevant commit timeline:

e3a2b3f931f5 ("blk-mq: allow changing of queue depth through sysfs")    
         v3.16-rc1

28307d938fb2 ("percpu: make pcpu_alloc() aware of current gfp context")  
        v5.7-rc5

9a5b183941b5 ("mm, percpu: do not consider sleepable allocations 
atomic")    v6.15-rc1

b86433721f46 ("blk-mq: fix potential deadlock while nr_requests grown")  
         v6.18-rc1


This means that in the Linux master branch, this deadlock issue *did not 
exist* during the version window from v5.7-rc5 to v6.15-rc1. After 
analyzing the LTS versions, I found that linux-5.7.y through 
linux-6.13.y should also not have this deadlock issue.

If you have any questions or concerns, please feel free to discuss further.


Best regards,

Qixing