[syzbot] [crypto?] possible deadlock in padata_do_serial

[syzbot] [crypto?] possible deadlock in padata_do_serial

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    2ae2aaafb214 Add linux-next specific files for 20250624
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=10deb70c580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=d85d3c440a754f86
dashboard link: https://syzkaller.appspot.com/bug?extid=bd936ccd4339cea66e6b
compiler:       Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/689f558e5251/disk-2ae2aaaf.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/355aae0f4972/vmlinux-2ae2aaaf.xz
kernel image: https://storage.googleapis.com/syzbot-assets/94e736ebe604/bzImage-2ae2aaaf.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+bd936ccd4339cea66e6b@syzkaller.appspotmail.com

============================================
WARNING: possible recursive locking detected
6.16.0-rc3-next-20250624-syzkaller #0 Not tainted
--------------------------------------------
kworker/u8:11/20346 is trying to acquire lock:
ffffe8ffffd55fd8 (&pd_list->lock){+...}-{3:3}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffffe8ffffd55fd8 (&pd_list->lock){+...}-{3:3}, at: padata_find_next kernel/padata.c:256 [inline]
ffffe8ffffd55fd8 (&pd_list->lock){+...}-{3:3}, at: padata_reorder kernel/padata.c:309 [inline]
ffffe8ffffd55fd8 (&pd_list->lock){+...}-{3:3}, at: padata_do_serial+0x73e/0xb80 kernel/padata.c:379

but task is already holding lock:
ffffe8ffffd5f4d8 (&pd_list->lock){+...}-{3:3}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffffe8ffffd5f4d8 (&pd_list->lock){+...}-{3:3}, at: padata_reorder kernel/padata.c:300 [inline]
ffffe8ffffd5f4d8 (&pd_list->lock){+...}-{3:3}, at: padata_do_serial+0x5b5/0xb80 kernel/padata.c:379

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&pd_list->lock);
  lock(&pd_list->lock);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

3 locks held by kworker/u8:11/20346:
 #0: ffff888146e84948 ((wq_completion)pencrypt_parallel){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3214 [inline]
 #0: ffff888146e84948 ((wq_completion)pencrypt_parallel){+.+.}-{0:0}, at: process_scheduled_works+0x9b4/0x17b0 kernel/workqueue.c:3322
 #1: ffffc9000bf7fbc0 ((work_completion)(&pw->pw_work)){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3215 [inline]
 #1: ffffc9000bf7fbc0 ((work_completion)(&pw->pw_work)){+.+.}-{0:0}, at: process_scheduled_works+0x9ef/0x17b0 kernel/workqueue.c:3322
 #2: ffffe8ffffd5f4d8 (&pd_list->lock){+...}-{3:3}, at: spin_lock include/linux/spinlock.h:351 [inline]
 #2: ffffe8ffffd5f4d8 (&pd_list->lock){+...}-{3:3}, at: padata_reorder kernel/padata.c:300 [inline]
 #2: ffffe8ffffd5f4d8 (&pd_list->lock){+...}-{3:3}, at: padata_do_serial+0x5b5/0xb80 kernel/padata.c:379

stack backtrace:
CPU: 1 UID: 0 PID: 20346 Comm: kworker/u8:11 Not tainted 6.16.0-rc3-next-20250624-syzkaller #0 PREEMPT(full) 
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/07/2025
Workqueue: pencrypt_parallel padata_parallel_worker
Call Trace:
 <TASK>
 dump_stack_lvl+0x189/0x250 lib/dump_stack.c:120
 print_deadlock_bug+0x28b/0x2a0 kernel/locking/lockdep.c:3044
 check_deadlock kernel/locking/lockdep.c:3096 [inline]
 validate_chain+0x1a3f/0x2140 kernel/locking/lockdep.c:3898
 __lock_acquire+0xab9/0xd20 kernel/locking/lockdep.c:5240
 lock_acquire+0x120/0x360 kernel/locking/lockdep.c:5871
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:351 [inline]
 padata_find_next kernel/padata.c:256 [inline]
 padata_reorder kernel/padata.c:309 [inline]
 padata_do_serial+0x73e/0xb80 kernel/padata.c:379
 padata_parallel_worker+0x75/0x1d0 kernel/padata.c:157
 process_one_work kernel/workqueue.c:3239 [inline]
 process_scheduled_works+0xade/0x17b0 kernel/workqueue.c:3322
 worker_thread+0x8a0/0xda0 kernel/workqueue.c:3403
 kthread+0x711/0x8a0 kernel/kthread.c:463
 ret_from_fork+0x3fc/0x770 arch/x86/kernel/process.c:148
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
 </TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [crypto?] possible deadlock in padata_do_serial

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit:    02ffd6f89c50 Merge tag 'bpf-fixes' of git://git.kernel.org..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=124f6934580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=c1f4909b95fa1ed
dashboard link: https://syzkaller.appspot.com/bug?extid=bd936ccd4339cea66e6b
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12e46b12580000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=1598a47c580000

Downloadable assets:
disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/d900f083ada3/non_bootable_disk-02ffd6f8.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/56f3c676fa83/vmlinux-02ffd6f8.xz
kernel image: https://storage.googleapis.com/syzbot-assets/b17e95e57bfa/bzImage-02ffd6f8.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+bd936ccd4339cea66e6b@syzkaller.appspotmail.com

============================================
WARNING: possible recursive locking detected
syzkaller #0 Not tainted
--------------------------------------------
kworker/u32:5/96 is trying to acquire lock:
ffffe8fefc53dbc8 (&pd_list->lock){+...}-{3:3}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffffe8fefc53dbc8 (&pd_list->lock){+...}-{3:3}, at: padata_find_next kernel/padata.c:256 [inline]
ffffe8fefc53dbc8 (&pd_list->lock){+...}-{3:3}, at: padata_reorder kernel/padata.c:309 [inline]
ffffe8fefc53dbc8 (&pd_list->lock){+...}-{3:3}, at: padata_do_serial+0x7bd/0xd20 kernel/padata.c:379

but task is already holding lock:
ffffe8fefc53dc18 (&pd_list->lock){+...}-{3:3}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffffe8fefc53dc18 (&pd_list->lock){+...}-{3:3}, at: padata_reorder kernel/padata.c:300 [inline]
ffffe8fefc53dc18 (&pd_list->lock){+...}-{3:3}, at: padata_do_serial+0x697/0xd20 kernel/padata.c:379

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&pd_list->lock);
  lock(&pd_list->lock);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

3 locks held by kworker/u32:5/96:
 #0: ffff888022495148 ((wq_completion)pdecrypt_parallel){+.+.}-{0:0}, at: process_one_work+0x12a2/0x1b70 kernel/workqueue.c:3211
 #1: ffffc9000167fd10 ((work_completion)(&pw->pw_work)){+.+.}-{0:0}, at: process_one_work+0x929/0x1b70 kernel/workqueue.c:3212
 #2: ffffe8fefc53dc18 (&pd_list->lock){+...}-{3:3}, at: spin_lock include/linux/spinlock.h:351 [inline]
 #2: ffffe8fefc53dc18 (&pd_list->lock){+...}-{3:3}, at: padata_reorder kernel/padata.c:300 [inline]
 #2: ffffe8fefc53dc18 (&pd_list->lock){+...}-{3:3}, at: padata_do_serial+0x697/0xd20 kernel/padata.c:379

stack backtrace:
CPU: 2 UID: 0 PID: 96 Comm: kworker/u32:5 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014
Workqueue: pdecrypt_parallel padata_parallel_worker
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x116/0x1f0 lib/dump_stack.c:120
 print_deadlock_bug+0x1e9/0x240 kernel/locking/lockdep.c:3041
 check_deadlock kernel/locking/lockdep.c:3093 [inline]
 validate_chain kernel/locking/lockdep.c:3895 [inline]
 __lock_acquire+0x1133/0x1ce0 kernel/locking/lockdep.c:5237
 lock_acquire kernel/locking/lockdep.c:5868 [inline]
 lock_acquire+0x179/0x350 kernel/locking/lockdep.c:5825
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:351 [inline]
 padata_find_next kernel/padata.c:256 [inline]
 padata_reorder kernel/padata.c:309 [inline]
 padata_do_serial+0x7bd/0xd20 kernel/padata.c:379
 pcrypt_aead_dec+0x5b/0x70 crypto/pcrypt.c:140
 padata_parallel_worker+0x62/0xb0 kernel/padata.c:157
 process_one_work+0x9cc/0x1b70 kernel/workqueue.c:3236
 process_scheduled_works kernel/workqueue.c:3319 [inline]
 worker_thread+0x6c8/0xf10 kernel/workqueue.c:3400
 kthread+0x3c2/0x780 kernel/kthread.c:463
 ret_from_fork+0x56a/0x730 arch/x86/kernel/process.c:148
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
 </TASK>


---
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

Re: [syzbot] [crypto?] possible deadlock in padata_do_serial

[Tetsuo Handa]

#syz test

diff --git a/kernel/padata.c b/kernel/padata.c
index f4def028c48c..d2dce316e607 100644
--- a/kernel/padata.c
+++ b/kernel/padata.c
@@ -253,7 +253,7 @@ static struct padata_priv *padata_find_next(struct parallel_data *pd, int cpu,
 
 	reorder = per_cpu_ptr(pd->reorder_list, cpu);
 
-	spin_lock(&reorder->lock);
+	spin_lock_nested(&reorder->lock, 1);
 	if (list_empty(&reorder->list))
 		goto notfound;
 

Re: [syzbot] [crypto?] possible deadlock in padata_do_serial

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-by: syzbot+bd936ccd4339cea66e6b@syzkaller.appspotmail.com
Tested-by: syzbot+bd936ccd4339cea66e6b@syzkaller.appspotmail.com

Tested on:

commit:         6146a0f1 Linux 6.18-rc4
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=16836342580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=f009a9a8d50667
dashboard link: https://syzkaller.appspot.com/bug?extid=bd936ccd4339cea66e6b
compiler:       gcc (Debian 12.2.0-14+deb12u1) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=11f3cc2c580000

Note: testing is done by a robot and is best-effort only.

padata: Is padata_find_next() thread-safe?

[Tetsuo Handa]

syzbot is reporting possibility of recursive locking at
https://syzkaller.appspot.com/bug?extid=bd936ccd4339cea66e6b .
If this is a false positive report, the fix will be as simple as

--- a/kernel/padata.c
+++ b/kernel/padata.c
@@ -253,7 +253,7 @@ static struct padata_priv *padata_find_next(struct parallel_data *pd, int cpu,
 
 	reorder = per_cpu_ptr(pd->reorder_list, cpu);
 
-	spin_lock(&reorder->lock);
+	spin_lock_nested(&reorder->lock, 1);
 	if (list_empty(&reorder->list))
 		goto notfound;

. But I don't know if there is a possibility of AB-BA deadlock.

Can a sequence shown below possible? If it is possible, how is calling
padata_find_next() with a spinlock already held by caller introduced by
commit 71203f68c774 ("padata: Fix pd UAF once and for all") thread-safe?

  struct padata_list list[2];

  CPU 0:                         CPU 1:

  spin_lock(&list[0].lock);
                                 spin_lock(&list[1].lock);
  spin_lock_nested(&list[1].lock, 1);
                                 spin_lock_nested(&list[0].lock, 1);
  do_something();
                                 do_something();
  spin_unlock(&list[1].lock);
                                 spin_unlock(&list[0].lock);
  spin_unlock(&list[0].lock);
                                 spin_unlock(&list[1].lock);

Re: padata: Is padata_find_next() thread-safe?

[Herbert Xu]

On Tue, Nov 04, 2025 at 08:44:53PM +0900, Tetsuo Handa wrote:
> syzbot is reporting possibility of recursive locking at
> https://syzkaller.appspot.com/bug?extid=bd936ccd4339cea66e6b .
> If this is a false positive report, the fix will be as simple as

Yes it's a false positive as reorder->lock is never the same as
squeue->serial.lock.

However, they both have the same data type which is why lockdep
is confused.

Please provide a patch that sets the class for one of them to
something different.  For example, change the lockdep class for
reorder->lock using lockdep_set_class and the problem should go
away.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: padata: Is padata_find_next() thread-safe?

[Tetsuo Handa]

On 2025/11/06 18:28, Herbert Xu wrote:
> On Tue, Nov 04, 2025 at 08:44:53PM +0900, Tetsuo Handa wrote:
>> syzbot is reporting possibility of recursive locking at
>> https://syzkaller.appspot.com/bug?extid=bd936ccd4339cea66e6b .
>> If this is a false positive report, the fix will be as simple as
> 
> Yes it's a false positive as reorder->lock is never the same as
> squeue->serial.lock.

OK. But what is about the "Can a sequence shown below possible?" part?

> 
> However, they both have the same data type which is why lockdep
> is confused.
> 
> Please provide a patch that sets the class for one of them to
> something different.  For example, change the lockdep class for
> reorder->lock using lockdep_set_class and the problem should go
> away.

Is using _nested(lock, 1) wrong?

Re: padata: Is padata_find_next() thread-safe?

[Herbert Xu]

On Thu, Nov 06, 2025 at 06:41:19PM +0900, Tetsuo Handa wrote:
>
> OK. But what is about the "Can a sequence shown below possible?" part?

It's a false positive, i.e., it cannot dead-lock as shown.

> Is using _nested(lock, 1) wrong?

That should kill this particular warning but there could still be
similar warnings if you treat these two locks as the same class.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

[PATCH] padata: use different lock_class_key for padata_list lock

[Tetsuo Handa]

syzbot is reporting possibility of deadlock due to calling spin_lock_init()
in __padata_list_init() from both padata_init_reorder_list() and
padata_init_squeues(). This is a false positive, for reorder->lock is never
the same as squeue->serial.lock.

Reported-by: syzbot+bd936ccd4339cea66e6b@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=bd936ccd4339cea66e6b
Suggested-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
---
 kernel/padata.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/padata.c b/kernel/padata.c
index f4def028c48c..465d25020e1e 100644
--- a/kernel/padata.c
+++ b/kernel/padata.c
@@ -529,12 +529,14 @@ static void padata_init_squeues(struct parallel_data *pd)
 /* Initialize per-CPU reorder lists */
 static void padata_init_reorder_list(struct parallel_data *pd)
 {
+	static struct lock_class_key padata_list_key;
 	int cpu;
 	struct padata_list *list;
 
 	for_each_cpu(cpu, pd->cpumask.pcpu) {
 		list = per_cpu_ptr(pd->reorder_list, cpu);
 		__padata_list_init(list);
+		lockdep_set_class(&list->lock, &padata_list_key);
 	}
 }
 
-- 
2.47.3

[PATCH v2] padata: remove __padata_list_init()

[Tetsuo Handa]

syzbot is reporting possibility of deadlock due to sharing lock_class_key
between padata_init_squeues() and padata_init_reorder_list(). This is a
false positive, for these callers initialize different object. Unshare
lock_class_key by embedding __padata_list_init() into these callers.

Reported-by: syzbot+bd936ccd4339cea66e6b@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=bd936ccd4339cea66e6b
Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
---
This version might be easier to understand, for __padata_list_init() is
cheap (which is likely inlined by compiler) and has only two callers.

 kernel/padata.c | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/kernel/padata.c b/kernel/padata.c
index f4def028c48c..aa66d91e20f9 100644
--- a/kernel/padata.c
+++ b/kernel/padata.c
@@ -506,12 +506,6 @@ void __init padata_do_multithreaded(struct padata_mt_job *job)
 	padata_works_free(&works);
 }
 
-static void __padata_list_init(struct padata_list *pd_list)
-{
-	INIT_LIST_HEAD(&pd_list->list);
-	spin_lock_init(&pd_list->lock);
-}
-
 /* Initialize all percpu queues used by serial workers */
 static void padata_init_squeues(struct parallel_data *pd)
 {
@@ -521,7 +515,8 @@ static void padata_init_squeues(struct parallel_data *pd)
 	for_each_cpu(cpu, pd->cpumask.cbcpu) {
 		squeue = per_cpu_ptr(pd->squeue, cpu);
 		squeue->pd = pd;
-		__padata_list_init(&squeue->serial);
+		INIT_LIST_HEAD(&squeue->serial.list);
+		spin_lock_init(&squeue->serial.lock);
 		INIT_WORK(&squeue->work, padata_serial_worker);
 	}
 }
@@ -534,7 +529,8 @@ static void padata_init_reorder_list(struct parallel_data *pd)
 
 	for_each_cpu(cpu, pd->cpumask.pcpu) {
 		list = per_cpu_ptr(pd->reorder_list, cpu);
-		__padata_list_init(list);
+		INIT_LIST_HEAD(&list->list);
+		spin_lock_init(&list->lock);
 	}
 }
 
-- 
2.47.3

Re: [PATCH v2] padata: remove __padata_list_init()

[Herbert Xu]

On Fri, Nov 07, 2025 at 11:49:37PM +0900, Tetsuo Handa wrote:
> syzbot is reporting possibility of deadlock due to sharing lock_class_key
> between padata_init_squeues() and padata_init_reorder_list(). This is a
> false positive, for these callers initialize different object. Unshare
> lock_class_key by embedding __padata_list_init() into these callers.
> 
> Reported-by: syzbot+bd936ccd4339cea66e6b@syzkaller.appspotmail.com
> Closes: https://syzkaller.appspot.com/bug?extid=bd936ccd4339cea66e6b
> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> ---
> This version might be easier to understand, for __padata_list_init() is
> cheap (which is likely inlined by compiler) and has only two callers.
> 
>  kernel/padata.c | 12 ++++--------
>  1 file changed, 4 insertions(+), 8 deletions(-)

This looks much nicer.

Patch applied.  Thanks.
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt