[bug report] BUG: kernel NULL pointer dereference, address: 0000000000000060

[bug report] BUG: kernel NULL pointer dereference, address: 0000000000000060

[Changhui Zhong]

Hello,

the following kernel panic was triggered by 'ubdsrv  make test T=generic' tests,
please help check and let me know if you need any info/test, thanks.

repo: https://github.com/torvalds/linux.git
branch: master
INFO: HEAD of cloned kernel:
commit d0b3b7b22dfa1f4b515fd3a295b3fd958f9e81af
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date:   Sun Jun 29 13:09:04 2025 -0700

    Linux 6.16-rc4

dmesg log:
[ 3431.347957] BUG: kernel NULL pointer dereference, address: 0000000000000060
[ 3431.355744] #PF: supervisor read access in kernel mode
[ 3431.361484] #PF: error_code(0x0000) - not-present page
[ 3431.367224] PGD 119ffa067 P4D 0
[ 3431.370830] Oops: Oops: 0000 [#1] SMP NOPTI
[ 3431.375503] CPU: 22 UID: 0 PID: 397273 Comm: fio Tainted: G S
           6.16.0-rc4 #1 PREEMPT(voluntary)
[ 3431.386864] Tainted: [S]=CPU_OUT_OF_SPEC
[ 3431.391243] Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW,
BIOS AFE118M-1.32 06/29/2022
[ 3431.400954] RIP: 0010:ublk_queue_rqs+0x7d/0x1c0 [ublk_drv]
[ 3431.407085] Code: 00 00 4c 8b b8 c8 00 00 00 48 63 43 20 48 c1 e0
05 4d 8d 6c 07 30 48 85 d2 0f 84 c1 00 00 00 4c 01 f8 48 8b 7a 10 48
8b 70 40 <48> 8b 4e 60 48 39 4f 60 0f 84 9a 00 00 00 4d 85 e4 0f 84 9f
00 00
[ 3431.428035] RSP: 0018:ff711b900c5379d8 EFLAGS: 00010282
[ 3431.433869] RAX: ff300f05d5c534b0 RBX: ff300f05ea3b2940 RCX: ff300f05d5c53090
[ 3431.441834] RDX: ff300f05d5c534c0 RSI: 0000000000000000 RDI: 0000000000000000
[ 3431.449799] RBP: ff300f05ea3b2800 R08: 0000000000000000 R09: 0000000000000028
[ 3431.457766] R10: ff711b900c537a60 R11: ff300f062256ec20 R12: 0000000000000000
[ 3431.465723] R13: ff300f05d5c534e0 R14: ff711b900c537af0 R15: ff300f05d5c53090
[ 3431.473687] FS:  00007fddab6cd080(0000) GS:ff300f08df445000(0000)
knlGS:0000000000000000
[ 3431.482720] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3431.489137] CR2: 0000000000000060 CR3: 0000000122ff0001 CR4: 0000000000773ef0
[ 3431.497095] PKRU: 55555554
[ 3431.500108] Call Trace:
[ 3431.502829]  <TASK>
[ 3431.505172]  blk_mq_dispatch_queue_requests+0x15a/0x190
[ 3431.511011]  blk_mq_flush_plug_list+0x78/0x190
[ 3431.515971]  ? io_submit_one+0xee/0x370
[ 3431.520257]  __blk_flush_plug+0xf2/0x150
[ 3431.524639]  blk_finish_plug+0x28/0x40
[ 3431.528826]  __x64_sys_io_submit+0xd5/0x1e0
[ 3431.533500]  do_syscall_64+0x7f/0x980
[ 3431.537591]  ? blk_mq_start_request+0x48/0x190
[ 3431.542554]  ? __io_req_task_work_add+0x35/0x1f0
[ 3431.547711]  ? ublk_queue_rqs+0x103/0x1c0 [ublk_drv]
[ 3431.553255]  ? blk_mq_dispatch_queue_requests+0x162/0x190
[ 3431.559283]  ? blk_mq_flush_plug_list+0x78/0x190
[ 3431.564435]  ? io_submit_one+0xee/0x370
[ 3431.568717]  ? __blk_flush_plug+0xf2/0x150
[ 3431.573292]  ? rseq_get_rseq_cs.isra.0+0x16/0x210
[ 3431.578546]  ? rseq_ip_fixup+0x90/0x1d0
[ 3431.582820]  ? __rseq_handle_notify_resume+0x35/0x60
[ 3431.588363]  ? arch_exit_to_user_mode_prepare.isra.0+0x82/0xb0
[ 3431.594877]  ? do_syscall_64+0xb1/0x980
[ 3431.599160]  ? blk_mq_dispatch_queue_requests+0x162/0x190
[ 3431.605189]  ? blk_mq_flush_plug_list+0x78/0x190
[ 3431.610344]  ? io_submit_one+0xee/0x370
[ 3431.614625]  ? __blk_flush_plug+0xf2/0x150
[ 3431.619201]  ? blk_finish_plug+0x28/0x40
[ 3431.623581]  ? __x64_sys_io_submit+0x104/0x1e0
[ 3431.628542]  ? syscall_exit_work+0x108/0x140
[ 3431.633302]  ? clear_bhb_loop+0x50/0xa0
[ 3431.637585]  ? clear_bhb_loop+0x50/0xa0
[ 3431.641860]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 3431.647499] RIP: 0033:0x7fddab7d1a3d
[ 3431.651490] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e
fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24
08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a3 b3 0c 00 f7 d8 64 89
01 48
[ 3431.672448] RSP: 002b:00007ffd378a7a08 EFLAGS: 00000246 ORIG_RAX:
00000000000000d1
[ 3431.680900] RAX: ffffffffffffffda RBX: 00007fddab6ccff8 RCX: 00007fddab7d1a3d
[ 3431.688867] RDX: 000055ff413149c0 RSI: 0000000000000010 RDI: 00007fdda3226000
[ 3431.696824] RBP: 00007fdda3226000 R08: 00007fdda3240000 R09: 0000000000000280
[ 3431.704789] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000010
[ 3431.712755] R13: 0000000000000000 R14: 000055ff413149c0 R15: 000055ff4131b740
[ 3431.720723]  </TASK>
[ 3431.723160] Modules linked in: ublk_drv raid10 raid1 raid0 dm_raid
raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx
raid6_pq nf_tables rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs
lockd grace nfs_localio netfs sunrpc rfkill intel_rapl_msr
intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common
i10nm_edac skx_edac_common nfit libnvdimm x86_pkg_temp_thermal
intel_powerclamp coretemp kvm_intel kvm ipmi_ssif iTCO_wdt irqbypass
dax_hmem rapl cxl_acpi mgag200 cdc_ether isst_if_mbox_pci
iTCO_vendor_support cxl_port intel_cstate isst_if_mmio cxl_core tg3
usbnet intel_th_gth i2c_algo_bit i2c_i801 mei_me ioatdma intel_uncore
mii intel_th_pci einj mei pcspkr acpi_power_meter isst_if_common
i2c_smbus intel_vsec intel_pch_thermal intel_th dca ipmi_si acpi_ipmi
ipmi_devintf ipmi_msghandler acpi_pad sg fuse loop nfnetlink xfs
sd_mod ahci libahci libata ghash_clmulni_intel wmi dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: null_blk]
[ 3431.818860] CR2: 0000000000000060
[ 3431.822559] ---[ end trace 0000000000000000 ]---
[ 3431.856214] RIP: 0010:ublk_queue_rqs+0x7d/0x1c0 [ublk_drv]
[ 3431.862342] Code: 00 00 4c 8b b8 c8 00 00 00 48 63 43 20 48 c1 e0
05 4d 8d 6c 07 30 48 85 d2 0f 84 c1 00 00 00 4c 01 f8 48 8b 7a 10 48
8b 70 40 <48> 8b 4e 60 48 39 4f 60 0f 84 9a 00 00 00 4d 85 e4 0f 84 9f
00 00
[ 3431.883302] RSP: 0018:ff711b900c5379d8 EFLAGS: 00010282
[ 3431.889136] RAX: ff300f05d5c534b0 RBX: ff300f05ea3b2940 RCX: ff300f05d5c53090
[ 3431.897094] RDX: ff300f05d5c534c0 RSI: 0000000000000000 RDI: 0000000000000000
[ 3431.905052] RBP: ff300f05ea3b2800 R08: 0000000000000000 R09: 0000000000000028
[ 3431.913009] R10: ff711b900c537a60 R11: ff300f062256ec20 R12: 0000000000000000
[ 3431.920974] R13: ff300f05d5c534e0 R14: ff711b900c537af0 R15: ff300f05d5c53090
[ 3431.928940] FS:  00007fddab6cd080(0000) GS:ff300f08df445000(0000)
knlGS:0000000000000000
[ 3431.937973] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3431.944388] CR2: 0000000000000060 CR3: 0000000122ff0001 CR4: 0000000000773ef0
[ 3431.952354] PKRU: 55555554
[ 3431.955374] Kernel panic - not syncing: Fatal exception
[ 3431.961292] Kernel Offset: 0xd000000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 3432.002335] ---[ end Kernel panic - not syncing: Fatal exception ]---

Best Regards,
Changhui

Re: [bug report] BUG: kernel NULL pointer dereference, address: 0000000000000060

[Ming Lei]

Hi Changhui,

Thanks for the report!

On Tue, Jul 01, 2025 at 09:55:23AM +0800, Changhui Zhong wrote:
> Hello,
> 
> the following kernel panic was triggered by 'ubdsrv  make test T=generic' tests,
> please help check and let me know if you need any info/test, thanks.
> 
> repo: https://github.com/torvalds/linux.git
> branch: master
> INFO: HEAD of cloned kernel:
> commit d0b3b7b22dfa1f4b515fd3a295b3fd958f9e81af
> Author: Linus Torvalds <torvalds@linux-foundation.org>
> Date:   Sun Jun 29 13:09:04 2025 -0700
> 
>     Linux 6.16-rc4
> 
> dmesg log:
> [ 3431.347957] BUG: kernel NULL pointer dereference, address: 0000000000000060
> [ 3431.355744] #PF: supervisor read access in kernel mode
> [ 3431.361484] #PF: error_code(0x0000) - not-present page
> [ 3431.367224] PGD 119ffa067 P4D 0
> [ 3431.370830] Oops: Oops: 0000 [#1] SMP NOPTI
> [ 3431.375503] CPU: 22 UID: 0 PID: 397273 Comm: fio Tainted: G S
>            6.16.0-rc4 #1 PREEMPT(voluntary)
> [ 3431.386864] Tainted: [S]=CPU_OUT_OF_SPEC
> [ 3431.391243] Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW,
> BIOS AFE118M-1.32 06/29/2022
> [ 3431.400954] RIP: 0010:ublk_queue_rqs+0x7d/0x1c0 [ublk_drv]

It is one regression of commit 524346e9d79f ("ublk: build batch from IOs in same io_ring_ctx and io task").

io->cmd can't be derefered unless the uring cmd is live, and the following patch
should fix the oops:

diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
index c3e3c3b65a6d..99894d712c1f 100644
--- a/drivers/block/ublk_drv.c
+++ b/drivers/block/ublk_drv.c
@@ -1442,15 +1442,14 @@ static void ublk_queue_rqs(struct rq_list *rqlist)
 		struct ublk_queue *this_q = req->mq_hctx->driver_data;
 		struct ublk_io *this_io = &this_q->ios[req->tag];
 
-		if (io && !ublk_belong_to_same_batch(io, this_io) &&
-				!rq_list_empty(&submit_list))
-			ublk_queue_cmd_list(io, &submit_list);
-		io = this_io;
-
-		if (ublk_prep_req(this_q, req, true) == BLK_STS_OK)
+		if (ublk_prep_req(this_q, req, true) == BLK_STS_OK) {
+			if (io && !ublk_belong_to_same_batch(io, this_io) &&
+					!rq_list_empty(&submit_list))
+				ublk_queue_cmd_list(io, &submit_list);
 			rq_list_add_tail(&submit_list, req);
-		else
+		} else
 			rq_list_add_tail(&requeue_list, req);
+		io = this_io;
 	}
 
 	if (!rq_list_empty(&submit_list))


Thanks,
Ming

Re: [bug report] BUG: kernel NULL pointer dereference, address: 0000000000000060

[Changhui Zhong]

On Tue, Jul 1, 2025 at 11:08 AM Ming Lei <ming.lei@redhat.com> wrote:
>
> Hi Changhui,
>
> Thanks for the report!
>
> On Tue, Jul 01, 2025 at 09:55:23AM +0800, Changhui Zhong wrote:
> > Hello,
> >
> > the following kernel panic was triggered by 'ubdsrv  make test T=generic' tests,
> > please help check and let me know if you need any info/test, thanks.
> >
> > repo: https://github.com/torvalds/linux.git
> > branch: master
> > INFO: HEAD of cloned kernel:
> > commit d0b3b7b22dfa1f4b515fd3a295b3fd958f9e81af
> > Author: Linus Torvalds <torvalds@linux-foundation.org>
> > Date:   Sun Jun 29 13:09:04 2025 -0700
> >
> >     Linux 6.16-rc4
> >
> > dmesg log:
> > [ 3431.347957] BUG: kernel NULL pointer dereference, address: 0000000000000060
> > [ 3431.355744] #PF: supervisor read access in kernel mode
> > [ 3431.361484] #PF: error_code(0x0000) - not-present page
> > [ 3431.367224] PGD 119ffa067 P4D 0
> > [ 3431.370830] Oops: Oops: 0000 [#1] SMP NOPTI
> > [ 3431.375503] CPU: 22 UID: 0 PID: 397273 Comm: fio Tainted: G S
> >            6.16.0-rc4 #1 PREEMPT(voluntary)
> > [ 3431.386864] Tainted: [S]=CPU_OUT_OF_SPEC
> > [ 3431.391243] Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW,
> > BIOS AFE118M-1.32 06/29/2022
> > [ 3431.400954] RIP: 0010:ublk_queue_rqs+0x7d/0x1c0 [ublk_drv]
>
> It is one regression of commit 524346e9d79f ("ublk: build batch from IOs in same io_ring_ctx and io task").
>
> io->cmd can't be derefered unless the uring cmd is live, and the following patch
> should fix the oops:
>
> diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
> index c3e3c3b65a6d..99894d712c1f 100644
> --- a/drivers/block/ublk_drv.c
> +++ b/drivers/block/ublk_drv.c
> @@ -1442,15 +1442,14 @@ static void ublk_queue_rqs(struct rq_list *rqlist)
>                 struct ublk_queue *this_q = req->mq_hctx->driver_data;
>                 struct ublk_io *this_io = &this_q->ios[req->tag];
>
> -               if (io && !ublk_belong_to_same_batch(io, this_io) &&
> -                               !rq_list_empty(&submit_list))
> -                       ublk_queue_cmd_list(io, &submit_list);
> -               io = this_io;
> -
> -               if (ublk_prep_req(this_q, req, true) == BLK_STS_OK)
> +               if (ublk_prep_req(this_q, req, true) == BLK_STS_OK) {
> +                       if (io && !ublk_belong_to_same_batch(io, this_io) &&
> +                                       !rq_list_empty(&submit_list))
> +                               ublk_queue_cmd_list(io, &submit_list);
>                         rq_list_add_tail(&submit_list, req);
> -               else
> +               } else
>                         rq_list_add_tail(&requeue_list, req);
> +               io = this_io;
>         }
>
>         if (!rq_list_empty(&submit_list))
>
>
> Thanks,
> Ming
>

Hi，Ming

thanks for fix patch，I ran the test 30 times with your patch and did
not hit this issue again.
I saw you sent a new patch
https://lore.kernel.org/linux-block/20250701072325.1458109-1-ming.lei@redhat.com/T/#u
will re-run the tests for the new patch，

Thanks，