nvme discard issue

nvme discard issue

[Jens Axboe]

Hi,

Ran into this on my test box:

WARNING: CPU: 271 PID: 6837 at drivers/nvme/host/core.c:874 nvme_setup_discard+0x166/0x250 [nvme_core]
Modules linked in: ipmi_ssif nls_ascii nls_cp437 vfat fat joydev wmi_bmof evdev platform_profile dell_smbios dell_wmi_descriptor dcdbas acpi_power_meter ccp button rng_core acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler pcspkr wmi k10temp fuse efi_pstore configfs dm_mod efivarfs autofs4 btrfs xor hid_generic usbhid hid raid6_pq zstd_compress ahci libahci xhci_pci libata xhci_hcd bnxt_en tg3 scsi_mod nvme libphy ptp usbcore scsi_common nvme_core mdio_bus usb_common pps_core sp5100_tco watchdog i2c_piix4 i2c_smbus
CPU: 271 UID: 0 PID: 6837 Comm: kworker/271:1H Not tainted 6.18.0-rc6+ #325 NONE 
Hardware name: Dell Inc. PowerEdge R7625/06444F, BIOS 1.14.1 08/08/2025
Workqueue: xfs-log/nvme0n1p6 xlog_ioend_work
RIP: 0010:nvme_setup_discard+0x166/0x250 [nvme_core]
Code: ac 00 00 00 81 4b 1c 00 10 00 00 48 c1 e8 0c 48 c1 e0 06 48 03 05 2a 4f be bf 48 89 83 a0 00 00 00 31 c0 5b 5d 41 5c 41 5d c3 <0f> 0b b8 00 00 00 80 48 01 f0 0f 82 c1 00 00 00 48 c7 c2 00 00 00
RSP: 0018:ff5a3bf8a8c9bbd0 EFLAGS: 00010202
RAX: 0000000000000000 RBX: ff48de72c98aa000 RCX: 0000000000000ea0
RDX: 00000000000000ec RSI: ff48de78433d1000 RDI: 00000000000000eb
RBP: ff48de72c98aa120 R08: 00000000342cfaf8 R09: 0000000000000008
R10: ff5a3bf8a8c9bbd0 R11: ff48de78433d1000 R12: ff48deba4347a800
R13: 00000000000000eb R14: 0000000000000000 R15: ff5a3bf8a8c9bd60
FS:  0000000000000000(0000) GS:ff48de7ea3419000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe64dffeb10 CR3: 0000008197429006 CR4: 0000000000371ef0
Call Trace:
 <TASK>
 nvme_setup_cmd+0x1aa/0x500 [nvme_core]
 ? __submit_bio+0x94/0x1f0
 nvme_prep_rq+0x39/0x90 [nvme]
 nvme_queue_rqs+0xb5/0x190 [nvme]
 blk_mq_dispatch_queue_requests+0x125/0x140
 ? bio_alloc_bioset+0xc5/0x360
 blk_mq_flush_plug_list+0x54/0x160
 __blk_flush_plug+0xbc/0x100
 ? __blkdev_issue_discard+0x6c/0xd0
 blk_finish_plug+0x1e/0x40
 xfs_discard_extents+0x1a6/0x1e0
 ? __free_frozen_pages+0x403/0x5b0
 xlog_cil_process_committed+0x44/0x60
 xlog_state_do_callback+0x135/0x290
 xlog_ioend_work+0x2c/0x70
 process_one_work+0x13f/0x2a0
 worker_thread+0x2e7/0x420
 ? process_one_work+0x2a0/0x2a0
 kthread+0xd4/0x1c0
 ? kthread_fetch_affinity.isra.0+0x90/0x90
 ? kthread_fetch_affinity.isra.0+0x90/0x90
 ret_from_fork+0x157/0x180
 ? kthread_fetch_affinity.isra.0+0x90/0x90
 ret_from_fork_asm+0x11/0x20
 </TASK>
---[ end trace 0000000000000000 ]---
I/O error, dev nvme0c0n1, sector 429422568 op 0x3:(DISCARD) flags 0x2000000 phys_seg 235 prio class 2
I/O error, dev nvme0c0n1, sector 692087256 op 0x3:(DISCARD) flags 0x2004000 phys_seg 256 prio class 2
I/O error, dev nvme0c0n1, sector 827951352 op 0x3:(DISCARD) flags 0x2000000 phys_seg 239 prio class 2
I/O error, dev nvme0c0n1, sector 811257984 op 0x3:(DISCARD) flags 0x2000000 phys_seg 18 prio class 2
traps: objtool[191612] general protection fault ip:562d82a6bdfc sp:7ffd15cd0858 error:0 in objtool[15dfc,562d82a5a000+1c000]
traps: objtool[200442] general protection fault ip:55bbf5846dfc sp:7fffe542f3f8 error:0 in objtool[15dfc,55bbf5835000+1c000]

which seems to be from:

commit 2516c246d01c23a5f5310e9ac78d9f8aad9b1d0e
Author: Keith Busch <kbusch@kernel.org>
Date:   Fri Nov 14 10:31:45 2025 -0800

    block: consider discard merge last

This was just doing an allmodconfig build, using XFS as per the trace
above. The fs is mounted:

/dev/nvme0n1p6 on /home type xfs (rw,relatime,discard,inode64,logbufs=8,logbsize=32k,noquota)

-- 
Jens Axboe

Re: nvme discard issue

[Keith Busch]

On Tue, Nov 18, 2025 at 07:24:59AM -0700, Jens Axboe wrote:
> commit 2516c246d01c23a5f5310e9ac78d9f8aad9b1d0e
> Author: Keith Busch <kbusch@kernel.org>
> Date:   Fri Nov 14 10:31:45 2025 -0800
> 
>     block: consider discard merge last
> 
> This was just doing an allmodconfig build, using XFS as per the trace
> above. The fs is mounted:

Huh, xfs was the only filesystem I tested, but obviously not enough. So
the segment accounting is off now, I'll take a look.

Re: nvme discard issue

[Jens Axboe]

On 11/18/25 8:22 AM, Keith Busch wrote:
> On Tue, Nov 18, 2025 at 07:24:59AM -0700, Jens Axboe wrote:
>> commit 2516c246d01c23a5f5310e9ac78d9f8aad9b1d0e
>> Author: Keith Busch <kbusch@kernel.org>
>> Date:   Fri Nov 14 10:31:45 2025 -0800
>>
>>     block: consider discard merge last
>>
>> This was just doing an allmodconfig build, using XFS as per the trace
>> above. The fs is mounted:
> 
> Huh, xfs was the only filesystem I tested, but obviously not enough. So
> the segment accounting is off now, I'll take a look.

It's all very strange - reverted the above commit, and ran into other
issues. So may be something else entirely and your commit is fine. My
for-6.19/block branch seems fine (?!?), but merged into master it's not.

-- 
Jens Axboe

Re: nvme discard issue

[Keith Busch]

On Tue, Nov 18, 2025 at 08:28:16AM -0700, Jens Axboe wrote:
> On 11/18/25 8:22 AM, Keith Busch wrote:
> > On Tue, Nov 18, 2025 at 07:24:59AM -0700, Jens Axboe wrote:
> >> commit 2516c246d01c23a5f5310e9ac78d9f8aad9b1d0e
> >> Author: Keith Busch <kbusch@kernel.org>
> >> Date:   Fri Nov 14 10:31:45 2025 -0800
> >>
> >>     block: consider discard merge last
> >>
> >> This was just doing an allmodconfig build, using XFS as per the trace
> >> above. The fs is mounted:
> > 
> > Huh, xfs was the only filesystem I tested, but obviously not enough. So
> > the segment accounting is off now, I'll take a look.
> 
> It's all very strange - reverted the above commit, and ran into other
> issues. So may be something else entirely and your commit is fine. My
> for-6.19/block branch seems fine (?!?), but merged into master it's not.

Interesting, I'll keep testing futher back too.

But I do see a non-trivial problem with my patch, so I think you should
either drop or revert it at this time. For the discard back merge case,
ideally we'd just adjust the bio's bi_size and drop the second bio
completely.

Anyway, sorry, that was my mistake. I was trying to get nvme to hit a
previous bug from merging data-less bio's that nvme had dodged.

Re: nvme discard issue

[Jens Axboe]

On 11/18/25 8:34 AM, Keith Busch wrote:
> On Tue, Nov 18, 2025 at 08:28:16AM -0700, Jens Axboe wrote:
>> On 11/18/25 8:22 AM, Keith Busch wrote:
>>> On Tue, Nov 18, 2025 at 07:24:59AM -0700, Jens Axboe wrote:
>>>> commit 2516c246d01c23a5f5310e9ac78d9f8aad9b1d0e
>>>> Author: Keith Busch <kbusch@kernel.org>
>>>> Date:   Fri Nov 14 10:31:45 2025 -0800
>>>>
>>>>     block: consider discard merge last
>>>>
>>>> This was just doing an allmodconfig build, using XFS as per the trace
>>>> above. The fs is mounted:
>>>
>>> Huh, xfs was the only filesystem I tested, but obviously not enough. So
>>> the segment accounting is off now, I'll take a look.
>>
>> It's all very strange - reverted the above commit, and ran into other
>> issues. So may be something else entirely and your commit is fine. My
>> for-6.19/block branch seems fine (?!?), but merged into master it's not.
> 
> Interesting, I'll keep testing futher back too.

Thanks!

> But I do see a non-trivial problem with my patch, so I think you should
> either drop or revert it at this time. For the discard back merge case,
> ideally we'd just adjust the bio's bi_size and drop the second bio
> completely.
> 
> Anyway, sorry, that was my mistake. I was trying to get nvme to hit a
> previous bug from merging data-less bio's that nvme had dodged.

OK, I'll revert it for now. Done.

-- 
Jens Axboe