[bug report] nvme: fio workload hangs on 16m random read

[bug report] nvme: fio workload hangs on 16m random read

[Andreas Hindborg]

Hi All,

My CI system has found an issue for the following fio workload:

root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme-core.ko
root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme.ko poll_queues=1

"fio" "--group_reporting" "--name=default" "--filename=/dev/nvme0n1"
"--readwrite=randread" "--blocksize=16777216" "--direct=1" "--numjobs=1"
"--ioengine=io_uring" "--iodepth=1" "--hipri=1" "--time_based=1"
"--runtime=3000"

fio will hang, some times immediately, some times after a few minutes.

I observed this on v6.14-rc1 and v6.14-rc5. It is not present in v6.13.

Can any of you reproduce this? I was able to reproduce this on real
hardware and in qemu.


Best regards,
Andreas Hindborg

Re: [bug report] nvme: fio workload hangs on 16m random read

[Keith Busch]

On Thu, Mar 06, 2025 at 09:46:37PM +0100, Andreas Hindborg wrote:
> Hi All,
> 
> My CI system has found an issue for the following fio workload:
> 
> root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme-core.ko
> root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme.ko poll_queues=1
> 
> "fio" "--group_reporting" "--name=default" "--filename=/dev/nvme0n1"
> "--readwrite=randread" "--blocksize=16777216" "--direct=1" "--numjobs=1"
> "--ioengine=io_uring" "--iodepth=1" "--hipri=1" "--time_based=1"
> "--runtime=3000"
> 
> fio will hang, some times immediately, some times after a few minutes.
> 
> I observed this on v6.14-rc1 and v6.14-rc5. It is not present in v6.13.
> 
> Can any of you reproduce this? I was able to reproduce this on real
> hardware and in qemu.

Could you try adding this patch to your kernel?

https://lore.kernel.org/io-uring/92b0a330-4782-45e9-8de7-3b90a94208c2@kernel.dk/

Re: [bug report] nvme: fio workload hangs on 16m random read

[Andreas Hindborg]

"Keith Busch" <kbusch@kernel.org> writes:

> On Thu, Mar 06, 2025 at 09:46:37PM +0100, Andreas Hindborg wrote:
>> Hi All,
>>
>> My CI system has found an issue for the following fio workload:
>>
>> root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme-core.ko
>> root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme.ko poll_queues=1
>>
>> "fio" "--group_reporting" "--name=default" "--filename=/dev/nvme0n1"
>> "--readwrite=randread" "--blocksize=16777216" "--direct=1" "--numjobs=1"
>> "--ioengine=io_uring" "--iodepth=1" "--hipri=1" "--time_based=1"
>> "--runtime=3000"
>>
>> fio will hang, some times immediately, some times after a few minutes.
>>
>> I observed this on v6.14-rc1 and v6.14-rc5. It is not present in v6.13.
>>
>> Can any of you reproduce this? I was able to reproduce this on real
>> hardware and in qemu.
>
> Could you try adding this patch to your kernel?
>
> https://lore.kernel.org/io-uring/92b0a330-4782-45e9-8de7-3b90a94208c2@kernel.dk/

Thanks Keith, at first glance that seems to solve the issue. Will
restart my CI with this applied.


Best regards,
Andreas Hindborg