* Weird delay introduced in v6.16-rc only, possible regression
@ 2025-06-29 21:45 Qu Wenruo
2025-07-01 16:30 ` Jens Axboe
0 siblings, 1 reply; 6+ messages in thread
From: Qu Wenruo @ 2025-06-29 21:45 UTC (permalink / raw)
To: linux-block@vger.kernel.org, LKML
Hi,
Recently I'm hitting a very weird delay when doing development inside a
x86_64 VM.
The dmesg shows the delay (10+ sec) between virtio blk and device-mapper:
[ 3.651377] virtio_blk virtio4: 10/0/0 default/read/poll queues
[ 3.653075] virtio_scsi virtio2: 10/0/0 default/read/poll queues
[ 3.670269] virtio_blk virtio4: [vda] 83886080 512-byte logical
blocks (42.9 GB/40.0 GiB)
[ 3.672096] scsi host6: Virtio SCSI HBA
[ 3.708452] vda: vda1 vda2
[ 3.711073] virtio_blk virtio5: 10/0/0 default/read/poll queues
[ 3.729879] virtio_blk virtio5: [vdb] 167772160 512-byte logical
blocks (85.9 GB/80.0 GiB)
[ 3.737535] virtio_blk virtio8: 10/0/0 default/read/poll queues
[ 3.747045] virtio_blk virtio8: [vdc] 83886080 512-byte logical
blocks (42.9 GB/40.0 GiB)
[ 17.453833] device-mapper: uevent: version 1.0.3
[ 17.455689] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28)
initialised: dm-devel@lists.linux.dev
:: performing fsck on '/dev/os/root'
/dev/mapper/os-root: clean, 240299/1048576 files, 3372218/4194304 blocks
:: mounting '/dev/os/root' on real root
[ 17.871671] EXT4-fs (dm-0): mounted filesystem
00a85626-d289-4817-8183-ee828e221f76 r/w with ordered data mode. Quota
mode: none.
The VM is running kernel based on upstream commit 78f4e737a53e ("Merge
tag 'for-6.16/dm-fixes' of
git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm"),
with a lot of extra btrfs patches.
The v6.15 kernel from Archlinux is totally fine without any delay.
The v6.16-rc kernel may have some different configs, but the config is
used for a long long time, way before v6.15, so it looks like it's
something in the v6.16 cycle causing problems.
I can definitely do a bisection, but any clue would be appreciated.
Thanks,
Qu
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Weird delay introduced in v6.16-rc only, possible regression
2025-06-29 21:45 Weird delay introduced in v6.16-rc only, possible regression Qu Wenruo
@ 2025-07-01 16:30 ` Jens Axboe
2025-07-01 22:07 ` Qu Wenruo
0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2025-07-01 16:30 UTC (permalink / raw)
To: Qu Wenruo, linux-block@vger.kernel.org, LKML
On 6/29/25 3:45 PM, Qu Wenruo wrote:
> Hi,
>
> Recently I'm hitting a very weird delay when doing development inside a x86_64 VM.
>
> The dmesg shows the delay (10+ sec) between virtio blk and device-mapper:
>
> [ 3.651377] virtio_blk virtio4: 10/0/0 default/read/poll queues
> [ 3.653075] virtio_scsi virtio2: 10/0/0 default/read/poll queues
> [ 3.670269] virtio_blk virtio4: [vda] 83886080 512-byte logical blocks (42.9 GB/40.0 GiB)
> [ 3.672096] scsi host6: Virtio SCSI HBA
> [ 3.708452] vda: vda1 vda2
> [ 3.711073] virtio_blk virtio5: 10/0/0 default/read/poll queues
> [ 3.729879] virtio_blk virtio5: [vdb] 167772160 512-byte logical blocks (85.9 GB/80.0 GiB)
> [ 3.737535] virtio_blk virtio8: 10/0/0 default/read/poll queues
> [ 3.747045] virtio_blk virtio8: [vdc] 83886080 512-byte logical blocks (42.9 GB/40.0 GiB)
> [ 17.453833] device-mapper: uevent: version 1.0.3
> [ 17.455689] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev
> :: performing fsck on '/dev/os/root'
> /dev/mapper/os-root: clean, 240299/1048576 files, 3372218/4194304 blocks
> :: mounting '/dev/os/root' on real root
> [ 17.871671] EXT4-fs (dm-0): mounted filesystem 00a85626-d289-4817-8183-ee828e221f76 r/w with ordered data mode. Quota mode: none.
>
> The VM is running kernel based on upstream commit 78f4e737a53e ("Merge tag 'for-6.16/dm-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm"), with a lot of extra btrfs patches.
>
>
> The v6.15 kernel from Archlinux is totally fine without any delay.
>
> The v6.16-rc kernel may have some different configs, but the config is used for a long long time, way before v6.15, so it looks like it's something in the v6.16 cycle causing problems.
>
> I can definitely do a bisection, but any clue would be appreciated.
Probably a good idea to go ahead with a bisect to help pin it down.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Weird delay introduced in v6.16-rc only, possible regression
2025-07-01 16:30 ` Jens Axboe
@ 2025-07-01 22:07 ` Qu Wenruo
2025-07-01 22:11 ` Jens Axboe
0 siblings, 1 reply; 6+ messages in thread
From: Qu Wenruo @ 2025-07-01 22:07 UTC (permalink / raw)
To: Jens Axboe, linux-block@vger.kernel.org, LKML
在 2025/7/2 02:00, Jens Axboe 写道:
> On 6/29/25 3:45 PM, Qu Wenruo wrote:
>> Hi,
>>
>> Recently I'm hitting a very weird delay when doing development inside a x86_64 VM.
>>
>> The dmesg shows the delay (10+ sec) between virtio blk and device-mapper:
>>
>> [ 3.651377] virtio_blk virtio4: 10/0/0 default/read/poll queues
>> [ 3.653075] virtio_scsi virtio2: 10/0/0 default/read/poll queues
>> [ 3.670269] virtio_blk virtio4: [vda] 83886080 512-byte logical blocks (42.9 GB/40.0 GiB)
>> [ 3.672096] scsi host6: Virtio SCSI HBA
>> [ 3.708452] vda: vda1 vda2
>> [ 3.711073] virtio_blk virtio5: 10/0/0 default/read/poll queues
>> [ 3.729879] virtio_blk virtio5: [vdb] 167772160 512-byte logical blocks (85.9 GB/80.0 GiB)
>> [ 3.737535] virtio_blk virtio8: 10/0/0 default/read/poll queues
>> [ 3.747045] virtio_blk virtio8: [vdc] 83886080 512-byte logical blocks (42.9 GB/40.0 GiB)
>> [ 17.453833] device-mapper: uevent: version 1.0.3
>> [ 17.455689] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev
>> :: performing fsck on '/dev/os/root'
>> /dev/mapper/os-root: clean, 240299/1048576 files, 3372218/4194304 blocks
>> :: mounting '/dev/os/root' on real root
>> [ 17.871671] EXT4-fs (dm-0): mounted filesystem 00a85626-d289-4817-8183-ee828e221f76 r/w with ordered data mode. Quota mode: none.
>>
>> The VM is running kernel based on upstream commit 78f4e737a53e ("Merge tag 'for-6.16/dm-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm"), with a lot of extra btrfs patches.
>>
>>
>> The v6.15 kernel from Archlinux is totally fine without any delay.
>>
>> The v6.16-rc kernel may have some different configs, but the config is used for a long long time, way before v6.15, so it looks like it's something in the v6.16 cycle causing problems.
>>
>> I can definitely do a bisection, but any clue would be appreciated.
>
> Probably a good idea to go ahead with a bisect to help pin it down.
>
BTW, a little more digging shows it's the `udevadm settle` causing the
long delay in the initramfs.
The rootfs is an ext4 on a LVM lv, so initramfs is required to mount the
rootfs.
So it may not be the block/dm layer causing the problem.
Thanks,
Qu
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Weird delay introduced in v6.16-rc only, possible regression
2025-07-01 22:07 ` Qu Wenruo
@ 2025-07-01 22:11 ` Jens Axboe
2025-07-02 3:52 ` Qu Wenruo
0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2025-07-01 22:11 UTC (permalink / raw)
To: Qu Wenruo, linux-block@vger.kernel.org, LKML
On 7/1/25 4:07 PM, Qu Wenruo wrote:
>
>
> ? 2025/7/2 02:00, Jens Axboe ??:
>> On 6/29/25 3:45 PM, Qu Wenruo wrote:
>>> Hi,
>>>
>>> Recently I'm hitting a very weird delay when doing development inside a x86_64 VM.
>>>
>>> The dmesg shows the delay (10+ sec) between virtio blk and device-mapper:
>>>
>>> [ 3.651377] virtio_blk virtio4: 10/0/0 default/read/poll queues
>>> [ 3.653075] virtio_scsi virtio2: 10/0/0 default/read/poll queues
>>> [ 3.670269] virtio_blk virtio4: [vda] 83886080 512-byte logical blocks (42.9 GB/40.0 GiB)
>>> [ 3.672096] scsi host6: Virtio SCSI HBA
>>> [ 3.708452] vda: vda1 vda2
>>> [ 3.711073] virtio_blk virtio5: 10/0/0 default/read/poll queues
>>> [ 3.729879] virtio_blk virtio5: [vdb] 167772160 512-byte logical blocks (85.9 GB/80.0 GiB)
>>> [ 3.737535] virtio_blk virtio8: 10/0/0 default/read/poll queues
>>> [ 3.747045] virtio_blk virtio8: [vdc] 83886080 512-byte logical blocks (42.9 GB/40.0 GiB)
>>> [ 17.453833] device-mapper: uevent: version 1.0.3
>>> [ 17.455689] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev
>>> :: performing fsck on '/dev/os/root'
>>> /dev/mapper/os-root: clean, 240299/1048576 files, 3372218/4194304 blocks
>>> :: mounting '/dev/os/root' on real root
>>> [ 17.871671] EXT4-fs (dm-0): mounted filesystem 00a85626-d289-4817-8183-ee828e221f76 r/w with ordered data mode. Quota mode: none.
>>>
>>> The VM is running kernel based on upstream commit 78f4e737a53e ("Merge tag 'for-6.16/dm-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm"), with a lot of extra btrfs patches.
>>>
>>>
>>> The v6.15 kernel from Archlinux is totally fine without any delay.
>>>
>>> The v6.16-rc kernel may have some different configs, but the config is used for a long long time, way before v6.15, so it looks like it's something in the v6.16 cycle causing problems.
>>>
>>> I can definitely do a bisection, but any clue would be appreciated.
>>
>> Probably a good idea to go ahead with a bisect to help pin it down.
>>
>
> BTW, a little more digging shows it's the `udevadm settle` causing the long delay in the initramfs.
>
> The rootfs is an ext4 on a LVM lv, so initramfs is required to mount the rootfs.
>
> So it may not be the block/dm layer causing the problem.
Even the more reason to bisect it then, if we don't quite know why it's
slow.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Weird delay introduced in v6.16-rc only, possible regression
2025-07-01 22:11 ` Jens Axboe
@ 2025-07-02 3:52 ` Qu Wenruo
2025-07-02 4:14 ` Qu Wenruo
0 siblings, 1 reply; 6+ messages in thread
From: Qu Wenruo @ 2025-07-02 3:52 UTC (permalink / raw)
To: Jens Axboe, linux-block@vger.kernel.org, LKML
在 2025/7/2 07:41, Jens Axboe 写道:
> On 7/1/25 4:07 PM, Qu Wenruo wrote:
>>
>>
>> ? 2025/7/2 02:00, Jens Axboe ??:
>>> On 6/29/25 3:45 PM, Qu Wenruo wrote:
>>>> Hi,
>>>>
>>>> Recently I'm hitting a very weird delay when doing development inside a x86_64 VM.
>>>>
>>>> The dmesg shows the delay (10+ sec) between virtio blk and device-mapper:
>>>>
>>>> [ 3.651377] virtio_blk virtio4: 10/0/0 default/read/poll queues
>>>> [ 3.653075] virtio_scsi virtio2: 10/0/0 default/read/poll queues
>>>> [ 3.670269] virtio_blk virtio4: [vda] 83886080 512-byte logical blocks (42.9 GB/40.0 GiB)
>>>> [ 3.672096] scsi host6: Virtio SCSI HBA
>>>> [ 3.708452] vda: vda1 vda2
>>>> [ 3.711073] virtio_blk virtio5: 10/0/0 default/read/poll queues
>>>> [ 3.729879] virtio_blk virtio5: [vdb] 167772160 512-byte logical blocks (85.9 GB/80.0 GiB)
>>>> [ 3.737535] virtio_blk virtio8: 10/0/0 default/read/poll queues
>>>> [ 3.747045] virtio_blk virtio8: [vdc] 83886080 512-byte logical blocks (42.9 GB/40.0 GiB)
>>>> [ 17.453833] device-mapper: uevent: version 1.0.3
>>>> [ 17.455689] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev
>>>> :: performing fsck on '/dev/os/root'
>>>> /dev/mapper/os-root: clean, 240299/1048576 files, 3372218/4194304 blocks
>>>> :: mounting '/dev/os/root' on real root
>>>> [ 17.871671] EXT4-fs (dm-0): mounted filesystem 00a85626-d289-4817-8183-ee828e221f76 r/w with ordered data mode. Quota mode: none.
>>>>
>>>> The VM is running kernel based on upstream commit 78f4e737a53e ("Merge tag 'for-6.16/dm-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm"), with a lot of extra btrfs patches.
>>>>
>>>>
>>>> The v6.15 kernel from Archlinux is totally fine without any delay.
>>>>
>>>> The v6.16-rc kernel may have some different configs, but the config is used for a long long time, way before v6.15, so it looks like it's something in the v6.16 cycle causing problems.
>>>>
>>>> I can definitely do a bisection, but any clue would be appreciated.
>>>
>>> Probably a good idea to go ahead with a bisect to help pin it down.
>>>
>>
>> BTW, a little more digging shows it's the `udevadm settle` causing the long delay in the initramfs.
>>
>> The rootfs is an ext4 on a LVM lv, so initramfs is required to mount the rootfs.
>>
>> So it may not be the block/dm layer causing the problem.
>
> Even the more reason to bisect it then, if we don't quite know why it's
> slow.
>
It looks like my memory is blurry, I tried as old as v6.14 (v6.13 failed
to compile using the latest gcc 15.1.1), all the same delay.
Thus I believe it may be some missing kernel config causing the problem.
Let me retry with the base Archlinux kernel config and see what's going
wrong.
Thanks,
Qu
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Weird delay introduced in v6.16-rc only, possible regression
2025-07-02 3:52 ` Qu Wenruo
@ 2025-07-02 4:14 ` Qu Wenruo
0 siblings, 0 replies; 6+ messages in thread
From: Qu Wenruo @ 2025-07-02 4:14 UTC (permalink / raw)
To: Jens Axboe, linux-block@vger.kernel.org, LKML
在 2025/7/2 13:22, Qu Wenruo 写道:
>
>
> 在 2025/7/2 07:41, Jens Axboe 写道:
>> On 7/1/25 4:07 PM, Qu Wenruo wrote:
>>>
>>>
>>> ? 2025/7/2 02:00, Jens Axboe ??:
>>>> On 6/29/25 3:45 PM, Qu Wenruo wrote:
>>>>> Hi,
>>>>>
>>>>> Recently I'm hitting a very weird delay when doing development
>>>>> inside a x86_64 VM.
>>>>>
>>>>> The dmesg shows the delay (10+ sec) between virtio blk and device-
>>>>> mapper:
>>>>>
>>>>> [ 3.651377] virtio_blk virtio4: 10/0/0 default/read/poll queues
>>>>> [ 3.653075] virtio_scsi virtio2: 10/0/0 default/read/poll queues
>>>>> [ 3.670269] virtio_blk virtio4: [vda] 83886080 512-byte logical
>>>>> blocks (42.9 GB/40.0 GiB)
>>>>> [ 3.672096] scsi host6: Virtio SCSI HBA
>>>>> [ 3.708452] vda: vda1 vda2
>>>>> [ 3.711073] virtio_blk virtio5: 10/0/0 default/read/poll queues
>>>>> [ 3.729879] virtio_blk virtio5: [vdb] 167772160 512-byte logical
>>>>> blocks (85.9 GB/80.0 GiB)
>>>>> [ 3.737535] virtio_blk virtio8: 10/0/0 default/read/poll queues
>>>>> [ 3.747045] virtio_blk virtio8: [vdc] 83886080 512-byte logical
>>>>> blocks (42.9 GB/40.0 GiB)
>>>>> [ 17.453833] device-mapper: uevent: version 1.0.3
>>>>> [ 17.455689] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28)
>>>>> initialised: dm-devel@lists.linux.dev
>>>>> :: performing fsck on '/dev/os/root'
>>>>> /dev/mapper/os-root: clean, 240299/1048576 files, 3372218/4194304
>>>>> blocks
>>>>> :: mounting '/dev/os/root' on real root
>>>>> [ 17.871671] EXT4-fs (dm-0): mounted filesystem 00a85626-
>>>>> d289-4817-8183-ee828e221f76 r/w with ordered data mode. Quota mode:
>>>>> none.
>>>>>
>>>>> The VM is running kernel based on upstream commit 78f4e737a53e
>>>>> ("Merge tag 'for-6.16/dm-fixes' of git://git.kernel.org/pub/scm/
>>>>> linux/kernel/git/device-mapper/linux-dm"), with a lot of extra
>>>>> btrfs patches.
>>>>>
>>>>>
>>>>> The v6.15 kernel from Archlinux is totally fine without any delay.
>>>>>
>>>>> The v6.16-rc kernel may have some different configs, but the config
>>>>> is used for a long long time, way before v6.15, so it looks like
>>>>> it's something in the v6.16 cycle causing problems.
>>>>>
>>>>> I can definitely do a bisection, but any clue would be appreciated.
>>>>
>>>> Probably a good idea to go ahead with a bisect to help pin it down.
>>>>
>>>
>>> BTW, a little more digging shows it's the `udevadm settle` causing
>>> the long delay in the initramfs.
>>>
>>> The rootfs is an ext4 on a LVM lv, so initramfs is required to mount
>>> the rootfs.
>>>
>>> So it may not be the block/dm layer causing the problem.
>>
>> Even the more reason to bisect it then, if we don't quite know why it's
>> slow.
>>
>
> It looks like my memory is blurry, I tried as old as v6.14 (v6.13 failed
> to compile using the latest gcc 15.1.1), all the same delay.
>
> Thus I believe it may be some missing kernel config causing the problem.
>
> Let me retry with the base Archlinux kernel config and see what's going
> wrong.
Confirmed, with minimal change/trimming to the Archlinux kernel config,
everything is back fine.
So there must be some config being trimmed and causing the `udevadm
settle` to wait for some sysfs interface that will never come up.
Sorry for the noise, and thankfully it's not a kernel regression.
Thanks,
Qu
>
> Thanks,
> Qu
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-07-02 4:14 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-06-29 21:45 Weird delay introduced in v6.16-rc only, possible regression Qu Wenruo
2025-07-01 16:30 ` Jens Axboe
2025-07-01 22:07 ` Qu Wenruo
2025-07-01 22:11 ` Jens Axboe
2025-07-02 3:52 ` Qu Wenruo
2025-07-02 4:14 ` Qu Wenruo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox