Re: Weird delay introduced in v6.16-rc only, possible regression

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

在 2025/7/2 13:22, Qu Wenruo 写道:
> 
> 
> 在 2025/7/2 07:41, Jens Axboe 写道:
>> On 7/1/25 4:07 PM, Qu Wenruo wrote:
>>>
>>>
>>> ? 2025/7/2 02:00, Jens Axboe ??:
>>>> On 6/29/25 3:45 PM, Qu Wenruo wrote:
>>>>> Hi,
>>>>>
>>>>> Recently I'm hitting a very weird delay when doing development 
>>>>> inside a x86_64 VM.
>>>>>
>>>>> The dmesg shows the delay (10+ sec) between virtio blk and device- 
>>>>> mapper:
>>>>>
>>>>> [    3.651377] virtio_blk virtio4: 10/0/0 default/read/poll queues
>>>>> [    3.653075] virtio_scsi virtio2: 10/0/0 default/read/poll queues
>>>>> [    3.670269] virtio_blk virtio4: [vda] 83886080 512-byte logical 
>>>>> blocks (42.9 GB/40.0 GiB)
>>>>> [    3.672096] scsi host6: Virtio SCSI HBA
>>>>> [    3.708452]  vda: vda1 vda2
>>>>> [    3.711073] virtio_blk virtio5: 10/0/0 default/read/poll queues
>>>>> [    3.729879] virtio_blk virtio5: [vdb] 167772160 512-byte logical 
>>>>> blocks (85.9 GB/80.0 GiB)
>>>>> [    3.737535] virtio_blk virtio8: 10/0/0 default/read/poll queues
>>>>> [    3.747045] virtio_blk virtio8: [vdc] 83886080 512-byte logical 
>>>>> blocks (42.9 GB/40.0 GiB)
>>>>> [   17.453833] device-mapper: uevent: version 1.0.3
>>>>> [   17.455689] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) 
>>>>> initialised: dm-devel@lists.linux.dev
>>>>> :: performing fsck on '/dev/os/root'
>>>>> /dev/mapper/os-root: clean, 240299/1048576 files, 3372218/4194304 
>>>>> blocks
>>>>> :: mounting '/dev/os/root' on real root
>>>>> [   17.871671] EXT4-fs (dm-0): mounted filesystem 00a85626- 
>>>>> d289-4817-8183-ee828e221f76 r/w with ordered data mode. Quota mode: 
>>>>> none.
>>>>>
>>>>> The VM is running kernel based on upstream commit 78f4e737a53e 
>>>>> ("Merge tag 'for-6.16/dm-fixes' of git://git.kernel.org/pub/scm/ 
>>>>> linux/kernel/git/device-mapper/linux-dm"), with a lot of extra 
>>>>> btrfs patches.
>>>>>
>>>>>
>>>>> The v6.15 kernel from Archlinux is totally fine without any delay.
>>>>>
>>>>> The v6.16-rc kernel may have some different configs, but the config 
>>>>> is used for a long long time, way before v6.15, so it looks like 
>>>>> it's something in the v6.16 cycle causing problems.
>>>>>
>>>>> I can definitely do a bisection, but any clue would be appreciated.
>>>>
>>>> Probably a good idea to go ahead with a bisect to help pin it down.
>>>>
>>>
>>> BTW, a little more digging shows it's the `udevadm settle` causing 
>>> the long delay in the initramfs.
>>>
>>> The rootfs is an ext4 on a LVM lv, so initramfs is required to mount 
>>> the rootfs.
>>>
>>> So it may not be the block/dm layer causing the problem.
>>
>> Even the more reason to bisect it then, if we don't quite know why it's
>> slow.
>>
> 
> It looks like my memory is blurry, I tried as old as v6.14 (v6.13 failed 
> to compile using the latest gcc 15.1.1), all the same delay.
> 
> Thus I believe it may be some missing kernel config causing the problem.
> 
> Let me retry with the base Archlinux kernel config and see what's going 
> wrong.

Confirmed, with minimal change/trimming to the Archlinux kernel config, 
everything is back fine.

So there must be some config being trimmed and causing the `udevadm 
settle` to wait for some sysfs interface that will never come up.

Sorry for the noise, and thankfully it's not a kernel regression.

Thanks,
Qu

> 
> Thanks,
> Qu