Re: lvm2 deadlock

[Zdenek Kabelac <zdenek.kabelac@gmail.com>]

Dne 03. 06. 24 v 14:56 Jaco Kroon napsal(a):
> Hi,
> 
> Thanks for the insight.  Please refer below.
> 
> On 2024/05/31 14:34, Zdenek Kabelac wrote:
>> Dne 30. 05. 24 v 12:21 Jaco Kroon napsal(a):
>>> Hi,
>>>
>> I'm kind of missing here to see your 'deadlock' scenario from this description.
> Well, stuff blocks, until the cookie is released by using the dmset 
> udevcomplete command, so wrong wording perhaps?
>>
>> Lvm2 takes the VG lock - creates LV - waits for udev till it's finished with 
>> its job and confirms all the udev work with dmsetup udevcomplete.
> 
> So what I understand from this is that udevcomplete ends up never executing?  
> Is there some way of confirming this?

udevcomplete needs  someone to create 'semaphore' for completion in the first 
place.


>>
>> It's also unclear which OS are you using - Debian, Fedora, ???
> 
> Gentoo.
> 
>> Version of your packages ?
> 
> I thought I did provide this:
> 
> Kernel version was 6.4.12 when this hapened, is now 6.9.3.
> 
> crowsnest [12:19:47] /run/lvm # udevadm --version
> 254
> 
> aka systemd-utils-254.10
> 
> lvm2-2.03.22

Since this is most likely your personal build - please provide full output of
'lvm version'  command.

For the 'udev' synchronization, there needs to be '--enable-udev_sync' 
configure option. So let's check which configure/build option were used here.
And also preferably upstream udev rules.

> 
> Thanks for the feedback, what you say makes perfect sense, and the implication 
> is that there are only a few options:
> 
> 1.  Something is resulting in the udev trigger to take longer than three 
> minutes, and the dmsetup udevcomplete never being executed.

systemd simply kills udev worker if takes too long.

However on properly running system, it would be very very unusual to hit these 
timeouts  - you would need to work with thousands of devices....

> 
> This could potentially be due to extremely heavy disk IO, or LVM itself 
> freezing IO.

well reducing the percentage of '/proc/sys/vm/dirty_ration' may possibly help
when your disk system is too slow and you create a very lengthy 'sync' io 
queues...

> I don't see the default value for udev_log from the config. Explicitly set to 
> debug now, but still not seeing anything logged to syslog. Running with udevd 
> --debug, which logs to a ramdisk on /run.  Hopefully (if/when this happens 
> again) that may shed some light.  There is 256GB of RAM available, so as long 
> as the log doesn't grow too quickly should be fine.

A lot of RAM may possibly create a huge amount of dirty pages...

Regards

Zdenek