Request For Suggestion: how to handle udevd timeout for bcache registration

Request For Suggestion: how to handle udevd timeout for bcache registration

[Coly Li]

Hi folk,

I want to listen to your suggestion on how to handle the udevd timeout
for bcache registration.

First of all let me introduce the background of this timeout problem.

Now the bcache registration is synchronized, the registering process
will be blocked until the whole registration done. In boot up time, such
registration can be initiated from a bcache udev rule. Normally it won't
be problem, but for very large cached data size there might be a large
internal btree on the cache device. During the registration checking all
the btree nodes may take 50+ minutes as a udev task, it exceeds 180
seconds timeout and udevd will kill it. The killing signal will make
kthread_create() fail during bcache initialization, then the automatic
bcache registration in boot up time will fail.

The above text describes the problem I need to solve: make boot up time
automatic bache registration always success no mater how long it will take.

I know there are several solutions to solve such problem, I do
appreciate if you may share the solution so that I may learn good ideas
from them.

Thank you in advance for the information sharing of my request of
suggestion.


Coly Li

Re: Request For Suggestion: how to handle udevd timeout for bcache registration

[Jens Axboe]

On 4/23/20 5:23 AM, Coly Li wrote:
> Hi folk,
> 
> I want to listen to your suggestion on how to handle the udevd timeout
> for bcache registration.
> 
> First of all let me introduce the background of this timeout problem.
> 
> Now the bcache registration is synchronized, the registering process
> will be blocked until the whole registration done. In boot up time, such
> registration can be initiated from a bcache udev rule. Normally it won't
> be problem, but for very large cached data size there might be a large
> internal btree on the cache device. During the registration checking all
> the btree nodes may take 50+ minutes as a udev task, it exceeds 180
> seconds timeout and udevd will kill it. The killing signal will make
> kthread_create() fail during bcache initialization, then the automatic
> bcache registration in boot up time will fail.
> 
> The above text describes the problem I need to solve: make boot up time
> automatic bache registration always success no mater how long it will take.
> 
> I know there are several solutions to solve such problem, I do
> appreciate if you may share the solution so that I may learn good ideas
> from them.
> 
> Thank you in advance for the information sharing of my request of
> suggestion.

The way I see it, you have only two choices:

1) Make the registration async (or lazy), so that starting the device is
   fast, but the btree verification happens on-demand or in the
   background.

2) Increase udev timeout.

That's about it, I don't think there's any clever tricks to be had, and
I definitely don't want to go down the path of trying to work around the
udev killing in the kernel.

-- 
Jens Axboe

Re: Request For Suggestion: how to handle udevd timeout for bcache registration

[Coly Li]

On 2020/4/23 23:08, Jens Axboe wrote:
> On 4/23/20 5:23 AM, Coly Li wrote:
>> Hi folk,
>>
>> I want to listen to your suggestion on how to handle the udevd timeout
>> for bcache registration.
>>
>> First of all let me introduce the background of this timeout problem.
>>
>> Now the bcache registration is synchronized, the registering process
>> will be blocked until the whole registration done. In boot up time, such
>> registration can be initiated from a bcache udev rule. Normally it won't
>> be problem, but for very large cached data size there might be a large
>> internal btree on the cache device. During the registration checking all
>> the btree nodes may take 50+ minutes as a udev task, it exceeds 180
>> seconds timeout and udevd will kill it. The killing signal will make
>> kthread_create() fail during bcache initialization, then the automatic
>> bcache registration in boot up time will fail.
>>
>> The above text describes the problem I need to solve: make boot up time
>> automatic bache registration always success no mater how long it will take.
>>
>> I know there are several solutions to solve such problem, I do
>> appreciate if you may share the solution so that I may learn good ideas
>> from them.
>>
>> Thank you in advance for the information sharing of my request of
>> suggestion.
> 
> The way I see it, you have only two choices:
> 
> 1) Make the registration async (or lazy), so that starting the device is
>    fast, but the btree verification happens on-demand or in the
>    background.
> 

Yes, this is what I plan to do now, make whole initialization to be
asynchronous.

Currently there are points not clear to me.
- During the boot up time, if a bcache device is listed in /etc/fstab.
How can I block the fs mount step before the bcache device
/dev/bcache<N> shows up. I guess it should be done in systemd, and not
sure whether there is another timeout value.
- When the bcache device registration done and /dev/bcache<N> show up,
if it is listed in /etc/fstab, how to only mount this bcache device only
and not touch other mount points.

udev rules and systemd are both magic to me at this moment. If anybody
may give a hint, or some similar example to learn and understand, it
will be very helpful.

Thanks.

Coly Li

Re: Request For Suggestion: how to handle udevd timeout for bcache registration

[Hannes Reinecke]

On 4/24/20 10:57 AM, Coly Li wrote:
> On 2020/4/23 23:08, Jens Axboe wrote:
>> On 4/23/20 5:23 AM, Coly Li wrote:
>>> Hi folk,
>>>
>>> I want to listen to your suggestion on how to handle the udevd timeout
>>> for bcache registration.
>>>
>>> First of all let me introduce the background of this timeout problem.
>>>
>>> Now the bcache registration is synchronized, the registering process
>>> will be blocked until the whole registration done. In boot up time, such
>>> registration can be initiated from a bcache udev rule. Normally it won't
>>> be problem, but for very large cached data size there might be a large
>>> internal btree on the cache device. During the registration checking all
>>> the btree nodes may take 50+ minutes as a udev task, it exceeds 180
>>> seconds timeout and udevd will kill it. The killing signal will make
>>> kthread_create() fail during bcache initialization, then the automatic
>>> bcache registration in boot up time will fail.
>>>
>>> The above text describes the problem I need to solve: make boot up time
>>> automatic bache registration always success no mater how long it will take.
>>>
>>> I know there are several solutions to solve such problem, I do
>>> appreciate if you may share the solution so that I may learn good ideas
>>> from them.
>>>
>>> Thank you in advance for the information sharing of my request of
>>> suggestion.
>>
>> The way I see it, you have only two choices:
>>
>> 1) Make the registration async (or lazy), so that starting the device is
>>     fast, but the btree verification happens on-demand or in the
>>     background.
>>
> 
> Yes, this is what I plan to do now, make whole initialization to be
> asynchronous.
> 
> Currently there are points not clear to me.
> - During the boot up time, if a bcache device is listed in /etc/fstab.
> How can I block the fs mount step before the bcache device
> /dev/bcache<N> shows up. I guess it should be done in systemd, and not
> sure whether there is another timeout value.
In short: you don't.

Asynchronous initialization means that the device node /dev/bcache<N>
only shows up once registration is complete.

> - When the bcache device registration done and /dev/bcache<N> show up,
> if it is listed in /etc/fstab, how to only mount this bcache device only
> and not touch other mount points.
> 
That's done by systemd, and typically nothing to worry about.
If we can stick with the udev rules and start initialization from an 
udev event everything should 'just work' (tm).

> udev rules and systemd are both magic to me at this moment. If anybody
> may give a hint, or some similar example to learn and understand, it
> will be very helpful.
> 
The proposed sequence of events is:

- backing device generates uevent
-> udev rule triggers bcache initialisation
    -> bcache driver starts initialistion workqueue
    -> bcache driver returns
-> udev event completed

bcache initialisation workqueue starts
    -> bcache driver registers backing device

- cache device generates uevent
-> udev rule triggers bcache initialisation
   -> bcache driver starts initialisation workqueue
   -> bcache driver returns

- bcache initialisation workqueue starts
-> bcache driver registers cache device
   -> bcache driver starts bcache initialisation
     -> bcache driver registers /dev/bcache<N>


With that the uevent handling / bcache userspace initialisation
just submits a workqueue element, and the uevent won't be held off
by an overly long initialisation process.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare@suse.de                               +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer