multipath'd bcache device

multipath'd bcache device

[Mitchell Dzurick]

Hello bcache team.

I know this project is done and stable as [0] says, but I have a
question if anyone is around to answer.

Has bcache devices been tested and supported on multipath'd disks? I'm
looking into an Ubuntu bug[1], where these 2 projects are clashing.
I'm wondering if there was any consideration or support for
multipathing when this project was made.

Also, your new project, bcachefs, might be hitting the same scenario.
I haven't had the time to test this though unfortunately.

Thanks for your time,
-Mitch

[0] - https://bcache.evilpiepirate.org/#index4h1
[1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558

Re: multipath'd bcache device

[Coly Li]

> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> 
> Hello bcache team.
> 
> I know this project is done and stable as [0] says, but I have a
> question if anyone is around to answer.
> 
> Has bcache devices been tested and supported on multipath'd disks? I'm
> looking into an Ubuntu bug[1], where these 2 projects are clashing.
> I'm wondering if there was any consideration or support for
> multipathing when this project was made.
> 
> Also, your new project, bcachefs, might be hitting the same scenario.
> I haven't had the time to test this though unfortunately.
> 
> Thanks for your time,
> -Mitch
> 
> [0] - https://bcache.evilpiepirate.org/#index4h1
> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
> 

From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.

It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.

Thanks.

Coly Li

Re: multipath'd bcache device

[Mitchell Dzurick]

Thanks for the reply Coly.

I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
&& 24.10). It should be an issue in Jammy but haven't tested that yet.
The current kernel used in Oracular is 6.8.0-31.31 and the current
kernel used in Noble is 6.8.0-40.40.

Unfortunately you need an account to access pastebin. I can copy that
information elsewhere for you if that would be helpful, but I can also
just gather any extra information you may want from my testbed.

I also have some steps in the bug report to reproduce this issue using kvm.

Lastly, if there's any steps you'd like me to try or look into, I'd be
glad to hear :)

-Mitch

On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
>
>
>
> > 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >
> > Hello bcache team.
> >
> > I know this project is done and stable as [0] says, but I have a
> > question if anyone is around to answer.
> >
> > Has bcache devices been tested and supported on multipath'd disks? I'm
> > looking into an Ubuntu bug[1], where these 2 projects are clashing.
> > I'm wondering if there was any consideration or support for
> > multipathing when this project was made.
> >
> > Also, your new project, bcachefs, might be hitting the same scenario.
> > I haven't had the time to test this though unfortunately.
> >
> > Thanks for your time,
> > -Mitch
> >
> > [0] - https://bcache.evilpiepirate.org/#index4h1
> > [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
> >
>
> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
>
> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
>
> Thanks.
>
> Coly Li

Re: multipath'd bcache device

[Coly Li]

Hi Mitchell,


It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.

If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices. 

BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.

Just FYI.

Coly Li 

> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> 
> Thanks for the reply Coly.
> 
> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
> && 24.10). It should be an issue in Jammy but haven't tested that yet.
> The current kernel used in Oracular is 6.8.0-31.31 and the current
> kernel used in Noble is 6.8.0-40.40.
> 
> Unfortunately you need an account to access pastebin. I can copy that
> information elsewhere for you if that would be helpful, but I can also
> just gather any extra information you may want from my testbed.
> 
> I also have some steps in the bug report to reproduce this issue using kvm.
> 
> Lastly, if there's any steps you'd like me to try or look into, I'd be
> glad to hear :)
> 
> -Mitch
> 
> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
>> 
>> 
>> 
>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>> 
>>> Hello bcache team.
>>> 
>>> I know this project is done and stable as [0] says, but I have a
>>> question if anyone is around to answer.
>>> 
>>> Has bcache devices been tested and supported on multipath'd disks? I'm
>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
>>> I'm wondering if there was any consideration or support for
>>> multipathing when this project was made.
>>> 
>>> Also, your new project, bcachefs, might be hitting the same scenario.
>>> I haven't had the time to test this though unfortunately.
>>> 
>>> Thanks for your time,
>>> -Mitch
>>> 
>>> [0] - https://bcache.evilpiepirate.org/#index4h1
>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
>>> 
>> 
>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
>> 
>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
>> 
>> Thanks.
>> 
>> Coly Li
> 

Re: multipath'd bcache device

[Mitchell Dzurick]

Thanks for the response Coly!

Apologies on my delay, I was out for a bit and then was sick == lots
of catching up to do.

Getting back to this, I do see that we have the bcache KCONFIG value enabled

CONFIG_BCACHE_ASYNC_REGISTRATION=y

I don't believe this is a race condition issue. If I try to reload the
devmap table with multipath-tools well after boot, I see the following
in dmesg:

$ sudo multipath -r
[ 8758.157075] device-mapper: table: 252:3: multipath: error getting
device (-EBUSY)
[ 8758.158039] device-mapper: ioctl: error adding target to table
[ 8758.256206] device-mapper: table: 252:3: multipath: error getting
device (-EBUSY)
[ 8758.256758] device-mapper: ioctl: error adding target to table

FYI - Since I'm not sure if this is a bug in bcache-tools or
multipath-tools, I also made an upstream bug report for
multipath-tools at[0].

If there is anything else you'd like me to try, let me know :)

[0] - https://github.com/opensvc/multipath-tools/issues/96

On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@suse.de> wrote:
>
> Hi Mitchell,
>
>
> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
>
> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
>
> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
>
> Just FYI.
>
> Coly Li
>
> > 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >
> > Thanks for the reply Coly.
> >
> > I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
> > && 24.10). It should be an issue in Jammy but haven't tested that yet.
> > The current kernel used in Oracular is 6.8.0-31.31 and the current
> > kernel used in Noble is 6.8.0-40.40.
> >
> > Unfortunately you need an account to access pastebin. I can copy that
> > information elsewhere for you if that would be helpful, but I can also
> > just gather any extra information you may want from my testbed.
> >
> > I also have some steps in the bug report to reproduce this issue using kvm.
> >
> > Lastly, if there's any steps you'd like me to try or look into, I'd be
> > glad to hear :)
> >
> > -Mitch
> >
> > On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
> >>
> >>
> >>
> >>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>
> >>> Hello bcache team.
> >>>
> >>> I know this project is done and stable as [0] says, but I have a
> >>> question if anyone is around to answer.
> >>>
> >>> Has bcache devices been tested and supported on multipath'd disks? I'm
> >>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
> >>> I'm wondering if there was any consideration or support for
> >>> multipathing when this project was made.
> >>>
> >>> Also, your new project, bcachefs, might be hitting the same scenario.
> >>> I haven't had the time to test this though unfortunately.
> >>>
> >>> Thanks for your time,
> >>> -Mitch
> >>>
> >>> [0] - https://bcache.evilpiepirate.org/#index4h1
> >>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
> >>>
> >>
> >> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
> >>
> >> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
> >>
> >> Thanks.
> >>
> >> Coly Li
> >
>

Re: multipath'd bcache device

[Coly Li]

> 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> 
> Thanks for the response Coly!
> 
> Apologies on my delay, I was out for a bit and then was sick == lots
> of catching up to do.
> 
> Getting back to this, I do see that we have the bcache KCONFIG value enabled
> 
> CONFIG_BCACHE_ASYNC_REGISTRATION=y
> 
> I don't believe this is a race condition issue. If I try to reload the
> devmap table with multipath-tools well after boot, I see the following
> in dmesg:
> 
> $ sudo multipath -r
> [ 8758.157075] device-mapper: table: 252:3: multipath: error getting
> device (-EBUSY)
> [ 8758.158039] device-mapper: ioctl: error adding target to table
> [ 8758.256206] device-mapper: table: 252:3: multipath: error getting
> device (-EBUSY)
> [ 8758.256758] device-mapper: ioctl: error adding target to table

When you see the above kernel message, can you check whether bcache device is initialized already?
Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look.

Thanks.


Coly Li




> 
> FYI - Since I'm not sure if this is a bug in bcache-tools or
> multipath-tools, I also made an upstream bug report for
> multipath-tools at[0].
> 
> If there is anything else you'd like me to try, let me know :)
> 
> [0] - https://github.com/opensvc/multipath-tools/issues/96
> 
> On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@suse.de> wrote:
>> 
>> Hi Mitchell,
>> 
>> 
>> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
>> 
>> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
>> 
>> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
>> 
>> Just FYI.
>> 
>> Coly Li
>> 
>>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>> 
>>> Thanks for the reply Coly.
>>> 
>>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
>>> && 24.10). It should be an issue in Jammy but haven't tested that yet.
>>> The current kernel used in Oracular is 6.8.0-31.31 and the current
>>> kernel used in Noble is 6.8.0-40.40.
>>> 
>>> Unfortunately you need an account to access pastebin. I can copy that
>>> information elsewhere for you if that would be helpful, but I can also
>>> just gather any extra information you may want from my testbed.
>>> 
>>> I also have some steps in the bug report to reproduce this issue using kvm.
>>> 
>>> Lastly, if there's any steps you'd like me to try or look into, I'd be
>>> glad to hear :)
>>> 
>>> -Mitch
>>> 
>>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
>>>> 
>>>> 
>>>> 
>>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>>> 
>>>>> Hello bcache team.
>>>>> 
>>>>> I know this project is done and stable as [0] says, but I have a
>>>>> question if anyone is around to answer.
>>>>> 
>>>>> Has bcache devices been tested and supported on multipath'd disks? I'm
>>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
>>>>> I'm wondering if there was any consideration or support for
>>>>> multipathing when this project was made.
>>>>> 
>>>>> Also, your new project, bcachefs, might be hitting the same scenario.
>>>>> I haven't had the time to test this though unfortunately.
>>>>> 
>>>>> Thanks for your time,
>>>>> -Mitch
>>>>> 
>>>>> [0] - https://bcache.evilpiepirate.org/#index4h1
>>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
>>>>> 
>>>> 
>>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
>>>> 
>>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
>>>> 
>>>> Thanks.
>>>> 
>>>> Coly Li
>>> 
>> 
> 

Re: multipath'd bcache device

[Mitchell Dzurick]

Coly,

Yes, I made that observation in multipath-tools upstream bug[0].

Ultimately the issue is that the bcache-tools udev rule runs
bcache-register on the device and therefore it can't be opened by
multipath tools.

Therefore, the udev rule should wait to run until after multipath has
a chance to make the maps. I'm not sure if there's any complications
with moving the rules later into the boot process though, WDYT?

-Mitch

On Mon, Sep 9, 2024 at 8:56 AM Coly Li <colyli@suse.de> wrote:
>
>
>
> > 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >
> > Thanks for the response Coly!
> >
> > Apologies on my delay, I was out for a bit and then was sick == lots
> > of catching up to do.
> >
> > Getting back to this, I do see that we have the bcache KCONFIG value enabled
> >
> > CONFIG_BCACHE_ASYNC_REGISTRATION=y
> >
> > I don't believe this is a race condition issue. If I try to reload the
> > devmap table with multipath-tools well after boot, I see the following
> > in dmesg:
> >
> > $ sudo multipath -r
> > [ 8758.157075] device-mapper: table: 252:3: multipath: error getting
> > device (-EBUSY)
> > [ 8758.158039] device-mapper: ioctl: error adding target to table
> > [ 8758.256206] device-mapper: table: 252:3: multipath: error getting
> > device (-EBUSY)
> > [ 8758.256758] device-mapper: ioctl: error adding target to table
>
> When you see the above kernel message, can you check whether bcache device is initialized already?
> Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look.
>
> Thanks.
>
>
> Coly Li
>
>
>
>
> >
> > FYI - Since I'm not sure if this is a bug in bcache-tools or
> > multipath-tools, I also made an upstream bug report for
> > multipath-tools at[0].
> >
> > If there is anything else you'd like me to try, let me know :)
> >
> > [0] - https://github.com/opensvc/multipath-tools/issues/96
> >
> > On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@suse.de> wrote:
> >>
> >> Hi Mitchell,
> >>
> >>
> >> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
> >>
> >> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
> >>
> >> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
> >>
> >> Just FYI.
> >>
> >> Coly Li
> >>
> >>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>
> >>> Thanks for the reply Coly.
> >>>
> >>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
> >>> && 24.10). It should be an issue in Jammy but haven't tested that yet.
> >>> The current kernel used in Oracular is 6.8.0-31.31 and the current
> >>> kernel used in Noble is 6.8.0-40.40.
> >>>
> >>> Unfortunately you need an account to access pastebin. I can copy that
> >>> information elsewhere for you if that would be helpful, but I can also
> >>> just gather any extra information you may want from my testbed.
> >>>
> >>> I also have some steps in the bug report to reproduce this issue using kvm.
> >>>
> >>> Lastly, if there's any steps you'd like me to try or look into, I'd be
> >>> glad to hear :)
> >>>
> >>> -Mitch
> >>>
> >>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
> >>>>
> >>>>
> >>>>
> >>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>>
> >>>>> Hello bcache team.
> >>>>>
> >>>>> I know this project is done and stable as [0] says, but I have a
> >>>>> question if anyone is around to answer.
> >>>>>
> >>>>> Has bcache devices been tested and supported on multipath'd disks? I'm
> >>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
> >>>>> I'm wondering if there was any consideration or support for
> >>>>> multipathing when this project was made.
> >>>>>
> >>>>> Also, your new project, bcachefs, might be hitting the same scenario.
> >>>>> I haven't had the time to test this though unfortunately.
> >>>>>
> >>>>> Thanks for your time,
> >>>>> -Mitch
> >>>>>
> >>>>> [0] - https://bcache.evilpiepirate.org/#index4h1
> >>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
> >>>>>
> >>>>
> >>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
> >>>>
> >>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
> >>>>
> >>>> Thanks.
> >>>>
> >>>> Coly Li
> >>>
> >>
> >
>

Re: multipath'd bcache device

[Coly Li]

On my distribution, I see multipath rules is 56-multipath.rules, it should be executed early than bcache one. I am not an systemd and boot expert, but what is the order of your multipath rules?

Coly Li

> 2024年9月20日 19:24，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> 
> Coly,
> 
> Yes, I made that observation in multipath-tools upstream bug[0].
> 
> Ultimately the issue is that the bcache-tools udev rule runs
> bcache-register on the device and therefore it can't be opened by
> multipath tools.
> 
> Therefore, the udev rule should wait to run until after multipath has
> a chance to make the maps. I'm not sure if there's any complications
> with moving the rules later into the boot process though, WDYT?
> 
> -Mitch
> 
> On Mon, Sep 9, 2024 at 8:56 AM Coly Li <colyli@suse.de> wrote:
>> 
>> 
>> 
>>> 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>> 
>>> Thanks for the response Coly!
>>> 
>>> Apologies on my delay, I was out for a bit and then was sick == lots
>>> of catching up to do.
>>> 
>>> Getting back to this, I do see that we have the bcache KCONFIG value enabled
>>> 
>>> CONFIG_BCACHE_ASYNC_REGISTRATION=y
>>> 
>>> I don't believe this is a race condition issue. If I try to reload the
>>> devmap table with multipath-tools well after boot, I see the following
>>> in dmesg:
>>> 
>>> $ sudo multipath -r
>>> [ 8758.157075] device-mapper: table: 252:3: multipath: error getting
>>> device (-EBUSY)
>>> [ 8758.158039] device-mapper: ioctl: error adding target to table
>>> [ 8758.256206] device-mapper: table: 252:3: multipath: error getting
>>> device (-EBUSY)
>>> [ 8758.256758] device-mapper: ioctl: error adding target to table
>> 
>> When you see the above kernel message, can you check whether bcache device is initialized already?
>> Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look.
>> 
>> Thanks.
>> 
>> 
>> Coly Li
>> 
>> 
>> 
>> 
>>> 
>>> FYI - Since I'm not sure if this is a bug in bcache-tools or
>>> multipath-tools, I also made an upstream bug report for
>>> multipath-tools at[0].
>>> 
>>> If there is anything else you'd like me to try, let me know :)
>>> 
>>> [0] - https://github.com/opensvc/multipath-tools/issues/96
>>> 
>>> On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@suse.de> wrote:
>>>> 
>>>> Hi Mitchell,
>>>> 
>>>> 
>>>> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
>>>> 
>>>> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
>>>> 
>>>> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
>>>> 
>>>> Just FYI.
>>>> 
>>>> Coly Li
>>>> 
>>>>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>>> 
>>>>> Thanks for the reply Coly.
>>>>> 
>>>>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
>>>>> && 24.10). It should be an issue in Jammy but haven't tested that yet.
>>>>> The current kernel used in Oracular is 6.8.0-31.31 and the current
>>>>> kernel used in Noble is 6.8.0-40.40.
>>>>> 
>>>>> Unfortunately you need an account to access pastebin. I can copy that
>>>>> information elsewhere for you if that would be helpful, but I can also
>>>>> just gather any extra information you may want from my testbed.
>>>>> 
>>>>> I also have some steps in the bug report to reproduce this issue using kvm.
>>>>> 
>>>>> Lastly, if there's any steps you'd like me to try or look into, I'd be
>>>>> glad to hear :)
>>>>> 
>>>>> -Mitch
>>>>> 
>>>>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>>>>> 
>>>>>>> Hello bcache team.
>>>>>>> 
>>>>>>> I know this project is done and stable as [0] says, but I have a
>>>>>>> question if anyone is around to answer.
>>>>>>> 
>>>>>>> Has bcache devices been tested and supported on multipath'd disks? I'm
>>>>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
>>>>>>> I'm wondering if there was any consideration or support for
>>>>>>> multipathing when this project was made.
>>>>>>> 
>>>>>>> Also, your new project, bcachefs, might be hitting the same scenario.
>>>>>>> I haven't had the time to test this though unfortunately.
>>>>>>> 
>>>>>>> Thanks for your time,
>>>>>>> -Mitch
>>>>>>> 
>>>>>>> [0] - https://bcache.evilpiepirate.org/#index4h1
>>>>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
>>>>>>> 
>>>>>> 
>>>>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
>>>>>> 
>>>>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
>>>>>> 
>>>>>> Thanks.
>>>>>> 
>>>>>> Coly Li
>>>>> 
>>>> 
>>> 
>> 

Re: multipath'd bcache device

[Mitchell Dzurick]

You are right, it seems to be something else than the udev rules. Let
me paste the boot messages real quick.

On Fri, Sep 20, 2024 at 5:42 AM Coly Li <colyli@suse.de> wrote:
>
> On my distribution, I see multipath rules is 56-multipath.rules, it should be executed early than bcache one. I am not an systemd and boot expert, but what is the order of your multipath rules?
>
> Coly Li
>
> > 2024年9月20日 19:24，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >
> > Coly,
> >
> > Yes, I made that observation in multipath-tools upstream bug[0].
> >
> > Ultimately the issue is that the bcache-tools udev rule runs
> > bcache-register on the device and therefore it can't be opened by
> > multipath tools.
> >
> > Therefore, the udev rule should wait to run until after multipath has
> > a chance to make the maps. I'm not sure if there's any complications
> > with moving the rules later into the boot process though, WDYT?
> >
> > -Mitch
> >
> > On Mon, Sep 9, 2024 at 8:56 AM Coly Li <colyli@suse.de> wrote:
> >>
> >>
> >>
> >>> 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>
> >>> Thanks for the response Coly!
> >>>
> >>> Apologies on my delay, I was out for a bit and then was sick == lots
> >>> of catching up to do.
> >>>
> >>> Getting back to this, I do see that we have the bcache KCONFIG value enabled
> >>>
> >>> CONFIG_BCACHE_ASYNC_REGISTRATION=y
> >>>
> >>> I don't believe this is a race condition issue. If I try to reload the
> >>> devmap table with multipath-tools well after boot, I see the following
> >>> in dmesg:
> >>>
> >>> $ sudo multipath -r
> >>> [ 8758.157075] device-mapper: table: 252:3: multipath: error getting
> >>> device (-EBUSY)
> >>> [ 8758.158039] device-mapper: ioctl: error adding target to table
> >>> [ 8758.256206] device-mapper: table: 252:3: multipath: error getting
> >>> device (-EBUSY)
> >>> [ 8758.256758] device-mapper: ioctl: error adding target to table
> >>
> >> When you see the above kernel message, can you check whether bcache device is initialized already?
> >> Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look.
> >>
> >> Thanks.
> >>
> >>
> >> Coly Li
> >>
> >>
> >>
> >>
> >>>
> >>> FYI - Since I'm not sure if this is a bug in bcache-tools or
> >>> multipath-tools, I also made an upstream bug report for
> >>> multipath-tools at[0].
> >>>
> >>> If there is anything else you'd like me to try, let me know :)
> >>>
> >>> [0] - https://github.com/opensvc/multipath-tools/issues/96
> >>>
> >>> On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@suse.de> wrote:
> >>>>
> >>>> Hi Mitchell,
> >>>>
> >>>>
> >>>> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
> >>>>
> >>>> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
> >>>>
> >>>> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
> >>>>
> >>>> Just FYI.
> >>>>
> >>>> Coly Li
> >>>>
> >>>>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>>
> >>>>> Thanks for the reply Coly.
> >>>>>
> >>>>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
> >>>>> && 24.10). It should be an issue in Jammy but haven't tested that yet.
> >>>>> The current kernel used in Oracular is 6.8.0-31.31 and the current
> >>>>> kernel used in Noble is 6.8.0-40.40.
> >>>>>
> >>>>> Unfortunately you need an account to access pastebin. I can copy that
> >>>>> information elsewhere for you if that would be helpful, but I can also
> >>>>> just gather any extra information you may want from my testbed.
> >>>>>
> >>>>> I also have some steps in the bug report to reproduce this issue using kvm.
> >>>>>
> >>>>> Lastly, if there's any steps you'd like me to try or look into, I'd be
> >>>>> glad to hear :)
> >>>>>
> >>>>> -Mitch
> >>>>>
> >>>>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>>>>
> >>>>>>> Hello bcache team.
> >>>>>>>
> >>>>>>> I know this project is done and stable as [0] says, but I have a
> >>>>>>> question if anyone is around to answer.
> >>>>>>>
> >>>>>>> Has bcache devices been tested and supported on multipath'd disks? I'm
> >>>>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
> >>>>>>> I'm wondering if there was any consideration or support for
> >>>>>>> multipathing when this project was made.
> >>>>>>>
> >>>>>>> Also, your new project, bcachefs, might be hitting the same scenario.
> >>>>>>> I haven't had the time to test this though unfortunately.
> >>>>>>>
> >>>>>>> Thanks for your time,
> >>>>>>> -Mitch
> >>>>>>>
> >>>>>>> [0] - https://bcache.evilpiepirate.org/#index4h1
> >>>>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
> >>>>>>>
> >>>>>>
> >>>>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
> >>>>>>
> >>>>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>> Coly Li
> >>>>>
> >>>>
> >>>
> >>
>

Re: multipath'd bcache device

[Mitchell Dzurick]

Coly,

Are you able to view this?

https://gist.github.com/mitchdz/addb44425709fd6df5a6b1b91611b234

-Mitch

On Fri, Sep 20, 2024 at 5:51 AM Mitchell Dzurick
<mitchell.dzurick@canonical.com> wrote:
>
> You are right, it seems to be something else than the udev rules. Let
> me paste the boot messages real quick.
>
> On Fri, Sep 20, 2024 at 5:42 AM Coly Li <colyli@suse.de> wrote:
> >
> > On my distribution, I see multipath rules is 56-multipath.rules, it should be executed early than bcache one. I am not an systemd and boot expert, but what is the order of your multipath rules?
> >
> > Coly Li
> >
> > > 2024年9月20日 19:24，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> > >
> > > Coly,
> > >
> > > Yes, I made that observation in multipath-tools upstream bug[0].
> > >
> > > Ultimately the issue is that the bcache-tools udev rule runs
> > > bcache-register on the device and therefore it can't be opened by
> > > multipath tools.
> > >
> > > Therefore, the udev rule should wait to run until after multipath has
> > > a chance to make the maps. I'm not sure if there's any complications
> > > with moving the rules later into the boot process though, WDYT?
> > >
> > > -Mitch
> > >
> > > On Mon, Sep 9, 2024 at 8:56 AM Coly Li <colyli@suse.de> wrote:
> > >>
> > >>
> > >>
> > >>> 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> > >>>
> > >>> Thanks for the response Coly!
> > >>>
> > >>> Apologies on my delay, I was out for a bit and then was sick == lots
> > >>> of catching up to do.
> > >>>
> > >>> Getting back to this, I do see that we have the bcache KCONFIG value enabled
> > >>>
> > >>> CONFIG_BCACHE_ASYNC_REGISTRATION=y
> > >>>
> > >>> I don't believe this is a race condition issue. If I try to reload the
> > >>> devmap table with multipath-tools well after boot, I see the following
> > >>> in dmesg:
> > >>>
> > >>> $ sudo multipath -r
> > >>> [ 8758.157075] device-mapper: table: 252:3: multipath: error getting
> > >>> device (-EBUSY)
> > >>> [ 8758.158039] device-mapper: ioctl: error adding target to table
> > >>> [ 8758.256206] device-mapper: table: 252:3: multipath: error getting
> > >>> device (-EBUSY)
> > >>> [ 8758.256758] device-mapper: ioctl: error adding target to table
> > >>
> > >> When you see the above kernel message, can you check whether bcache device is initialized already?
> > >> Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look.
> > >>
> > >> Thanks.
> > >>
> > >>
> > >> Coly Li
> > >>
> > >>
> > >>
> > >>
> > >>>
> > >>> FYI - Since I'm not sure if this is a bug in bcache-tools or
> > >>> multipath-tools, I also made an upstream bug report for
> > >>> multipath-tools at[0].
> > >>>
> > >>> If there is anything else you'd like me to try, let me know :)
> > >>>
> > >>> [0] - https://github.com/opensvc/multipath-tools/issues/96
> > >>>
> > >>> On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@suse.de> wrote:
> > >>>>
> > >>>> Hi Mitchell,
> > >>>>
> > >>>>
> > >>>> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
> > >>>>
> > >>>> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
> > >>>>
> > >>>> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
> > >>>>
> > >>>> Just FYI.
> > >>>>
> > >>>> Coly Li
> > >>>>
> > >>>>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> > >>>>>
> > >>>>> Thanks for the reply Coly.
> > >>>>>
> > >>>>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
> > >>>>> && 24.10). It should be an issue in Jammy but haven't tested that yet.
> > >>>>> The current kernel used in Oracular is 6.8.0-31.31 and the current
> > >>>>> kernel used in Noble is 6.8.0-40.40.
> > >>>>>
> > >>>>> Unfortunately you need an account to access pastebin. I can copy that
> > >>>>> information elsewhere for you if that would be helpful, but I can also
> > >>>>> just gather any extra information you may want from my testbed.
> > >>>>>
> > >>>>> I also have some steps in the bug report to reproduce this issue using kvm.
> > >>>>>
> > >>>>> Lastly, if there's any steps you'd like me to try or look into, I'd be
> > >>>>> glad to hear :)
> > >>>>>
> > >>>>> -Mitch
> > >>>>>
> > >>>>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> > >>>>>>>
> > >>>>>>> Hello bcache team.
> > >>>>>>>
> > >>>>>>> I know this project is done and stable as [0] says, but I have a
> > >>>>>>> question if anyone is around to answer.
> > >>>>>>>
> > >>>>>>> Has bcache devices been tested and supported on multipath'd disks? I'm
> > >>>>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
> > >>>>>>> I'm wondering if there was any consideration or support for
> > >>>>>>> multipathing when this project was made.
> > >>>>>>>
> > >>>>>>> Also, your new project, bcachefs, might be hitting the same scenario.
> > >>>>>>> I haven't had the time to test this though unfortunately.
> > >>>>>>>
> > >>>>>>> Thanks for your time,
> > >>>>>>> -Mitch
> > >>>>>>>
> > >>>>>>> [0] - https://bcache.evilpiepirate.org/#index4h1
> > >>>>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
> > >>>>>>>
> > >>>>>>
> > >>>>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
> > >>>>>>
> > >>>>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
> > >>>>>>
> > >>>>>> Thanks.
> > >>>>>>
> > >>>>>> Coly Li
> > >>>>>
> > >>>>
> > >>>
> > >>
> >

Re: multipath'd bcache device

[Coly Li]

Hi Mitchell,

Yes I can see the log.

[ 1.226296] device-mapper: multipath service-time: version 0.3.0 loaded
[ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)
[ 1.277086] device-mapper: ioctl: error adding target to table
[ 1.300407] bcache: bch_journal_replay() journal replay done, 0 keys in 1 entries, seq 1
[ 1.301650] bcache: register_cache() registered cache device sdd
[ 1.301748] bcache: register_cache_worker() error /dev/sdc: fail to register cache device


From the above lines, it seems device mapper starts early, but both dm and bcache encounter a busy device.
Then bcache registered sdd successfully, and not register sdc. But I don’t see other information about how the backing device was registered. 

From the following message, 
[ 4.049395] bcache: register_bcache() error : device already registered
[ 4.268997] bcache: register_cache_worker() error /dev/sdc: fail to register cache device

I assume sdc is the another path of sdd?
From the logs, it seems all bcache behaviors are expected, except for no backing device existed/registered.

Also I see multipath rule might execute before bcache one, but explains “[ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)”. I don’t know how it happens, but this is a suspicious one.

Do you know why multipath failed to get device 252:1 ?  And why I don’t see backing device of bcache here ?

Thanks.

Coly Li

> 2024年9月20日 21:11，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> 
> Coly,
> 
> Are you able to view this?
> 
> https://gist.github.com/mitchdz/addb44425709fd6df5a6b1b91611b234
> 
> -Mitch
> 
> On Fri, Sep 20, 2024 at 5:51 AM Mitchell Dzurick
> <mitchell.dzurick@canonical.com> wrote:
>> 
>> You are right, it seems to be something else than the udev rules. Let
>> me paste the boot messages real quick.
>> 
>> On Fri, Sep 20, 2024 at 5:42 AM Coly Li <colyli@suse.de> wrote:
>>> 
>>> On my distribution, I see multipath rules is 56-multipath.rules, it should be executed early than bcache one. I am not an systemd and boot expert, but what is the order of your multipath rules?
>>> 
>>> Coly Li
>>> 
>>>> 2024年9月20日 19:24，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>> 
>>>> Coly,
>>>> 
>>>> Yes, I made that observation in multipath-tools upstream bug[0].
>>>> 
>>>> Ultimately the issue is that the bcache-tools udev rule runs
>>>> bcache-register on the device and therefore it can't be opened by
>>>> multipath tools.
>>>> 
>>>> Therefore, the udev rule should wait to run until after multipath has
>>>> a chance to make the maps. I'm not sure if there's any complications
>>>> with moving the rules later into the boot process though, WDYT?
>>>> 
>>>> -Mitch
>>>> 
>>>> On Mon, Sep 9, 2024 at 8:56 AM Coly Li <colyli@suse.de> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>>> 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>>>> 
>>>>>> Thanks for the response Coly!
>>>>>> 
>>>>>> Apologies on my delay, I was out for a bit and then was sick == lots
>>>>>> of catching up to do.
>>>>>> 
>>>>>> Getting back to this, I do see that we have the bcache KCONFIG value enabled
>>>>>> 
>>>>>> CONFIG_BCACHE_ASYNC_REGISTRATION=y
>>>>>> 
>>>>>> I don't believe this is a race condition issue. If I try to reload the
>>>>>> devmap table with multipath-tools well after boot, I see the following
>>>>>> in dmesg:
>>>>>> 
>>>>>> $ sudo multipath -r
>>>>>> [ 8758.157075] device-mapper: table: 252:3: multipath: error getting
>>>>>> device (-EBUSY)
>>>>>> [ 8758.158039] device-mapper: ioctl: error adding target to table
>>>>>> [ 8758.256206] device-mapper: table: 252:3: multipath: error getting
>>>>>> device (-EBUSY)
>>>>>> [ 8758.256758] device-mapper: ioctl: error adding target to table
>>>>> 
>>>>> When you see the above kernel message, can you check whether bcache device is initialized already?
>>>>> Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look.
>>>>> 
>>>>> Thanks.
>>>>> 
>>>>> 
>>>>> Coly Li
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> FYI - Since I'm not sure if this is a bug in bcache-tools or
>>>>>> multipath-tools, I also made an upstream bug report for
>>>>>> multipath-tools at[0].
>>>>>> 
>>>>>> If there is anything else you'd like me to try, let me know :)
>>>>>> 
>>>>>> [0] - https://github.com/opensvc/multipath-tools/issues/96
>>>>>> 
>>>>>> On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@suse.de> wrote:
>>>>>>> 
>>>>>>> Hi Mitchell,
>>>>>>> 
>>>>>>> 
>>>>>>> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
>>>>>>> 
>>>>>>> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
>>>>>>> 
>>>>>>> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
>>>>>>> 
>>>>>>> Just FYI.
>>>>>>> 
>>>>>>> Coly Li
>>>>>>> 
>>>>>>>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>>>>>> 
>>>>>>>> Thanks for the reply Coly.
>>>>>>>> 
>>>>>>>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
>>>>>>>> && 24.10). It should be an issue in Jammy but haven't tested that yet.
>>>>>>>> The current kernel used in Oracular is 6.8.0-31.31 and the current
>>>>>>>> kernel used in Noble is 6.8.0-40.40.
>>>>>>>> 
>>>>>>>> Unfortunately you need an account to access pastebin. I can copy that
>>>>>>>> information elsewhere for you if that would be helpful, but I can also
>>>>>>>> just gather any extra information you may want from my testbed.
>>>>>>>> 
>>>>>>>> I also have some steps in the bug report to reproduce this issue using kvm.
>>>>>>>> 
>>>>>>>> Lastly, if there's any steps you'd like me to try or look into, I'd be
>>>>>>>> glad to hear :)
>>>>>>>> 
>>>>>>>> -Mitch
>>>>>>>> 
>>>>>>>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>>>>>>>> 
>>>>>>>>>> Hello bcache team.
>>>>>>>>>> 
>>>>>>>>>> I know this project is done and stable as [0] says, but I have a
>>>>>>>>>> question if anyone is around to answer.
>>>>>>>>>> 
>>>>>>>>>> Has bcache devices been tested and supported on multipath'd disks? I'm
>>>>>>>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
>>>>>>>>>> I'm wondering if there was any consideration or support for
>>>>>>>>>> multipathing when this project was made.
>>>>>>>>>> 
>>>>>>>>>> Also, your new project, bcachefs, might be hitting the same scenario.
>>>>>>>>>> I haven't had the time to test this though unfortunately.
>>>>>>>>>> 
>>>>>>>>>> Thanks for your time,
>>>>>>>>>> -Mitch
>>>>>>>>>> 
>>>>>>>>>> [0] - https://bcache.evilpiepirate.org/#index4h1
>>>>>>>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
>>>>>>>>> 
>>>>>>>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
>>>>>>>>> 
>>>>>>>>> Thanks.
>>>>>>>>> 
>>>>>>>>> Coly Li
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
> 

Re: multipath'd bcache device

[Mitchell Dzurick]

Yes, sdc is another path to sdd. The lsblk command looks like this:

NAME             MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
fd0                2:0    1    4K  0 disk
sda                8:0    0   20G  0 disk
└─mpatha         252:1    0   20G  0 mpath
  ├─mpatha-part1 252:2    0    1M  0 part
  └─mpatha-part2 252:3    0   20G  0 part  /
sdb                8:16   0   20G  0 disk
└─mpatha         252:1    0   20G  0 mpath
  ├─mpatha-part1 252:2    0    1M  0 part
  └─mpatha-part2 252:3    0   20G  0 part  /
sdc                8:32   0    1G  0 disk
└─mpathb         252:0    0    1G  0 mpath
sdd                8:48   0    1G  0 disk
└─mpathb         252:0    0    1G  0 mpath
sr0               11:0    1  2.6G  0 rom

Where I invoke the KVM command like this:
$ kvm -m 2048 -boot c \
    -cdrom ./oracular-live-server-amd64.iso \
    -device virtio-scsi-pci,id=scsi \
    -drive file=image.img,if=none,id=sda,format=raw,file.locking=off \
    -device scsi-hd,drive=sda,serial=0001 \
    -drive if=none,id=sdb,file=image.img,format=raw,file.locking=off \
    -device scsi-hd,drive=sdb,serial=0001 \
    -drive file=image2.img,if=none,id=sdc,format=raw,file.locking=off \
    -device scsi-hd,drive=sdc,serial=0002 \
    -drive if=none,id=sdd,file=image2.img,format=raw,file.locking=off \
    -device scsi-hd,drive=sdd,serial=0002 \
    -netdev user,id=net0,hostfwd=tcp::2222-:22 \
    -device virtio-net-pci,netdev=net0

> Do you know why multipath failed to get device 252:1 ?  And why I don’t see backing device of bcache here ?
I'm not sure at the moment, that's what I'm trying to get to the bottom of.

The one thing that lead me down this path of bcache being involved
with this issue, is simply that by `sudo apt purge bcache-tools` I see
the device is picked up by multipath-tools successfully, so it seems
something in the bcache-tools package is causing this.

Is there any specific thing you'd like me to look for, or a command
you'd like me to run?

I originally assumed that the reason multipath failed was that the
bcache udev rule opens the device file, but that is clearly a
misunderstanding since the multipath udev rules run first, so
_something_ is intereacting with the device early and making it busy,
I'm just not sure what right now.

On Sun, Sep 22, 2024 at 6:55 PM Coly Li <colyli@suse.de> wrote:
>
> Hi Mitchell,
>
> Yes I can see the log.
>
> [ 1.226296] device-mapper: multipath service-time: version 0.3.0 loaded
> [ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)
> [ 1.277086] device-mapper: ioctl: error adding target to table
> [ 1.300407] bcache: bch_journal_replay() journal replay done, 0 keys in 1 entries, seq 1
> [ 1.301650] bcache: register_cache() registered cache device sdd
> [ 1.301748] bcache: register_cache_worker() error /dev/sdc: fail to register cache device
>
>
> From the above lines, it seems device mapper starts early, but both dm and bcache encounter a busy device.
> Then bcache registered sdd successfully, and not register sdc. But I don’t see other information about how the backing device was registered.
>
> From the following message,
> [ 4.049395] bcache: register_bcache() error : device already registered
> [ 4.268997] bcache: register_cache_worker() error /dev/sdc: fail to register cache device
>
> I assume sdc is the another path of sdd?
> From the logs, it seems all bcache behaviors are expected, except for no backing device existed/registered.
>
> Also I see multipath rule might execute before bcache one, but explains “[ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)”. I don’t know how it happens, but this is a suspicious one.
>
> Do you know why multipath failed to get device 252:1 ?  And why I don’t see backing device of bcache here ?
>
> Thanks.
>
> Coly Li
>
> > 2024年9月20日 21:11，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >
> > Coly,
> >
> > Are you able to view this?
> >
> > https://gist.github.com/mitchdz/addb44425709fd6df5a6b1b91611b234
> >
> > -Mitch
> >
> > On Fri, Sep 20, 2024 at 5:51 AM Mitchell Dzurick
> > <mitchell.dzurick@canonical.com> wrote:
> >>
> >> You are right, it seems to be something else than the udev rules. Let
> >> me paste the boot messages real quick.
> >>
> >> On Fri, Sep 20, 2024 at 5:42 AM Coly Li <colyli@suse.de> wrote:
> >>>
> >>> On my distribution, I see multipath rules is 56-multipath.rules, it should be executed early than bcache one. I am not an systemd and boot expert, but what is the order of your multipath rules?
> >>>
> >>> Coly Li
> >>>
> >>>> 2024年9月20日 19:24，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>
> >>>> Coly,
> >>>>
> >>>> Yes, I made that observation in multipath-tools upstream bug[0].
> >>>>
> >>>> Ultimately the issue is that the bcache-tools udev rule runs
> >>>> bcache-register on the device and therefore it can't be opened by
> >>>> multipath tools.
> >>>>
> >>>> Therefore, the udev rule should wait to run until after multipath has
> >>>> a chance to make the maps. I'm not sure if there's any complications
> >>>> with moving the rules later into the boot process though, WDYT?
> >>>>
> >>>> -Mitch
> >>>>
> >>>> On Mon, Sep 9, 2024 at 8:56 AM Coly Li <colyli@suse.de> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>>>
> >>>>>> Thanks for the response Coly!
> >>>>>>
> >>>>>> Apologies on my delay, I was out for a bit and then was sick == lots
> >>>>>> of catching up to do.
> >>>>>>
> >>>>>> Getting back to this, I do see that we have the bcache KCONFIG value enabled
> >>>>>>
> >>>>>> CONFIG_BCACHE_ASYNC_REGISTRATION=y
> >>>>>>
> >>>>>> I don't believe this is a race condition issue. If I try to reload the
> >>>>>> devmap table with multipath-tools well after boot, I see the following
> >>>>>> in dmesg:
> >>>>>>
> >>>>>> $ sudo multipath -r
> >>>>>> [ 8758.157075] device-mapper: table: 252:3: multipath: error getting
> >>>>>> device (-EBUSY)
> >>>>>> [ 8758.158039] device-mapper: ioctl: error adding target to table
> >>>>>> [ 8758.256206] device-mapper: table: 252:3: multipath: error getting
> >>>>>> device (-EBUSY)
> >>>>>> [ 8758.256758] device-mapper: ioctl: error adding target to table
> >>>>>
> >>>>> When you see the above kernel message, can you check whether bcache device is initialized already?
> >>>>> Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>>
> >>>>> Coly Li
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> FYI - Since I'm not sure if this is a bug in bcache-tools or
> >>>>>> multipath-tools, I also made an upstream bug report for
> >>>>>> multipath-tools at[0].
> >>>>>>
> >>>>>> If there is anything else you'd like me to try, let me know :)
> >>>>>>
> >>>>>> [0] - https://github.com/opensvc/multipath-tools/issues/96
> >>>>>>
> >>>>>> On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@suse.de> wrote:
> >>>>>>>
> >>>>>>> Hi Mitchell,
> >>>>>>>
> >>>>>>>
> >>>>>>> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
> >>>>>>>
> >>>>>>> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
> >>>>>>>
> >>>>>>> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
> >>>>>>>
> >>>>>>> Just FYI.
> >>>>>>>
> >>>>>>> Coly Li
> >>>>>>>
> >>>>>>>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>>>>>
> >>>>>>>> Thanks for the reply Coly.
> >>>>>>>>
> >>>>>>>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
> >>>>>>>> && 24.10). It should be an issue in Jammy but haven't tested that yet.
> >>>>>>>> The current kernel used in Oracular is 6.8.0-31.31 and the current
> >>>>>>>> kernel used in Noble is 6.8.0-40.40.
> >>>>>>>>
> >>>>>>>> Unfortunately you need an account to access pastebin. I can copy that
> >>>>>>>> information elsewhere for you if that would be helpful, but I can also
> >>>>>>>> just gather any extra information you may want from my testbed.
> >>>>>>>>
> >>>>>>>> I also have some steps in the bug report to reproduce this issue using kvm.
> >>>>>>>>
> >>>>>>>> Lastly, if there's any steps you'd like me to try or look into, I'd be
> >>>>>>>> glad to hear :)
> >>>>>>>>
> >>>>>>>> -Mitch
> >>>>>>>>
> >>>>>>>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>>>>>>>
> >>>>>>>>>> Hello bcache team.
> >>>>>>>>>>
> >>>>>>>>>> I know this project is done and stable as [0] says, but I have a
> >>>>>>>>>> question if anyone is around to answer.
> >>>>>>>>>>
> >>>>>>>>>> Has bcache devices been tested and supported on multipath'd disks? I'm
> >>>>>>>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
> >>>>>>>>>> I'm wondering if there was any consideration or support for
> >>>>>>>>>> multipathing when this project was made.
> >>>>>>>>>>
> >>>>>>>>>> Also, your new project, bcachefs, might be hitting the same scenario.
> >>>>>>>>>> I haven't had the time to test this though unfortunately.
> >>>>>>>>>>
> >>>>>>>>>> Thanks for your time,
> >>>>>>>>>> -Mitch
> >>>>>>>>>>
> >>>>>>>>>> [0] - https://bcache.evilpiepirate.org/#index4h1
> >>>>>>>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
> >>>>>>>>>
> >>>>>>>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
> >>>>>>>>>
> >>>>>>>>> Thanks.
> >>>>>>>>>
> >>>>>>>>> Coly Li
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >
>

Re: multipath'd bcache device

[Mitchell Dzurick]

Is it perhaps the bcache_allocator interacting with the block device?
I dug around a little bit but couldn't conclude if the allocator was
interacting with the block device, but there might be a better way to
confirm that, which I am not aware of.

On Wed, Sep 25, 2024 at 5:55 PM Mitchell Dzurick
<mitchell.dzurick@canonical.com> wrote:
>
> Yes, sdc is another path to sdd. The lsblk command looks like this:
>
> NAME             MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
> fd0                2:0    1    4K  0 disk
> sda                8:0    0   20G  0 disk
> └─mpatha         252:1    0   20G  0 mpath
>   ├─mpatha-part1 252:2    0    1M  0 part
>   └─mpatha-part2 252:3    0   20G  0 part  /
> sdb                8:16   0   20G  0 disk
> └─mpatha         252:1    0   20G  0 mpath
>   ├─mpatha-part1 252:2    0    1M  0 part
>   └─mpatha-part2 252:3    0   20G  0 part  /
> sdc                8:32   0    1G  0 disk
> └─mpathb         252:0    0    1G  0 mpath
> sdd                8:48   0    1G  0 disk
> └─mpathb         252:0    0    1G  0 mpath
> sr0               11:0    1  2.6G  0 rom
>
> Where I invoke the KVM command like this:
> $ kvm -m 2048 -boot c \
>     -cdrom ./oracular-live-server-amd64.iso \
>     -device virtio-scsi-pci,id=scsi \
>     -drive file=image.img,if=none,id=sda,format=raw,file.locking=off \
>     -device scsi-hd,drive=sda,serial=0001 \
>     -drive if=none,id=sdb,file=image.img,format=raw,file.locking=off \
>     -device scsi-hd,drive=sdb,serial=0001 \
>     -drive file=image2.img,if=none,id=sdc,format=raw,file.locking=off \
>     -device scsi-hd,drive=sdc,serial=0002 \
>     -drive if=none,id=sdd,file=image2.img,format=raw,file.locking=off \
>     -device scsi-hd,drive=sdd,serial=0002 \
>     -netdev user,id=net0,hostfwd=tcp::2222-:22 \
>     -device virtio-net-pci,netdev=net0
>
> > Do you know why multipath failed to get device 252:1 ?  And why I don’t see backing device of bcache here ?
> I'm not sure at the moment, that's what I'm trying to get to the bottom of.
>
> The one thing that lead me down this path of bcache being involved
> with this issue, is simply that by `sudo apt purge bcache-tools` I see
> the device is picked up by multipath-tools successfully, so it seems
> something in the bcache-tools package is causing this.
>
> Is there any specific thing you'd like me to look for, or a command
> you'd like me to run?
>
> I originally assumed that the reason multipath failed was that the
> bcache udev rule opens the device file, but that is clearly a
> misunderstanding since the multipath udev rules run first, so
> _something_ is intereacting with the device early and making it busy,
> I'm just not sure what right now.
>
> On Sun, Sep 22, 2024 at 6:55 PM Coly Li <colyli@suse.de> wrote:
> >
> > Hi Mitchell,
> >
> > Yes I can see the log.
> >
> > [ 1.226296] device-mapper: multipath service-time: version 0.3.0 loaded
> > [ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)
> > [ 1.277086] device-mapper: ioctl: error adding target to table
> > [ 1.300407] bcache: bch_journal_replay() journal replay done, 0 keys in 1 entries, seq 1
> > [ 1.301650] bcache: register_cache() registered cache device sdd
> > [ 1.301748] bcache: register_cache_worker() error /dev/sdc: fail to register cache device
> >
> >
> > From the above lines, it seems device mapper starts early, but both dm and bcache encounter a busy device.
> > Then bcache registered sdd successfully, and not register sdc. But I don’t see other information about how the backing device was registered.
> >
> > From the following message,
> > [ 4.049395] bcache: register_bcache() error : device already registered
> > [ 4.268997] bcache: register_cache_worker() error /dev/sdc: fail to register cache device
> >
> > I assume sdc is the another path of sdd?
> > From the logs, it seems all bcache behaviors are expected, except for no backing device existed/registered.
> >
> > Also I see multipath rule might execute before bcache one, but explains “[ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)”. I don’t know how it happens, but this is a suspicious one.
> >
> > Do you know why multipath failed to get device 252:1 ?  And why I don’t see backing device of bcache here ?
> >
> > Thanks.
> >
> > Coly Li
> >
> > > 2024年9月20日 21:11，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> > >
> > > Coly,
> > >
> > > Are you able to view this?
> > >
> > > https://gist.github.com/mitchdz/addb44425709fd6df5a6b1b91611b234
> > >
> > > -Mitch
> > >
> > > On Fri, Sep 20, 2024 at 5:51 AM Mitchell Dzurick
> > > <mitchell.dzurick@canonical.com> wrote:
> > >>
> > >> You are right, it seems to be something else than the udev rules. Let
> > >> me paste the boot messages real quick.
> > >>
> > >> On Fri, Sep 20, 2024 at 5:42 AM Coly Li <colyli@suse.de> wrote:
> > >>>
> > >>> On my distribution, I see multipath rules is 56-multipath.rules, it should be executed early than bcache one. I am not an systemd and boot expert, but what is the order of your multipath rules?
> > >>>
> > >>> Coly Li
> > >>>
> > >>>> 2024年9月20日 19:24，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> > >>>>
> > >>>> Coly,
> > >>>>
> > >>>> Yes, I made that observation in multipath-tools upstream bug[0].
> > >>>>
> > >>>> Ultimately the issue is that the bcache-tools udev rule runs
> > >>>> bcache-register on the device and therefore it can't be opened by
> > >>>> multipath tools.
> > >>>>
> > >>>> Therefore, the udev rule should wait to run until after multipath has
> > >>>> a chance to make the maps. I'm not sure if there's any complications
> > >>>> with moving the rules later into the boot process though, WDYT?
> > >>>>
> > >>>> -Mitch
> > >>>>
> > >>>> On Mon, Sep 9, 2024 at 8:56 AM Coly Li <colyli@suse.de> wrote:
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>> 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> > >>>>>>
> > >>>>>> Thanks for the response Coly!
> > >>>>>>
> > >>>>>> Apologies on my delay, I was out for a bit and then was sick == lots
> > >>>>>> of catching up to do.
> > >>>>>>
> > >>>>>> Getting back to this, I do see that we have the bcache KCONFIG value enabled
> > >>>>>>
> > >>>>>> CONFIG_BCACHE_ASYNC_REGISTRATION=y
> > >>>>>>
> > >>>>>> I don't believe this is a race condition issue. If I try to reload the
> > >>>>>> devmap table with multipath-tools well after boot, I see the following
> > >>>>>> in dmesg:
> > >>>>>>
> > >>>>>> $ sudo multipath -r
> > >>>>>> [ 8758.157075] device-mapper: table: 252:3: multipath: error getting
> > >>>>>> device (-EBUSY)
> > >>>>>> [ 8758.158039] device-mapper: ioctl: error adding target to table
> > >>>>>> [ 8758.256206] device-mapper: table: 252:3: multipath: error getting
> > >>>>>> device (-EBUSY)
> > >>>>>> [ 8758.256758] device-mapper: ioctl: error adding target to table
> > >>>>>
> > >>>>> When you see the above kernel message, can you check whether bcache device is initialized already?
> > >>>>> Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look.
> > >>>>>
> > >>>>> Thanks.
> > >>>>>
> > >>>>>
> > >>>>> Coly Li
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>>
> > >>>>>> FYI - Since I'm not sure if this is a bug in bcache-tools or
> > >>>>>> multipath-tools, I also made an upstream bug report for
> > >>>>>> multipath-tools at[0].
> > >>>>>>
> > >>>>>> If there is anything else you'd like me to try, let me know :)
> > >>>>>>
> > >>>>>> [0] - https://github.com/opensvc/multipath-tools/issues/96
> > >>>>>>
> > >>>>>> On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@suse.de> wrote:
> > >>>>>>>
> > >>>>>>> Hi Mitchell,
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
> > >>>>>>>
> > >>>>>>> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
> > >>>>>>>
> > >>>>>>> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
> > >>>>>>>
> > >>>>>>> Just FYI.
> > >>>>>>>
> > >>>>>>> Coly Li
> > >>>>>>>
> > >>>>>>>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> > >>>>>>>>
> > >>>>>>>> Thanks for the reply Coly.
> > >>>>>>>>
> > >>>>>>>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
> > >>>>>>>> && 24.10). It should be an issue in Jammy but haven't tested that yet.
> > >>>>>>>> The current kernel used in Oracular is 6.8.0-31.31 and the current
> > >>>>>>>> kernel used in Noble is 6.8.0-40.40.
> > >>>>>>>>
> > >>>>>>>> Unfortunately you need an account to access pastebin. I can copy that
> > >>>>>>>> information elsewhere for you if that would be helpful, but I can also
> > >>>>>>>> just gather any extra information you may want from my testbed.
> > >>>>>>>>
> > >>>>>>>> I also have some steps in the bug report to reproduce this issue using kvm.
> > >>>>>>>>
> > >>>>>>>> Lastly, if there's any steps you'd like me to try or look into, I'd be
> > >>>>>>>> glad to hear :)
> > >>>>>>>>
> > >>>>>>>> -Mitch
> > >>>>>>>>
> > >>>>>>>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> > >>>>>>>>>>
> > >>>>>>>>>> Hello bcache team.
> > >>>>>>>>>>
> > >>>>>>>>>> I know this project is done and stable as [0] says, but I have a
> > >>>>>>>>>> question if anyone is around to answer.
> > >>>>>>>>>>
> > >>>>>>>>>> Has bcache devices been tested and supported on multipath'd disks? I'm
> > >>>>>>>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
> > >>>>>>>>>> I'm wondering if there was any consideration or support for
> > >>>>>>>>>> multipathing when this project was made.
> > >>>>>>>>>>
> > >>>>>>>>>> Also, your new project, bcachefs, might be hitting the same scenario.
> > >>>>>>>>>> I haven't had the time to test this though unfortunately.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for your time,
> > >>>>>>>>>> -Mitch
> > >>>>>>>>>>
> > >>>>>>>>>> [0] - https://bcache.evilpiepirate.org/#index4h1
> > >>>>>>>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
> > >>>>>>>>>
> > >>>>>>>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
> > >>>>>>>>>
> > >>>>>>>>> Thanks.
> > >>>>>>>>>
> > >>>>>>>>> Coly Li
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >
> >

Re: multipath'd bcache device

[Coly Li]

I doubt bcache_allocator involved in this issue. After the cache device registered, an extra refcount will be added to the block device, this is expected.
And Bcache is fine with multipath, because another path has identical UUID, the registration just fails and exits.

Why multipath was not initialized at the first place, this is an open question for me.

Coly Li



> 2024年9月26日 09:15，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> 
> Is it perhaps the bcache_allocator interacting with the block device?
> I dug around a little bit but couldn't conclude if the allocator was
> interacting with the block device, but there might be a better way to
> confirm that, which I am not aware of.
> 
> On Wed, Sep 25, 2024 at 5:55 PM Mitchell Dzurick
> <mitchell.dzurick@canonical.com> wrote:
>> 
>> Yes, sdc is another path to sdd. The lsblk command looks like this:
>> 
>> NAME             MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
>> fd0                2:0    1    4K  0 disk
>> sda                8:0    0   20G  0 disk
>> └─mpatha         252:1    0   20G  0 mpath
>>  ├─mpatha-part1 252:2    0    1M  0 part
>>  └─mpatha-part2 252:3    0   20G  0 part  /
>> sdb                8:16   0   20G  0 disk
>> └─mpatha         252:1    0   20G  0 mpath
>>  ├─mpatha-part1 252:2    0    1M  0 part
>>  └─mpatha-part2 252:3    0   20G  0 part  /
>> sdc                8:32   0    1G  0 disk
>> └─mpathb         252:0    0    1G  0 mpath
>> sdd                8:48   0    1G  0 disk
>> └─mpathb         252:0    0    1G  0 mpath
>> sr0               11:0    1  2.6G  0 rom
>> 
>> Where I invoke the KVM command like this:
>> $ kvm -m 2048 -boot c \
>>    -cdrom ./oracular-live-server-amd64.iso \
>>    -device virtio-scsi-pci,id=scsi \
>>    -drive file=image.img,if=none,id=sda,format=raw,file.locking=off \
>>    -device scsi-hd,drive=sda,serial=0001 \
>>    -drive if=none,id=sdb,file=image.img,format=raw,file.locking=off \
>>    -device scsi-hd,drive=sdb,serial=0001 \
>>    -drive file=image2.img,if=none,id=sdc,format=raw,file.locking=off \
>>    -device scsi-hd,drive=sdc,serial=0002 \
>>    -drive if=none,id=sdd,file=image2.img,format=raw,file.locking=off \
>>    -device scsi-hd,drive=sdd,serial=0002 \
>>    -netdev user,id=net0,hostfwd=tcp::2222-:22 \
>>    -device virtio-net-pci,netdev=net0
>> 
>>> Do you know why multipath failed to get device 252:1 ?  And why I don’t see backing device of bcache here ?
>> I'm not sure at the moment, that's what I'm trying to get to the bottom of.
>> 
>> The one thing that lead me down this path of bcache being involved
>> with this issue, is simply that by `sudo apt purge bcache-tools` I see
>> the device is picked up by multipath-tools successfully, so it seems
>> something in the bcache-tools package is causing this.
>> 
>> Is there any specific thing you'd like me to look for, or a command
>> you'd like me to run?
>> 
>> I originally assumed that the reason multipath failed was that the
>> bcache udev rule opens the device file, but that is clearly a
>> misunderstanding since the multipath udev rules run first, so
>> _something_ is intereacting with the device early and making it busy,
>> I'm just not sure what right now.
>> 
>> On Sun, Sep 22, 2024 at 6:55 PM Coly Li <colyli@suse.de> wrote:
>>> 
>>> Hi Mitchell,
>>> 
>>> Yes I can see the log.
>>> 
>>> [ 1.226296] device-mapper: multipath service-time: version 0.3.0 loaded
>>> [ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)
>>> [ 1.277086] device-mapper: ioctl: error adding target to table
>>> [ 1.300407] bcache: bch_journal_replay() journal replay done, 0 keys in 1 entries, seq 1
>>> [ 1.301650] bcache: register_cache() registered cache device sdd
>>> [ 1.301748] bcache: register_cache_worker() error /dev/sdc: fail to register cache device
>>> 
>>> 
>>> From the above lines, it seems device mapper starts early, but both dm and bcache encounter a busy device.
>>> Then bcache registered sdd successfully, and not register sdc. But I don’t see other information about how the backing device was registered.
>>> 
>>> From the following message,
>>> [ 4.049395] bcache: register_bcache() error : device already registered
>>> [ 4.268997] bcache: register_cache_worker() error /dev/sdc: fail to register cache device
>>> 
>>> I assume sdc is the another path of sdd?
>>> From the logs, it seems all bcache behaviors are expected, except for no backing device existed/registered.
>>> 
>>> Also I see multipath rule might execute before bcache one, but explains “[ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)”. I don’t know how it happens, but this is a suspicious one.
>>> 
>>> Do you know why multipath failed to get device 252:1 ?  And why I don’t see backing device of bcache here ?
>>> 
>>> Thanks.
>>> 
>>> Coly Li
>>> 
>>>> 2024年9月20日 21:11，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>> 
>>>> Coly,
>>>> 
>>>> Are you able to view this?
>>>> 
>>>> https://gist.github.com/mitchdz/addb44425709fd6df5a6b1b91611b234
>>>> 
>>>> -Mitch
>>>> 
>>>> On Fri, Sep 20, 2024 at 5:51 AM Mitchell Dzurick
>>>> <mitchell.dzurick@canonical.com> wrote:
>>>>> 
>>>>> You are right, it seems to be something else than the udev rules. Let
>>>>> me paste the boot messages real quick.
>>>>> 
>>>>> On Fri, Sep 20, 2024 at 5:42 AM Coly Li <colyli@suse.de> wrote:
>>>>>> 
>>>>>> On my distribution, I see multipath rules is 56-multipath.rules, it should be executed early than bcache one. I am not an systemd and boot expert, but what is the order of your multipath rules?
>>>>>> 
>>>>>> Coly Li
>>>>>> 
>>>>>>> 2024年9月20日 19:24，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>>>>> 
>>>>>>> Coly,
>>>>>>> 
>>>>>>> Yes, I made that observation in multipath-tools upstream bug[0].
>>>>>>> 
>>>>>>> Ultimately the issue is that the bcache-tools udev rule runs
>>>>>>> bcache-register on the device and therefore it can't be opened by
>>>>>>> multipath tools.
>>>>>>> 
>>>>>>> Therefore, the udev rule should wait to run until after multipath has
>>>>>>> a chance to make the maps. I'm not sure if there's any complications
>>>>>>> with moving the rules later into the boot process though, WDYT?
>>>>>>> 
>>>>>>> -Mitch
>>>>>>> 
>>>>>>> On Mon, Sep 9, 2024 at 8:56 AM Coly Li <colyli@suse.de> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>>>>>>> 
>>>>>>>>> Thanks for the response Coly!
>>>>>>>>> 
>>>>>>>>> Apologies on my delay, I was out for a bit and then was sick == lots
>>>>>>>>> of catching up to do.
>>>>>>>>> 
>>>>>>>>> Getting back to this, I do see that we have the bcache KCONFIG value enabled
>>>>>>>>> 
>>>>>>>>> CONFIG_BCACHE_ASYNC_REGISTRATION=y
>>>>>>>>> 
>>>>>>>>> I don't believe this is a race condition issue. If I try to reload the
>>>>>>>>> devmap table with multipath-tools well after boot, I see the following
>>>>>>>>> in dmesg:
>>>>>>>>> 
>>>>>>>>> $ sudo multipath -r
>>>>>>>>> [ 8758.157075] device-mapper: table: 252:3: multipath: error getting
>>>>>>>>> device (-EBUSY)
>>>>>>>>> [ 8758.158039] device-mapper: ioctl: error adding target to table
>>>>>>>>> [ 8758.256206] device-mapper: table: 252:3: multipath: error getting
>>>>>>>>> device (-EBUSY)
>>>>>>>>> [ 8758.256758] device-mapper: ioctl: error adding target to table
>>>>>>>> 
>>>>>>>> When you see the above kernel message, can you check whether bcache device is initialized already?
>>>>>>>> Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look.
>>>>>>>> 
>>>>>>>> Thanks.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Coly Li
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> FYI - Since I'm not sure if this is a bug in bcache-tools or
>>>>>>>>> multipath-tools, I also made an upstream bug report for
>>>>>>>>> multipath-tools at[0].
>>>>>>>>> 
>>>>>>>>> If there is anything else you'd like me to try, let me know :)
>>>>>>>>> 
>>>>>>>>> [0] - https://github.com/opensvc/multipath-tools/issues/96
>>>>>>>>> 
>>>>>>>>> On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@suse.de> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Mitchell,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
>>>>>>>>>> 
>>>>>>>>>> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
>>>>>>>>>> 
>>>>>>>>>> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
>>>>>>>>>> 
>>>>>>>>>> Just FYI.
>>>>>>>>>> 
>>>>>>>>>> Coly Li
>>>>>>>>>> 
>>>>>>>>>>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the reply Coly.
>>>>>>>>>>> 
>>>>>>>>>>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
>>>>>>>>>>> && 24.10). It should be an issue in Jammy but haven't tested that yet.
>>>>>>>>>>> The current kernel used in Oracular is 6.8.0-31.31 and the current
>>>>>>>>>>> kernel used in Noble is 6.8.0-40.40.
>>>>>>>>>>> 
>>>>>>>>>>> Unfortunately you need an account to access pastebin. I can copy that
>>>>>>>>>>> information elsewhere for you if that would be helpful, but I can also
>>>>>>>>>>> just gather any extra information you may want from my testbed.
>>>>>>>>>>> 
>>>>>>>>>>> I also have some steps in the bug report to reproduce this issue using kvm.
>>>>>>>>>>> 
>>>>>>>>>>> Lastly, if there's any steps you'd like me to try or look into, I'd be
>>>>>>>>>>> glad to hear :)
>>>>>>>>>>> 
>>>>>>>>>>> -Mitch
>>>>>>>>>>> 
>>>>>>>>>>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello bcache team.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I know this project is done and stable as [0] says, but I have a
>>>>>>>>>>>>> question if anyone is around to answer.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Has bcache devices been tested and supported on multipath'd disks? I'm
>>>>>>>>>>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
>>>>>>>>>>>>> I'm wondering if there was any consideration or support for
>>>>>>>>>>>>> multipathing when this project was made.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Also, your new project, bcachefs, might be hitting the same scenario.
>>>>>>>>>>>>> I haven't had the time to test this though unfortunately.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for your time,
>>>>>>>>>>>>> -Mitch
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [0] - https://bcache.evilpiepirate.org/#index4h1
>>>>>>>>>>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
>>>>>>>>>>>> 
>>>>>>>>>>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>> 
>>>>>>>>>>>> Coly Li
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>>> 
> 

Re: multipath'd bcache device

[Mitchell Dzurick]

There is some more conversation with the multipath-tools bug, but it
seems the udev rules might actually be at play here.

Making the bcache-tools udev rules respect SYSTEMD_READY has
alleviated the problems for me in my testbed. This is a generic
mechanism that other udev rules use (btrfs as an example). Would you
consider looking into including something like this?

To do it is pretty easy, you just add this at the top of the udev rule:

ENV{SYSTEMD_READY}=="0", GOTO="bcache_end"

-Mitch

On Thu, Sep 26, 2024 at 8:55 AM Coly Li <colyli@suse.de> wrote:
>
> I doubt bcache_allocator involved in this issue. After the cache device registered, an extra refcount will be added to the block device, this is expected.
> And Bcache is fine with multipath, because another path has identical UUID, the registration just fails and exits.
>
> Why multipath was not initialized at the first place, this is an open question for me.
>
> Coly Li
>
>
>
> > 2024年9月26日 09:15，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >
> > Is it perhaps the bcache_allocator interacting with the block device?
> > I dug around a little bit but couldn't conclude if the allocator was
> > interacting with the block device, but there might be a better way to
> > confirm that, which I am not aware of.
> >
> > On Wed, Sep 25, 2024 at 5:55 PM Mitchell Dzurick
> > <mitchell.dzurick@canonical.com> wrote:
> >>
> >> Yes, sdc is another path to sdd. The lsblk command looks like this:
> >>
> >> NAME             MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
> >> fd0                2:0    1    4K  0 disk
> >> sda                8:0    0   20G  0 disk
> >> └─mpatha         252:1    0   20G  0 mpath
> >>  ├─mpatha-part1 252:2    0    1M  0 part
> >>  └─mpatha-part2 252:3    0   20G  0 part  /
> >> sdb                8:16   0   20G  0 disk
> >> └─mpatha         252:1    0   20G  0 mpath
> >>  ├─mpatha-part1 252:2    0    1M  0 part
> >>  └─mpatha-part2 252:3    0   20G  0 part  /
> >> sdc                8:32   0    1G  0 disk
> >> └─mpathb         252:0    0    1G  0 mpath
> >> sdd                8:48   0    1G  0 disk
> >> └─mpathb         252:0    0    1G  0 mpath
> >> sr0               11:0    1  2.6G  0 rom
> >>
> >> Where I invoke the KVM command like this:
> >> $ kvm -m 2048 -boot c \
> >>    -cdrom ./oracular-live-server-amd64.iso \
> >>    -device virtio-scsi-pci,id=scsi \
> >>    -drive file=image.img,if=none,id=sda,format=raw,file.locking=off \
> >>    -device scsi-hd,drive=sda,serial=0001 \
> >>    -drive if=none,id=sdb,file=image.img,format=raw,file.locking=off \
> >>    -device scsi-hd,drive=sdb,serial=0001 \
> >>    -drive file=image2.img,if=none,id=sdc,format=raw,file.locking=off \
> >>    -device scsi-hd,drive=sdc,serial=0002 \
> >>    -drive if=none,id=sdd,file=image2.img,format=raw,file.locking=off \
> >>    -device scsi-hd,drive=sdd,serial=0002 \
> >>    -netdev user,id=net0,hostfwd=tcp::2222-:22 \
> >>    -device virtio-net-pci,netdev=net0
> >>
> >>> Do you know why multipath failed to get device 252:1 ?  And why I don’t see backing device of bcache here ?
> >> I'm not sure at the moment, that's what I'm trying to get to the bottom of.
> >>
> >> The one thing that lead me down this path of bcache being involved
> >> with this issue, is simply that by `sudo apt purge bcache-tools` I see
> >> the device is picked up by multipath-tools successfully, so it seems
> >> something in the bcache-tools package is causing this.
> >>
> >> Is there any specific thing you'd like me to look for, or a command
> >> you'd like me to run?
> >>
> >> I originally assumed that the reason multipath failed was that the
> >> bcache udev rule opens the device file, but that is clearly a
> >> misunderstanding since the multipath udev rules run first, so
> >> _something_ is intereacting with the device early and making it busy,
> >> I'm just not sure what right now.
> >>
> >> On Sun, Sep 22, 2024 at 6:55 PM Coly Li <colyli@suse.de> wrote:
> >>>
> >>> Hi Mitchell,
> >>>
> >>> Yes I can see the log.
> >>>
> >>> [ 1.226296] device-mapper: multipath service-time: version 0.3.0 loaded
> >>> [ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)
> >>> [ 1.277086] device-mapper: ioctl: error adding target to table
> >>> [ 1.300407] bcache: bch_journal_replay() journal replay done, 0 keys in 1 entries, seq 1
> >>> [ 1.301650] bcache: register_cache() registered cache device sdd
> >>> [ 1.301748] bcache: register_cache_worker() error /dev/sdc: fail to register cache device
> >>>
> >>>
> >>> From the above lines, it seems device mapper starts early, but both dm and bcache encounter a busy device.
> >>> Then bcache registered sdd successfully, and not register sdc. But I don’t see other information about how the backing device was registered.
> >>>
> >>> From the following message,
> >>> [ 4.049395] bcache: register_bcache() error : device already registered
> >>> [ 4.268997] bcache: register_cache_worker() error /dev/sdc: fail to register cache device
> >>>
> >>> I assume sdc is the another path of sdd?
> >>> From the logs, it seems all bcache behaviors are expected, except for no backing device existed/registered.
> >>>
> >>> Also I see multipath rule might execute before bcache one, but explains “[ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)”. I don’t know how it happens, but this is a suspicious one.
> >>>
> >>> Do you know why multipath failed to get device 252:1 ?  And why I don’t see backing device of bcache here ?
> >>>
> >>> Thanks.
> >>>
> >>> Coly Li
> >>>
> >>>> 2024年9月20日 21:11，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>
> >>>> Coly,
> >>>>
> >>>> Are you able to view this?
> >>>>
> >>>> https://gist.github.com/mitchdz/addb44425709fd6df5a6b1b91611b234
> >>>>
> >>>> -Mitch
> >>>>
> >>>> On Fri, Sep 20, 2024 at 5:51 AM Mitchell Dzurick
> >>>> <mitchell.dzurick@canonical.com> wrote:
> >>>>>
> >>>>> You are right, it seems to be something else than the udev rules. Let
> >>>>> me paste the boot messages real quick.
> >>>>>
> >>>>> On Fri, Sep 20, 2024 at 5:42 AM Coly Li <colyli@suse.de> wrote:
> >>>>>>
> >>>>>> On my distribution, I see multipath rules is 56-multipath.rules, it should be executed early than bcache one. I am not an systemd and boot expert, but what is the order of your multipath rules?
> >>>>>>
> >>>>>> Coly Li
> >>>>>>
> >>>>>>> 2024年9月20日 19:24，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>>>>
> >>>>>>> Coly,
> >>>>>>>
> >>>>>>> Yes, I made that observation in multipath-tools upstream bug[0].
> >>>>>>>
> >>>>>>> Ultimately the issue is that the bcache-tools udev rule runs
> >>>>>>> bcache-register on the device and therefore it can't be opened by
> >>>>>>> multipath tools.
> >>>>>>>
> >>>>>>> Therefore, the udev rule should wait to run until after multipath has
> >>>>>>> a chance to make the maps. I'm not sure if there's any complications
> >>>>>>> with moving the rules later into the boot process though, WDYT?
> >>>>>>>
> >>>>>>> -Mitch
> >>>>>>>
> >>>>>>> On Mon, Sep 9, 2024 at 8:56 AM Coly Li <colyli@suse.de> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>>>>>>
> >>>>>>>>> Thanks for the response Coly!
> >>>>>>>>>
> >>>>>>>>> Apologies on my delay, I was out for a bit and then was sick == lots
> >>>>>>>>> of catching up to do.
> >>>>>>>>>
> >>>>>>>>> Getting back to this, I do see that we have the bcache KCONFIG value enabled
> >>>>>>>>>
> >>>>>>>>> CONFIG_BCACHE_ASYNC_REGISTRATION=y
> >>>>>>>>>
> >>>>>>>>> I don't believe this is a race condition issue. If I try to reload the
> >>>>>>>>> devmap table with multipath-tools well after boot, I see the following
> >>>>>>>>> in dmesg:
> >>>>>>>>>
> >>>>>>>>> $ sudo multipath -r
> >>>>>>>>> [ 8758.157075] device-mapper: table: 252:3: multipath: error getting
> >>>>>>>>> device (-EBUSY)
> >>>>>>>>> [ 8758.158039] device-mapper: ioctl: error adding target to table
> >>>>>>>>> [ 8758.256206] device-mapper: table: 252:3: multipath: error getting
> >>>>>>>>> device (-EBUSY)
> >>>>>>>>> [ 8758.256758] device-mapper: ioctl: error adding target to table
> >>>>>>>>
> >>>>>>>> When you see the above kernel message, can you check whether bcache device is initialized already?
> >>>>>>>> Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look.
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Coly Li
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> FYI - Since I'm not sure if this is a bug in bcache-tools or
> >>>>>>>>> multipath-tools, I also made an upstream bug report for
> >>>>>>>>> multipath-tools at[0].
> >>>>>>>>>
> >>>>>>>>> If there is anything else you'd like me to try, let me know :)
> >>>>>>>>>
> >>>>>>>>> [0] - https://github.com/opensvc/multipath-tools/issues/96
> >>>>>>>>>
> >>>>>>>>> On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@suse.de> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi Mitchell,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
> >>>>>>>>>>
> >>>>>>>>>> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
> >>>>>>>>>>
> >>>>>>>>>> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
> >>>>>>>>>>
> >>>>>>>>>> Just FYI.
> >>>>>>>>>>
> >>>>>>>>>> Coly Li
> >>>>>>>>>>
> >>>>>>>>>>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the reply Coly.
> >>>>>>>>>>>
> >>>>>>>>>>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
> >>>>>>>>>>> && 24.10). It should be an issue in Jammy but haven't tested that yet.
> >>>>>>>>>>> The current kernel used in Oracular is 6.8.0-31.31 and the current
> >>>>>>>>>>> kernel used in Noble is 6.8.0-40.40.
> >>>>>>>>>>>
> >>>>>>>>>>> Unfortunately you need an account to access pastebin. I can copy that
> >>>>>>>>>>> information elsewhere for you if that would be helpful, but I can also
> >>>>>>>>>>> just gather any extra information you may want from my testbed.
> >>>>>>>>>>>
> >>>>>>>>>>> I also have some steps in the bug report to reproduce this issue using kvm.
> >>>>>>>>>>>
> >>>>>>>>>>> Lastly, if there's any steps you'd like me to try or look into, I'd be
> >>>>>>>>>>> glad to hear :)
> >>>>>>>>>>>
> >>>>>>>>>>> -Mitch
> >>>>>>>>>>>
> >>>>>>>>>>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@suse.de> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@canonical.com> 写道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hello bcache team.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I know this project is done and stable as [0] says, but I have a
> >>>>>>>>>>>>> question if anyone is around to answer.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Has bcache devices been tested and supported on multipath'd disks? I'm
> >>>>>>>>>>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
> >>>>>>>>>>>>> I'm wondering if there was any consideration or support for
> >>>>>>>>>>>>> multipathing when this project was made.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Also, your new project, bcachefs, might be hitting the same scenario.
> >>>>>>>>>>>>> I haven't had the time to test this though unfortunately.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for your time,
> >>>>>>>>>>>>> -Mitch
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [0] - https://bcache.evilpiepirate.org/#index4h1
> >>>>>>>>>>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
> >>>>>>>>>>>>
> >>>>>>>>>>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Coly Li
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>>
> >
>