RBD volumes for bare-metal hosts

RBD volumes for bare-metal hosts

[Radoslaw Zarzynski]

Hello Guys,

I wonder about a method for providing bare-metal hosts with an access
to RBD volumes.

The most obvious solution is krbd. Unfortunately, it has a substantial
feature gap in comparison to librbd [1][2]. exclusive-lock seems to be
the basic dependency that should be satisfied to even start thinking
about the rest. Is someone working on this feature? What are the plans
in general?

I also saw that Jewel brought a new interesting concept: rbd-nbd.
It was mentioned in the pull request [2] that an internal testing
shows performance comparable with krbd. Could someone share
a bit more detail in that matter?

Best regards,
Radoslaw Zarzynski

[1] https://github.com/ceph/ceph-client/blob/for-linus/drivers/block/rbd.c#L121
[2] https://github.com/ceph/ceph/blob/master/src/include/rbd/features.h#L25
[3] https://github.com/ceph/ceph/pull/6657

Re: RBD volumes for bare-metal hosts

[Ilya Dryomov]

On Wed, Jul 6, 2016 at 12:16 AM, Radoslaw Zarzynski
<rzarzynski@mirantis.com> wrote:
> Hello Guys,
>
> I wonder about a method for providing bare-metal hosts with an access
> to RBD volumes.
>
> The most obvious solution is krbd. Unfortunately, it has a substantial
> feature gap in comparison to librbd [1][2]. exclusive-lock seems to be
> the basic dependency that should be satisfied to even start thinking
> about the rest. Is someone working on this feature? What are the plans
> in general?

The code for exclusive-lock, object-map and fast-diff is pretty much
ready.  We had a large prerequisite that had to go in before any of
these and it's been merged into 4.7, so it's now a matter of review and
more comprehensive testing.  deep-flatten isn't ready, but should
follow shortly after.

Note that you can disable all of these features dynamically and that
the image is perfectly usable with none of them enabled.  Unless you
have clones or discard a lot, in which case object-map can help, the
I/O path is the same.

No concrete plans for mirroring support yet.

Thanks,

                Ilya

Re: RBD volumes for bare-metal hosts

[Radoslaw Zarzynski]

Hi Ilya,

Thanks for the quick and detailed response.

On Wed, Jul 6, 2016 at 1:41 AM, Ilya Dryomov <idryomov@gmail.com> wrote:
> The code for exclusive-lock, object-map and fast-diff is pretty much
> ready.  We had a large prerequisite that had to go in before any of
> these and it's been merged into 4.7, so it's now a matter of review and
> more comprehensive testing.  deep-flatten isn't ready, but should
> follow shortly after.

This sounds great.

> No concrete plans for mirroring support yet.

AFAIK the sole dependency for mirroring (from RBD volume
consumer's POV) is journaling that in turn requires exclusive-lock.
Having the last one in the code base opens a door for the whole
feature. The user-space daemon (rbd-mirror) and tooling might be
reused. Is that correct?

Regards,
Radek

Re: RBD volumes for bare-metal hosts

[Ilya Dryomov]

On Wed, Jul 6, 2016 at 2:21 AM, Radoslaw Zarzynski
<rzarzynski@mirantis.com> wrote:
> Hi Ilya,
>
> Thanks for the quick and detailed response.
>
> On Wed, Jul 6, 2016 at 1:41 AM, Ilya Dryomov <idryomov@gmail.com> wrote:
>> The code for exclusive-lock, object-map and fast-diff is pretty much
>> ready.  We had a large prerequisite that had to go in before any of
>> these and it's been merged into 4.7, so it's now a matter of review and
>> more comprehensive testing.  deep-flatten isn't ready, but should
>> follow shortly after.
>
> This sounds great.
>
>> No concrete plans for mirroring support yet.
>
> AFAIK the sole dependency for mirroring (from RBD volume
> consumer's POV) is journaling that in turn requires exclusive-lock.
> Having the last one in the code base opens a door for the whole
> feature. The user-space daemon (rbd-mirror) and tooling might be
> reused. Is that correct?

Yes, that should be correct.

Thanks,

                Ilya

Re: RBD volumes for bare-metal hosts

[Dongsheng Yang]

On 07/06/2016 07:41 AM, Ilya Dryomov wrote:
> On Wed, Jul 6, 2016 at 12:16 AM, Radoslaw Zarzynski
> <rzarzynski@mirantis.com> wrote:
>> Hello Guys,
>>
>> I wonder about a method for providing bare-metal hosts with an access
>> to RBD volumes.
>>
>> The most obvious solution is krbd. Unfortunately, it has a substantial
>> feature gap in comparison to librbd [1][2]. exclusive-lock seems to be
>> the basic dependency that should be satisfied to even start thinking
>> about the rest. Is someone working on this feature? What are the plans
>> in general?
> The code for exclusive-lock, object-map and fast-diff is pretty much
> ready.

Hi Ilya,
     Sounds great! But where can I read these code, I did not find it
in https://github.com/ceph/ceph-client. could you give a reference
for it. thanx a lot.

Yang
>   We had a large prerequisite that had to go in before any of
> these and it's been merged into 4.7, so it's now a matter of review and
> more comprehensive testing.  deep-flatten isn't ready, but should
> follow shortly after.
>
> Note that you can disable all of these features dynamically and that
> the image is perfectly usable with none of them enabled.  Unless you
> have clones or discard a lot, in which case object-map can help, the
> I/O path is the same.
>
> No concrete plans for mirroring support yet.
>
> Thanks,
>
>                  Ilya
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: RBD volumes for bare-metal hosts

[Ilya Dryomov]

On Tue, Jul 12, 2016 at 9:46 AM, Dongsheng Yang
<dongsheng.yang@easystack.cn> wrote:
>
>
> On 07/06/2016 07:41 AM, Ilya Dryomov wrote:
>>
>> On Wed, Jul 6, 2016 at 12:16 AM, Radoslaw Zarzynski
>> <rzarzynski@mirantis.com> wrote:
>>>
>>> Hello Guys,
>>>
>>> I wonder about a method for providing bare-metal hosts with an access
>>> to RBD volumes.
>>>
>>> The most obvious solution is krbd. Unfortunately, it has a substantial
>>> feature gap in comparison to librbd [1][2]. exclusive-lock seems to be
>>> the basic dependency that should be satisfied to even start thinking
>>> about the rest. Is someone working on this feature? What are the plans
>>> in general?
>>
>> The code for exclusive-lock, object-map and fast-diff is pretty much
>> ready.
>
>
> Hi Ilya,
>     Sounds great! But where can I read these code, I did not find it
> in https://github.com/ceph/ceph-client. could you give a reference
> for it. thanx a lot.

It's not a final version - there are some bugs and sharp edges in
there.  But if you just want to take look:

    https://github.com/fullerdj/ceph-client/commits/wip-djf-osdc

Thanks,

                Ilya

Re: RBD volumes for bare-metal hosts

[Dongsheng Yang]

On 07/12/2016 04:00 PM, Ilya Dryomov wrote:
> On Tue, Jul 12, 2016 at 9:46 AM, Dongsheng Yang
> <dongsheng.yang@easystack.cn> wrote:
>>
>> On 07/06/2016 07:41 AM, Ilya Dryomov wrote:
>>> On Wed, Jul 6, 2016 at 12:16 AM, Radoslaw Zarzynski
>>> <rzarzynski@mirantis.com> wrote:
>>>> Hello Guys,
>>>>
>>>> I wonder about a method for providing bare-metal hosts with an access
>>>> to RBD volumes.
>>>>
>>>> The most obvious solution is krbd. Unfortunately, it has a substantial
>>>> feature gap in comparison to librbd [1][2]. exclusive-lock seems to be
>>>> the basic dependency that should be satisfied to even start thinking
>>>> about the rest. Is someone working on this feature? What are the plans
>>>> in general?
>>> The code for exclusive-lock, object-map and fast-diff is pretty much
>>> ready.
>>
>> Hi Ilya,
>>      Sounds great! But where can I read these code, I did not find it
>> in https://github.com/ceph/ceph-client. could you give a reference
>> for it. thanx a lot.
> It's not a final version - there are some bugs and sharp edges in
> there.  But if you just want to take look:
>
>      https://github.com/fullerdj/ceph-client/commits/wip-djf-osdc

Got it. Thanx
>
> Thanks,
>
>                  Ilya
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>