SCSI inquiry interface for RBD devices

SCSI inquiry interface for RBD devices

[Kamble, Nitin A]

SCSI block devices has SG_IO IOCTL interface by which applications can identify the device uniquely. This helps applications identify the device uniquely without relying on the device file names or any other mechanisms.

For more details l ook at the manpage of sg_inq commnad. https://manned.org/sg_inq

Some of the non SCSI devices also support this SG_IO interface for inquiry. Examples include SATA, NVME, Virtual Disks.

It is desired to identify every RBD uniquely just from the device file. It will help in using RBD devices as drop in replacement for SCSI/SATA devices. Currently the RBD code in ceph-client does not support the SG_IO ioctl interface for inquiry to identify the device uniquely. Has adding the SG_IO ioctl interface to the RBD devices ever been considered before?



Thanks,
Nitin

Re: SCSI inquiry interface for RBD devices

[Ilya Dryomov]

On Fri, Aug 12, 2016 at 1:38 AM, Kamble, Nitin A
<Nitin.Kamble@teradata.com> wrote:
>
> SCSI block devices has SG_IO IOCTL interface by which applications can identify the device uniquely. This helps applications identify the device uniquely without relying on the device file names or any other mechanisms.
>
> For more details l ook at the manpage of sg_inq commnad. https://manned.org/sg_inq
>
> Some of the non SCSI devices also support this SG_IO interface for inquiry. Examples include SATA, NVME, Virtual Disks.
>
> It is desired to identify every RBD uniquely just from the device file. It will help in using RBD devices as drop in replacement for SCSI/SATA devices. Currently the RBD code in ceph-client does not support the SG_IO ioctl interface for inquiry to identify the device uniquely. Has adding the SG_IO ioctl interface to the RBD devices ever been considered before?

No, SG_IO hasn't come up before, and we don't really have a good
established way of doing that at all - if we had one, wiring up SG_IO
(assuming no objections from SCSI maintainers) wouldn't be a problem.
One approach is to concatenate a cluster-wide UUID with pool, image and
snapshot IDs, but the problem there is that images can be moved between
pools and clusters and v1 images don't have an image ID.

Mike recently ran into the need to uniquely identify RBD devices for
his multipath-tools work and I think he ended up using some variant of
the above approach, grabbing IDs from /sys/bus/rbd.  (The cluster_uuid
and snap_id attributes currently don't exist, but will be added in the
next kernel release.)

It would make this a lot easier if we started generating a unique UUID
for the image and each snapshot at image/snapshot creation time.  Jason?

Thanks,

                Ilya

Re: SCSI inquiry interface for RBD devices

[Mike Christie]

On 08/11/2016 06:38 PM, Kamble, Nitin A wrote:
> 
> SCSI block devices has SG_IO IOCTL interface by which applications can identify the device uniquely. This helps applications identify the device uniquely without relying on the device file names or any other mechanisms.
> 
> For more details l ook at the manpage of sg_inq commnad. https://manned.org/sg_inq
> 
> Some of the non SCSI devices also support this SG_IO interface for inquiry. Examples include SATA, NVME, Virtual Disks.
> 
> It is desired to identify every RBD uniquely just from the device file. It will help in using RBD devices as drop in replacement for SCSI/SATA devices. Currently the RBD code in ceph-client does not support the SG_IO ioctl interface for inquiry to identify the device uniquely. Has adding the SG_IO ioctl interface to the RBD devices ever been considered before?
> 

What application do you need this for right now? Is it udev's scsi_id?

What pages do you need? Just 0x83?

Re: SCSI inquiry interface for RBD devices

[Ilya Dryomov]

On Fri, Aug 12, 2016 at 4:17 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
> On Fri, Aug 12, 2016 at 1:38 AM, Kamble, Nitin A
> <Nitin.Kamble@teradata.com> wrote:
>>
>> SCSI block devices has SG_IO IOCTL interface by which applications can identify the device uniquely. This helps applications identify the device uniquely without relying on the device file names or any other mechanisms.
>>
>> For more details l ook at the manpage of sg_inq commnad. https://manned.org/sg_inq
>>
>> Some of the non SCSI devices also support this SG_IO interface for inquiry. Examples include SATA, NVME, Virtual Disks.
>>
>> It is desired to identify every RBD uniquely just from the device file. It will help in using RBD devices as drop in replacement for SCSI/SATA devices. Currently the RBD code in ceph-client does not support the SG_IO ioctl interface for inquiry to identify the device uniquely. Has adding the SG_IO ioctl interface to the RBD devices ever been considered before?
>
> No, SG_IO hasn't come up before, and we don't really have a good
> established way of doing that at all - if we had one, wiring up SG_IO
> (assuming no objections from SCSI maintainers) wouldn't be a problem.
> One approach is to concatenate a cluster-wide UUID with pool, image and
> snapshot IDs, but the problem there is that images can be moved between
> pools and clusters and v1 images don't have an image ID.
>
> Mike recently ran into the need to uniquely identify RBD devices for
> his multipath-tools work and I think he ended up using some variant of
> the above approach, grabbing IDs from /sys/bus/rbd.  (The cluster_uuid
> and snap_id attributes currently don't exist, but will be added in the
> next kernel release.)
>
> It would make this a lot easier if we started generating a unique UUID
> for the image and each snapshot at image/snapshot creation time.  Jason?

I've filed http://tracker.ceph.com/issues/17012 for UUIDs.  Once it's
in, exporting that in various ways should be trivial.

Thanks,

                Ilya

Re: SCSI inquiry interface for RBD devices

[Kamble, Nitin A]

> On Aug 12, 2016, at 7:17 AM, Ilya Dryomov <idryomov@gmail.com> wrote:
> 
> On Fri, Aug 12, 2016 at 1:38 AM, Kamble, Nitin A
> <Nitin.Kamble@teradata.com> wrote:
>> 
>> SCSI block devices has SG_IO IOCTL interface by which applications can identify the device uniquely. This helps applications identify the device uniquely without relying on the device file names or any other mechanisms.
>> 
>> For more details l ook at the manpage of sg_inq commnad. https://manned.org/sg_inq
>> 
>> Some of the non SCSI devices also support this SG_IO interface for inquiry. Examples include SATA, NVME, Virtual Disks.
>> 
>> It is desired to identify every RBD uniquely just from the device file. It will help in using RBD devices as drop in replacement for SCSI/SATA devices. Currently the RBD code in ceph-client does not support the SG_IO ioctl interface for inquiry to identify the device uniquely. Has adding the SG_IO ioctl interface to the RBD devices ever been considered before?
> 
> No, SG_IO hasn't come up before, and we don't really have a good
> established way of doing that at all - if we had one, wiring up SG_IO
> (assuming no objections from SCSI maintainers) wouldn't be a problem.
> One approach is to concatenate a cluster-wide UUID with pool, image and
> snapshot IDs, but the problem there is that images can be moved between
> pools and clusters and v1 images don't have an image ID.

I am surprised to know that images can be moved across pools and clusters. The feature request for moving images across pools was rejected here: http://tracker.ceph.com/issues/2305 . Also the rbd man page says rbd rename across pools is not supported. (http://docs.ceph.com/docs/master/man/8/rbd/)

Is it a currently available feature?

> 
> Mike recently ran into the need to uniquely identify RBD devices for
> his multipath-tools work and I think he ended up using some variant of
> the above approach, grabbing IDs from /sys/bus/rbd.  (The cluster_uuid
> and snap_id attributes currently don't exist, but will be added in the
> next kernel release.)
> 
> It would make this a lot easier if we started generating a unique UUID
> for the image and each snapshot at image/snapshot creation time.  Jason?
> 

Image UUID seems like the right direction.
If that UUID needs to be really unique, then image copy also need to create a new UUID.

Thanks,
Nitin


> Thanks,
> 
>                Ilya

Re: SCSI inquiry interface for RBD devices

[Ilya Dryomov]

On Fri, Aug 12, 2016 at 7:42 PM, Kamble, Nitin A
<Nitin.Kamble@teradata.com> wrote:
>
>> On Aug 12, 2016, at 7:17 AM, Ilya Dryomov <idryomov@gmail.com> wrote:
>>
>> On Fri, Aug 12, 2016 at 1:38 AM, Kamble, Nitin A
>> <Nitin.Kamble@teradata.com> wrote:
>>>
>>> SCSI block devices has SG_IO IOCTL interface by which applications can identify the device uniquely. This helps applications identify the device uniquely without relying on the device file names or any other mechanisms.
>>>
>>> For more details l ook at the manpage of sg_inq commnad. https://manned.org/sg_inq
>>>
>>> Some of the non SCSI devices also support this SG_IO interface for inquiry. Examples include SATA, NVME, Virtual Disks.
>>>
>>> It is desired to identify every RBD uniquely just from the device file. It will help in using RBD devices as drop in replacement for SCSI/SATA devices. Currently the RBD code in ceph-client does not support the SG_IO ioctl interface for inquiry to identify the device uniquely. Has adding the SG_IO ioctl interface to the RBD devices ever been considered before?
>>
>> No, SG_IO hasn't come up before, and we don't really have a good
>> established way of doing that at all - if we had one, wiring up SG_IO
>> (assuming no objections from SCSI maintainers) wouldn't be a problem.
>> One approach is to concatenate a cluster-wide UUID with pool, image and
>> snapshot IDs, but the problem there is that images can be moved between
>> pools and clusters and v1 images don't have an image ID.
>
> I am surprised to know that images can be moved across pools and clusters. The feature request for moving images across pools was rejected here: http://tracker.ceph.com/issues/2305 . Also the rbd man page says rbd rename across pools is not supported. (http://docs.ceph.com/docs/master/man/8/rbd/)
>
> Is it a currently available feature?

There are a couple of hacky, undocumented and unsupported ways which
I won't describe.  The above was just a very bad way of saying that
current v2 image IDs are guaranteed to be unique only within a pool and
can be the duplicated between different pools on the same cluster and
different clusters.

>
>>
>> Mike recently ran into the need to uniquely identify RBD devices for
>> his multipath-tools work and I think he ended up using some variant of
>> the above approach, grabbing IDs from /sys/bus/rbd.  (The cluster_uuid
>> and snap_id attributes currently don't exist, but will be added in the
>> next kernel release.)
>>
>> It would make this a lot easier if we started generating a unique UUID
>> for the image and each snapshot at image/snapshot creation time.  Jason?
>>
>
> Image UUID seems like the right direction.

With UUIDs we would be in much better shape - current v2 image ID is
relatively very short.  We can couple it with a cluster UUID too.

> If that UUID needs to be really unique, then image copy also need to create a new UUID.

It would be akin to the UUID of a partition.  I suppose including
a tune2fs -U functionality wouldn't hurt.  I'll add that to the ticket.

Thanks,

                Ilya

Re: SCSI inquiry interface for RBD devices

[Kamble, Nitin A]

> On Aug 12, 2016, at 10:19 AM, Mike Christie <mchristi@redhat.com> wrote:
> 
> On 08/11/2016 06:38 PM, Kamble, Nitin A wrote:
>> 
>> SCSI block devices has SG_IO IOCTL interface by which applications can identify the device uniquely. This helps applications identify the device uniquely without relying on the device file names or any other mechanisms.
>> 
>> For more details l ook at the manpage of sg_inq commnad. https://manned.org/sg_inq
>> 
>> Some of the non SCSI devices also support this SG_IO interface for inquiry. Examples include SATA, NVME, Virtual Disks.
>> 
>> It is desired to identify every RBD uniquely just from the device file. It will help in using RBD devices as drop in replacement for SCSI/SATA devices. Currently the RBD code in ceph-client does not support the SG_IO ioctl interface for inquiry to identify the device uniquely. Has adding the SG_IO ioctl interface to the RBD devices ever been considered before?
>> 
> 
> What application do you need this for right now? Is it udev's scsi_id?

The sg_inq command is one of such application. Or any application calling sg_simple_inquiry() function from /usr/include/scsi/sg_cmds_basic.h

> 
> What pages do you need? Just 0x83?
> 

The pages 0x83 & 0x80 are good to have. The Inquiry command 0x12 is of most interest. It is described in the section 3.6.1 of this SCSI commands reference manual: http://www.seagate.com/files/staticfiles/support/docs/manual/Interface%20manuals/100293068h.pdf .

Thanks,
Nitin

Re: SCSI inquiry interface for RBD devices

[Kamble, Nitin A]

> On Aug 12, 2016, at 11:15 AM, Ilya Dryomov <idryomov@gmail.com> wrote:
> 
> On Fri, Aug 12, 2016 at 7:42 PM, Kamble, Nitin A
> <Nitin.Kamble@teradata.com> wrote:
>> 
>>> On Aug 12, 2016, at 7:17 AM, Ilya Dryomov <idryomov@gmail.com> wrote:
>>> 
>>> On Fri, Aug 12, 2016 at 1:38 AM, Kamble, Nitin A
>>> <Nitin.Kamble@teradata.com> wrote:
>>>> 
>>>> SCSI block devices has SG_IO IOCTL interface by which applications can identify the device uniquely. This helps applications identify the device uniquely without relying on the device file names or any other mechanisms.
>>>> 
>>>> For more details l ook at the manpage of sg_inq commnad. https://manned.org/sg_inq
>>>> 
>>>> Some of the non SCSI devices also support this SG_IO interface for inquiry. Examples include SATA, NVME, Virtual Disks.
>>>> 
>>>> It is desired to identify every RBD uniquely just from the device file. It will help in using RBD devices as drop in replacement for SCSI/SATA devices. Currently the RBD code in ceph-client does not support the SG_IO ioctl interface for inquiry to identify the device uniquely. Has adding the SG_IO ioctl interface to the RBD devices ever been considered before?
>>> 
>>> No, SG_IO hasn't come up before, and we don't really have a good
>>> established way of doing that at all - if we had one, wiring up SG_IO
>>> (assuming no objections from SCSI maintainers) wouldn't be a problem.
>>> One approach is to concatenate a cluster-wide UUID with pool, image and
>>> snapshot IDs, but the problem there is that images can be moved between
>>> pools and clusters and v1 images don't have an image ID.
>> 
>> I am surprised to know that images can be moved across pools and clusters. The feature request for moving images across pools was rejected here: http://tracker.ceph.com/issues/2305 . Also the rbd man page says rbd rename across pools is not supported. (http://docs.ceph.com/docs/master/man/8/rbd/)
>> 
>> Is it a currently available feature?
> 
> There are a couple of hacky, undocumented and unsupported ways which
> I won't describe.  The above was just a very bad way of saying that
> current v2 image IDs are guaranteed to be unique only within a pool and
> can be the duplicated between different pools on the same cluster and
> different clusters.
> 
>> 
>>> 
>>> Mike recently ran into the need to uniquely identify RBD devices for
>>> his multipath-tools work and I think he ended up using some variant of
>>> the above approach, grabbing IDs from /sys/bus/rbd.  (The cluster_uuid
>>> and snap_id attributes currently don't exist, but will be added in the
>>> next kernel release.)
>>> 
>>> It would make this a lot easier if we started generating a unique UUID
>>> for the image and each snapshot at image/snapshot creation time.  Jason?
>>> 
>> 
>> Image UUID seems like the right direction.
> 
> With UUIDs we would be in much better shape - current v2 image ID is
> relatively very short.  We can couple it with a cluster UUID too.
> 
>> If that UUID needs to be really unique, then image copy also need to create a new UUID.
> 
> It would be akin to the UUID of a partition.  I suppose including
> a tune2fs -U functionality wouldn't hurt.  I'll add that to the ticket.
> 

The rbd creation creation tool having an ability to assign a specific UUID to the image being created would be useful too.

> Thanks,
> 
>                Ilya

Re: SCSI inquiry interface for RBD devices

[Alex Elder]

On 08/12/2016 01:40 PM, Kamble, Nitin A wrote:
>> It would make this a lot easier if we started generating a unique UUID
>> >>> for the image and each snapshot at image/snapshot creation time.  Jason?
>> >>> 
>>> 
>>> Image UUID seems like the right direction.

Yes.  It's not a snapshot you want, but the basic image, regardless
of the state of its content.  It needs to be created at image
creation time, and no matter how many times the image grows or
shrinks, or how the cluster gets reorganized along the way, that
UUID uniquely defines the image (and it ought not to be possible,
as a rule, to *change* an image's UUID).  Coping an entire image
(or cloning) would require a new UUID (as has been said).

I think the UUID should be defined independent of the cluster as
well.  In that case, there may be a reason to think about a "move"
operation, where it an image is transferred to a new cluster, with
the new one retaining the original UUID and the old one going away.

					-Alex

Re: SCSI inquiry interface for RBD devices

[Ilya Dryomov]

On Sat, Aug 13, 2016 at 5:18 PM, Alex Elder <elder@ieee.org> wrote:
> On 08/12/2016 01:40 PM, Kamble, Nitin A wrote:
>>> It would make this a lot easier if we started generating a unique UUID
>>> >>> for the image and each snapshot at image/snapshot creation time.  Jason?
>>> >>>
>>>>
>>>> Image UUID seems like the right direction.
>
> Yes.  It's not a snapshot you want, but the basic image, regardless
> of the state of its content.  It needs to be created at image

I'm not sure I follow.  We need to have an identifier for each mappable
entity - that's images (HEADs) and snapshots.

> creation time, and no matter how many times the image grows or
> shrinks, or how the cluster gets reorganized along the way, that
> UUID uniquely defines the image (and it ought not to be possible,
> as a rule, to *change* an image's UUID).  Coping an entire image

Why?  IMO it shouldn't be any different from a filesystem UUID - a user
should be able to create an image (snapshot) with the specified UUID,
change it at any time and maybe even clear it (allowing clear makes
older images a non-issue).  We already have an unchangeable pool-wide
ID for ourselves, this is aimed at users and they should be in full
control here.

> (or cloning) would require a new UUID (as has been said).
>
> I think the UUID should be defined independent of the cluster as
> well.  In that case, there may be a reason to think about a "move"
> operation, where it an image is transferred to a new cluster, with
> the new one retaining the original UUID and the old one going away.

By default, it would be a randomly-generated UUID.  What I had meant
when I said "we can couple it with a cluster UUID too" was that we can
export a cluster UUID along with an image UUID.

Thanks,

                Ilya

Re: SCSI inquiry interface for RBD devices

[Alex Elder]

On 08/13/2016 02:14 PM, Ilya Dryomov wrote:
> On Sat, Aug 13, 2016 at 5:18 PM, Alex Elder <elder@ieee.org> wrote:
>> On 08/12/2016 01:40 PM, Kamble, Nitin A wrote:
>>>> It would make this a lot easier if we started generating a unique UUID
>>>>>>> for the image and each snapshot at image/snapshot creation time.  Jason?
>>>>>>>
>>>>>
>>>>> Image UUID seems like the right direction.

I'm really sorry I didn't respond to this earlier.  I sent out
my message and promptly went on vacation for a week.

>> Yes.  It's not a snapshot you want, but the basic image, regardless
>> of the state of its content.  It needs to be created at image
> 
> I'm not sure I follow.  We need to have an identifier for each mappable
> entity - that's images (HEADs) and snapshots.

I guess it really boils down to what you intend this
unique identifier to represent.  SCSI has VPD data
pages for device identification.  I *think* these
are intended to allow a physical device or logical
unit to have a persistent, globally unique identifier.
I haven't worked with the SCSI specs in a *long* time
though, and I'm just scanning an SPC draft right now.

Anyway, my earlier response was based on the notion that
you wanted to identify some RBD image equivalent of a
hardware unique identifier, much as you might have for
identifying a single physical disk drive.  In that case,
the identifier should be generated at image creation time.
If a copy of an image were created, it might contain the
same data, but would necessarily have a different UUID
for the duplicate image.

The complication is that we really aren't dealing with
physical entities here.  It is not possible to truly
duplicate a physical disk drive in the same way we
can duplicate an RBD image.  And there is no concept
of a snapshot or clone for a physical disk.

I don't actually know how exactly this identifier is
intended to be used, and I probably just spoke with
an improper understanding.

>> creation time, and no matter how many times the image grows or
>> shrinks, or how the cluster gets reorganized along the way, that
>> UUID uniquely defines the image (and it ought not to be possible,
>> as a rule, to *change* an image's UUID).  Coping an entire image
> 
> Why?  IMO it shouldn't be any different from a filesystem UUID - a user
> should be able to create an image (snapshot) with the specified UUID,
> change it at any time and maybe even clear it (allowing clear makes
> older images a non-issue).  We already have an unchangeable pool-wide
> ID for ourselves, this is aimed at users and they should be in full
> control here.

That may be the better model in this case.  That is, the
UUID you're talking about is a soft label that's unique,
but changeable.  The model I had in my mind was more like
an airplane serial number plate; you can replace every
part on an airplane, but it remains "the same airplane"
as long as it has on it the plate that shows the original
serial number.

>> (or cloning) would require a new UUID (as has been said).
>>
>> I think the UUID should be defined independent of the cluster as
>> well.  In that case, there may be a reason to think about a "move"
>> operation, where it an image is transferred to a new cluster, with
>> the new one retaining the original UUID and the old one going away.
> 
> By default, it would be a randomly-generated UUID.  What I had meant
> when I said "we can couple it with a cluster UUID too" was that we can
> export a cluster UUID along with an image UUID.

Again, sorry for the delay in responding to you.  I don't
claim to know the answer, but I think you need to be very
clear on what the UUID you're talking about really represents.
Mimicking the SCSI inquiry interface is reasonable, but we
are dealing with something that isn't exactly like a SCSI
device, so I think precision on the meaning will be important.

					-Alex


> 
> Thanks,
> 
>                 Ilya
>