Best practice for new linux block driver device naming?

Best practice for new linux block driver device naming?

[scameron]

Hi,

I'm just wondering if there are best pratices for new linux block
drivers that are adding new devices nodes of which grub is currently
not cognizant.

E.g. when we added the HP Smart Array cciss driver to the kernel
many years ago, it had device nodes like /dev/cciss/c*d*, and there's
code in grub to handle this in util/getroot.c, in
convert_system_partition_to_system_disk():

      /* If this is a CCISS disk.  */
      if (strncmp ("cciss/c", p, sizeof ("cciss/c") - 1) == 0)
        {
          /* /dev/cciss/c[0-9]+d[0-9]+(p[0-9]+)? */
          p = strchr (p, 'p');
          if (p)
            {
              *is_part = 1;
              *p = '\0';
            }

          return path;
        }

And there is similar code for other weird device names is in there
as well.

Ideally, I'm hoping there's a way to introduce new devices nodes
with a new block driver which does not any require grub modifications.
Looking over the code, it's not clear to me whether or not this is
possible, and if it is, how to do it, what the constraints may be, etc.

Currently I have a new driver which adds devices like /dev/sop0  with
partitions like /dev/sop0p1, /dev/sop0p2, etc.

If there is some better way to do this to enable grub to work with
these devices, it is not yet too late for me to change it.

Or on the other hand, if it turns out that it is not possible to add
new block devices to linux and have grub support for those devices without
also modifying grub, then I wonder if it might be worth looking into to
adding some kind of shared device namespace for block devices to linux, so
new block drivers could use that and have a common naming system for block devices,
and grub could be modified to support this new common naming system,
much as scsi hba device drivers share the /dev/sd* namespace for their attached
disks, so it is easy to add new scsi hba drivers to linux and automatically have
grub support for them.  It would be nice if it were similarly easy to add new
block device drivers to linux without also requiring modifications to grub.
(It also occurs to me that this is such an obvious desire that if it is not 
already supported, perhaps there's a good reason why not, but if that's the
case, I'm don't know what the reason might be.)

Thoughts?

Thanks,

-- steve

Re: Best practice for new linux block driver device naming?

[Lennart Sorensen]

On Fri, Mar 08, 2013 at 02:07:18PM -0600, scameron@beardog.cce.hp.com wrote:
> I'm just wondering if there are best pratices for new linux block
> drivers that are adding new devices nodes of which grub is currently
> not cognizant.
> 
> E.g. when we added the HP Smart Array cciss driver to the kernel
> many years ago, it had device nodes like /dev/cciss/c*d*, and there's
> code in grub to handle this in util/getroot.c, in
> convert_system_partition_to_system_disk():
> 
>       /* If this is a CCISS disk.  */
>       if (strncmp ("cciss/c", p, sizeof ("cciss/c") - 1) == 0)
>         {
>           /* /dev/cciss/c[0-9]+d[0-9]+(p[0-9]+)? */
>           p = strchr (p, 'p');
>           if (p)
>             {
>               *is_part = 1;
>               *p = '\0';
>             }
> 
>           return path;
>         }
> 
> And there is similar code for other weird device names is in there
> as well.
> 
> Ideally, I'm hoping there's a way to introduce new devices nodes
> with a new block driver which does not any require grub modifications.
> Looking over the code, it's not clear to me whether or not this is
> possible, and if it is, how to do it, what the constraints may be, etc.
> 
> Currently I have a new driver which adds devices like /dev/sop0  with
> partitions like /dev/sop0p1, /dev/sop0p2, etc.
> 
> If there is some better way to do this to enable grub to work with
> these devices, it is not yet too late for me to change it.
> 
> Or on the other hand, if it turns out that it is not possible to add
> new block devices to linux and have grub support for those devices without
> also modifying grub, then I wonder if it might be worth looking into to
> adding some kind of shared device namespace for block devices to linux, so
> new block drivers could use that and have a common naming system for block devices,
> and grub could be modified to support this new common naming system,
> much as scsi hba device drivers share the /dev/sd* namespace for their attached
> disks, so it is easy to add new scsi hba drivers to linux and automatically have
> grub support for them.  It would be nice if it were similarly easy to add new
> block device drivers to linux without also requiring modifications to grub.
> (It also occurs to me that this is such an obvious desire that if it is not 
> already supported, perhaps there's a good reason why not, but if that's the
> case, I'm don't know what the reason might be.)
> 
> Thoughts?

Well currently, SCSI, SATA, IDE, most well behaved raid controllers,
USB storage, and many others all show up simple as /dev/sd*.  You better
have a really good reason to not do so if you make a new controller.
Certainly the IBM serveraid cards I have worked with just present sd*
devices (as well as some sg* devices for the controller and hotswap
backplane and such).  I think the CCISS is a sample of a horrible design
as far as the device names in linux are concerned.

By creating a new type of block device you force everyone else to do work
to support it (or choose to ignore your device because no one cares).
By emulating a plain old scsi device interface, everything else just
works, and all the work is done by you in your device driver to pretend
to my just a scsi disk.  If you want your device taken seriously,
I don't think you have a choice.  Sysadmin's hate things that are
different for no good reason.

-- 
Len Sorensen

Re: Best practice for new linux block driver device naming?

[scameron]

On Fri, Mar 08, 2013 at 04:56:32PM -0500, Lennart Sorensen wrote:
> On Fri, Mar 08, 2013 at 02:07:18PM -0600, scameron@beardog.cce.hp.com wrote:
> > I'm just wondering if there are best pratices for new linux block
> > drivers that are adding new devices nodes of which grub is currently
> > not cognizant.
> > 
> > E.g. when we added the HP Smart Array cciss driver to the kernel
> > many years ago, it had device nodes like /dev/cciss/c*d*, and there's
> > code in grub to handle this in util/getroot.c, in
> > convert_system_partition_to_system_disk():
> > 
> >       /* If this is a CCISS disk.  */
> >       if (strncmp ("cciss/c", p, sizeof ("cciss/c") - 1) == 0)
> >         {
> >           /* /dev/cciss/c[0-9]+d[0-9]+(p[0-9]+)? */
> >           p = strchr (p, 'p');
> >           if (p)
> >             {
> >               *is_part = 1;
> >               *p = '\0';
> >             }
> > 
> >           return path;
> >         }
> > 
> > And there is similar code for other weird device names is in there
> > as well.
> > 
> > Ideally, I'm hoping there's a way to introduce new devices nodes
> > with a new block driver which does not any require grub modifications.
> > Looking over the code, it's not clear to me whether or not this is
> > possible, and if it is, how to do it, what the constraints may be, etc.
> > 
> > Currently I have a new driver which adds devices like /dev/sop0  with
> > partitions like /dev/sop0p1, /dev/sop0p2, etc.
> > 
> > If there is some better way to do this to enable grub to work with
> > these devices, it is not yet too late for me to change it.
> > 
> > Or on the other hand, if it turns out that it is not possible to add
> > new block devices to linux and have grub support for those devices without
> > also modifying grub, then I wonder if it might be worth looking into to
> > adding some kind of shared device namespace for block devices to linux, so
> > new block drivers could use that and have a common naming system for block devices,
> > and grub could be modified to support this new common naming system,
> > much as scsi hba device drivers share the /dev/sd* namespace for their attached
> > disks, so it is easy to add new scsi hba drivers to linux and automatically have
> > grub support for them.  It would be nice if it were similarly easy to add new
> > block device drivers to linux without also requiring modifications to grub.
> > (It also occurs to me that this is such an obvious desire that if it is not 
> > already supported, perhaps there's a good reason why not, but if that's the
> > case, I'm don't know what the reason might be.)
> > 
> > Thoughts?
> 
> Well currently, SCSI, SATA, IDE, most well behaved raid controllers,
> USB storage, and many others all show up simple as /dev/sd*.  You better
> have a really good reason to not do so if you make a new controller.
> Certainly the IBM serveraid cards I have worked with just present sd*
> devices (as well as some sg* devices for the controller and hotswap
> backplane and such).  

I get ~4x the IOPSs with a block driver vs. scsi driver due to contention
for locks in the scsi mid layer (in scsi_request_fn).  It's the
difference between the device being worth manufacturing vs. not.

See this thread: http://marc.info/?l=linux-scsi&m=135518042125008&w=2

Driver is similar to nvme (also a new block driver), but this one is
for SCSI over PCIe, basically highly parallelized access to very low
latency devices and trying to use the SCSI midlayer kills the IOPS.
 
> I think the CCISS is a sample of a horrible design
> as far as the device names in linux are concerned.

There were reasons back then for doing that one as a block driver
which are no longer extant (hence the existence of the hpsa driver
which supplanted cciss for new smart array devices.)

> 
> By creating a new type of block device you force everyone else to do work
> to support it (or choose to ignore your device because no one cares).
> By emulating a plain old scsi device interface, everything else just
> works, and all the work is done by you in your device driver to pretend
> to my just a scsi disk.  If you want your device taken seriously,
> I don't think you have a choice.  Sysadmin's hate things that are
> different for no good reason.

All other things being equal, I would also prefer a scsi driver.
Heck, it's called SCSI over PCIe -- I tried like hell to get it 
to perform adequately as a SCSI driver but all other things are
not equal, not even close, the block driver was ~4x as fast.

So we reluctantly go with a block driver, just like nvme did.

-- steve

Re: Best practice for new linux block driver device naming?

[Lennart Sorensen]

On Fri, Mar 08, 2013 at 04:34:28PM -0600, scameron@beardog.cce.hp.com wrote:
> I get ~4x the IOPSs with a block driver vs. scsi driver due to contention
> for locks in the scsi mid layer (in scsi_request_fn).  It's the
> difference between the device being worth manufacturing vs. not.

Well that starts to qualify as a good reason I suppose.  Of course it
also makes you wonder if perhaps some work on optimizing that part of
the scsi stack is oin order (I have no idea if that's even plausible).

> See this thread: http://marc.info/?l=linux-scsi&m=135518042125008&w=2
> 
> Driver is similar to nvme (also a new block driver), but this one is
> for SCSI over PCIe, basically highly parallelized access to very low
> latency devices and trying to use the SCSI midlayer kills the IOPS.

Some nifty hardware that's for sure.

> There were reasons back then for doing that one as a block driver
> which are no longer extant (hence the existence of the hpsa driver
> which supplanted cciss for new smart array devices.)
> 
> All other things being equal, I would also prefer a scsi driver.
> Heck, it's called SCSI over PCIe -- I tried like hell to get it 
> to perform adequately as a SCSI driver but all other things are
> not equal, not even close, the block driver was ~4x as fast.
> 
> So we reluctantly go with a block driver, just like nvme did.

Makes sense.  Perhaps that does mean having to teach grub about it then.

-- 
Len Sorensen

Re: Best practice for new linux block driver device naming?

[scameron]

On Fri, Mar 08, 2013 at 05:49:11PM -0500, Lennart Sorensen wrote:
> On Fri, Mar 08, 2013 at 04:34:28PM -0600, scameron@beardog.cce.hp.com wrote:
> > I get ~4x the IOPSs with a block driver vs. scsi driver due to contention
> > for locks in the scsi mid layer (in scsi_request_fn).  It's the
> > difference between the device being worth manufacturing vs. not.
> 
> Well that starts to qualify as a good reason I suppose.  Of course it
> also makes you wonder if perhaps some work on optimizing that part of
> the scsi stack is oin order (I have no idea if that's even plausible).
> 
> > See this thread: http://marc.info/?l=linux-scsi&m=135518042125008&w=2
> > 
> > Driver is similar to nvme (also a new block driver), but this one is
> > for SCSI over PCIe, basically highly parallelized access to very low
> > latency devices and trying to use the SCSI midlayer kills the IOPS.
> 
> Some nifty hardware that's for sure.
> 
> > There were reasons back then for doing that one as a block driver
> > which are no longer extant (hence the existence of the hpsa driver
> > which supplanted cciss for new smart array devices.)
> > 
> > All other things being equal, I would also prefer a scsi driver.
> > Heck, it's called SCSI over PCIe -- I tried like hell to get it 
> > to perform adequately as a SCSI driver but all other things are
> > not equal, not even close, the block driver was ~4x as fast.
> > 
> > So we reluctantly go with a block driver, just like nvme did.
> 
> Makes sense.  Perhaps that does mean having to teach grub about it then.

We are not expecting to be able to boot from the device in the first iteration,
so it's not as if we would need support instantly (not that I imagine we could
get it instantly anyway), and it's not clear that it makes sense for such a high
IOPS device to be used as a boot device in most imaginable use cases anyway, but
OTOH, we would prefer not to rule out booting from it.

So, that being said, are there any best practices for naming new block device nodes?
Or is any scheme like /dev/sop[0-9a-z] about as good/bad as any other?

And, is it a worthwhile idea to pursue adding some sort of shared device namespace
for block devices to the kernel (maintaining backwards compatibility of device node
names would of course be required) so that block devices could have some shared
namespace as scsi devices do?

Typically the block devices themselves don't care what the device nodes are named, 
only the userland apps do, though it falls to the block drivers to specify a device
name via struct gendisk's ->disk_name member being set prior to calling add_disk().

If there were some kernel interface the block driver could use to get a disk name
assigned from a shared name space, something like blk_get_new_disk_name(disk->disk_name);
that could hand out device names like b%s, so you'd get all the block devices which use
this interface having device names like /dev/bda, /dev/bdb, /dev/bdc, analogous to
/dev/sda, /dev/sdb, etc. -- the specifics here don't matter to me, the above is just
an idea off the top of my head -- then, we teach grub about this new block device
namespace *once*, and force all new block drivers to use it.  Thereafter, adding a
block driver to the kernel causes no more grub related pain to grub and distro
developers and users than adding a new scsi hba driver -- i.e. none. 

Would such a thing be worth pursuing?  Or is there some good reason such a thing
doesn't already exist?  (Maybe this is more a question for lkml.)

-- steve

 

Re: Best practice for new linux block driver device naming?

[Lennart Sorensen]

On Fri, Mar 08, 2013 at 05:05:33PM -0600, scameron@beardog.cce.hp.com wrote:
> We are not expecting to be able to boot from the device in the first iteration,
> so it's not as if we would need support instantly (not that I imagine we could
> get it instantly anyway), and it's not clear that it makes sense for such a high
> IOPS device to be used as a boot device in most imaginable use cases anyway, but
> OTOH, we would prefer not to rule out booting from it.
> 
> So, that being said, are there any best practices for naming new block device nodes?
> Or is any scheme like /dev/sop[0-9a-z] about as good/bad as any other?
> 
> And, is it a worthwhile idea to pursue adding some sort of shared device namespace
> for block devices to the kernel (maintaining backwards compatibility of device node
> names would of course be required) so that block devices could have some shared
> namespace as scsi devices do?
> 
> Typically the block devices themselves don't care what the device nodes are named, 
> only the userland apps do, though it falls to the block drivers to specify a device
> name via struct gendisk's ->disk_name member being set prior to calling add_disk().
> 
> If there were some kernel interface the block driver could use to get a disk name
> assigned from a shared name space, something like blk_get_new_disk_name(disk->disk_name);
> that could hand out device names like b%s, so you'd get all the block devices which use
> this interface having device names like /dev/bda, /dev/bdb, /dev/bdc, analogous to
> /dev/sda, /dev/sdb, etc. -- the specifics here don't matter to me, the above is just
> an idea off the top of my head -- then, we teach grub about this new block device
> namespace *once*, and force all new block drivers to use it.  Thereafter, adding a
> block driver to the kernel causes no more grub related pain to grub and distro
> developers and users than adding a new scsi hba driver -- i.e. none. 
> 
> Would such a thing be worth pursuing?  Or is there some good reason such a thing
> doesn't already exist?  (Maybe this is more a question for lkml.)

Oh it certainly sounds like a topic for lkml.

-- 
Len Sorensen