RDMA miniconf: Persistent device naming

RDMA miniconf: Persistent device naming

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

Hi All,

I would like to propose the discussion about ability to add
persistent device naming to RDMA subsystem.

Right now, the devices receive their names by simple addition of new free
index to the name (mlx5_0, mlx5_1 ...). Such naming scheme depends on the
PCI probing and in case of device reset can rename device.

Such situation doesn't help to provide constant udev rules, reliable
ibverbs hotplug and advanced "set command" of RDMAtool which at some
point of time will require reinit of ib_device.

In this discussion, I want to present netdev implementation of such
feature, talk about implications on libibverbs and suggest possible
device name convention.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: RDMA miniconf: Persistent device naming

[Dennis Dalessandro]

On 7/31/2017 7:36 AM, Leon Romanovsky wrote:
> Hi All,
> 
> I would like to propose the discussion about ability to add
> persistent device naming to RDMA subsystem.
> 
> Right now, the devices receive their names by simple addition of new free
> index to the name (mlx5_0, mlx5_1 ...). Such naming scheme depends on the
> PCI probing and in case of device reset can rename device.
> 
> Such situation doesn't help to provide constant udev rules, reliable
> ibverbs hotplug and advanced "set command" of RDMAtool which at some
> point of time will require reinit of ib_device.
> 
> In this discussion, I want to present netdev implementation of such
> feature, talk about implications on libibverbs and suggest possible
> device name convention.
> 
> Thanks
> 

Sounds like a good topic to me.

-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA miniconf: Persistent device naming

[Bart Van Assche]

On Mon, 2017-07-31 at 14:36 +0300, Leon Romanovsky wrote:
> I would like to propose the discussion about ability to add
> persistent device naming to RDMA subsystem.
> 
> Right now, the devices receive their names by simple addition of new free
> index to the name (mlx5_0, mlx5_1 ...). Such naming scheme depends on the
> PCI probing and in case of device reset can rename device.
> 
> Such situation doesn't help to provide constant udev rules, reliable
> ibverbs hotplug and advanced "set command" of RDMAtool which at some
> point of time will require reinit of ib_device.
> 
> In this discussion, I want to present netdev implementation of such
> feature, talk about implications on libibverbs and suggest possible
> device name convention.

Hello Leon,

Does this mean that there is user space software that depends on these names?
Shouldn't all user space software be independent of the device names assigned
by the kernel? For e.g. SCSI disks this was solved a long time ago by making
/dev/disk/by-id/${name derived from persistent ID} soft links available. An
example:

$ ls -l /dev/disk/by-id/scsi-35001b44c698c2947
lrwxrwxrwx 1 root root 9 Jul 31 07:53 /dev/disk/by-id/scsi-35001b44c698c2947 -> ../../sda

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA miniconf: Persistent device naming

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1799 bytes --]

On Mon, Jul 31, 2017 at 03:00:47PM +0000, Bart Van Assche wrote:
> On Mon, 2017-07-31 at 14:36 +0300, Leon Romanovsky wrote:
> > I would like to propose the discussion about ability to add
> > persistent device naming to RDMA subsystem.
> >
> > Right now, the devices receive their names by simple addition of new free
> > index to the name (mlx5_0, mlx5_1 ...). Such naming scheme depends on the
> > PCI probing and in case of device reset can rename device.
> >
> > Such situation doesn't help to provide constant udev rules, reliable
> > ibverbs hotplug and advanced "set command" of RDMAtool which at some
> > point of time will require reinit of ib_device.
> >
> > In this discussion, I want to present netdev implementation of such
> > feature, talk about implications on libibverbs and suggest possible
> > device name convention.
>
> Hello Leon,
>
> Does this mean that there is user space software that depends on these names?

Yes, during initialization phase libibverbs reads sysfs entry [1] and
the ibverbs hotplug relies on this information for every
ibv_get_device_list call.

In the RDMA netlink patches, I added notion of device index and the
RDMAtool operates on it, instead of the device name.

> Shouldn't all user space software be independent of the device names assigned
> by the kernel? For e.g. SCSI disks this was solved a long time ago by making
> /dev/disk/by-id/${name derived from persistent ID} soft links available. An
> example:

Yes, it should, and I don't have enough knowledge to say why RDMA is "special"
in this specific case.

>
> $ ls -l /dev/disk/by-id/scsi-35001b44c698c2947
> lrwxrwxrwx 1 root root 9 Jul 31 07:53 /dev/disk/by-id/scsi-35001b44c698c2947 -> ../../sda
>

[1] https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/init.c#L138

> Bart.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: RDMA miniconf: Persistent device naming

[Jason Gunthorpe]

On Mon, Jul 31, 2017 at 03:00:47PM +0000, Bart Van Assche wrote:
> On Mon, 2017-07-31 at 14:36 +0300, Leon Romanovsky wrote:
> > I would like to propose the discussion about ability to add
> > persistent device naming to RDMA subsystem.
> > 
> > Right now, the devices receive their names by simple addition of new free
> > index to the name (mlx5_0, mlx5_1 ...). Such naming scheme depends on the
> > PCI probing and in case of device reset can rename device.
> > 
> > Such situation doesn't help to provide constant udev rules, reliable
> > ibverbs hotplug and advanced "set command" of RDMAtool which at some
> > point of time will require reinit of ib_device.
> > 
> > In this discussion, I want to present netdev implementation of such
> > feature, talk about implications on libibverbs and suggest possible
> > device name convention.
> 
> Hello Leon,
> 
> Does this mean that there is user space software that depends on these names?
> Shouldn't all user space software be independent of the device names assigned
> by the kernel? For e.g. SCSI disks this was solved a long time ago by making
> /dev/disk/by-id/${name derived from persistent ID} soft links available. An
> example:
> 
> $ ls -l /dev/disk/by-id/scsi-35001b44c698c2947
> lrwxrwxrwx 1 root root 9 Jul 31 07:53 /dev/disk/by-id/scsi-35001b44c698c2947 -> ../../sda

The problem with RDMA is it relies heavily on sysfs, and the sysfs
path name is what is being proposed to be changable.

The best fix is going to be to consistently use the new stable index
to access all information, which means effectively obsoleting sysfs.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA miniconf: Persistent device naming

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2157 bytes --]

On Mon, Jul 31, 2017 at 10:47:05AM -0600, Jason Gunthorpe wrote:
> On Mon, Jul 31, 2017 at 03:00:47PM +0000, Bart Van Assche wrote:
> > On Mon, 2017-07-31 at 14:36 +0300, Leon Romanovsky wrote:
> > > I would like to propose the discussion about ability to add
> > > persistent device naming to RDMA subsystem.
> > >
> > > Right now, the devices receive their names by simple addition of new free
> > > index to the name (mlx5_0, mlx5_1 ...). Such naming scheme depends on the
> > > PCI probing and in case of device reset can rename device.
> > >
> > > Such situation doesn't help to provide constant udev rules, reliable
> > > ibverbs hotplug and advanced "set command" of RDMAtool which at some
> > > point of time will require reinit of ib_device.
> > >
> > > In this discussion, I want to present netdev implementation of such
> > > feature, talk about implications on libibverbs and suggest possible
> > > device name convention.
> >
> > Hello Leon,
> >
> > Does this mean that there is user space software that depends on these names?
> > Shouldn't all user space software be independent of the device names assigned
> > by the kernel? For e.g. SCSI disks this was solved a long time ago by making
> > /dev/disk/by-id/${name derived from persistent ID} soft links available. An
> > example:
> >
> > $ ls -l /dev/disk/by-id/scsi-35001b44c698c2947
> > lrwxrwxrwx 1 root root 9 Jul 31 07:53 /dev/disk/by-id/scsi-35001b44c698c2947 -> ../../sda
>
> The problem with RDMA is it relies heavily on sysfs, and the sysfs
> path name is what is being proposed to be changable.
>
> The best fix is going to be to consistently use the new stable index
> to access all information, which means effectively obsoleting sysfs.
>

I want to take RDMA one step more,
https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

I called the topic name "persistent", but the right name is "predictable".

Thanks

> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: RDMA miniconf: Persistent device naming

[Jason Gunthorpe]

On Mon, Jul 31, 2017 at 07:54:53PM +0300, Leon Romanovsky wrote:
> I want to take RDMA one step more,
> https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
> 
> I called the topic name "persistent", but the right name is "predictable".

Once renaming is possible then having udev rework the names isn't a
big step anymore.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA miniconf: Persistent device naming

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]

On Mon, Jul 31, 2017 at 11:00:18AM -0600, Jason Gunthorpe wrote:
> On Mon, Jul 31, 2017 at 07:54:53PM +0300, Leon Romanovsky wrote:
> > I want to take RDMA one step more,
> > https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
> >
> > I called the topic name "persistent", but the right name is "predictable".
>
> Once renaming is possible then having udev rework the names isn't a
> big step anymore.

You need predictable starting point to perform automatic rename,
currently neither index, neither names are not.

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: RDMA miniconf: Persistent device naming

[Jason Gunthorpe]

On Mon, Jul 31, 2017 at 08:08:04PM +0300, Leon Romanovsky wrote:
> On Mon, Jul 31, 2017 at 11:00:18AM -0600, Jason Gunthorpe wrote:
> > On Mon, Jul 31, 2017 at 07:54:53PM +0300, Leon Romanovsky wrote:
> > > I want to take RDMA one step more,
> > > https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
> > >
> > > I called the topic name "persistent", but the right name is "predictable".
> >
> > Once renaming is possible then having udev rework the names isn't a
> > big step anymore.
> 
> You need predictable starting point to perform automatic rename,
> currently neither index, neither names are not.

udev has all the information needed to produce the naming pattern in
the link above.

You'd end up with names like rdmap3s1f1 for instance.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA miniconf: Persistent device naming

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]

On Mon, Jul 31, 2017 at 11:24:36AM -0600, Jason Gunthorpe wrote:
> On Mon, Jul 31, 2017 at 08:08:04PM +0300, Leon Romanovsky wrote:
> > On Mon, Jul 31, 2017 at 11:00:18AM -0600, Jason Gunthorpe wrote:
> > > On Mon, Jul 31, 2017 at 07:54:53PM +0300, Leon Romanovsky wrote:
> > > > I want to take RDMA one step more,
> > > > https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
> > > >
> > > > I called the topic name "persistent", but the right name is "predictable".
> > >
> > > Once renaming is possible then having udev rework the names isn't a
> > > big step anymore.
> >
> > You need predictable starting point to perform automatic rename,
> > currently neither index, neither names are not.
>
> udev has all the information needed to produce the naming pattern in
> the link above.
>
> You'd end up with names like rdmap3s1f1 for instance.

I'm not so convinced that udev scripts digging into various outputs
(lspci, e.t.c.) are better solution than kernel handles everything
exactly as netdev.

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: RDMA miniconf: Persistent device naming

[Jason Gunthorpe]

On Mon, Jul 31, 2017 at 08:38:13PM +0300, Leon Romanovsky wrote:
> On Mon, Jul 31, 2017 at 11:24:36AM -0600, Jason Gunthorpe wrote:
> > On Mon, Jul 31, 2017 at 08:08:04PM +0300, Leon Romanovsky wrote:
> > > On Mon, Jul 31, 2017 at 11:00:18AM -0600, Jason Gunthorpe wrote:
> > > > On Mon, Jul 31, 2017 at 07:54:53PM +0300, Leon Romanovsky wrote:
> > > > > I want to take RDMA one step more,
> > > > > https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
> > > > >
> > > > > I called the topic name "persistent", but the right name is "predictable".
> > > >
> > > > Once renaming is possible then having udev rework the names isn't a
> > > > big step anymore.
> > >
> > > You need predictable starting point to perform automatic rename,
> > > currently neither index, neither names are not.
> >
> > udev has all the information needed to produce the naming pattern in
> > the link above.
> >
> > You'd end up with names like rdmap3s1f1 for instance.
> 
> I'm not so convinced that udev scripts digging into various outputs
> (lspci, e.t.c.) are better solution than kernel handles everything
> exactly as netdev.

I'm confused. You linked to PredictableNetworkInterfaceNames, which is
entirely implemented by udev.

The only kernel support for netdev is the ability to rename netdevs,
everything else is done in userspace, by udev on modern systems.

The sooner we get the same kernel API for RDMA the better, I'd run it
over rdma netlink I think.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA miniconf: Persistent device naming

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1891 bytes --]

On Mon, Jul 31, 2017 at 11:43:12AM -0600, Jason Gunthorpe wrote:
> On Mon, Jul 31, 2017 at 08:38:13PM +0300, Leon Romanovsky wrote:
> > On Mon, Jul 31, 2017 at 11:24:36AM -0600, Jason Gunthorpe wrote:
> > > On Mon, Jul 31, 2017 at 08:08:04PM +0300, Leon Romanovsky wrote:
> > > > On Mon, Jul 31, 2017 at 11:00:18AM -0600, Jason Gunthorpe wrote:
> > > > > On Mon, Jul 31, 2017 at 07:54:53PM +0300, Leon Romanovsky wrote:
> > > > > > I want to take RDMA one step more,
> > > > > > https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
> > > > > >
> > > > > > I called the topic name "persistent", but the right name is "predictable".
> > > > >
> > > > > Once renaming is possible then having udev rework the names isn't a
> > > > > big step anymore.
> > > >
> > > > You need predictable starting point to perform automatic rename,
> > > > currently neither index, neither names are not.
> > >
> > > udev has all the information needed to produce the naming pattern in
> > > the link above.
> > >
> > > You'd end up with names like rdmap3s1f1 for instance.
> >
> > I'm not so convinced that udev scripts digging into various outputs
> > (lspci, e.t.c.) are better solution than kernel handles everything
> > exactly as netdev.
>
> I'm confused. You linked to PredictableNetworkInterfaceNames, which is
> entirely implemented by udev.

I linked to that article to save copy/paste of advantages of such scheme.

>
> The only kernel support for netdev is the ability to rename netdevs,
> everything else is done in userspace, by udev on modern systems.

I planned to take a deep look on the gaps prior to LPC, but in meanwhile, you
are right and it is done in udev since 197.

>
> The sooner we get the same kernel API for RDMA the better, I'd run it
> over rdma netlink I think.

Nice, I found who will present it :) Can you prepare short topic about it?

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]