Proposal for adding ib_usa to the Linux Infiniband Subsystem

Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Mike Heinz]

Hello, all.

For the past few years, we (QLogic) have been using the IB user space SA module that was originally written by Sean Hefty. We think it would be great if this module were to become part of the official Linux Infiniband subsystem - but rather than simply dumping this on Roland as a surprise, I thought it might be a good idea to talk about it, first, and find out what the best approach would be.


What is ib_usa?
---------------
The ib_usa module adds two features to the Infiniband Subsystem. It allows user applications to subscribe to SA notifications (remote port up, remote port down, etc.) and it allows user applications to subscribe to multicast groups. It does this via a device file, /dev/infiniband/ib_usa, similar to /dev/infiniband/uverbs and /dev/infiniband/umad.


Why is this needed?
-------------------
The SA protocol allows notices and multicast membership to be managed at a CA port level.  As a result, if multiple processes want to receive notices or join multicast groups, that registration must be coordinated per CA port.  Existing code in the Linux kernel exists for this purpose, coordinating the needs of various Infiniband kernel modules and presenting a single "per server CA port" perspective to the SA.  The IB user space SA module provides a mechanism for user space applications to take advantage of this existing kernel code for managing SA registrations.

By providing this mechanism, ib_usa provides a convenient and useful way for user applications to detect when nodes enter or leave the fabric. This is useful for fabric monitoring applications (like a system admin dashboard) or, conceivably, to allow a cluster application to detect problems and to dynamically balance a computational load across the fabric.

In addition, the multicast functionality will allow user applications to use properly verbs to receive multicast traffic.


How will it integrate into the existing Infiniband subsystem?
-------------------------------------------------------------
We recommend including Sean's original module, and we can provide a version of it that works fine with the latest versions of OFED. This will add a new kernel module to the ofa_kernel package, ib_usa.ko. 

In addition, there will be a need for a shared library to provide an API to provide access to ib_usa. I would suggest adding the needed functions to OFED management module, as part of the libibumad library, or to their own libibusa library. In either case, I would provide the needed code and a sample/test application.

So, how about it? What's the best way forward here?



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Sean Hefty]

>Why is this needed?
>-------------------
>The SA protocol allows notices and multicast membership to be managed at a CA
>port level.  As a result, if multiple processes want to receive notices or join
>multicast groups, that registration must be coordinated per CA port.  Existing
>code in the Linux kernel exists for this purpose, coordinating the needs of
>various Infiniband kernel modules and presenting a single "per server CA port"
>perspective to the SA.  The IB user space SA module provides a mechanism for
>user space applications to take advantage of this existing kernel code for
>managing SA registrations.

The AF_IB support that is being requested for the rdma_cm would provide greater
support for IB multicast join/leave membership.  Some additional work would be
needed to allow the user to specify the full MCMemberRecord, but the basic
infrastructure should be there.  Just mentioning this as a possibility.  (The IB
ACM code joins multicast groups using libibumad and sends MADs directly to the
SA.)

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Mike Heinz]

Would it be better to omit the multicast support from ib_usa and simply add it as a way to handle notifications?

-----Original Message-----
From: Sean Hefty [mailto:sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org] 
Sent: Friday, May 21, 2010 11:30 AM
To: Mike Heinz; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: RE: Proposal for adding ib_usa to the Linux Infiniband Subsystem

>Why is this needed?
>-------------------
>The SA protocol allows notices and multicast membership to be managed at a CA
>port level.  As a result, if multiple processes want to receive notices or join
>multicast groups, that registration must be coordinated per CA port.  Existing
>code in the Linux kernel exists for this purpose, coordinating the needs of
>various Infiniband kernel modules and presenting a single "per server CA port"
>perspective to the SA.  The IB user space SA module provides a mechanism for
>user space applications to take advantage of this existing kernel code for
>managing SA registrations.

The AF_IB support that is being requested for the rdma_cm would provide greater
support for IB multicast join/leave membership.  Some additional work would be
needed to allow the user to specify the full MCMemberRecord, but the basic
infrastructure should be there.  Just mentioning this as a possibility.  (The IB
ACM code joins multicast groups using libibumad and sends MADs directly to the
SA.)

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Sean Hefty]

>Would it be better to omit the multicast support from ib_usa and simply add it
>as a way to handle notifications?

I think so, since it limits the number of interfaces and libraries that a user
needs to learn.

I'm not sure of the best userspace interface for notifications.  Maybe there's a
way to extend the async event reporting in libibverbs?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Jason Gunthorpe]

I've said this before, but again..

There are too many orthogonal interfaces here. We really don't need
more libraries or kernel modules or /dev/ devices. Really.

Subscribing and multiplexing notifications from a GSI service should
just be a general function in the mad code..

On Fri, May 21, 2010 at 10:57:48AM -0500, Mike Heinz wrote:
> Would it be better to omit the multicast support from ib_usa and simply add it as a way to handle notifications?
> 
> From: Sean Hefty [mailto:sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org] 
> Sent: Friday, May 21, 2010 11:30 AM
> To: Mike Heinz; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: RE: Proposal for adding ib_usa to the Linux Infiniband Subsystem
> 
> >Why is this needed?
> >The SA protocol allows notices and multicast membership to be managed at a CA
> >port level.? As a result, if multiple processes want to receive notices or join
> >multicast groups, that registration must be coordinated per CA port.? Existing
> >code in the Linux kernel exists for this purpose, coordinating the needs of
> >various Infiniband kernel modules and presenting a single "per server CA port"
> >perspective to the SA.? The IB user space SA module provides a mechanism for
> >user space applications to take advantage of this existing kernel code for
> >managing SA registrations.
> 
> The AF_IB support that is being requested for the rdma_cm would provide greater
> support for IB multicast join/leave membership.  Some additional work would be
> needed to allow the user to specify the full MCMemberRecord, but the basic
> infrastructure should be there.  Just mentioning this as a possibility.  (The IB
> ACM code joins multicast groups using libibumad and sends MADs directly to the
> SA.)
> 
> - Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Mike Heinz]

I have to agree that in some ways this seems a good fit with ibverbs. Also, without the multicast support, the whole thing is well under 1 kloc in the kernel, making it a good candidate for being combined with another module.

My biggest objections, though, would be:

1. If it's been working for a couple of years now, do we really need to change it?
2. The async events in ibverbs are all local events, are we really keeping things simple by adding remote notifications to mix, or making a simple interface much more complicated?

-----Original Message-----
From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Jason Gunthorpe
Sent: Friday, May 21, 2010 12:50 PM
To: Mike Heinz
Cc: Sean Hefty; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: Proposal for adding ib_usa to the Linux Infiniband Subsystem

I've said this before, but again..

There are too many orthogonal interfaces here. We really don't need
more libraries or kernel modules or /dev/ devices. Really.

Subscribing and multiplexing notifications from a GSI service should
just be a general function in the mad code..

On Fri, May 21, 2010 at 10:57:48AM -0500, Mike Heinz wrote:
> Would it be better to omit the multicast support from ib_usa and simply add it as a way to handle notifications?
> 
> From: Sean Hefty [mailto:sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org] 
> Sent: Friday, May 21, 2010 11:30 AM
> To: Mike Heinz; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: RE: Proposal for adding ib_usa to the Linux Infiniband Subsystem
> 
> >Why is this needed?
> >The SA protocol allows notices and multicast membership to be managed at a CA
> >port level.? As a result, if multiple processes want to receive notices or join
> >multicast groups, that registration must be coordinated per CA port.? Existing
> >code in the Linux kernel exists for this purpose, coordinating the needs of
> >various Infiniband kernel modules and presenting a single "per server CA port"
> >perspective to the SA.? The IB user space SA module provides a mechanism for
> >user space applications to take advantage of this existing kernel code for
> >managing SA registrations.
> 
> The AF_IB support that is being requested for the rdma_cm would provide greater
> support for IB multicast join/leave membership.  Some additional work would be
> needed to allow the user to specify the full MCMemberRecord, but the basic
> infrastructure should be there.  Just mentioning this as a possibility.  (The IB
> ACM code joins multicast groups using libibumad and sends MADs directly to the
> SA.)
> 
> - Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Jason Gunthorpe]

On Fri, May 21, 2010 at 12:04:17PM -0500, Mike Heinz wrote:

> I have to agree that in some ways this seems a good fit with
> ibverbs. Also, without the multicast support, the whole thing is
> well under 1 kloc in the kernel, making it a good candidate for
> being combined with another module.

verbs? Why verbs?

umad. The only thing that is needed is to subscribe to a GSI
notification, replicate incoming notifications and route them as
normal mads over the umad interface to all subscribers..

With some cleverness you could probably make the subscribe/unsubscribe
fairly general..

Remember SA notifications are not the only unsolicited events that can
occure in the GSI framework.

> 1. If it's been working for a couple of years now, do we really need
> to change it?

We need to live with it forever once it is accepted..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Mike Heinz]

Jason,

I mentioned verbs because Sean had suggested it and, after all, he's the original author of ib_usa.

However - you're wrong about "the only thing needed" being to subscribe to a GSI notification; notifications are sent to ports, not to individual queue pairs or processes - that's why the ib_sa module has a mechanism for for managing notifications. Like ib_umad, ib_usa is the mechanism for extending ib_sa into user space. 

-----Original Message-----
From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org] 
Sent: Friday, May 21, 2010 1:20 PM
To: Mike Heinz
Cc: Sean Hefty; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: Proposal for adding ib_usa to the Linux Infiniband Subsystem

On Fri, May 21, 2010 at 12:04:17PM -0500, Mike Heinz wrote:

> I have to agree that in some ways this seems a good fit with
> ibverbs. Also, without the multicast support, the whole thing is
> well under 1 kloc in the kernel, making it a good candidate for
> being combined with another module.

verbs? Why verbs?

umad. The only thing that is needed is to subscribe to a GSI
notification, replicate incoming notifications and route them as
normal mads over the umad interface to all subscribers..

With some cleverness you could probably make the subscribe/unsubscribe
fairly general..

Remember SA notifications are not the only unsolicited events that can
occure in the GSI framework.

> 1. If it's been working for a couple of years now, do we really need
> to change it?

We need to live with it forever once it is accepted..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Jason Gunthorpe]

On Fri, May 21, 2010 at 12:29:31PM -0500, Mike Heinz wrote:

>> umad. The only thing that is needed is to subscribe to a GSI
>> notification, replicate incoming notifications and route them as
>> normal mads over the umad interface to all subscribers..

> However - you're wrong about "the only thing needed" being to
> subscribe to a GSI notification; notifications are sent to ports,
> not to individual queue pairs or processes - that's why the ib_sa
> module has a mechanism for for managing notifications. Like ib_umad,
> ib_usa is the mechanism for extending ib_sa into user space.

To be fair I had two things in that statement..

umad instances are per-port. You do an ioctl or something on the umad
FD to register for the notification, and the umad module replicates an
incoming notification GMP to all registrants. This seems totally
straightforward to me..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Sean Hefty]

>verbs? Why verbs?
>
>umad. The only thing that is needed is to subscribe to a GSI
>notification, replicate incoming notifications and route them as
>normal mads over the umad interface to all subscribers..

IMO, someone needs to propose a specific user space API that provides this
capability.  Personally, I don't think MADs provide the desired interface.

I suggested looking at the libibverbs async events, but I didn't take the time
to look at the details of how well that API would fit.  It's used to report
client_reregister events, so using it to report notifications doesn't seem too
out of line.

What's missing is a mechanism to register for the event, which does map better
to umad.  Maybe the kernel user_mad module snoops MADs and reacts accordingly.

Isn't access to the umad interface usually privileged?  If so, is that
acceptable for the anticipated usage?

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Mike Heinz]

> IMO, someone needs to propose a specific user space API that provides this
> capability.  Personally, I don't think MADs provide the desired interface.

Agreed. I also agree with Jason that the proliferation of interfaces and modules is an impediment to IB adoption.

And if we need to get the API nailed down first, so be it, that's why I made the proposal the way I did - to get feedback on what was needed.

Let me spend a few days reviewing libibumad and libibverbs, see if can get an idea of what would be the better fit.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Proposal for adding ib_usa to the Linux Infiniband Subsystem

[Jason Gunthorpe]

On Fri, May 21, 2010 at 10:57:17AM -0700, Sean Hefty wrote:

> IMO, someone needs to propose a specific user space API that provides this
> capability.  Personally, I don't think MADs provide the desired interface.

My thinking was sort of:

fd = open("/dev/..umad..");
ioctl(fd, SUBSCRIBE_TRAP, &trap_description)
read(fd..); // Events arrive
close(fd);  // Event is unsubscribed

The reason this fits nicely with umad is that you really actually want
the GMP trap, not some processed version.. So this is like a multicast
subscribe in IP.

So.. The main question is how to do formulate a SUBSCRIBE_TRAP
operation that:
 1) Can describe GSI traps for multiple managers, not just SA
 2) Specifies how to filter incoming GMPs
 3) Lets the kernel send and refcount the trap subscribe message
 4) Lets the kernel send an unsubscribe message.
 5) Lets the kernel send a single repress message if necessary

The first two seem easy, use the trap ID, class ID, and possibly LID
as well.

I think the latter three are not too bad, the subscribes/unsubscribe
messages are pretty uniform. Tricky bit might be the ref counting?

Mostly just reading the spec to see..

> I suggested looking at the libibverbs async events, but I didn't
> take the time to look at the details of how well that API would fit.
> It's used to report client_reregister events, so using it to report
> notifications doesn't seem too out of line.

This seems reasonable to me for some high level notices that are
commonly used and shouldn't require privilege to access. But if you
want to access raw trap/notices then umad seems a better bet.

To me, if it is OK for the kernel to always subscribe to the
trap/notice then async events seem to be a good choice. I'm not sure
the kernel should unconditionally subscribe to things like gid in/out
of service, etc..

> Isn't access to the umad interface usually privileged?  If so, is that
> acceptable for the anticipated usage?

Subscribing to an arbitary trap should be privileged.. IHMO.

But I agree, is there a single specific trap you are trying to access
with ib_usa??

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html