Re: [PATCH for-next V2 02/22] IB/core: change pkey table lookups to support full and partial membership for the same pkey

[Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>]

On Wednesday 12 September 2012 19:48, Doug Ledford wrote:
> > On the Hypervisor, however, we assume that if both versions of the pkey are in its pkey table,
> > then for its own infiniband operation (as opposed to performing its pkey virtualizing function),
> > it should operate with the highest membership type in its table for a given 15-bit pkey.
> 
> That's what I was looking for.  So, how can you know this assumption is
> correct?  It seems to me that if someone wanted to restrict membership
> of the hypervisor as part of a security lockdown, then give full
> membership to a guest because that guest is some high security, single
> task guest, then this assumption would break things (the user would be
> able to assign the full membership key to the guest OK, but regardless
> of how they wanted the hypervisor to be subscribed to that particular
> pkey, it would always get the full membership from the guest).
> 
This issue, unfortunately, opens up a real "can of worms".
ib_find_cached_pkey() is used by the CM in determining the (15-bit) p_key
to be used for the connection (although 16-bit pkeys are placed in the
CM_REQ message (see IB Spec 1.2.1, table 99, page 667). The REQ handler on
the remote side finds the index in its pkey table which contains the
(15-bit) pkey enclosed in the REQ message.  This index is then used
when creating the local RC qp as its pkey_index.
AFAIK, no check is performed regarding compatible membership forms (i.e.,
that at least one of the two sides has the full membership form of the
15-bit pkey) -- it is the network administrator's responsibility to see
that the pkey configurations are correct.

Up to now, the ib_core driver has assumed that only 1 membership form per
15-bit pkey is contained in the local pkey table.

Now that both forms may exist in the Hypervisor's table, we do have a problem.
1. We cannot depend on ib_find_cached_pkey simply finding the first 15-bit pkey
   which matches its search, since we cannot depend on the order of full vs limited
   membership forms appearing in the pkey table (the order is not controllable -- see
   the example at the end of this post).
2. We therefore need a consistent policy for preferentially retrieving either the full
   or the partial member for a given 15-bit pkey.
3. This policy must be configurable by the administrator per host, per HCA.

The problems that I see:
1. What if the Hypervisor needs to have limited membership for some connections, but
   full membership for others?  Such a split policy is complex to implement -- it would
   require specifying PER PKEY whether the preferential return should be full, or should
   be limited.
2. What if the Hypervisor has several HCAs, and wishes a different policy for each HCA?

There may be more problems as well.  Liran?

In any event, the issue of multiple pkey forms in a given pkey table is actually separate
from SRIOV (it's just that SRIOV needs multiple forms, so the issue came up).
Solving it will require changes in the ib core driver.

I don't yet have a concrete proposal to fix this at present.  Any ideas would be appreciated.

-Jack

P.S. We have the same issue in procedure ib_find_pkey(), also in this patch, in core/device.c .
This procedure is used in lots of places: ipoib, core/multicast, sa_query.

I seem to recall that there were problems with IPoIB when partial membership pkeys are used.
Liran, do you recall something?

======================================
It is also impossible to demand that the pkey table be ordered so that if both limited
and full pkeys are present, then the full membership value always appears first.

A simple example:
  1. Fill the entire pkey table with full-membership pkeys. Say pkey A and pkey B are both
     in the table, and that pkey A appears first.

  2. Delete full-membership pkey A, and add the limited-membership value of Pkey B. This
     new limited-pkey-B will be entered into the pkey table by OpenSM in the position just
     vacated by the deleted pkey A -- it is the only available slot!.

  3. The Limited-membership pkey-B value will therefore unavoidably appear in the table
     before the full-membership pkey-B value.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html