[PATCH 0/3] Add support for specific Lustre Qos

[PATCH 0/3] Add support for specific Lustre Qos

[sebastien dugue]

  These patches allow to have a specific QoS for Lustre traffic.

   - Patch 1 adds a new Lustre port space to the rdma_cm kernel module

   - Patch 2 defines a new Lustre service ID (matching the new RDMA CMA port
     space) and a new 'lustre' keyword in the OpenSM QoS policy parser.

   - Patch 3 adds a new port_space parameter for the Lustre ko2iblnd kernel
     module that allows to specify the port space to be used by LNET. This
     patch has already been submitted to the Lustre community and is only
     given here as a reference. The relevant discussion is at
     https://bugzilla.lustre.org/show_bug.cgi?id=21732

  In order the exercise QoS using this Lustre port space, the following setup
may be used:

  1. OpenSM configuration - etc/opensm/opensm.conf

    #
    # QoS OPTIONS
    #
    # Enable QoS setup
    qos TRUE
    # QoS policy file to be used
    qos_policy_file /etc/opensm/qos­policy.conf
    # QoS default options
    qos_max_vls    8
    qos_high_limit 1
    qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
    qos_vlarb_low  0:1,1:16,2:32,3:64,4:1,5:0
    qos_sl2vl      0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

    The VLARB weights are only indicative.


  2. OpenSM QoS policy configuration - /etc/opensm/qos-policy.conf:

    qos­ulps 
        default                 	: 0 # default SL
        lustre, target­port­guid 0x1234 : 1 # lustre traffic to a specific
					    # port guid (ex MDS traffic)
        lustre                  	: 2 # lustre default traffic
    end­qos­ulps 

  3. Lustre module configuration - /etc/modprobe.d/lustre.conf

     Add the following line to all the Lustre nodes:

     options ko2iblnd port_space=339


  Sebastien.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[sebastien dugue]

  This patch adds a new port space for use by Lustre traffic.

  This, along with patches to OpenSM and Lustre, allow to define a
specific QoS for lustre.

Signed-off-by: Sebastien Dugue <sebastien.dugue-6ktuUTfB/bM@public.gmane.org>
---
 drivers/infiniband/core/cma.c |    5 +++++
 include/rdma/rdma_cm.h        |   11 ++++++-----
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index cc9b594..6b9e75e 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -78,6 +78,7 @@ static DEFINE_IDR(sdp_ps);
 static DEFINE_IDR(tcp_ps);
 static DEFINE_IDR(udp_ps);
 static DEFINE_IDR(ipoib_ps);
+static DEFINE_IDR(lustre_ps);
 static int next_port;
 
 struct cma_device {
@@ -2066,6 +2067,9 @@ static int cma_get_port(struct rdma_id_private *id_priv)
 	case RDMA_PS_IPOIB:
 		ps = &ipoib_ps;
 		break;
+	case RDMA_PS_LUSTRE:
+		ps = &lustre_ps;
+		break;
 	default:
 		return -EPROTONOSUPPORT;
 	}
@@ -3034,6 +3038,7 @@ static void __exit cma_cleanup(void)
 	idr_destroy(&tcp_ps);
 	idr_destroy(&udp_ps);
 	idr_destroy(&ipoib_ps);
+	idr_destroy(&lustre_ps);
 }
 
 module_init(cma_init);
diff --git a/include/rdma/rdma_cm.h b/include/rdma/rdma_cm.h
index c6b2962..d662799 100644
--- a/include/rdma/rdma_cm.h
+++ b/include/rdma/rdma_cm.h
@@ -63,11 +63,12 @@ enum rdma_cm_event_type {
 };
 
 enum rdma_port_space {
-	RDMA_PS_SDP   = 0x0001,
-	RDMA_PS_IPOIB = 0x0002,
-	RDMA_PS_TCP   = 0x0106,
-	RDMA_PS_UDP   = 0x0111,
-	RDMA_PS_SCTP  = 0x0183
+	RDMA_PS_SDP    = 0x0001,
+	RDMA_PS_IPOIB  = 0x0002,
+	RDMA_PS_TCP    = 0x0106,
+	RDMA_PS_UDP    = 0x0111,
+	RDMA_PS_LUSTRE = 0x0153,
+	RDMA_PS_SCTP   = 0x0183
 };
 
 struct rdma_addr {
-- 
1.6.0.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[Sean Hefty]

>  This patch adds a new port space for use by Lustre traffic.
>
>  This, along with patches to OpenSM and Lustre, allow to define a
>specific QoS for lustre.

In general, I don't like the idea of application port spaces for QoS
support.  Can't this be done using port numbers in the existing port
space?

>+	RDMA_PS_LUSTRE = 0x0153,

It looks like this number maps to the VINES protocol.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[Roland Dreier]

 > In general, I don't like the idea of application port spaces for QoS
 > support.  Can't this be done using port numbers in the existing port
 > space?

I agree.  If setting QoS requires a kernel patch for each application, I
think we've messed up QoS somehow.  There needs to be a way to control
QoS via configuration, rather than rebuilding.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[sebastien dugue]

  Hi Roland.

On Wed, 13 Jan 2010 09:04:19 -0800
Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:

> 
>  > In general, I don't like the idea of application port spaces for QoS
>  > support.  Can't this be done using port numbers in the existing port
>  > space?
> 
> I agree.  If setting QoS requires a kernel patch for each application, I
> think we've messed up QoS somehow.  There needs to be a way to control
> QoS via configuration, rather than rebuilding.

  Right, it's possible via configuration only, but you then cannot separate
the different Lustre traffics (MDS and OSS for example).

  Sebastien.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[Roland Dreier]

 > > I agree.  If setting QoS requires a kernel patch for each application, I
 > > think we've messed up QoS somehow.  There needs to be a way to control
 > > QoS via configuration, rather than rebuilding.
 > 
 >   Right, it's possible via configuration only, but you then cannot separate
 > the different Lustre traffics (MDS and OSS for example).

I guess I don't know enough about Lustre to know why this would be so.
How does creating a new port space for Lustre help with this?  Why can't
one do the same thing in one of the existing port spaces?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[sebastien dugue]

On Thu, 14 Jan 2010 09:25:31 -0800
Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:

> 
>  > > I agree.  If setting QoS requires a kernel patch for each application, I
>  > > think we've messed up QoS somehow.  There needs to be a way to control
>  > > QoS via configuration, rather than rebuilding.
>  > 
>  >   Right, it's possible via configuration only, but you then cannot separate
>  > the different Lustre traffics (MDS and OSS for example).
> 
> I guess I don't know enough about Lustre to know why this would be so.
> How does creating a new port space for Lustre help with this?  Why can't
> one do the same thing in one of the existing port spaces?

  Well, without a specific port space, the default for Lustre is to use the
TCP port space so you cannot distinguish Lustre traffic from other traffic using
that same port space.

  Sebastien.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[Roland Dreier]

 >   Well, without a specific port space, the default for Lustre is to use the
 > TCP port space so you cannot distinguish Lustre traffic from other traffic using
 > that same port space.

I'm still a bit confused.  The problem as I understand it is that Lustre
always uses the same TCP port, so there's no way to apply different QoS
to different types of Lustre traffic.  But if we create a new port space
and don't change the ports that Lustre uses, then there's still no way
to apply different QoS for different Lustre traffic.  So I guess you
need to change the ports used within the new port space -- but then why
can't you just stay in the TCP space but change the ports used?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[sebastien dugue]

  Hi Roland,

On Tue, 19 Jan 2010 17:12:43 -0800
Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:

> 
>  >   Well, without a specific port space, the default for Lustre is to use the
>  > TCP port space so you cannot distinguish Lustre traffic from other traffic using
>  > that same port space.
> 
> I'm still a bit confused.  The problem as I understand it is that Lustre
> always uses the same TCP port, so there's no way to apply different QoS
> to different types of Lustre traffic.

  Right, all Lustre servers (MDS and OSS) use the same TCP port, so there's no
way to distinguish between the 2 types of traffic.

>  But if we create a new port space
> and don't change the ports that Lustre uses, then there's still no way
> to apply different QoS for different Lustre traffic.

  With the OpenSM patch and the Lustre port space, you can map Lustre traffic
to specific target port GUIDs onto a specific SL hence separating the flows.

>  So I guess you
> need to change the ports used within the new port space -- but then why
> can't you just stay in the TCP space but change the ports used?

  No, with the new port space, there's no need to change ports. You only need
to specify the target GUIDs.

  For example:

    qos-­ulps 
        default                 	      : 0 # default SL
        lustre, target-­port­guid 0x1234,0x1235 : 1 # lustre traffic to MDSs
        lustre                  	      : 2 # default lustre traffic (to OSSs)
    end­-qos-­ulps


  Hope this helps clarify things a bit.

  Thanks,

  Sébastien.




--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[Or Gerlitz]

sebastien dugue wrote:
>> So I guess you need to change the ports used within the new port space -- but then 
>> why can't you just stay in the TCP space but change the ports used?
>>     
>
> No, with the new port space, there's no need to change ports. You only need to specify the target GUIDs. For example:
>         lustre, target-­port­guid 0x1234,0x1235 : 1 # lustre traffic to MDSs
>         lustre                  	      : 2 # default lustre traffic (to OSSs)
> Hope this helps clarify things a bit.
>   
sorry, but it doesn't,  as far as I understand there are three 
possibilities for what the string "lustre" is being translated to
by the opensm QoS logic:

(A) lustre port in the TCP port space
(B) lustre port space
(C) nothing (that is not a service, in the same manner that ipoib just 
doesn't mean anything to opensm)

Assuming C is not the case, then either A or B will yield the same 
result and as such the new port space buys you nothing.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[sebastien dugue]

On Wed, 20 Jan 2010 10:03:47 +0200
Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:

> sebastien dugue wrote:
> >> So I guess you need to change the ports used within the new port space -- but then 
> >> why can't you just stay in the TCP space but change the ports used?
> >>     
> >
> > No, with the new port space, there's no need to change ports. You only need to specify the target GUIDs. For example:
> >         lustre, target-­port­guid 0x1234,0x1235 : 1 # lustre traffic to MDSs
> >         lustre                  	      : 2 # default lustre traffic (to OSSs)
> > Hope this helps clarify things a bit.
> >   
> sorry, but it doesn't,  as far as I understand there are three 
> possibilities for what the string "lustre" is being translated to
> by the opensm QoS logic:
> 
> (A) lustre port in the TCP port space
> (B) lustre port space
> (C) nothing (that is not a service, in the same manner that ipoib just 
> doesn't mean anything to opensm)

  I'm not sure I'm following you here.

  With the patches, the 'lustre' keyword is being translated to the lustre
port space and only to that port space, never to the TCP port space. So if
I'm understanding you correctly, it's B. 

> 
> Assuming C is not the case, then either A or B will yield the same 
> result and as such the new port space buys you nothing.

  No, because in OpenSM's QoS logic, there's no way to map the TCP port
space with specific target GUIDs onto an SL. You have keywords for SDP, SRP,
RDS, ISER, ... but not for the TCP port space (or am I missing something?).

  What B buys you is the ability to map Lustre traffic to a specific
target GUID onto an SL.

  An alternative would be to be able to map the TCP port space to a specific
port and specific target GUIDs onto an SL in OpenSM's QoS configuration, but
that's currently not possible.

  Sébastien.




--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[Or Gerlitz]

sebastien dugue wrote:
>  No, because in OpenSM's QoS logic, there's no way to map the TCP port
> space with specific target GUIDs onto an SL. You have keywords for SDP, SRP,
> RDS, ISER, ... but not for the TCP port space (or am I missing something?).

going with this, what prevents you from patching opensm qos engine to support
the lustre service under the tcp port-space and/or support a combination of service 
and target port-guid? all in all, first, I don't see what a kernel patch buys you
and second, if it buys you something you should be able to gain the same effect with
patching open-sm.

thinking on this a bit more, since the rules are processed by order wouldn't the 
following scheme let you achieve the same effect?

target-­port­guid 0x1234,0x1235 : 1 # traffic to MDSs
lustre                        : 2 # default lustre traffic (to OSSs)

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[sebastien dugue]

  Hi Or,

On Wed, 20 Jan 2010 17:26:17 +0200
Or Gerlitz <ogerlitz-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> wrote:

> sebastien dugue wrote:
> >  No, because in OpenSM's QoS logic, there's no way to map the TCP port
> > space with specific target GUIDs onto an SL. You have keywords for SDP, SRP,
> > RDS, ISER, ... but not for the TCP port space (or am I missing something?).
> 
> going with this, what prevents you from patching opensm qos engine to support
> the lustre service under the tcp port-space and/or support a combination of service 
> and target port-guid? all in all, first, I don't see what a kernel patch buys you
> and second, if it buys you something you should be able to gain the same effect with
> patching open-sm.

  Right, I can indeed achieve that with only a patch to OpenSM. Going with a
specific Lustre port space is simple and less intrusive although you have to
patch the kernel and OpenSM.

  OK, then going with the TCP port space, what we need in OpenSM is a
combination of service id (TCP) _and_ TCP port _and_ target GUID.

  Something like:

  tcp port <lustre TCP port> target-port-guid <Lustre servers GUIDs>

  That way, as you point out, there's absolutely no need for a specific
Lustre specific port space and kernel patch. I just need to extend the
QoS parser logic to add the semantic above.

  I will look into this.

  Thanks, and sorry for the hassle.

  Sebastien.

> 
> thinking on this a bit more, since the rules are processed by order wouldn't the 
> following scheme let you achieve the same effect?
> 
> target-­port­guid 0x1234,0x1235 : 1 # traffic to MDSs
> lustre                        : 2 # default lustre traffic (to OSSs)
> 
> Or.
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[Or Gerlitz]

sebastien dugue wrote:
> OK, then going with the TCP port space, what we need in OpenSM is a
> combination of service id (TCP) _and_ TCP port _and_ target GUID.

I believe that you can have a 'lustre' keyword in opensm qos parser which stands for the combination of tcp port space + lustre tcp port (maybe it exists now), so in the policy file this would translate to X,{Z1,Z2,..,Zm} (as was in your example) and not to X,Y,{Z1,Z2,..,Zm}. 

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[sebastien dugue]

On Thu, 21 Jan 2010 11:12:08 +0200
Or Gerlitz <ogerlitz-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> wrote:

> sebastien dugue wrote:
> > OK, then going with the TCP port space, what we need in OpenSM is a
> > combination of service id (TCP) _and_ TCP port _and_ target GUID.
> 
> I believe that you can have a 'lustre' keyword in opensm qos parser which stands for the combination of tcp port space + lustre tcp port (maybe it exists now), so in the policy file this would translate to X,{Z1,Z2,..,Zm} (as was in your example) and not to X,Y,{Z1,Z2,..,Zm}.

  Right, that'd be a nice shortcut. I also think being able to specify both port
and target GUID to the TCP port space might be interesting for other ULPs, not
just Lustre.

  Sebastien.
 
> 
> Or.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[sebastien dugue]

  Hi Sean,

On Wed, 13 Jan 2010 08:56:30 -0800
"Sean Hefty" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> >  This patch adds a new port space for use by Lustre traffic.
> >
> >  This, along with patches to OpenSM and Lustre, allow to define a
> >specific QoS for lustre.
> 
> In general, I don't like the idea of application port spaces for QoS
> support.  Can't this be done using port numbers in the existing port
> space?

  That can be done with port numbers, except that we cannot separate
traffic to Lustre MDS and traffic to Lustre OSS as can be done with
a target-port-guid.


> 
> >+	RDMA_PS_LUSTRE = 0x0153,
> 
> It looks like this number maps to the VINES protocol.

  Argh, haven't seen that. From what I understand, the low 8 bit are to
be taken from IANA IP protocol ID when bit 8 is 1, but what about when
bit 8 is 0 (as is the case with SDP and IPOIB)?

  Thanks,

  Sebastien.


> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[Or Gerlitz]

sebastien dugue wrote:
> That can be done with port numbers, except that we cannot separate
> traffic to Lustre MDS and traffic to Lustre OSS 

Looking on these patches and going with you for a minute, I don't see how this patch set serves you to assign a different QoS level (e.g SL) to MDS vs OSS related traffic. Can you elaborate on that a bit?

Sean Hefty wrote:
> Can't this be done using port numbers in the existing port space?

Indeed, Sebastien what prevents you from using the TCP port space, with one port used for MDS traffic and another port for OSS traffic? how does Lustre get ports to listen on, are they well known or you call bind with port zero and use the port allocated by the rdma-cm?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space

[sebastien dugue]

  Hi Or,

On Thu, 14 Jan 2010 16:09:43 +0200
Or Gerlitz <ogerlitz-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> wrote:

> sebastien dugue wrote:
> > That can be done with port numbers, except that we cannot separate
> > traffic to Lustre MDS and traffic to Lustre OSS 
> 
> Looking on these patches and going with you for a minute, I don't see how this patch set serves you to assign a different QoS level (e.g SL) to MDS vs OSS related traffic. Can you elaborate on that a bit?

  If you can specify a target-port-guid to the Lustre service ID in OpenSM's
qos-policy.conf then you're all set.

  For example:

    qos-­ulps 
        default                 	      : 0 # default SL
        lustre, target-­port­guid 0x1234,0x1235 : 1 # lustre traffic to MDSs
        lustre                  	      : 2 # default lustre traffic (to OSSs)
    end­-qos-­ulps


> 
> Sean Hefty wrote:
> > Can't this be done using port numbers in the existing port space?
> 
> Indeed, Sebastien what prevents you from using the TCP port space, with one port used for MDS traffic and another port for OSS traffic? how does Lustre get ports to listen on, are they well known or you call bind with port zero and use the port allocated by the rdma-cm?

  I'm not a Lustre expert (far from it), but after talking with our Lustre guys,
it seems that all Lustre servers, OSS or MDS, use the same TCP port. So there's
no possible way to separate traffic based on TCP port only.

  Thanks,

  Sébastien.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 2/3] opensm/qos_policy: Add a new service ID and keyword for Lustre QoS

[sebastien dugue]

  This patch adds a new service ID for the Lustre ULP as well as the
lexer and parser bits for the policy definition simplified syntax.

  The QoS policy parser now supports the following 'qos-ulps' keywords:

   lustre                          : 0 #default SL for Lustre
   lustre, target-port-guid 0x1234 : 1 #SL for Lustre with target port guid
   lustre, port-num 10000-20000    : 2 #SL for Lustre port range

  This, along with patches to the kernel rdma cma and Lustre, allow to define a
specific QoS for lustre.

Signed-off-by: Sebastien Dugue <sebastien.dugue-6ktuUTfB/bM@public.gmane.org>
---
 opensm/include/opensm/osm_qos_policy.h |   11 ++--
 opensm/opensm/osm_qos_parser_l.l       |    8 +++
 opensm/opensm/osm_qos_parser_y.y       |  108 ++++++++++++++++++++++++++++++++
 3 files changed, 122 insertions(+), 5 deletions(-)

diff --git a/opensm/include/opensm/osm_qos_policy.h b/opensm/include/opensm/osm_qos_policy.h
index 03ee891..ea3998a 100644
--- a/opensm/include/opensm/osm_qos_policy.h
+++ b/opensm/include/opensm/osm_qos_policy.h
@@ -54,11 +54,12 @@
 #define OSM_QOS_POLICY_MAX_PORTS_ON_SWITCH  128
 #define OSM_QOS_POLICY_DEFAULT_LEVEL_NAME   "default"
 
-#define OSM_QOS_POLICY_ULP_SDP_SERVICE_ID   0x0000000000010000ULL
-#define OSM_QOS_POLICY_ULP_RDS_SERVICE_ID   0x0000000001060000ULL
-#define OSM_QOS_POLICY_ULP_RDS_PORT         0x48CA
-#define OSM_QOS_POLICY_ULP_ISER_SERVICE_ID  0x0000000001060000ULL
-#define OSM_QOS_POLICY_ULP_ISER_PORT        0x0CBC
+#define OSM_QOS_POLICY_ULP_SDP_SERVICE_ID     0x0000000000010000ULL
+#define OSM_QOS_POLICY_ULP_RDS_SERVICE_ID     0x0000000001060000ULL
+#define OSM_QOS_POLICY_ULP_RDS_PORT           0x48CA
+#define OSM_QOS_POLICY_ULP_ISER_SERVICE_ID    0x0000000001060000ULL
+#define OSM_QOS_POLICY_ULP_ISER_PORT          0x0CBC
+#define OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID  0x0000000001530000ULL
 
 #define OSM_QOS_POLICY_NODE_TYPE_CA        (((uint8_t)1)<<IB_NODE_TYPE_CA)
 #define OSM_QOS_POLICY_NODE_TYPE_SWITCH    (((uint8_t)1)<<IB_NODE_TYPE_SWITCH)
diff --git a/opensm/opensm/osm_qos_parser_l.l b/opensm/opensm/osm_qos_parser_l.l
index 4633e0e..8f5d147 100644
--- a/opensm/opensm/osm_qos_parser_l.l
+++ b/opensm/opensm/osm_qos_parser_l.l
@@ -117,6 +117,9 @@ static void reset_new_line_flags();
 #define START_ULP_SRP_GUID      {in_list_of_num_ranges = TRUE;} /* comma-separated list of hex or dec num ranges */
 #define START_ULP_IPOIB_DEFAULT {in_single_number = TRUE;}      /* single number */
 #define START_ULP_IPOIB_PKEY    {in_list_of_num_ranges = TRUE;} /* comma-separated list of hex or dec num ranges */
+#define START_ULP_LUSTRE_DEFAULT {in_single_number = TRUE;}      /* single number */
+#define START_ULP_LUSTRE_PORT    {in_list_of_num_ranges = TRUE;} /* comma-separated list of hex or dec num ranges */
+#define START_ULP_LUSTRE_PORT_GUID {in_list_of_num_ranges = TRUE;} /* comma-separated list of hex or dec num ranges */
 
 
 %}
@@ -187,6 +190,7 @@ ULP_SDP                 [Ss][Dd][Pp]
 ULP_SRP                 [Ss][Rr][Pp]
 ULP_RDS                 [Rr][Dd][Ss]
 ULP_IPOIB               [Ii][Pp][Oo][Ii][Bb]
+ULP_LUSTRE              [Ll][Uu][Ss][Tt][Rr][Ee]
 ULP_ISER                [Ii][Ss][Ee][Rr]
 ULP_ANY                 [Aa][Nn][Yy]
 ULP_DEFAULT             [Dd][Ee][Ff][Aa][Uu][Ll][Tt]
@@ -289,6 +293,10 @@ QUOTED_TEXT             \"[^\"]*\"
 {ULP_IPOIB}{WHITE_DOTDOT_WHITE}                { SAVE_POS; HANDLE_IF_IN_DESCRIPTION; START_ULP_IPOIB_DEFAULT; return TK_ULP_IPOIB_DEFAULT; }
 {ULP_IPOIB}{WHITE_COMMA_WHITE}{PKEY}           { SAVE_POS; HANDLE_IF_IN_DESCRIPTION; START_ULP_IPOIB_PKEY; return TK_ULP_IPOIB_PKEY; }
 
+{ULP_LUSTRE}{WHITE_DOTDOT_WHITE}               { SAVE_POS; HANDLE_IF_IN_DESCRIPTION; START_ULP_LUSTRE_DEFAULT; return TK_ULP_LUSTRE_DEFAULT; }
+{ULP_LUSTRE}{WHITE_COMMA_WHITE}{PORT_NUM}      { SAVE_POS; HANDLE_IF_IN_DESCRIPTION; START_ULP_LUSTRE_PORT; return TK_ULP_LUSTRE_PORT; }
+{ULP_LUSTRE}{WHITE_COMMA_WHITE}{TARGET_PORT_GUID}      { SAVE_POS; HANDLE_IF_IN_DESCRIPTION; START_ULP_LUSTRE_PORT_GUID; return TK_ULP_LUSTRE_PORT_GUID; }
+
 0[xX][0-9a-fA-F]+  {
                         SAVE_POS;
                         yylval = strdup(yytext);
diff --git a/opensm/opensm/osm_qos_parser_y.y b/opensm/opensm/osm_qos_parser_y.y
index 7118b79..ccaac91 100644
--- a/opensm/opensm/osm_qos_parser_y.y
+++ b/opensm/opensm/osm_qos_parser_y.y
@@ -268,6 +268,9 @@ static cl_list_t __ulp_match_rules;
 %token TK_ULP_SRP_GUID
 %token TK_ULP_IPOIB_DEFAULT
 %token TK_ULP_IPOIB_PKEY
+%token TK_ULP_LUSTRE_DEFAULT
+%token TK_ULP_LUSTRE_PORT
+%token TK_ULP_LUSTRE_PORT_GUID
 
 %start head
 
@@ -302,6 +305,9 @@ qos_policy_entry:     qos_ulps_section
      *      iser                          : 4 #default SL for iSER
      *      ipoib, pkey 0x0001            : 5 #SL for IPoIB on partition with pkey 0x0001
      *      ipoib                         : 6 #default IPoIB partition - pkey=0x7FFF
+     *      lustre                        : 0 #default SL for Lustre
+     *      lustre, port-num 10000-20000  : 2 #SL for Lustre port range
+     *      lustre, target-port-guid 0x1234 : 3 #SL for Lustre with target port guid
      *      any, service-id 0x6234        : 2
      *      any, pkey 0x0ABC              : 3
      *      any, target-port-guid 0x0ABC-0xFFFFF : 6
@@ -628,6 +634,9 @@ qos_match_rule_entry: qos_match_rule_use
      *   iser with port-num
      *   ipoib
      *   ipoib with pkey
+     *   lustre
+     *   lustre with port-num
+     *   lustre with port-guid
      *   any with service-id
      *   any with pkey
      *   any with target-port-guid
@@ -951,6 +960,96 @@ qos_ulp:            TK_ULP_DEFAULT single_number {
                         p_current_qos_match_rule->pkey_range_len = range_len;
 
                     } qos_ulp_sl
+
+		    | qos_ulp_type_lustre_default {
+			/* "lustre : sl" - default SL for Lustre */
+			uint64_t ** range_arr =
+                               (uint64_t **)malloc(sizeof(uint64_t *));
+                        range_arr[0] = (uint64_t *)malloc(2*sizeof(uint64_t));
+                        range_arr[0][0] = OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID;
+                        range_arr[0][1] = OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID + 0xFFFF;
+
+                        p_current_qos_match_rule->service_id_range_arr = range_arr;
+                        p_current_qos_match_rule->service_id_range_len = 1;
+
+		    } qos_ulp_sl
+
+                    | qos_ulp_type_lustre_port list_of_ranges TK_DOTDOT {
+                        /* Lustre with port numbers */
+                        uint64_t ** range_arr;
+                        unsigned    range_len;
+                        unsigned    i;
+
+                        if (!cl_list_count(&tmp_parser_struct.num_pair_list))
+                        {
+                            yyerror("Lustre ULP rule doesn't have port numbers");
+                            return 1;
+                        }
+
+                        /* get all the port ranges */
+                        __rangelist2rangearr( &tmp_parser_struct.num_pair_list,
+                                              &range_arr,
+                                              &range_len );
+                        /* now translate these port numbers into service ids */
+                        for (i = 0; i < range_len; i++)
+                        {
+                            if (range_arr[i][0] > 0xFFFF || range_arr[i][1] > 0xFFFF)
+                            {
+                                yyerror("Lustre port number out of range");
+                                return 1;
+                            }
+                            range_arr[i][0] += OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID;
+                            range_arr[i][1] += OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID;
+                        }
+
+                        p_current_qos_match_rule->service_id_range_arr = range_arr;
+                        p_current_qos_match_rule->service_id_range_len = range_len;
+
+                    } qos_ulp_sl
+
+		    | qos_ulp_type_lustre_port_guid list_of_ranges TK_DOTDOT {
+			/* lustre, port-guid ... : sl */
+                        uint64_t ** range_arr;
+                        unsigned    range_len;
+
+                        if (!cl_list_count(&tmp_parser_struct.num_pair_list))
+                        {
+                            yyerror("Lustre ULP rule doesn't have port guids");
+                            return 1;
+                        }
+
+                        /* create a new port group with these ports */
+                        __parser_port_group_start();
+
+                        p_current_port_group->name = strdup("_Lustre_Targets_");
+                        p_current_port_group->use = strdup("Generated from ULP rules");
+
+                        __rangelist2rangearr( &tmp_parser_struct.num_pair_list,
+                                              &range_arr,
+                                              &range_len );
+
+                        __parser_add_guid_range_to_port_map(
+                                              &p_current_port_group->port_map,
+                                              range_arr,
+                                              range_len);
+
+                        /* add this port group to the destination
+                           groups of the current match rule */
+                        cl_list_insert_tail(&p_current_qos_match_rule->destination_group_list,
+                                            p_current_port_group);
+
+                        __parser_port_group_end();
+
+			/* setup ranges as in qos_ulp_type_lustre_default */
+			range_arr = (uint64_t **)malloc(sizeof(uint64_t *));
+                        range_arr[0] = (uint64_t *)malloc(2*sizeof(uint64_t));
+                        range_arr[0][0] = OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID;
+                        range_arr[0][1] = OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID + 0xFFFF;
+
+                        p_current_qos_match_rule->service_id_range_arr = range_arr;
+                        p_current_qos_match_rule->service_id_range_len = 1;
+
+		    } qos_ulp_sl
                     ;
 
 qos_ulp_type_any_service: TK_ULP_ANY_SERVICE_ID
@@ -989,6 +1088,15 @@ qos_ulp_type_ipoib_default: TK_ULP_IPOIB_DEFAULT
 qos_ulp_type_ipoib_pkey: TK_ULP_IPOIB_PKEY
                     { __parser_ulp_match_rule_start(); };
 
+qos_ulp_type_lustre_default: TK_ULP_LUSTRE_DEFAULT
+                    { __parser_ulp_match_rule_start(); };
+
+qos_ulp_type_lustre_port: TK_ULP_LUSTRE_PORT
+                    { __parser_ulp_match_rule_start(); };
+
+qos_ulp_type_lustre_port_guid: TK_ULP_LUSTRE_PORT_GUID
+                    { __parser_ulp_match_rule_start(); };
+
 
 qos_ulp_sl:   single_number {
                         /* get the SL for ULP rules */
-- 
1.6.0.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 3/3] Add new port_space parameter to ko2iblnd module

[sebastien dugue]

  This patch is only given as a reference.


Index: b/lnet/klnds/o2iblnd/o2iblnd.c
===================================================================
--- a/lnet/klnds/o2iblnd/o2iblnd.c
+++ b/lnet/klnds/o2iblnd/o2iblnd.c
@@ -2503,7 +2503,7 @@ kiblnd_startup (lnet_ni_t *ni)
                 ibdev->ibd_ifip = ip;
                 strcpy(&ibdev->ibd_ifname[0], ifname);
 
-                id = rdma_create_id(kiblnd_cm_callback, ibdev, RDMA_PS_TCP);
+                id = rdma_create_id(kiblnd_cm_callback, ibdev, *kiblnd_tunables.kib_port_space);
                 if (IS_ERR(id)) {
                         CERROR("Can't create listen ID: %ld\n", PTR_ERR(id));
                         goto failed;
Index: b/lnet/klnds/o2iblnd/o2iblnd.h
===================================================================
--- a/lnet/klnds/o2iblnd/o2iblnd.h
+++ b/lnet/klnds/o2iblnd/o2iblnd.h
@@ -125,6 +125,7 @@ typedef struct
 #if defined(CONFIG_SYSCTL) && !CFS_SYSFS_MODULE_PARM
         cfs_sysctl_table_header_t *kib_sysctl;  /* sysctl interface */
 #endif
+        int              *kib_port_space;       /* port space */
 } kib_tunables_t;
 
 extern kib_tunables_t  kiblnd_tunables;
Index: b/lnet/klnds/o2iblnd/o2iblnd_cb.c
===================================================================
--- a/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -1188,7 +1188,7 @@ kiblnd_connect_peer (kib_peer_t *peer)
         LASSERT (net != NULL);
         LASSERT (peer->ibp_connecting > 0);
 
-        cmid = rdma_create_id(kiblnd_cm_callback, peer, RDMA_PS_TCP);
+        cmid = rdma_create_id(kiblnd_cm_callback, peer, *kiblnd_tunables.kib_port_space);
         if (IS_ERR(cmid)) {
                 CERROR("Can't create CMID for %s: %ld\n",
                        libcfs_nid2str(peer->ibp_nid), PTR_ERR(cmid));
Index: b/lnet/klnds/o2iblnd/o2iblnd_modparams.c
===================================================================
--- a/lnet/klnds/o2iblnd/o2iblnd_modparams.c
+++ b/lnet/klnds/o2iblnd/o2iblnd_modparams.c
@@ -120,6 +120,10 @@ static int pmr_pool_size = 512;
 CFS_MODULE_PARM(pmr_pool_size, "i", int, 0444,
                 "size of the MR cache pmr pool");
 
+static int port_space = RDMA_PS_TCP;
+CFS_MODULE_PARM(port_space, "i", int, 0444,
+                "Port space");
+
 kib_tunables_t kiblnd_tunables = {
         .kib_service                = &service,
         .kib_cksum                  = &cksum,
@@ -141,6 +145,7 @@ kib_tunables_t kiblnd_tunables = {
         .kib_fmr_flush_trigger      = &fmr_flush_trigger,
         .kib_fmr_cache              = &fmr_cache,
         .kib_pmr_pool_size          = &pmr_pool_size,
+        .kib_port_space             = &port_space,
 };
 
 #if defined(CONFIG_SYSCTL) && !CFS_SYSFS_MODULE_PARM
@@ -169,7 +174,8 @@ enum {
         O2IBLND_FMR_POOL_SIZE,
         O2IBLND_FMR_FLUSH_TRIGGER,
         O2IBLND_FMR_CACHE,
-        O2IBLND_PMR_POOL_SIZE
+        O2IBLND_PMR_POOL_SIZE,
+        O2IBLND_PORT_SPACE
 };
 #else
 
@@ -193,6 +199,7 @@ enum {
 #define O2IBLND_FMR_FLUSH_TRIGGER CTL_UNNUMBERED
 #define O2IBLND_FMR_CACHE        CTL_UNNUMBERED
 #define O2IBLND_PMR_POOL_SIZE    CTL_UNNUMBERED
+#define O2IBLND_PORT_SPACE       CTL_UNNUMBERED
 
 #endif
 
@@ -358,6 +365,14 @@ static cfs_sysctl_table_t kiblnd_ctl_tab
                 .mode     = 0444,
                 .proc_handler = &proc_dointvec
         },
+        {
+                .ctl_name = O2IBLND_PORT_SPACE,
+                .procname = "port_space",
+                .data     = &port_space,
+                .maxlen   = sizeof(int),
+                .mode     = 0444,
+                .proc_handler = &proc_dointvec
+        },
         {0}
 };
 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html