QOS setup issue with opensm

QOS setup issue with opensm

[Vincent Ficet]

Hello,

I'm running into trouble when trying to configure QoS on opensm 3.3.2. I
tried several QoS setups, but unfortunately without any luck.
Could someone please tell me what I'm doing wrong ?

The different combinations I tried are listed below:

###############################################################

qos-policy.conf:

qos-ulps
    default          : 0 # default SL
    ipoib            : 4 # TCP tests over IB
end-qos-ulps

Nov 24 09:26:01 065823 [96B5D910] 0x01 -> __qos_policy_validate_pkey:
ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
QoS Level SL (4)

###############################################################

qos-policy.conf:

qos-ulps
    default               : 0 # default SL
    ipoib, pkey 0x0001    : 4 # TCP tests over IB
end-qos-ulps

partitions.conf:

default=0x7fff,ipoib                    : ALL=full;
backbone=0x0001,ipoib,defmember=full    : 0x2c9000100d000679,
0x2c9000100d00056d;

Nov 24 09:21:43 526886 [C85A1910] 0x01 -> __qos_policy_validate_pkey:
ERR AC15: pkey 0x0001 in match rule - overriding partition SL (0) with
QoS Level SL (4)

###############################################################

qos-policy.conf

qos-ulps
    default          : 0 # default SL
    ipoib            : 4 # TCP tests over IB
end-qos-ulps

partitions.conf:

default=0x7fff                          : ALL=full;
backbone=0x0001,ipoib,defmember=full    : 0x2c9000100d000679,
0x2c9000100d00056d;

Nov 24 09:46:36 576946 [640C7910] 0x01 -> __qos_policy_validate_pkey:
ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
QoS Level SL (4)

###############################################################

qos-policy.conf

qos-ulps
    default          : 0 # default SL
    ipoib            : 4 # TCP tests over IB
end-qos-ulps

default=0x7fff                        : ALL=full;
backbone=0x0001,defmember=full        : 0x2c9000100d000679,
0x2c9000100d00056d;

Nov 24 09:51:23 666860 [6EA6B910] 0x01 -> __qos_policy_validate_pkey:
ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
QoS Level SL (4)

###############################################################

Did I make a mistake, or does that sound like an opensm bug ?

Thanks for your help,

Vincent


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: QOS setup issue with opensm

[Yevgeny Kliteynik]

Vincent Ficet wrote:
> Hello,
> 
> I'm running into trouble when trying to configure QoS on opensm 3.3.2. I
> tried several QoS setups, but unfortunately without any luck.
> Could someone please tell me what I'm doing wrong ?
> 
> The different combinations I tried are listed below:
> 
> ###############################################################
> 
> qos-policy.conf:
> 
> qos-ulps
>     default          : 0 # default SL
>     ipoib            : 4 # TCP tests over IB
> end-qos-ulps
> 
> Nov 24 09:26:01 065823 [96B5D910] 0x01 -> __qos_policy_validate_pkey:
> ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
> QoS Level SL (4)
> 
> ###############################################################
> 
> qos-policy.conf:
> 
> qos-ulps
>     default               : 0 # default SL
>     ipoib, pkey 0x0001    : 4 # TCP tests over IB
> end-qos-ulps
> 
> partitions.conf:
> 
> default=0x7fff,ipoib                    : ALL=full;
> backbone=0x0001,ipoib,defmember=full    : 0x2c9000100d000679,
> 0x2c9000100d00056d;
> 
> Nov 24 09:21:43 526886 [C85A1910] 0x01 -> __qos_policy_validate_pkey:
> ERR AC15: pkey 0x0001 in match rule - overriding partition SL (0) with
> QoS Level SL (4)
> 
> ###############################################################
> 
> qos-policy.conf
> 
> qos-ulps
>     default          : 0 # default SL
>     ipoib            : 4 # TCP tests over IB
> end-qos-ulps
> 
> partitions.conf:
> 
> default=0x7fff                          : ALL=full;
> backbone=0x0001,ipoib,defmember=full    : 0x2c9000100d000679,
> 0x2c9000100d00056d;
> 
> Nov 24 09:46:36 576946 [640C7910] 0x01 -> __qos_policy_validate_pkey:
> ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
> QoS Level SL (4)
> 
> ###############################################################
> 
> qos-policy.conf
> 
> qos-ulps
>     default          : 0 # default SL
>     ipoib            : 4 # TCP tests over IB
> end-qos-ulps
> 
> default=0x7fff                        : ALL=full;
> backbone=0x0001,defmember=full        : 0x2c9000100d000679,
> 0x2c9000100d00056d;
> 
> Nov 24 09:51:23 666860 [6EA6B910] 0x01 -> __qos_policy_validate_pkey:
> ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
> QoS Level SL (4)
> 
> ###############################################################
> 
> Did I make a mistake, or does that sound like an opensm bug ?

It's neither. It's an outcome of OSM being not very user
friendly in QoS configuration :(

You're trying to assign a certain SL to IPoIB, so you
define a partition with a certain pkey, and add a rule
for this pkey in the QoS policy file. So far everything
is OK.
The problem is that SL is part of partition's parameters,
so when you defined the partition as follows:

  default=0x7fff,ipoib       : ALL=full;

you didn't specify SL, and SM used default SL for this
partition. In short, it is as if you defined the partition
as follows:

  default=0x7fff,ipoib,sl=0  : ALL=full;

Now comes the QoS policy rule that says the following:

  ipoib            : 4 # TCP tests over IB

So you have a mismatch between SLs from partition
configuration and QoS policy. SM is aware if this
and uses SL from QoS policy to overwrite the one
that is stored in the partition structure.

So bottom line - your configuration is correct.

-- Yevgeny


> Thanks for your help,
> 
> Vincent
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: QOS setup issue with opensm

[Vincent Ficet]

Hello Yevgeny,

> Vincent Ficet wrote:
>> Hello,
>>
>> I'm running into trouble when trying to configure QoS on opensm 3.3.2. I
>> tried several QoS setups, but unfortunately without any luck.
>> Could someone please tell me what I'm doing wrong ?
>>
>> The different combinations I tried are listed below:
>>
>> ###############################################################
>>
>> qos-policy.conf:
>>
>> qos-ulps
>>     default          : 0 # default SL
>>     ipoib            : 4 # TCP tests over IB
>> end-qos-ulps
>>
>> Nov 24 09:26:01 065823 [96B5D910] 0x01 -> __qos_policy_validate_pkey:
>> ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
>> QoS Level SL (4)
>>
>> ###############################################################
>>
>> qos-policy.conf:
>>
>> qos-ulps
>>     default               : 0 # default SL
>>     ipoib, pkey 0x0001    : 4 # TCP tests over IB
>> end-qos-ulps
>>
>> partitions.conf:
>>
>> default=0x7fff,ipoib                    : ALL=full;
>> backbone=0x0001,ipoib,defmember=full    : 0x2c9000100d000679,
>> 0x2c9000100d00056d;
>>
>> Nov 24 09:21:43 526886 [C85A1910] 0x01 -> __qos_policy_validate_pkey:
>> ERR AC15: pkey 0x0001 in match rule - overriding partition SL (0) with
>> QoS Level SL (4)
>>
>> ###############################################################
>>
>> qos-policy.conf
>>
>> qos-ulps
>>     default          : 0 # default SL
>>     ipoib            : 4 # TCP tests over IB
>> end-qos-ulps
>>
>> partitions.conf:
>>
>> default=0x7fff                          : ALL=full;
>> backbone=0x0001,ipoib,defmember=full    : 0x2c9000100d000679,
>> 0x2c9000100d00056d;
>>
>> Nov 24 09:46:36 576946 [640C7910] 0x01 -> __qos_policy_validate_pkey:
>> ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
>> QoS Level SL (4)
>>
>> ###############################################################
>>
>> qos-policy.conf
>>
>> qos-ulps
>>     default          : 0 # default SL
>>     ipoib            : 4 # TCP tests over IB
>> end-qos-ulps
>>
>> default=0x7fff                        : ALL=full;
>> backbone=0x0001,defmember=full        : 0x2c9000100d000679,
>> 0x2c9000100d00056d;
>>
>> Nov 24 09:51:23 666860 [6EA6B910] 0x01 -> __qos_policy_validate_pkey:
>> ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
>> QoS Level SL (4)
>>
>> ###############################################################
>>
>> Did I make a mistake, or does that sound like an opensm bug ?
>
> It's neither. It's an outcome of OSM being not very user
> friendly in QoS configuration :(
>
> You're trying to assign a certain SL to IPoIB, so you
> define a partition with a certain pkey, and add a rule
> for this pkey in the QoS policy file. So far everything
> is OK.
> The problem is that SL is part of partition's parameters,
> so when you defined the partition as follows:
>
>  default=0x7fff,ipoib       : ALL=full;
>
> you didn't specify SL, and SM used default SL for this
> partition. In short, it is as if you defined the partition
> as follows:
>
>  default=0x7fff,ipoib,sl=0  : ALL=full;
>
> Now comes the QoS policy rule that says the following:
>
>  ipoib            : 4 # TCP tests over IB
>
> So you have a mismatch between SLs from partition
> configuration and QoS policy. SM is aware if this
> and uses SL from QoS policy to overwrite the one
> that is stored in the partition structure.
>
> So bottom line - your configuration is correct.
Just to make sure I understand correctly. What you are saying is that
the following configuration (among the 4 alternatives I listed above) is
correct, although opensm reports an error ?

partitions.conf:

default=0x7fff,ipoib            : ALL=full;
backbone=0x0001,ipoib,defmember=full    : 0x2c9000100d000679,
0x2c9000100d00056d;

qos-ulps
        default            : 0 # default SL
    ipoib            : 4 # TCP tests over IB
end-qos-ulps

Thanks for your help,

Vincent

>
> -- Yevgeny
>
>
>> Thanks for your help,
>>
>> Vincent
>>
>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: QOS setup issue with opensm

[Yevgeny Kliteynik]

Hi Vincent,

Vincent Ficet wrote:
> Hello Yevgeny,
> 
>> Vincent Ficet wrote:
>>> Hello,
>>>
>>> I'm running into trouble when trying to configure QoS on opensm 3.3.2. I
>>> tried several QoS setups, but unfortunately without any luck.
>>> Could someone please tell me what I'm doing wrong ?
>>>
>>> The different combinations I tried are listed below:
>>>
>>> ###############################################################
>>>
>>> qos-policy.conf:
>>>
>>> qos-ulps
>>>     default          : 0 # default SL
>>>     ipoib            : 4 # TCP tests over IB
>>> end-qos-ulps
>>>
>>> Nov 24 09:26:01 065823 [96B5D910] 0x01 -> __qos_policy_validate_pkey:
>>> ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
>>> QoS Level SL (4)
>>>
>>> ###############################################################
>>>
>>> qos-policy.conf:
>>>
>>> qos-ulps
>>>     default               : 0 # default SL
>>>     ipoib, pkey 0x0001    : 4 # TCP tests over IB
>>> end-qos-ulps
>>>
>>> partitions.conf:
>>>
>>> default=0x7fff,ipoib                    : ALL=full;
>>> backbone=0x0001,ipoib,defmember=full    : 0x2c9000100d000679,
>>> 0x2c9000100d00056d;
>>>
>>> Nov 24 09:21:43 526886 [C85A1910] 0x01 -> __qos_policy_validate_pkey:
>>> ERR AC15: pkey 0x0001 in match rule - overriding partition SL (0) with
>>> QoS Level SL (4)
>>>
>>> ###############################################################
>>>
>>> qos-policy.conf
>>>
>>> qos-ulps
>>>     default          : 0 # default SL
>>>     ipoib            : 4 # TCP tests over IB
>>> end-qos-ulps
>>>
>>> partitions.conf:
>>>
>>> default=0x7fff                          : ALL=full;
>>> backbone=0x0001,ipoib,defmember=full    : 0x2c9000100d000679,
>>> 0x2c9000100d00056d;
>>>
>>> Nov 24 09:46:36 576946 [640C7910] 0x01 -> __qos_policy_validate_pkey:
>>> ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
>>> QoS Level SL (4)
>>>
>>> ###############################################################
>>>
>>> qos-policy.conf
>>>
>>> qos-ulps
>>>     default          : 0 # default SL
>>>     ipoib            : 4 # TCP tests over IB
>>> end-qos-ulps
>>>
>>> default=0x7fff                        : ALL=full;
>>> backbone=0x0001,defmember=full        : 0x2c9000100d000679,
>>> 0x2c9000100d00056d;
>>>
>>> Nov 24 09:51:23 666860 [6EA6B910] 0x01 -> __qos_policy_validate_pkey:
>>> ERR AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with
>>> QoS Level SL (4)
>>>
>>> ###############################################################
>>>
>>> Did I make a mistake, or does that sound like an opensm bug ?
>> It's neither. It's an outcome of OSM being not very user
>> friendly in QoS configuration :(
>>
>> You're trying to assign a certain SL to IPoIB, so you
>> define a partition with a certain pkey, and add a rule
>> for this pkey in the QoS policy file. So far everything
>> is OK.
>> The problem is that SL is part of partition's parameters,
>> so when you defined the partition as follows:
>>
>>  default=0x7fff,ipoib       : ALL=full;
>>
>> you didn't specify SL, and SM used default SL for this
>> partition. In short, it is as if you defined the partition
>> as follows:
>>
>>  default=0x7fff,ipoib,sl=0  : ALL=full;
>>
>> Now comes the QoS policy rule that says the following:
>>
>>  ipoib            : 4 # TCP tests over IB
>>
>> So you have a mismatch between SLs from partition
>> configuration and QoS policy. SM is aware if this
>> and uses SL from QoS policy to overwrite the one
>> that is stored in the partition structure.
>>
>> So bottom line - your configuration is correct.
> Just to make sure I understand correctly. What you are saying is that
> the following configuration (among the 4 alternatives I listed above) is
> correct, although opensm reports an error ?
> 
> partitions.conf:
> 
> default=0x7fff,ipoib            : ALL=full;
> backbone=0x0001,ipoib,defmember=full    : 0x2c9000100d000679,
> 0x2c9000100d00056d;
> 
> qos-ulps
>         default            : 0 # default SL
>     ipoib            : 4 # TCP tests over IB
> end-qos-ulps

I don't know what exactly are you trying to do,
so I don't know which option is the correct one.
If you just want to have default IPoIB to use
SL4, you can use the first option:

	qos-ulps
		default          : 0 # default SL
		ipoib            : 4 # TCP tests over IB
	end-qos-ulps

As you already noted, you will see the error message
about SL mismatch in the OSM log. The reason for this
is default partition configuration: if no partition
file was provided, SM defines default IPoIB partition
with default pkey, which is 0x7fff, and other default
parameters, one of them is SL0.
So you get the SL mismatch again, but SM uses what you
provided in the QoS policy as an SL with higher precedence
and replaces the SL0 in the partition.

-- Yevgeny

 
> Thanks for your help,
> 
> Vincent
> 
>> -- Yevgeny
>>
>>
>>> Thanks for your help,
>>>
>>> Vincent
>>>
>>>
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: QOS setup issue with opensm

[Vincent Ficet]

Hello Yevgeny,

> I don't know what exactly are you trying to do,
> so I don't know which option is the correct one.
> If you just want to have default IPoIB to use
> SL4, you can use the first option:
>
>     qos-ulps
>         default          : 0 # default SL
>         ipoib            : 4 # TCP tests over IB
>     end-qos-ulps
>
> As you already noted, you will see the error message
> about SL mismatch in the OSM log. The reason for this
> is default partition configuration: if no partition
> file was provided, SM defines default IPoIB partition
> with default pkey, which is 0x7fff, and other default
> parameters, one of them is SL0.
> So you get the SL mismatch again, but SM uses what you
> provided in the QoS policy as an SL with higher precedence
> and replaces the SL0 in the partition.
>
OK, this makes sense. I carried out some tests and it did work.

Thanks for your help,

Vincent
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html