OpenSM Failover

OpenSM Failover

[Aaron Knister]

I'm not sure if this is the right place to post about this issue, but  
here goes-

I'm having problems with OpenSM failover.

I have two nodes running opensmd version "3.2.6_20090317" from RHEL  
5.4. I'm using a configuration file on both generated using opensm -c.  
When I start the subnet manager on node a, everything is fine. It  
appears to reassign itself a lid of 1 which I think is expected. When  
I started the subnet manager on node b everything is fine. If you  
query its lid it shows the subnet manager in a standby state. Now for  
the fun. If I stop opensmd on node a (service opensmd stop) then all  
of the traffic on the fabric stops. It takes node b's OpenSM instance  
about 30 seconds to realize that node a's subnet manager is dead and  
come up in the master state. Now when node a's subnet manager comes  
back (service opensmd start), all traffic on the fabric stops and node  
b's subnet manager goes into the standby state...but node a's subnet  
manager doesn't take over the fabric and come up as master for about  
another 40 seconds (during this time the no traffic passes over the  
fabric). The below logs should help illustrate what I'm seeing


Oct 10 19:14:14 node-a OpenSM[14132]: Entering DISCOVERING state
Oct 10 19:14:14 node-a OpenSM[14132]: Entering MASTER state
Oct 10 19:14:14 node-a OpenSM[14132]: SUBNET UP

Oct 10 19:14:25 node-b OpenSM[11197]: /var/log/opensm.log log file  
opened
Oct 10 19:14:25 node-b OpenSM[11197]: OpenSM 3.2.6_20090317
Oct 10 19:14:25 node-b OpenSM[11197]: Entering DISCOVERING state
Oct 10 19:14:26 node-b OpenSM[11197]: Entering STANDBY state

Oct 10 19:15:44 node-a OpenSM[14132]: Exiting SM
Oct 10 19:16:16 node-b OpenSM[11197]: Entering DISCOVERING state
Oct 10 19:16:16 node-b OpenSM[11197]: Entering MASTER state

Oct 10 19:18:52 node-a OpenSM[14213]: /var/log/opensm.log log file  
opened
Oct 10 19:18:52 node-a OpenSM[14213]: OpenSM 3.2.6_20090317
Oct 10 19:18:52 node-a OpenSM[14213]: Entering DISCOVERING state
Oct 10 19:18:53 node-b OpenSM[11197]: Entering STANDBY state
Oct 10 19:18:53 node-a OpenSM[14213]: Entering STANDBY state
Oct 10 19:19:33 node-a OpenSM[14213]: Entering DISCOVERING state
Oct 10 19:19:33 node-a OpenSM[14213]: Entering MASTER state
Oct 10 19:19:33 node-a OpenSM[14213]: SUBNET UP

We have opensm 3.2.5_20081207 (ofed 1.4) on another cluster and it  
fails over and fails back almost instantly with seemingly no traffic  
interruption if you gracefully stopped the active opensmd instance  
(service opensmd stop). Is the behavior I'm seeing considered normal?  
I can understand the 30 seconds for the initial failover but why the  
40 second failback when the original master comes back? Any help is  
appreciated :)

BTW my switch is a Qlogic 12800-180 with the latest firmware and the  
HCAs are Mellanox MT26428 running firmware version 2.6.648.

Thanks!

-Aaron
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Aaron Knister]

I just stumbled across this in the release notes for opensm 3.2.6-

"* SMs do not hand-over when running on ConnectX in a switch-based  
topology."

So I guess that answers the question of whether or not what I'm seeing  
is "expected behavior". Out of curiosity what are the technical  
reasons for this? I just tried opensm 3.3.2 and I still experience the  
same behavior.

On Oct 10, 2009, at 7:38 PM, Aaron Knister wrote:

> I'm not sure if this is the right place to post about this issue,  
> but here goes-
>
> I'm having problems with OpenSM failover.
>
> I have two nodes running opensmd version "3.2.6_20090317" from RHEL  
> 5.4. I'm using a configuration file on both generated using opensm - 
> c. When I start the subnet manager on node a, everything is fine. It  
> appears to reassign itself a lid of 1 which I think is expected.  
> When I started the subnet manager on node b everything is fine. If  
> you query its lid it shows the subnet manager in a standby state.  
> Now for the fun. If I stop opensmd on node a (service opensmd stop)  
> then all of the traffic on the fabric stops. It takes node b's  
> OpenSM instance about 30 seconds to realize that node a's subnet  
> manager is dead and come up in the master state. Now when node a's  
> subnet manager comes back (service opensmd start), all traffic on  
> the fabric stops and node b's subnet manager goes into the standby  
> state...but node a's subnet manager doesn't take over the fabric and  
> come up as master for about another 40 seconds (during this time the  
> no traffic passes over the fabric). The below logs should help  
> illustrate what I'm seeing
>
>
> Oct 10 19:14:14 node-a OpenSM[14132]: Entering DISCOVERING state
> Oct 10 19:14:14 node-a OpenSM[14132]: Entering MASTER state
> Oct 10 19:14:14 node-a OpenSM[14132]: SUBNET UP
>
> Oct 10 19:14:25 node-b OpenSM[11197]: /var/log/opensm.log log file  
> opened
> Oct 10 19:14:25 node-b OpenSM[11197]: OpenSM 3.2.6_20090317
> Oct 10 19:14:25 node-b OpenSM[11197]: Entering DISCOVERING state
> Oct 10 19:14:26 node-b OpenSM[11197]: Entering STANDBY state
>
> Oct 10 19:15:44 node-a OpenSM[14132]: Exiting SM
> Oct 10 19:16:16 node-b OpenSM[11197]: Entering DISCOVERING state
> Oct 10 19:16:16 node-b OpenSM[11197]: Entering MASTER state
>
> Oct 10 19:18:52 node-a OpenSM[14213]: /var/log/opensm.log log file  
> opened
> Oct 10 19:18:52 node-a OpenSM[14213]: OpenSM 3.2.6_20090317
> Oct 10 19:18:52 node-a OpenSM[14213]: Entering DISCOVERING state
> Oct 10 19:18:53 node-b OpenSM[11197]: Entering STANDBY state
> Oct 10 19:18:53 node-a OpenSM[14213]: Entering STANDBY state
> Oct 10 19:19:33 node-a OpenSM[14213]: Entering DISCOVERING state
> Oct 10 19:19:33 node-a OpenSM[14213]: Entering MASTER state
> Oct 10 19:19:33 node-a OpenSM[14213]: SUBNET UP
>
> We have opensm 3.2.5_20081207 (ofed 1.4) on another cluster and it  
> fails over and fails back almost instantly with seemingly no traffic  
> interruption if you gracefully stopped the active opensmd instance  
> (service opensmd stop). Is the behavior I'm seeing considered  
> normal? I can understand the 30 seconds for the initial failover but  
> why the 40 second failback when the original master comes back? Any  
> help is appreciated :)
>
> BTW my switch is a Qlogic 12800-180 with the latest firmware and the  
> HCAs are Mellanox MT26428 running firmware version 2.6.648.
>
> Thanks!
>
> -Aaron

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Yevgeny Kliteynik]

Aaron,

Aaron Knister wrote:
> I just stumbled across this in the release notes for opensm 3.2.6-
> 
> "* SMs do not hand-over when running on ConnectX in a switch-based 
> topology."
> 
> So I guess that answers the question of whether or not what I'm seeing 
> is "expected behavior". Out of curiosity what are the technical reasons 
> for this? I just tried opensm 3.3.2 and I still experience the same 
> behavior.

There was a hand-over problem in OFED 1.4, but later it turned
out to be FW issue. The thing is, FW version 2.6.648 doesn't
have this bug any more...

The 30 seconds for the initial failover is expected, but the 
40 second failback when the original master comes back is a problem.

Can you please double check that the FW version 2.6.648 is used on
both HCAs that run OSM?
And what is the FW version of HCAs that don't have this problem?

Also, can you please reproduce the issue running OSM as follows:
	opensm -V -e -s 0
on both nodes and attach the /var/log/opensm.log files?

-- Yevgeny

 
> On Oct 10, 2009, at 7:38 PM, Aaron Knister wrote:
> 
>> I'm not sure if this is the right place to post about this issue, but 
>> here goes-
>>
>> I'm having problems with OpenSM failover.
>>
>> I have two nodes running opensmd version "3.2.6_20090317" from RHEL 
>> 5.4. I'm using a configuration file on both generated using opensm -c. 
>> When I start the subnet manager on node a, everything is fine. It 
>> appears to reassign itself a lid of 1 which I think is expected. When 
>> I started the subnet manager on node b everything is fine. If you 
>> query its lid it shows the subnet manager in a standby state. Now for 
>> the fun. If I stop opensmd on node a (service opensmd stop) then all 
>> of the traffic on the fabric stops. It takes node b's OpenSM instance 
>> about 30 seconds to realize that node a's subnet manager is dead and 
>> come up in the master state. Now when node a's subnet manager comes 
>> back (service opensmd start), all traffic on the fabric stops and node 
>> b's subnet manager goes into the standby state...but node a's subnet 
>> manager doesn't take over the fabric and come up as master for about 
>> another 40 seconds (during this time the no traffic passes over the 
>> fabric). The below logs should help illustrate what I'm seeing
>>
>>
>> Oct 10 19:14:14 node-a OpenSM[14132]: Entering DISCOVERING state
>> Oct 10 19:14:14 node-a OpenSM[14132]: Entering MASTER state
>> Oct 10 19:14:14 node-a OpenSM[14132]: SUBNET UP
>>
>> Oct 10 19:14:25 node-b OpenSM[11197]: /var/log/opensm.log log file opened
>> Oct 10 19:14:25 node-b OpenSM[11197]: OpenSM 3.2.6_20090317
>> Oct 10 19:14:25 node-b OpenSM[11197]: Entering DISCOVERING state
>> Oct 10 19:14:26 node-b OpenSM[11197]: Entering STANDBY state
>>
>> Oct 10 19:15:44 node-a OpenSM[14132]: Exiting SM
>> Oct 10 19:16:16 node-b OpenSM[11197]: Entering DISCOVERING state
>> Oct 10 19:16:16 node-b OpenSM[11197]: Entering MASTER state
>>
>> Oct 10 19:18:52 node-a OpenSM[14213]: /var/log/opensm.log log file opened
>> Oct 10 19:18:52 node-a OpenSM[14213]: OpenSM 3.2.6_20090317
>> Oct 10 19:18:52 node-a OpenSM[14213]: Entering DISCOVERING state
>> Oct 10 19:18:53 node-b OpenSM[11197]: Entering STANDBY state
>> Oct 10 19:18:53 node-a OpenSM[14213]: Entering STANDBY state
>> Oct 10 19:19:33 node-a OpenSM[14213]: Entering DISCOVERING state
>> Oct 10 19:19:33 node-a OpenSM[14213]: Entering MASTER state
>> Oct 10 19:19:33 node-a OpenSM[14213]: SUBNET UP
>>
>> We have opensm 3.2.5_20081207 (ofed 1.4) on another cluster and it 
>> fails over and fails back almost instantly with seemingly no traffic 
>> interruption if you gracefully stopped the active opensmd instance 
>> (service opensmd stop). Is the behavior I'm seeing considered normal? 
>> I can understand the 30 seconds for the initial failover but why the 
>> 40 second failback when the original master comes back? Any help is 
>> appreciated :)
>>
>> BTW my switch is a Qlogic 12800-180 with the latest firmware and the 
>> HCAs are Mellanox MT26428 running firmware version 2.6.648.
>>
>> Thanks!
>>
>> -Aaron
> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Or Gerlitz]

Yevgeny Kliteynik wrote:
> There was a hand-over problem in OFED 1.4, but later it turned  out to 
> be FW issue. The thing is, FW version 2.6.648 doesn't  have this bug 
> any more...
so things should work fine with the newly released 2.7 firmware? if this 
is still under question, Aaron, I suggest you open a bugzilla case @ 
https://bugs.openfabrics.org and we can track from there.

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Yevgeny Kliteynik]

Or Gerlitz wrote:
> Yevgeny Kliteynik wrote:
>> There was a hand-over problem in OFED 1.4, but later it turned  out to 
>> be FW issue. The thing is, FW version 2.6.648 doesn't  have this bug 
>> any more...
> so things should work fine with the newly released 2.7 firmware?

Yes

> if this 
> is still under question, Aaron, I suggest you open a bugzilla case @ 
> https://bugs.openfabrics.org and we can track from there.

Good idea.

-- Yevgeny

> Or.
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Aaron Knister]

While the adapters have mellanox chipsets their actually IBM OEM
branded and IBM hasn't released the 2.7 fw yet. I'm a little hesitant
to apply the generic Mellanox FW.

On Mon, Oct 12, 2009 at 4:22 AM, Yevgeny Kliteynik
<kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> Or Gerlitz wrote:
>>
>> Yevgeny Kliteynik wrote:
>>>
>>> There was a hand-over problem in OFED 1.4, but later it turned  out to be
>>> FW issue. The thing is, FW version 2.6.648 doesn't  have this bug any
>>> more...
>>
>> so things should work fine with the newly released 2.7 firmware?
>
> Yes
>
>> if this is still under question, Aaron, I suggest you open a bugzilla case
>> @ https://bugs.openfabrics.org and we can track from there.
>
> Good idea.
>
> -- Yevgeny
>
>> Or.
>>
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Yevgeny Kliteynik]

Aaron,

Thanks for the logs, this was really helpful.
Looks like there is a handover race in the OSM -
SM on node A misses the fact that SM on node B
have gave up its mastership.

There is a bugzilla issue the describes all the
details of this race:

https://bugs.openfabrics.org/show_bug.cgi?id=1499

I've updated the issue form with your case, and 
we will continue following this bug there.

-- Yevgeny

Aaron Knister wrote:
> While the adapters have mellanox chipsets their actually IBM OEM
> branded and IBM hasn't released the 2.7 fw yet. I'm a little hesitant
> to apply the generic Mellanox FW.
> 
> On Mon, Oct 12, 2009 at 4:22 AM, Yevgeny Kliteynik
> <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>> Or Gerlitz wrote:
>>> Yevgeny Kliteynik wrote:
>>>> There was a hand-over problem in OFED 1.4, but later it turned  out to be
>>>> FW issue. The thing is, FW version 2.6.648 doesn't  have this bug any
>>>> more...
>>> so things should work fine with the newly released 2.7 firmware?
>> Yes
>>
>>> if this is still under question, Aaron, I suggest you open a bugzilla case
>>> @ https://bugs.openfabrics.org and we can track from there.
>> Good idea.
>>
>> -- Yevgeny
>>
>>> Or.
>>>
>>>
>>
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Aaron Knister]

Thanks! I really appreciate that.

I still have a question about the initial failover- I'm still
wondering why there's a 30 second delay. Wouldn't nodeA send some type
of handover message (my IB knowledge is limited) to notify a subnet
manager of a lower priority to take over? As I said, the older opensms
on the older mellanox model HCAs failsover and failsback instantly.

On Tue, Oct 13, 2009 at 11:32 AM, Yevgeny Kliteynik
<kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> Aaron,
>
> Thanks for the logs, this was really helpful.
> Looks like there is a handover race in the OSM -
> SM on node A misses the fact that SM on node B
> have gave up its mastership.
>
> There is a bugzilla issue the describes all the
> details of this race:
>
> https://bugs.openfabrics.org/show_bug.cgi?id=1499
>
> I've updated the issue form with your case, and we will continue following
> this bug there.
>
> -- Yevgeny
>
> Aaron Knister wrote:
>>
>> While the adapters have mellanox chipsets their actually IBM OEM
>> branded and IBM hasn't released the 2.7 fw yet. I'm a little hesitant
>> to apply the generic Mellanox FW.
>>
>> On Mon, Oct 12, 2009 at 4:22 AM, Yevgeny Kliteynik
>> <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>>>
>>> Or Gerlitz wrote:
>>>>
>>>> Yevgeny Kliteynik wrote:
>>>>>
>>>>> There was a hand-over problem in OFED 1.4, but later it turned  out to
>>>>> be
>>>>> FW issue. The thing is, FW version 2.6.648 doesn't  have this bug any
>>>>> more...
>>>>
>>>> so things should work fine with the newly released 2.7 firmware?
>>>
>>> Yes
>>>
>>>> if this is still under question, Aaron, I suggest you open a bugzilla
>>>> case
>>>> @ https://bugs.openfabrics.org and we can track from there.
>>>
>>> Good idea.
>>>
>>> -- Yevgeny
>>>
>>>> Or.
>>>>
>>>>
>>>
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Yevgeny Kliteynik]

Aaron Knister wrote:
> Thanks! I really appreciate that.
> 
> I still have a question about the initial failover- I'm still
> wondering why there's a 30 second delay. Wouldn't nodeA send some type
> of handover message (my IB knowledge is limited) to notify a subnet
> manager of a lower priority to take over?

When master SM dies, it can't notify anyone that it is dead.
Standby SM keeps polling the master SM, and if the latter is
dead, it will be getting timeouts on these polls.
The number of polls, and the time that passes between these
polls is configurable.
Default is 4 polls with 10 seconds waiting in between.

Note that SM fail-over might be very destructive to the 
traffic. Depending on the SM configuration, it can change LIDs,
routing, it will require flushing of all the path resolutions
on all the fabric nodes and refreshing all the multicast
membership in the subnet.

> As I said, the older opensms
> on the older mellanox model HCAs failsover and failsback instantly.

The instant failback is expected, and this is the bug that
we're discussing. As for the instant failover - I'll check
how the things supposed to work and get back to you.

-- Yevgeny

> On Tue, Oct 13, 2009 at 11:32 AM, Yevgeny Kliteynik
> <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>> Aaron,
>>
>> Thanks for the logs, this was really helpful.
>> Looks like there is a handover race in the OSM -
>> SM on node A misses the fact that SM on node B
>> have gave up its mastership.
>>
>> There is a bugzilla issue the describes all the
>> details of this race:
>>
>> https://bugs.openfabrics.org/show_bug.cgi?id=1499
>>
>> I've updated the issue form with your case, and we will continue following
>> this bug there.
>>
>> -- Yevgeny
>>
>> Aaron Knister wrote:
>>> While the adapters have mellanox chipsets their actually IBM OEM
>>> branded and IBM hasn't released the 2.7 fw yet. I'm a little hesitant
>>> to apply the generic Mellanox FW.
>>>
>>> On Mon, Oct 12, 2009 at 4:22 AM, Yevgeny Kliteynik
>>> <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>>>> Or Gerlitz wrote:
>>>>> Yevgeny Kliteynik wrote:
>>>>>> There was a hand-over problem in OFED 1.4, but later it turned  out to
>>>>>> be
>>>>>> FW issue. The thing is, FW version 2.6.648 doesn't  have this bug any
>>>>>> more...
>>>>> so things should work fine with the newly released 2.7 firmware?
>>>> Yes
>>>>
>>>>> if this is still under question, Aaron, I suggest you open a bugzilla
>>>>> case
>>>>> @ https://bugs.openfabrics.org and we can track from there.
>>>> Good idea.
>>>>
>>>> -- Yevgeny
>>>>
>>>>> Or.
>>>>>
>>>>>
>>
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Aaron Knister]

I wouldn't expect an SM that goes belly up to failover instantly, but
I would have thought that if you shut it down gracefully (with a
regular kill signal) that it could notify a standby SM that it's going
away. Either way, I appreciate the help!

On Tue, Oct 13, 2009 at 12:13 PM, Yevgeny Kliteynik
<kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> Aaron Knister wrote:
>>
>> Thanks! I really appreciate that.
>>
>> I still have a question about the initial failover- I'm still
>> wondering why there's a 30 second delay. Wouldn't nodeA send some type
>> of handover message (my IB knowledge is limited) to notify a subnet
>> manager of a lower priority to take over?
>
> When master SM dies, it can't notify anyone that it is dead.
> Standby SM keeps polling the master SM, and if the latter is
> dead, it will be getting timeouts on these polls.
> The number of polls, and the time that passes between these
> polls is configurable.
> Default is 4 polls with 10 seconds waiting in between.
>
> Note that SM fail-over might be very destructive to the traffic. Depending
> on the SM configuration, it can change LIDs,
> routing, it will require flushing of all the path resolutions
> on all the fabric nodes and refreshing all the multicast
> membership in the subnet.
>
>> As I said, the older opensms
>> on the older mellanox model HCAs failsover and failsback instantly.
>
> The instant failback is expected, and this is the bug that
> we're discussing. As for the instant failover - I'll check
> how the things supposed to work and get back to you.
>
> -- Yevgeny
>
>> On Tue, Oct 13, 2009 at 11:32 AM, Yevgeny Kliteynik
>> <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>>>
>>> Aaron,
>>>
>>> Thanks for the logs, this was really helpful.
>>> Looks like there is a handover race in the OSM -
>>> SM on node A misses the fact that SM on node B
>>> have gave up its mastership.
>>>
>>> There is a bugzilla issue the describes all the
>>> details of this race:
>>>
>>> https://bugs.openfabrics.org/show_bug.cgi?id=1499
>>>
>>> I've updated the issue form with your case, and we will continue
>>> following
>>> this bug there.
>>>
>>> -- Yevgeny
>>>
>>> Aaron Knister wrote:
>>>>
>>>> While the adapters have mellanox chipsets their actually IBM OEM
>>>> branded and IBM hasn't released the 2.7 fw yet. I'm a little hesitant
>>>> to apply the generic Mellanox FW.
>>>>
>>>> On Mon, Oct 12, 2009 at 4:22 AM, Yevgeny Kliteynik
>>>> <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>>>>>
>>>>> Or Gerlitz wrote:
>>>>>>
>>>>>> Yevgeny Kliteynik wrote:
>>>>>>>
>>>>>>> There was a hand-over problem in OFED 1.4, but later it turned  out
>>>>>>> to
>>>>>>> be
>>>>>>> FW issue. The thing is, FW version 2.6.648 doesn't  have this bug any
>>>>>>> more...
>>>>>>
>>>>>> so things should work fine with the newly released 2.7 firmware?
>>>>>
>>>>> Yes
>>>>>
>>>>>> if this is still under question, Aaron, I suggest you open a bugzilla
>>>>>> case
>>>>>> @ https://bugs.openfabrics.org and we can track from there.
>>>>>
>>>>> Good idea.
>>>>>
>>>>> -- Yevgeny
>>>>>
>>>>>> Or.
>>>>>>
>>>>>>
>>>
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Yevgeny Kliteynik]

Aaron,

Aaron Knister wrote:
>>> As I said, the older opensms
>>> on the older mellanox model HCAs failsover and failsback instantly.
>> The instant failback is expected, and this is the bug that
>> we're discussing. As for the instant failover - I'll check
>> how the things supposed to work and get back to you.

After checking this thing, I don't understand how the
instant failover is possible. The only case it can
work is if you don't have a switch in your subnet - 
just two HCAs connected directly to each other.
Is this the case?

If not, then I'd like to see opensm logs.
Please run opensm as before (-V -s 0 -e).
Start OSM on node A with high priority.
Start OSM on node B with low priority.
Kill OSM on node A, and see that OSM on
node B becomes master.

I need only the log of the opensm on node B.
Best if you could just attach it to the bugzilla
issue form, but if you can't - you can mail it to me.

-- Yevgeny

>> -- Yevgeny
>>
>>> On Tue, Oct 13, 2009 at 11:32 AM, Yevgeny Kliteynik
>>> <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>>>> Aaron,
>>>>
>>>> Thanks for the logs, this was really helpful.
>>>> Looks like there is a handover race in the OSM -
>>>> SM on node A misses the fact that SM on node B
>>>> have gave up its mastership.
>>>>
>>>> There is a bugzilla issue the describes all the
>>>> details of this race:
>>>>
>>>> https://bugs.openfabrics.org/show_bug.cgi?id=1499
>>>>
>>>> I've updated the issue form with your case, and we will continue
>>>> following
>>>> this bug there.
>>>>
>>>> -- Yevgeny
>>>>
>>>> Aaron Knister wrote:
>>>>> While the adapters have mellanox chipsets their actually IBM OEM
>>>>> branded and IBM hasn't released the 2.7 fw yet. I'm a little hesitant
>>>>> to apply the generic Mellanox FW.
>>>>>
>>>>> On Mon, Oct 12, 2009 at 4:22 AM, Yevgeny Kliteynik
>>>>> <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>>>>>> Or Gerlitz wrote:
>>>>>>> Yevgeny Kliteynik wrote:
>>>>>>>> There was a hand-over problem in OFED 1.4, but later it turned  out
>>>>>>>> to
>>>>>>>> be
>>>>>>>> FW issue. The thing is, FW version 2.6.648 doesn't  have this bug any
>>>>>>>> more...
>>>>>>> so things should work fine with the newly released 2.7 firmware?
>>>>>> Yes
>>>>>>
>>>>>>> if this is still under question, Aaron, I suggest you open a bugzilla
>>>>>>> case
>>>>>>> @ https://bugs.openfabrics.org and we can track from there.
>>>>>> Good idea.
>>>>>>
>>>>>> -- Yevgeny
>>>>>>
>>>>>>> Or.
>>>>>>>
>>>>>>>
>>
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Aaron Knister]

It will be difficult to get you those logs until the older cluster is
decommissioned (which in theory should be soon), but as soon as I am
able I will get them too you.

On Wed, Oct 14, 2009 at 4:46 AM, Yevgeny Kliteynik
<kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> Aaron,
>
> Aaron Knister wrote:
>>>>
>>>> As I said, the older opensms
>>>> on the older mellanox model HCAs failsover and failsback instantly.
>>>
>>> The instant failback is expected, and this is the bug that
>>> we're discussing. As for the instant failover - I'll check
>>> how the things supposed to work and get back to you.
>
> After checking this thing, I don't understand how the
> instant failover is possible. The only case it can
> work is if you don't have a switch in your subnet - just two HCAs connected
> directly to each other.
> Is this the case?
>
> If not, then I'd like to see opensm logs.
> Please run opensm as before (-V -s 0 -e).
> Start OSM on node A with high priority.
> Start OSM on node B with low priority.
> Kill OSM on node A, and see that OSM on
> node B becomes master.
>
> I need only the log of the opensm on node B.
> Best if you could just attach it to the bugzilla
> issue form, but if you can't - you can mail it to me.
>
> -- Yevgeny
>
>>> -- Yevgeny
>>>
>>>> On Tue, Oct 13, 2009 at 11:32 AM, Yevgeny Kliteynik
>>>> <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>>>>>
>>>>> Aaron,
>>>>>
>>>>> Thanks for the logs, this was really helpful.
>>>>> Looks like there is a handover race in the OSM -
>>>>> SM on node A misses the fact that SM on node B
>>>>> have gave up its mastership.
>>>>>
>>>>> There is a bugzilla issue the describes all the
>>>>> details of this race:
>>>>>
>>>>> https://bugs.openfabrics.org/show_bug.cgi?id=1499
>>>>>
>>>>> I've updated the issue form with your case, and we will continue
>>>>> following
>>>>> this bug there.
>>>>>
>>>>> -- Yevgeny
>>>>>
>>>>> Aaron Knister wrote:
>>>>>>
>>>>>> While the adapters have mellanox chipsets their actually IBM OEM
>>>>>> branded and IBM hasn't released the 2.7 fw yet. I'm a little hesitant
>>>>>> to apply the generic Mellanox FW.
>>>>>>
>>>>>> On Mon, Oct 12, 2009 at 4:22 AM, Yevgeny Kliteynik
>>>>>> <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>>>>>>>
>>>>>>> Or Gerlitz wrote:
>>>>>>>>
>>>>>>>> Yevgeny Kliteynik wrote:
>>>>>>>>>
>>>>>>>>> There was a hand-over problem in OFED 1.4, but later it turned  out
>>>>>>>>> to
>>>>>>>>> be
>>>>>>>>> FW issue. The thing is, FW version 2.6.648 doesn't  have this bug
>>>>>>>>> any
>>>>>>>>> more...
>>>>>>>>
>>>>>>>> so things should work fine with the newly released 2.7 firmware?
>>>>>>>
>>>>>>> Yes
>>>>>>>
>>>>>>>> if this is still under question, Aaron, I suggest you open a
>>>>>>>> bugzilla
>>>>>>>> case
>>>>>>>> @ https://bugs.openfabrics.org and we can track from there.
>>>>>>>
>>>>>>> Good idea.
>>>>>>>
>>>>>>> -- Yevgeny
>>>>>>>
>>>>>>>> Or.
>>>>>>>>
>>>>>>>>
>>>
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM Failover

[Aaron Knister]

Alright I'm sending this for the 3rd time in plain text because
apparently vger wont' take html mail. Fair 'nuff.


I double checked and firmware version 2.6.648 is in use on the cluster
with the failover issue. The cluster that doesn't have the issue is
running a MT25204 Mellanox HCAs with firmware 1.1.0. I'm also curious
about the initial 30 second delay if the active subnet manager is shut
down gracefully. The OpenSMs on the cluster with the MT25204 HCAs
failover instantly when one disappears. I ran opensm with the options
you provided and re-created the failover failback scenario.

On Mon, Oct 12, 2009 at 3:14 AM, Yevgeny Kliteynik
<kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>
> Aaron,
>
> Aaron Knister wrote:
>>
>> I just stumbled across this in the release notes for opensm 3.2.6-
>>
>> "* SMs do not hand-over when running on ConnectX in a switch-based topology."
>>
>> So I guess that answers the question of whether or not what I'm seeing is "expected behavior". Out of curiosity what are the technical reasons for this? I just tried opensm 3.3.2 and I still experience the same behavior.
>
> There was a hand-over problem in OFED 1.4, but later it turned
> out to be FW issue. The thing is, FW version 2.6.648 doesn't
> have this bug any more...
>
> The 30 seconds for the initial failover is expected, but the 40 second failback when the original master comes back is a problem.
>
> Can you please double check that the FW version 2.6.648 is used on
> both HCAs that run OSM?
> And what is the FW version of HCAs that don't have this problem?
>
> Also, can you please reproduce the issue running OSM as follows:
>        opensm -V -e -s 0
> on both nodes and attach the /var/log/opensm.log files?
>
> -- Yevgeny
>
>
>> On Oct 10, 2009, at 7:38 PM, Aaron Knister wrote:
>>
>>> I'm not sure if this is the right place to post about this issue, but here goes-
>>>
>>> I'm having problems with OpenSM failover.
>>>
>>> I have two nodes running opensmd version "3.2.6_20090317" from RHEL 5.4. I'm using a configuration file on both generated using opensm -c. When I start the subnet manager on node a, everything is fine. It appears to reassign itself a lid of 1 which I think is expected. When I started the subnet manager on node b everything is fine. If you query its lid it shows the subnet manager in a standby state. Now for the fun. If I stop opensmd on node a (service opensmd stop) then all of the traffic on the fabric stops. It takes node b's OpenSM instance about 30 seconds to realize that node a's subnet manager is dead and come up in the master state. Now when node a's subnet manager comes back (service opensmd start), all traffic on the fabric stops and node b's subnet manager goes into the standby state...but node a's subnet manager doesn't take over the fabric and come up as master for about another 40 seconds (during this time the no traffic passes over the fabric). The below logs should help illustrate what I'm seeing
>>>
>>>
>>> Oct 10 19:14:14 node-a OpenSM[14132]: Entering DISCOVERING state
>>> Oct 10 19:14:14 node-a OpenSM[14132]: Entering MASTER state
>>> Oct 10 19:14:14 node-a OpenSM[14132]: SUBNET UP
>>>
>>> Oct 10 19:14:25 node-b OpenSM[11197]: /var/log/opensm.log log file opened
>>> Oct 10 19:14:25 node-b OpenSM[11197]: OpenSM 3.2.6_20090317
>>> Oct 10 19:14:25 node-b OpenSM[11197]: Entering DISCOVERING state
>>> Oct 10 19:14:26 node-b OpenSM[11197]: Entering STANDBY state
>>>
>>> Oct 10 19:15:44 node-a OpenSM[14132]: Exiting SM
>>> Oct 10 19:16:16 node-b OpenSM[11197]: Entering DISCOVERING state
>>> Oct 10 19:16:16 node-b OpenSM[11197]: Entering MASTER state
>>>
>>> Oct 10 19:18:52 node-a OpenSM[14213]: /var/log/opensm.log log file opened
>>> Oct 10 19:18:52 node-a OpenSM[14213]: OpenSM 3.2.6_20090317
>>> Oct 10 19:18:52 node-a OpenSM[14213]: Entering DISCOVERING state
>>> Oct 10 19:18:53 node-b OpenSM[11197]: Entering STANDBY state
>>> Oct 10 19:18:53 node-a OpenSM[14213]: Entering STANDBY state
>>> Oct 10 19:19:33 node-a OpenSM[14213]: Entering DISCOVERING state
>>> Oct 10 19:19:33 node-a OpenSM[14213]: Entering MASTER state
>>> Oct 10 19:19:33 node-a OpenSM[14213]: SUBNET UP
>>>
>>> We have opensm 3.2.5_20081207 (ofed 1.4) on another cluster and it fails over and fails back almost instantly with seemingly no traffic interruption if you gracefully stopped the active opensmd instance (service opensmd stop). Is the behavior I'm seeing considered normal? I can understand the 30 seconds for the initial failover but why the 40 second failback when the original master comes back? Any help is appreciated :)
>>>
>>> BTW my switch is a Qlogic 12800-180 with the latest firmware and the HCAs are Mellanox MT26428 running firmware version 2.6.648.
>>>
>>> Thanks!
>>>
>>> -Aaron
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html