Re: Failure to reconnect after cluster failvoer

[Ross Lagerwall <ross.lagerwall@citrix.com>]

On 2/22/19 11:25 PM, Tom Talpey wrote:
>> -----Original Message-----
>> From: Ross Lagerwall <ross.lagerwall@citrix.com>
>> Sent: Friday, February 22, 2019 9:17 AM
>> To: Tom Talpey <ttalpey@microsoft.com>; Steve French
>> <smfrench@gmail.com>
>> Cc: CIFS <linux-cifs@vger.kernel.org>
>> Subject: Re: Failure to reconnect after cluster failvoer
>>
>> On 2/21/19 5:59 PM, Tom Talpey wrote:
>>> The reconnect is apparently using a dotted-quad as the servername, and you
>> can see the auth is forced to NTLM as a consequence. Is that the way you
>> initially mounted the share (i.e. mount 10.71.217.50:/smbshare /mnt)?
>>>
>>> -----Original Message-----
>>> From: linux-cifs-owner@vger.kernel.org <linux-cifs-owner@vger.kernel.org>
>> On Behalf Of Steve French
>>> Sent: Thursday, February 21, 2019 9:07 AM
>>> To: Ross Lagerwall <ross.lagerwall@citrix.com>
>>> Cc: CIFS <linux-cifs@vger.kernel.org>
>>> Subject: Re: Failure to reconnect after cluster failvoer
>>>
>>> Couple quick thoughts.
>>>
>>> Does this work on current kernels (5.0 for example).
>>>
>>> Was thinking about patches that might affect this like:
>>> - "cifs: connect to servername instead of IP for IPC$ share"
>>> - "smb3: on reconnect set PreviousSessionId field"
>>> - Paulo's patches (has cifs-utils coreq) to reconnect to new IP
>>> address if hostname's IP address changed and his add support for
>>> failover
>>> - Paulo's patch to remove trailing slashes from server UNC name
>>>
>> I've reproduced this with 5.0-rc7 and the latest cifs-utils from git.
>> The share was mounted as follows (yes, by IP):
>>
>> mount.cifs -o
>> vers=3.0,cache=loose,actimeo=0,username=x,domain=y,password=z
>> '//10.71.217.31/smbshare' /mnt
>>
>> Here is the tcpdump when it fails to reconnect properly:
> ...
>>
>> The initial connection is at timestamp 0s, reconnection at 13s,
>> STATUS_NETWORK_NAME_DELETED at 60s.
>>
>> For comparison, here is a tcpdump using the "fix" from my previous mail:
> ...
>>
>> The initial connection is at timestamp 0s, reconnection at 34s,
>> successful read request at 215s.
>>
>> Note that the tree connect for IPC$ only happens _after_ the tree
>> connect for the share succeeds.
> 
> Thanks for the full traces, they clarify the situation. But, I don’t see any
> meaningful difference in the client behavior. The ordering of the two
> treeconnects is the same between the two - initially, "IPC$" then
> "smbshare", and on reconnect, the other way around. So, I'm unclear
> whether your patch did anything.

There is definitely a difference. Before the patch, on reconnect the client:

* Connects to "smbshare" which fails
* Then connects to "IPC$" which succeeds
* Then tries again to connect to smbshare which fails repeatedly

After the patch, on reconnect the client:

* Connects to "smbshare" which fails
* Then tries again to connect to "smbshare" which succeeds after several 
retries
* Then tries to connect to "IPC$" which succeeds

This subtle reordering somehow makes it work. It may indeed be a server 
bug rather than a client bug. I was hoping someone could shed some light 
on this.

> 
> The STATUS_NETWORK_NAME_DELETED is a consequence of the failed
> re-establishment of the tree connect, and is not itself the problem. The
> server is simply timing out the treeid, since the client did not successfully
> reclaim it. The repeated STATUS_BAD_NETWORK_NAME is the issue.
> 
> Are you sure the clustered server is recovering properly when you are
> forcing the failover? For example, if it's a two-node cluster, maybe node A
> can take over node B, but node B has issues taking over node A. Is there
> anything relevant in the server logs?
> 

It's a two node cluster. The behaviour happens reliably when failing 
over either way. After failover, the server state is consistent. E.g. 
after a failover from node A to node B, node B shows itself as the 
primary server and the node A is marked as down. I couldn't find 
anything interesting in the server logs.

Thanks,
-- 
Ross Lagerwall