[REGRESSION] failure to reconnect on SMB server restart with custom TCP port (not 445): Host is down (at least since 6.6.95)

["Dr. Thomas Orgis" <thomas.orgis@uni-hamburg.de>]

Dear Linux-CIFS maintainer(s),

I stumbled upon a regression in the Linux cifs/smb3 client when working
with a smbd using a non-standard port. I am not the first to note this, see

	https://bbs.archlinux.org/viewtopic.php?id=306712

which is a report from mid last year, indicating the problem sometime
after Linux 6.6.72. It is a very simple issue, where details of the
kernel builds or mount setup don't seem to matter much: Older kernels
reconnect to a SMB server that was restarted (old processes killed and
replaced), newer kernels do not and just have a defunct mount.

I reproduced this in our HPC cluster environment with such smb.conf on
the server side

[global]
security = user
map to guest = Bad Password
server role = standalone server
smb ports = 1445

[public]
path = /some/path
guest ok = yes
read only = yes

and such a mount command on the client:

mount -t smb3 -o port=1445,user=guest,password=foo //server/public dir

When I kill and re-start smbd on the server, older client kernels
reconnect and continue to return listings and files from the share,
while newer kernels give this:

$ ls dir
ls: cannot access 'dir': Host is down

I also see that no smbd process is spawned for the client connection.
The client simply seems not to try to talk to the server anymore. This
apparently is only the case with the changed SMB port. With the default
445, the same setup works. I once tested that with root running smbd or
by forwarding port 445 to 1445 via socat on the server, things are back
to normal. The share continues working, server processes are spawned
and talked to by the client.

Also, it seems that this only covers cases with actual network
transfer. Mounting on the server from itself (localhost) seems to be
fine on either port.

I narrowed this down to these kernels working in the same system (which
is a HPC node booting a Debian 12 image with our custom kernel built
from vanilla kernel.org sources, barely changed):

6.6.53: good
6.6.78: good
6.6.86: good

Also, an Ubuntu client with 5.15.0-164-generic in a distinct setup,
talking to a differing server, was also fine. So I assume clients
reconnecting is baseline behaviour. I tested these kernel versions to
have the regression:

6.6.95: bad
6.6.97: bad
6.6.101: bad
6.12.47: bad
6.12.55: bad
6.18.16: bad
7.0.0-rc3: bad

The last one is torvalds/linux.git as of today, commit
1f318b96cc84d7c2ab792fcc0bfd42a7ca890681.

So it looks like something backported into 6.6 series between 6.6.86
and 6.6.95 broke the behaviour and this stayed with us up to the
current RC. This matches the report in the Arch Linux forums. I tested
by building kernels and booting real hardware … so this already
took some time and effort. I can test a specific patch/version
quite easily, but don't have a fully automatic bisecting setup.

The issue should be obvious in any environment, though. It seems to be
universal once you choose a custom port and have some real (or
emulated?) network between client and server.

Hopefully an easy fix for someone close to the code?


Alrighty then,

Thomas

PS: $ grep -e SMB -e CIFS  .config |grep -i -v -e SMBUS -e SMBIOS
CONFIG_RMI4_SMB=m
# CONFIG_CHARGER_SMB347 is not set
CONFIG_CIFS=m
CONFIG_CIFS_STATS2=y
CONFIG_CIFS_ALLOW_INSECURE_LEGACY=y
CONFIG_CIFS_UPCALL=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_CIFS_DEBUG=y
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_DEBUG_DUMP_KEYS is not set
CONFIG_CIFS_DFS_UPCALL=y
# CONFIG_CIFS_SWN_UPCALL is not set
# CONFIG_CIFS_SMB_DIRECT is not set
CONFIG_CIFS_FSCACHE=y
# CONFIG_CIFS_COMPRESSION is not set
CONFIG_SMB_SERVER=m
# CONFIG_SMB_SERVER_SMBDIRECT is not set
CONFIG_SMB_SERVER_CHECK_CAP_NET_ADMIN=y
# CONFIG_SMB_SERVER_KERBEROS5 is not set
CONFIG_SMBFS=m

PPS: I am testing running Samba smbd as unprivileged user on a
non-standard port to find a cheap/simple least-privilege way to
re-export a custom FUSE fs to clients, read-only. Samba smbd doesn't
really like that, but can be coerced, esp. in a mount namespace where
the hardcoded paths it uses can be remapped. I could add more layers to
map the port back to 445 or do something else entirely, but maybe this
is a half-reasonable scenario why one would like to talk SMB on a
different port.

-- 
Dr. Thomas Orgis
HPC @ Universität Hamburg