Re: Potential smb/dfs regression introduced in kernel 6.6

["Steinbeißer, Sebastian" <Sebastian.Steinbeisser@lrz.de>]

[-- Attachment #1.1: Type: text/plain, Size: 6282 bytes --]

Dear Greg,
Dear Paulo,

Have you been able to take a look at the logs I sent?
Is there any additional information I can provide?

Thanks!

Best
Sebastian
+-------------------------------------------------------------+
 Sebastian Steinbeißer
 Leibniz-Rechenzentrum   Tel.:+49-89-35831-8779
 Boltzmannstraße 1       E-Mail: Sebastian.Steinbeisser@lrz.de
 D-85748 Garching
+-------------------------------------------------------------+


-----Original Message-----
From: Sebastian Steinbeißer <Sebastian.Steinbeisser@lrz.de>
To: Greg KH <gregkh@linuxfoundation.org>, Paulo Alcantara
<pc@manguebit.com>
Cc: regressions@lists.linux.dev <regressions@lists.linux.dev>
Subject: Re: Potential smb/dfs regression introduced in kernel 6.6
Date: 28.06.2024 10:19:24

Dear Greg,
Dear Paulo,

thanks for the quick reply and sorry for my late one - getting
information internally and from the customer who reported the issue
took some time.
Here we go:

1. We do currently not have the capacity to bisect, but the customer
reporting the issue speculated that it might have been introduced
between commits 9a49e221a64111535f65e9c8804db24d11eaff8b and
539aad7f14dab7f947e5ab81901c0b20513a50db

2. I can not for sure confirm or deny if the subvolume is realized as a
DFS link.

3. The logs as requested are attached.

4. About the setup used in the example:
Two fresh virtual machines running lubuntu 22.04 and 24.04 with kernels
6.5.0 and 6.8.0, respectively.
Both of them were installed today and updated.
After that the smb-client and cifs-utils packages were installed and
both VMs were rebooted.
The tracing was done in parallel.

4. About the infrastructure in the example:
The "badw" share contains several folders, three of which are realized
as their own volumes:

volume         junction-path
-------------- ---------------
mwn_share_badw /mwn_share_badw
mwn_share_badwifv
               /mwn_share_badw/ifv
mwn_share_badwlrz
               /mwn_share_badw/lrz
mwn_share_badwver
               /mwn_share_badw/ver
These are the ones that can not be accessed -> see the ls_error_message
attachement.
We're exporting shares with CIFS 3.2 and the following security
settings:
                            Kerberos Clock Skew: 5 minutes
                            Kerberos Ticket Age: 10 hours
                           Kerberos Renewal Age: 7 days
                           Kerberos KDC Timeout: 3 seconds
                            Is Signing Required: true
                Is Password Complexity Required: true
           Use start_tls for AD LDAP connection: true
         (DEPRECATED)-Is AES Encryption Enabled: true
                         LM Compatibility Level: lm-ntlm-ntlmv2-krb
                     Is SMB Encryption Required: true
                        Client Session Security: seal
   (DEPRECATED)-SMB1 Enabled for DC Connections: false
                SMB2 Enabled for DC Connections: true
  LDAP Referral Enabled For AD LDAP connections: false
               Use LDAPS for AD LDAP connection: false
      Encryption is required for DC Connections: true
   AES session key enabled for NetLogon channel: false
    Try Channel Binding For AD LDAP Connections: true
        Encryption Types Advertised to Kerberos: aes-256, aes-128, rc4,
des


Thus traces are most likely encrypted.

Thanks a lot!

Best
Sebastian
+-------------------------------------------------------------+
 Sebastian Steinbeißer
 Leibniz-Rechenzentrum   Tel.:+49-89-35831-8779
 Boltzmannstraße 1       E-Mail: Sebastian.Steinbeisser@lrz.de
 D-85748 Garching
+-------------------------------------------------------------+


-----Original Message-----
From: Paulo Alcantara <pc@manguebit.com>
To: "Steinbeißer, Sebastian" <Sebastian.Steinbeisser@lrz.de>
Subject: Re: Potential smb/dfs regression introduced in kernel 6.6
Date: 27.06.2024 17:50:07

Paulo Alcantara <pc@manguebit.com> writes:

> Steinbeißer, Sebastian  <Sebastian.Steinbeisser@lrz.de> writes:
> 
> > The way this works with NetApp is that a volume, e.g., myvol,
> > within
> > the storage virtual machine (SVM - vserver) is exported and can be
> > accessed by mounting //SVM_IP/myvol into, e.g., /mnt/myvol
> > Subvolumes can be added to the junction path, e.g., mysubvol can be
> > mounted under myvol and used to be transparently accessable via
> > /mnt/myvol/mysubvol just like any directory within myvol would be.
> > This now is no longer the case and the subvolume needs to be
> > mounted
> > separately.
> > 
> > This has started when customers upgraded distributions that shipped
> > with kernel >=6.6.
> > https://wiki.samba.org/index.php/LinuxCIFSKernel#6.6_(module_version_2.45)_(47_changesets)
> > 
> > We've been able to reproduce the behavior with Fedora/Ubuntu and
> > kernels 6.6 to 6.9.
> 
> Are you talking about DFS shares?  The "mysubvol" would be an DFS
> link?

Can you please provide verbose logs and network traces of both working
and non-working kernels:

 # umount all SMB shares in the system
 dmesg --clear
 modprobe dns_resolver
 modprobe cifs
 echo 'module cifs +p' > /sys/kernel/debug/dynamic_debug/control
 echo 'file fs/cifs/* +p' > /sys/kernel/debug/dynamic_debug/control
 echo 1 > /proc/fs/cifs/cifsFYI
 echo 1 > /sys/module/dns_resolver/parameters/debug
 echo 0 > /proc/sys/kernel/printk_ratelimit
 echo 0 > /proc/fs/cifs/dfscache
 tcpdump -s 0 -w trace.pcap port 445 & pid=$!
 cat /proc/fs/cifs/DebugData > trace.log
 cat /proc/fs/cifs/dfscache >> trace.log
 dmesg >> trace.log
 # mount.cifs ... /mnt -o ...
 # attempt to subvol under mounted share (i.e. ls /mnt/mysubvol)
 cat /proc/fs/cifs/DebugData >> trace.log
 cat /proc/fs/cifs/dfscache >> trace.log
 dmesg >> trace.log
 sleep 3
 kill $pid

Then send trace.log and trace.pcap files of both kernels.

Thanks.

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5874 bytes --]