Linux CIFS client, Windows 2012 CIFS file server => files not found?

Linux CIFS client, Windows 2012 CIFS file server => files not found?

[Kristian Rink]

Folks;

the last few years we've been locally serving files using a NetApp 
storage system exposing CIFS and several (Windows, Linux) clients 
mounting these shares in order to host data and run an internal business 
application.

The application itself (binaries, libraries, resources, a bunch of ksh 
scripts) entirely lives on the CIFS share. So far (having NetApp CIFS 
mounted), this worked on all machines.

Now, these CIFS shares are to be provided by a Windows 2012 Storage 
Server. Outcome: Linux machines can't run this application anymore. 
Having mounted the W2012 CIFS share, the application won't start 
complaining that some resources (scripts and libraries) are missing, 
even while they are obviously around and also, permission-wise, accessible.

It's difficult to really figure out what happens here. Basically, 
looking at strace output, it seems the application does stat64() one of 
these files which in both cases seems to provide a valid outcome...

stat64("/opt/.../file.oeb", {st_mode=S_IFREG|0755, st_size=36405, ...}) = 0

... but while mounting the share off the NetApp the application works as 
desired, mounting the share off the W2012 server the application will 
terminate here complaining about this particular resource missing.

Vague as it is, does anyone have any idea what could be causing this? 
I've been playing with various parameters of mount.cifs so far, but I'm 
not deep enough into either NetApps nor Windows 2012s understanding of 
CIFS to have an idea what might have changed here...

TIA,
Kristian

Re: Linux CIFS client, Windows 2012 CIFS file server => files not found?

[Steve French]

any chance you could send me a small capture file of each (preferably
the smallest possible reproduction scenario so the trace isn't big and
cluttered) so I can compare the two server's responses to see what
might be going on.

(see the "wire captures" section of
https://wiki.samba.org/index.php/LinuxCIFS_troubleshooting)

On Thu, Aug 20, 2015 at 8:32 AM, Kristian Rink <kawazu428-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Folks;
>
> the last few years we've been locally serving files using a NetApp storage
> system exposing CIFS and several (Windows, Linux) clients mounting these
> shares in order to host data and run an internal business application.
>
> The application itself (binaries, libraries, resources, a bunch of ksh
> scripts) entirely lives on the CIFS share. So far (having NetApp CIFS
> mounted), this worked on all machines.
>
> Now, these CIFS shares are to be provided by a Windows 2012 Storage Server.
> Outcome: Linux machines can't run this application anymore. Having mounted
> the W2012 CIFS share, the application won't start complaining that some
> resources (scripts and libraries) are missing, even while they are obviously
> around and also, permission-wise, accessible.
>
> It's difficult to really figure out what happens here. Basically, looking at
> strace output, it seems the application does stat64() one of these files
> which in both cases seems to provide a valid outcome...
>
> stat64("/opt/.../file.oeb", {st_mode=S_IFREG|0755, st_size=36405, ...}) = 0
>
> ... but while mounting the share off the NetApp the application works as
> desired, mounting the share off the W2012 server the application will
> terminate here complaining about this particular resource missing.
>
> Vague as it is, does anyone have any idea what could be causing this? I've
> been playing with various parameters of mount.cifs so far, but I'm not deep
> enough into either NetApps nor Windows 2012s understanding of CIFS to have
> an idea what might have changed here...
>
> TIA,
> Kristian
> --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Thanks,

Steve

Re: Linux CIFS client, Windows 2012 CIFS file server => files not found?

[Kristian Rink]

Hi Steve;

and first off, thanks loads for your feedback.

Am 20.08.2015 um 18:11 schrieb Steve French:
> any chance you could send me a small capture file of each (preferably
> the smallest possible reproduction scenario so the trace isn't big and
> cluttered) so I can compare the two server's responses to see what
> might be going on.

Sure, it's easy to reproduce, and I ended up with two pcap files. Mind 
if I send these off-list, or should I rather upload them anywhere?

Thanks again and all the best,
Kristian

Re: Linux CIFS client, Windows 2012 CIFS file server => files not found?

[Steve French]

On Thu, Aug 20, 2015 at 2:36 PM, Kristian Rink <kawazu428-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi Steve;
>
> and first off, thanks loads for your feedback.
>
> Am 20.08.2015 um 18:11 schrieb Steve French:
>>
>> any chance you could send me a small capture file of each (preferably
>> the smallest possible reproduction scenario so the trace isn't big and
>> cluttered) so I can compare the two server's responses to see what
>> might be going on.
>
>
> Sure, it's easy to reproduce, and I ended up with two pcap files.

Last thing in the trace is a parallelized read with lots of reads in
flight at once which should be ok and I do see a response.

Two obvious things to ask:

1) What dialect are you using when you mount to Windows?  It looks
like SMB2.0 (and would be MUCH better to use SMB2.1 or SMB3 or
SMB3.02, or even CIFS when mounting to Windows, SMB2.0 is old and does
not support large reads/writes and it looks like it is failing )
rather than the default CIFS.  To NetApp you are mounting using CIFS
(not SMB2 or SMB3) which seems to work ok for your workload looking at
the trace.

Note that you should try the same dialect on both the mount to NetApp
and Windows to make the comparison easier:  e.g. after -o in the mount
command specify "vers=2.1" or "vers=3.0" (or don't specify any vers
(or "vers=1.0") and it should default to cifs)

2) Are there any things logged in the message log on the Linux client
(type "dmesg" to see what is logged) by cifs.ko during the failure on
running to Windows.   Does the status of the session show disconnected
at the end of the failure to Windows?   (See the "SMB session status"
and "TCP status" in the output of "cat /proc/fs/cifs/DebugData")


-- 
Thanks,

Steve

Re: Linux CIFS client, Windows 2012 CIFS file server => files not found?

[Kristian Rink]

Hi Steve, *;

Am 20.08.2015 um 22:23 schrieb Steve French:
>
> Last thing in the trace is a parallelized read with lots of reads in
> flight at once which should be ok and I do see a response.

Ok. It's a bit strange, we used to have a similar behaviour a while ago
after switching the NetApp filer to SMB2.0, which would end up in a
situation in which some (Linux) part applications created files on the
CIFS share which this particular application was unable to see. In that
particular case however it seemed a timing problem in the application
and happened when creating files on CIFS from Linux and trying to access
them from Windows clients.


>
> Note that you should try the same dialect on both the mount to
> NetApp and Windows to make the comparison easier:  e.g. after -o in
> the mount command specify "vers=2.1" or "vers=3.0" (or don't specify
> any vers (or "vers=1.0") and it should default to cifs)

I tried all dialects while connecting to Windows, which didn't change 
this behaviour, just mounting with vers=3.0 makes it fail considerably 
faster.

Actually, to use the same dialect for both mounts means using vers=1.0; 
though the NetApp filer does support some understanding of at least 
SMB2.0 (and the shares can be accessed using SMB2.0 from Windows 
clients), mount.cifs can't mount any of our NetApp shares using SMB2.0.

We figured this out only a few weeks ago when starting moving our stores 
and tried to mount things consistently using SMB2.0 for performance 
reasons, and mounting the NetApp shares using anything else than CIFS 
will reproducibly end up in errors like this:

-------------------
mount error(95): Operation not supported
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

[3959136.173555] CIFS VFS: cifs_read_super: get root inode failed
-------------------

That's a different problem however and I am unsure whether this is a 
general problem or just an issue our fairly old NetApp / ONTAP version 
is to blame for. I surely can provide you with more information on that 
if that is of interest.


> 2) Are there any things logged in the message log on the Linux
> client (type "dmesg" to see what is logged) by cifs.ko during the
> failure on running to Windows.

No. Unfortunately not.

> Does the status of the session show
> disconnected at the end of the failure to Windows?   (See the "SMB
> session status" and "TCP status" in the output of "cat
> /proc/fs/cifs/DebugData")

Unsure. This is what DebugData so far says for this particular filer 
connection:


CIFS Version 2.03
Features: dfs fscache lanman posix spnego xattr acl
Active VFS Requests: 0
Servers:
1) entry for 192.168.1.138 not fully displayed
         TCP status: 1
         Local Users To Server: 1 SecMode: 0x3 Req On Wire: 0
         Shares:
         1) \\192.168.1.138\Store Mounts: 1 Type: NTFS DevInfo: 0x60020
		Attributes: 0xc700ff
         PathComponentMax: 255 Status: 0x1 type: DISK
         MIDs:


Can you make anything of this?

Thanks bunches and all the best,
Kristian

Re: Linux CIFS client, Windows 2012 CIFS file server => files not found?

[Kristian Rink]

Hi Benjamin;

and thanks for your feedback. :)

Am 21.08.2015 um 10:57 schrieb Benjamin Bellec:
 >
> I have exactly the same error and kernel message when trying to mount
> our NetApp (FAS2240) on Fedora 20 or RHEL7. I already mentioned this
> issue some months ago, look at the Steve answers here :
> https://bugzilla.samba.org/show_bug.cgi?id=8914#c37

Oh, ok. I see. Well, so I'm sorted here - not much of a problem as our 
FAS2020 (running ONTAP 7.3.something) is on its way out also due to 
other issues. Not sure whether a Windows based file server is a better 
choice, though. Let's see... ;)

Best,
Kristian