regressions & flakiness make for ugly symlink links

regressions & flakiness make for ugly symlink links

[L A Walsh]

In my Windows root directory I have several links.  This is how
they look on the Windows machine's "root".  Several months ago I
sent an email to this list complimenting you on how they looked.
(without user+group)

> ll |grep -- '->'
lrwxrwxrwx   1           17 Jun 13  2016 Documents -> //Bliss/Documents/
lrwxrwxrwx   1            6 Jul 13  2009 Documents and Settings -> /Users/
lrwxrwxrwx   1           16 Jun  5  2015 FolderChanger -> /m/FolderChanger/
lrwxrwxrwx   1            5 May 13  2017 Home -> Users/
lrwxrwxrwx   1           20 Nov  6  2014 Prog -> /Program Files (x86)/
lrwxrwxrwx   1           13 Apr 21  2013 Prog64 -> Program Files/
lrwxrwxrwx   1           12 Aug  9  2015 ProgD -> /ProgramData/
lrwxrwxrwx   1            2 Apr 17  2017 Share -> /s/
lrwxrwxrwx   1           22 Aug 22 16:08 Symbols -> //Bliss/Share/Symbols//
lrwxrwxrwx   1            7 Mar  1  2018 WINNT -> Windows/
lrwxrwxrwx   1            3 May 13  2017 lib64 -> lib/
lrwxrwxrwx   1            3 Jan 12  2014 temp -> tmp/

Now, I'm getting flakiresults -- with them not resolving 1 time,
then resolving,  then not again...etc.  By flakey I mean
output like random can't access messages, followed by question
marks in all fields of links.

# ll|more
ls: cannot access 'D': Operation not supported
ls: cannot access 'Documents': Operation not supported
ls: cannot access 'FolderChanger': Operation not supported
ls: cannot access 'Home': Operation not supported
ls: cannot access 'lib64': Operation not supported
ls: cannot access 'M': Operation not supported
ls: cannot access 'P': Operation not supported
ls: cannot access 'pagefile.sys': Device or resource busy
ls: cannot access 'Prog64': Operation not supported
ls: cannot access 'Share': Operation not supported
ls: cannot access 'temp': Operation not supported
ls: cannot access 'WINNT': Operation not supported
d????????? ?       ?            ? D/
d????????? ?       ?            ? Documents/
d????????? ?       ?            ? FolderChanger/
d????????? ?       ?            ? Home/
d????????? ?       ?            ? M/
d????????? ?       ?            ? P/
d????????? ?       ?            ? Prog64/
d????????? ?       ?            ? Share/
d????????? ?       ?            ? WINNT/
d????????? ?       ?            ? lib64/
d????????? ?       ?            ? temp/

Depending on timing, Sometimes I will get most or all of
them appearing the same as they would on windows.

If I can't resolve them more quickly, is there a know
or two to increase cache size and item lifetime?

Thanks

Re: regressions & flakiness make for ugly symlink links

[Pavel Shilovsky]

пн, 18 февр. 2019 г. в 22:16, L A Walsh <cifs@tlinx.org>:
>
>
> In my Windows root directory I have several links.  This is how
> they look on the Windows machine's "root".  Several months ago I
> sent an email to this list complimenting you on how they looked.
> (without user+group)
>
> > ll |grep -- '->'
> lrwxrwxrwx   1           17 Jun 13  2016 Documents -> //Bliss/Documents/
> lrwxrwxrwx   1            6 Jul 13  2009 Documents and Settings -> /Users/
> lrwxrwxrwx   1           16 Jun  5  2015 FolderChanger -> /m/FolderChanger/
> lrwxrwxrwx   1            5 May 13  2017 Home -> Users/
> lrwxrwxrwx   1           20 Nov  6  2014 Prog -> /Program Files (x86)/
> lrwxrwxrwx   1           13 Apr 21  2013 Prog64 -> Program Files/
> lrwxrwxrwx   1           12 Aug  9  2015 ProgD -> /ProgramData/
> lrwxrwxrwx   1            2 Apr 17  2017 Share -> /s/
> lrwxrwxrwx   1           22 Aug 22 16:08 Symbols -> //Bliss/Share/Symbols//
> lrwxrwxrwx   1            7 Mar  1  2018 WINNT -> Windows/
> lrwxrwxrwx   1            3 May 13  2017 lib64 -> lib/
> lrwxrwxrwx   1            3 Jan 12  2014 temp -> tmp/
>
> Now, I'm getting flakiresults -- with them not resolving 1 time,
> then resolving,  then not again...etc.  By flakey I mean
> output like random can't access messages, followed by question
> marks in all fields of links.
>
> # ll|more
> ls: cannot access 'D': Operation not supported
> ls: cannot access 'Documents': Operation not supported
> ls: cannot access 'FolderChanger': Operation not supported
> ls: cannot access 'Home': Operation not supported
> ls: cannot access 'lib64': Operation not supported
> ls: cannot access 'M': Operation not supported
> ls: cannot access 'P': Operation not supported
> ls: cannot access 'pagefile.sys': Device or resource busy
> ls: cannot access 'Prog64': Operation not supported
> ls: cannot access 'Share': Operation not supported
> ls: cannot access 'temp': Operation not supported
> ls: cannot access 'WINNT': Operation not supported
> d????????? ?       ?            ? D/
> d????????? ?       ?            ? Documents/
> d????????? ?       ?            ? FolderChanger/
> d????????? ?       ?            ? Home/
> d????????? ?       ?            ? M/
> d????????? ?       ?            ? P/
> d????????? ?       ?            ? Prog64/
> d????????? ?       ?            ? Share/
> d????????? ?       ?            ? WINNT/
> d????????? ?       ?            ? lib64/
> d????????? ?       ?            ? temp/
>
> Depending on timing, Sometimes I will get most or all of
> them appearing the same as they would on windows.
>
> If I can't resolve them more quickly, is there a know
> or two to increase cache size and item lifetime?
>
> Thanks
>

Which kernel version shows symlinks right and which - wrong?

--
Best regards,
Pavel Shilovsky

Re: regressions & flakiness make for ugly symlink links

[L A Walsh]

On 2/19/2019 11:46 AM, Pavel Shilovsky wrote:
> пн, 18 февр. 2019 г. в 22:16, L A Walsh <cifs@tlinx.org>:
>   
>> In my Windows root directory I have several links...
>> lrwxrwxrwx   1           17 Jun 13  2016 Documents -> //Bliss/Documents/    
>> lrwxrwxrwx   1           20 Nov  6  2014 Prog -> /Program Files (x86)/
>> lrwxrwxrwx   1           13 Apr 21  2013 Prog64 -> Program Files/
>> lrwxrwxrwx   1           12 Aug  9  2015 ProgD -> /ProgramData/
>>     
...
>> ---
>>
>> Now, I'm getting flaki results -- with them not resolving 1 time,
>> then resolving,  then not again...etc
>>
>> # ll|more
>> ls: cannot access 'Documents': Operation not supported
>> ...
>> ls: cannot access 'Prog64': Operation not supported
>> ...
---
>> d????????? ?       ?            ? Documents/
>> ...
>> d????????? ?       ?            ? Prog64/
>> ...
>> Depending on timing, Sometimes I will get most or all of
>> them appearing the same as they would on windows.
>>
>> If I can't resolve them more quickly, is there a know
>> or two to increase cache size and item lifetime?
>>     
>
> Which kernel version shows symlinks right and which - wrong?
>   
I only recently switched on getting the access info 'live',
vs. mounting it all as 1 user.  Since it works and doesn't
work based on what's in the cache, the same kernel can provide
both symptoms.

I don't remember the exact date I switched, maybe in the 4.19.xx
series, but at the moment, am running 4.20.3.

Sometimes after a client reboot, I'll see question marks for all
files or about 60 entries in the root dir.  Vs. only the symlinks
where it has to do 2 lookups -- the item and the one pointed to,
and see about 10-15 entries.

I figured that it's a timing prob related to whether or not the entries
are in some cache, so I need to look more closely at that
UID<->RID resolution path, but meanwhile, I thought I'd increase
the cache size and maybe look at the timeout value and if I could
tolerate a longer timeout.

So -- don't think it is the kernel version, but the fact that it's
going a long path to do the lookups -- especially the links --
and that optimizing that path would likely bring the most benefits.

There used to be documentation about doing the resolution through
a separate program, but that documenation disappeared as someone
wrote a .dll to do the resolving, which got plopped in there after
an accidental system update attempt (long story -- power outage,
ups needs new batteries, and more).

Anyway, UID/GID<=>RID map is relatively small (<10-15K) with
long-lived entries.  The server has >100GB mem and the
client, 92GB.

That's why I wanted to increase the UID<->RID cache size &
raise the timeout to a long value.  Maybe ideally with a way
to manually invalidate the cache and/or 1 entry for rare updates.

Thanks!

Re:upcalls seem to have problems with symlinks, junctions et al.

[L A Walsh]

On 2/20/2019 11:48 AM, L A Walsh wrote:
>
>>> ---
>>> ls: cannot access 'Documents': Operation not supported
>>> ...
>>> ls: cannot access 'Prog64': Operation not supported
>>>       
>>> d????????? ?       ?            ? Documents/
>>> ...
>>> d????????? ?       ?            ? Prog64/
>>> ...
>>> Depending on timing, Sometimes I will get most or all of
>>> them appearing the same as they would on windows.
>>>
>>> If I can't resolve them more quickly, is there a know
>>> or two to increase cache size and item lifetime?
>>>     
>>>       
>> Which kernel version shows symlinks right and which - wrong?
>>     
----
    Wrong is 4.20.3... I'm not sure, using the "upcall" for the linux
client reading Metadata(owner+ACL) has ever worked right
for the various forms of symlink/symlinkd/mountd/junction, since
I see a message from me on the cifs list trying 4.10.1 and
having resolution problems related to file-system redirects.

    They "should" work the same way...as when I list them locally
since I'm accessing the machine as the same user (the linux
machine is a samba4 PDC and I'm using the same user-id to
access it on both sides).

    The only time they fully work is when I assigned the mount
ownership to my local user that owns or is in a group that
owns most of the links.  Is there something flakey
about reading the link info -- since it works when I mount
it using 1 UID.

    There may be cache timing issues as well, but they are too
ephemeral to pin down.  The question marks seem to be fairly
consistent on these names (all of which are some form of
NT-pointer to another place - junctions


  

Re: upcalls seem to have problems with symlinks, junctions et al.

[Steve French]

I have wanted to change this code and improve it for a while - one
thing which is tricky is showing mode bits when no permissions to read
permissions though, and we also need to clean up and simplify some of
this code.   Let's follow up on this in a few days, if you are
flexible and can install some test patches

On Thu, Feb 21, 2019 at 5:17 PM L A Walsh <cifs@tlinx.org> wrote:
>
> On 2/20/2019 11:48 AM, L A Walsh wrote:
> >
> >>> ---
> >>> ls: cannot access 'Documents': Operation not supported
> >>> ...
> >>> ls: cannot access 'Prog64': Operation not supported
> >>>
> >>> d????????? ?       ?            ? Documents/
> >>> ...
> >>> d????????? ?       ?            ? Prog64/
> >>> ...
> >>> Depending on timing, Sometimes I will get most or all of
> >>> them appearing the same as they would on windows.
> >>>
> >>> If I can't resolve them more quickly, is there a know
> >>> or two to increase cache size and item lifetime?
> >>>
> >>>
> >> Which kernel version shows symlinks right and which - wrong?
> >>
> ----
>     Wrong is 4.20.3... I'm not sure, using the "upcall" for the linux
> client reading Metadata(owner+ACL) has ever worked right
> for the various forms of symlink/symlinkd/mountd/junction, since
> I see a message from me on the cifs list trying 4.10.1 and
> having resolution problems related to file-system redirects.
>
>     They "should" work the same way...as when I list them locally
> since I'm accessing the machine as the same user (the linux
> machine is a samba4 PDC and I'm using the same user-id to
> access it on both sides).
>
>     The only time they fully work is when I assigned the mount
> ownership to my local user that owns or is in a group that
> owns most of the links.  Is there something flakey
> about reading the link info -- since it works when I mount
> it using 1 UID.
>
>     There may be cache timing issues as well, but they are too
> ephemeral to pin down.  The question marks seem to be fairly
> consistent on these names (all of which are some form of
> NT-pointer to another place - junctions
>
>
>
>


-- 
Thanks,

Steve

Re: upcalls seem to have problems with symlinks, junctions et al.

[L. A. Walsh]

On 2/21/2019 8:30 PM, Steve French wrote:
> I have wanted to change this code and improve it for a while - one
> thing which is tricky is showing mode bits when no permissions to read
> permissions though, and we also need to clean up and simplify some of
> this code.   Let's follow up on this in a few days, if you are
> flexible and can install some test patches
>   
Usually links on linux have 777 permissions and aren't
changeable, though things are changing.

As for testing -- should be able, but maybe not
too fast.

Dunno if you rememeber, but I had some probs w/links back
around Feb 2-3 2017...and you asked for a wireshark
trace that I sent on on the 3rd...then we both got
distracted.
(same list)

I sometimes have a long cycle time...you too?  :-)

I could send the dump again... ;-)  Seriously, I went
back to using a single UID mount and that works around
the problem, as in my case I have perms to read all
of the links.

I just recently tried it again -- and hit the same
type of behavior. 

Re: upcalls seem to have problems with symlinks, junctions et al.

[L A Walsh]

On 2/21/2019 8:30 PM, Steve French wrote:
> I have wanted to change this code and improve it for a while - one
> thing which is tricky is showing mode bits when no permissions to read
> permissions though,
>   
---
    How often would it be the case that you could read the content
of a directory, but not be able to read the permissions? 

    While that can happen on Windows... and not so much on linux
what is the content of the link considered?  I.e. where it is pointing?
Is that something that would usually be readable because it is
pointing to the next item in the path and most people have
the privilege to "Bypass Traverse Checking"?

    I.e. -- is the access to the path information in a symlink controlled
by read permissions to item content, or is it considered part of the
information needed for Traverse Checking?

    I think my issue I had in Feb 2017 had to do with getting
different results depending on the type of the redirect -- junctions,
mountd/vol, and symlink/symlinkd's -- and perhaps that might be
related to how they store store the meta-information?  I suppose
there is similar redirect information at a DFS mount point?

    Part of what I expect, "as a user", to be available to a remote
user, is based on how things look from a user perspective using the
POSIXish-designed-Cygwin.  If I'm signed in with the same
domain-credential (and same SID), remotely, vs. locally, my
expectation is that I should see the same access.  That forms my
expectations with CIFS.

Re: upcalls seem to have problems with symlinks, junctions et al.

[L A Walsh]

On 2/21/2019 8:30 PM, Steve French wrote:
> I have wanted to change this code and improve it for a while - one
> thing which is tricky is showing mode bits when no permissions to read
> permissions though, and we also need to clean up and simplify some of
> this code.   Let's follow up on this in a few days, if you are
> flexible and can install some test patches
>   
FWIW, it seems to be symlinks that are unreadable when perms on
the mount are set to 'multi-user'.
Using the following two examples from my system,

in cmd.exe they look like:
04/17/2017  08:45 AM    <SYMLINKD>     Share [S:\]
08/22/2018  03:08 PM    <JUNCTION>     Symbols [\\Bliss\Share\Symbols\]*
*

in Cygwin, they look like: (domain+user+groups are shortened to fit)
       (domain name, B (Bliss), username 'l' and group 'lg')
lrwxrwxrwx   1 B\l  B\lg  2 Apr 17  2017 Share -> /s
lrwxrwxrwx   1 B\l  B\lg 22 Aug 22  2018 Symbols -> //Bliss/Share/Symbols/

So both are links owned by me, one pointing to a locally mounted drive
and the other a remotely mounted Share on the Domain server.

On linux with single user mounting, I see:

l--------- 1 law Administrators            0 Apr 17  2017 Share -> /??/S://
drwxr-xr-x 2 law Administrators            0 Dec 17 19:14 Symbols/

Amusingly, I can read directory Symbols which results in a read
from the same server I'm doing the 'ls' from -- sorta the long way
around.

Oddly, I can't read the permissions on the symlink even though they
are usable (created directory named '??' and pointers and directories
underneath it as appropriate).

If I mount the cifs share "multiuser", I lose the capacity to see or
follow the symlinks as well as getting "Operation not supported" errors,
but the junctions still work fine:

ls: cannot access 'Share': Operation not supported
d????????? ? ?    ?                    ?            ? Share/
drwxrwxr-x 2 root Domain Admins        0 Dec 17 19:14 Symbols/

1) the Operation not supported message is dubious, since
I can read the link in single-user mount, even though the link
shows no access.

2) If the permissions on the links really are unreadable, the
'l-----' format is certainly easier on the eyes than
'd?????'.


I'd like to see the multi-user mount give at least as much information
as the single-user mount.  That would make sense, wouldn't it?
Can your patch/cleanup at least solve at problem?










**