[PATCH] Use a separate superblock if mount requires a different security flavor

[PATCH] Use a separate superblock if mount requires a different security flavor

[Frank Filz]

If a server has two exports from the same filesystem but with different
security flavors allowed, when the client mounts first one and then the
second, the same super block was being used. This resulted in the
security flavor for the first export being applied to access to the
second export.

The fix is simply to check the security flavor of the nfs_server
temporarily constructed for the second mount within nfs_compare_super.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
---
 fs/nfs/super.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/fs/nfs/super.c b/fs/nfs/super.c
index 084af10..44d60f1 100644
--- a/fs/nfs/super.c
+++ b/fs/nfs/super.c
@@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct super_block
*sb, void *data)
        struct nfs_server *server = sb_mntdata->server, *old =
NFS_SB(sb);
        int mntflags = sb_mntdata->mntflags;
 
+       if(old->client->cl_auth->au_flavor
+          != server->client->cl_auth->au_flavor)
+               return 0;
        if (!nfs_compare_super_address(old, server))
                return 0;
        /* Note: NFS_MOUNT_UNSHARED == NFS4_MOUNT_UNSHARED */
-- 
1.9.3

Re: [PATCH] Use a separate superblock if mount requires a different security flavor

[Trond Myklebust]

On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com> wrote:
> If a server has two exports from the same filesystem but with different
> security flavors allowed, when the client mounts first one and then the
> second, the same super block was being used. This resulted in the
> security flavor for the first export being applied to access to the
> second export.
>
> The fix is simply to check the security flavor of the nfs_server
> temporarily constructed for the second mount within nfs_compare_super.
>
> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
> ---
>  fs/nfs/super.c | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/fs/nfs/super.c b/fs/nfs/super.c
> index 084af10..44d60f1 100644
> --- a/fs/nfs/super.c
> +++ b/fs/nfs/super.c
> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct super_block
> *sb, void *data)
>         struct nfs_server *server = sb_mntdata->server, *old =
> NFS_SB(sb);
>         int mntflags = sb_mntdata->mntflags;
>
> +       if(old->client->cl_auth->au_flavor
> +          != server->client->cl_auth->au_flavor)
> +               return 0;

Isn't this check already being performed in
nfs_compare_mount_options()? As far as I can see, the difference is
that you are checking unconditionally, whereas
nfs_compare_mount_options only does so if there was a 'sec=' line
specified in the mount options.

Cheers
  Trond

Re: [PATCH] Use a separate superblock if mount requires a different security flavor

[Chuck Lever]

On Sep 16, 2015, at 4:52 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:

> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com> wrote:
>> If a server has two exports from the same filesystem but with different
>> security flavors allowed, when the client mounts first one and then the
>> second, the same super block was being used. This resulted in the
>> security flavor for the first export being applied to access to the
>> second export.
>> 
>> The fix is simply to check the security flavor of the nfs_server
>> temporarily constructed for the second mount within nfs_compare_super.
>> 
>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
>> ---
>> fs/nfs/super.c | 3 +++
>> 1 file changed, 3 insertions(+)
>> 
>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c
>> index 084af10..44d60f1 100644
>> --- a/fs/nfs/super.c
>> +++ b/fs/nfs/super.c
>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct super_block
>> *sb, void *data)
>>        struct nfs_server *server = sb_mntdata->server, *old =
>> NFS_SB(sb);
>>        int mntflags = sb_mntdata->mntflags;
>> 
>> +       if(old->client->cl_auth->au_flavor
>> +          != server->client->cl_auth->au_flavor)
>> +               return 0;
> 
> Isn't this check already being performed in
> nfs_compare_mount_options()? As far as I can see, the difference is
> that you are checking unconditionally, whereas
> nfs_compare_mount_options only does so if there was a 'sec=' line
> specified in the mount options.

Right. If the user doesn't provide a sec=, the security flavor
is autonegotiated. In the case Frank describes, there are two
directories shared on the server, each from the same FSID but
using distinct security policies.

So the mount options comparison is inadequate if no sec= is
specified on the mount command line.

--
Chuck Lever

Re: [PATCH] Use a separate superblock if mount requires a different security flavor

[Trond Myklebust]

On Wed, Sep 16, 2015 at 4:55 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
>
> On Sep 16, 2015, at 4:52 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:
>
>> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com> wrote:
>>> If a server has two exports from the same filesystem but with different
>>> security flavors allowed, when the client mounts first one and then the
>>> second, the same super block was being used. This resulted in the
>>> security flavor for the first export being applied to access to the
>>> second export.
>>>
>>> The fix is simply to check the security flavor of the nfs_server
>>> temporarily constructed for the second mount within nfs_compare_super.
>>>
>>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
>>> ---
>>> fs/nfs/super.c | 3 +++
>>> 1 file changed, 3 insertions(+)
>>>
>>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c
>>> index 084af10..44d60f1 100644
>>> --- a/fs/nfs/super.c
>>> +++ b/fs/nfs/super.c
>>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct super_block
>>> *sb, void *data)
>>>        struct nfs_server *server = sb_mntdata->server, *old =
>>> NFS_SB(sb);
>>>        int mntflags = sb_mntdata->mntflags;
>>>
>>> +       if(old->client->cl_auth->au_flavor
>>> +          != server->client->cl_auth->au_flavor)
>>> +               return 0;
>>
>> Isn't this check already being performed in
>> nfs_compare_mount_options()? As far as I can see, the difference is
>> that you are checking unconditionally, whereas
>> nfs_compare_mount_options only does so if there was a 'sec=' line
>> specified in the mount options.
>
> Right. If the user doesn't provide a sec=, the security flavor
> is autonegotiated. In the case Frank describes, there are two
> directories shared on the server, each from the same FSID but
> using distinct security policies.
>
> So the mount options comparison is inadequate if no sec= is
> specified on the mount command line.

We don't claim to support autonegotiation of multiple security
policies per filesystem, in general. We only claim to support one auth
flavour per super block.

If I understand you correctly, you are knowingly trying to work around
that limitation by performing multiple mounts of the same filesystem
and force it to use multiple super blocks. Why can't you then also
specify the 'sec='?

Cheers
  Trond

RE: [PATCH] Use a separate superblock if mount requires a different security flavor

[Frank Filz]

> On Wed, Sep 16, 2015 at 4:55 PM, Chuck Lever <chuck.lever@oracle.com>
> wrote:
> >
> > On Sep 16, 2015, at 4:52 PM, Trond Myklebust
> <trond.myklebust@primarydata.com> wrote:
> >
> >> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com>
> wrote:
> >>> If a server has two exports from the same filesystem but with
> >>> different security flavors allowed, when the client mounts first one
> >>> and then the second, the same super block was being used. This
> >>> resulted in the security flavor for the first export being applied
> >>> to access to the second export.
> >>>
> >>> The fix is simply to check the security flavor of the nfs_server
> >>> temporarily constructed for the second mount within
> nfs_compare_super.
> >>>
> >>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
> >>> ---
> >>> fs/nfs/super.c | 3 +++
> >>> 1 file changed, 3 insertions(+)
> >>>
> >>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c index 084af10..44d60f1
> >>> 100644
> >>> --- a/fs/nfs/super.c
> >>> +++ b/fs/nfs/super.c
> >>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct
> >>> super_block *sb, void *data)
> >>>        struct nfs_server *server = sb_mntdata->server, *old =
> >>> NFS_SB(sb);
> >>>        int mntflags = sb_mntdata->mntflags;
> >>>
> >>> +       if(old->client->cl_auth->au_flavor
> >>> +          != server->client->cl_auth->au_flavor)
> >>> +               return 0;
> >>
> >> Isn't this check already being performed in
> >> nfs_compare_mount_options()? As far as I can see, the difference is
> >> that you are checking unconditionally, whereas
> >> nfs_compare_mount_options only does so if there was a 'sec=' line
> >> specified in the mount options.
> >
> > Right. If the user doesn't provide a sec=, the security flavor is
> > autonegotiated. In the case Frank describes, there are two directories
> > shared on the server, each from the same FSID but using distinct
> > security policies.
> >
> > So the mount options comparison is inadequate if no sec= is specified
> > on the mount command line.
> 
> We don't claim to support autonegotiation of multiple security policies per
> filesystem, in general. We only claim to support one auth flavour per super
> block.
> 
> If I understand you correctly, you are knowingly trying to work around that
> limitation by performing multiple mounts of the same filesystem and force it
> to use multiple super blocks. Why can't you then also specify the 'sec='?

I see that point, but why not just make this case work smoothly rather than force the user to go back and specify -o sec on the mount?

Actually all that is necessary is SOME difference in mount options (or use -o nosharedcache, which could be used on all the mounts so all can have the same mount options...) and allow security negotiation to work.

An interesting question is if there are any servers out there that don't typically provide different FSID for different portions of the namespace, but also provide a mechanism to specify different security policies for different portions of the namespace?

Frank


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: [PATCH] Use a separate superblock if mount requires a different security flavor

[Trond Myklebust]

On Wed, Sep 16, 2015 at 5:36 PM, Frank Filz <ffilzlnx@mindspring.com> wrote:
>> On Wed, Sep 16, 2015 at 4:55 PM, Chuck Lever <chuck.lever@oracle.com>
>> wrote:
>> >
>> > On Sep 16, 2015, at 4:52 PM, Trond Myklebust
>> <trond.myklebust@primarydata.com> wrote:
>> >
>> >> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com>
>> wrote:
>> >>> If a server has two exports from the same filesystem but with
>> >>> different security flavors allowed, when the client mounts first one
>> >>> and then the second, the same super block was being used. This
>> >>> resulted in the security flavor for the first export being applied
>> >>> to access to the second export.
>> >>>
>> >>> The fix is simply to check the security flavor of the nfs_server
>> >>> temporarily constructed for the second mount within
>> nfs_compare_super.
>> >>>
>> >>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
>> >>> ---
>> >>> fs/nfs/super.c | 3 +++
>> >>> 1 file changed, 3 insertions(+)
>> >>>
>> >>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c index 084af10..44d60f1
>> >>> 100644
>> >>> --- a/fs/nfs/super.c
>> >>> +++ b/fs/nfs/super.c
>> >>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct
>> >>> super_block *sb, void *data)
>> >>>        struct nfs_server *server = sb_mntdata->server, *old =
>> >>> NFS_SB(sb);
>> >>>        int mntflags = sb_mntdata->mntflags;
>> >>>
>> >>> +       if(old->client->cl_auth->au_flavor
>> >>> +          != server->client->cl_auth->au_flavor)
>> >>> +               return 0;
>> >>
>> >> Isn't this check already being performed in
>> >> nfs_compare_mount_options()? As far as I can see, the difference is
>> >> that you are checking unconditionally, whereas
>> >> nfs_compare_mount_options only does so if there was a 'sec=' line
>> >> specified in the mount options.
>> >
>> > Right. If the user doesn't provide a sec=, the security flavor is
>> > autonegotiated. In the case Frank describes, there are two directories
>> > shared on the server, each from the same FSID but using distinct
>> > security policies.
>> >
>> > So the mount options comparison is inadequate if no sec= is specified
>> > on the mount command line.
>>
>> We don't claim to support autonegotiation of multiple security policies per
>> filesystem, in general. We only claim to support one auth flavour per super
>> block.
>>
>> If I understand you correctly, you are knowingly trying to work around that
>> limitation by performing multiple mounts of the same filesystem and force it
>> to use multiple super blocks. Why can't you then also specify the 'sec='?
>
> I see that point, but why not just make this case work smoothly rather than force the user to go back and specify -o sec on the mount?

The main issue is that it violates the policy that we must try our
best not to set up situations where the client has cache consistency
issues due to the existence of multiple superblocks that all have page
caches for the same file.

> Actually all that is necessary is SOME difference in mount options (or use -o nosharedcache, which could be used on all the mounts so all can have the same mount options...) and allow security negotiation to work.

I'd expect there to be no problems if the admin specifies -o
nosharedcache. Please do let me know if that fails to work.

> An interesting question is if there are any servers out there that don't typically provide different FSID for different portions of the namespace, but also provide a mechanism to specify different security policies for different portions of the namespace?

That sort of situation is difficult to manage. How do you deal with
hard links that cross the security policy boundary? I haven't seen any
discussion of how that sort of case is expected to work, and of what
requirements it imposes on clients and servers.

Cheers
  Trond

Re: [PATCH] Use a separate superblock if mount requires a different security flavor

[Chuck Lever]

On Sep 16, 2015, at 11:32 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:

> On Wed, Sep 16, 2015 at 5:36 PM, Frank Filz <ffilzlnx@mindspring.com> wrote:
>>> On Wed, Sep 16, 2015 at 4:55 PM, Chuck Lever <chuck.lever@oracle.com>
>>> wrote:
>>>> 
>>>> On Sep 16, 2015, at 4:52 PM, Trond Myklebust
>>> <trond.myklebust@primarydata.com> wrote:
>>>> 
>>>>> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com>
>>> wrote:
>>>>>> If a server has two exports from the same filesystem but with
>>>>>> different security flavors allowed, when the client mounts first one
>>>>>> and then the second, the same super block was being used. This
>>>>>> resulted in the security flavor for the first export being applied
>>>>>> to access to the second export.
>>>>>> 
>>>>>> The fix is simply to check the security flavor of the nfs_server
>>>>>> temporarily constructed for the second mount within
>>> nfs_compare_super.
>>>>>> 
>>>>>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
>>>>>> ---
>>>>>> fs/nfs/super.c | 3 +++
>>>>>> 1 file changed, 3 insertions(+)
>>>>>> 
>>>>>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c index 084af10..44d60f1
>>>>>> 100644
>>>>>> --- a/fs/nfs/super.c
>>>>>> +++ b/fs/nfs/super.c
>>>>>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct
>>>>>> super_block *sb, void *data)
>>>>>>       struct nfs_server *server = sb_mntdata->server, *old =
>>>>>> NFS_SB(sb);
>>>>>>       int mntflags = sb_mntdata->mntflags;
>>>>>> 
>>>>>> +       if(old->client->cl_auth->au_flavor
>>>>>> +          != server->client->cl_auth->au_flavor)
>>>>>> +               return 0;
>>>>> 
>>>>> Isn't this check already being performed in
>>>>> nfs_compare_mount_options()? As far as I can see, the difference is
>>>>> that you are checking unconditionally, whereas
>>>>> nfs_compare_mount_options only does so if there was a 'sec=' line
>>>>> specified in the mount options.
>>>> 
>>>> Right. If the user doesn't provide a sec=, the security flavor is
>>>> autonegotiated. In the case Frank describes, there are two directories
>>>> shared on the server, each from the same FSID but using distinct
>>>> security policies.
>>>> 
>>>> So the mount options comparison is inadequate if no sec= is specified
>>>> on the mount command line.
>>> 
>>> We don't claim to support autonegotiation of multiple security policies per
>>> filesystem, in general. We only claim to support one auth flavour per super
>>> block.
>>> 
>>> If I understand you correctly, you are knowingly trying to work around that
>>> limitation by performing multiple mounts of the same filesystem and force it
>>> to use multiple super blocks. Why can't you then also specify the 'sec='?
>> 
>> I see that point, but why not just make this case work smoothly rather than force the user to go back and specify -o sec on the mount?
> 
> The main issue is that it violates the policy that we must try our
> best not to set up situations where the client has cache consistency
> issues due to the existence of multiple superblocks that all have page
> caches for the same file.

The parts of the physical FS's namespace that are accessible
by each security flavor are disjoint. Aside from hardlinks, is
there any possibility for cache aliasing in this scenario?


>> Actually all that is necessary is SOME difference in mount options (or use -o nosharedcache, which could be used on all the mounts so all can have the same mount options...) and allow security negotiation to work.
> 
> I'd expect there to be no problems if the admin specifies -o
> nosharedcache. Please do let me know if that fails to work.
> 
>> An interesting question is if there are any servers out there that don't typically provide different FSID for different portions of the namespace, but also provide a mechanism to specify different security policies for different portions of the namespace?
> 
> That sort of situation is difficult to manage.

But appears to be allowed by Solaris, and likely also by Linux
and Ganesha. I think we are going to have to consider it,
particularly if there is no prohibition against it in the RFCs.

Meanwhile, I need to document this behavior (internally) and
thus need to understand if this is a server bug, a client bug,
or a protocol bug. Thoughts about which?


> How do you deal with
> hard links that cross the security policy boundary? I haven't seen any
> discussion of how that sort of case is expected to work, and of what
> requirements it imposes on clients and servers.
> 
> Cheers
>  Trond

--
Chuck Lever

Re: [PATCH] Use a separate superblock if mount requires a different security flavor

[Chuck Lever]

> On Sep 17, 2015, at 11:01 AM, Chuck Lever <chuck.lever@oracle.com> wrote:
> 
> 
> On Sep 16, 2015, at 11:32 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> 
>> On Wed, Sep 16, 2015 at 5:36 PM, Frank Filz <ffilzlnx@mindspring.com> wrote:
>>>> On Wed, Sep 16, 2015 at 4:55 PM, Chuck Lever <chuck.lever@oracle.com>
>>>> wrote:
>>>>> 
>>>>> On Sep 16, 2015, at 4:52 PM, Trond Myklebust
>>>> <trond.myklebust@primarydata.com> wrote:
>>>>> 
>>>>>> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com>
>>>> wrote:
>>>>>>> If a server has two exports from the same filesystem but with
>>>>>>> different security flavors allowed, when the client mounts first one
>>>>>>> and then the second, the same super block was being used. This
>>>>>>> resulted in the security flavor for the first export being applied
>>>>>>> to access to the second export.
>>>>>>> 
>>>>>>> The fix is simply to check the security flavor of the nfs_server
>>>>>>> temporarily constructed for the second mount within
>>>> nfs_compare_super.
>>>>>>> 
>>>>>>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
>>>>>>> ---
>>>>>>> fs/nfs/super.c | 3 +++
>>>>>>> 1 file changed, 3 insertions(+)
>>>>>>> 
>>>>>>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c index 084af10..44d60f1
>>>>>>> 100644
>>>>>>> --- a/fs/nfs/super.c
>>>>>>> +++ b/fs/nfs/super.c
>>>>>>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct
>>>>>>> super_block *sb, void *data)
>>>>>>>      struct nfs_server *server = sb_mntdata->server, *old =
>>>>>>> NFS_SB(sb);
>>>>>>>      int mntflags = sb_mntdata->mntflags;
>>>>>>> 
>>>>>>> +       if(old->client->cl_auth->au_flavor
>>>>>>> +          != server->client->cl_auth->au_flavor)
>>>>>>> +               return 0;
>>>>>> 
>>>>>> Isn't this check already being performed in
>>>>>> nfs_compare_mount_options()? As far as I can see, the difference is
>>>>>> that you are checking unconditionally, whereas
>>>>>> nfs_compare_mount_options only does so if there was a 'sec=' line
>>>>>> specified in the mount options.
>>>>> 
>>>>> Right. If the user doesn't provide a sec=, the security flavor is
>>>>> autonegotiated. In the case Frank describes, there are two directories
>>>>> shared on the server, each from the same FSID but using distinct
>>>>> security policies.
>>>>> 
>>>>> So the mount options comparison is inadequate if no sec= is specified
>>>>> on the mount command line.
>>>> 
>>>> We don't claim to support autonegotiation of multiple security policies per
>>>> filesystem, in general. We only claim to support one auth flavour per super
>>>> block.
>>>> 
>>>> If I understand you correctly, you are knowingly trying to work around that
>>>> limitation by performing multiple mounts of the same filesystem and force it
>>>> to use multiple super blocks. Why can't you then also specify the 'sec='?
>>> 
>>> I see that point, but why not just make this case work smoothly rather than force the user to go back and specify -o sec on the mount?
>> 
>> The main issue is that it violates the policy that we must try our
>> best not to set up situations where the client has cache consistency
>> issues due to the existence of multiple superblocks that all have page
>> caches for the same file.
> 
> The parts of the physical FS's namespace that are accessible
> by each security flavor are disjoint. Aside from hardlinks, is
> there any possibility for cache aliasing in this scenario?
> 
> 
>>> Actually all that is necessary is SOME difference in mount options (or use -o nosharedcache, which could be used on all the mounts so all can have the same mount options...) and allow security negotiation to work.
>> 
>> I'd expect there to be no problems if the admin specifies -o
>> nosharedcache. Please do let me know if that fails to work.
>> 
>>> An interesting question is if there are any servers out there that don't typically provide different FSID for different portions of the namespace, but also provide a mechanism to specify different security policies for different portions of the namespace?
>> 
>> That sort of situation is difficult to manage.
> 
> But appears to be allowed by Solaris, and likely also by Linux
> and Ganesha. I think we are going to have to consider it,
> particularly if there is no prohibition against it in the RFCs.

It is certainly possible to document best practices or
add admin UI limits to prevent servers from being
configured this way.

Meanwhile, the Linux client does allow mounting such
exports when both mounts specify unique “sec=“. If this
is a dangerous or unsupported scenario, perhaps this
should be disallowed somehow.

When security is negotiated, the second mount is not
allowed. It could display an informative error message
when if fails.


> Meanwhile, I need to document this behavior (internally) and
> thus need to understand if this is a server bug, a client bug,
> or a protocol bug. Thoughts about which?
> 
> 
>> How do you deal with
>> hard links that cross the security policy boundary? I haven't seen any
>> discussion of how that sort of case is expected to work, and of what
>> requirements it imposes on clients and servers.
>> 
>> Cheers
>> Trond
> 
> --
> Chuck Lever
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

—
Chuck Lever

Re: [PATCH] Use a separate superblock if mount requires a different security flavor

[Trond Myklebust]

On Mon, Sep 21, 2015 at 7:10 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
>
>> On Sep 17, 2015, at 11:01 AM, Chuck Lever <chuck.lever@oracle.com> wrote:
>>
>>
>> On Sep 16, 2015, at 11:32 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:
>>
>>> On Wed, Sep 16, 2015 at 5:36 PM, Frank Filz <ffilzlnx@mindspring.com> wrote:
>>>>> On Wed, Sep 16, 2015 at 4:55 PM, Chuck Lever <chuck.lever@oracle.com>
>>>>> wrote:
>>>>>>
>>>>>> On Sep 16, 2015, at 4:52 PM, Trond Myklebust
>>>>> <trond.myklebust@primarydata.com> wrote:
>>>>>>
>>>>>>> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com>
>>>>> wrote:
>>>>>>>> If a server has two exports from the same filesystem but with
>>>>>>>> different security flavors allowed, when the client mounts first one
>>>>>>>> and then the second, the same super block was being used. This
>>>>>>>> resulted in the security flavor for the first export being applied
>>>>>>>> to access to the second export.
>>>>>>>>
>>>>>>>> The fix is simply to check the security flavor of the nfs_server
>>>>>>>> temporarily constructed for the second mount within
>>>>> nfs_compare_super.
>>>>>>>>
>>>>>>>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
>>>>>>>> ---
>>>>>>>> fs/nfs/super.c | 3 +++
>>>>>>>> 1 file changed, 3 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c index 084af10..44d60f1
>>>>>>>> 100644
>>>>>>>> --- a/fs/nfs/super.c
>>>>>>>> +++ b/fs/nfs/super.c
>>>>>>>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct
>>>>>>>> super_block *sb, void *data)
>>>>>>>>      struct nfs_server *server = sb_mntdata->server, *old =
>>>>>>>> NFS_SB(sb);
>>>>>>>>      int mntflags = sb_mntdata->mntflags;
>>>>>>>>
>>>>>>>> +       if(old->client->cl_auth->au_flavor
>>>>>>>> +          != server->client->cl_auth->au_flavor)
>>>>>>>> +               return 0;
>>>>>>>
>>>>>>> Isn't this check already being performed in
>>>>>>> nfs_compare_mount_options()? As far as I can see, the difference is
>>>>>>> that you are checking unconditionally, whereas
>>>>>>> nfs_compare_mount_options only does so if there was a 'sec=' line
>>>>>>> specified in the mount options.
>>>>>>
>>>>>> Right. If the user doesn't provide a sec=, the security flavor is
>>>>>> autonegotiated. In the case Frank describes, there are two directories
>>>>>> shared on the server, each from the same FSID but using distinct
>>>>>> security policies.
>>>>>>
>>>>>> So the mount options comparison is inadequate if no sec= is specified
>>>>>> on the mount command line.
>>>>>
>>>>> We don't claim to support autonegotiation of multiple security policies per
>>>>> filesystem, in general. We only claim to support one auth flavour per super
>>>>> block.
>>>>>
>>>>> If I understand you correctly, you are knowingly trying to work around that
>>>>> limitation by performing multiple mounts of the same filesystem and force it
>>>>> to use multiple super blocks. Why can't you then also specify the 'sec='?
>>>>
>>>> I see that point, but why not just make this case work smoothly rather than force the user to go back and specify -o sec on the mount?
>>>
>>> The main issue is that it violates the policy that we must try our
>>> best not to set up situations where the client has cache consistency
>>> issues due to the existence of multiple superblocks that all have page
>>> caches for the same file.
>>
>> The parts of the physical FS's namespace that are accessible
>> by each security flavor are disjoint. Aside from hardlinks, is
>> there any possibility for cache aliasing in this scenario?
>>
>>
>>>> Actually all that is necessary is SOME difference in mount options (or use -o nosharedcache, which could be used on all the mounts so all can have the same mount options...) and allow security negotiation to work.
>>>
>>> I'd expect there to be no problems if the admin specifies -o
>>> nosharedcache. Please do let me know if that fails to work.
>>>
>>>> An interesting question is if there are any servers out there that don't typically provide different FSID for different portions of the namespace, but also provide a mechanism to specify different security policies for different portions of the namespace?
>>>
>>> That sort of situation is difficult to manage.
>>
>> But appears to be allowed by Solaris, and likely also by Linux
>> and Ganesha. I think we are going to have to consider it,
>> particularly if there is no prohibition against it in the RFCs.
>
> It is certainly possible to document best practices or
> add admin UI limits to prevent servers from being
> configured this way.
>
> Meanwhile, the Linux client does allow mounting such
> exports when both mounts specify unique “sec=“. If this
> is a dangerous or unsupported scenario, perhaps this
> should be disallowed somehow.
>
> When security is negotiated, the second mount is not
> allowed. It could display an informative error message
> when if fails.

My main worry with the patch you proposed is that we're only
considering a small part of the problem here, namely what happens on
mount.
What if the user were to mount '/' instead of the particular path you
chose, and then simply walk down to the problematic directory? As far
as I can see, that will fail just as badly both with and without this
patch. Why would users expect that behaviour to be any different to
the new mount behaviour?

IOW: what is the full set of expectations would we be setting by
applying the patch, and what is the rationale for changing some
behaviour, but not all?

Also, while I know that the Linux client has never supported this
behaviour. What do the Solaris and FreeBSD clients do, and what are
their limitations?

Cheers
  Trond

Re: [PATCH] Use a separate superblock if mount requires a different security flavor

[Chuck Lever]

> On Sep 21, 2015, at 5:43 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> 
> On Mon, Sep 21, 2015 at 7:10 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
>> 
>>> On Sep 17, 2015, at 11:01 AM, Chuck Lever <chuck.lever@oracle.com> wrote:
>>> 
>>> 
>>> On Sep 16, 2015, at 11:32 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:
>>> 
>>>> On Wed, Sep 16, 2015 at 5:36 PM, Frank Filz <ffilzlnx@mindspring.com> wrote:
>>>>>> On Wed, Sep 16, 2015 at 4:55 PM, Chuck Lever <chuck.lever@oracle.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> On Sep 16, 2015, at 4:52 PM, Trond Myklebust
>>>>>> <trond.myklebust@primarydata.com> wrote:
>>>>>>> 
>>>>>>>> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com>
>>>>>> wrote:
>>>>>>>>> If a server has two exports from the same filesystem but with
>>>>>>>>> different security flavors allowed, when the client mounts first one
>>>>>>>>> and then the second, the same super block was being used. This
>>>>>>>>> resulted in the security flavor for the first export being applied
>>>>>>>>> to access to the second export.
>>>>>>>>> 
>>>>>>>>> The fix is simply to check the security flavor of the nfs_server
>>>>>>>>> temporarily constructed for the second mount within
>>>>>> nfs_compare_super.
>>>>>>>>> 
>>>>>>>>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
>>>>>>>>> ---
>>>>>>>>> fs/nfs/super.c | 3 +++
>>>>>>>>> 1 file changed, 3 insertions(+)
>>>>>>>>> 
>>>>>>>>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c index 084af10..44d60f1
>>>>>>>>> 100644
>>>>>>>>> --- a/fs/nfs/super.c
>>>>>>>>> +++ b/fs/nfs/super.c
>>>>>>>>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct
>>>>>>>>> super_block *sb, void *data)
>>>>>>>>>     struct nfs_server *server = sb_mntdata->server, *old =
>>>>>>>>> NFS_SB(sb);
>>>>>>>>>     int mntflags = sb_mntdata->mntflags;
>>>>>>>>> 
>>>>>>>>> +       if(old->client->cl_auth->au_flavor
>>>>>>>>> +          != server->client->cl_auth->au_flavor)
>>>>>>>>> +               return 0;
>>>>>>>> 
>>>>>>>> Isn't this check already being performed in
>>>>>>>> nfs_compare_mount_options()? As far as I can see, the difference is
>>>>>>>> that you are checking unconditionally, whereas
>>>>>>>> nfs_compare_mount_options only does so if there was a 'sec=' line
>>>>>>>> specified in the mount options.
>>>>>>> 
>>>>>>> Right. If the user doesn't provide a sec=, the security flavor is
>>>>>>> autonegotiated. In the case Frank describes, there are two directories
>>>>>>> shared on the server, each from the same FSID but using distinct
>>>>>>> security policies.
>>>>>>> 
>>>>>>> So the mount options comparison is inadequate if no sec= is specified
>>>>>>> on the mount command line.
>>>>>> 
>>>>>> We don't claim to support autonegotiation of multiple security policies per
>>>>>> filesystem, in general. We only claim to support one auth flavour per super
>>>>>> block.
>>>>>> 
>>>>>> If I understand you correctly, you are knowingly trying to work around that
>>>>>> limitation by performing multiple mounts of the same filesystem and force it
>>>>>> to use multiple super blocks. Why can't you then also specify the 'sec='?
>>>>> 
>>>>> I see that point, but why not just make this case work smoothly rather than force the user to go back and specify -o sec on the mount?
>>>> 
>>>> The main issue is that it violates the policy that we must try our
>>>> best not to set up situations where the client has cache consistency
>>>> issues due to the existence of multiple superblocks that all have page
>>>> caches for the same file.
>>> 
>>> The parts of the physical FS's namespace that are accessible
>>> by each security flavor are disjoint. Aside from hardlinks, is
>>> there any possibility for cache aliasing in this scenario?
>>> 
>>> 
>>>>> Actually all that is necessary is SOME difference in mount options (or use -o nosharedcache, which could be used on all the mounts so all can have the same mount options...) and allow security negotiation to work.
>>>> 
>>>> I'd expect there to be no problems if the admin specifies -o
>>>> nosharedcache. Please do let me know if that fails to work.
>>>> 
>>>>> An interesting question is if there are any servers out there that don't typically provide different FSID for different portions of the namespace, but also provide a mechanism to specify different security policies for different portions of the namespace?
>>>> 
>>>> That sort of situation is difficult to manage.
>>> 
>>> But appears to be allowed by Solaris, and likely also by Linux
>>> and Ganesha. I think we are going to have to consider it,
>>> particularly if there is no prohibition against it in the RFCs.
>> 
>> It is certainly possible to document best practices or
>> add admin UI limits to prevent servers from being
>> configured this way.
>> 
>> Meanwhile, the Linux client does allow mounting such
>> exports when both mounts specify unique “sec=“. If this
>> is a dangerous or unsupported scenario, perhaps this
>> should be disallowed somehow.
>> 
>> When security is negotiated, the second mount is not
>> allowed. It could display an informative error message
>> when if fails.
> 
> My main worry with the patch you proposed is that we're only
> considering a small part of the problem here, namely what happens on
> mount.
> What if the user were to mount '/' instead of the particular path you
> chose, and then simply walk down to the problematic directory? As far
> as I can see, that will fail just as badly both with and without this
> patch. Why would users expect that behaviour to be any different to
> the new mount behaviour?

<shrug> Maybe that’s a little different, though I don’t
have direct experience of how this is supposed to work.

If the client mounts “/“ with an explicit security flavor
that does not work with all the server’s shares, then
that’s clearly an admin choice. The root directories of
inaccessible shares will simply not allow users to cd
into them.

When no “sec=“ is specified when mounting “/“ then is the
client allowed to renegotiate when crossing into real
shares, or are the real shares accessed only via the same
flavor that was negotiated for “/“ ? If it’s the former,
do we know that works well for any set of possible
security policies specified on the server’s shares? If
it’s the latter, is it a surprise when some shares are
not accessible? Dunno.


> IOW: what is the full set of expectations would we be setting by
> applying the patch, and what is the rationale for changing some
> behaviour, but not all?

Our testing turned up this inconsistency. I don’t think
we have a particular favorite outcome here, other than
safe, consistent behavior.

Since specifying different “sec=“ works now, Frank and I
thought it was indeed a supported configuration.

If it’s not, then let’s fix the Linux client to behave
consistently and/or fail gracefully. The client (or an
appropriate man page) should help admins understand
what might be going wrong.


> Also, while I know that the Linux client has never supported this
> behaviour. What do the Solaris and FreeBSD clients do, and what are
> their limitations?

I’m told the Solaris client has a similar design as
Linux, in that it attempts to match one superblock to
each FSID on the server.

It tries to behave in a consistent manner by using the
WRONGSEC errors to flip back and forth between security
flavors as necessary. The Solaris folks recognize this
is not a good permanent solution, and have similar
concerns about whether this is a reasonable scenario
to try to support with more complex logic on the client.

I suggested some server changes before. It would be
useful to know whether sharing more than one directory
on a server’s physical filesystem using distinct
security policies is a common usage scenario. If it
isn’t, then preventing such multi-security mounts on
the client doesn’t seem onerous.


—
Chuck Lever

Re: [PATCH] Use a separate superblock if mount requires a different security flavor

[Tom Haynes]

On Mon, Sep 21, 2015 at 06:31:48PM -0700, Chuck Lever wrote:
> 
> > On Sep 21, 2015, at 5:43 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> > 
> > On Mon, Sep 21, 2015 at 7:10 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
> >> 
> >>> On Sep 17, 2015, at 11:01 AM, Chuck Lever <chuck.lever@oracle.com> wrote:
> >>> 
> >>> 
> >>> On Sep 16, 2015, at 11:32 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> >>> 
> >>>> On Wed, Sep 16, 2015 at 5:36 PM, Frank Filz <ffilzlnx@mindspring.com> wrote:
> >>>>>> On Wed, Sep 16, 2015 at 4:55 PM, Chuck Lever <chuck.lever@oracle.com>
> >>>>>> wrote:
> >>>>>>> 
> >>>>>>> On Sep 16, 2015, at 4:52 PM, Trond Myklebust
> >>>>>> <trond.myklebust@primarydata.com> wrote:
> >>>>>>> 
> >>>>>>>> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com>
> >>>>>> wrote:
> >>>>>>>>> If a server has two exports from the same filesystem but with
> >>>>>>>>> different security flavors allowed, when the client mounts first one
> >>>>>>>>> and then the second, the same super block was being used. This
> >>>>>>>>> resulted in the security flavor for the first export being applied
> >>>>>>>>> to access to the second export.
> >>>>>>>>> 
> >>>>>>>>> The fix is simply to check the security flavor of the nfs_server
> >>>>>>>>> temporarily constructed for the second mount within
> >>>>>> nfs_compare_super.
> >>>>>>>>> 
> >>>>>>>>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
> >>>>>>>>> ---
> >>>>>>>>> fs/nfs/super.c | 3 +++
> >>>>>>>>> 1 file changed, 3 insertions(+)
> >>>>>>>>> 
> >>>>>>>>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c index 084af10..44d60f1
> >>>>>>>>> 100644
> >>>>>>>>> --- a/fs/nfs/super.c
> >>>>>>>>> +++ b/fs/nfs/super.c
> >>>>>>>>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct
> >>>>>>>>> super_block *sb, void *data)
> >>>>>>>>>     struct nfs_server *server = sb_mntdata->server, *old =
> >>>>>>>>> NFS_SB(sb);
> >>>>>>>>>     int mntflags = sb_mntdata->mntflags;
> >>>>>>>>> 
> >>>>>>>>> +       if(old->client->cl_auth->au_flavor
> >>>>>>>>> +          != server->client->cl_auth->au_flavor)
> >>>>>>>>> +               return 0;
> >>>>>>>> 
> >>>>>>>> Isn't this check already being performed in
> >>>>>>>> nfs_compare_mount_options()? As far as I can see, the difference is
> >>>>>>>> that you are checking unconditionally, whereas
> >>>>>>>> nfs_compare_mount_options only does so if there was a 'sec=' line
> >>>>>>>> specified in the mount options.
> >>>>>>> 
> >>>>>>> Right. If the user doesn't provide a sec=, the security flavor is
> >>>>>>> autonegotiated. In the case Frank describes, there are two directories
> >>>>>>> shared on the server, each from the same FSID but using distinct
> >>>>>>> security policies.
> >>>>>>> 
> >>>>>>> So the mount options comparison is inadequate if no sec= is specified
> >>>>>>> on the mount command line.
> >>>>>> 
> >>>>>> We don't claim to support autonegotiation of multiple security policies per
> >>>>>> filesystem, in general. We only claim to support one auth flavour per super
> >>>>>> block.
> >>>>>> 
> >>>>>> If I understand you correctly, you are knowingly trying to work around that
> >>>>>> limitation by performing multiple mounts of the same filesystem and force it
> >>>>>> to use multiple super blocks. Why can't you then also specify the 'sec='?
> >>>>> 
> >>>>> I see that point, but why not just make this case work smoothly rather than force the user to go back and specify -o sec on the mount?
> >>>> 
> >>>> The main issue is that it violates the policy that we must try our
> >>>> best not to set up situations where the client has cache consistency
> >>>> issues due to the existence of multiple superblocks that all have page
> >>>> caches for the same file.
> >>> 
> >>> The parts of the physical FS's namespace that are accessible
> >>> by each security flavor are disjoint. Aside from hardlinks, is
> >>> there any possibility for cache aliasing in this scenario?
> >>> 
> >>> 
> >>>>> Actually all that is necessary is SOME difference in mount options (or use -o nosharedcache, which could be used on all the mounts so all can have the same mount options...) and allow security negotiation to work.
> >>>> 
> >>>> I'd expect there to be no problems if the admin specifies -o
> >>>> nosharedcache. Please do let me know if that fails to work.
> >>>> 
> >>>>> An interesting question is if there are any servers out there that don't typically provide different FSID for different portions of the namespace, but also provide a mechanism to specify different security policies for different portions of the namespace?

Yes, this is quite common.


> >>>> 
> >>>> That sort of situation is difficult to manage.
> >>> 
> >>> But appears to be allowed by Solaris, and likely also by Linux
> >>> and Ganesha. I think we are going to have to consider it,
> >>> particularly if there is no prohibition against it in the RFCs.
> >> 
> >> It is certainly possible to document best practices or
> >> add admin UI limits to prevent servers from being
> >> configured this way.
> >> 
> >> Meanwhile, the Linux client does allow mounting such
> >> exports when both mounts specify unique “sec=“. If this
> >> is a dangerous or unsupported scenario, perhaps this
> >> should be disallowed somehow.
> >> 
> >> When security is negotiated, the second mount is not
> >> allowed. It could display an informative error message
> >> when if fails.
> > 
> > My main worry with the patch you proposed is that we're only
> > considering a small part of the problem here, namely what happens on
> > mount.
> > What if the user were to mount '/' instead of the particular path you
> > chose, and then simply walk down to the problematic directory? As far
> > as I can see, that will fail just as badly both with and without this
> > patch. Why would users expect that behaviour to be any different to
> > the new mount behaviour?
> 
> <shrug> Maybe that’s a little different, though I don’t
> have direct experience of how this is supposed to work.
> 
> If the client mounts “/“ with an explicit security flavor
> that does not work with all the server’s shares, then
> that’s clearly an admin choice. The root directories of
> inaccessible shares will simply not allow users to cd
> into them.
> 
> When no “sec=“ is specified when mounting “/“ then is the
> client allowed to renegotiate when crossing into real
> shares, or are the real shares accessed only via the same
> flavor that was negotiated for “/“ ? 

5661 is pretty clear about this:

   Because NFSv4 clients possess the ability to change the security
   mechanisms used, after determining what is allowed, by using SECINFO
   and SECINFO_NONAME, the server SHOULD NOT present a different view of
   the namespace based on the security mechanism being used by a client.


> If it’s the former,
> do we know that works well for any set of possible
> security policies specified on the server’s shares? If
> it’s the latter, is it a surprise when some shares are
> not accessible? Dunno.
> 
> 
> > IOW: what is the full set of expectations would we be setting by
> > applying the patch, and what is the rationale for changing some
> > behaviour, but not all?
> 
> Our testing turned up this inconsistency. I don’t think
> we have a particular favorite outcome here, other than
> safe, consistent behavior.
> 
> Since specifying different “sec=“ works now, Frank and I
> thought it was indeed a supported configuration.
> 
> If it’s not, then let’s fix the Linux client to behave
> consistently and/or fail gracefully. The client (or an
> appropriate man page) should help admins understand
> what might be going wrong.
> 
> 
> > Also, while I know that the Linux client has never supported this
> > behaviour. What do the Solaris and FreeBSD clients do, and what are
> > their limitations?
> 
> I’m told the Solaris client has a similar design as
> Linux, in that it attempts to match one superblock to
> each FSID on the server.
> 
> It tries to behave in a consistent manner by using the
> WRONGSEC errors to flip back and forth between security
> flavors as necessary. The Solaris folks recognize this
> is not a good permanent solution, and have similar
> concerns about whether this is a reasonable scenario
> to try to support with more complex logic on the client.
> 
> I suggested some server changes before. It would be
> useful to know whether sharing more than one directory
> on a server’s physical filesystem using distinct
> security policies is a common usage scenario. If it
> isn’t, then preventing such multi-security mounts on
> the client doesn’t seem onerous.

With Data ONTAP 7-mode, this is very easy to do, especially
with qtrees.

With Data ONTAP C-mode, the norm is to export only at the
root of the filesystem. But note that it can massage the
fsids to make it look like there is only one filesystem. So
you would have to be very careful that such a change did
not break access here.

Also, with ZFS under Solaris, it is not very painful to
create a new "filessytem" per directory, etc. I.e., each
for a pool serving up homedirs, each user can appear to
have their own filesystem and their own share/export.

I don't know how easy this would be with a Linux server.
E.g., in a set-top deployment, you would want to restrict
each homedir to a specific IP. While you wouldn't necessarily
have different security-flavors here, you would have different
export options per directory.

> 
> 
> —
> Chuck Lever
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Use a separate superblock if mount requires a different security flavor

[Trond Myklebust]

On Mon, Sep 21, 2015 at 9:31 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
>
>> On Sep 21, 2015, at 5:43 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:
>>
>> On Mon, Sep 21, 2015 at 7:10 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
>>>
>>>> On Sep 17, 2015, at 11:01 AM, Chuck Lever <chuck.lever@oracle.com> wrote:
>>>>
>>>>
>>>> On Sep 16, 2015, at 11:32 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:
>>>>
>>>>> On Wed, Sep 16, 2015 at 5:36 PM, Frank Filz <ffilzlnx@mindspring.com> wrote:
>>>>>>> On Wed, Sep 16, 2015 at 4:55 PM, Chuck Lever <chuck.lever@oracle.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Sep 16, 2015, at 4:52 PM, Trond Myklebust
>>>>>>> <trond.myklebust@primarydata.com> wrote:
>>>>>>>>
>>>>>>>>> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com>
>>>>>>> wrote:
>>>>>>>>>> If a server has two exports from the same filesystem but with
>>>>>>>>>> different security flavors allowed, when the client mounts first one
>>>>>>>>>> and then the second, the same super block was being used. This
>>>>>>>>>> resulted in the security flavor for the first export being applied
>>>>>>>>>> to access to the second export.
>>>>>>>>>>
>>>>>>>>>> The fix is simply to check the security flavor of the nfs_server
>>>>>>>>>> temporarily constructed for the second mount within
>>>>>>> nfs_compare_super.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
>>>>>>>>>> ---
>>>>>>>>>> fs/nfs/super.c | 3 +++
>>>>>>>>>> 1 file changed, 3 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c index 084af10..44d60f1
>>>>>>>>>> 100644
>>>>>>>>>> --- a/fs/nfs/super.c
>>>>>>>>>> +++ b/fs/nfs/super.c
>>>>>>>>>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct
>>>>>>>>>> super_block *sb, void *data)
>>>>>>>>>>     struct nfs_server *server = sb_mntdata->server, *old =
>>>>>>>>>> NFS_SB(sb);
>>>>>>>>>>     int mntflags = sb_mntdata->mntflags;
>>>>>>>>>>
>>>>>>>>>> +       if(old->client->cl_auth->au_flavor
>>>>>>>>>> +          != server->client->cl_auth->au_flavor)
>>>>>>>>>> +               return 0;
>>>>>>>>>
>>>>>>>>> Isn't this check already being performed in
>>>>>>>>> nfs_compare_mount_options()? As far as I can see, the difference is
>>>>>>>>> that you are checking unconditionally, whereas
>>>>>>>>> nfs_compare_mount_options only does so if there was a 'sec=' line
>>>>>>>>> specified in the mount options.
>>>>>>>>
>>>>>>>> Right. If the user doesn't provide a sec=, the security flavor is
>>>>>>>> autonegotiated. In the case Frank describes, there are two directories
>>>>>>>> shared on the server, each from the same FSID but using distinct
>>>>>>>> security policies.
>>>>>>>>
>>>>>>>> So the mount options comparison is inadequate if no sec= is specified
>>>>>>>> on the mount command line.
>>>>>>>
>>>>>>> We don't claim to support autonegotiation of multiple security policies per
>>>>>>> filesystem, in general. We only claim to support one auth flavour per super
>>>>>>> block.
>>>>>>>
>>>>>>> If I understand you correctly, you are knowingly trying to work around that
>>>>>>> limitation by performing multiple mounts of the same filesystem and force it
>>>>>>> to use multiple super blocks. Why can't you then also specify the 'sec='?
>>>>>>
>>>>>> I see that point, but why not just make this case work smoothly rather than force the user to go back and specify -o sec on the mount?
>>>>>
>>>>> The main issue is that it violates the policy that we must try our
>>>>> best not to set up situations where the client has cache consistency
>>>>> issues due to the existence of multiple superblocks that all have page
>>>>> caches for the same file.
>>>>
>>>> The parts of the physical FS's namespace that are accessible
>>>> by each security flavor are disjoint. Aside from hardlinks, is
>>>> there any possibility for cache aliasing in this scenario?
>>>>
>>>>
>>>>>> Actually all that is necessary is SOME difference in mount options (or use -o nosharedcache, which could be used on all the mounts so all can have the same mount options...) and allow security negotiation to work.
>>>>>
>>>>> I'd expect there to be no problems if the admin specifies -o
>>>>> nosharedcache. Please do let me know if that fails to work.
>>>>>
>>>>>> An interesting question is if there are any servers out there that don't typically provide different FSID for different portions of the namespace, but also provide a mechanism to specify different security policies for different portions of the namespace?
>>>>>
>>>>> That sort of situation is difficult to manage.
>>>>
>>>> But appears to be allowed by Solaris, and likely also by Linux
>>>> and Ganesha. I think we are going to have to consider it,
>>>> particularly if there is no prohibition against it in the RFCs.
>>>
>>> It is certainly possible to document best practices or
>>> add admin UI limits to prevent servers from being
>>> configured this way.
>>>
>>> Meanwhile, the Linux client does allow mounting such
>>> exports when both mounts specify unique “sec=“. If this
>>> is a dangerous or unsupported scenario, perhaps this
>>> should be disallowed somehow.
>>>
>>> When security is negotiated, the second mount is not
>>> allowed. It could display an informative error message
>>> when if fails.
>>
>> My main worry with the patch you proposed is that we're only
>> considering a small part of the problem here, namely what happens on
>> mount.
>> What if the user were to mount '/' instead of the particular path you
>> chose, and then simply walk down to the problematic directory? As far
>> as I can see, that will fail just as badly both with and without this
>> patch. Why would users expect that behaviour to be any different to
>> the new mount behaviour?
>
> <shrug> Maybe that’s a little different, though I don’t
> have direct experience of how this is supposed to work.
>
> If the client mounts “/“ with an explicit security flavor
> that does not work with all the server’s shares, then
> that’s clearly an admin choice. The root directories of
> inaccessible shares will simply not allow users to cd
> into them.
>

Right now, I can do

mkdir -p /mnt; umount -a -t nfs,nfs4; mount server:/ /mnt; cd /mnt/foo/bar

or I can do

mkdir -p /mnt/foo/bar; umount -a -t nfs,nfs4; mount server:/foo/mnt
/mnt/foo/bar; cd /mnt/foo/bar

I can expect both approaches to lead to the exact same behaviour in
directory /mnt/foo/bar. This is true no matter what the path I'm
mounting or looking up, as long as I use the default mount options.

Any patches need to preserve this property that mount and lookup are
the same operation...

RE: [PATCH] Use a separate superblock if mount requires a different security flavor

[Frank Filz]

From: Trond Myklebust [mailto:trond.myklebust@primarydata.com]
> Sent: Tuesday, September 22, 2015 6:00 AM
> To: Chuck Lever <chuck.lever@oracle.com>
> Cc: Frank Filz <ffilzlnx@mindspring.com>; Linux NFS Mailing List <linux-
> nfs@vger.kernel.org>
> Subject: Re: [PATCH] Use a separate superblock if mount requires a different
> security flavor
> 
> On Mon, Sep 21, 2015 at 9:31 PM, Chuck Lever <chuck.lever@oracle.com>
> wrote:
> >
> >> On Sep 21, 2015, at 5:43 PM, Trond Myklebust
> <trond.myklebust@primarydata.com> wrote:
> >>
> >> On Mon, Sep 21, 2015 at 7:10 PM, Chuck Lever <chuck.lever@oracle.com>
> wrote:
> >>>
> >>>> On Sep 17, 2015, at 11:01 AM, Chuck Lever <chuck.lever@oracle.com>
> wrote:
> >>>>
> >>>>
> >>>> On Sep 16, 2015, at 11:32 PM, Trond Myklebust
> <trond.myklebust@primarydata.com> wrote:
> >>>>
> >>>>> On Wed, Sep 16, 2015 at 5:36 PM, Frank Filz
> <ffilzlnx@mindspring.com> wrote:
> >>>>>>> On Wed, Sep 16, 2015 at 4:55 PM, Chuck Lever
> >>>>>>> <chuck.lever@oracle.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> On Sep 16, 2015, at 4:52 PM, Trond Myklebust
> >>>>>>> <trond.myklebust@primarydata.com> wrote:
> >>>>>>>>
> >>>>>>>>> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz
> >>>>>>>>> <ffilzlnx@mindspring.com>
> >>>>>>> wrote:
> >>>>>>>>>> If a server has two exports from the same filesystem but with
> >>>>>>>>>> different security flavors allowed, when the client mounts
> >>>>>>>>>> first one and then the second, the same super block was being
> >>>>>>>>>> used. This resulted in the security flavor for the first
> >>>>>>>>>> export being applied to access to the second export.
> >>>>>>>>>>
> >>>>>>>>>> The fix is simply to check the security flavor of the
> >>>>>>>>>> nfs_server temporarily constructed for the second mount
> >>>>>>>>>> within
> >>>>>>> nfs_compare_super.
> >>>>>>>>>>
> >>>>>>>>>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
> >>>>>>>>>> ---
> >>>>>>>>>> fs/nfs/super.c | 3 +++
> >>>>>>>>>> 1 file changed, 3 insertions(+)
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c index
> >>>>>>>>>> 084af10..44d60f1
> >>>>>>>>>> 100644
> >>>>>>>>>> --- a/fs/nfs/super.c
> >>>>>>>>>> +++ b/fs/nfs/super.c
> >>>>>>>>>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct
> >>>>>>>>>> super_block *sb, void *data)
> >>>>>>>>>>     struct nfs_server *server = sb_mntdata->server, *old =
> >>>>>>>>>> NFS_SB(sb);
> >>>>>>>>>>     int mntflags = sb_mntdata->mntflags;
> >>>>>>>>>>
> >>>>>>>>>> +       if(old->client->cl_auth->au_flavor
> >>>>>>>>>> +          != server->client->cl_auth->au_flavor)
> >>>>>>>>>> +               return 0;
> >>>>>>>>>
> >>>>>>>>> Isn't this check already being performed in
> >>>>>>>>> nfs_compare_mount_options()? As far as I can see, the
> >>>>>>>>> difference is that you are checking unconditionally, whereas
> >>>>>>>>> nfs_compare_mount_options only does so if there was a 'sec='
> >>>>>>>>> line specified in the mount options.
> >>>>>>>>
> >>>>>>>> Right. If the user doesn't provide a sec=, the security flavor
> >>>>>>>> is autonegotiated. In the case Frank describes, there are two
> >>>>>>>> directories shared on the server, each from the same FSID but
> >>>>>>>> using distinct security policies.
> >>>>>>>>
> >>>>>>>> So the mount options comparison is inadequate if no sec= is
> >>>>>>>> specified on the mount command line.
> >>>>>>>
> >>>>>>> We don't claim to support autonegotiation of multiple security
> >>>>>>> policies per filesystem, in general. We only claim to support
> >>>>>>> one auth flavour per super block.
> >>>>>>>
> >>>>>>> If I understand you correctly, you are knowingly trying to work
> >>>>>>> around that limitation by performing multiple mounts of the same
> >>>>>>> filesystem and force it to use multiple super blocks. Why can't you
> then also specify the 'sec='?
> >>>>>>
> >>>>>> I see that point, but why not just make this case work smoothly
> rather than force the user to go back and specify -o sec on the mount?
> >>>>>
> >>>>> The main issue is that it violates the policy that we must try our
> >>>>> best not to set up situations where the client has cache
> >>>>> consistency issues due to the existence of multiple superblocks
> >>>>> that all have page caches for the same file.
> >>>>
> >>>> The parts of the physical FS's namespace that are accessible by
> >>>> each security flavor are disjoint. Aside from hardlinks, is there
> >>>> any possibility for cache aliasing in this scenario?
> >>>>
> >>>>
> >>>>>> Actually all that is necessary is SOME difference in mount options (or
> use -o nosharedcache, which could be used on all the mounts so all can have
> the same mount options...) and allow security negotiation to work.
> >>>>>
> >>>>> I'd expect there to be no problems if the admin specifies -o
> >>>>> nosharedcache. Please do let me know if that fails to work.
> >>>>>
> >>>>>> An interesting question is if there are any servers out there that
> don't typically provide different FSID for different portions of the
> namespace, but also provide a mechanism to specify different security
> policies for different portions of the namespace?
> >>>>>
> >>>>> That sort of situation is difficult to manage.
> >>>>
> >>>> But appears to be allowed by Solaris, and likely also by Linux and
> >>>> Ganesha. I think we are going to have to consider it, particularly
> >>>> if there is no prohibition against it in the RFCs.
> >>>
> >>> It is certainly possible to document best practices or add admin UI
> >>> limits to prevent servers from being configured this way.
> >>>
> >>> Meanwhile, the Linux client does allow mounting such exports when
> >>> both mounts specify unique “sec=“. If this is a dangerous or
> >>> unsupported scenario, perhaps this should be disallowed somehow.
> >>>
> >>> When security is negotiated, the second mount is not allowed. It
> >>> could display an informative error message when if fails.
> >>
> >> My main worry with the patch you proposed is that we're only
> >> considering a small part of the problem here, namely what happens on
> >> mount.
> >> What if the user were to mount '/' instead of the particular path you
> >> chose, and then simply walk down to the problematic directory? As far
> >> as I can see, that will fail just as badly both with and without this
> >> patch. Why would users expect that behaviour to be any different to
> >> the new mount behaviour?
> >
> > <shrug> Maybe that’s a little different, though I don’t have direct
> > experience of how this is supposed to work.
> >
> > If the client mounts “/“ with an explicit security flavor that does
> > not work with all the server’s shares, then that’s clearly an admin
> > choice. The root directories of inaccessible shares will simply not
> > allow users to cd into them.
> >
> 
> Right now, I can do
> 
> mkdir -p /mnt; umount -a -t nfs,nfs4; mount server:/ /mnt; cd /mnt/foo/bar
> 
> or I can do
> 
> mkdir -p /mnt/foo/bar; umount -a -t nfs,nfs4; mount server:/foo/mnt
> /mnt/foo/bar; cd /mnt/foo/bar
> 
> I can expect both approaches to lead to the exact same behaviour in
> directory /mnt/foo/bar. This is true no matter what the path I'm mounting or
> looking up, as long as I use the default mount options.
> 
> Any patches need to preserve this property that mount and lookup are the
> same operation...

I just verified this. I have two exports (of interest):

/export/sys
/export/none

Without the patch, if I mount /export/sys, I can not subsequently mount /export/none. If I mount /, I can ls /mnt/sys, but not /mnt/none (getting the same EPERM error).

With the patch, I can mount both /export/sys and /export/none, or I can mount / and ls both /mnt/sys and /mnt/none.

Looking at a wireshark trace, in the mount / and then ls the two directories scenario, I see LOOKUP failing with NFS4ERR_WRONGSEC followed by SECINFO followed by a new LOOKUP that succeeds.

Note that after the two ls, this is what I see in mount:

192.168.0.110:/ on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.117,local_lock=none,addr=192.168.0.110)
192.168.0.110:/export/sys on /mnt/export/sys type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.117,local_lock=none,addr=192.168.0.110)
192.168.0.110:/export/none on /mnt/export/none type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=null,clientaddr=192.168.0.117,local_lock=none,addr=192.168.0.110)

It looks to me like everything is working as I would have expected.

I think it actually speaks to the robustness of the mounting code that all we needed to do was add a check on the security flavor for the "server" to identify that a new superblock was necessary.

Frank


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: [PATCH] Use a separate superblock if mount requires a different security flavor

[Frank Filz]

I'll see what happens with mounting / and then navigating into the two exports. My gut feel is my patch will actually make that work but I acknowledge the automatic mounts that will occur might follow a different path that isn't helped by my patch. 

I can also try and find out what freebsd client does. 

I also agree with Chuck that some understanding if this configuration is something sysadmins genuinely expect to work (as opposed to QA folks trying the scenario because no documentation says it isn't supported).

If we decide it shouldn't be supported then we should consider making server side changes. 

Frank

Sent from my iPhone

On Sep 21, 2015, at 5:43 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:

> On Mon, Sep 21, 2015 at 7:10 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
>> 
>>> On Sep 17, 2015, at 11:01 AM, Chuck Lever <chuck.lever@oracle.com> wrote:
>>> 
>>> 
>>> On Sep 16, 2015, at 11:32 PM, Trond Myklebust <trond.myklebust@primarydata.com> wrote:
>>> 
>>>> On Wed, Sep 16, 2015 at 5:36 PM, Frank Filz <ffilzlnx@mindspring.com> wrote:
>>>>>> On Wed, Sep 16, 2015 at 4:55 PM, Chuck Lever <chuck.lever@oracle.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> On Sep 16, 2015, at 4:52 PM, Trond Myklebust
>>>>>> <trond.myklebust@primarydata.com> wrote:
>>>>>>> 
>>>>>>>> On Wed, Sep 16, 2015 at 2:49 PM, Frank Filz <ffilzlnx@mindspring.com>
>>>>>> wrote:
>>>>>>>>> If a server has two exports from the same filesystem but with
>>>>>>>>> different security flavors allowed, when the client mounts first one
>>>>>>>>> and then the second, the same super block was being used. This
>>>>>>>>> resulted in the security flavor for the first export being applied
>>>>>>>>> to access to the second export.
>>>>>>>>> 
>>>>>>>>> The fix is simply to check the security flavor of the nfs_server
>>>>>>>>> temporarily constructed for the second mount within
>>>>>> nfs_compare_super.
>>>>>>>>> 
>>>>>>>>> Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
>>>>>>>>> ---
>>>>>>>>> fs/nfs/super.c | 3 +++
>>>>>>>>> 1 file changed, 3 insertions(+)
>>>>>>>>> 
>>>>>>>>> diff --git a/fs/nfs/super.c b/fs/nfs/super.c index 084af10..44d60f1
>>>>>>>>> 100644
>>>>>>>>> --- a/fs/nfs/super.c
>>>>>>>>> +++ b/fs/nfs/super.c
>>>>>>>>> @@ -2455,6 +2455,9 @@ static int nfs_compare_super(struct
>>>>>>>>> super_block *sb, void *data)
>>>>>>>>>     struct nfs_server *server = sb_mntdata->server, *old =
>>>>>>>>> NFS_SB(sb);
>>>>>>>>>     int mntflags = sb_mntdata->mntflags;
>>>>>>>>> 
>>>>>>>>> +       if(old->client->cl_auth->au_flavor
>>>>>>>>> +          != server->client->cl_auth->au_flavor)
>>>>>>>>> +               return 0;
>>>>>>>> 
>>>>>>>> Isn't this check already being performed in
>>>>>>>> nfs_compare_mount_options()? As far as I can see, the difference is
>>>>>>>> that you are checking unconditionally, whereas
>>>>>>>> nfs_compare_mount_options only does so if there was a 'sec=' line
>>>>>>>> specified in the mount options.
>>>>>>> 
>>>>>>> Right. If the user doesn't provide a sec=, the security flavor is
>>>>>>> autonegotiated. In the case Frank describes, there are two directories
>>>>>>> shared on the server, each from the same FSID but using distinct
>>>>>>> security policies.
>>>>>>> 
>>>>>>> So the mount options comparison is inadequate if no sec= is specified
>>>>>>> on the mount command line.
>>>>>> 
>>>>>> We don't claim to support autonegotiation of multiple security policies per
>>>>>> filesystem, in general. We only claim to support one auth flavour per super
>>>>>> block.
>>>>>> 
>>>>>> If I understand you correctly, you are knowingly trying to work around that
>>>>>> limitation by performing multiple mounts of the same filesystem and force it
>>>>>> to use multiple super blocks. Why can't you then also specify the 'sec='?
>>>>> 
>>>>> I see that point, but why not just make this case work smoothly rather than force the user to go back and specify -o sec on the mount?
>>>> 
>>>> The main issue is that it violates the policy that we must try our
>>>> best not to set up situations where the client has cache consistency
>>>> issues due to the existence of multiple superblocks that all have page
>>>> caches for the same file.
>>> 
>>> The parts of the physical FS's namespace that are accessible
>>> by each security flavor are disjoint. Aside from hardlinks, is
>>> there any possibility for cache aliasing in this scenario?
>>> 
>>> 
>>>>> Actually all that is necessary is SOME difference in mount options (or use -o nosharedcache, which could be used on all the mounts so all can have the same mount options...) and allow security negotiation to work.
>>>> 
>>>> I'd expect there to be no problems if the admin specifies -o
>>>> nosharedcache. Please do let me know if that fails to work.
>>>> 
>>>>> An interesting question is if there are any servers out there that don't typically provide different FSID for different portions of the namespace, but also provide a mechanism to specify different security policies for different portions of the namespace?
>>>> 
>>>> That sort of situation is difficult to manage.
>>> 
>>> But appears to be allowed by Solaris, and likely also by Linux
>>> and Ganesha. I think we are going to have to consider it,
>>> particularly if there is no prohibition against it in the RFCs.
>> 
>> It is certainly possible to document best practices or
>> add admin UI limits to prevent servers from being
>> configured this way.
>> 
>> Meanwhile, the Linux client does allow mounting such
>> exports when both mounts specify unique “sec=“. If this
>> is a dangerous or unsupported scenario, perhaps this
>> should be disallowed somehow.
>> 
>> When security is negotiated, the second mount is not
>> allowed. It could display an informative error message
>> when if fails.
> 
> My main worry with the patch you proposed is that we're only
> considering a small part of the problem here, namely what happens on
> mount.
> What if the user were to mount '/' instead of the particular path you
> chose, and then simply walk down to the problematic directory? As far
> as I can see, that will fail just as badly both with and without this
> patch. Why would users expect that behaviour to be any different to
> the new mount behaviour?
> 
> IOW: what is the full set of expectations would we be setting by
> applying the patch, and what is the rationale for changing some
> behaviour, but not all?
> 
> Also, while I know that the Linux client has never supported this
> behaviour. What do the Solaris and FreeBSD clients do, and what are
> their limitations?
> 
> Cheers
>  Trond