CAP(abilities) and NFS mounted storage

CAP(abilities) and NFS mounted storage

[Sander Smeenk]

Hi,

I'm struggling with a permission issue involving NFS-mounted storage and
a certain set of capabilities set by cap_set_flags(). The behaviour
differs for local storage versus NFS mounted storage.

I have this structure on local storage (spinning disks):
| # namei -l /opt/home/sites/t/test/dir/structure/.htaccess
| f: /opt/home/sites/t/test/dir/structure/.htaccess
| drwxr-xr-x root      root                 /
| drwxr-xr-x root      root                 opt
| drwxr-x--x root      root                 home
| drwx--x--x root      root                 sites
| drwx--x--x root      root                 t
| drwx------ http-test http-linux_http-test test
| drwx--x--x http-test http-linux_http-test dir
| drwx------ http-test http-linux_http-test structure
| -rw------- http-test http-linux_http-test .htaccess

And this same structure on NFS-mounted storage:
| # namei -l /mnt/home/sites/t/test/dir/structure/.htaccess
| f: /mnt/home/sites/t/test/dir/structure/.htaccess
| drwxr-xr-x root      root                 /
| drwxr-xr-x root      root                 mnt
| drwxr-x--x root      root                 home
| drwx--x--x root      root                 sites
| drwx--x--x root      root                 t
| drwx------ http-test http-linux_http-test test
| drwx--x--x http-test http-linux_http-test dir
| drwx------ http-test http-linux_http-test structure
| -rw------- http-test http-linux_http-test .htaccess

The NFS server is a NetApp filer (-sec=sys,rw=clientip,root=clientip).
I tried this with a Linux server too (rw,no_root_squash,no_subtree_check).
The client is always a Linux machine (rw,vers=3,tcp,bg).

I made a little C program to illustrate the issue. It drops privileges
to www-data and tries to access the file specified with a certain set of
capabilties[*].

This works for local storage, fails on NFS:

LOCAL:
| # ./capset /opt/home/sites/t/test/dir/structure/.htaccess
| euid:33 uid:33 egid:33 gid:33
| Process capabilities: = cap_dac_read_search,cap_setgid,cap_setuid,cap_sys_nice+ep;
| Access: success!

NFS:
| # ./capset /mnt/home/sites/t/test/dir/structure/.htaccess
| euid:33 uid:33 egid:33 gid:33
| Process capabilities: = cap_dac_read_search,cap_setgid,cap_setuid,cap_sys_nice+ep;
| Access: error (13): Permission denied

The source for capset can be seen pasted at https://8n1.org/10831/12f0
Lines >=42

I've experimented with different capabilties, but CAP_DAC_OVERRIDE is
not enough. I'd very much like to hear if it is possible for this to
work on NFS like it does on local storage.

Any ideas? 

Thanks in advance.
-Sndr.

[*] This issue popped up since Apache module 'mpm_itk' started using
libcap capabilties to further enhance the security. The capabilties set
was taken from mpm_itk source to 'prove the point' w/o the entire Apache
setup. 
-- 
| I wish i was a glow worm, a glow worm's never glum.
| How can you be unhappy when the sun shines out your bum!
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7  FBD6 F3A9 9442 20CC 6CD2

Re: CAP(abilities) and NFS mounted storage

[Trond Myklebust]

On Tue, Oct 13, 2015 at 8:21 AM, Sander Smeenk <ssmeenk@freshdot.net> wrote:
> Hi,
>
> I'm struggling with a permission issue involving NFS-mounted storage and
> a certain set of capabilities set by cap_set_flags(). The behaviour
> differs for local storage versus NFS mounted storage.
>
> I have this structure on local storage (spinning disks):
> | # namei -l /opt/home/sites/t/test/dir/structure/.htaccess
> | f: /opt/home/sites/t/test/dir/structure/.htaccess
> | drwxr-xr-x root      root                 /
> | drwxr-xr-x root      root                 opt
> | drwxr-x--x root      root                 home
> | drwx--x--x root      root                 sites
> | drwx--x--x root      root                 t
> | drwx------ http-test http-linux_http-test test
> | drwx--x--x http-test http-linux_http-test dir
> | drwx------ http-test http-linux_http-test structure
> | -rw------- http-test http-linux_http-test .htaccess
>
> And this same structure on NFS-mounted storage:
> | # namei -l /mnt/home/sites/t/test/dir/structure/.htaccess
> | f: /mnt/home/sites/t/test/dir/structure/.htaccess
> | drwxr-xr-x root      root                 /
> | drwxr-xr-x root      root                 mnt
> | drwxr-x--x root      root                 home
> | drwx--x--x root      root                 sites
> | drwx--x--x root      root                 t
> | drwx------ http-test http-linux_http-test test
> | drwx--x--x http-test http-linux_http-test dir
> | drwx------ http-test http-linux_http-test structure
> | -rw------- http-test http-linux_http-test .htaccess
>
> The NFS server is a NetApp filer (-sec=sys,rw=clientip,root=clientip).
> I tried this with a Linux server too (rw,no_root_squash,no_subtree_check).
> The client is always a Linux machine (rw,vers=3,tcp,bg).
>
> I made a little C program to illustrate the issue. It drops privileges
> to www-data and tries to access the file specified with a certain set of
> capabilties[*].
>
> This works for local storage, fails on NFS:
>
> LOCAL:
> | # ./capset /opt/home/sites/t/test/dir/structure/.htaccess
> | euid:33 uid:33 egid:33 gid:33
> | Process capabilities: = cap_dac_read_search,cap_setgid,cap_setuid,cap_sys_nice+ep;
> | Access: success!
>
> NFS:
> | # ./capset /mnt/home/sites/t/test/dir/structure/.htaccess
> | euid:33 uid:33 egid:33 gid:33
> | Process capabilities: = cap_dac_read_search,cap_setgid,cap_setuid,cap_sys_nice+ep;
> | Access: error (13): Permission denied
>
> The source for capset can be seen pasted at https://8n1.org/10831/12f0
> Lines >=42
>
> I've experimented with different capabilties, but CAP_DAC_OVERRIDE is
> not enough. I'd very much like to hear if it is possible for this to
> work on NFS like it does on local storage.

This will not work on NFS. The server, which enforces permissions, has
no way to know what capabilities your process has on the client.

Cheers
  Trond

Re: CAP(abilities) and NFS mounted storage

[Sander Smeenk]

Quoting Trond Myklebust (trond.myklebust@primarydata.com):

> > I've experimented with different capabilties, but CAP_DAC_OVERRIDE is
> > not enough. I'd very much like to hear if it is possible for this to
> > work on NFS like it does on local storage.
> This will not work on NFS. The server, which enforces permissions, has
> no way to know what capabilities your process has on the client.

Thanks. I feared this answer. But i understand that the NFS-server cant
know if the process on the NFS-client has CAP_DAC_READ_SEARCH
capabilities set.

Would setfsuid() help anything in this case? Or is it just a big no-go?

-Sndr.
-- 
| Daylight savings time - why are they saving it and where do they keep it?
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7  FBD6 F3A9 9442 20CC 6CD2

Re: CAP(abilities) and NFS mounted storage

[Olga Kornievskaia]

On Tue, Oct 13, 2015 at 10:34 AM, Sander Smeenk <ssmeenk@freshdot.net> wrote:
> Quoting Trond Myklebust (trond.myklebust@primarydata.com):
>
>> > I've experimented with different capabilties, but CAP_DAC_OVERRIDE is
>> > not enough. I'd very much like to hear if it is possible for this to
>> > work on NFS like it does on local storage.
>> This will not work on NFS. The server, which enforces permissions, has
>> no way to know what capabilities your process has on the client.
>
> Thanks. I feared this answer. But i understand that the NFS-server cant
> know if the process on the NFS-client has CAP_DAC_READ_SEARCH
> capabilities set.
>
> Would setfsuid() help anything in this case? Or is it just a big no-go?
>

Are you looking for something like labeled NFS that supports
capabilities? I think Redhat7 has SElinux labeled NFS support.

> -Sndr.
> --
> | Daylight savings time - why are they saving it and where do they keep it?
> | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7  FBD6 F3A9 9442 20CC 6CD2
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CAP(abilities) and NFS mounted storage

[Trond Myklebust]

On Tue, Oct 13, 2015 at 11:02 AM, Olga Kornievskaia <aglo@umich.edu> wrote:
> On Tue, Oct 13, 2015 at 10:34 AM, Sander Smeenk <ssmeenk@freshdot.net> wrote:
>> Quoting Trond Myklebust (trond.myklebust@primarydata.com):
>>
>>> > I've experimented with different capabilties, but CAP_DAC_OVERRIDE is
>>> > not enough. I'd very much like to hear if it is possible for this to
>>> > work on NFS like it does on local storage.
>>> This will not work on NFS. The server, which enforces permissions, has
>>> no way to know what capabilities your process has on the client.
>>
>> Thanks. I feared this answer. But i understand that the NFS-server cant
>> know if the process on the NFS-client has CAP_DAC_READ_SEARCH
>> capabilities set.
>>
>> Would setfsuid() help anything in this case? Or is it just a big no-go?

setfsuid() would allow you to set a user up with root privileges on
the fs. That's better than giving overall root privileges, but is
still a risk, since a user could use it to overwrite /etc/passwd etc.

> Are you looking for something like labeled NFS that supports
> capabilities? I think Redhat7 has SElinux labeled NFS support.
>

The labeled NFS implementation is client enforced and confers no extra
privileges on the server. Plus (please correct me if I'm wrong), I
believe NetApp has yet to announce support for labeled NFS in OnTAP.

Re: CAP(abilities) and NFS mounted storage

[Sander Smeenk]

Quoting Trond Myklebust (trond.myklebust@primarydata.com):
> >>> > I've experimented with different capabilties, but CAP_DAC_OVERRIDE is
> >>> > not enough. I'd very much like to hear if it is possible for this to
> >>> > work on NFS like it does on local storage.
> >>> This will not work on NFS. The server, which enforces permissions, has
> >>> no way to know what capabilities your process has on the client.
> >> Thanks. I feared this answer. But i understand that the NFS-server cant
> >> know if the process on the NFS-client has CAP_DAC_READ_SEARCH
> >> capabilities set.
> >> Would setfsuid() help anything in this case? Or is it just a big no-go?
> 
> setfsuid() would allow you to set a user up with root privileges on
> the fs. That's better than giving overall root privileges, but is
> still a risk, since a user could use it to overwrite /etc/passwd etc.

I'm not going to run anything i'm trying in a production environment yet. ;)

But, should setfsuid() work with NFS-mounted filesystems too?
'Cause i tried, but it didn't seem to change much (w/ caps set).
Might be i need more capabilities for that to work. Hmm.


> > Are you looking for something like labeled NFS that supports
> > capabilities? I think Redhat7 has SElinux labeled NFS support.
> 
> The labeled NFS implementation is client enforced and confers no extra
> privileges on the server. Plus (please correct me if I'm wrong), I
> believe NetApp has yet to announce support for labeled NFS in OnTAP.

Yes. This is not really an option. ;)


Regards,
-Sndr.
-- 
| Spend some time trying hard not to think about giraffes.
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7  FBD6 F3A9 9442 20CC 6CD2

Re: CAP(abilities) and NFS mounted storage

[Sander Smeenk]

Quoting Olga Kornievskaia (aglo@umich.edu):

> >> > I've experimented with different capabilties, but CAP_DAC_OVERRIDE is
> >> > not enough. I'd very much like to hear if it is possible for this to
> >> > work on NFS like it does on local storage.
> >> This will not work on NFS. The server, which enforces permissions, has
> >> no way to know what capabilities your process has on the client.
> > Thanks. I feared this answer. But i understand that the NFS-server cant
> > know if the process on the NFS-client has CAP_DAC_READ_SEARCH
> > capabilities set.
> > Would setfsuid() help anything in this case? Or is it just a big no-go?
> 
> Are you looking for something like labeled NFS that supports
> capabilities? I think Redhat7 has SElinux labeled NFS support.

SELinux might be a bit too much for our situation. ;)

I'm looking for an answer to who exactly controls filesystem access
permission checks when dealing with files on an NFS mount, and perhaps
an answer to why these permission checks seem to differ for files on NFS
mounts and files on locally attached storage w/ strict capabilities set.

Regards,
-Sndr.
-- 
| Have you ever imagined a world with no hypothetical situations?
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7  FBD6 F3A9 9442 20CC 6CD2