Using overlayfs in (unprivileged) namespace

Using overlayfs in (unprivileged) namespace

[Philipp Wendler]

Hello all,

I would like to mount an overlayfs inside unprivileged user and mount
namespaces (i.e., the user creating the namespaces is a regular user
with no special privileges).
This works mostly fine, but it fails as soon as I try to delete a file
which exists in the "lower" directory of the overlay,
because overlayfs then needs to create a "whiteout" file,
for which it uses a device node with 0/0 device number
(https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt),
but I do not have the permission to create device nodes.

Is there any way to make overlayfs work fully in my situation,
without requiring additional privileges?
If not, is this something that could be made work in the future?
Of course, creating arbitrary devices nodes is something that cannot be
granted to an unprivileged user, but in this case it is only a specific
device node with device numbers 0/0, and it is a kernel module creating
the device node on behalf of me.

I am currently using Linux 4.2. To reproduce the problem,
you can use the following steps:
Create the mount and user namespaces with the example program from the
user_namespaces man page
(http://man7.org/linux/man-pages/man7/user_namespaces.7.html),
mapping the user root inside the namespace to my user:

$ ./userns_child_exec -m -U -z bash

Then execute the following commands:

mkdir /tmp/namespace-overlay
cd /tmp/namespace-overlay
mkdir mount lower upper work
touch lower/test
mount -t overlayfs n -o lowerdir=lower,upperdir=upper,workdir=work mount
rm mount/test

The last command gives:
> rm: cannot remove 'mount/test': Operation not permitted

This fails even if /tmp does not have "nodev" set (with "nodev" it would
be expected to fail of course).
Interestingly, it even fails if I start userns_child_exec as root,
not sure why.
Outside namespaces everything works as expected.

Kind regards,
Philipp

Re: Using overlayfs in (unprivileged) namespace

[Nikolay Borisov]

On 02/15/2016 01:04 PM, Philipp Wendler wrote:
> Hello all,
> 
> I would like to mount an overlayfs inside unprivileged user and mount
> namespaces (i.e., the user creating the namespaces is a regular user
> with no special privileges).
> This works mostly fine, but it fails as soon as I try to delete a file
> which exists in the "lower" directory of the overlay,
> because overlayfs then needs to create a "whiteout" file,
> for which it uses a device node with 0/0 device number
> (https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt),
> but I do not have the permission to create device nodes.
> 
> Is there any way to make overlayfs work fully in my situation,
> without requiring additional privileges?
> If not, is this something that could be made work in the future?
> Of course, creating arbitrary devices nodes is something that cannot be
> granted to an unprivileged user, but in this case it is only a specific
> device node with device numbers 0/0, and it is a kernel module creating
> the device node on behalf of me.
> 
> I am currently using Linux 4.2. To reproduce the problem,
> you can use the following steps:
> Create the mount and user namespaces with the example program from the
> user_namespaces man page
> (http://man7.org/linux/man-pages/man7/user_namespaces.7.html),
> mapping the user root inside the namespace to my user:
> 
> $ ./userns_child_exec -m -U -z bash
> 
> Then execute the following commands:
> 
> mkdir /tmp/namespace-overlay
> cd /tmp/namespace-overlay
> mkdir mount lower upper work
> touch lower/test
> mount -t overlayfs n -o lowerdir=lower,upperdir=upper,workdir=work mount
> rm mount/test
> 
> The last command gives:
>> rm: cannot remove 'mount/test': Operation not permitted
> 
> This fails even if /tmp does not have "nodev" set (with "nodev" it would
> be expected to fail of course).
> Interestingly, it even fails if I start userns_child_exec as root,
> not sure why.
> Outside namespaces everything works as expected.

Wouldn't using the device cgroup with the respective major/minor numbers
allowed rectify the situation? Also, have you done any tracing trying to
figure out where exactly is this failing? E.g. using ftrace?


> 
> Kind regards,
> Philipp
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers
> 

Re: Using overlayfs in (unprivileged) namespace

[Philipp Wendler]

Dear Nikolay,

thank you for your answer.

Am 15.02.2016 um 12:47 schrieb Nikolay Borisov:
> On 02/15/2016 01:04 PM, Philipp Wendler wrote:

>> $ ./userns_child_exec -m -U -z bash
>>
>> Then execute the following commands:
>>
>> mkdir /tmp/namespace-overlay
>> cd /tmp/namespace-overlay
>> mkdir mount lower upper work
>> touch lower/test
>> mount -t overlayfs n -o lowerdir=lower,upperdir=upper,workdir=work mount
>> rm mount/test
>>
>> The last command gives:
>>> rm: cannot remove 'mount/test': Operation not permitted
>>
>> This fails even if /tmp does not have "nodev" set (with "nodev" it would
>> be expected to fail of course).
>> Interestingly, it even fails if I start userns_child_exec as root,
>> not sure why.
>> Outside namespaces everything works as expected.
> 
> Wouldn't using the device cgroup with the respective major/minor numbers
> allowed rectify the situation?

I am not sure how.
I have no special setup for the devices cgroup.
The file devices.list of the cgroup in which my process is contains
"a *:* rwm".
So I think the devices cgroup would already allow me to create that
device node, and I have no way to grant additional permissions with the
cgroup, right?
At least that is how I understand the documentation for the devices cgroup.

> Also, have you done any tracing trying to
> figure out where exactly is this failing? E.g. using ftrace?

Sorry, no, I don't know how to do this.
I have looked into ftrace now, but I didn't find a way how to see which
function is responsible for letting the rm fail.
The kernel documentation on ftrace is quite overwhelming, so maybe I
have missed something.
Do you have by chance a more specific pointer to what would help me?

Greetings, Philipp

Re: Using overlayfs in (unprivileged) namespace

[Nikolay Borisov]

On 02/15/2016 02:30 PM, Philipp Wendler wrote:
> I have looked into ftrace now, but I didn't find a way how to see which
> function is responsible for letting the rm fail.
> The kernel documentation on ftrace is quite overwhelming, so maybe I
> have missed something.
> Do you have by chance a more specific pointer to what would help me?

If I were you I would run something along the lines of (inside the
container):

trace-cmd record -p function_graph -F rm some-file

And then you would do :

trace-cmd report

and see the resulting call trace and quite possibly it might be failing
in an ovl_* prefixed function.

Re: Using overlayfs in (unprivileged) namespace

[Philipp Wendler]

Hello,

Am 15.02.2016 um 13:37 schrieb Nikolay Borisov:
> On 02/15/2016 02:30 PM, Philipp Wendler wrote:
>> I have looked into ftrace now, but I didn't find a way how to see which
>> function is responsible for letting the rm fail.
>> The kernel documentation on ftrace is quite overwhelming, so maybe I
>> have missed something.
>> Do you have by chance a more specific pointer to what would help me?
> 
> If I were you I would run something along the lines of (inside the
> container):
> 
> trace-cmd record -p function_graph -F rm some-file
> 
> And then you would do :
> 
> trace-cmd report
> 
> and see the resulting call trace and quite possibly it might be failing
> in an ovl_* prefixed function.

Thanks, this was very helpful, and I know now that I was on a wrong path.

From the trace, I noticed that my Ubuntu kernel actually has an
additional legacy mode of overlayfs which I was using (unintentionally),
and which instead of creating a device node, creates a symlink and tries
to set the "trusted.overlay.whiteout" xattr on it, which fails.
So the failure has nothing to do with devices, sorry for the confusion.

The legacy mode got triggered by the "-t overlayfs" parameter to mount.
If I specify "-t overlay" instead, the normal mode gets used (which
seems to match the one in the vanilla kernel), and with this everything
appears to work nicely, regardless of whether the user namespace was
created by a privileged or an unprivileged user.

So sorry for the confusion and thank you very much for helping me
debugging this!

Greetings, Philipp

Re: Using overlayfs in (unprivileged) namespace

[Serge Hallyn]

Quoting Philipp Wendler (ml-PRuqubkS1MrCxoAYUeDZubNAH6kLmebB@public.gmane.org):
> Hello all,
> 
> I would like to mount an overlayfs inside unprivileged user and mount
> namespaces (i.e., the user creating the namespaces is a regular user
> with no special privileges).
> This works mostly fine, but it fails as soon as I try to delete a file
> which exists in the "lower" directory of the overlay,
> because overlayfs then needs to create a "whiteout" file,
> for which it uses a device node with 0/0 device number
> (https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt),
> but I do not have the permission to create device nodes.
> 
> Is there any way to make overlayfs work fully in my situation,
> without requiring additional privileges?
> If not, is this something that could be made work in the future?
> Of course, creating arbitrary devices nodes is something that cannot be
> granted to an unprivileged user, but in this case it is only a specific
> device node with device numbers 0/0, and it is a kernel module creating
> the device node on behalf of me.
> 
> I am currently using Linux 4.2. To reproduce the problem,

Exactly what version from where?

It sounds to me like you're hitting

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531747

> you can use the following steps:
> Create the mount and user namespaces with the example program from the
> user_namespaces man page
> (http://man7.org/linux/man-pages/man7/user_namespaces.7.html),
> mapping the user root inside the namespace to my user:
> 
> $ ./userns_child_exec -m -U -z bash
> 
> Then execute the following commands:
> 
> mkdir /tmp/namespace-overlay
> cd /tmp/namespace-overlay
> mkdir mount lower upper work
> touch lower/test
> mount -t overlayfs n -o lowerdir=lower,upperdir=upper,workdir=work mount
> rm mount/test
> 
> The last command gives:
> > rm: cannot remove 'mount/test': Operation not permitted
> 
> This fails even if /tmp does not have "nodev" set (with "nodev" it would
> be expected to fail of course).
> Interestingly, it even fails if I start userns_child_exec as root,
> not sure why.
> Outside namespaces everything works as expected.
> 
> Kind regards,
> Philipp
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers

Re: Using overlayfs in (unprivileged) namespace

[Philipp Wendler]

Hello Serge,

Am 15.02.2016 um 19:18 schrieb Serge Hallyn:
> Quoting Philipp Wendler (ml-PRuqubkS1MrCxoAYUeDZubNAH6kLmebB@public.gmane.org):
>> I would like to mount an overlayfs inside unprivileged user and mount
>> namespaces (i.e., the user creating the namespaces is a regular user
>> with no special privileges).
>> This works mostly fine, but it fails as soon as I try to delete a file
>> which exists in the "lower" directory of the overlay,
>> because overlayfs then needs to create a "whiteout" file,
>> for which it uses a device node with 0/0 device number
>> (https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt),
>> but I do not have the permission to create device nodes.
>>
>> Is there any way to make overlayfs work fully in my situation,
>> without requiring additional privileges?
>> If not, is this something that could be made work in the future?
>> Of course, creating arbitrary devices nodes is something that cannot be
>> granted to an unprivileged user, but in this case it is only a specific
>> device node with device numbers 0/0, and it is a kernel module creating
>> the device node on behalf of me.
>>
>> I am currently using Linux 4.2. To reproduce the problem,
> 
> Exactly what version from where?
> 
> It sounds to me like you're hitting
> 
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531747

Thank you for pointing this out!
This bug is different from my original problem, but now that that one is
solved I tried to reproduce this Ubuntu bug, and indeed I am hitting it,
too.

Greetings, Philipp