how to do not allow to mount /cgroup inside container?

how to do not allow to mount /cgroup inside container?

[Krzysztof Taraszka]

Hi,

I was looking for possibility to secure lxc container to do not allow 'root
container user'  from changing limits from cgroup. Right now without STACK64
or SELinux he can do this easily.
I read the http://www.ibm.com/developerworks/linux/library/l-lxc-security/cookbook
and decided to use STACK64 kernel mechanism.
Well... mounting cgroup inside container fails (great!, i am looked for that
;)) but networking fails too (interface bring up, sshd bring up, connection
beetween host and container is, but 'mtr', 'ping' even 'apt-get update'
fails and I do not know why). I secure my container exactly like in the
cookbook.

Is there any other possilbility to have secure container without network
problems or any hint now to enable networking with stack64 enabled? If so,
maybe the l-lxc-security cookbook have to updated? Maybe another kernel
patch to do not allow container to mount cgroup when the mount call come
from container?

Any ideas?

-- 
Krzysztof Taraszka

Re: how to do not allow to mount /cgroup inside container?

[Daniel Lezcano]

Krzysztof Taraszka wrote:
> Hi,
>
> I was looking for possibility to secure lxc container to do not allow 'root
> container user'  from changing limits from cgroup. Right now without STACK64
> or SELinux he can do this easily.
> I read the http://www.ibm.com/developerworks/linux/library/l-lxc-security/cookbook
> and decided to use STACK64 kernel mechanism.
> Well... mounting cgroup inside container fails (great!, i am looked for that
> ;)) but networking fails too (interface bring up, sshd bring up, connection
> beetween host and container is, but 'mtr', 'ping' even 'apt-get update'
> fails and I do not know why). I secure my container exactly like in the
> cookbook.
>
> Is there any other possilbility to have secure container without network
> problems or any hint now to enable networking with stack64 enabled? If so,
> maybe the l-lxc-security cookbook have to updated? Maybe another kernel
> patch to do not allow container to mount cgroup when the mount call come
> from container?
>
> Any ideas?
>   
I think Serge can help you on this area (Cc'ed).

Re: how to do not allow to mount /cgroup inside container?

[Serge E. Hallyn]

Quoting Daniel Lezcano (daniel.lezcano-GANU6spQydw@public.gmane.org):
> Krzysztof Taraszka wrote:
>> Hi,
>>
>> I was looking for possibility to secure lxc container to do not allow 'root
>> container user'  from changing limits from cgroup. Right now without STACK64
>> or SELinux he can do this easily.
>> I read the http://www.ibm.com/developerworks/linux/library/l-lxc-security/cookbook
>> and decided to use STACK64 kernel mechanism.
>> Well... mounting cgroup inside container fails (great!, i am looked for that
>> ;)) but networking fails too (interface bring up, sshd bring up, connection
>> beetween host and container is, but 'mtr', 'ping' even 'apt-get update'
>> fails and I do not know why). I secure my container exactly like in the
>> cookbook.

Yeah, smack's use of cipso can make things tricky, and it's possible things
have changed a bit recently.  Although I'm currently running smack in my
everyday s390 kernel to test checkpointing of its labels, and networking
is working fine.

Can you give me a few details - what distro, smack policy, and precise kernel
version are you using, for starters?

>> Is there any other possilbility to have secure container without network
>> problems or any hint now to enable networking with stack64 enabled? If so,
>> maybe the l-lxc-security cookbook have to updated? Maybe another kernel
>> patch to do not allow container to mount cgroup when the mount call come
>> from container?
>>
>> Any ideas?
>>   
> I think Serge can help you on this area (Cc'ed).

Well the idea is that user namespaces will provide this.  The files in
the cgroupfs will be labeled as being owned by users in the initial
user namespace.  The users in the container would be in a child user
namespace, the namespaces being hierarchical, so for instance the
container might have been created by uid 500 in the initial namespace
(ns=1), with the new namespace being (ns=2).  Then uid 0 in the container
is actually (1:500,2:0) and uid 1000 in the container is (1:500,2:1000).
Now tasks in the container will only own files which are owned by 1:500,
and root tasks in the container can only get privileges (CAP_DAC_OVERRIDE
etc) to files owned by 1:500.  So regular DAC permissions then suffice to
prevent containers from messing with their cgroup constraints.

Unfortunately that is (still) a ways off.  If you have time to work
on that, I think the last time Eric and I discussed how to go about
introducing this functionality was in the thread starting here:
https://lists.linux-foundation.org/pipermail/containers/2008-August/thread.html#12675
and i.e. https://lists.linux-foundation.org/pipermail/containers/2008-August/012691.html
where Eric suggests starting with sorting out 'capable' with respect to
namespaces first.

-serge

Re: how to do not allow to mount /cgroup inside container?

[Krzysztof Taraszka]

2009/8/25 Serge E. Hallyn <serue-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>

> Quoting Daniel Lezcano (daniel.lezcano-GANU6spQydw@public.gmane.org):
> > Krzysztof Taraszka wrote:
> >> Hi,
> >>
> >> I was looking for possibility to secure lxc container to do not allow
> 'root
> >> container user'  from changing limits from cgroup. Right now without
> STACK64
> >> or SELinux he can do this easily.
> >> I read the
> http://www.ibm.com/developerworks/linux/library/l-lxc-security/cookbook
> >> and decided to use STACK64 kernel mechanism.
> >> Well... mounting cgroup inside container fails (great!, i am looked for
> that
> >> ;)) but networking fails too (interface bring up, sshd bring up,
> connection
> >> beetween host and container is, but 'mtr', 'ping' even 'apt-get update'
> >> fails and I do not know why). I secure my container exactly like in the
> >> cookbook.
>
> Yeah, smack's use of cipso can make things tricky, and it's possible things
> have changed a bit recently.  Although I'm currently running smack in my
> everyday s390 kernel to test checkpointing of its labels, and networking
> is working fine.


> Can you give me a few details - what distro, smack policy, and precise
> kernel
> version are you using, for starters?
>

debian lenny amd64,
kernel 2.6.30.5
lxc-tools from git

lxc1amd64:~# cat /etc/smackaccesses
debian _ rwa
_ debian rwa
_ host rwax
host _ rwax

where "debian" is container, "host" is a host.

I did this:

for f in `find /root/rootfs.debian`; do
    attr -S -s SMACK64 -V debian $f
done

on the container fs.

container startup script look like here:

lxc1amd64:~# cat vs1.sh
#!/bin/sh
cp /bin/dropmacadmin /root/rootfs.debian/bin/
attr -S -s SMACK64 -V debian /root/rootfs.debian/bin/dropmacadmin
echo debian > /proc/self/attr/current
lxc-start -n debian /bin/dropmacadmin /sbin/init

/etc/fstab inside container look like:

debian:~# cat /etc/fstab
tmpfs  /dev/shm   tmpfs  defaults,smackfsroot=debian,smackfsdef=debian  0 0

And here is some output when I tried to do ping to the wp.pl, tried to
apt-get update and tried to ping gateway

debian:~# ping wp.pl
PING wp.pl (212.77.100.101) 56(84) bytes of data.
From 10.177.128.1 icmp_seq=1 Parameter problem: pointer = 20
From 10.177.128.1 icmp_seq=2 Parameter problem: pointer = 20
^C
--- wp.pl ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1001ms

debian:~# apt-get update
Err http://ftp.debian.org lenny Release.gpg
  Could not connect to ftp.debian.org:80 (130.89.149.226). - connect (71
Protocol error)
Err http://ftp.debian.org lenny/main Translation-en_US
  Could not connect to ftp.debian.org:80 (130.89.149.226). - connect (71
Protocol error)
Ign http://ftp.debian.org lenny Release
Ign http://ftp.debian.org lenny/main Packages/DiffIndex
Ign http://ftp.debian.org lenny/main Packages
Err http://ftp.debian.org lenny/main Packages
  Could not connect to ftp.debian.org:80 (130.89.149.226). - connect (71
Protocol error)
W: Failed to fetch http://ftp.debian.org/debian/dists/lenny/Release.gpg
Could not connect to ftp.debian.org:80 (130.89.149.226). - connect (71
Protocol error)

W: Failed to fetch
http://ftp.debian.org/debian/dists/lenny/main/i18n/Translation-en_US.gz
Could not connect to ftp.debian.org:80 (130.89.149.226). - connect (71
Protocol error)

W: Failed to fetch
http://ftp.debian.org/debian/dists/lenny/main/binary-amd64/Packages  Could
not connect to ftp.debian.org:80 (130.89.149.226). - connect (71 Protocol
error)

E: Some index files failed to download, they have been ignored, or old ones
used instead.
debian:~# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.085 ms
unknown option 86
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.136 ms
unknown option 86
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.116 ms
unknown option 86
^C
--- 192.168.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2005ms
rtt min/avg/max/mdev = 0.085/0.112/0.136/0.022 ms

did you changed your smack policy or you have the same as mine?

-- 
Krzysztof Taraszka

Re: how to do not allow to mount /cgroup inside container?

[Krzysztof Taraszka]

2009/8/25 Krzysztof Taraszka <krzysztof.taraszka-S89nZTSLPHGGdvJs77BJ7Q@public.gmane.org>

> 2009/8/25 Serge E. Hallyn <serue-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
>
>> Quoting Daniel Lezcano (daniel.lezcano-GANU6spQydw@public.gmane.org):
>> > Krzysztof Taraszka wrote:
>> >> Hi,
>> >>
>> >> I was looking for possibility to secure lxc container to do not allow
>> 'root
>> >> container user'  from changing limits from cgroup. Right now without
>> STACK64
>> >> or SELinux he can do this easily.
>> >> I read the
>> http://www.ibm.com/developerworks/linux/library/l-lxc-security/cookbook
>> >> and decided to use STACK64 kernel mechanism.
>> >> Well... mounting cgroup inside container fails (great!, i am looked for
>> that
>> >> ;)) but networking fails too (interface bring up, sshd bring up,
>> connection
>> >> beetween host and container is, but 'mtr', 'ping' even 'apt-get update'
>> >> fails and I do not know why). I secure my container exactly like in the
>> >> cookbook.
>>
>> Yeah, smack's use of cipso can make things tricky, and it's possible
>> things
>> have changed a bit recently.  Although I'm currently running smack in my
>> everyday s390 kernel to test checkpointing of its labels, and networking
>> is working fine.
>
>
>> Can you give me a few details - what distro, smack policy, and precise
>> kernel
>> version are you using, for starters?
>>
>
> debian lenny amd64,
> kernel 2.6.30.5
> lxc-tools from git
>
> lxc1amd64:~# cat /etc/smackaccesses
> debian _ rwa
> _ debian rwa
> _ host rwax
> host _ rwax
>
> where "debian" is container, "host" is a host.
>
> I did this:
>
> for f in `find /root/rootfs.debian`; do
>     attr -S -s SMACK64 -V debian $f
> done
>
> on the container fs.
>
> container startup script look like here:
>
> lxc1amd64:~# cat vs1.sh
> #!/bin/sh
> cp /bin/dropmacadmin /root/rootfs.debian/bin/
> attr -S -s SMACK64 -V debian /root/rootfs.debian/bin/dropmacadmin
> echo debian > /proc/self/attr/current
> lxc-start -n debian /bin/dropmacadmin /sbin/init
>
> /etc/fstab inside container look like:
>
> debian:~# cat /etc/fstab
> tmpfs  /dev/shm   tmpfs  defaults,smackfsroot=debian,smackfsdef=debian  0 0
>
> And here is some output when I tried to do ping to the wp.pl, tried to
> apt-get update and tried to ping gateway
>
> debian:~# ping wp.pl
> PING wp.pl (212.77.100.101) 56(84) bytes of data.
> From 10.177.128.1 icmp_seq=1 Parameter problem: pointer = 20
> From 10.177.128.1 icmp_seq=2 Parameter problem: pointer = 20
> ^C
> --- wp.pl ping statistics ---
> 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1001ms
>
> debian:~# apt-get update
> Err http://ftp.debian.org lenny Release.gpg
>   Could not connect to ftp.debian.org:80 (130.89.149.226). - connect (71
> Protocol error)
> Err http://ftp.debian.org lenny/main Translation-en_US
>   Could not connect to ftp.debian.org:80 (130.89.149.226). - connect (71
> Protocol error)
> Ign http://ftp.debian.org lenny Release
> Ign http://ftp.debian.org lenny/main Packages/DiffIndex
> Ign http://ftp.debian.org lenny/main Packages
> Err http://ftp.debian.org lenny/main Packages
>   Could not connect to ftp.debian.org:80 (130.89.149.226). - connect (71
> Protocol error)
> W: Failed to fetch http://ftp.debian.org/debian/dists/lenny/Release.gpg
> Could not connect to ftp.debian.org:80 (130.89.149.226). - connect (71
> Protocol error)
>
> W: Failed to fetch
> http://ftp.debian.org/debian/dists/lenny/main/i18n/Translation-en_US.gz
> Could not connect to ftp.debian.org:80 (130.89.149.226). - connect (71
> Protocol error)
>
> W: Failed to fetch
> http://ftp.debian.org/debian/dists/lenny/main/binary-amd64/Packages  Could
> not connect to ftp.debian.org:80 (130.89.149.226). - connect (71 Protocol
> error)
>
> E: Some index files failed to download, they have been ignored, or old ones
> used instead.
> debian:~# ping 192.168.1.1
> PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
> 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.085 ms
> unknown option 86
> 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.136 ms
> unknown option 86
> 64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.116 ms
> unknown option 86
> ^C
> --- 192.168.1.1 ping statistics ---
> 3 packets transmitted, 3 received, 0% packet loss, time 2005ms
> rtt min/avg/max/mdev = 0.085/0.112/0.136/0.022 ms
>
> did you changed your smack policy or you have the same as mine?
>
>

Oh, I forgot to add that smack-utils I got from here:

https://launchpad.net/~anthonywrather/+archive/ppa/+files/smack-util_0.2-0ubuntu0~ppa3.tar.gz

because this link won't work:
http://schaufler-ca.com/data/080616/smack-util-0.1.tar

-- 
Krzysztof Taraszka

Re: how to do not allow to mount /cgroup inside container?

[Serge E. Hallyn]

Quoting Krzysztof Taraszka (krzysztof.taraszka-S89nZTSLPHGGdvJs77BJ7Q@public.gmane.org):
> 2009/8/25 Serge E. Hallyn <serue-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
> 
> > Quoting Daniel Lezcano (daniel.lezcano-GANU6spQydw@public.gmane.org):
> > > Krzysztof Taraszka wrote:
> > >> Hi,
> > >>
> > >> I was looking for possibility to secure lxc container to do not allow
> > 'root
> > >> container user'  from changing limits from cgroup. Right now without
> > STACK64
> > >> or SELinux he can do this easily.
> > >> I read the
> > http://www.ibm.com/developerworks/linux/library/l-lxc-security/cookbook
> > >> and decided to use STACK64 kernel mechanism.
> > >> Well... mounting cgroup inside container fails (great!, i am looked for
> > that
> > >> ;)) but networking fails too (interface bring up, sshd bring up,
> > connection
> > >> beetween host and container is, but 'mtr', 'ping' even 'apt-get update'
> > >> fails and I do not know why). I secure my container exactly like in the
> > >> cookbook.
> >
> > Yeah, smack's use of cipso can make things tricky, and it's possible things
> > have changed a bit recently.  Although I'm currently running smack in my
> > everyday s390 kernel to test checkpointing of its labels, and networking
> > is working fine.
> 
> 
> > Can you give me a few details - what distro, smack policy, and precise
> > kernel
> > version are you using, for starters?
> >
> 
> debian lenny amd64,
> kernel 2.6.30.5
> lxc-tools from git
> 
> lxc1amd64:~# cat /etc/smackaccesses
> debian _ rwa
> _ debian rwa
> _ host rwax
> host _ rwax

Ok, I think what you want to do is use /smack/netlabel as shown around line
425 in linux-2.6/Documentation/Smack.txt.  I haven't played with it yet,
but will tomorrow if you don't get a chance.  So basically I think you should
be able to do:

echo 127.0.0.1 -CIPSO > /smack/netlabel
echo 0.0.0.0/0 @      > /smack/netlabel

to open up the network.

Does that work?

-serge

Re: how to do not allow to mount /cgroup inside container?

[Krzysztof Taraszka]

2009/8/25 Serge E. Hallyn <serue-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>

> Quoting Krzysztof Taraszka (krzysztof.taraszka-S89nZTSLPHGGdvJs77BJ7Q@public.gmane.org):
> > 2009/8/25 Serge E. Hallyn <serue-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
> >
> > > Quoting Daniel Lezcano (daniel.lezcano-GANU6spQydw@public.gmane.org):
> > > > Krzysztof Taraszka wrote:
> > > >> Hi,
> > > >>
> > > >> I was looking for possibility to secure lxc container to do not
> allow
> > > 'root
> > > >> container user'  from changing limits from cgroup. Right now without
> > > STACK64
> > > >> or SELinux he can do this easily.
> > > >> I read the
> > >
> http://www.ibm.com/developerworks/linux/library/l-lxc-security/cookbook
> > > >> and decided to use STACK64 kernel mechanism.
> > > >> Well... mounting cgroup inside container fails (great!, i am looked
> for
> > > that
> > > >> ;)) but networking fails too (interface bring up, sshd bring up,
> > > connection
> > > >> beetween host and container is, but 'mtr', 'ping' even 'apt-get
> update'
> > > >> fails and I do not know why). I secure my container exactly like in
> the
> > > >> cookbook.
> > >
> > > Yeah, smack's use of cipso can make things tricky, and it's possible
> things
> > > have changed a bit recently.  Although I'm currently running smack in
> my
> > > everyday s390 kernel to test checkpointing of its labels, and
> networking
> > > is working fine.
> >
> >
> > > Can you give me a few details - what distro, smack policy, and precise
> > > kernel
> > > version are you using, for starters?
> > >
> >
> > debian lenny amd64,
> > kernel 2.6.30.5
> > lxc-tools from git
> >
> > lxc1amd64:~# cat /etc/smackaccesses
> > debian _ rwa
> > _ debian rwa
> > _ host rwax
> > host _ rwax
>
> Ok, I think what you want to do is use /smack/netlabel as shown around line
> 425 in linux-2.6/Documentation/Smack.txt.  I haven't played with it yet,
> but will tomorrow if you don't get a chance.  So basically I think you
> should
> be able to do:
>
> echo 127.0.0.1 -CIPSO > /smack/netlabel
> echo 0.0.0.0/0 @      > /smack/netlabel
>
> to open up the network.
>
> Does that work?
>

Yep :))
Works.
Anyway - are you going to update your cookbook on the ibm webpage?

-- 
Krzysztof Taraszka

Re: how to do not allow to mount /cgroup inside container?

[Serge E. Hallyn]

Quoting Krzysztof Taraszka (krzysztof.taraszka-S89nZTSLPHGGdvJs77BJ7Q@public.gmane.org):
> 2009/8/25 Serge E. Hallyn <serue-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
> 
> > Quoting Krzysztof Taraszka (krzysztof.taraszka-S89nZTSLPHGGdvJs77BJ7Q@public.gmane.org):
> > > 2009/8/25 Serge E. Hallyn <serue-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
> > >
> > > > Quoting Daniel Lezcano (daniel.lezcano-GANU6spQydw@public.gmane.org):
> > > > > Krzysztof Taraszka wrote:
> > > > >> Hi,
> > > > >>
> > > > >> I was looking for possibility to secure lxc container to do not
> > allow
> > > > 'root
> > > > >> container user'  from changing limits from cgroup. Right now without
> > > > STACK64
> > > > >> or SELinux he can do this easily.
> > > > >> I read the
> > > >
> > http://www.ibm.com/developerworks/linux/library/l-lxc-security/cookbook
> > > > >> and decided to use STACK64 kernel mechanism.
> > > > >> Well... mounting cgroup inside container fails (great!, i am looked
> > for
> > > > that
> > > > >> ;)) but networking fails too (interface bring up, sshd bring up,
> > > > connection
> > > > >> beetween host and container is, but 'mtr', 'ping' even 'apt-get
> > update'
> > > > >> fails and I do not know why). I secure my container exactly like in
> > the
> > > > >> cookbook.
> > > >
> > > > Yeah, smack's use of cipso can make things tricky, and it's possible
> > things
> > > > have changed a bit recently.  Although I'm currently running smack in
> > my
> > > > everyday s390 kernel to test checkpointing of its labels, and
> > networking
> > > > is working fine.
> > >
> > >
> > > > Can you give me a few details - what distro, smack policy, and precise
> > > > kernel
> > > > version are you using, for starters?
> > > >
> > >
> > > debian lenny amd64,
> > > kernel 2.6.30.5
> > > lxc-tools from git
> > >
> > > lxc1amd64:~# cat /etc/smackaccesses
> > > debian _ rwa
> > > _ debian rwa
> > > _ host rwax
> > > host _ rwax
> >
> > Ok, I think what you want to do is use /smack/netlabel as shown around line
> > 425 in linux-2.6/Documentation/Smack.txt.  I haven't played with it yet,
> > but will tomorrow if you don't get a chance.  So basically I think you
> > should
> > be able to do:
> >
> > echo 127.0.0.1 -CIPSO > /smack/netlabel
> > echo 0.0.0.0/0 @      > /smack/netlabel
> >
> > to open up the network.
> >
> > Does that work?
> >
> 
> Yep :))
> Works.

excellent.

> Anyway - are you going to update your cookbook on the ibm webpage?

That's problematic, actually... there is no good process for that.  I've been
waiting for updates to an selinux paper for 2 or 3 years.  I was thinking I'd
post a comment at the bottom of the article instead.

thanks,
-serge