prevent breaking a chroot() jail?

prevent breaking a chroot() jail?

[Shaya Potter]

I'm trying to develop a way to ensure that one can't break out of a
chroot() jail, even as root.  I'm willing to change the way the syscalls
work (most likely only for a subset of processes, i.e. processes that
are run in the jail end up getting a marker which is passed down to all
their children that causes the syscalls to behave differently).

What should I be aware of?  I figure devices (no need to run mknod in
this jail) and chroot (as per man page), is there any other way of
breaking the chroot jail (at a syscall level or otherwise)?

or is this 100% impossible?

thanks,

shaya

Re: prevent breaking a chroot() jail?

[Miquel van Smoorenburg]

In article <1025877004.11004.59.camel@zaphod>,
Shaya Potter  <spotter@cs.columbia.edu> wrote:
>I'm trying to develop a way to ensure that one can't break out of a
>chroot() jail, even as root.  I'm willing to change the way the syscalls
>work (most likely only for a subset of processes, i.e. processes that
>are run in the jail end up getting a marker which is passed down to all
>their children that causes the syscalls to behave differently).
>What should I be aware of?  I figure devices (no need to run mknod in
>this jail) and chroot (as per man page), is there any other way of
>breaking the chroot jail (at a syscall level or otherwise)?

int main()
{
	chdir("/");
	mkdir("foo");
	chroot("foo");
	chdir("../../../../../../..");
	chroot(".");
	execl("/bin/sh", "sh", NULL);
}

Run as root and you're out of the chroot jail. This is because
chroot() doesn't chdir() to the new root, so after a chroot() in
the chroot jail you're suddenly out of it.

Mike.

Re: prevent breaking a chroot() jail?

[Shaya Potter]

On Fri, 2002-07-05 at 10:02, Miquel van Smoorenburg wrote:
> In article <1025877004.11004.59.camel@zaphod>,
> Shaya Potter  <spotter@cs.columbia.edu> wrote:
> >I'm trying to develop a way to ensure that one can't break out of a
> >chroot() jail, even as root.  I'm willing to change the way the syscalls
> >work (most likely only for a subset of processes, i.e. processes that
> >are run in the jail end up getting a marker which is passed down to all
> >their children that causes the syscalls to behave differently).
> >What should I be aware of?  I figure devices (no need to run mknod in
> >this jail) and chroot (as per man page), is there any other way of
> >breaking the chroot jail (at a syscall level or otherwise)?
> 
> int main()
> {
> 	chdir("/");
> 	mkdir("foo");
> 	chroot("foo");
> 	chdir("../../../../../../..");
> 	chroot(".");
> 	execl("/bin/sh", "sh", NULL);
> }
> 
> Run as root and you're out of the chroot jail. This is because
> chroot() doesn't chdir() to the new root, so after a chroot() in
> the chroot jail you're suddenly out of it.

yes, that's what the man page says.  Is that the only hole? i.e. if one
changed the semantics of chroot() to also do a chdir() to the new root,
would that be fixed? (not arguing on changing this for everything, just
for something specific)

thanks,

shaya

Re: prevent breaking a chroot() jail?

[Jesse Pollard]

Shaya Potter <spotter@cs.columbia.edu>:
> 
> On Fri, 2002-07-05 at 10:02, Miquel van Smoorenburg wrote:
> > In article <1025877004.11004.59.camel@zaphod>,
> > Shaya Potter  <spotter@cs.columbia.edu> wrote:
> > >I'm trying to develop a way to ensure that one can't break out of a
> > >chroot() jail, even as root.  I'm willing to change the way the syscalls
> > >work (most likely only for a subset of processes, i.e. processes that
> > >are run in the jail end up getting a marker which is passed down to all
> > >their children that causes the syscalls to behave differently).
> > >What should I be aware of?  I figure devices (no need to run mknod in
> > >this jail) and chroot (as per man page), is there any other way of
> > >breaking the chroot jail (at a syscall level or otherwise)?
> > 
> > int main()
> > {
> > 	chdir("/");
> > 	mkdir("foo");
> > 	chroot("foo");
> > 	chdir("../../../../../../..");
> > 	chroot(".");
> > 	execl("/bin/sh", "sh", NULL);
> > }
> > 
> > Run as root and you're out of the chroot jail. This is because
> > chroot() doesn't chdir() to the new root, so after a chroot() in
> > the chroot jail you're suddenly out of it.

Usually, I see the above sequence more like:
	...
	chdir("/");
	mkdir("foo");
	chdir("/foo");
	chroot("/foo");
	...
Though this doesn't necessarily change the situation.

> yes, that's what the man page says.  Is that the only hole? i.e. if one
> changed the semantics of chroot() to also do a chdir() to the new root,
> would that be fixed? (not arguing on changing this for everything, just
> for something specific)

sure, but it depends on what you are trying to prevent... all of these assume
root, and are only possiblilities, not necessarily realities.

signal is one way to bring the system down (as is reboot).

swapon is a way to get to other processes memory (create a swap file, then
start swapping to it). This one is difficult to implement effectively.

ptrace may allow access to other processes (not sure - see PTRACE_ATTACH)

Modify the shared libraries (in memory) and you modify all processes that
are using it (yes it is read only, but root may be able to change the memory
tables). I havent tried this one, and it may not be possible

Use open/read/write/close to get the inode of the real parent directory,
then the program creates a new file entry (bypassing mkdir... use read/write)
to create a subdirectory link to then cd into. Yes it is an invalid entry.

mmap may allow access to devices or the IO memory mapped area.

use iopl/ioperm to gain access to the I/O ports for devices.

mount may allow a loopback nfs mount (not tested either). This would only
grant access to NFS filesystems exported globally, and would fail for those
with explicit exports (host specified in the exports). It may also allow
the mounting of /proc /devfs (or /devices) which would give access to
everything.

There is no good way to trap root. You really need to move to something
more secure.. such as a non priviledged root user (capability based, or
LSM/SELinux based) and simply remove the privileges from the process before/as
it is handed over to the chroot "jail".
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: prevent breaking a chroot() jail?

[David Wagner]

Shaya Potter  wrote:
>if one changed the semantics of chroot() to also do a chdir() to the new root,
>would that be fixed?

There are lots of other ways for root to escape from a chroot jail.
For example: Take control of another process using ptrace, mknod a new
device, use the loopback filesystem to create new devices, mount /proc and
modify /dev/kmem, bind to privileged ports and escape over the network,
kill servers and re-bind to their addresses, mount a new filesystem,
bang on IPC or shared memory, load a loadable kernel module, directly
access hardware with iopl(), and so on.

Moreover, note that chrooted root processes can do lots of other harm:
kill all the rest of the processes on the box, set the time of day,
set the local hostname, reboot your machine, use sysctl to turn on
some dangerous option (e.g., IP forwarding), renice other processes,
and the like.  This is probably not so good, either.

Chroot is a lot better than nothing, but it doesn't provide a
secure jail, especially not for root.  However, the following
tools are intended to provide a secure jail, and may be of interest
to you: SubDomain (http://www.immunix.org/subdomain.html), Janus
(http://www.cs.berkeley.edu/~daw/janus/), and BSD's jail() system call
come to mind.  Also, may I point you to the Linux Security Modules project
(http://lsm.immunix.org/)?  I think you may find it of interest.

Re: prevent breaking a chroot() jail?

[Martin Josefsson]

On Fri, 2002-07-05 at 23:00, David Wagner wrote:

> Chroot is a lot better than nothing, but it doesn't provide a
> secure jail, especially not for root.  However, the following
> tools are intended to provide a secure jail, and may be of interest
> to you: SubDomain (http://www.immunix.org/subdomain.html), Janus
> (http://www.cs.berkeley.edu/~daw/janus/), and BSD's jail() system call
> come to mind.  Also, may I point you to the Linux Security Modules project
> (http://lsm.immunix.org/)?  I think you may find it of interest.

I havn't seen vserver mentioned in this thread.

http://www.solucorp.qc.ca/miscprj/s_context.hc

It disables a lot of capabilities (configurable) and other stuff.
Worth taking a look at.

-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

Re: prevent breaking a chroot() jail?

[Thunder from the hill]

Hi,

On 5 Jul 2002, Shaya Potter wrote:
> What should I be aware of?  I figure devices (no need to run mknod in
> this jail) and chroot (as per man page), is there any other way of
> breaking the chroot jail (at a syscall level or otherwise)?
> 
> or is this 100% impossible?

Well, since we're talking about root:

 - If you had saved the old root before chroot()ing, use that one.
 - If you have your whole disk exported via NFS, the prisoner process 
   could use nfs to read files outside the jail
 - If you have access to a /dev directory, use /dev/sd?? to do the disc
   access
 - If not, use mknod("dideldei", 600, {68,1}); open("dideldei", O_SYNC);
   and do as you like.

However, if you aren't running anything you find as root, it's relatively 
secure.

							Regards,
							Thunder
-- 
(Use http://www.ebb.org/ungeek if you can't decode)
------BEGIN GEEK CODE BLOCK------
Version: 3.12
GCS/E/G/S/AT d- s++:-- a? C++$ ULAVHI++++$ P++$ L++++(+++++)$ E W-$
N--- o?  K? w-- O- M V$ PS+ PE- Y- PGP+ t+ 5+ X+ R- !tv b++ DI? !D G
e++++ h* r--- y- 
------END GEEK CODE BLOCK------

Re: prevent breaking a chroot() jail?

[Vance Lankhaar]

Take a look at the GR Security patch. It has a whole bunch of chroot
restrictions, and might just do exactly what you want.
http://www.grsecurity.net

Vance

On Fri, 2002-07-05 at 06:50, Shaya Potter wrote:
> I'm trying to develop a way to ensure that one can't break out of a
> chroot() jail, even as root.  I'm willing to change the way the syscalls
> work (most likely only for a subset of processes, i.e. processes that
> are run in the jail end up getting a marker which is passed down to all
> their children that causes the syscalls to behave differently).
> 
> What should I be aware of?  I figure devices (no need to run mknod in
> this jail) and chroot (as per man page), is there any other way of
> breaking the chroot jail (at a syscall level or otherwise)?
> 
> or is this 100% impossible?
> 
> thanks,
> 
> shaya

-- 
------------------------------------------------------------
 Vance Lankhaar                        vance@pcsscreston.ca
 PCSS Yearbook                      yearbook@pcsscreston.ca
 PCSS Computers                    sysadmins@pcsscreston.ca
 http://www.crestonbc.com/pcss/   http://www.pcsscreston.ca
------------------------------------------------------------

Re: prevent breaking a chroot() jail?

[H. Peter Anvin]

Followup to:  <1025877004.11004.59.camel@zaphod>
By author:    Shaya Potter <spotter@cs.columbia.edu>
In newsgroup: linux.dev.kernel
>
> I'm trying to develop a way to ensure that one can't break out of a
> chroot() jail, even as root.  I'm willing to change the way the syscalls
> work (most likely only for a subset of processes, i.e. processes that
> are run in the jail end up getting a marker which is passed down to all
> their children that causes the syscalls to behave differently).
> 
> What should I be aware of?  I figure devices (no need to run mknod in
> this jail) and chroot (as per man page), is there any other way of
> breaking the chroot jail (at a syscall level or otherwise)?
> 
> or is this 100% impossible?
> 

This sounds like a job for [dum de dum dum] capabilities... remember,
on Linux root hasn't been almighty for a very long time, it's just a
matter of which capabilities you retain.  Of course, if you really
want to be safe, you might end up with a rather castrated root inside
the chroot shell.

If you really want to jail something, use UML.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: prevent breaking a chroot() jail?

[Ville Herva]

On Fri, Jul 05, 2002 at 11:15:39AM -0700, you [H. Peter Anvin] wrote:
> 
> This sounds like a job for [dum de dum dum] capabilities... remember,
> on Linux root hasn't been almighty for a very long time, it's just a
> matter of which capabilities you retain.  Of course, if you really
> want to be safe, you might end up with a rather castrated root inside
> the chroot shell.
> 
> If you really want to jail something, use UML.

ISTR UML had some security problems (guest processes being able to disrupt
host processes or just guest processes being able to disrupt other guest
processes). Have those been resolved yet? 

Do people use it in production? Last I heard someone had evaluated it, it
had ended up consuming way too much CPU per "jail" for whatever reason.
Perhaps things are better already...


-- v --

v@iki.fi

Re: prevent breaking a chroot() jail?

[Alan Cox]

> ISTR UML had some security problems (guest processes being able to disrupt
> host processes or just guest processes being able to disrupt other guest
> processes). Have those been resolved yet? 

Yes

> Do people use it in production? Last I heard someone had evaluated it, it
> had ended up consuming way too much CPU per "jail" for whatever reason.
> Perhaps things are better already...

It needs some work, and probably ultimately a couple of assists to
do sigaltmm and VM style pagex

Re: prevent breaking a chroot() jail?

[Jeff Dike]

vherva@niksula.hut.fi said:
> ISTR UML had some security problems (guest processes being able to
> disrupt host processes or just guest processes being able to disrupt
> other guest processes). Have those been resolved yet?  

Can you be more specific?  That's not ringing any bells with me.

As far as I know, there's nothing that needs to be resolved.

				Jeff

Re: prevent breaking a chroot() jail?

[Ville Herva]

On Fri, Jul 05, 2002 at 04:48:43PM -0500, you [Jeff Dike] wrote:
> vherva@niksula.hut.fi said:
> > ISTR UML had some security problems (guest processes being able to
> > disrupt host processes or just guest processes being able to disrupt
> > other guest processes). Have those been resolved yet?  
> 
> Can you be more specific?  That's not ringing any bells with me.

Sorry, I should've searched the archives before asking. It was just a thing
I had read somewhere, quite a while ago. Can't find the exact reference now,
but google does give a couple of hits:

http://online.securityfocus.com/bid/3973/discussion/

I just hadn't heard these having been addressed, but that's most likely just
me not following closely enough.

> As far as I know, there's nothing that needs to be resolved.

Glad to hear. I'll try it next time I'll need to jail something.


-- v --

v@iki.fi

Re: prevent breaking a chroot() jail?

[Alan Cox]

> work (most likely only for a subset of processes, i.e. processes that
> are run in the jail end up getting a marker which is passed down to all
> their children that causes the syscalls to behave differently).
> 
> What should I be aware of?  I figure devices (no need to run mknod in
> this jail) and chroot (as per man page), is there any other way of
> breaking the chroot jail (at a syscall level or otherwise)?
> 
> or is this 100% impossible?

Its hairy. You need to avoid devices, root, fd passing from untrusted agents
outside the chroot world and shared uid (ptrace) attacks. If you are outside the
USA you can do this with NSA SE Linux although in the US you may hit the
type enforcement patents.

Alan

Re: prevent breaking a chroot() jail?

[Pavel Machek]

Hi!

> I'm trying to develop a way to ensure that one can't break out of a
> chroot() jail, even as root.  I'm willing to change the way the syscalls
> work (most likely only for a subset of processes, i.e. processes that
> are run in the jail end up getting a marker which is passed down to all
> their children that causes the syscalls to behave differently).
> 
> What should I be aware of?  I figure devices (no need to run mknod in
> this jail) and chroot (as per man page), is there any other way of
> breaking the chroot jail (at a syscall level or otherwise)?

subterfugue.sf.net, or UML. Its hard to do with chroot(). Think
kill(-9, -1).
								Pavel
-- 
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?

Re: prevent breaking a chroot() jail?

[Hank Leininger]

On 2002-07-05, Shaya Potter <spotter@cs.columbia.edu> wrote:

> On Fri, 2002-07-05 at 10:02, Miquel van Smoorenburg wrote:
> > In article <1025877004.11004.59.camel@zaphod>,
> > Shaya Potter  <spotter@cs.columbia.edu> wrote:
> > > I'm trying to develop a way to ensure that one can't break out of a
> > > chroot() jail, even as root.  I'm willing to change the way the
[snip]
> > Run as root and you're out of the chroot jail. This is because
> > chroot() doesn't chdir() to the new root, so after a chroot() in
> > the chroot jail you're suddenly out of it.

> yes, that's what the man page says.  Is that the only hole? i.e. if one
> changed the semantics of chroot() to also do a chdir() to the new root,
> would that be fixed? (not arguing on changing this for everything, just
> for something specific)

No, there are many ways that root can break out of chroot(2).  I maintain
some patches[1] against 2.2 (and grsecurity[2] has ported most of them to
2.4) which aim to try to make it harder for root to break out of chroot(2),
but I won't say I've got them all--in fact I'll say I'm sure I *don't* have
them all, and I'd like to hear suggestions for more.  Here are some things
to worry about:

-chroot(2)'ing with an open directory fd
-prevent chroot(2) by a process already chrooted ("double-chroot")
-block mount(2) attempts inside chroot ("chroot(../..)" ...)
-block mknod of char or block devices inside chroot ("mknod /dev/hda",
   "mknod /dev/kmem")
-block chmod +s by a chrooted process
-block ptrace(2) by a chrooted process of processes outside the jail
-block most signals by a chrooted process to processes outside the jail
-block setting capabilities (capset) by a chrooted process of processes 
   outside the jail
-drop "dangerous" capabilities when chroot(2)'ing.  (See the patch, but
   basically, various *_ADMIN, *RAW*, etc to block ioctl, sysctl for
   dangerous things.)

One area I have not looked at sufficiently is sysv IPC (shared memory,
semaphores...).  It's quite possible that a chrooted process can tamper
with shared memory segments that other, outside-chroot processes are using
(especially if some app is designed to use them to communicate across the
chroot boundary; I don't know of any but they could exist) and use that
vector to attack and try to subvert the other, non-chrooted process(es).

I'd appreciate any suggestions in addition to the above, and/or holes poked
in the implementation (which I'm sure isn't the best...).

[1] http://www.theaimsgroup.com/~hlein/hap-linux/
[2] http://www.grsecurity.org/

Thanks,

--
Hank Leininger <hlein@progressive-comp.com> 

Re: prevent breaking a chroot() jail?

[Ville Herva]

On Fri, Jul 05, 2002 at 11:16:34AM -0400, you [Hank Leininger] wrote:
>
> No, there are many ways that root can break out of chroot(2).  I maintain
> some patches[1] against 2.2 (and grsecurity[2] has ported most of them to
> 2.4) which aim to try to make it harder for root to break out of chroot(2),
> but I won't say I've got them all--in fact I'll say I'm sure I *don't* have
> them all, and I'd like to hear suggestions for more.  Here are some things
> to worry about:

I was skimming through FreeBSD jail(2) documents
(http://docs.freebsd.org/44doc/papers/jail/jail.html). 

Compared to jail
(http://docs.freebsd.org/44doc/papers/jail/jail-5.html#section5):
 
> -chroot(2)'ing with an open directory fd
> -prevent chroot(2) by a process already chrooted ("double-chroot")

Jail thwarts these.

> -block mount(2) attempts inside chroot ("chroot(../..)" ...)
> -block mknod of char or block devices inside chroot ("mknod /dev/hda",
>    "mknod /dev/kmem")

Also prohibited in jail.

> -block chmod +s by a chrooted process

Jail appears to allow this, and you can't get out of jail as (jailed) root
anyway.

> -block ptrace(2) by a chrooted process of processes outside the jail

I believe jail prohibits this as well (through its p_trespass() mechanism).

> -block most signals by a chrooted process to processes outside the jail

Likewise - it blocks all signals in and out from jail.

> -block setting capabilities (capset) by a chrooted process of processes 
>    outside the jail

(No idea)

> -drop "dangerous" capabilities when chroot(2)'ing.  (See the patch, but
>    basically, various *_ADMIN, *RAW*, etc to block ioctl, sysctl for
>    dangerous things.)

Jail takes care of this by only allowing 35 operations for jailed root (out
of the 260 allowed for normal root). Capabilies are propably better in linux
context.

> One area I have not looked at sufficiently is sysv IPC (shared memory,
> semaphores...).  It's quite possible that a chrooted process can tamper
> with shared memory segments that other, outside-chroot processes are using
> (especially if some app is designed to use them to communicate across the
> chroot boundary; I don't know of any but they could exist) and use that
> vector to attack and try to subvert the other, non-chrooted process(es).

I would imagine the p_trespass() check is used in FreeBSD to disallow any
memory sharing between processes that are not in the same jail.

In addition to what has been mentioned above, jail(2) notably limits jailed
processes to one ip number (that of jail's). That way the jailed processes
can't connect to hosts services (even through localhost interface) unless
they listen to jail's ip, nor bind to any ip but the one that has been
granted to the jail. They also do not allow any ipconfig, routing or kernel
parameter changes etc from within a jail.
 
In general, I wonder if it would make sense to aim for something like
jail(2). Chroot has its shortcomings, and I take it that many of them have
to be preserved to maintain standard compliance. Jail isolates processes
more completely than chroot is ever ment to.

FreeBSD implements jail by adding a jail pointer to struct proc - I'm not
sure how much of that should/could be done with mere capabilities in linux,
and how much of the "fortificated chroot" implementation jail has overlaps
with Al Viro's namespaces.

All in all, I've seen suprisingly little conversation about jail on
lkml. What do people think of it?


-- v --

v@iki.fi

Re: prevent breaking a chroot() jail?

[Bill Davidsen]

On Fri, 5 Jul 2002, Ville Herva wrote:

> In general, I wonder if it would make sense to aim for something like
> jail(2). Chroot has its shortcomings, and I take it that many of them have
> to be preserved to maintain standard compliance. Jail isolates processes
> more completely than chroot is ever ment to.
> 
> FreeBSD implements jail by adding a jail pointer to struct proc - I'm not
> sure how much of that should/could be done with mere capabilities in linux,
> and how much of the "fortificated chroot" implementation jail has overlaps
> with Al Viro's namespaces.
> 
> All in all, I've seen suprisingly little conversation about jail on
> lkml. What do people think of it?

Not having had the problem of jailing things for almost a decade, I
haven't been following this field, but certainly after looking at POSIX
and chroot, I don't think we can both be secure and compliant. Adding
jail() would be a better approach, preferably with a BSD-compatible API so
people could move programs here.

I have been doing some reading on UML, and it sounds as if that could be
at least as secure as jail, but it is almost certainly higher overhead. I
want to try running multiple machines in UML and see how well it works. If
disk is the only issue it's cheap enough to to have enough per
application, but I doubt if it's practical per-user in most cases.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.