[kernel-hardening] [PATCH/RFC] user_ns: fix missing limiting of user_ns counts

[kernel-hardening] [PATCH/RFC] user_ns: fix missing limiting of user_ns counts

[Vasily Kulikov]

Currently there is completely no limiting in number of user namespaces
created by unprivileged users.  One can freely create thousands of
user_ns'es and exhaust kernel memory without even bumping in
RLIMIT_NPROC or similar.

Even more -- it allows user to overflow kernel stack theoretically
allowing user to overwrite some important kernel data.  The problem is
that free_user_ns() may also free its parent user_namespace recursively
calling free_user_ns().  As kernel stack is very limited, it leads to
kernel stack overflow.

The code needs several checks.  First, noone should be able to create
user_ns of arbitrary depth.  Besides kernel stack overflow one could
create too big depth to DoS processes belonging to other users by
forcing them to loop a long time in cap_capable called from some
ns_capable() (e.g. in case one does smth like "ls -R /proc").  Second,
non-privileged users must not be able to overlimit some count of
namespaces to not be able to exhaust kernel memory.

The included patch is a basic fix for both or them.  Both values are
hardcoded here to 100 max depth and 1000 max in total.  I'm not sure how
better to make them configurable.  Looks like it needs some sysctl value
like kernel.max_user_ns_per_user, but also something more configurable
like new rlimit'ish limit may be created for user_ns needs.  E.g. in
case root wants one user to contain hundreds of private containers
(container owner user), but he doesn't want anybody to fill the kernel
with hundreds of containers multiplied by number of system users (equals
to thousands).

I'm not sure how it is an approved way for user_ns.  Eric?

A related issue which is NOT FIXED HERE is limits for all resources
available for containerized pseudo roots.  E.g. I succeeded creating
thousands of veth network devices without problems by a non-root user,
there seems no limit in number of network devices.  I suspect it is
possible to setup routing and net_ns'es the way it will be very
time-consuming for kernel to handle IP packets inside of ksoftirq, which
is not counted as this user scheduler time.   I suppose the issue is not
veth-specific, almost all newly available for unprivileged users code
pathes are vulnerable to DoS attacks.

Signed-off-by: Vasily Kulikov <segoon@openwall.com>
-- 
 include/linux/sched.h   |    3 +++
 kernel/user_namespace.c |   26 ++++++++++++++++++++++++++
 2 files changed, 29 insertions(+)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 206bb08..479940e 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -706,6 +706,9 @@ struct user_struct {
 #ifdef CONFIG_EPOLL
 	atomic_long_t epoll_watches; /* The number of file descriptors currently watched */
 #endif
+#ifdef CONFIG_USER_NS
+	atomic_t user_namespaces; /* How many user_ns does this user created? */
+#endif
 #ifdef CONFIG_POSIX_MQUEUE
 	/* protected by mq_lock	*/
 	unsigned long mq_bytes;	/* How many bytes can be allocated to mqueue? */
diff --git a/kernel/user_namespace.c b/kernel/user_namespace.c
index 2b042c4..a52c4e8 100644
--- a/kernel/user_namespace.c
+++ b/kernel/user_namespace.c
@@ -45,6 +45,16 @@ static void set_cred_user_ns(struct cred *cred, struct user_namespace *user_ns)
 	cred->user_ns = user_ns;
 }
 
+static long get_user_ns_depth(struct user_namespace *ns)
+{
+	long depth;
+
+	for (depth = 1; ns != &init_user_ns; ns = ns->parent)
+		depth++;
+
+	return depth;
+}
+
 /*
  * Create a new user namespace, deriving the creator from the user in the
  * passed credentials, and replacing that user with the new root user for the
@@ -56,6 +66,7 @@ static void set_cred_user_ns(struct cred *cred, struct user_namespace *user_ns)
 int create_user_ns(struct cred *new)
 {
 	struct user_namespace *ns, *parent_ns = new->user_ns;
+	struct user_struct *user = current->cred->user;
 	kuid_t owner = new->euid;
 	kgid_t group = new->egid;
 	int ret;
@@ -68,6 +79,18 @@ int create_user_ns(struct cred *new)
 	    !kgid_has_mapping(parent_ns, group))
 		return -EPERM;
 
+	/* Too long user_ns chains, might overflow kernel stack on kref_put() */
+	if (get_user_ns_depth(parent_ns) > 100)
+		return -ENOMEM;
+
+	atomic_inc(&user->user_namespaces);
+	/* FIXME: probably it's better to configure the number
+	 *        instead of hardcoding 1000 */
+	if (atomic_read(&user->user_namespaces) > 1000) {
+		atomic_dec(&user->user_namespaces);
+		return -ENOMEM;
+	}
+
 	ns = kmem_cache_zalloc(user_ns_cachep, GFP_KERNEL);
 	if (!ns)
 		return -ENOMEM;
@@ -108,10 +131,13 @@ void free_user_ns(struct kref *kref)
 {
 	struct user_namespace *parent, *ns =
 		container_of(kref, struct user_namespace, kref);
+	struct user_struct *user = find_user(ns->owner);
 
 	parent = ns->parent;
 	proc_free_inum(ns->proc_inum);
 	kmem_cache_free(user_ns_cachep, ns);
+	if (user)
+		atomic_dec(&user->user_namespaces);
 	put_user_ns(parent);
 }
 EXPORT_SYMBOL(free_user_ns);

-- 
Vasily Kulikov
http://www.openwall.com - bringing security into open computing environments

[kernel-hardening] Re: [PATCH/RFC] user_ns: fix missing limiting of user_ns counts

[Al Viro]

On Fri, Dec 28, 2012 at 09:56:27PM +0400, Vasily Kulikov wrote:
> The included patch is a basic fix for both or them.  Both values are
> hardcoded here to 100 max depth and 1000 max in total.  I'm not sure how
> better to make them configurable.  Looks like it needs some sysctl value
> like kernel.max_user_ns_per_user, but also something more configurable
> like new rlimit'ish limit may be created for user_ns needs.  E.g. in
> case root wants one user to contain hundreds of private containers
> (container owner user), but he doesn't want anybody to fill the kernel
> with hundreds of containers multiplied by number of system users (equals
> to thousands).

I'm sorry, but this is not a solution.  Kernel is not x86-only; there are
architectures with far bigger minimal stack frame size.  E.g. on sparc64
every fucking stack frame is at least 176 bytes.  So your 100 calls deep
call chain will happily overflow the damn stack all by itself - kernel
stack on sparc64 is 16Kb total, including struct thread_info living there.

[kernel-hardening] Re: [PATCH/RFC] user_ns: fix missing limiting of user_ns counts

[Vasily Kulikov]

On Fri, Dec 28, 2012 at 18:43 +0000, Al Viro wrote:
> On Fri, Dec 28, 2012 at 09:56:27PM +0400, Vasily Kulikov wrote:
> > The included patch is a basic fix for both or them.  Both values are
> > hardcoded here to 100 max depth and 1000 max in total.  I'm not sure how
> > better to make them configurable.  Looks like it needs some sysctl value
> > like kernel.max_user_ns_per_user, but also something more configurable
> > like new rlimit'ish limit may be created for user_ns needs.  E.g. in
> > case root wants one user to contain hundreds of private containers
> > (container owner user), but he doesn't want anybody to fill the kernel
> > with hundreds of containers multiplied by number of system users (equals
> > to thousands).
> 
> I'm sorry, but this is not a solution.  Kernel is not x86-only; there are
> architectures with far bigger minimal stack frame size.  E.g. on sparc64
> every fucking stack frame is at least 176 bytes.  So your 100 calls deep
> call chain will happily overflow the damn stack all by itself - kernel
> stack on sparc64 is 16Kb total, including struct thread_info living there.

Understood.  How to properly fix it then?  Looks like there are quite
many kernel structures which may reference other structures which
indirectly reference each other via kref, IOW it is not user_ns specific
issue.  With unprivileged user_ns the way it should be freed must be
somehow changed.

Thanks,

-- 
Vasily Kulikov
http://www.openwall.com - bringing security into open computing environments

[kernel-hardening] Re: [PATCH/RFC] user_ns: fix missing limiting of user_ns counts

[Al Viro]

On Fri, Dec 28, 2012 at 11:04:35PM +0400, Vasily Kulikov wrote:

> > I'm sorry, but this is not a solution.  Kernel is not x86-only; there are
> > architectures with far bigger minimal stack frame size.  E.g. on sparc64
> > every fucking stack frame is at least 176 bytes.  So your 100 calls deep
> > call chain will happily overflow the damn stack all by itself - kernel
> > stack on sparc64 is 16Kb total, including struct thread_info living there.
> 
> Understood.  How to properly fix it then?  Looks like there are quite
> many kernel structures which may reference other structures which
> indirectly reference each other via kref, IOW it is not user_ns specific
> issue.  With unprivileged user_ns the way it should be freed must be
> somehow changed.

	There are many damn good reasons why kref should *not* be used without
thinking.  It's been oversold as easy solution to all refcounting problems;
it isn't one.  Don't use it here.

[kernel-hardening] Re: [PATCH/RFC] user_ns: fix missing limiting of user_ns counts

[Eric W. Biederman]

Vasily Kulikov <segoon@openwall.com> writes:

> Currently there is completely no limiting in number of user namespaces
> created by unprivileged users.  One can freely create thousands of
> user_ns'es and exhaust kernel memory without even bumping in
> RLIMIT_NPROC or similar.

First for a proper sense of scale it will take roughly 14,000 to consume
a megabyte.  So it will take hundreds of millions of user namespaces to
eat up all of kernel memory.

That said I have no objects to a patch that implemnts sysctls for
maximum limits.

> Even more -- it allows user to overflow kernel stack theoretically
> allowing user to overwrite some important kernel data.  The problem is
> that free_user_ns() may also free its parent user_namespace recursively
> calling free_user_ns().  As kernel stack is very limited, it leads to
> kernel stack overflow.

Yes.  Gcc can't turn a tail call into a jump in even the most basic
cases apparently.  So we need to adopt the solution of the pid
namespace.  Patch to follow shortly.

> The code needs several checks.  First, noone should be able to create
> user_ns of arbitrary depth.  Besides kernel stack overflow one could
> create too big depth to DoS processes belonging to other users by
> forcing them to loop a long time in cap_capable called from some
> ns_capable() (e.g. in case one does smth like "ls -R /proc").

Where do you get a ns_capable call from "ls -R /proc" ?

> Second,
> non-privileged users must not be able to overlimit some count of
> namespaces to not be able to exhaust kernel memory.

> The included patch is a basic fix for both or them.  Both values are
> hardcoded here to 100 max depth and 1000 max in total.  I'm not sure how
> better to make them configurable.  Looks like it needs some sysctl value
> like kernel.max_user_ns_per_user, but also something more configurable
> like new rlimit'ish limit may be created for user_ns needs.  E.g. in
> case root wants one user to contain hundreds of private containers
> (container owner user), but he doesn't want anybody to fill the kernel
> with hundreds of containers multiplied by number of system users (equals
> to thousands).
>
> I'm not sure how it is an approved way for user_ns.  Eric?

An per user limit for user namespaces is pretty much useless, as it is
expected that many user namespaces will be allocated multiple uids to play
with.  My current target is to modify newuser allocate 10,000 uids for
each user by default.

Other than a global limit the recommended solution is some kind of
control group.

With that said I am starting to think there may be a good argument for
per userns limits that apply to a user namespace and all of it's
children.  But for that to really make sense requires showing that
control groups can't do the job well.   I think there might be a
reasonable argument there.

> A related issue which is NOT FIXED HERE is limits for all resources
> available for containerized pseudo roots.  E.g. I succeeded creating
> thousands of veth network devices without problems by a non-root user,
> there seems no limit in number of network devices.  I suspect it is
> possible to setup routing and net_ns'es the way it will be very
> time-consuming for kernel to handle IP packets inside of ksoftirq, which
> is not counted as this user scheduler time.   I suppose the issue is not
> veth-specific, almost all newly available for unprivileged users code
> pathes are vulnerable to DoS attacks.

veth at least should process packets synchronously so I don't see how
you will get softirq action.  There is also for whatever it is worth
the network memory control group, that should limit networking things.
I haven't had a chance to look how sane it is in practice.

> Signed-off-by: Vasily Kulikov <segoon@openwall.com>
> -- 
>  include/linux/sched.h   |    3 +++
>  kernel/user_namespace.c |   26 ++++++++++++++++++++++++++
>  2 files changed, 29 insertions(+)
>
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 206bb08..479940e 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -706,6 +706,9 @@ struct user_struct {
>  #ifdef CONFIG_EPOLL
>  	atomic_long_t epoll_watches; /* The number of file descriptors currently watched */
>  #endif
> +#ifdef CONFIG_USER_NS
> +	atomic_t user_namespaces; /* How many user_ns does this user created? */
> +#endif
>  #ifdef CONFIG_POSIX_MQUEUE
>  	/* protected by mq_lock	*/
>  	unsigned long mq_bytes;	/* How many bytes can be allocated to mqueue? */
> diff --git a/kernel/user_namespace.c b/kernel/user_namespace.c
> index 2b042c4..a52c4e8 100644
> --- a/kernel/user_namespace.c
> +++ b/kernel/user_namespace.c
> @@ -45,6 +45,16 @@ static void set_cred_user_ns(struct cred *cred, struct user_namespace *user_ns)
>  	cred->user_ns = user_ns;
>  }
>  
> +static long get_user_ns_depth(struct user_namespace *ns)
> +{
> +	long depth;
> +
> +	for (depth = 1; ns != &init_user_ns; ns = ns->parent)
> +		depth++;
> +
> +	return depth;
> +}
> +
>  /*
>   * Create a new user namespace, deriving the creator from the user in the
>   * passed credentials, and replacing that user with the new root user for the
> @@ -56,6 +66,7 @@ static void set_cred_user_ns(struct cred *cred, struct user_namespace *user_ns)
>  int create_user_ns(struct cred *new)
>  {
>  	struct user_namespace *ns, *parent_ns = new->user_ns;
> +	struct user_struct *user = current->cred->user;
>  	kuid_t owner = new->euid;
>  	kgid_t group = new->egid;
>  	int ret;
> @@ -68,6 +79,18 @@ int create_user_ns(struct cred *new)
>  	    !kgid_has_mapping(parent_ns, group))
>  		return -EPERM;
>  
> +	/* Too long user_ns chains, might overflow kernel stack on kref_put() */
> +	if (get_user_ns_depth(parent_ns) > 100)
> +		return -ENOMEM;
> +
> +	atomic_inc(&user->user_namespaces);
> +	/* FIXME: probably it's better to configure the number
> +	 *        instead of hardcoding 1000 */
> +	if (atomic_read(&user->user_namespaces) > 1000) {
> +		atomic_dec(&user->user_namespaces);
> +		return -ENOMEM;
> +	}
> +
>  	ns = kmem_cache_zalloc(user_ns_cachep, GFP_KERNEL);
>  	if (!ns)
>  		return -ENOMEM;
> @@ -108,10 +131,13 @@ void free_user_ns(struct kref *kref)
>  {
>  	struct user_namespace *parent, *ns =
>  		container_of(kref, struct user_namespace, kref);
> +	struct user_struct *user = find_user(ns->owner);
>  
>  	parent = ns->parent;
>  	proc_free_inum(ns->proc_inum);
>  	kmem_cache_free(user_ns_cachep, ns);
> +	if (user)
> +		atomic_dec(&user->user_namespaces);
>  	put_user_ns(parent);
>  }
>  EXPORT_SYMBOL(free_user_ns);

[kernel-hardening] Re: [PATCH/RFC] user_ns: fix missing limiting of user_ns counts

[Al Viro]

On Fri, Dec 28, 2012 at 08:05:32PM -0800, Eric W. Biederman wrote:

> Yes.  Gcc can't turn a tail call into a jump in even the most basic
> cases apparently.

What.  The.  Fuck?

You have introduced unlimited recursion on kernel stack.  OK, it's
unpleasant, but it can happen to anybody.  But then you have the gall
to complain about gcc optimizations not saving your broken code.  Are
you serious?

It's not a question of gcc being not smart enough, and while we are
at it, it's *not* a basic case at all - we have recursion going through
the callback and we have non-trivial return value on top of that; if
the caller of kref_put() had checked said return value, you would've
been unable to turn that into a loop at all.

Are you seriously saying that you relied on compiler being smart enough
to
	* notice that return value of kref_put() is ignored
	* notice that therefore the return value of kref_sub() is ignored
	* notice that you have this call of put_user_ns() calling
kref_put() calling kref_sub() inside the callback we'd passed through the
last two levels of calls and that we are dealing with the tail recursion
here
... so that your code wouldn't have stepped into unlimited recursion?
I sincerely hope that you are not that much of an idiot.  If nothing else,
even if gcc did spot that one, the result would've been extremely brittle -
minor change to kref.h several year down the road and we'd get the problem.

[kernel-hardening] Re: [PATCH/RFC] user_ns: fix missing limiting of user_ns counts

[Vasily Kulikov]

On Fri, Dec 28, 2012 at 20:05 -0800, Eric W. Biederman wrote:
> Vasily Kulikov <segoon@openwall.com> writes:
> 
> > Currently there is completely no limiting in number of user namespaces
> > created by unprivileged users.  One can freely create thousands of
> > user_ns'es and exhaust kernel memory without even bumping in
> > RLIMIT_NPROC or similar.
> 
> First for a proper sense of scale it will take roughly 14,000 to consume
> a megabyte.  So it will take hundreds of millions of user namespaces to
> eat up all of kernel memory.

Yes, but you can freely create *any* number of nested userns by a loop:

    for() {
        unshare()
        write to /proc/self/{u,g}id_map
    }

> > The code needs several checks.  First, noone should be able to create
> > user_ns of arbitrary depth.  Besides kernel stack overflow one could
> > create too big depth to DoS processes belonging to other users by
> > forcing them to loop a long time in cap_capable called from some
> > ns_capable() (e.g. in case one does smth like "ls -R /proc").
> 
> Where do you get a ns_capable call from "ls -R /proc" ?

E.g. if procfs is mounted with hidepid=2 then ls does
ptrace_may_access() check. 

Thanks,

-- 
Vasily Kulikov
http://www.openwall.com - bringing security into open computing environments

[kernel-hardening] Re: [PATCH/RFC] user_ns: fix missing limiting of user_ns counts

[Vasily Kulikov]

On Fri, Dec 28, 2012 at 20:05 -0800, Eric W. Biederman wrote:
> > A related issue which is NOT FIXED HERE is limits for all resources
> > available for containerized pseudo roots.  E.g. I succeeded creating
> > thousands of veth network devices without problems by a non-root user,
> > there seems no limit in number of network devices.  I suspect it is
> > possible to setup routing and net_ns'es the way it will be very
> > time-consuming for kernel to handle IP packets inside of ksoftirq, which
> > is not counted as this user scheduler time.   I suppose the issue is not
> > veth-specific, almost all newly available for unprivileged users code
> > pathes are vulnerable to DoS attacks.
> 
> veth at least should process packets synchronously so I don't see how
> you will get softirq action.

What do you mean -- synchronously?  From my limited understanding of
veth job, it is handled like every network packet in system, via:

    veth_xmit() -> dev_forward_skb() -> netif_rx() -> enqueue_to_backlog()

enqueue_to_backlog() adds the packet to softnet_data->input_pkt_queue. 

Then inside of softirq process_backlog() moves ->input_pkt_queue to
->process_queue and calls __netif_receive_skb(), which does all networking
stack magic.

AFAICS, one could create user_ns, net_ns inside of it, and setup routing
tables and netfilter to infinitely pass few network packets from and to
veth, abusing ksoftirq.

-- 
Vasily Kulikov
http://www.openwall.com - bringing security into open computing environments