[SECURITY] Fix leaking of kernel heap addresses via /proc

[SECURITY] Fix leaking of kernel heap addresses via /proc

[Dan Rosenberg]

Maintainers,

I am planning on submitting a series of patches to resolve the leakage
of kernel heap addresses to userspace via network protocol /proc
interfaces.  Revealing this information is a bad idea from a security
perspective for a number of reasons, the most obvious of which is it
provides unprivileged users a mechanism by which to create a structure
in the kernel heap containing function pointers, obtain the address of
that structure, and overwrite those function pointers by leveraging
other vulnerabilities.  It is my hope that by eliminating this
information leakage, in conjunction with making statically-declared
function pointer tables read-only (to be done in a separate patch
series), we can at least add a small hurdle for the exploitation of a
subset of kernel vulnerabilities.

Here's the list of protocols that I've identified as leaking socket
structure addresses via their /proc interfaces:

atm
ipv4
ipv6
key
netlink
packet
phonet
unix

There may be others, and I will continue looking for more examples, but
this is at least a good start.

Clearly, in most cases we cannot just remove the field from the /proc
output, as this would break a number of userspace programs that rely on
consistency.  However, I propose that we replace the address with a "0"
rather than leaking this information.

I'm writing this email before submitting a patch to ask if anyone would
have any objections to this change, and if anyone can foresee anything
breaking as a result of this.  Please let me know if you're on board,
have some doubts, or are totally opposed, and I can get this patch ready
for submission.

Thanks,
Dan Rosenberg

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Linus Torvalds]

On Saturday, November 6, 2010, Dan Rosenberg <drosenberg@vsecurity.com> wrote:
>
> Clearly, in most cases we cannot just remove the field from the /proc
> output, as this would break a number of userspace programs that rely on
> consistency.  However, I propose that we replace the address with a "0"
> rather than leaking this information.

I really think it would be much better to use the unidentified number
or similar.

Just replacing with zeroes is annoying, and has the potential of
losing actual information.

     Linus

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Ted Ts'o]

On Sat, Nov 06, 2010 at 01:50:32PM -0700, Linus Torvalds wrote:
> On Saturday, November 6, 2010, Dan Rosenberg <drosenberg@vsecurity.com> wrote:
> > Clearly, in most cases we cannot just remove the field from the /proc
> > output, as this would break a number of userspace programs that rely on
> > consistency.  However, I propose that we replace the address with a "0"
> > rather than leaking this information.
> 
> I really think it would be much better to use the unidentified number
> or similar.
> 
> Just replacing with zeroes is annoying, and has the potential of
> losing actual information.

Are there any userspace programs that might be reasonably expected to
_use_ this information?  If there is, we could just pick a random
number at boot time, and then XOR the heap adddress with that random
number.

						- Ted

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[David Miller]

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Sat, 6 Nov 2010 13:50:32 -0700

> On Saturday, November 6, 2010, Dan Rosenberg <drosenberg@vsecurity.com> wrote:
>>
>> Clearly, in most cases we cannot just remove the field from the /proc
>> output, as this would break a number of userspace programs that rely on
>> consistency.  However, I propose that we replace the address with a "0"
>> rather than leaking this information.
> 
> I really think it would be much better to use the unidentified number
> or similar.
> 
> Just replacing with zeroes is annoying, and has the potential of
> losing actual information.

I would really like to see the specific examples of where this is
happening, it sounds like something very silly to me.

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Oliver Hartkopp]

On 07.11.2010 00:57, David Miller wrote:
> From: Linus Torvalds <torvalds@linux-foundation.org>
> Date: Sat, 6 Nov 2010 13:50:32 -0700
> 
>> On Saturday, November 6, 2010, Dan Rosenberg <drosenberg@vsecurity.com> wrote:
>>>
>>> Clearly, in most cases we cannot just remove the field from the /proc
>>> output, as this would break a number of userspace programs that rely on
>>> consistency.  However, I propose that we replace the address with a "0"
>>> rather than leaking this information.
>>
>> I really think it would be much better to use the unidentified number
>> or similar.
>>
>> Just replacing with zeroes is annoying, and has the potential of
>> losing actual information.
> 
> I would really like to see the specific examples of where this is
> happening, it sounds like something very silly to me.

Indeed Urs and me had a similar discussion before we decided to put different
(user relevant) content into the procfs output and break the current layout:

http://patchwork.ozlabs.org/patch/70282/

The layout break was ok in this case as the people using the CAN procfs stuff
do this only when facing problems (with their applications) at runtime.

A discussed approach that won't break the procfs layout was to set the values
to "0" and only fill them with real content depending on CONFIG_DEBUG_INFO .

Would that fit here?

Or maybe a different config option CONFIG_DEBUG_KERNEL_ADDR would do the job,
as i don't know which distros enable CONFIG_DEBUG_INFO by default ...

Regards,
Oliver

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On Sun, 2010-11-07 at 11:28 +0100, Oliver Hartkopp wrote:
[...]
> The layout break was ok in this case as the people using the CAN procfs stuff
> do this only when facing problems (with their applications) at runtime.
> 
> A discussed approach that won't break the procfs layout was to set the values
> to "0" and only fill them with real content depending on CONFIG_DEBUG_INFO .
> 
> Would that fit here?
> 
> Or maybe a different config option CONFIG_DEBUG_KERNEL_ADDR would do the job,
> as i don't know which distros enable CONFIG_DEBUG_INFO by default ...

I believe most distributions now enable CONFIG_DEBUG_INFO (though the
debug information is then stripped and shipped in separate packages).  I
also dislike this idea of an implicit trade-off between debugging and
security; such a trade-off may be necessary but then it should be
explicit.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Andi Kleen]

Ted Ts'o <tytso@mit.edu> writes:

> Are there any userspace programs that might be reasonably expected to
> _use_ this information?  If there is, we could just pick a random
> number at boot time, and then XOR the heap adddress with that random
> number.

If any of the addresses can be guessed ever (and that is likely if it's
allocated at boot) determining the random value will be trivial
for everyone.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Chas Williams (CONTRACTOR)]

In message <87sjzcssx5.fsf@basil.nowhere.org>,Andi Kleen writes:
>Ted Ts'o <tytso@mit.edu> writes:
>
>> Are there any userspace programs that might be reasonably expected to
>> _use_ this information?  If there is, we could just pick a random
>> number at boot time, and then XOR the heap adddress with that random
>> number.
>
>If any of the addresses can be guessed ever (and that is likely if it's
>allocated at boot) determining the random value will be trivial
>for everyone.

i suppose one could use idr to map the pointers to unique values.  the
infiniband code uses this technique>

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Andi Kleen]

On Sun, Nov 07, 2010 at 05:48:45PM -0500, Chas Williams (CONTRACTOR) wrote:
> In message <87sjzcssx5.fsf@basil.nowhere.org>,Andi Kleen writes:
> >Ted Ts'o <tytso@mit.edu> writes:
> >
> >> Are there any userspace programs that might be reasonably expected to
> >> _use_ this information?  If there is, we could just pick a random
> >> number at boot time, and then XOR the heap adddress with that random
> >> number.
> >
> >If any of the addresses can be guessed ever (and that is likely if it's
> >allocated at boot) determining the random value will be trivial
> >for everyone.
> 
> i suppose one could use idr to map the pointers to unique values.  the
> infiniband code uses this technique>

idr requires allocating memory, and it's unclear you can do that
in all the situations where debugging printks are used. And
how would you get the idr table out of a broken kernel? And further
the memory allocations would eventually fill up your memory
if they go on. I don't think idr is a solution.

I don't really have a good solution either. Even if the
the individual pointers were removed from printks (which
probably doesn't make much difference anyways because those
printks usually happen only in unlikely debug situations):

The information about the memory layout early on in dmesg
is probably enough to make a good educated guess about
the locations of standard slab caches on a known kernel
image. For example the first N sockets opened are very likely
easy to guess this way.

I guess one could make dmesg root only, although I personally
use it often as non root.

Or maybe add some more randomization to the buddy allocator.
The drawback of that is that they tend to make
benchmarks unstable due to cache coloring differences.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Linus Torvalds]

On Sun, Nov 7, 2010 at 2:48 PM, Chas Williams (CONTRACTOR)
<chas@cmf.nrl.navy.mil> wrote:
>
> i suppose one could use idr to map the pointers to unique values.  the
> infiniband code uses this technique>

We already _have_ the unique value that /proc uses - the inode number.
It's what lsof and friends use to match things across different files
anyway.

Why are people arguing about stupid things? If it's wrong to expose
kernel pointers in /proc (and I do think it generally is something we
should try to avoid), then we already do have the natural alternative.
Which happens to be what the patch already does.

So the only question is whether kernel pointers are an information
leak worth worrying about. Personally, I think it's just damn stupid
to expose a kernel pointer unless you _have_ to. There is absolutely
no reason to expose the address of a socket in /proc, perhaps unless
you're in some super-duper-debugging mode that no sane person would
ever use outside of specialized network debugging (and I bet that in
that case you still shouldn't expose it in a _normal_ proc file, but
in debugfs or something).

                          Linus

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Dan Rosenberg]

Based on the feedback so far, I think a few things need to be worked
out.  Firstly, I'm going to separate out printk leaks as an issue to be
worked on at a later time - it seems there is some interest in
evaluating whether dmesg should be restricted, but that is out of scope
of my plans for this patch series.

That leaves the /proc leakage.  I don't think XOR-ing is a viable
option, since it would be relatively easy to infer the constant value.
The criticism raised so far is that cutting out the pointers entirely
results in the omission of potentially useful debugging information.  I
see two viable options to address this: either print out or omit
addresses based on privileges (CAP_NET_ADMIN, for example), or have it
controllable via sysctl.  I'm leaning towards the sysctl
option...thoughts?

-Dan

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Andi Kleen]

> So the only question is whether kernel pointers are an information
> leak worth worrying about. Personally, I think it's just damn stupid
> to expose a kernel pointer unless you _have_ to. There is absolutely

Agreed.

> no reason to expose the address of a socket in /proc, perhaps unless
> you're in some super-duper-debugging mode that no sane person would
> ever use outside of specialized network debugging (and I bet that in
> that case you still shouldn't expose it in a _normal_ proc file, but
> in debugfs or something).

In most cases you can get them by running gdb or crash on /proc/kcore 
anyways.

Still overall I suspect there may be a case that making
dmesg root only is a good idea.  At least optionally for the non 
kernel hackers out there.

The early memory map information can be likely used to guess a lot of 
locations and perhaps really make that heap overflow easier to write.
And perhaps some more mapping randomization inside the kernel.

I'm not sure the case is very strong though.

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Andi Kleen]

> The criticism raised so far is that cutting out the pointers entirely
> results in the omission of potentially useful debugging information.  I
> see two viable options to address this: either print out or omit
> addresses based on privileges (CAP_NET_ADMIN, for example), or have it
> controllable via sysctl.  I'm leaning towards the sysctl
> option...thoughts?

I would just remove the pointers from /proc and supply 
gdb macros that extract the equivalent information from /proc/kcore.
This is a bit racy, but for debugging it should be no
problem to run them multiple times as needed.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Willy Tarreau]

On Sat, Nov 06, 2010 at 04:57:03PM -0700, David Miller wrote:
> From: Linus Torvalds <torvalds@linux-foundation.org>
> Date: Sat, 6 Nov 2010 13:50:32 -0700
> 
> > On Saturday, November 6, 2010, Dan Rosenberg <drosenberg@vsecurity.com> wrote:
> >>
> >> Clearly, in most cases we cannot just remove the field from the /proc
> >> output, as this would break a number of userspace programs that rely on
> >> consistency.  However, I propose that we replace the address with a "0"
> >> rather than leaking this information.
> > 
> > I really think it would be much better to use the unidentified number
> > or similar.
> > 
> > Just replacing with zeroes is annoying, and has the potential of
> > losing actual information.
> 
> I would really like to see the specific examples of where this is
> happening, it sounds like something very silly to me.

It has happened to me several times to use an hex editor to check some
socket's parameters (eg: backlog) based on the pointer. Sometimes I had
even change some parameters at runtime as part of debugging sessions.

In fact we could consider than many places that return pointers could
return 0 to normal users and the real value only to root (or any special
capability). I find it important not to reduce the observability of the
kernel for the sake of security.

Regards,
Willy

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[David Miller]

From: Andi Kleen <andi@firstfloor.org>
Date: Mon, 8 Nov 2010 00:56:10 +0100

> I would just remove the pointers from /proc and supply 
> gdb macros that extract the equivalent information from /proc/kcore.
> This is a bit racy, but for debugging it should be no
> problem to run them multiple times as needed.

I do not think at all that this is tenable for the kind of
things people use the socket pointers for when debugging
problems.

I defeinitely prefer the inode number to this idea.

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Eric Dumazet]

Le dimanche 07 novembre 2010 à 18:01 -0800, David Miller a écrit :
> From: Andi Kleen <andi@firstfloor.org>
> Date: Mon, 8 Nov 2010 00:56:10 +0100
> 
> > I would just remove the pointers from /proc and supply 
> > gdb macros that extract the equivalent information from /proc/kcore.
> > This is a bit racy, but for debugging it should be no
> > problem to run them multiple times as needed.
> 
> I do not think at all that this is tenable for the kind of
> things people use the socket pointers for when debugging
> problems.
> 
> I defeinitely prefer the inode number to this idea.

We currently have no guarantee of sockets inode numbers unicity.
I admit chances of clash are low.

When a printk() happens right before a BUG(), how are we going to check
the dumped registers are possibly close the socket involved, if we dont
have access to the machine, and only the crashlog ?

BTW, any local user can look at "dmesg", and crash reports. These
reports are even published on a remote site (bugzilla) so that hostile
hackers can be feeded.

I am OK to delete socket pointers from /proc files for non root users
(after checking things like lsof continue to work correctly).
I dont remember using them while doing debugging stuff.

BTW, rtnetlink also expose socket pointers to non root users :

$ ss -e dst 192.168.20.108
State      Recv-Q Send-Q    Local Address:Port    Peer Address:Port   
ESTAB      0      0         10.150.51.210:46979   192.168.20.108:ssh 
timer:(keepalive,119min,0) ino:136919 sk:ffff88002129d7c0


Mixing in same patch /proc pointers removal and printk() pointers
removal seems wrong to me. Very different problems.

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Andi Kleen]

> When a printk() happens right before a BUG(), how are we going to check
> the dumped registers are possibly close the socket involved, if we dont
> have access to the machine, and only the crashlog ?

Is that really something you do regularly? It seems highly obscure
to me.

Besides if the kernel has timestamps enabled you can easily
guess based on the timestamps if the printk and the oops
are related.

-Andi

Re: [Security] [SECURITY] Fix leaking of kernel heap addresses via /proc

[Eric Dumazet]

Le lundi 08 novembre 2010 à 10:43 +0100, Andi Kleen a écrit :
> > When a printk() happens right before a BUG(), how are we going to check
> > the dumped registers are possibly close the socket involved, if we dont
> > have access to the machine, and only the crashlog ?
> 
> Is that really something you do regularly? It seems highly obscure
> to me.

Yes, very regularly, I can find bugs thanks to every bit of information
found in kernel logs, including code around the fault.

If people now say : "I have a kernel bug, but am not able to provide you
a kernel stack trace and previous printk() messages because of security.
You cannot have an access to this machine, and the bug happens once in a
while. Kernel version is also hidden. Please help me."

Oh well, thats a challenge, maybe use this cristal ball I have somewhere
in the attic ;)