Security: information leaks in /proc enable keystroke recovery

Security: information leaks in /proc enable keystroke recovery

[David Wagner]

At Usenix Security 2009, two researchers announced last week a new
security vulnerability in multi-user Linux systems.  They demonstrated
that one user can, in many cases, recover partial information about
the keystrokes that another user types into applications running on
that system.  For instance, they demonstrate how a malicious user can
recover partial information about SSH passwords typed by other users,
reducing the password search space by a factor of 250-2000x in
their experiments.  Thus, this could facilitate password recovery.

Question: Are there any plans to modify the Linux kernel to defend
against this kind of attack?

The paper is here:

http://www.usenix.org/events/sec09/tech/full_papers/zhang.pdf 

In a nutshell, they exploit the fact that many files in /proc are
world-readable yet contain sensitive information that can leak information
about inter-keystroke timings.  For instance, /proc/$PID/stat reveals the
ESP and EIP registers of the associated process, and is world-readable.
/proc/pid/status is also mentioned as revealing information that could
be exploited in these attacks.

Based on my understanding of their work, it sounds like some of
the information on those files should perhaps not be world-readable.
It's not clear to me that it's reasonable for the kernel to reveal ESP,
EIP, and other sensitive information about process behavior to everyone
on the same system.

Are folks already aware of these vulnerabilities?  Is there any work
underway to try to address the issues identified in the Usenix Security
paper?

Re: Security: information leaks in /proc enable keystroke recovery

[Oliver Pinter]

On 8/16/09, David Wagner <daw@cs.berkeley.edu> wrote:
> At Usenix Security 2009, two researchers announced last week a new
> security vulnerability in multi-user Linux systems.  They demonstrated
> that one user can, in many cases, recover partial information about
> the keystrokes that another user types into applications running on
> that system.  For instance, they demonstrate how a malicious user can
> recover partial information about SSH passwords typed by other users,
> reducing the password search space by a factor of 250-2000x in
> their experiments.  Thus, this could facilitate password recovery.
>
> Question: Are there any plans to modify the Linux kernel to defend
> against this kind of attack?
>
> The paper is here:
>
> http://www.usenix.org/events/sec09/tech/full_papers/zhang.pdf
>
> In a nutshell, they exploit the fact that many files in /proc are
> world-readable yet contain sensitive information that can leak information
> about inter-keystroke timings.  For instance, /proc/$PID/stat reveals the
> ESP and EIP registers of the associated process, and is world-readable.
> /proc/pid/status is also mentioned as revealing information that could
> be exploited in these attacks.
>
> Based on my understanding of their work, it sounds like some of
> the information on those files should perhaps not be world-readable.
> It's not clear to me that it's reasonable for the kernel to reveal ESP,
> EIP, and other sensitive information about process behavior to everyone
> on the same system.
>
> Are folks already aware of these vulnerabilities?  Is there any work
> underway to try to address the issues identified in the Usenix Security
> paper?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

[snip]
6.2 Information Leaks in the Procfs of
       Other UNIX-like Systems
Besides Linux, most other UNIX-like systems also im-
plement procfs. These implementations vary from case
to case, and as a result, their susceptibilities to side-
channel attacks also differ. Here we discuss such privacy
risks on two systems, FreeBSD and OpenSolaris.
   FreeBSD manages its process files more cautiously
than Linux12 : it puts all register values into the file
/proc/pid/regs that can only be read by the owner
of a process, which blocks the information used by
our attack. However, we found that other informa-
tion released by the procfs can lead to similar attacks.
A prominent example is the system time reported by
/proc/pid/status, a file open to every user. Fig-
ure 11 shows the correlations between the time con-
sumed by vim and the keystrokes it received, as ob-
served in our research. This demonstrates that keystroke
events within the process can be identified from the
change of its system time, which makes keystroke eaves-
dropping possible. A problem here is that we may not
be able to detect special keys a user enters, for example,
“MOV CURSOR”, which is determined from ESP/EIP in-
formation on Linux. A possible solution is using the dis-
crepancies of system-time increments triggered by dif-
ferent keys being entered to fingerprint these individual
keys. Further study of this technique is left to our future
research.
[/snip]

Re: Security: information leaks in /proc enable keystroke recovery

[Robert Watson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3665 bytes --]


On Sun, 16 Aug 2009, Oliver Pinter wrote:

> FreeBSD manages its process files more cautiously than Linux12 : it puts all 
> register values into the file /proc/pid/regs that can only be read by the 
> owner of a process, which blocks the information used by

This is inaccurate, but largely in an academic sense.  The FreeBSD kernels 
computes debug permissions between two processes from a number of factors, 
including:

- Comparison of uids (effective, real, saved)
- Comparison of gids (effective, real, saved)
- Subset check on the additional group set
- Jail information
- Mandatory access control policies

There are some other checks that fit the pattern of credential comparison less 
well:

- We deny debugger activity during execve(2) as a robustness protection
   against possible race conditions.

- We have global policy controls, security.bsd.see_other_gids and
   security.bsd.see_other_uids, which allow administrators to scope not just
   debugging facilities, but also monitoring facilities.

- We have a global policy control, security.bsd.unprivileged_proc_debug, to
   disable debugging facilities for unprivileged users, on the grounds that (a)
   this is sometimes a desirable system policy, and (b) all UNIX systems have
   historically suffered from significant debugger security vulnerabilities and
   this provides an easy work-around to use if that happens in the future.

Which is to say: the UNIX file system permissions appearing in procfs are 
purely decorative -- they roughly summarize, but do not implement, the above 
checks.

procfs is also deprecated in FreeBSD, and has not been mounted by default for 
several major releases.  Instead, the system call interfaces ktrace(2), 
ptrace(2), and sysctl(2) provide access to trace data, process debugging, and 
process state (such as address space layout and file descriptor information). 
Some legacy setgid kvm tools exist that use libkvm and /dev/kmem, but we are 
eliminating these as quickly as we can; they may not follow the same policies 
as those implemented in the kernel.

The see_other_uids and see_other_gids policy sysctls narrow the policy on 
inter-process visibility via monitoring controls -- however, 
additional hardening is required to enforce this policy universally.  For 
example, administrators will also need to limit access to logs, accounting 
data, and so on, for this to be fully effective.

Beyond this, and assuming the correct implementation of the above, we're into 
the grounds of classic trusted OS covert channel analysis, against which no 
COTS UNIX OSes I'm aware of are hardened.  This isn't to dismiss these attacks 
as purely hypothetical -- we've seen some rather compelling examples of covert 
channels being exploited in unexpected and remarkably practical ways in the 
last few years (Steven Murdoch's "Hot or Not" paper takes the cake in that 
regard, I think).

However, this next step up from "the kernel doesn't reveal information on 
processes from other users" involves scheduler hardening, consideration of 
inter-CPU/core/thread cache interactions, and so on -- things that we don't 
have a good research, let alone production, OS understanding of.  There are 
tools in FreeBSD that can help with some of these issues -- for example, you 
can use login classes to pin different users to different CPU threads, cores, 
or packages.  However, this leaves the implementation of policy up to the 
administrator, rather than simply allowing the administator to specify the 
policy that mutually untrusting processes can't share CPUs with each other in 
some window.

Robert N M Watson
Computer Laboratory
University of Cambridge

Re: Security: information leaks in /proc enable keystroke recovery

[David Wagner]

Thanks for the comments.

> Beyond this, and assuming the correct implementation of the above,
> we're into the grounds of classic trusted OS covert channel analysis,
> against which no COTS UNIX OSes I'm aware of are hardened.  This isn't to
> dismiss these attacks as purely hypothetical -- we've seen some rather
> compelling examples of covert channels being exploited in unexpected
> and remarkably practical ways in the last few years (Steven Murdoch's
> "Hot or Not" paper takes the cake in that regard, I think).

To be pedantic, I'd say that the Usenix Security paper describes a side
channel, not a covert channel.  The paper shows how a malicious attacker
can attack an unsuspecting victim application.

Covert channels are when both the sender and the receiver are malicious
and cooperating to transmit information from the sender to the receiver.
In contrast, side channels arise when the "sender" is unintentionally
and inadvertently leaking information that can be decoded by a malicious
receiver, against the interests of the "sender".  The attack in the
paper is a side channel, not a covert channel.  When it comes to covert
channels, it is indeed reasonable to throw up your hands and say that
defending against covert channels is basically impossible, so why bother?
For side channels, though, it's less clear that this is a persuasive
argument.

I agree we certainly shouldn't discuss the keystroke recovery attack
as hypothetical, because it is clearly not hypothetical: the authors
implemented it and found that it works.

> However, this next step up from "the kernel doesn't reveal information on
> processes from other users" involves scheduler hardening, consideration
> of inter-CPU/core/thread cache interactions, and so on -- things that we
> don't have a good research, let alone production, OS understanding of.

Indeed.  A major challenge.  Good to hear that, in its default
configuration, FreeBSD does eliminate the attack vector described in
the Usenix Security paper (the EIP/ESP leak).  It seems a good starting
point would be to limit access to information that we know can enable
keystroke recovery -- as FreeBSD apparently already does, but Linux
apparently does not.

Re: Security: information leaks in /proc enable keystroke recovery

[Robert N. M. Watson]

On 16 Aug 2009, at 22:09, David Wagner wrote:

>> Beyond this, and assuming the correct implementation of the above,
>> we're into the grounds of classic trusted OS covert channel analysis,
>> against which no COTS UNIX OSes I'm aware of are hardened.  This  
>> isn't to
>> dismiss these attacks as purely hypothetical -- we've seen some  
>> rather
>> compelling examples of covert channels being exploited in unexpected
>> and remarkably practical ways in the last few years (Steven Murdoch's
>> "Hot or Not" paper takes the cake in that regard, I think).
>
> To be pedantic, I'd say that the Usenix Security paper describes a  
> side
> channel, not a covert channel.  The paper shows how a malicious  
> attacker
> can attack an unsuspecting victim application.
>
> Covert channels are when both the sender and the receiver are  
> malicious
> and cooperating to transmit information from the sender to the  
> receiver.
> In contrast, side channels arise when the "sender" is unintentionally
> and inadvertently leaking information that can be decoded by a  
> malicious
> receiver, against the interests of the "sender".  The attack in the
> paper is a side channel, not a covert channel.  When it comes to  
> covert
> channels, it is indeed reasonable to throw up your hands and say that
> defending against covert channels is basically impossible, so why  
> bother?
> For side channels, though, it's less clear that this is a persuasive
> argument.

Hi David--

I see what you're saying, but I'm not sure I entirely agree on the  
pedantic definitions front ("two can play at this game"). Historically  
interesting definitions in DoD 5200.28-STD ("The Orange Book") and  
NCSC-TG-030 ("A Guide to Understanding Covert Channel Analysis of  
Trusted Systems") are decidedly hazy on the concept of intention, with  
some portions specific on its involvement and others entirely  
disregarding it. These definitions come out of trusted OS research/ 
development, and might be considered historically anachronistic by  
some. To my mind, the OS timing issue we're discussing meet two of the  
definitions of "covert channel" presented in NCSC-TG-030:

   Definition 4: Covert channels are those that "use entities not  
normally viewed as data objects to transfer information from one  
subject to another."

   Definition 5: Given a non-discretionary (e.g., mandatory) security  
policy model M and its interpretation I(M) in an operating system, any  
potential communication between two subjects I(Sh) and I(Si) of I(m)  
is covert if and only if any communication between the corresponding  
subject Sh and Si of the model M is illegal in M.

Which is to say: what makes something a "covert channel" is not the  
intention to communicate in violation of a policy, but the  
*possibility* of communication in violation of system design or  
security policy. Both documents are concerned primarily with  
intentional leakage/corruption of information in confidentiality and  
integrity policies, but their definitions appear more general.

The use of the word "mandatory" here is certainly one that also bears  
consideration: I would argue that the system integrity constraints of  
historic UNIX, such as those on inter-process control vectors  
(debugging, signals, ...) do in fact constitute a mandatory policy,  
albeit not based on information flow control or a particularly clean  
or well-documented model. As an example: unprivileged users are  
permitted only limited scope under the discretionary access control  
policy to delegate rights for objects they own. They can delegate to  
another user write access to a file via open(2), but not the right to  
chown the file using chown(2), signal a process using kill(2), etc,  
making the protections that prevent these operations "mandatory".

I would argue that undesired information leakage via I/O timing across  
process monitoring interfaces qualifies as a covert channel under both  
definitions above: it's not an intended communication channel provided  
by the OS design, and that the OS security policy is not intended to  
allow unintentional communication of I/O data without explicit  
delegation. The historic covert channel analysis of the timing  
problem, drawn out in somewhat painful detail in NCSC-TG-030, seems  
pretty much to apply to this problem.

I wouldn't argue that EIP leakage in procfs counted, on the other  
hand, as it appears to be an intentional, if in retrospect  
unfortunate, part of the design and policy. I wouldn't doubt that  
countless other similar "oh, perhaps not" cases of information leakage  
exist across countless variations on the UNIX theme due to monitoring  
and debugging interfaces -- for example, netstat's reporting of TCP  
pending send/receive queues seems likely subject to quite similar  
problems, as with timestamps on device nodes in /dev, even network  
interface or protocol statistics from netstat.

Coming down on the other side of pedantic, BTW, the Orange Book's  
definition of "Covert storage channels" seems to ignore intention,  
"Covert timing channels" seems to require it.

>> However, this next step up from "the kernel doesn't reveal  
>> information on
>> processes from other users" involves scheduler hardening,  
>> consideration
>> of inter-CPU/core/thread cache interactions, and so on -- things  
>> that we
>> don't have a good research, let alone production, OS understanding  
>> of.
>
> Indeed.  A major challenge.  Good to hear that, in its default
> configuration, FreeBSD does eliminate the attack vector described in
> the Usenix Security paper (the EIP/ESP leak).  It seems a good  
> starting
> point would be to limit access to information that we know can enable
> keystroke recovery -- as FreeBSD apparently already does, but Linux
> apparently does not.


NCSC-TG-030 has quite a bit to say on the topic of these sorts of  
things, albeit addressed at a different context and in a different  
time. I find myself skeptical that the sorts of protections we are  
waving our hands at apply all that well to UNIX-like systems due to  
their origin as time-sharing systems. However, I think a more  
interesting place to direct this analysis would be the current flush  
of hypervisors, which appear (possibly even claim) to offer much  
stronger separation in a less time-sharesque way.

Robert

Re: Security: information leaks in /proc enable keystroke recovery

[David Wagner]

I still think my definitions of "covert channel" vs "side channel"
better reflect accepted usage these days, but whatever.  I don't
have any great desire to debate the definitions.  That doesn't seem
like a good use of everyone's time.

I was trying to define some shorthand to more concisely make my point.
Since it appears my preferred shorthand turned out to be a barrier to
communication, rather than an aid, I'll try to make my point again, this
time spelling it out without using the problematic shorthand.  I care
more about the ultimate point than the language we use to communicate it.

My broader point is this:

I accept your argument that there is no point trying to defend
against deliberate communication of information between two cooperating
processes via some sneaky channel; there is no hope of stopping that
in general-purpose commodity OS's.  If process X and Y are both
colluding to send information from X to Y, they will succeed, no
matter how hard we try.  We have no hope of closing all such channels,
for general-purpose commodity OS's (like FreeBSD or Linux).

However I do not accept that this argument means we should throw up
our hands and ignore cases where the kernel allows malicious process
Y to spy on process X, against X's will.  If the kernel has a leak that
lets process Y eavesdrop on keystrokes typed into process X, that's
arguably worth fixing.  Trying to prevent that is not clearly hopeless.

There is a significant difference in threat model between

  "both X and Y are malicious and colluding with each other to
   facilitate some joint purpose shared by both X and Y"

vs

  "Y is malicious and is attempting to subvert the security of
   process X, against X's will".

If the designers deliberately intended to allow process Y to snoop on
the ESP and EIP of process X, even when there is no relationship between
X and Y (e.g., they don't have the same uid, and Y isn't root), well,
I would claim that was a design error.  Facilitating keystroke recovery
does not seem like a good design goal.

It's possible that the impact could be broader than discussed in the
Usenix Security paper.  Imagine if process X is doing crypto, say an RSA
decryption, and process Y is running on the same machine and is malicious.
If process Y is allowed to observe the EIP of process X, then process Y
may be able to observe which path process X has taken through the code.
In some cases, such as a naive implementation of RSA decryption, this
may reveal X's private key.

Leaking EIP and ESP to every other user on the same system strikes me
as pretty dubious.

Re: Security: information leaks in /proc enable keystroke recovery

[Robert Watson]

On Sun, 16 Aug 2009, David Wagner wrote:

> I accept your argument that there is no point trying to defend against 
> deliberate communication of information between two cooperating processes 
> via some sneaky channel; there is no hope of stopping that in 
> general-purpose commodity OS's.  If process X and Y are both colluding to 
> send information from X to Y, they will succeed, no matter how hard we try. 
> We have no hope of closing all such channels, for general-purpose commodity 
> OS's (like FreeBSD or Linux).

Moving beyind EIP/ESP, which are clearly a bad idea:

The OS community has not engaged well with the concerns raised by past 
cache-based crypto side channels, in part because it seemed the least complex 
solution was hardening crypto against having key-driven footprints in the 
cache.  However, the problem they represent (avoiding the use of shared 
resources between mutually untrusting processes, and then mitigating efects 
that remain) definitely sounds like the covert channel problem, with very 
similar concerns extensively discussed in the documents I referred to.

In an interactive system, the scheduling of threads in a process reflect the 
completion of various events: user I/O, network I/O, disk I/O, or perhaps the 
expiration of a timer associated with application-internal events (animations, 
statistics, etc).  Monitoring these from another process is intentionally easy 
on commodity OS's -- there are a variety of monitoring statistics, from the 
already mentioned process/thread execution time, to context switch counters, 
wait channels/addresses, lock states, timestamps on special devices, etc, not 
to mention having CPU sink processes that nice themselves appropriately and 
hang around monitoring execution of other processes/threads/the kernel through 
gaps in its own scheduling.  Some of the intentional mechanisms are specific 
to processes, and easy to block by policy.  Others are global, and begin the 
sliding down the slope of "making the system and applications a lot harder to 
analyze and debug", something that sites frequently hosting large numbers of 
mutually untrusting users (web farms) may not be willing to deal do.

Into the area of techniques that annoy people: my guess is that you may also 
be able to measure the context switching of processes on other CPUs through 
very careful timing of events in the kernel on your local CPU.  For example, 
it's a reasonable bet that using the TSC and carefully selected system 
calls/arguments, you can measure cache line behavior associated with kernel 
scheduler/statistic lines that will be pulled to another CPU when a context 
switch takes place.  For example, consider per-CPU run queue locks or context 
switch statistics, which may in edge cases be pulled to another CPU, such as 
when monitoring takes place.  If they are already local to the attacking CPU, 
no context switch has taken place on the other CPU since you last checked; if 
they're non-local, a context switch has taken place.

Following Colin Percival's paper on cache side channels for RSA, there was a 
lot of discussion about how the OS could help mitigate these problems: do you 
provide "security critical sections" around cryptography which introduce 
temporary but performance-degrading mutual exclusion of caches based on 
knowledge of the CPU topology, for example.  Identifying and offering similar 
trade-offs between performance and security, avoiding excess complexity, and 
in particular, limiting the scope of those performance losses to only critical 
moments will be key if the security community wants to engage the OS community 
here.  Otherwise I suspect these concerns will pass by, unaddressed, again.

Robert N M Watson
Computer Laboratory
University of Cambridge

Re: Security: information leaks in /proc enable keystroke recovery

[Dag-Erling Smørgrav]

David Wagner <daw@cs.berkeley.edu> writes:
> I agree we certainly shouldn't discuss the keystroke recovery attack
> as hypothetical, because it is clearly not hypothetical: the authors
> implemented it and found that it works.

It *is* hypothetical.  They were able to collect some keystrokes (but
not all) in Linux, but IIUC, all they could do in FreeBSD was figure out
whether or not a key was pressed at a certain time (or during a certain
interval).  They *hypothesize* that the interval between keystrokes can
be used to identify the keys being pressed, but they haven't actually
done it.  I can imagine - purely hypothetically - that it would be
possible, but only while the user was typing running text; the
parameters would vary greatly from typist to typist, and between
keyboard layouts, and you could probably defeat it pretty easily (at
least some of the time) by deliberately typing slowly and arythmically,
e.g. typing in your password with only one finger.

DES
-- 
Dag-Erling Smørgrav - des@des.no

Re: Security: information leaks in /proc enable keystroke recovery

[Theodore Tso]

On Sat, Aug 15, 2009 at 03:21:27PM -0700, David Wagner wrote:
> At Usenix Security 2009, two researchers announced last week a new
> security vulnerability in multi-user Linux systems.  They demonstrated
> that one user can, in many cases, recover partial information about
> the keystrokes that another user types into applications running on
> that system.  For instance, they demonstrate how a malicious user can
> recover partial information about SSH passwords typed by other users,
> reducing the password search space by a factor of 250-2000x in
> their experiments.  Thus, this could facilitate password recovery.
> 
> In a nutshell, they exploit the fact that many files in /proc are
> world-readable yet contain sensitive information that can leak information
> about inter-keystroke timings.  For instance, /proc/$PID/stat reveals the
> ESP and EIP registers of the associated process, and is world-readable.
> /proc/pid/status is also mentioned as revealing information that could
> be exploited in these attacks.

A configuration option which defaults to disabling ESP and EIP would
be a simple way to prevent this specific instance of information
leakage.  The problem is there are other files that might reveal
timing information, but which are very useful for a system
administrator.  A key example of this is /proc/$pid/wchan, which is
responsible for the WCHAN column is a ps listing.

Maybe the better example, othter than making some of these files
non-world-readable, or zeroing out certain fields in /proc/$PID/stat,
is to put in a limit to how quickly specific files can be queried.  If
a process's /proc files which could be used to extract keystroke
timing information, is accessed more than say, ten times a second, a
random delay could be added before the information is returned.

Essentially, we can treat this as a covert channel that needs to be
restricted off, without necessarily completely removing the ability
for a system administrator to see what processes are blocked on via
"ps lx".

							- Ted

Re: Security: information leaks in /proc enable keystroke recovery

[David Wagner]

Theodore Tso  wrote:
> A configuration option which defaults to disabling ESP and EIP would
> be a simple way to prevent this specific instance of information
> leakage.  The problem is there are other files that might reveal
> timing information, but which are very useful for a system
> administrator.  A key example of this is /proc/$pid/wchan, which is
> responsible for the WCHAN column is a ps listing.

If they're useful for system administrators, would making them
readable to root (but not everyone) be enough?

Re: Security: information leaks in /proc enable keystroke recovery

[Theodore Tso]

On Sun, Aug 16, 2009 at 12:44:14AM +0000, David Wagner wrote:
> Theodore Tso  wrote:
> > A configuration option which defaults to disabling ESP and EIP would
> > be a simple way to prevent this specific instance of information
> > leakage.  The problem is there are other files that might reveal
> > timing information, but which are very useful for a system
> > administrator.  A key example of this is /proc/$pid/wchan, which is
> > responsible for the WCHAN column is a ps listing.
> 
> If they're useful for system administrators, would making them
> readable to root (but not everyone) be enough?

I suppose sysadmins use "sudo ps" instead of "ps", but that would be a
bit inconvenient and would certainly clutter the sudo logs.  It might
also cause people to decide it's more trouble than it's worth, and
configure their system not restrict the various proc files.

That's why I think it would be useful to restrict the number of times
any process in the system can sample the files before the system
starts inserting random delays, thus destroying the usefulness of
using wchan for mounting timing attacks, without removing system
functionality.

						- Ted

Re: Security: information leaks in /proc enable keystroke recovery

[david]

On Sat, 15 Aug 2009, Theodore Tso wrote:

> On Sun, Aug 16, 2009 at 12:44:14AM +0000, David Wagner wrote:
>> Theodore Tso  wrote:
>>> A configuration option which defaults to disabling ESP and EIP would
>>> be a simple way to prevent this specific instance of information
>>> leakage.  The problem is there are other files that might reveal
>>> timing information, but which are very useful for a system
>>> administrator.  A key example of this is /proc/$pid/wchan, which is
>>> responsible for the WCHAN column is a ps listing.
>>
>> If they're useful for system administrators, would making them
>> readable to root (but not everyone) be enough?
>
> I suppose sysadmins use "sudo ps" instead of "ps", but that would be a
> bit inconvenient and would certainly clutter the sudo logs.  It might
> also cause people to decide it's more trouble than it's worth, and
> configure their system not restrict the various proc files.
>
> That's why I think it would be useful to restrict the number of times
> any process in the system can sample the files before the system
> starts inserting random delays, thus destroying the usefulness of
> using wchan for mounting timing attacks, without removing system
> functionality.

would it make sense to limit the precision of the times reported? or cache 
the results for some time period?

David Lang

Re: Security: information leaks in /proc enable keystroke recovery

[David Wagner]

Theodore Tso  wrote:
>On Sun, Aug 16, 2009 at 12:44:14AM +0000, David Wagner wrote:
>> Theodore Tso  wrote:
>> > A configuration option which defaults to disabling ESP and EIP would
>> > be a simple way to prevent this specific instance of information
>> > leakage.  The problem is there are other files that might reveal
>> > timing information, but which are very useful for a system
>> > administrator.  A key example of this is /proc/$pid/wchan, which is
>> > responsible for the WCHAN column is a ps listing.
>> 
>> If they're useful for system administrators, would making them
>> readable to root (but not everyone) be enough?
>
>I suppose sysadmins use "sudo ps" instead of "ps", but that would be a
>bit inconvenient and would certainly clutter the sudo logs.  It might
>also cause people to decide it's more trouble than it's worth, and
>configure their system not restrict the various proc files.

OK.  What about this:

(a) Remove ESP and EIP from /proc/$pid/stat{,us} entirely.  Put them in
some other file that is only readable by root and by the owner of the
process, but is not world-readable.  We know that ESP and EIP can be used
for keystroke recovery, and they are not usually used by administrators,
so the first step is to lock them down tightly: there is no downside.

(b) Keep /proc/$pid/wchan.  We suspect that it's possible this might
be usable for keystroke recovery, but we're not sure, and we do know
that blocking this entirely might annoy administrators, so rather than
blocking access entirely, apply a more lenient policy.  If the user
is not the owner of the process and is not root, allow up to about 10
accesses to /proc/$pid/wchan per, say, 10 minutes; if a user makes more
than 10 accesses per 10-minute period, insert random delays on the order
of a few hundred milliseconds into every access to /proc/$pid/wchan.

Notes: It is critical to do this by user, rather than by process;
otherwise a malicious user can fork a bazillion processes, each of which
reads /proc/$pid/wchan a constant number of times.  The "few hundred
milliseconds" comes from the fact that, for touch typists, typing two
keys with alternating hands typically takes something like 0-150ms,
whereas typing two keys with the same hand typically takes something
like 150-300ms; so if we want to obfuscate the difference between these
two, we need several hundreds of milliseconds of delay.  Allowing more
than a small number of undelayed accesses to /proc/$pid/wchan may not
be safe: passwords typically have less than a dozen characters in them,
and it may be possible to mount non-trivial attacks with only a few
probes per keypress (perhaps as few as two probes).

These two steps could be implemented separately.  Given that there are
demonstrated attacks that exploit ESP and EIP, it might be reasonable
to undertake step (a) right away.

Re: Security: information leaks in /proc enable keystroke recovery

[Theodore Tso]

On Mon, Aug 17, 2009 at 12:31:38AM +0000, David Wagner wrote:
> 
> OK.  What about this:
> 
> (a) Remove ESP and EIP from /proc/$pid/stat{,us} entirely.  Put them in
> some other file that is only readable by root and by the owner of the
> process, but is not world-readable.  We know that ESP and EIP can be used
> for keystroke recovery, and they are not usually used by administrators,
> so the first step is to lock them down tightly: there is no downside.

Agreed.

> (b) Keep /proc/$pid/wchan.  We suspect that it's possible this might
> be usable for keystroke recovery, but we're not sure, and we do know
> that blocking this entirely might annoy administrators, so rather than
> blocking access entirely, apply a more lenient policy.  If the user
> is not the owner of the process and is not root, allow up to about 10
> accesses to /proc/$pid/wchan per, say, 10 minutes; if a user makes more
> than 10 accesses per 10-minute period, insert random delays on the order
> of a few hundred milliseconds into every access to /proc/$pid/wchan.
> 
> Notes: It is critical to do this by user, rather than by process;
> otherwise a malicious user can fork a bazillion processes, each of which
> reads /proc/$pid/wchan a constant number of times.  The "few hundred
> milliseconds" comes from the fact that, for touch typists, typing two
> keys with alternating hands typically takes something like 0-150ms,

I was going to suggest instead of doing it by user or by process,
doing it globally; that is, *any* non-root process tries to access a
particular task's wchan /proc file more than X accesses per Y minutes,
we start inserting random delays.  Note that the "top" program by
default will refresh the screen once every 3 seconds, so allowing at
least 20 accesses per minute would be necessary to prevent "top" from
getting hit by a pretty massive slowdown.

That should be fine, I think; even if someone were to query wchan once
a second, it doesn't seem likely that would be enough to be able to
carry out a keystroke timing attack.

As far as "not being sure" whether it can be used for keystroke
recovery, after reading the Usenix security paper, I'm quite certain
that it would be enough; all the attack relies on is knowing when the
user's keystrokes have been recorded.  The EIP/ESP is one way of
figuring this out; but so is checking to see whether the process is
blocked on n_tty_wait or not.  In fact, arguably it would be
**easier** to use wchan, since with the EIP/ESP route you need to
figure out what EIP is associated with the read system call, where as
knowledge of whether the process is blocked on n_tty_wait far more
telling.

In addition, if you know that the user is entering a password at that
point in time, simply looking to see whether the process is in state
'R' or 'S' might be enough to know when the keystrokes are being
entered.  I might be willing to put that in the category, "We suspect
that it's possible this might be usable for keystroke recovery, but
we're not sure" --- but it would be a pretty strong suspicion.  As far
as whether you could use WCHAN for keystroke recovery, I'd call that a
near-certainty.

There are other timing channels I'd be wary of as well; for example,
simply using stat on the pseudo-tty will record the last access time
to a very high degree of accuracy:

% stat  /dev/pts/0
  File: `/dev/pts/0'
  Size: 0		Blocks: 0          IO Block: 1024   character special file
Device: ah/10d		Inode: 3           Links: 1     Device type: 88,0
Access: (0620/crw--w----)  Uid: (15806/   tytso)   Gid: (    5/     tty)
Access: 2009-08-16 22:21:06.847035909 -0400
Modify: 2009-08-16 22:21:06.847035909 -0400
Change: 2009-08-14 09:13:58.915700017 -0400

Maybe we should only be recording the last access and last modified
time on pseudo-tty's to a second granularity, hmmm?

						- Ted

Re: Security: information leaks in /proc enable keystroke recovery

[James Morris]

On Sun, 16 Aug 2009, Theodore Tso wrote:

> On Mon, Aug 17, 2009 at 12:31:38AM +0000, David Wagner wrote:
> > 
> > OK.  What about this:
> > 
> > (a) Remove ESP and EIP from /proc/$pid/stat{,us} entirely.  Put them in
> > some other file that is only readable by root and by the owner of the
> > process, but is not world-readable.  We know that ESP and EIP can be used
> > for keystroke recovery, and they are not usually used by administrators,
> > so the first step is to lock them down tightly: there is no downside.
> 
> Agreed.

It might be best to require a capability for this if not root.

The candidates seem to be CAP_SYS_ADMIN and CAP_SYS_PTRACE.

> Maybe we should only be recording the last access and last modified
> time on pseudo-tty's to a second granularity, hmmm?

I can't think of a valid case for finer granularity here, although I guess 
it could be made a sysctl if necessary.

I wonder if there are any other channels?


- James
-- 
James Morris
<jmorris@namei.org>

Re: Security: information leaks in /proc enable keystroke recovery

[Arjan van de Ven]

On Mon, 17 Aug 2009 12:45:45 +1000 (EST)
James Morris <jmorris@namei.org> wrote:

> On Sun, 16 Aug 2009, Theodore Tso wrote:
> 
> > On Mon, Aug 17, 2009 at 12:31:38AM +0000, David Wagner wrote:
> > > 
> > > OK.  What about this:
> > > 
> > > (a) Remove ESP and EIP from /proc/$pid/stat{,us} entirely.  Put
> > > them in some other file that is only readable by root and by the
> > > owner of the process, but is not world-readable.  We know that
> > > ESP and EIP can be used for keystroke recovery, and they are not
> > > usually used by administrators, so the first step is to lock them
> > > down tightly: there is no downside.
> > 
> > Agreed.
> 
> It might be best to require a capability for this if not root.
> 
> The candidates seem to be CAP_SYS_ADMIN and CAP_SYS_PTRACE.

I would suggest we just use the "can ptrace" permission function
altogether... (after all, if you can ptrace an app you already own all
its keys)


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: Security: information leaks in /proc enable keystroke recovery

[Pavel Machek]

Hi!

> > (b) Keep /proc/$pid/wchan.  We suspect that it's possible this might
> > be usable for keystroke recovery, but we're not sure, and we do know
> > that blocking this entirely might annoy administrators, so rather than
> > blocking access entirely, apply a more lenient policy.  If the user
> > is not the owner of the process and is not root, allow up to about 10
> > accesses to /proc/$pid/wchan per, say, 10 minutes; if a user makes more
> > than 10 accesses per 10-minute period, insert random delays on the order
> > of a few hundred milliseconds into every access to /proc/$pid/wchan.
> > 
> > Notes: It is critical to do this by user, rather than by process;
> > otherwise a malicious user can fork a bazillion processes, each of which
> > reads /proc/$pid/wchan a constant number of times.  The "few hundred
> > milliseconds" comes from the fact that, for touch typists, typing two
> > keys with alternating hands typically takes something like 0-150ms,
> 
> I was going to suggest instead of doing it by user or by process,
> doing it globally; that is, *any* non-root process tries to access a
> particular task's wchan /proc file more than X accesses per Y minutes,
> we start inserting random delays.  Note that the "top" program by
> default will refresh the screen once every 3 seconds, so allowing at
> least 20 accesses per minute would be necessary to prevent "top" from
> getting hit by a pretty massive slowdown.
> 
> That should be fine, I think; even if someone were to query wchan once
> a second, it doesn't seem likely that would be enough to be able to
> carry out a keystroke timing attack.

It will still be complex accounting :-(. Is really wchan so useful?

> In addition, if you know that the user is entering a password at that
> point in time, simply looking to see whether the process is in state
> 'R' or 'S' might be enough to know when the keystrokes are being
> entered.  I might be willing to put that in the category, "We suspect
> that it's possible this might be usable for keystroke recovery, but
> we're not sure" --- but it would be a pretty strong suspicion.  As far
> as whether you could use WCHAN for keystroke recovery, I'd call that a
> near-certainty.
> 
> There are other timing channels I'd be wary of as well; for example,
> simply using stat on the pseudo-tty will record the last access time
> to a very high degree of accuracy:

/proc/interrupts ? various interfaces powertop uses?

I did not have chance to read the paper yet, but... does not ssh have
bad problem in that case?

								Pavel


-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Security: information leaks in /proc enable keystroke recovery

[Henrique de Moraes Holschuh]

On Fri, 21 Aug 2009, Pavel Machek wrote:
> /proc/interrupts ? various interfaces powertop uses?
> 
> I did not have chance to read the paper yet, but... does not ssh have
> bad problem in that case?

I am told that SSH tries it best to take constant time for everything
that is even remotely sensitive (or at least, it should)... but it, as
well as openssl, gpg, gnutls and libc for that matter, can't really
protect themselves from sidechannels and information leaks that show up
when one abuses other CPUs (especially if SMT is operational) to gather
behaviour data of the other tasks/processes/threads in the system, or
when one gets to watch the cache misses, CPU pipeline stalls, etc.

I would also expect that *any* timing-related, power-related or
cache-/brach-prediction-/pipeline-stall-/CPU-behaviour- related high-
and medium-precision statistic to be viable side-channels to leak data.

These things are way too subtle to play the "prove it to me then" crap:
it often takes a stroke of geniality on top of extreme skill to find,
let alone exploit them.  Therefore, it is really difficult to say "they
don't exist".

IMO, it is too hard a problem to try to close for real, but we really
should try to have all our crypto be constant-everything as much as we
can (constant CPU effort, constant timing, constant memory pattern
access), and to clamp the precision of measurements sent to userspace to
the strictly needed for them to be useful for system administration,
unless the kernel was compiled to expose high-precision statistics for
development work.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: Security: information leaks in /proc enable keystroke recovery

[Amerigo Wang]

On Sat, Aug 15, 2009 at 03:21:27PM -0700, David Wagner wrote:
>
>In a nutshell, they exploit the fact that many files in /proc are
>world-readable yet contain sensitive information that can leak information
>about inter-keystroke timings.  For instance, /proc/$PID/stat reveals the
>ESP and EIP registers of the associated process, and is world-readable.
>/proc/pid/status is also mentioned as revealing information that could
>be exploited in these attacks.
>
>Based on my understanding of their work, it sounds like some of
>the information on those files should perhaps not be world-readable.
>It's not clear to me that it's reasonable for the kernel to reveal ESP,
>EIP, and other sensitive information about process behavior to everyone
>on the same system.


Nope, just make sure EIP, ESP, WCHAN etc. info will not be leaked.


>
>Are folks already aware of these vulnerabilities?

Yes, we have already fixed this in commit f83ce3 by Jake, so EIP, ESP
and WCHAN will not be leaked for non-privileged users.