Impact of HW vulnerabilities & Implications on Security Vulnerability Process

Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[Lars Kurth]

Dear community members,

A few years ago it was discovered that much of the RAM shipped in our
computers contains flaws which allow "leakage" across rows; effectively
allowing programs to use access to one bit of memory to flip bits in
other parts of memory to which they have been specifically denied
access.  This has attack on faulty hardware has been dubbed "rowhammer" 
[1].

As time has gone on, researchers have continued to refine rowhammer
techniques and attacks, increasing the probability and speed of finding
rowhammer vulnerabilities, their effectiveness at making unauthorized
changes to memory, and the methods of using these changes in order to
escalate privilege.  Last year Google published demonstrated an attack
on Linux [2]; and a recent Usenix conference had three papers containing
rowhammer attacks, one of which demonstrated an attack on KVM [3], and
one of which demonstrated an attack on Xen PV guests [4].  It is only to
be expected that the techniques will continue to advance over time.

[1] http://users.ece.cmu.edu/%7Eyoonguk/papers/kim-isca14.pdf
[2] http://googleprojectzero.blogspot.com/2015/03/exploiting-dram-rowhammer-bug-to-gain.html
[3] https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_razavi.pdf
[4] http://web.cse.ohio-state.edu/~yinqian/papers/sec16.pdf

The underlying bug is clearly not a software bug, which raises a number of technical as well as process questions:
* Process: whether we should handle such vulnerabilities as XSAs
* Technical: to which degree is rowhammer (or similar HW   
  vulnerabilities) affecting Xen, is the threat theoretical or real 
  and are there any technical mitigations we could implement other than 
  not using PV (using the rowhammer example)

I am starting this discussion, as the information is already in the public domain. 

Process
=======
From a process perspective, these new class of hardware vulnerabilities 
raise the question whether the project should treat them as XSAs (even 
if just informational purposes), or not. My understanding of the issue 
is that this is a hardware bug, which cannot be fixed by a Xen software 
patch (please correct me if wrong).

Looking at the process 
[5] https://www.xenproject.org/security-policy.html
the scope of the process currently is "vulnerability in Xen Project software" - see bullet 1 

A general observation in [5] is that the "Introduction" and "Scope of 
this process" sections are a little unclear and could probably be 
improved. In particular with a view of how we deal with vulnerabilities 
in upstreams such as QEMU, Linux, ...), vulnerabilities in experimental/preview code, and HW vulnerabilities, ...

From my perspective, there are a number of differing goals we are trying 
to achieve with the process

a) Ensuring that pre-disclosure members can deploy patches before 
   information becomes public

b) If already public (or at disclosure time), ensure that our users have 
   all the information to make the right choices

In this case, a) does not apply as there are no patches that can be 
applied, but b) would.

However, we are also other considerations/trade-offs

c) Widening the scope of the process, obviously creates more work-load   
   for the security team
   We had some issues with upstream vulnerabilities in QEMU in the past:    
   for a while we did not have a clear policy, which led to a growing
   influx of upstream vulnerabilities being reported
   to security@xenproject instead of the upstream projects. This 
   eventually forced us to tighten our criteria to only cover 
   vulnerabilities of upstream that affect supported configurations.

d) Also, XSAs are monitored by the tech press, and frequently result in 
   media coverage.
   The coverage is not always fair, and often technically incorrect. 
   Treating HW vulnerabilities like XSAs carries the risk that 
   vulnerabilities like rowhammer will be closely linked with
   Xen, due to mistakes by reporters, and ultimately damage the 
   reputation of the project. This would in particular be true if 
   competing projects/commercial products, do not treat Hardware 
   issues in the same way. 

As an aside, there has already been ample coverage on rowhammer/Flip 
Feng Shui attacks, but none of the stories published were Xen specific. 
If we were to publish an XSA, that would VERY LIKELY change. 

Technical
=========
On the technical front, it would be good to understand whether
a) This is a real threat and whether thus, we as a community need to 
   take action 
b) Whether the technique described in [3] is serious on big iron with 
   different core/cache properties compared to some of the machines this 
   was tested on
c) Whether there is any mitigation that we can develop, if necessary

In any case, it is probably a new type of attack vector, that we need to 
consider and keep in mind, when developing new features.

Best Regards
Lars
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[Ian Jackson]

Lars Kurth writes ("Impact of HW vulnerabilities & Implications on Security Vulnerability Process"):
> A few years ago it was discovered that much of the RAM shipped in our
> computers contains flaws which allow "leakage" across rows; effectively
> allowing programs to use access to one bit of memory to flip bits in
> other parts of memory to which they have been specifically denied
> access.  This has attack on faulty hardware has been dubbed "rowhammer" 
> [1].
...

> From my perspective, there are a number of differing goals we are trying 
> to achieve with the process
...
> b) If already public (or at disclosure time), ensure that our users have 
>    all the information to make the right choices

This is my concern.

From my POV XSAs are a convenient established format and process.

However, I don't think this necessarily needs to be dealt with by
issuing an actual XSA, particularly if there are other reasons for
doing things differently.  We could brief our users by writing some
other kind of message, perhaps posted on xen-announce.

Indeed several aspects of the XSA process are not really applicable.

One main reason for issuing an XSA for an ordinary software bug is
that it allows the issue, and its fix, to be tracked in a standardised
way.  CVEs perform the same function, with a more general scope.

But, we would not expect to get a CVE for what really amounts to a
hardware quality issue.  And where there can be little useful way of
avoiding a hardware bug by adding workarounds to the software
(specifically, in our case, by modifying Xen), there is no need to
track whether any particular codebase has the mitigation.

So there is little benefit in assigning a number.

Unlike with software bugs, there is also relatively little benefit in
having rowhammer listed on a web page alongside software bugs.

The XSA advisory template format does not lend itself well to this
issue, as I found when I tried to write a draft.  While does have the
benefit of being in a format which is familiar to users, user response
will have to be quite different.  And the level of uncertainty and
hardware-dependence means that the usual questions such as `Impact'
and `Vulnerable systems' have unsatisfactory non-answers, in such a
bulletin.

We did issue XSA-3 for a mitigationless hardware design problem.  But
that was in a very different environment: the XSA process was not as
formally established as it is now, and the publicity implications were
different.

> Technical
> =========
> On the technical front, it would be good to understand whether
> a) This is a real threat and whether thus, we as a community need to 
>    take action 

It is unclear what action the Xen upstream community can usefully
take, other than providing users with information.

But, users with deployments on actual hardware ought to try to find
out whether they are vulnerable.  If they are then they could seek
replacement non-faulty hardware from their vendor, or take unpleasant
migitation measures (like switching to HVM, perhaps).

> b) Whether the technique described in [3] is serious on big iron with 
>    different core/cache properties compared to some of the machines this 
>    was tested on

This is a big question.

> c) Whether there is any mitigation that we can develop, if necessary

AIUI there is little to be done.  But, I look forward to being proven
wrong.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[Jan Beulich]

>>> On 07.09.16 at 17:33, <ian.jackson@eu.citrix.com> wrote:
> Lars Kurth writes ("Impact of HW vulnerabilities & Implications on Security 
> Vulnerability Process"):
>> c) Whether there is any mitigation that we can develop, if necessary
> 
> AIUI there is little to be done.  But, I look forward to being proven
> wrong.

Well, I think David's PV-with-stub-Xen-in-a-HVM-container outlined
on the last hackathon would be a possible aid, albeit presumably
one requiring quite a bit of work.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[George Dunlap]

On 07/09/16 16:33, Ian Jackson wrote:
> Lars Kurth writes ("Impact of HW vulnerabilities & Implications on Security Vulnerability Process"):
>> A few years ago it was discovered that much of the RAM shipped in our
>> computers contains flaws which allow "leakage" across rows; effectively
>> allowing programs to use access to one bit of memory to flip bits in
>> other parts of memory to which they have been specifically denied
>> access.  This has attack on faulty hardware has been dubbed "rowhammer" 
>> [1].
> ...
> 
>> From my perspective, there are a number of differing goals we are trying 
>> to achieve with the process
> ...
>> b) If already public (or at disclosure time), ensure that our users have 
>>    all the information to make the right choices
> 
> This is my concern.
> 
> From my POV XSAs are a convenient established format and process.
> 
> However, I don't think this necessarily needs to be dealt with by
> issuing an actual XSA, particularly if there are other reasons for
> doing things differently.  We could brief our users by writing some
> other kind of message, perhaps posted on xen-announce.
> 
> Indeed several aspects of the XSA process are not really applicable.
> 
> One main reason for issuing an XSA for an ordinary software bug is
> that it allows the issue, and its fix, to be tracked in a standardised
> way.  CVEs perform the same function, with a more general scope.
> 
> But, we would not expect to get a CVE for what really amounts to a
> hardware quality issue.  And where there can be little useful way of
> avoiding a hardware bug by adding workarounds to the software
> (specifically, in our case, by modifying Xen), there is no need to
> track whether any particular codebase has the mitigation.
> 
> So there is little benefit in assigning a number.
> 
> Unlike with software bugs, there is also relatively little benefit in
> having rowhammer listed on a web page alongside software bugs.
> 
> The XSA advisory template format does not lend itself well to this
> issue, as I found when I tried to write a draft.  While does have the
> benefit of being in a format which is familiar to users, user response
> will have to be quite different.  And the level of uncertainty and
> hardware-dependence means that the usual questions such as `Impact'
> and `Vulnerable systems' have unsatisfactory non-answers, in such a
> bulletin.
> 
> We did issue XSA-3 for a mitigationless hardware design problem.  But
> that was in a very different environment: the XSA process was not as
> formally established as it is now, and the publicity implications were
> different.
> 
>> Technical
>> =========
>> On the technical front, it would be good to understand whether
>> a) This is a real threat and whether thus, we as a community need to 
>>    take action 
> 
> It is unclear what action the Xen upstream community can usefully
> take, other than providing users with information.
> 
> But, users with deployments on actual hardware ought to try to find
> out whether they are vulnerable.  If they are then they could seek
> replacement non-faulty hardware from their vendor, or take unpleasant
> migitation measures (like switching to HVM, perhaps).
> 
>> b) Whether the technique described in [3] is serious on big iron with 
>>    different core/cache properties compared to some of the machines this 
>>    was tested on
> 
> This is a big question.
> 
>> c) Whether there is any mitigation that we can develop, if necessary
> 
> AIUI there is little to be done.  But, I look forward to being proven
> wrong.

The attack described in [4] relies on the fact that the attacker knows
the mfn of the L2 pagetables being used by the hardware.  Using shadows
for the L2+ pagetables would thwart that particular attack, and
shouldn't in theory cause too much overhead.

But that would only thwart a particular attack.  Other attacks are
certain to develop; we can only strongly advise all our users that if
they expect to have determined adversaries inside their guests, that
they should make every effort to use RAM which is not vulnerable to
rowhammer attacks.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[George Dunlap]

On 07/09/16 16:33, Ian Jackson wrote:
> Lars Kurth writes ("Impact of HW vulnerabilities & Implications on Security Vulnerability Process"):
>> A few years ago it was discovered that much of the RAM shipped in our
>> computers contains flaws which allow "leakage" across rows; effectively
>> allowing programs to use access to one bit of memory to flip bits in
>> other parts of memory to which they have been specifically denied
>> access.  This has attack on faulty hardware has been dubbed "rowhammer" 
>> [1].
> ...
> 
>> From my perspective, there are a number of differing goals we are trying 
>> to achieve with the process
> ...
>> b) If already public (or at disclosure time), ensure that our users have 
>>    all the information to make the right choices
> 
> This is my concern.
> 
> From my POV XSAs are a convenient established format and process.
> 
> However, I don't think this necessarily needs to be dealt with by
> issuing an actual XSA, particularly if there are other reasons for
> doing things differently.  We could brief our users by writing some
> other kind of message, perhaps posted on xen-announce.
> 
> Indeed several aspects of the XSA process are not really applicable.
> 
> One main reason for issuing an XSA for an ordinary software bug is
> that it allows the issue, and its fix, to be tracked in a standardised
> way.  CVEs perform the same function, with a more general scope.
> 
> But, we would not expect to get a CVE for what really amounts to a
> hardware quality issue.  And where there can be little useful way of
> avoiding a hardware bug by adding workarounds to the software
> (specifically, in our case, by modifying Xen), there is no need to
> track whether any particular codebase has the mitigation.
> 
> So there is little benefit in assigning a number.
> 
> Unlike with software bugs, there is also relatively little benefit in
> having rowhammer listed on a web page alongside software bugs.
> 
> The XSA advisory template format does not lend itself well to this
> issue, as I found when I tried to write a draft.  While does have the
> benefit of being in a format which is familiar to users, user response
> will have to be quite different.  And the level of uncertainty and
> hardware-dependence means that the usual questions such as `Impact'
> and `Vulnerable systems' have unsatisfactory non-answers, in such a
> bulletin.
> 
> We did issue XSA-3 for a mitigationless hardware design problem.  But
> that was in a very different environment: the XSA process was not as
> formally established as it is now, and the publicity implications were
> different.

What's the conclusion here -- are you inclined to say that we shouldn't
issue an XSA, but perhaps do some other sort of announcement?

 -George


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[Ian Jackson]

George Dunlap writes ("Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process"):
> What's the conclusion here -- are you inclined to say that we shouldn't
> issue an XSA, but perhaps do some other sort of announcement?

I would like us to _either_ issue an XSA or some other sort of
announcement.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[Wei Liu]

On Thu, Sep 08, 2016 at 12:12:22PM +0100, Ian Jackson wrote:
> George Dunlap writes ("Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process"):
> > What's the conclusion here -- are you inclined to say that we shouldn't
> > issue an XSA, but perhaps do some other sort of announcement?
> 
> I would like us to _either_ issue an XSA or some other sort of
> announcement.
> 

+1 for some other sort of announcement.

Wei.

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[Lars Kurth]

> On 8 Sep 2016, at 12:12, Ian Jackson <ian.jackson@eu.citrix.com> wrote:
> 
> George Dunlap writes ("Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process"):
>> What's the conclusion here -- are you inclined to say that we shouldn't
>> issue an XSA, but perhaps do some other sort of announcement?
> 
> I would like us to _either_ issue an XSA or some other sort of
> announcement.

xen-announce@ and XSA's go to the same group of people: with the exception that xen-announce@  may not
cover all people on the pre-disclosure list and we may not hit the people who poll http://xenbits.xen.org/xsa/

I would prefer not to use an XSA, as I laid out before. 
It seems that Ian has a slight preference not to be constrained by the XSA format. 

Using xen-announce@ allows us to set up more context (e.g. including to some of the 
related studies covering other hypervisors, ...). Secondly xen-announce@ is less formal 
and thus the risk that the media will pick it up is significantly lower. 

But I also think that this should contain some practical and useful advice.

Regards
Lars


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[Stefano Stabellini]

On Wed, 7 Sep 2016, Ian Jackson wrote:
> > Technical
> > =========
> > On the technical front, it would be good to understand whether
> > a) This is a real threat and whether thus, we as a community need to 
> >    take action 
> 
> It is unclear what action the Xen upstream community can usefully
> take, other than providing users with information.
> 
> But, users with deployments on actual hardware ought to try to find
> out whether they are vulnerable.  If they are then they could seek
> replacement non-faulty hardware from their vendor, or take unpleasant
> migitation measures (like switching to HVM, perhaps).

How difficult is to check for it?

Is there a simple test, maybe a little executable, that users could use
to find out whether their ram is vulnerable? That would be extremely
valuable.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[Meng Xu]

On Wed, Sep 7, 2016 at 3:08 PM, Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> On Wed, 7 Sep 2016, Ian Jackson wrote:
> > > Technical
> > > =========
> > > On the technical front, it would be good to understand whether
> > > a) This is a real threat and whether thus, we as a community need to
> > >    take action
> >
> > It is unclear what action the Xen upstream community can usefully
> > take, other than providing users with information.
> >
> > But, users with deployments on actual hardware ought to try to find
> > out whether they are vulnerable.  If they are then they could seek
> > replacement non-faulty hardware from their vendor, or take unpleasant
> > migitation measures (like switching to HVM, perhaps).
>
> How difficult is to check for it?
>
> Is there a simple test, maybe a little executable, that users could use
> to find out whether their ram is vulnerable? That would be extremely
> valuable.

Google does have a github repo to do the rowhammer test:
https://github.com/google/rowhammer-test

Meng



-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[Stefano Stabellini]

On Wed, 7 Sep 2016, Meng Xu wrote:
> On Wed, Sep 7, 2016 at 3:08 PM, Stefano Stabellini
> <sstabellini@kernel.org> wrote:
> >
> > On Wed, 7 Sep 2016, Ian Jackson wrote:
> > > > Technical
> > > > =========
> > > > On the technical front, it would be good to understand whether
> > > > a) This is a real threat and whether thus, we as a community need to
> > > >    take action
> > >
> > > It is unclear what action the Xen upstream community can usefully
> > > take, other than providing users with information.
> > >
> > > But, users with deployments on actual hardware ought to try to find
> > > out whether they are vulnerable.  If they are then they could seek
> > > replacement non-faulty hardware from their vendor, or take unpleasant
> > > migitation measures (like switching to HVM, perhaps).
> >
> > How difficult is to check for it?
> >
> > Is there a simple test, maybe a little executable, that users could use
> > to find out whether their ram is vulnerable? That would be extremely
> > valuable.
> 
> Google does have a github repo to do the rowhammer test:
> https://github.com/google/rowhammer-test

Nice! It would be good to document this in a Xen Project document
somewhere.

The code is small enough that we could even consider pulling it in Xen
and running it at boot time (obviously it would be a kconfig option to
compile and a xen command line option to run the test). In case of
failure we could WARN the sysadmin and refuse to continue.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[Andrew Cooper]

On 07/09/2016 22:02, Stefano Stabellini wrote:
> On Wed, 7 Sep 2016, Meng Xu wrote:
>> On Wed, Sep 7, 2016 at 3:08 PM, Stefano Stabellini
>> <sstabellini@kernel.org> wrote:
>>> On Wed, 7 Sep 2016, Ian Jackson wrote:
>>>>> Technical
>>>>> =========
>>>>> On the technical front, it would be good to understand whether
>>>>> a) This is a real threat and whether thus, we as a community need to
>>>>>    take action
>>>> It is unclear what action the Xen upstream community can usefully
>>>> take, other than providing users with information.
>>>>
>>>> But, users with deployments on actual hardware ought to try to find
>>>> out whether they are vulnerable.  If they are then they could seek
>>>> replacement non-faulty hardware from their vendor, or take unpleasant
>>>> migitation measures (like switching to HVM, perhaps).
>>> How difficult is to check for it?
>>>
>>> Is there a simple test, maybe a little executable, that users could use
>>> to find out whether their ram is vulnerable? That would be extremely
>>> valuable.
>> Google does have a github repo to do the rowhammer test:
>> https://github.com/google/rowhammer-test
> Nice! It would be good to document this in a Xen Project document
> somewhere.
>
> The code is small enough that we could even consider pulling it in Xen
> and running it at boot time (obviously it would be a kconfig option to
> compile and a xen command line option to run the test). In case of
> failure we could WARN the sysadmin and refuse to continue.

-10 for any code in the hypervisor.  That is a knee-jerk reaction.  By
the same logic, we should also embed memtest and run that during boot.

Documentation, and making a test easy to run is of course fine, but
noone will compile it in or use it in production.  In the best case, it
comes back quickly and identifies that you can't run untrusted guests,
and in the worse case runs forever.

It is very unfortunate that the very design and intention of the x86 PV
ABI makes it easier for most attackers than a typical user/kernel
interaction.  Hindsight is great, but there is no possible way that a
group of PHD students 15 years ago developing a mutually-cooperative
method of virtualisation could have predicted that in this modern day
and age, commercial DRAM is pushed too close to the limit to function
correctly in all cases software can provoke.

That said, there are both software and hardware mitigations which can be
used.  ECC RAM is a good start, and anyone running a production system
without it has already lost.  There is TRR as an optional DRAM mechanism
which I can see becoming de-facto standard (and likely non-optional in
DDR5).  More intellegent placement of guest memory could prevent it
performing rowhammer on anything but itself, and the proposed option to
move PV guests into an HVM stub container resolves the physical
information leakage which makes rowhammer particularly easy for PV guests.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[Stefano Stabellini]

On Wed, 7 Sep 2016, Andrew Cooper wrote:
> On 07/09/2016 22:02, Stefano Stabellini wrote:
> > On Wed, 7 Sep 2016, Meng Xu wrote:
> >> On Wed, Sep 7, 2016 at 3:08 PM, Stefano Stabellini
> >> <sstabellini@kernel.org> wrote:
> >>> On Wed, 7 Sep 2016, Ian Jackson wrote:
> >>>>> Technical
> >>>>> =========
> >>>>> On the technical front, it would be good to understand whether
> >>>>> a) This is a real threat and whether thus, we as a community need to
> >>>>>    take action
> >>>> It is unclear what action the Xen upstream community can usefully
> >>>> take, other than providing users with information.
> >>>>
> >>>> But, users with deployments on actual hardware ought to try to find
> >>>> out whether they are vulnerable.  If they are then they could seek
> >>>> replacement non-faulty hardware from their vendor, or take unpleasant
> >>>> migitation measures (like switching to HVM, perhaps).
> >>> How difficult is to check for it?
> >>>
> >>> Is there a simple test, maybe a little executable, that users could use
> >>> to find out whether their ram is vulnerable? That would be extremely
> >>> valuable.
> >> Google does have a github repo to do the rowhammer test:
> >> https://github.com/google/rowhammer-test
> > Nice! It would be good to document this in a Xen Project document
> > somewhere.
> >
> > The code is small enough that we could even consider pulling it in Xen
> > and running it at boot time (obviously it would be a kconfig option to
> > compile and a xen command line option to run the test). In case of
> > failure we could WARN the sysadmin and refuse to continue.
> 
> -10 for any code in the hypervisor.  That is a knee-jerk reaction.  By
> the same logic, we should also embed memtest and run that during boot.

The difference is that errors typically caught by memtest, such as bad
ram regions, don't end up causing privilege escalations.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Impact of HW vulnerabilities & Implications on Security Vulnerability Process

[George Dunlap]

On 07/09/16 22:02, Stefano Stabellini wrote:
> On Wed, 7 Sep 2016, Meng Xu wrote:
>> On Wed, Sep 7, 2016 at 3:08 PM, Stefano Stabellini
>> <sstabellini@kernel.org> wrote:
>>>
>>> On Wed, 7 Sep 2016, Ian Jackson wrote:
>>>>> Technical
>>>>> =========
>>>>> On the technical front, it would be good to understand whether
>>>>> a) This is a real threat and whether thus, we as a community need to
>>>>>    take action
>>>>
>>>> It is unclear what action the Xen upstream community can usefully
>>>> take, other than providing users with information.
>>>>
>>>> But, users with deployments on actual hardware ought to try to find
>>>> out whether they are vulnerable.  If they are then they could seek
>>>> replacement non-faulty hardware from their vendor, or take unpleasant
>>>> migitation measures (like switching to HVM, perhaps).
>>>
>>> How difficult is to check for it?
>>>
>>> Is there a simple test, maybe a little executable, that users could use
>>> to find out whether their ram is vulnerable? That would be extremely
>>> valuable.
>>
>> Google does have a github repo to do the rowhammer test:
>> https://github.com/google/rowhammer-test
> 
> Nice! It would be good to document this in a Xen Project document
> somewhere.
> 
> The code is small enough that we could even consider pulling it in Xen
> and running it at boot time (obviously it would be a kconfig option to
> compile and a xen command line option to run the test). In case of
> failure we could WARN the sysadmin and refuse to continue.

The rowhammer test takes a long time; on the order of an hour or two.  I
don't think people would appreciate those kinds of boot times. ;-)

Additionally, the default version in the Google repo randomly corrupts
memory -- potentially including Xen memory.  And if you have ECC memory,
the result of an uncorrestable error is often a machine reboot.  So
there would be a risk that adding such a test on a vulnerable system
would cause Xen to always reboot; or worse, to boot but after having
corrupted its own data or text segments.

I've been playing around with it, but "unfortunately" both my test
machine and the machine under my desk have ECC RAM.  I ran the
double-sided rowhammer test for 3 hours yesterday on the machine under
my desk, and the Linux EDAC driver didn't report any errors corrected.
This could either be because no errors happened, or because the errors
weren't being reported to Linux.  If no errors happened, it could be
because I'm not vulnerable, or because the test doesn't work on my hardware.

So unfortunately, there are just too many unknowns at this point to give
useful advice, other than "ECC RAM is probably better than non-ECC RAM".

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel