Clarification regarding Meltdown and 64-bit PV guests

Clarification regarding Meltdown and 64-bit PV guests

[Andy Smith]

Hi,

In
<https://blog.xenproject.org/2018/01/04/xen-project-spectremeltdown-faq/>:

    "On Intel processors, only 64-bit PV mode guests can attack Xen
    using Variant 3. Guests running in 32-bit PV mode, HVM mode, and
    PVH mode (both v1 and v2) cannot attack the hypervisor using
    Variant 3. However, in 32-bit PV mode, HVM mode, and PVH mode
    (both v1 and v2), guest userspaces can attack guest kernels
    using Variant 3; so updating guest kernels is advisable.

    Interestingly, guest kernels running in 64-bit PV mode are not
    vulnerable to attack using Variant 3, because 64-bit PV guests
    already run in a KPTI-like mode."

However, in multiple other places, including
<https://xenbits.xen.org/xsa/xsa254/README.comet> and
<https://xenbits.xen.org/xsa/xsa254/README.vixen>:

    "Note that both of these shim-based approaches prevent attacks on
    the host, but leave the guest vulnerable to Meltdown attacks by
    its own unprivileged processes; this is true even if the guest
    OS has KPTI or similar Meltdown mitigation."

These seem to contradict each other.

The FAQ seems to suggest that:

- 32-bit PV guest userland processes can use Variant 3 against their
  own kernels and that the KPTI patch would protect against that.

- Without Comet/Vixen, 64-bit PV guests can't use Variant 3 on
  themselves but can use it on the hypervisor, and KPTI patches in
  the guest do not prevent that.

- Running PV guests inside Comet or Vixen prevents them making use
  of Variant 3, they still cannot use Variant 3 against their own
  kernels, and KPTI patches in the guest are not necessary.

The Comet and Vixen READMEs seem to suggest that:

- Use of Comet/Vixen prevents PV guests from using Variant 3 against
  the hypervisor (and thus other guests as well).

- The guest itself remains able to use Variant 3 on its own kernel
  and KPTI patches inside the guest cannot prevent this.

Which is correct, or have I misunderstood and they are somehow both
correct?

Cheers,
Andy

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Clarification regarding Meltdown and 64-bit PV guests

[Hans van Kranenburg]

On 01/13/2018 07:42 AM, Andy Smith wrote:
> Hi,
> 
> In
> <https://blog.xenproject.org/2018/01/04/xen-project-spectremeltdown-faq/>:
> 
>     "On Intel processors, only 64-bit PV mode guests can attack Xen
>     using Variant 3. [...]

64bit PV domU userspace can attack the hypervisor -> read "host memory"
which also contains contents of guest memory (including itself)

>     Interestingly, guest kernels running in 64-bit PV mode are not
>     vulnerable to attack using Variant 3, because 64-bit PV guests
>     already run in a KPTI-like mode."

64-bit PV domU userspace can't attack kernel inside guest
Quoting Andrew C (on IRC): "the 64bit PV ABI already has split
guest-user / guest-kernel pagetables; (because 64bit x86 took away
working segment limits from the 32bit world); it mitigates guest
userspace SP3 attacks against the guest kernel"

This is also the reason for adding the if-statement which prevents the
new kpti code from becoming active when booting your latest kernel in a
64-bit PV domU.

> However, in multiple other places, including
> <https://xenbits.xen.org/xsa/xsa254/README.comet> and
> <https://xenbits.xen.org/xsa/xsa254/README.vixen>:
> 
>     "Note that both of these shim-based approaches prevent attacks on
>     the host, but leave the guest vulnerable to Meltdown attacks by
>     its own unprivileged processes; this is true even if the guest
>     OS has KPTI or similar Meltdown mitigation."

So, even while it can't attack itself inside the guest, it can make a
"detour" over the hypervisor to again read its own memory.

> These seem to contradict each other.
> 
> The FAQ seems to suggest that:
> 
> - 32-bit PV guest userland processes can use Variant 3 against their
>   own kernels and that the KPTI patch would protect against that.
> 
> - Without Comet/Vixen, 64-bit PV guests can't use Variant 3 on
>   themselves but can use it on the hypervisor, and KPTI patches in
>   the guest do not prevent that.

^^

> - Running PV guests inside Comet or Vixen prevents them making use
>   of Variant 3, they still cannot use Variant 3 against their own
>   kernels, and KPTI patches in the guest are not necessary.

Like it was for a long time already, because of the "64bit PV ABI
already has split [...]"

> The Comet and Vixen READMEs seem to suggest that:
> 
> - Use of Comet/Vixen prevents PV guests from using Variant 3 against
>   the hypervisor (and thus other guests as well).

By injecting a copy of a hypervisor between the outer level hypervisor
(that's called L0 right?) (in HVM or PVH mode) and the guest, having it
just run 1 guest, that (64-bit PV) guest cannot attack its own kernel,
but it can attack the intermediate hypervisor which results in reading
it's own memory from the fake intermediate "host memory".

> - The guest itself remains able to use Variant 3 on its own kernel
>   and KPTI patches inside the guest cannot prevent this.

This one should be incorrect.

> Which is correct, or have I misunderstood and they are somehow both
> correct?

It's complicated. At least, the above is what I understood.

Hans

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Clarification regarding Meltdown and 64-bit PV guests

[Andy Smith]

Hi Hans,

On Sat, Jan 13, 2018 at 10:43:03AM +0100, Hans van Kranenburg wrote:
> By injecting a copy of a hypervisor between the outer level hypervisor
> (that's called L0 right?) (in HVM or PVH mode) and the guest, having it
> just run 1 guest, that (64-bit PV) guest cannot attack its own kernel,
> but it can attack the intermediate hypervisor which results in reading
> it's own memory from the fake intermediate "host memory".

So are you saying that, considering only SP3/Variant 3/Meltdown, it
works out like this:

== 64-bit PV mode guest ==

- Can't use SP3/Variant 3/Meltdown directly on its own kernel.

- Can use SP3/Variant 3/Meltdown on the hypervisor to read data from
  hypervisor so effectively everything including other kernels and
  its own kernel.

- Can't be mitigated by KPTI in the guest.

== PV-in-Comet and PV-in-Vixen ==

- Can't use SP3/Variant 3/Meltdown directly on its own kernel

- Can't use SP3/Variant 3/Meltdown on the real hypervisor.

- Can still use SP3/Variant 3/Meltdown on the shim hypervisor to
  still gain access to data from itself.

- Can't be mitigated by KPTI in the guest.

== HVM and PVHv2 ==

- Can use SP3/Variant 3/Meltdown directly on its own kernel.

- Can't use SP3/Variant 3/Meltdown on the hypervisor.

- Can be mitigated by KPTI in the guest (becomes not a Xen issue).

?

If so, then I can see how the FAQ, README.Comet and README.Vixen
can all be correct in this regard, but do note that this is
extremely confusing and a lot of people are only reading the
comments that say that Xen PV can't make use of SP3/Variant
3/Meltdown.

Cheers,
Andy

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Clarification regarding Meltdown and 64-bit PV guests

[Hans van Kranenburg]

On 01/13/2018 11:08 AM, Andy Smith wrote:
> Hi Hans,
> 
> On Sat, Jan 13, 2018 at 10:43:03AM +0100, Hans van Kranenburg wrote:
>> By injecting a copy of a hypervisor between the outer level hypervisor
>> (that's called L0 right?) (in HVM or PVH mode) and the guest, having it
>> just run 1 guest, that (64-bit PV) guest cannot attack its own kernel,
>> but it can attack the intermediate hypervisor which results in reading
>> it's own memory from the fake intermediate "host memory".
> 
> So are you saying that, considering only SP3/Variant 3/Meltdown, it
> works out like this:
> 
> == 64-bit PV mode guest ==
> 
> - Can't use SP3/Variant 3/Meltdown directly on its own kernel.
> 
> - Can use SP3/Variant 3/Meltdown on the hypervisor to read data from
>   hypervisor so effectively everything including other kernels and
>   its own kernel.
> 
> - Can't be mitigated by KPTI in the guest.
> 
> == PV-in-Comet and PV-in-Vixen ==
> 
> - Can't use SP3/Variant 3/Meltdown directly on its own kernel
> 
> - Can't use SP3/Variant 3/Meltdown on the real hypervisor.
> 
> - Can still use SP3/Variant 3/Meltdown on the shim hypervisor to
>   still gain access to data from itself.
> 
> - Can't be mitigated by KPTI in the guest.
> 
> == HVM and PVHv2 ==
> 
> - Can use SP3/Variant 3/Meltdown directly on its own kernel.
> 
> - Can't use SP3/Variant 3/Meltdown on the hypervisor.
> 
> - Can be mitigated by KPTI in the guest (becomes not a Xen issue).
> 
> ?

Exactly.

> If so, then I can see how the FAQ, README.Comet and README.Vixen
> can all be correct in this regard,

Well, what just happened is that I didn't provide any new information,
but presented the same information by just rewording it again, which
triggered a few missing connections to complete your mental image.

> but do note that this is extremely confusing

Yes it is.

A. Different types of Xen versions:
  1. <= 4.5, which is EOL and EOS, but still used (like AWS popping up
having things running based on Xen 3.4)
  2. 4.6 and 4.7, which have some security support
  3. 4.8 and 4.9, for which rumours are going that it might get PVHv2
from 4.10 backported
  4. 4.10, which can be used with PVHv2 (with guest kernel linux 4.14)

B. Different archs:
  1. x86 / 32-bit
  2. x64 / amd64 / 64-bit
  3. Add ARM things here...
  4. ...

C. Different CPU vendors:
  1. Intel
  2. AMD
  3. ...

D. Then different virtualization modes:
  1. PV
  2. HVM
  3. PVHv2

E. Different mitigation techniques for Xen (which are each only possible
with a subset of choices from other categories):
  1. Convert PV -> HVM
  2. Convert PV -> PVHv2
  3. Insert shim Vixen
  4. Insert shim Comet

F. Different kernels in the guest (only looking at linux here now):
  1. Any kernel released < Jan 3 2018
  2. Old version without any kaiser/kpti patch available
  3. Old version with kaiser backport patch (3.2, 3.16, 4.9 etc)
  4. 4.14 or 4.15 kernel with KPTI patch

G. Different mitigation techniques for inside the guest:
  1. Do nothing because 64-bit PV guest
  2. Get kernel with kaiser or kpti patch

H. And of course describing any situation as "vulnerable" has a
shortcoming, because we need to distinguish (4 combinations possible):
  1. Can the situation be used to attack?
  2. Is the situation vulnerable to attack by another guest/whatever?

ad H. Oh, and what if someone has a mix of HVM and PV guests? If there
are only HVM guests, they're safe from each other, but if you add 1
64-bit PV guest to it, suddenly you have one available to do an attack,
and all the HVM are suddenly vulnerable.

So it's a bit of an 7 or 8-dimensional space, and every situation of
different users is floating somewhere in there as a point. Every
combination of all of those can result in a wildly different outcome.

Oh wait, I forgot we're only talking about SP3. Let's add SP1 and SP2,
now we are 9-dimensional.

You know these kind of puzzles?
http://www.burre.nl/p/puzzels/logisch/logisch_1.gif

It's what we're doing here. :-) Well, instead of a boolean result, our
puzzle has multiple results, describing what's relevant for hypervisor,
for guest kernel etc...

> and a lot of people are only reading the
> comments that say that Xen PV can't make use of SP3/Variant
> 3/Meltdown.

Expressing all these things in text is really hard.

Lars made a new attempt with the tables, but you immediately see it
takes 4 or 5 tables below each other, and then it's still like "argh!"
because we can't flatten 8-dimensional to 2-dimensional that easy.

------------ >8 ------------

So, I have a new idea:

Let's make an interactive html/javascript thingie where you can play
around with all the combinations above. Based on everything you choose,
there's a different explanation as result, and different suggestions
about what to do next...

Some of the categories are radio buttons, like the CPU type of the
physical server. Others are checkboxes with multiple options, e.g. are
you running HVM guests, PV guests, or both?

I'm not a html/javascript guy, but I think that makes most sense, since
a user can also just git clone the thing and open the page in a local
browser. If there's some initial code that works, I can help adding all
options and results and then we can keep improving it. (E.g. if
limitations of shim implementations change or xen 4.8 and 4.9 get PVHv2
backported, it has to change again etc...)

Nice to have: If you have everything checked in the right way, show a
"code" (like A3-B1-C5-D6... or something more clever) that expresses the
exact combination, so you can save it and put it back in later to reset
the page to that state. And, you can share that code/sequence to explain
your exact situation to someone else.

Now we would have something that's easier to work with than hearing a
user out with 14 questions on IRC and then trying to explain everything
in words. Any change you make on the page refreshes the outcome immediately.

So, who has a better idea than this, or knows why this is a bad idea and
we shouldn't do this, or who wants to help creating it?

Hans

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Clarification regarding Meltdown and 64-bit PV guests

[Dongli Zhang]

Hi Hans,

On 01/13/2018 07:12 PM, Hans van Kranenburg wrote:
> On 01/13/2018 11:08 AM, Andy Smith wrote:
>> Hi Hans,
>>
>> On Sat, Jan 13, 2018 at 10:43:03AM +0100, Hans van Kranenburg wrote:
>>> By injecting a copy of a hypervisor between the outer level hypervisor
>>> (that's called L0 right?) (in HVM or PVH mode) and the guest, having it
>>> just run 1 guest, that (64-bit PV) guest cannot attack its own kernel,
>>> but it can attack the intermediate hypervisor which results in reading
>>> it's own memory from the fake intermediate "host memory".
>>
>> So are you saying that, considering only SP3/Variant 3/Meltdown, it
>> works out like this:
>>
>> == 64-bit PV mode guest ==
>>
>> - Can't use SP3/Variant 3/Meltdown directly on its own kernel.
>>
>> - Can use SP3/Variant 3/Meltdown on the hypervisor to read data from
>>   hypervisor so effectively everything including other kernels and
>>   its own kernel.
>>
>> - Can't be mitigated by KPTI in the guest.
>>
>> == PV-in-Comet and PV-in-Vixen ==
>>
>> - Can't use SP3/Variant 3/Meltdown directly on its own kernel
>>
>> - Can't use SP3/Variant 3/Meltdown on the real hypervisor.
>>
>> - Can still use SP3/Variant 3/Meltdown on the shim hypervisor to
>>   still gain access to data from itself.
>>
>> - Can't be mitigated by KPTI in the guest.
>>
>> == HVM and PVHv2 ==
>>
>> - Can use SP3/Variant 3/Meltdown directly on its own kernel.
>>
>> - Can't use SP3/Variant 3/Meltdown on the hypervisor.
>>
>> - Can be mitigated by KPTI in the guest (becomes not a Xen issue).
>>
>> ?
> 
> Exactly.

Does this indicate that there is no solution to prevent a malicious user space
program (running on 64-bit PV domU) from reading the memory of domU kernel space
(via meltdown), no matter whether comet/vixen is enabled or not?

Therefore, comet/vixen is only used to prevent the cross-VM meltdown attack.

> 
>> If so, then I can see how the FAQ, README.Comet and README.Vixen
>> can all be correct in this regard,
> 
> Well, what just happened is that I didn't provide any new information,
> but presented the same information by just rewording it again, which
> triggered a few missing connections to complete your mental image.
> 
>> but do note that this is extremely confusing
> 
> Yes it is.
> 
> A. Different types of Xen versions:
>   1. <= 4.5, which is EOL and EOS, but still used (like AWS popping up
> having things running based on Xen 3.4)
>   2. 4.6 and 4.7, which have some security support
>   3. 4.8 and 4.9, for which rumours are going that it might get PVHv2
> from 4.10 backported
>   4. 4.10, which can be used with PVHv2 (with guest kernel linux 4.14)
> 
> B. Different archs:
>   1. x86 / 32-bit
>   2. x64 / amd64 / 64-bit
>   3. Add ARM things here...
>   4. ...
> 
> C. Different CPU vendors:
>   1. Intel
>   2. AMD
>   3. ...
> 
> D. Then different virtualization modes:
>   1. PV
>   2. HVM
>   3. PVHv2
> 
> E. Different mitigation techniques for Xen (which are each only possible
> with a subset of choices from other categories):
>   1. Convert PV -> HVM
>   2. Convert PV -> PVHv2
>   3. Insert shim Vixen
>   4. Insert shim Comet
> 
> F. Different kernels in the guest (only looking at linux here now):
>   1. Any kernel released < Jan 3 2018
>   2. Old version without any kaiser/kpti patch available
>   3. Old version with kaiser backport patch (3.2, 3.16, 4.9 etc)
>   4. 4.14 or 4.15 kernel with KPTI patch
> 
> G. Different mitigation techniques for inside the guest:
>   1. Do nothing because 64-bit PV guest
>   2. Get kernel with kaiser or kpti patch
> 
> H. And of course describing any situation as "vulnerable" has a
> shortcoming, because we need to distinguish (4 combinations possible):
>   1. Can the situation be used to attack?
>   2. Is the situation vulnerable to attack by another guest/whatever?
> 
> ad H. Oh, and what if someone has a mix of HVM and PV guests? If there
> are only HVM guests, they're safe from each other, but if you add 1
> 64-bit PV guest to it, suddenly you have one available to do an attack,
> and all the HVM are suddenly vulnerable.
> 
> So it's a bit of an 7 or 8-dimensional space, and every situation of
> different users is floating somewhere in there as a point. Every
> combination of all of those can result in a wildly different outcome.
> 
> Oh wait, I forgot we're only talking about SP3. Let's add SP1 and SP2,
> now we are 9-dimensional.
> 
> You know these kind of puzzles?
> http://www.burre.nl/p/puzzels/logisch/logisch_1.gif
> 
> It's what we're doing here. :-) Well, instead of a boolean result, our
> puzzle has multiple results, describing what's relevant for hypervisor,
> for guest kernel etc...
> 
>> and a lot of people are only reading the
>> comments that say that Xen PV can't make use of SP3/Variant
>> 3/Meltdown.
> 
> Expressing all these things in text is really hard.
> 
> Lars made a new attempt with the tables, but you immediately see it
> takes 4 or 5 tables below each other, and then it's still like "argh!"
> because we can't flatten 8-dimensional to 2-dimensional that easy.
> 
> ------------ >8 ------------
> 
> So, I have a new idea:
> 
> Let's make an interactive html/javascript thingie where you can play
> around with all the combinations above. Based on everything you choose,
> there's a different explanation as result, and different suggestions
> about what to do next...
> 
> Some of the categories are radio buttons, like the CPU type of the
> physical server. Others are checkboxes with multiple options, e.g. are
> you running HVM guests, PV guests, or both?
> 
> I'm not a html/javascript guy, but I think that makes most sense, since
> a user can also just git clone the thing and open the page in a local
> browser. If there's some initial code that works, I can help adding all
> options and results and then we can keep improving it. (E.g. if
> limitations of shim implementations change or xen 4.8 and 4.9 get PVHv2
> backported, it has to change again etc...)
> 
> Nice to have: If you have everything checked in the right way, show a
> "code" (like A3-B1-C5-D6... or something more clever) that expresses the
> exact combination, so you can save it and put it back in later to reset
> the page to that state. And, you can share that code/sequence to explain
> your exact situation to someone else.
> 
> Now we would have something that's easier to work with than hearing a
> user out with 14 questions on IRC and then trying to explain everything
> in words. Any change you make on the page refreshes the outcome immediately.
> 
> So, who has a better idea than this, or knows why this is a bad idea and
> we shouldn't do this, or who wants to help creating it?
> 
> Hans
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel
> 

Dongli Zhang

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Clarification regarding Meltdown and 64-bit PV guests

[Hans van Kranenburg]

On 14/01/2018 15:00, Dongli Zhang wrote:
> Hi Hans,
> 
> On 01/13/2018 07:12 PM, Hans van Kranenburg wrote:
>> On 01/13/2018 11:08 AM, Andy Smith wrote:
>>> Hi Hans,
>>>
>>> On Sat, Jan 13, 2018 at 10:43:03AM +0100, Hans van Kranenburg wrote:
>>>> By injecting a copy of a hypervisor between the outer level hypervisor
>>>> (that's called L0 right?) (in HVM or PVH mode) and the guest, having it
>>>> just run 1 guest, that (64-bit PV) guest cannot attack its own kernel,
>>>> but it can attack the intermediate hypervisor which results in reading
>>>> it's own memory from the fake intermediate "host memory".
>>>
>>> So are you saying that, considering only SP3/Variant 3/Meltdown, it
>>> works out like this:
>>>
>>> == 64-bit PV mode guest ==
>>>
>>> - Can't use SP3/Variant 3/Meltdown directly on its own kernel.
>>>
>>> - Can use SP3/Variant 3/Meltdown on the hypervisor to read data from
>>>   hypervisor so effectively everything including other kernels and
>>>   its own kernel.
>>>
>>> - Can't be mitigated by KPTI in the guest.
>>>
>>> == PV-in-Comet and PV-in-Vixen ==
>>>
>>> - Can't use SP3/Variant 3/Meltdown directly on its own kernel
>>>
>>> - Can't use SP3/Variant 3/Meltdown on the real hypervisor.
>>>
>>> - Can still use SP3/Variant 3/Meltdown on the shim hypervisor to
>>>   still gain access to data from itself.
>>>
>>> - Can't be mitigated by KPTI in the guest.
>>>
>>> == HVM and PVHv2 ==
>>>
>>> - Can use SP3/Variant 3/Meltdown directly on its own kernel.
>>>
>>> - Can't use SP3/Variant 3/Meltdown on the hypervisor.
>>>
>>> - Can be mitigated by KPTI in the guest (becomes not a Xen issue).
>>>
>>> ?
>>
>> Exactly.
> 
> Does this indicate that there is no solution to prevent a malicious user space
> program (running on 64-bit PV domU) from reading the memory of domU kernel space
> (via meltdown), no matter whether comet/vixen is enabled or not?
> 
> Therefore, comet/vixen is only used to prevent the cross-VM meltdown attack.

Yes.

For reference, I looked back at my IRC logs. This is the related
conversation about it, #xen IRC on freenode, from Jan 6 2018, when I was
firing off questions to Andrew.

18:17 < Knorrie> with Xen 4.8+ and the pvshim method for non-pvh guest
kernels, is the xen implementation that does the same as KPTI inside the
guest for 64bit PV still present?
18:20 < andyhhp> hmm
18:20 < Knorrie> I think that's an interesting questino :)
18:20 < andyhhp> the 64bit PV ABI already has an isolation between guest
userspace and guest kernel
18:20 < andyhhp> so in effect, has had KPTI for 12 years already...
18:21 < andyhhp> however
18:21 < andyhhp> Xen's mappings are not currently isolated, so either
guest userspace or guest kernel can attack Xen using SP3/Meltdown
18:22 < Knorrie> yes, as I understood the reason of the pvshim feature
was to fix exactly that
18:23 < andyhhp> the Linux KPTI feature should turn itself off when it
detects it is running as a PV guest under Xen
18:24 < andyhhp> but this was quite late to the party, and it appears
that Redhat have published an update to RHEL 6.x which bricks PV guests :(
18:24 < Knorrie> ah right, I remember having seen that if statement one
of these days yes
18:24 < Knorrie> so no need to manually turn  it off in exactly that
scenario
18:26 < andyhhp> the PV-shim doesn't do anything clever
18:26 < andyhhp> all we do is stick a second version of Xen in there
18:26 < andyhhp> so we've got L0 Xen running a PVH guest (so in an HVM
container).
18:26 < andyhhp> that PVH guest is actually a Xen running a single
legacy PV guest
18:27 < Knorrie> well, the clever things is that you can't see outside
of your own memory any more right
18:27 < Knorrie> s/things/thing/
18:27 < andyhhp> the alteration is/should/will be transparent from the
point of view of the legacy PV guest
18:28 < andyhhp> the legacy PV guest can still attempt to pull off
SP3/Meltdown attacks, but they stop at the L1 Xen
18:29 < andyhhp> so all it can do is do is read details relating to
itself.  It can't read anything belonging to other VMs
18:29 < Knorrie> like, outgoing network traffic from other processes on
the same guest?
18:29 < Knorrie> (regardless of how practical and realistic those
attacks are)

---- >8 ----

18:30 < andyhhp> it should be noted that PV-SHIM isn't a perfect
replacement for some kind of Xen based KPTI solution.  A piece of
userspace inside the PV guest can still use SP3 to attack othe
r processes by bypassing the PV guest kernel and attacking via L1 Xen
18:30 < andyhhp> but that is a darn sight better than having said piece
of PV guest userspace being able to read every VM on the entire system

---- >8 ----

18:31 < Knorrie> yes
18:32 < andyhhp> however, I've come to the executive decision that Xen
can't do some kind of KPTI-based solution without a change in the PV ABI
18:32 < andyhhp> because PV guests currently have too much control over
their pagetables for Xen to implement KPTI behind them
18:34 < Knorrie> sounds like this is the moment when PV is going to be
declared dead
18:34 < Knorrie> well
18:34 < Knorrie> of course there's hardware that cannot do PVH
18:35 < Knorrie> but I mean, for who can do PVH whenever they're updated
to recent guest kernels the changes would come even later again, so for
them it's the PVH way
18:36 < Knorrie> and PV remains half-broken for a longer time for who cant
18:36 < andyhhp> PV isn't dead by any means, and I'm going to refuse to
declare it so
18:37 < andyhhp> there are still legitimate (and indeed, modern)
usecases where PV guests will unconditionally always outperform HVM guests
18:38 < andyhhp> but all such usecases have the PV guests at least in
the semi-TCB

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Clarification regarding Meltdown and 64-bit PV guests

[Stefano Stabellini]

On Sun, 14 Jan 2018, Hans van Kranenburg wrote:
> On 14/01/2018 15:00, Dongli Zhang wrote:
> > Hi Hans,
> > 
> > On 01/13/2018 07:12 PM, Hans van Kranenburg wrote:
> >> On 01/13/2018 11:08 AM, Andy Smith wrote:
> >>> Hi Hans,
> >>>
> >>> On Sat, Jan 13, 2018 at 10:43:03AM +0100, Hans van Kranenburg wrote:
> >>>> By injecting a copy of a hypervisor between the outer level hypervisor
> >>>> (that's called L0 right?) (in HVM or PVH mode) and the guest, having it
> >>>> just run 1 guest, that (64-bit PV) guest cannot attack its own kernel,
> >>>> but it can attack the intermediate hypervisor which results in reading
> >>>> it's own memory from the fake intermediate "host memory".
> >>>
> >>> So are you saying that, considering only SP3/Variant 3/Meltdown, it
> >>> works out like this:
> >>>
> >>> == 64-bit PV mode guest ==
> >>>
> >>> - Can't use SP3/Variant 3/Meltdown directly on its own kernel.
> >>>
> >>> - Can use SP3/Variant 3/Meltdown on the hypervisor to read data from
> >>>   hypervisor so effectively everything including other kernels and
> >>>   its own kernel.
> >>>
> >>> - Can't be mitigated by KPTI in the guest.
> >>>
> >>> == PV-in-Comet and PV-in-Vixen ==
> >>>
> >>> - Can't use SP3/Variant 3/Meltdown directly on its own kernel
> >>>
> >>> - Can't use SP3/Variant 3/Meltdown on the real hypervisor.
> >>>
> >>> - Can still use SP3/Variant 3/Meltdown on the shim hypervisor to
> >>>   still gain access to data from itself.
> >>>
> >>> - Can't be mitigated by KPTI in the guest.
> >>>
> >>> == HVM and PVHv2 ==
> >>>
> >>> - Can use SP3/Variant 3/Meltdown directly on its own kernel.
> >>>
> >>> - Can't use SP3/Variant 3/Meltdown on the hypervisor.
> >>>
> >>> - Can be mitigated by KPTI in the guest (becomes not a Xen issue).
> >>>
> >>> ?
> >>
> >> Exactly.
> > 
> > Does this indicate that there is no solution to prevent a malicious user space
> > program (running on 64-bit PV domU) from reading the memory of domU kernel space
> > (via meltdown), no matter whether comet/vixen is enabled or not?
> > 
> > Therefore, comet/vixen is only used to prevent the cross-VM meltdown attack.
> 
> Yes.

Keep an eye on this series, and future follow-ups:

https://marc.info/?l=xen-devel&m=151601415717228

It mitigates Meltdown/SP3 for Xen without introducing an intermediate
hypervisor. Thus, it protects Xen while retaining the property that the
guest kernel is already protected from the guest userspace because it
runs in a KPTI-like mode.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Clarification regarding Meltdown and 64-bit PV guests

[Dongli Zhang]

Hi Hans and Lars,

On 01/13/2018 07:12 PM, Hans van Kranenburg wrote:
> On 01/13/2018 11:08 AM, Andy Smith wrote:
>> Hi Hans,
>>
>> On Sat, Jan 13, 2018 at 10:43:03AM +0100, Hans van Kranenburg wrote:
>>> By injecting a copy of a hypervisor between the outer level hypervisor
>>> (that's called L0 right?) (in HVM or PVH mode) and the guest, having it
>>> just run 1 guest, that (64-bit PV) guest cannot attack its own kernel,
>>> but it can attack the intermediate hypervisor which results in reading
>>> it's own memory from the fake intermediate "host memory".
>>
>> So are you saying that, considering only SP3/Variant 3/Meltdown, it
>> works out like this:
>>
>> == 64-bit PV mode guest ==
>>
>> - Can't use SP3/Variant 3/Meltdown directly on its own kernel.
>>
>> - Can use SP3/Variant 3/Meltdown on the hypervisor to read data from
>>   hypervisor so effectively everything including other kernels and
>>   its own kernel.
>>
>> - Can't be mitigated by KPTI in the guest.
>>
>> == PV-in-Comet and PV-in-Vixen ==
>>
>> - Can't use SP3/Variant 3/Meltdown directly on its own kernel
>>
>> - Can't use SP3/Variant 3/Meltdown on the real hypervisor.
>>
>> - Can still use SP3/Variant 3/Meltdown on the shim hypervisor to
>>   still gain access to data from itself.
>>
>> - Can't be mitigated by KPTI in the guest.
>>
>> == HVM and PVHv2 ==
>>
>> - Can use SP3/Variant 3/Meltdown directly on its own kernel.
>>
>> - Can't use SP3/Variant 3/Meltdown on the hypervisor.
>>
>> - Can be mitigated by KPTI in the guest (becomes not a Xen issue).
>>
>> ?
> 
> Exactly.
> 
>> If so, then I can see how the FAQ, README.Comet and README.Vixen
>> can all be correct in this regard,
> 
> Well, what just happened is that I didn't provide any new information,
> but presented the same information by just rewording it again, which
> triggered a few missing connections to complete your mental image.
> 
>> but do note that this is extremely confusing
> 
> Yes it is.
> 
> A. Different types of Xen versions:
>   1. <= 4.5, which is EOL and EOS, but still used (like AWS popping up
> having things running based on Xen 3.4)
>   2. 4.6 and 4.7, which have some security support
>   3. 4.8 and 4.9, for which rumours are going that it might get PVHv2
> from 4.10 backported
>   4. 4.10, which can be used with PVHv2 (with guest kernel linux 4.14)

I suggest we add the source of attacker as well:

1. Arbitrary user space program on domU
2. Kernel space on domU

> 
> B. Different archs:
>   1. x86 / 32-bit
>   2. x64 / amd64 / 64-bit
>   3. Add ARM things here...
>   4. ...
> 
> C. Different CPU vendors:
>   1. Intel
>   2. AMD
>   3. ...
> 
> D. Then different virtualization modes:
>   1. PV
>   2. HVM
>   3. PVHv2
> 
> E. Different mitigation techniques for Xen (which are each only possible
> with a subset of choices from other categories):
>   1. Convert PV -> HVM
>   2. Convert PV -> PVHv2
>   3. Insert shim Vixen
>   4. Insert shim Comet
> 
> F. Different kernels in the guest (only looking at linux here now):
>   1. Any kernel released < Jan 3 2018
>   2. Old version without any kaiser/kpti patch available
>   3. Old version with kaiser backport patch (3.2, 3.16, 4.9 etc)
>   4. 4.14 or 4.15 kernel with KPTI patch
> 
> G. Different mitigation techniques for inside the guest:
>   1. Do nothing because 64-bit PV guest
>   2. Get kernel with kaiser or kpti patch
> 
> H. And of course describing any situation as "vulnerable" has a
> shortcoming, because we need to distinguish (4 combinations possible):
>   1. Can the situation be used to attack?
>   2. Is the situation vulnerable to attack by another guest/whatever?
> 
> ad H. Oh, and what if someone has a mix of HVM and PV guests? If there
> are only HVM guests, they're safe from each other, but if you add 1
> 64-bit PV guest to it, suddenly you have one available to do an attack,
> and all the HVM are suddenly vulnerable.
> 
> So it's a bit of an 7 or 8-dimensional space, and every situation of
> different users is floating somewhere in there as a point. Every
> combination of all of those can result in a wildly different outcome.
> 
> Oh wait, I forgot we're only talking about SP3. Let's add SP1 and SP2,
> now we are 9-dimensional.
> 
> You know these kind of puzzles?
> http://www.burre.nl/p/puzzels/logisch/logisch_1.gif
> 
> It's what we're doing here. :-) Well, instead of a boolean result, our
> puzzle has multiple results, describing what's relevant for hypervisor,
> for guest kernel etc...
> 
>> and a lot of people are only reading the
>> comments that say that Xen PV can't make use of SP3/Variant
>> 3/Meltdown.
> 
> Expressing all these things in text is really hard.
> 
> Lars made a new attempt with the tables, but you immediately see it
> takes 4 or 5 tables below each other, and then it's still like "argh!"
> because we can't flatten 8-dimensional to 2-dimensional that easy.
> 
> ------------ >8 ------------
> 
> So, I have a new idea:
> 
> Let's make an interactive html/javascript thingie where you can play
> around with all the combinations above. Based on everything you choose,
> there's a different explanation as result, and different suggestions
> about what to do next...
> 
> Some of the categories are radio buttons, like the CPU type of the
> physical server. Others are checkboxes with multiple options, e.g. are
> you running HVM guests, PV guests, or both?
> 
> I'm not a html/javascript guy, but I think that makes most sense, since
> a user can also just git clone the thing and open the page in a local
> browser. If there's some initial code that works, I can help adding all
> options and results and then we can keep improving it. (E.g. if
> limitations of shim implementations change or xen 4.8 and 4.9 get PVHv2
> backported, it has to change again etc...)
> 
> Nice to have: If you have everything checked in the right way, show a
> "code" (like A3-B1-C5-D6... or something more clever) that expresses the
> exact combination, so you can save it and put it back in later to reset
> the page to that state. And, you can share that code/sequence to explain
> your exact situation to someone else.
> 
> Now we would have something that's easier to work with than hearing a
> user out with 14 questions on IRC and then trying to explain everything
> in words. Any change you make on the page refreshes the outcome immediately.
> 
> So, who has a better idea than this, or knows why this is a bad idea and
> we shouldn't do this, or who wants to help creating it?
> 
> Hans
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel
> 

Dongli Zhang

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

What about dom0? (was: Re: Clarification regarding Meltdown and 64-bit PV guests)

[Hans van Kranenburg]

On 13/01/2018 07:42, Andy Smith wrote:
> [...]

Related...

Dom0 is a PV guest with more privileges than a domU, right?

Currently all documentation and FAQ do not contain a section about this.
It's all about the hypervisor and guests.

Probable reason for this is that most people on this list implicitly
think about dom0 as a thing that has as only purpose to manage starting
and stopping domUs, forwarding some network traffic using bridges or
openvswitch, and with its own network connection probably on a separate
management network...

But what about small-deployment-users who use the dom0 for more things,
and have a bunch of Xen domUs running inside. Who knows what they do,
maybe they even have their graphical desktop environment running in dom0
with web browser and everything?

Just throwing some dust in the air... :)

Hans

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel