xen/arm: On chip memory mappings

xen/arm: On chip memory mappings

[Edgar E. Iglesias]

Hi,

I'd like to support the assignment of on chip RAMs to guests, starting with dom0.

The mmio-sram compatible device kinda works already but the 2nd stage MMU
mapping is done with DEVICE memory attributes. This doesn't work well for
SRAMs for several reasons (e.g performance, alignment checks etc).

I guess we could add special treatment of these nodes to create Normal memory mappings.

The rules for combining the memory attributes from S1 and S2 translations
suggest that mapping things at S2 with Normal memory Inner/Outer WB cacheable
would give the guest/S1 flexibility in choosing the final attributes.
It seems to me like guest drivers have the best knowledge to decide how to
map the node memory regions.

Keeping the S2 shareability set to inner (like we already do for memory)
seems to be a good idea though.

So the question I had is, why do we map nodes at S2 with DEVICE attributes at all?
Am I missing something?

Thanks,
Edgar

Re: xen/arm: On chip memory mappings

[Ian Campbell]

On Tue, 2015-05-19 at 15:16 +1000, Edgar E. Iglesias wrote:
> Hi,
> 
> I'd like to support the assignment of on chip RAMs to guests, starting with dom0.
> 
> The mmio-sram compatible device kinda works already but the 2nd stage MMU
> mapping is done with DEVICE memory attributes. This doesn't work well for
> SRAMs for several reasons (e.g performance, alignment checks etc).
> 
> I guess we could add special treatment of these nodes to create Normal memory mappings.

Yes. Ideally we would figure out some sort of automated way to decide
this, but I'm not sure what that would look like.

Otherwise I think we are looking at a list of compatible values which
are mapped as normal memory.

> The rules for combining the memory attributes from S1 and S2 translations
> suggest that mapping things at S2 with Normal memory Inner/Outer WB cacheable
> would give the guest/S1 flexibility in choosing the final attributes.
> It seems to me like guest drivers have the best knowledge to decide how to
> map the node memory regions.
> 
> Keeping the S2 shareability set to inner (like we already do for memory)
> seems to be a good idea though.
> 
> So the question I had is, why do we map nodes at S2 with DEVICE attributes at all?
> Am I missing something?

I think the concern was exposing potentially UNPREDICTABLE /
IMPLEMENTATION DEFINED etc behaviour via a guest which maps MMIO regions
as normal memory in S1. By using a device memory mapping in S2 we force
a safe overall result.

I've not refreshed my memory on the way round this goes though, perhaps
the worry is/was unfounded. In particular perhaps on v8 this ends up as
CONSTRAINED UNPREDICTABLE which might be safe enough (again, I've not
checked).

I'd rather not have v7 and v8 differ in such a fundamental default, but
it might be justified I suppose. Likewise for e.g. doing something
different for dom0/hw-dom vs. others.

Ian.

Re: xen/arm: On chip memory mappings

[Julien Grall]

On 19/05/2015 10:09, Ian Campbell wrote:
> On Tue, 2015-05-19 at 15:16 +1000, Edgar E. Iglesias wrote:
>> Hi,

Hi,

>> The rules for combining the memory attributes from S1 and S2 translations
>> suggest that mapping things at S2 with Normal memory Inner/Outer WB cacheable
>> would give the guest/S1 flexibility in choosing the final attributes.
>> It seems to me like guest drivers have the best knowledge to decide how to
>> map the node memory regions.
>>
>> Keeping the S2 shareability set to inner (like we already do for memory)
>> seems to be a good idea though.
>>
>> So the question I had is, why do we map nodes at S2 with DEVICE attributes at all?
>> Am I missing something?
>
> I think the concern was exposing potentially UNPREDICTABLE /
> IMPLEMENTATION DEFINED etc behaviour via a guest which maps MMIO regions
> as normal memory in S1. By using a device memory mapping in S2 we force
> a safe overall result.
>
> I've not refreshed my memory on the way round this goes though, perhaps
> the worry is/was unfounded. In particular perhaps on v8 this ends up as
> CONSTRAINED UNPREDICTABLE which might be safe enough (again, I've not
> checked).
>
> I'd rather not have v7 and v8 differ in such a fundamental default, but
> it might be justified I suppose. Likewise for e.g. doing something
> different for dom0/hw-dom vs. others.

I remember a similar discussion with Christoffer few months ago (it was 
for ACPI). And the answer was:

"No, real access to MMIO regions of devices must be mapped as device
type in stage-2 if you don't want potential information leaks or weird
things to happen where a guest can tweak and time memory operations
such that they happen in a different context than the VM executing the
memory access.

You can argue that the latter is not necessary for Dom0 as Xen trusts
Dom0 completely, but I would still argue that it is the right approach
to take proper care of it, thus;"

Regards,

-- 
Julien Grall

Re: xen/arm: On chip memory mappings

[Edgar E. Iglesias]

On Tue, May 19, 2015 at 10:23:05AM +0100, Julien Grall wrote:
> 
> 
> On 19/05/2015 10:09, Ian Campbell wrote:
> >On Tue, 2015-05-19 at 15:16 +1000, Edgar E. Iglesias wrote:
> >>Hi,
> 
> Hi,
> 
> >>The rules for combining the memory attributes from S1 and S2 translations
> >>suggest that mapping things at S2 with Normal memory Inner/Outer WB cacheable
> >>would give the guest/S1 flexibility in choosing the final attributes.
> >>It seems to me like guest drivers have the best knowledge to decide how to
> >>map the node memory regions.
> >>
> >>Keeping the S2 shareability set to inner (like we already do for memory)
> >>seems to be a good idea though.
> >>
> >>So the question I had is, why do we map nodes at S2 with DEVICE attributes at all?
> >>Am I missing something?
> >
> >I think the concern was exposing potentially UNPREDICTABLE /
> >IMPLEMENTATION DEFINED etc behaviour via a guest which maps MMIO regions
> >as normal memory in S1. By using a device memory mapping in S2 we force
> >a safe overall result.
> >
> >I've not refreshed my memory on the way round this goes though, perhaps
> >the worry is/was unfounded. In particular perhaps on v8 this ends up as
> >CONSTRAINED UNPREDICTABLE which might be safe enough (again, I've not
> >checked).
> >
> >I'd rather not have v7 and v8 differ in such a fundamental default, but
> >it might be justified I suppose. Likewise for e.g. doing something
> >different for dom0/hw-dom vs. others.
> 
> I remember a similar discussion with Christoffer few months ago (it
> was for ACPI). And the answer was:
> 
> "No, real access to MMIO regions of devices must be mapped as device
> type in stage-2 if you don't want potential information leaks or weird
> things to happen where a guest can tweak and time memory operations
> such that they happen in a different context than the VM executing the
> memory access.
> 
> You can argue that the latter is not necessary for Dom0 as Xen trusts
> Dom0 completely, but I would still argue that it is the right approach
> to take proper care of it, thus;"
> 
> Regards,


Thanks for the pointers,

I agree that fundamental differences like these beteween v7 and v8 wouldn't
be good.

Possible unpredictable behaviour is worrysome...
I'm not aware of anything in the ARM architecture specs that would
cause it in this respect, but I may be missing something.

There might also very well be device/slave specific unpredictability. 
E.g unpredictable behaviour on specific AXI access patterns
(bursts, sizes etc) to specific devices...
On the other hand, I suppose giving direct device access to a guest
carries some kind of trust to behave nicely with the device.

I'm not sure I understand Christoffers arguments though.

A well behaved guest will map it's devices as DEVICE and there
won't be any difference at all wether S2 maps them as dev or mem.

A malicious guest could map things as cached memory and try to cause
cached accesses from other guests to flush out. But these cached accesses
would only contain data for other guests mapped as cacheable memory. AFAICT,
to really hurt another guest, the guest under attack has to participate
in the plot (by incorrectly mapping it's own devs as mem).

Anyway, at the moment it seems like doing a device-tree compatiblity prop
match for mmio-sram would be the path with least resistance...

Cheers,
Edgar

Re: xen/arm: On chip memory mappings

[Julien Grall]

On 20/05/15 14:40, Edgar E. Iglesias wrote:
> Thanks for the pointers,
> 
> I agree that fundamental differences like these beteween v7 and v8 wouldn't
> be good.

I didn't find any fundamental differences for device memory behavior in
the spec.

> Possible unpredictable behaviour is worrysome...
> I'm not aware of anything in the ARM architecture specs that would
> cause it in this respect, but I may be missing something.

AFAICT there is nothing in the spec which describe the behavior of a
region access with wrong memory attribute (i.e normal, device, strong).

I guess this would be described in the memory bus or processor spec.

> There might also very well be device/slave specific unpredictability. 
> E.g unpredictable behaviour on specific AXI access patterns
> (bursts, sizes etc) to specific devices...
> On the other hand, I suppose giving direct device access to a guest
> carries some kind of trust to behave nicely with the device.

The trust to the device is very limited. We got several Xen Security
Advisory ([1] [2] ...) related to PCI passthrough.

I agree that the guest may act badly with the device, although we don't
want to give him the opportunity to act more badly with a device and the
possibility to access another guest memory.

> I'm not sure I understand Christoffers arguments though.

It's not clear what is the behavior of a device memory mapped with
normal attribute. Many issue can appear.

Unless the behavior is clearly written in the spec, we should take the
worst case and not the best.

> A well behaved guest will map it's devices as DEVICE and there
> won't be any difference at all wether S2 maps them as dev or mem.

Agree.

> A malicious guest could map things as cached memory and try to cause
> cached accesses from other guests to flush out. But these cached accesses
> would only contain data for other guests mapped as cacheable memory. AFAICT,
> to really hurt another guest, the guest under attack has to participate
> in the plot (by incorrectly mapping it's own devs as mem).

Why not RAM?

> Anyway, at the moment it seems like doing a device-tree compatiblity prop
> match for mmio-sram would be the path with least resistance...

I think this is a good solution until we figure out the behavior of
device memory mapped with wrong attribute.

Regards,

[1] http://xenbits.xen.org/xsa/advisory-124.html
[2] http://xenbits.xen.org/xsa/advisory-126.html

-- 
Julien Grall

Re: xen/arm: On chip memory mappings

[Edgar E. Iglesias]

On Wed, May 20, 2015 at 03:12:24PM +0100, Julien Grall wrote:
> On 20/05/15 14:40, Edgar E. Iglesias wrote:
> > Thanks for the pointers,
> > 
> > I agree that fundamental differences like these beteween v7 and v8 wouldn't
> > be good.
> 
> I didn't find any fundamental differences for device memory behavior in
> the spec.
> 
> > Possible unpredictable behaviour is worrysome...
> > I'm not aware of anything in the ARM architecture specs that would
> > cause it in this respect, but I may be missing something.
> 
> AFAICT there is nothing in the spec which describe the behavior of a
> region access with wrong memory attribute (i.e normal, device, strong).
> 
> I guess this would be described in the memory bus or processor spec.
> 
> > There might also very well be device/slave specific unpredictability. 
> > E.g unpredictable behaviour on specific AXI access patterns
> > (bursts, sizes etc) to specific devices...
> > On the other hand, I suppose giving direct device access to a guest
> > carries some kind of trust to behave nicely with the device.
> 
> The trust to the device is very limited. We got several Xen Security
> Advisory ([1] [2] ...) related to PCI passthrough.
> 
> I agree that the guest may act badly with the device, although we don't
> want to give him the opportunity to act more badly with a device and the
> possibility to access another guest memory.
> 
> > I'm not sure I understand Christoffers arguments though.
> 
> It's not clear what is the behavior of a device memory mapped with
> normal attribute. Many issue can appear.

I think this is very device-specific.

For example, if SW is writing to a FIFO register but accesses are
getting buffered and combined, data-loss may happen (the cache consumes
multiple repeated stores into one single final store). That is pretty
harmless but for example if a slave is buggy or not protocol complete,
maybe not supporting specific AXI burst patterns and we allow guests to
(via caches) send such transactions to those slaves, things could go bad
in terms of locking up the device or maybe even the CPU. But not in terms
of getting access to other guests memory, AFAICT.

It sounds to me like this is more a question of trust model, i.e how
much do we "trust" guests with direct-device access.

Sounds like we should stick with the current mapping approach to play it
safe.


> Unless the behavior is clearly written in the spec, we should take the
> worst case and not the best.

Agreed.

> 
> > A well behaved guest will map it's devices as DEVICE and there
> > won't be any difference at all wether S2 maps them as dev or mem.
> 
> Agree.
> 
> > A malicious guest could map things as cached memory and try to cause
> > cached accesses from other guests to flush out. But these cached accesses
> > would only contain data for other guests mapped as cacheable memory. AFAICT,
> > to really hurt another guest, the guest under attack has to participate
> > in the plot (by incorrectly mapping it's own devs as mem).
> 
> Why not RAM?

Because RAM is already OK to map as Normal cached memory. I.e, there are no
side-effects, prefetching is OK, buffering and combining is OK. The malicious
guest cannot reach cached memory for another guest because the MMUs/SMMUs will
shoot down those accesses.

If the well-behaved guest for some reason needs to map memory with less caching,
it can do so by setting the right attributes in the S1 tables (i.e more restricted
access trumps over the relaxed S2 attributes).


> 
> > Anyway, at the moment it seems like doing a device-tree compatiblity prop
> > match for mmio-sram would be the path with least resistance...
> 
> I think this is a good solution until we figure out the behavior of
> device memory mapped with wrong attribute.

Thanks.

Best regards,
Edgar


> 
> Regards,
> 
> [1] http://xenbits.xen.org/xsa/advisory-124.html
> [2] http://xenbits.xen.org/xsa/advisory-126.html
> 
> -- 
> Julien Grall