Re: [PATCH RFC] xen: arm: use uncached foreign mappings when building guests

[David Vrabel <dvrabel@cantab.net>]

On 18/12/2013 17:28, Ian Campbell wrote:
> When building an ARM guest we need to take care of cache maintenance
> because the guest starts with MMU and cache disabled, which means we
> need to make sure that the initial images (kernel, initrd, dtb) which we
> write to guest memory are not in the cache.
> 
> We thought we had solved this with "tools: libxc: flush data cache after
> loading images into guest memory" (a0035ecc0d82) however it turns out
> that there are a couple of issues with this approach:
> 
> Firstly we need to do a cache flush from userspace, on arm64 this is
> possible by directly using the instructions from userspace, but on arm32
> this is not possible and so we need to use a system call. Unfortunately
> the system call provided by Linux for this purpose does not flush far
> enough down the cache hierarchy. Extending the system call would not be
> an insurmountable barrier, were it not for the second issue:
> 
> Secondly, and more importantly, Catalin Marinas points out (via Marc
> Zyngier) that there is a race between the cache flush and the point
> where we tear down the mappings, where the processor might speculatively
> pull some data into the cache (cache flushes are by Virtual Address, so
> this race is unavoidable).
> 
> If this happens then guest kernels which modify some code/data before
> enabling MMUs + caches may see stale data in the cache.

Would this same problem with occur with save/restore of a guest that has
caching disabled?  If so, this would require using uncached foreign
mappings for save/restore which I would think would make performance suck.

Does the same problem occur if the guest has MMU and caching enabled but
has uncached mappings?

Is there not some sort of cache flush you can do /after/ tearing down
the foreign mappings?  Flush all by ASID or similar?

Would this also require that granted pages must only have cachable
mappings in the granting domain?  If so, this should be documented as
part of the ABI.

David