Re: [PATCH v2] xen: arm: force guest memory accesses to cacheable when MMU is disabled

[Julien Grall <julien.grall@linaro.org>]

On 01/08/2014 02:09 PM, Ian Campbell wrote:
> On ARM guest OSes are started with MMU and Caches disables (as they are on
> native) however caching is enabled in the domain running the builder and
> therefore we must ensure cache consistency.
> 
> The existing solution to this problem (a0035ecc0d82 "tools: libxc: flush data
> cache after loading images into guest memory") is to flush the caches after
> loading the various blobs into guest RAM. However this approach has two short
> comings:
> 
>  - The cache flush primitives available to userspace on arm32 are not
>    sufficient for our needs.
>  - There is a race between the cache flush and the unmap of the guest page
>    where the processor might speculatively dirty the cache line again.
> 
> (of these the second is the more fundamental)
> 
> This patch makes use of the the hardware functionality to force all accesses
> made from guest mode to be cached (the HCR.DC == default cached bit). This
> means that we don't need to worry about the domain builder's writes being
> cached because the guests "uncached" accesses will actually be cached.
> 
> Unfortunately the use of HCR.DC is incompatible with the guest enabling its
> MMU (SCTLR.M bit). Therefore we must trap accesses to the SCTLR so that we can
> detect when this happens and disable HCR.DC. This is done with the HCR.TVM
> (trap virtual memory controls) bit which also causes various other registers
> to be trapped, all of which can be passed straight through to the underlying
> register. Once the guest has enabled its MMU we no longer need to trap so
> there is no ongoing overhead. In my tests Linux makes about half a dozen
> accesses to these registers before the MMU is enabled, I would expect other
> OSes to behave similarly (the sequence of writes needed to setup the MMU is
> pretty obvious).
> 
> Apart from this unfortunate need to trap these accesses this approach is
> incompatible with guests which attempt to do DMA operations with their MMU
> disabled. In practice this means guests with passthrough which we do not yet
> support. Since a typical guest (including dom0) does not access devices which
> require DMA until after it is fully up and running with paging enabled the
> main risk is to in-guest firmware which does DMA i.e. running EFI in a guest,
> with a disk passed through and booting from that disk. Since we know that dom0
> is not using any such firmware and we do not support device passthrough to
> guests yet we can live with this restriction. Once passthrough is implemented
> this will need to be revisited.
> 
> The patch includes a couple of seemingly unrelated but necessary changes:
> 
>  - HSR_SYSREG_CRN_MASK was incorrectly defined, which happened to be benign
>    with the existing set of system register we handled, but broke with the new
>    ones introduced here.
>  - The defines used to decode the HSR system register fields were named the
>    same as the register. This breaks the accessor macros. This had gone
>    unnoticed because the handling of the existing trapped registers did not
>    require accessing the underlying hardware register. Rename those constants
>    with an HSR_SYSREG prefix (in line with HSR_CP32/64 for 32-bit registers).
> 
> This patch has survived thousands of boot loops on a Midway system.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Julien Grall <julien.grall@linaro.org>

-- 
Julien Grall