Re: [Qemu-devel] [PATCH] roms: Flush icache when writing roms to guest memory

[Peter Maydell <peter.maydell@linaro.org>]

On 11 December 2013 13:23, Alexander Graf <agraf@suse.de> wrote:
> The guest expects that its data and instruction cache view of the world
> is 100% consistent when it initially boots. This works just fine on
> initial rom population for the first boot.
>
> However, when we reboot and then repopulate the rom region there could
> be old code still stuck in the instruction cache, giving the guest an
> inconsistent view of the world when we're using kvm.
>
> So we need to invalidate the icache every time we write a rom into guest
> address space. We do not need to do this for every DMA since the guest
> expects it has to flush the icache manually in that case.

> @@ -2033,6 +2034,13 @@ void cpu_physical_memory_write_rom(hwaddr addr,
>              ptr = qemu_get_ram_ptr(addr1);
>              memcpy(ptr, buf, l);
>              invalidate_and_set_dirty(addr1, l);
> +            if (kvm_enabled()) {
> +                /*
> +                 * The guest may want to directly execute from the rom region,
> +                 * so we better invalidate its icache
> +                 */
> +                flush_icache_range((uintptr_t)ptr, (uintptr_t)ptr + l);
> +            }

I bet these aren't the only places where code gets written
to guest memory. Also are you sure flush_icache_range()
works correctly when multiple threads (multiple vCPUs,
potentially executing on different host CPUs) are involved? The
TCG case only needs to care about "this thread writes code
to memory that it will itself later execute", not any kind of
cross-host-CPU flushing.

There was a huge thread on kvmarm earlier this year
https://lists.cs.columbia.edu/pipermail/kvmarm/2013-August/006716.html
about a similar sort of issue, and I think the conclusion was that
the kernel basically had to deal with the problem itself [though
the thread is rather confusing...]. I've cc'd Marc Z in the hope
he remembers the ARM specific detail...

thanks
-- PMM