How to expose caching policy to a para-virtualized guest?

How to expose caching policy to a para-virtualized guest?

[Gurchetan Singh]

Hi,

We're trying to implement Vulkan with virtio-gpu, and that API exposes
the difference between cached and uncached mappings to userspace (i.e,
some older GPUs can't snoop the CPU's caches).

We need to make sure that the guest and host caching attributes are
aligned, or there's a proper API between the virtio driver and device
to ensure coherence.

One issue that needs to be addressed is the caching policy is variable
dependent on the VM configuration and architecture.  For example, on
x86, it looks like a MTRR controls whether the guest caching attribute
predominates[1].  On ARM, it looks like the MMU registers control
whether the guest can override the host attribute, but in general it's
most restrictive attribute that makes a difference[2].  Let me if
that's incorrect.

I'm wondering if there's some standard kernel API to query such
attributes.  For example, something like
arch_does_guest_attribute_matter() or arch_can_guest_flush() would do
the trick.  Without this, we may need to introduce VIRTIO_GPU_F_*
flags set by the host, but that may make the already giant QEMU
command line even bigger.

[1] https://lwn.net/Articles/646712/
[2]  https://events.static.linuxfound.org/sites/events/files/slides/slides_10.pdf

Re: How to expose caching policy to a para-virtualized guest?

[Marc Zyngier]

Hi Gurchetan,

I don't know anything about graphics API, so please bear with me.

On 2019-12-11 01:32, Gurchetan Singh wrote:
> Hi,
>
> We're trying to implement Vulkan with virtio-gpu, and that API 
> exposes
> the difference between cached and uncached mappings to userspace 
> (i.e,
> some older GPUs can't snoop the CPU's caches).
>
> We need to make sure that the guest and host caching attributes are
> aligned, or there's a proper API between the virtio driver and device
> to ensure coherence.

I think you trying to cross two barriers at once here.

Virtio is always coherent between host and guest, and the guest should
use cacheable mappings. That's at least my expectation, and I don't
know of any exception to this rule.

You have then the coherency of the physical device, and that's a host
kernel (and maybe userspace) matter. Why should the guest ever know 
about
this constraint?

> One issue that needs to be addressed is the caching policy is 
> variable
> dependent on the VM configuration and architecture.  For example, on
> x86, it looks like a MTRR controls whether the guest caching 
> attribute
> predominates[1].  On ARM, it looks like the MMU registers control
> whether the guest can override the host attribute, but in general 
> it's
> most restrictive attribute that makes a difference[2].  Let me if
> that's incorrect.

For ARM, this is true up to ARMv8.3. Starting with ARMv8.4, FWB gives
the hypervisor the opportunity to force memory mappings as cacheable
write-back. None of that is visible to userspace, thankfully.

> I'm wondering if there's some standard kernel API to query such
> attributes.  For example, something like
> arch_does_guest_attribute_matter() or arch_can_guest_flush() would do
> the trick.  Without this, we may need to introduce VIRTIO_GPU_F_*
> flags set by the host, but that may make the already giant QEMU
> command line even bigger.

If something has to manage the coherency, it should be the host that
knows how the memory traffic flows between host and guest, and apply
cache management as required. Note that on arm64, cache management
instructions are available from userspace. On systems that are
fully coherent, they are expected to be little more than NOPs.

 From the above, it is pretty obvious that I don't understand what
problem you are trying to solve. Maybe you could explain how
you envisage things to work, who maps what where, and the expected
semantics. Once we have a common understanding of the problem,
maybe we can think of a decent solution.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

Re: How to expose caching policy to a para-virtualized guest?

[Gurchetan Singh]

On Wed, Dec 11, 2019 at 3:08 AM Marc Zyngier <maz@kernel.org> wrote:
>
> Hi Gurchetan,
>
> I don't know anything about graphics API, so please bear with me.

Vulkan exposes the concept of memory types to userspace.  These memory
types describe the memory properties of a heap.  For example, any
memory type with the HOST_COHERENT_BIT specifies the application isn't
required to send certain cache management commands
(vkFlushMappedMemoryRanges, vkInvalidateMappedMemoryRanges).

1) HOST_CACHED => cached, no cache management
2) HOST_COHERENT_BIT => write combine, no cache management, no snooping
3) HOST_COHERENT_BIT | HOST_CACHED => cached, gpu snoops cpu caches,
no cache management

Here's some more reading on that:

https://static.docs.arm.com/100019/0100/arm_mali_application_developer_best_practices_developer_guide_100019_0100_00_en2.pdf

(1) and (3) aren't too difficult -- just use
KVM_SET_USER_MEMORY_REGION. (2) is, since it may lead to inconsistent
mappings.

vmware has a very nice emulated solution for this:

https://lwn.net/Articles/804114/

We're planning on building off vmware's solution, and if possible, add
some additional optimizations.

>
> On 2019-12-11 01:32, Gurchetan Singh wrote:
> > Hi,
> >
> > We're trying to implement Vulkan with virtio-gpu, and that API
> > exposes
> > the difference between cached and uncached mappings to userspace
> > (i.e,
> > some older GPUs can't snoop the CPU's caches).
> >
> > We need to make sure that the guest and host caching attributes are
> > aligned, or there's a proper API between the virtio driver and device
> > to ensure coherence.
>
> I think you trying to cross two barriers at once here.
>
> Virtio is always coherent between host and guest, and the guest should
> use cacheable mappings. That's at least my expectation, and I don't
> know of any exception to this rule.

Where is this ruled stated?  We may need to modify the spec.  At the
very least, we need anything mapped on the host with HOST_COHERENT_BIT
only to be write combine in the guest.

However, there's an additional caveat: on x86, cache management
(clflush) is available to the guest.  So it's possible to map host
cached memory as write-combine, and import guest memory to Vulkan.
This is an optimization limited to non-zero amount of x86 devices.

So, something like arch_does_guest_attribute_matter() or
arch_can_guest_flush() in the guest kernel would be useful.  But it
doesn't sound like this is readily available?

> You have then the coherency of the physical device, and that's a host
> kernel (and maybe userspace) matter. Why should the guest ever know
> about
> this constraint?
>
> > One issue that needs to be addressed is the caching policy is
> > variable
> > dependent on the VM configuration and architecture.  For example, on
> > x86, it looks like a MTRR controls whether the guest caching
> > attribute
> > predominates[1].  On ARM, it looks like the MMU registers control
> > whether the guest can override the host attribute, but in general
> > it's
> > most restrictive attribute that makes a difference[2].  Let me if
> > that's incorrect.
>
> For ARM, this is true up to ARMv8.3. Starting with ARMv8.4, FWB gives
> the hypervisor the opportunity to force memory mappings as cacheable
> write-back. None of that is visible to userspace, thankfully.

Is there some open-source code available on ARM showing how the MMU is
configured (which forces guest attribute override)?

>
> > I'm wondering if there's some standard kernel API to query such
> > attributes.  For example, something like
> > arch_does_guest_attribute_matter() or arch_can_guest_flush() would do
> > the trick.  Without this, we may need to introduce VIRTIO_GPU_F_*
> > flags set by the host, but that may make the already giant QEMU
> > command line even bigger.
>
> If something has to manage the coherency, it should be the host that
> knows how the memory traffic flows between host and guest, and apply
> cache management as required. Note that on arm64, cache management
> instructions are available from userspace. On systems that are
> fully coherent, they are expected to be little more than NOPs.
>
>  From the above, it is pretty obvious that I don't understand what
> problem you are trying to solve. Maybe you could explain how
> you envisage things to work, who maps what where, and the expected
> semantics. Once we have a common understanding of the problem,
> maybe we can think of a decent solution.
>
> Thanks,
>
>          M.
> --
> Jazz is not dead. It just smells funny...

Re: How to expose caching policy to a para-virtualized guest?

[Marc Zyngier]

On Wed, 11 Dec 2019 23:34:30 +0000,
Gurchetan Singh <gurchetansingh@chromium.org> wrote:
> 
> On Wed, Dec 11, 2019 at 3:08 AM Marc Zyngier <maz@kernel.org> wrote:
> >
> > Hi Gurchetan,
> >
> > I don't know anything about graphics API, so please bear with me.
> 
> Vulkan exposes the concept of memory types to userspace.  These memory

Key word: userspace.

> types describe the memory properties of a heap.  For example, any
> memory type with the HOST_COHERENT_BIT specifies the application isn't
> required to send certain cache management commands
> (vkFlushMappedMemoryRanges, vkInvalidateMappedMemoryRanges).
> 
> 1) HOST_CACHED => cached, no cache management
> 2) HOST_COHERENT_BIT => write combine, no cache management, no snooping
> 3) HOST_COHERENT_BIT | HOST_CACHED => cached, gpu snoops cpu caches,
> no cache management

And that's absolutely fine between the CPU and the GPU. I don't see
why the guest, which is supposed to use a PV (virtio) driver should be
impacted by any of this.

> Here's some more reading on that:
> 
> https://static.docs.arm.com/100019/0100/arm_mali_application_developer_best_practices_developer_guide_100019_0100_00_en2.pdf

Sorry, but seeing the words "Mali" and "best practices" in the same
sentence is just hilarious. I haven't read any further.

> (1) and (3) aren't too difficult -- just use
> KVM_SET_USER_MEMORY_REGION. (2) is, since it may lead to
> inconsistent mappings.

How is that relevant to the above? Most importantly, why should the
guest be in *any way* influenced by the coherency (or lack thereof) of
the GPU driver on the host? The guest should see a cache-coherent PV
driver, and the host does its best to cope with it. After all, that
host driver knows exactly what level of coherency it has to deal with.

> vmware has a very nice emulated solution for this:
> 
> https://lwn.net/Articles/804114/
> 
> We're planning on building off vmware's solution, and if possible, add
> some additional optimizations.

Well, let's try to walk before we run, shall we?

> > On 2019-12-11 01:32, Gurchetan Singh wrote:
> > > Hi,
> > >
> > > We're trying to implement Vulkan with virtio-gpu, and that API
> > > exposes
> > > the difference between cached and uncached mappings to userspace
> > > (i.e,
> > > some older GPUs can't snoop the CPU's caches).
> > >
> > > We need to make sure that the guest and host caching attributes are
> > > aligned, or there's a proper API between the virtio driver and device
> > > to ensure coherence.
> >
> > I think you trying to cross two barriers at once here.
> >
> > Virtio is always coherent between host and guest, and the guest should
> > use cacheable mappings. That's at least my expectation, and I don't
> > know of any exception to this rule.
> 
> Where is this ruled stated?  We may need to modify the spec.

Sure. The spec is public, you just need to be convincing enough.

> At the very least, we need anything mapped on the host with
> HOST_COHERENT_BIT only to be write combine in the guest.

You don't get to choose. The guest does. And you haven't explained why
the guest should be involved in any way.

> However, there's an additional caveat: on x86, cache management
> (clflush) is available to the guest.  So it's possible to map host
> cached memory as write-combine, and import guest memory to Vulkan.
> This is an optimization limited to non-zero amount of x86 devices.

If you say so. I have no idea.

> So, something like arch_does_guest_attribute_matter() or
> arch_can_guest_flush() in the guest kernel would be useful.  But it
> doesn't sound like this is readily available?

Given that you haven't given the definition of what this is supposed
to do, and that you keep giving solutions without having described the
problem, I have no answer to give you.

> > You have then the coherency of the physical device, and that's a host
> > kernel (and maybe userspace) matter. Why should the guest ever know
> > about
> > this constraint?
> >
> > > One issue that needs to be addressed is the caching policy is
> > > variable
> > > dependent on the VM configuration and architecture.  For example, on
> > > x86, it looks like a MTRR controls whether the guest caching
> > > attribute
> > > predominates[1].  On ARM, it looks like the MMU registers control
> > > whether the guest can override the host attribute, but in general
> > > it's
> > > most restrictive attribute that makes a difference[2].  Let me if
> > > that's incorrect.
> >
> > For ARM, this is true up to ARMv8.3. Starting with ARMv8.4, FWB gives
> > the hypervisor the opportunity to force memory mappings as cacheable
> > write-back. None of that is visible to userspace, thankfully.
> 
> Is there some open-source code available on ARM showing how the MMU is
> configured (which forces guest attribute override)?

The kernel is all yours, and the ARMv8 ARM is an entertaining read.

> > > I'm wondering if there's some standard kernel API to query such
> > > attributes.  For example, something like
> > > arch_does_guest_attribute_matter() or arch_can_guest_flush() would do
> > > the trick.  Without this, we may need to introduce VIRTIO_GPU_F_*
> > > flags set by the host, but that may make the already giant QEMU
> > > command line even bigger.
> >
> > If something has to manage the coherency, it should be the host that
> > knows how the memory traffic flows between host and guest, and apply
> > cache management as required. Note that on arm64, cache management
> > instructions are available from userspace. On systems that are
> > fully coherent, they are expected to be little more than NOPs.
> >
> >  From the above, it is pretty obvious that I don't understand what
> > problem you are trying to solve. Maybe you could explain how
> > you envisage things to work, who maps what where, and the expected
> > semantics. Once we have a common understanding of the problem,
> > maybe we can think of a decent solution.

I'm still expecting answers to the above, as this is really key.

	M.

-- 
Jazz is not dead, it just smells funny.