userspace emulated smmu/vfio integration: how to trap updates to the table structures?

userspace emulated smmu/vfio integration: how to trap updates to the table structures?

[Auger Eric]

Hi Will, Robin, Jean-Philippe,

I have been working on the integration between user-space emulated
SMMU-v3 and VFIO in QEMU. At the moment I fail identifying a proper easy
way to trap page table updates. This is requested to keep the host
translation structures consistent to guest translation structures.

On Intel VTD there is a so-called "caching mode" (CM, see VTD spec
paragraph 6.1) that forces the OS to explicitly invalidate caches
whenever it updates any remapping structure (updates to not-present or
present entries). Those invalidation commands are used to trap and
update host structures. This mode was devised for virtualization. I was
not able to find such "caching mode" on ARM SMMU. Is there any?

If not, do you have any other suggestion, I mean, besides the
virtio-based solution.

Thanks

Eric

Re: userspace emulated smmu/vfio integration: how to trap updates to the table structures?

[Christoffer Dall]

On Tue, May 2, 2017 at 9:30 AM, Auger Eric <eric.auger@redhat.com> wrote:
> Hi Will, Robin, Jean-Philippe,
>
> I have been working on the integration between user-space emulated
> SMMU-v3 and VFIO in QEMU. At the moment I fail identifying a proper easy
> way to trap page table updates. This is requested to keep the host
> translation structures consistent to guest translation structures.
>
> On Intel VTD there is a so-called "caching mode" (CM, see VTD spec
> paragraph 6.1) that forces the OS to explicitly invalidate caches
> whenever it updates any remapping structure (updates to not-present or
> present entries). Those invalidation commands are used to trap and
> update host structures. This mode was devised for virtualization. I was
> not able to find such "caching mode" on ARM SMMU. Is there any?
>
> If not, do you have any other suggestion, I mean, besides the
> virtio-based solution.
>
>
Worst case, can you make the guest page tables read-only and catch the
faults and propagate changes to SMMU translations?

-Christoffer

Re: userspace emulated smmu/vfio integration: how to trap updates to the table structures?

[Auger Eric]

Hi Christoffer,

On 02/05/2017 09:53, Christoffer Dall wrote:
> On Tue, May 2, 2017 at 9:30 AM, Auger Eric <eric.auger@redhat.com> wrote:
>> Hi Will, Robin, Jean-Philippe,
>>
>> I have been working on the integration between user-space emulated
>> SMMU-v3 and VFIO in QEMU. At the moment I fail identifying a proper easy
>> way to trap page table updates. This is requested to keep the host
>> translation structures consistent to guest translation structures.
>>
>> On Intel VTD there is a so-called "caching mode" (CM, see VTD spec
>> paragraph 6.1) that forces the OS to explicitly invalidate caches
>> whenever it updates any remapping structure (updates to not-present or
>> present entries). Those invalidation commands are used to trap and
>> update host structures. This mode was devised for virtualization. I was
>> not able to find such "caching mode" on ARM SMMU. Is there any?
>>
>> If not, do you have any other suggestion, I mean, besides the
>> virtio-based solution.
>>
>>
> Worst case, can you make the guest page tables read-only and catch the
> faults and propagate changes to SMMU translations?

The issue I foresee is there are up to 4 level of page tables to trap.
This would lead to plenty of regions to "translate" on qemu side. Also,
besides the 1st level pointed by TTBR found in stage 1 context
descriptor, other page regions would be discovered dynamically as
mapping are built. To me this is the last resort solution if confirmed
feasible.

Thanks

Eric


> 
> -Christoffer
> 

Re: userspace emulated smmu/vfio integration: how to trap updates to the table structures?

[Christoffer Dall]

On Tue, May 2, 2017 at 10:13 AM, Auger Eric <eric.auger@redhat.com> wrote:
> Hi Christoffer,
>
> On 02/05/2017 09:53, Christoffer Dall wrote:
>> On Tue, May 2, 2017 at 9:30 AM, Auger Eric <eric.auger@redhat.com> wrote:
>>> Hi Will, Robin, Jean-Philippe,
>>>
>>> I have been working on the integration between user-space emulated
>>> SMMU-v3 and VFIO in QEMU. At the moment I fail identifying a proper easy
>>> way to trap page table updates. This is requested to keep the host
>>> translation structures consistent to guest translation structures.
>>>
>>> On Intel VTD there is a so-called "caching mode" (CM, see VTD spec
>>> paragraph 6.1) that forces the OS to explicitly invalidate caches
>>> whenever it updates any remapping structure (updates to not-present or
>>> present entries). Those invalidation commands are used to trap and
>>> update host structures. This mode was devised for virtualization. I was
>>> not able to find such "caching mode" on ARM SMMU. Is there any?
>>>
>>> If not, do you have any other suggestion, I mean, besides the
>>> virtio-based solution.
>>>
>>>
>> Worst case, can you make the guest page tables read-only and catch the
>> faults and propagate changes to SMMU translations?
>
> The issue I foresee is there are up to 4 level of page tables to trap.
> This would lead to plenty of regions to "translate" on qemu side. Also,
> besides the 1st level pointed by TTBR found in stage 1 context
> descriptor, other page regions would be discovered dynamically as
> mapping are built. To me this is the last resort solution if confirmed
> feasible.
>
Completely agree, it would be a terrible solution.

Thanks,
-Christoffer

Re: userspace emulated smmu/vfio integration: how to trap updates to the table structures?

[Michael S. Tsirkin]

On Tue, May 02, 2017 at 10:17:26AM +0200, Christoffer Dall wrote:
> On Tue, May 2, 2017 at 10:13 AM, Auger Eric <eric.auger@redhat.com> wrote:
> > Hi Christoffer,
> >
> > On 02/05/2017 09:53, Christoffer Dall wrote:
> >> On Tue, May 2, 2017 at 9:30 AM, Auger Eric <eric.auger@redhat.com> wrote:
> >>> Hi Will, Robin, Jean-Philippe,
> >>>
> >>> I have been working on the integration between user-space emulated
> >>> SMMU-v3 and VFIO in QEMU. At the moment I fail identifying a proper easy
> >>> way to trap page table updates. This is requested to keep the host
> >>> translation structures consistent to guest translation structures.
> >>>
> >>> On Intel VTD there is a so-called "caching mode" (CM, see VTD spec
> >>> paragraph 6.1) that forces the OS to explicitly invalidate caches
> >>> whenever it updates any remapping structure (updates to not-present or
> >>> present entries). Those invalidation commands are used to trap and
> >>> update host structures. This mode was devised for virtualization. I was
> >>> not able to find such "caching mode" on ARM SMMU. Is there any?
> >>>
> >>> If not, do you have any other suggestion, I mean, besides the
> >>> virtio-based solution.
> >>>
> >>>
> >> Worst case, can you make the guest page tables read-only and catch the
> >> faults and propagate changes to SMMU translations?
> >
> > The issue I foresee is there are up to 4 level of page tables to trap.
> > This would lead to plenty of regions to "translate" on qemu side. Also,
> > besides the 1st level pointed by TTBR found in stage 1 context
> > descriptor, other page regions would be discovered dynamically as
> > mapping are built. To me this is the last resort solution if confirmed
> > feasible.
> >
> Completely agree, it would be a terrible solution.
> 
> Thanks,
> -Christoffer

All "caching mode" is is simple a bit that the IOMMU device uses to tell
the OS "I might cache non-present/invalid entries, please use
invalidation even if you make non-present entries present".

So all we need from ARM is bit somewhere in one of MMU registers
and a promise not to use it for any other purpose preferably
by documenting it as caching mode bit in the next manual version.

Any chance of this happening?

-- 
MST

Re: userspace emulated smmu/vfio integration: how to trap updates to the table structures?

[Robin Murphy]

On 02/05/17 16:22, Michael S. Tsirkin wrote:
> On Tue, May 02, 2017 at 10:17:26AM +0200, Christoffer Dall wrote:
>> On Tue, May 2, 2017 at 10:13 AM, Auger Eric <eric.auger@redhat.com> wrote:
>>> Hi Christoffer,
>>>
>>> On 02/05/2017 09:53, Christoffer Dall wrote:
>>>> On Tue, May 2, 2017 at 9:30 AM, Auger Eric <eric.auger@redhat.com> wrote:
>>>>> Hi Will, Robin, Jean-Philippe,
>>>>>
>>>>> I have been working on the integration between user-space emulated
>>>>> SMMU-v3 and VFIO in QEMU. At the moment I fail identifying a proper easy
>>>>> way to trap page table updates. This is requested to keep the host
>>>>> translation structures consistent to guest translation structures.
>>>>>
>>>>> On Intel VTD there is a so-called "caching mode" (CM, see VTD spec
>>>>> paragraph 6.1) that forces the OS to explicitly invalidate caches
>>>>> whenever it updates any remapping structure (updates to not-present or
>>>>> present entries). Those invalidation commands are used to trap and
>>>>> update host structures. This mode was devised for virtualization. I was
>>>>> not able to find such "caching mode" on ARM SMMU. Is there any?
>>>>>
>>>>> If not, do you have any other suggestion, I mean, besides the
>>>>> virtio-based solution.
>>>>>
>>>>>
>>>> Worst case, can you make the guest page tables read-only and catch the
>>>> faults and propagate changes to SMMU translations?
>>>
>>> The issue I foresee is there are up to 4 level of page tables to trap.
>>> This would lead to plenty of regions to "translate" on qemu side. Also,
>>> besides the 1st level pointed by TTBR found in stage 1 context
>>> descriptor, other page regions would be discovered dynamically as
>>> mapping are built. To me this is the last resort solution if confirmed
>>> feasible.
>>>
>> Completely agree, it would be a terrible solution.
>>
>> Thanks,
>> -Christoffer
> 
> All "caching mode" is is simple a bit that the IOMMU device uses to tell
> the OS "I might cache non-present/invalid entries, please use
> invalidation even if you make non-present entries present".
> 
> So all we need from ARM is bit somewhere in one of MMU registers
> and a promise not to use it for any other purpose preferably
> by documenting it as caching mode bit in the next manual version.
> 
> Any chance of this happening?

It's unlikely as such, since the VMSAv8 translation formats which the
SMMU architecture specifies explicitly forbid such "negative caching",
so having an architected bit to say "I don't comply with the
architecture" comes out looking a little nonsensical.

However, doing the same via a firmware quirk would seem a lot more
reasonable. The io-pgtable code already has the IO_PGTABLE_TLBI_ON_MAP
quirk to cope with other not-quite-VMSA IOMMU implementations which do
cache invalid entries - supporting that in the io-pgtable-arm backend
(currently only the v7s backend implements it), then having the SMMUv3
driver detect the QEMU implementation and set that quirk on pagetables
accordingly would end up achieving the same thing.

Robin.

Re: userspace emulated smmu/vfio integration: how to trap updates to the table structures?

[Michael S. Tsirkin]

On Tue, May 02, 2017 at 04:42:56PM +0100, Robin Murphy wrote:
> On 02/05/17 16:22, Michael S. Tsirkin wrote:
> > On Tue, May 02, 2017 at 10:17:26AM +0200, Christoffer Dall wrote:
> >> On Tue, May 2, 2017 at 10:13 AM, Auger Eric <eric.auger-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> >>> Hi Christoffer,
> >>>
> >>> On 02/05/2017 09:53, Christoffer Dall wrote:
> >>>> On Tue, May 2, 2017 at 9:30 AM, Auger Eric <eric.auger-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> >>>>> Hi Will, Robin, Jean-Philippe,
> >>>>>
> >>>>> I have been working on the integration between user-space emulated
> >>>>> SMMU-v3 and VFIO in QEMU. At the moment I fail identifying a proper easy
> >>>>> way to trap page table updates. This is requested to keep the host
> >>>>> translation structures consistent to guest translation structures.
> >>>>>
> >>>>> On Intel VTD there is a so-called "caching mode" (CM, see VTD spec
> >>>>> paragraph 6.1) that forces the OS to explicitly invalidate caches
> >>>>> whenever it updates any remapping structure (updates to not-present or
> >>>>> present entries). Those invalidation commands are used to trap and
> >>>>> update host structures. This mode was devised for virtualization. I was
> >>>>> not able to find such "caching mode" on ARM SMMU. Is there any?
> >>>>>
> >>>>> If not, do you have any other suggestion, I mean, besides the
> >>>>> virtio-based solution.
> >>>>>
> >>>>>
> >>>> Worst case, can you make the guest page tables read-only and catch the
> >>>> faults and propagate changes to SMMU translations?
> >>>
> >>> The issue I foresee is there are up to 4 level of page tables to trap.
> >>> This would lead to plenty of regions to "translate" on qemu side. Also,
> >>> besides the 1st level pointed by TTBR found in stage 1 context
> >>> descriptor, other page regions would be discovered dynamically as
> >>> mapping are built. To me this is the last resort solution if confirmed
> >>> feasible.
> >>>
> >> Completely agree, it would be a terrible solution.
> >>
> >> Thanks,
> >> -Christoffer
> > 
> > All "caching mode" is is simple a bit that the IOMMU device uses to tell
> > the OS "I might cache non-present/invalid entries, please use
> > invalidation even if you make non-present entries present".
> > 
> > So all we need from ARM is bit somewhere in one of MMU registers
> > and a promise not to use it for any other purpose preferably
> > by documenting it as caching mode bit in the next manual version.
> > 
> > Any chance of this happening?
> 
> It's unlikely as such, since the VMSAv8 translation formats which the
> SMMU architecture specifies explicitly forbid such "negative caching",
> so having an architected bit to say "I don't comply with the
> architecture" comes out looking a little nonsensical.

Same's true for AMD/intel IOMMUs FWIW.

> However, doing the same via a firmware quirk would seem a lot more
> reasonable. The io-pgtable code already has the IO_PGTABLE_TLBI_ON_MAP
> quirk to cope with other not-quite-VMSA IOMMU implementations which do
> cache invalid entries - supporting that in the io-pgtable-arm backend
> (currently only the v7s backend implements it), then having the SMMUv3
> driver detect the QEMU implementation and set that quirk on pagetables
> accordingly would end up achieving the same thing.
> 
> Robin.

Without a spec specifying it, it does however mean that non-Linux
guests aren't likely to implement this in the same way.

-- 
MST

Re: userspace emulated smmu/vfio integration: how to trap updates to the table structures?

[Auger Eric]

Hi Robin, Michael,

On 02/05/2017 17:42, Robin Murphy wrote:
> On 02/05/17 16:22, Michael S. Tsirkin wrote:
>> On Tue, May 02, 2017 at 10:17:26AM +0200, Christoffer Dall wrote:
>>> On Tue, May 2, 2017 at 10:13 AM, Auger Eric <eric.auger-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>>>> Hi Christoffer,
>>>>
>>>> On 02/05/2017 09:53, Christoffer Dall wrote:
>>>>> On Tue, May 2, 2017 at 9:30 AM, Auger Eric <eric.auger-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>>>>>> Hi Will, Robin, Jean-Philippe,
>>>>>>
>>>>>> I have been working on the integration between user-space emulated
>>>>>> SMMU-v3 and VFIO in QEMU. At the moment I fail identifying a proper easy
>>>>>> way to trap page table updates. This is requested to keep the host
>>>>>> translation structures consistent to guest translation structures.
>>>>>>
>>>>>> On Intel VTD there is a so-called "caching mode" (CM, see VTD spec
>>>>>> paragraph 6.1) that forces the OS to explicitly invalidate caches
>>>>>> whenever it updates any remapping structure (updates to not-present or
>>>>>> present entries). Those invalidation commands are used to trap and
>>>>>> update host structures. This mode was devised for virtualization. I was
>>>>>> not able to find such "caching mode" on ARM SMMU. Is there any?
>>>>>>
>>>>>> If not, do you have any other suggestion, I mean, besides the
>>>>>> virtio-based solution.
>>>>>>
>>>>>>
>>>>> Worst case, can you make the guest page tables read-only and catch the
>>>>> faults and propagate changes to SMMU translations?
>>>>
>>>> The issue I foresee is there are up to 4 level of page tables to trap.
>>>> This would lead to plenty of regions to "translate" on qemu side. Also,
>>>> besides the 1st level pointed by TTBR found in stage 1 context
>>>> descriptor, other page regions would be discovered dynamically as
>>>> mapping are built. To me this is the last resort solution if confirmed
>>>> feasible.
>>>>
>>> Completely agree, it would be a terrible solution.
>>>
>>> Thanks,
>>> -Christoffer
>>
>> All "caching mode" is is simple a bit that the IOMMU device uses to tell
>> the OS "I might cache non-present/invalid entries, please use
>> invalidation even if you make non-present entries present".
>>
>> So all we need from ARM is bit somewhere in one of MMU registers
>> and a promise not to use it for any other purpose preferably
>> by documenting it as caching mode bit in the next manual version.
>>
>> Any chance of this happening?
> 
> It's unlikely as such, since the VMSAv8 translation formats which the
> SMMU architecture specifies explicitly forbid such "negative caching",
> so having an architected bit to say "I don't comply with the
> architecture" comes out looking a little nonsensical.
> 
> However, doing the same via a firmware quirk would seem a lot more
> reasonable. The io-pgtable code already has the IO_PGTABLE_TLBI_ON_MAP
> quirk to cope with other not-quite-VMSA IOMMU implementations which do
> cache invalid entries - supporting that in the io-pgtable-arm backend
> (currently only the v7s backend implements it), then having the SMMUv3
> driver detect the QEMU implementation and set that quirk on pagetables
> accordingly would end up achieving the same thing.

Thank you for your inputs.

I can see the quirk set in msm and mtk drivers. How would you describe
the quirk needs to be applied on arm-smmu-v3 driver? On dt side do you
suggest we add an optional property like quirk-tlbi-on-map. What about
ACPI? If you feel this approach is sensible I can submit something
starting with dt.

Thanks

Eric
> 
> Robin.
> 

Re: userspace emulated smmu/vfio integration: how to trap updates to the table structures?

[Michael S. Tsirkin]

On Tue, May 02, 2017 at 09:30:39AM +0200, Auger Eric wrote:
> 
> I have been working on the integration between user-space emulated
> SMMU-v3 and VFIO in QEMU. At the moment I fail identifying a proper easy
> way to trap page table updates. This is requested to keep the host
> translation structures consistent to guest translation structures.
> 
> On Intel VTD there is a so-called "caching mode" (CM, see VTD spec
> paragraph 6.1) that forces the OS to explicitly invalidate caches
> whenever it updates any remapping structure (updates to not-present or
> present entries). Those invalidation commands are used to trap and
> update host structures. This mode was devised for virtualization. I was
> not able to find such "caching mode" on ARM SMMU. Is there any?
> 
> If not, do you have any other suggestion, I mean, besides the
> virtio-based solution.

Nothing really prevents us from emulating e.g. an ARM system with
intel's or AMD's IOMMU.  Linux normally isn't building these for
ARM but it's mostly just tweaking the config.

Non-linux guest support might be harder though.

-- 
MST

Re: userspace emulated smmu/vfio integration: how to trap updates to the table structures?

[Michael S. Tsirkin]

On Tue, May 02, 2017 at 09:30:39AM +0200, Auger Eric wrote:
> Hi Will, Robin, Jean-Philippe,
> 
> I have been working on the integration between user-space emulated
> SMMU-v3 and VFIO in QEMU. At the moment I fail identifying a proper easy
> way to trap page table updates. This is requested to keep the host
> translation structures consistent to guest translation structures.
> 
> On Intel VTD there is a so-called "caching mode" (CM, see VTD spec
> paragraph 6.1) that forces the OS to explicitly invalidate caches
> whenever it updates any remapping structure (updates to not-present or
> present entries). Those invalidation commands are used to trap and
> update host structures. This mode was devised for virtualization. I was
> not able to find such "caching mode" on ARM SMMU. Is there any?
> 
> If not, do you have any other suggestion, I mean, besides the
> virtio-based solution.
> 
> Thanks
> 
> Eric

Does SMMU hardware have nested page table support?
Some IOMMUs have this.

If yes then for some workloads this has the potential to perform better
than the invalidation hack with host SMMU performing nested translations
using guest page tables.

-- 
MST

Re: userspace emulated smmu/vfio integration: how to trap updates to the table structures?

[Auger Eric]

Hi Michael,

On 02/05/2017 19:36, Michael S. Tsirkin wrote:
> On Tue, May 02, 2017 at 09:30:39AM +0200, Auger Eric wrote:
>> Hi Will, Robin, Jean-Philippe,
>>
>> I have been working on the integration between user-space emulated
>> SMMU-v3 and VFIO in QEMU. At the moment I fail identifying a proper easy
>> way to trap page table updates. This is requested to keep the host
>> translation structures consistent to guest translation structures.
>>
>> On Intel VTD there is a so-called "caching mode" (CM, see VTD spec
>> paragraph 6.1) that forces the OS to explicitly invalidate caches
>> whenever it updates any remapping structure (updates to not-present or
>> present entries). Those invalidation commands are used to trap and
>> update host structures. This mode was devised for virtualization. I was
>> not able to find such "caching mode" on ARM SMMU. Is there any?
>>
>> If not, do you have any other suggestion, I mean, besides the
>> virtio-based solution.
>>
>> Thanks
>>
>> Eric
> 
> Does SMMU hardware have nested page table support?
> Some IOMMUs have this.
Sorry for the delay. Yes it has and was supposed to serve that use case;
but using the feature was not advised as far as I understand.

Thanks

Eric
> 
> If yes then for some workloads this has the potential to perform better
> than the invalidation hack with host SMMU performing nested translations
> using guest page tables.
>