[RFC 0/2] arm-smmu-v3 tlbi-on-map option

[RFC 0/2] arm-smmu-v3 tlbi-on-map option

[Eric Auger]

This series adds a new tlbi-on-map option to the smmuv3 driver.
When set, the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk is applied for 
LPAE tables and the smmuv3 driver sends TLB invalidations on map.

This mode is useful when running the driver on a guest as it allows
the virtualizer to trap any change to the translation structures.
This is similar to the Intel vtd caching mode (CM).

This is especially needed for VFIO integration integration where
guest mappings must be applied to the physical IOMMU.

At the moment the option only is available for DT probing.

Best Regards

Eric

Eric Auger (2):
  iommu/io-pgtable-arm: flush TLBs when IO_PGTABLE_QUIRK_TLBI_ON_MAP
  arm-smmu-v3: Add tlbi_on_map option

 Documentation/devicetree/bindings/iommu/arm,smmu-v3.txt |  4 ++++
 drivers/iommu/arm-smmu-v3.c                             |  5 +++++
 drivers/iommu/io-pgtable-arm.c                          | 13 +++++++++++--
 3 files changed, 20 insertions(+), 2 deletions(-)

-- 
2.5.5

[RFC 1/2] iommu/io-pgtable-arm: flush TLBs when IO_PGTABLE_QUIRK_TLBI_ON_MAP

[Eric Auger]

This patch implements the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk for
LPAE page tables. It forces TLB invalidations on map.

Signed-off-by: Eric Auger <eric.auger-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 drivers/iommu/io-pgtable-arm.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c
index 6e5df5e..0ad10fc 100644
--- a/drivers/iommu/io-pgtable-arm.c
+++ b/drivers/iommu/io-pgtable-arm.c
@@ -395,6 +395,8 @@ static int arm_lpae_map(struct io_pgtable_ops *ops, unsigned long iova,
 	arm_lpae_iopte *ptep = data->pgd;
 	int ret, lvl = ARM_LPAE_START_LVL(data);
 	arm_lpae_iopte prot;
+	struct io_pgtable *iop = &data->iop;
+	struct io_pgtable_cfg *cfg = &data->iop.cfg;
 
 	/* If no access, then nothing to do */
 	if (!(iommu_prot & (IOMMU_READ | IOMMU_WRITE)))
@@ -402,6 +404,12 @@ static int arm_lpae_map(struct io_pgtable_ops *ops, unsigned long iova,
 
 	prot = arm_lpae_prot_to_pte(data, iommu_prot);
 	ret = __arm_lpae_map(data, iova, paddr, size, prot, lvl, ptep);
+
+	if (cfg->quirks & IO_PGTABLE_QUIRK_TLBI_ON_MAP) {
+		io_pgtable_tlb_add_flush(iop, iova, size,
+					 ARM_LPAE_GRANULE(data), false);
+		io_pgtable_tlb_sync(iop);
+	}
 	/*
 	 * Synchronise all PTE updates for the new mapping before there's
 	 * a chance for anything to kick off a table walk for the new iova.
@@ -673,7 +681,8 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *cfg, void *cookie)
 	u64 reg;
 	struct arm_lpae_io_pgtable *data;
 
-	if (cfg->quirks & ~IO_PGTABLE_QUIRK_ARM_NS)
+	if (cfg->quirks &
+		~(IO_PGTABLE_QUIRK_ARM_NS | IO_PGTABLE_QUIRK_TLBI_ON_MAP))
 		return NULL;
 
 	data = arm_lpae_alloc_pgtable(cfg);
@@ -762,7 +771,7 @@ arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *cfg, void *cookie)
 	struct arm_lpae_io_pgtable *data;
 
 	/* The NS quirk doesn't apply at stage 2 */
-	if (cfg->quirks)
+	if (cfg->quirks & ~(IO_PGTABLE_QUIRK_TLBI_ON_MAP))
 		return NULL;
 
 	data = arm_lpae_alloc_pgtable(cfg);
-- 
2.5.5

[RFC 2/2] arm-smmu-v3: Add tlbi_on_map option

[Eric Auger]

When running a virtual SMMU on a guest we sometimes need to trap
all changes to the translation structures. This is especially useful
to integrate with VFIO. This patch adds a new option that forces
the IO_PGTABLE_QUIRK_TLBI_ON_MAP to be applied on LPAE page tables.

TLBI commands then can be trapped.

Signed-off-by: Eric Auger <eric.auger-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 Documentation/devicetree/bindings/iommu/arm,smmu-v3.txt | 4 ++++
 drivers/iommu/arm-smmu-v3.c                             | 5 +++++
 2 files changed, 9 insertions(+)

diff --git a/Documentation/devicetree/bindings/iommu/arm,smmu-v3.txt b/Documentation/devicetree/bindings/iommu/arm,smmu-v3.txt
index be57550..a25337a 100644
--- a/Documentation/devicetree/bindings/iommu/arm,smmu-v3.txt
+++ b/Documentation/devicetree/bindings/iommu/arm,smmu-v3.txt
@@ -46,6 +46,10 @@ the PCIe specification.
                         devicetree/bindings/interrupt-controller/msi.txt
                       for a description of the msi-parent property.
 
+- tlbi-on-map       : invalidate caches whenever there is an update of
+                      any remapping structure (updates to not-present or
+                      present entries).
+
 - hisilicon,broken-prefetch-cmd
                     : Avoid sending CMD_PREFETCH_* commands to the SMMU.
 
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 380969a..5003ff7 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -597,6 +597,7 @@ struct arm_smmu_device {
 	u32				features;
 
 #define ARM_SMMU_OPT_SKIP_PREFETCH	(1 << 0)
+#define ARM_SMMU_OPT_TLBI_ON_MAP	(1 << 1)
 	u32				options;
 
 	struct arm_smmu_cmdq		cmdq;
@@ -663,6 +664,7 @@ struct arm_smmu_option_prop {
 
 static struct arm_smmu_option_prop arm_smmu_options[] = {
 	{ ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" },
+	{ ARM_SMMU_OPT_TLBI_ON_MAP, "tlbi-on-map" },
 	{ 0, NULL},
 };
 
@@ -1555,6 +1557,9 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain)
 		.iommu_dev	= smmu->dev,
 	};
 
+	if (smmu->options & ARM_SMMU_OPT_TLBI_ON_MAP)
+		pgtbl_cfg.quirks = IO_PGTABLE_QUIRK_TLBI_ON_MAP;
+
 	pgtbl_ops = alloc_io_pgtable_ops(fmt, &pgtbl_cfg, smmu_domain);
 	if (!pgtbl_ops)
 		return -ENOMEM;
-- 
2.5.5

Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option

[Will Deacon]

Hi Eric,

On Sun, Jul 09, 2017 at 05:15:01PM +0200, Eric Auger wrote:
> This series adds a new tlbi-on-map option to the smmuv3 driver.
> When set, the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk is applied for 
> LPAE tables and the smmuv3 driver sends TLB invalidations on map.
> 
> This mode is useful when running the driver on a guest as it allows
> the virtualizer to trap any change to the translation structures.
> This is similar to the Intel vtd caching mode (CM).
> 
> This is especially needed for VFIO integration integration where
> guest mappings must be applied to the physical IOMMU.
> 
> At the moment the option only is available for DT probing.

I'm really not a fan of this approach. If a virtual IOMMU implementation is
advertising itself as an SMMUv3, then it should adhere to the SMMUv3
architecture and not require non-standard behaviour from the driver. If
we're going to allow that, then we're better off going the extra mile and
using a PV approach. Given that the the SMMU3 architecture does *not*
require TLBI on map, then I don't think we should be quirking our behaviour
in this way. The fact that you only have this implemented for DT is the
canary in the coal mine imo.

Will

Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option

[Michael S. Tsirkin]

On Wed, Jul 12, 2017 at 06:54:56PM +0100, Will Deacon wrote:
> Hi Eric,
> 
> On Sun, Jul 09, 2017 at 05:15:01PM +0200, Eric Auger wrote:
> > This series adds a new tlbi-on-map option to the smmuv3 driver.
> > When set, the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk is applied for 
> > LPAE tables and the smmuv3 driver sends TLB invalidations on map.
> > 
> > This mode is useful when running the driver on a guest as it allows
> > the virtualizer to trap any change to the translation structures.
> > This is similar to the Intel vtd caching mode (CM).
> > 
> > This is especially needed for VFIO integration integration where
> > guest mappings must be applied to the physical IOMMU.
> > 
> > At the moment the option only is available for DT probing.
> 
> I'm really not a fan of this approach. If a virtual IOMMU implementation is
> advertising itself as an SMMUv3, then it should adhere to the SMMUv3
> architecture and not require non-standard behaviour from the driver.

You can also just enable e.g. building existing intel VTD code on ARM,
that emulation is already there.

> If
> we're going to allow that, then we're better off going the extra mile and
> using a PV approach. Given that the the SMMU3 architecture does *not*
> require TLBI on map, then I don't think we should be quirking our behaviour
> in this way. 

I think using hardware support for nesting is the right final
solution. It will take some time though. Given this, what should
we do meanwhile?

Assuming that's the final destination, a simple quirk like this
seems to be preferable to virtio-iommu which we'll never be
able to offload to hardware.

>The fact that you only have this implemented for DT is the
> canary in the coal mine imo.
> 
> Will

As to the last point, I agree absolutely. Need to support ACPI
as well or nothing. It would be nicer IMHO to get it from some
kind of device register, assuming ARM can promise to reserve
some bits for hypervisors to play with.

-- 
MST

Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option

[Jean-Philippe Brucker]

On 12/07/17 23:07, Michael S. Tsirkin wrote:
[...]
> I think using hardware support for nesting is the right final
> solution. It will take some time though. Given this, what should
> we do meanwhile?
> 
> Assuming that's the final destination, a simple quirk like this
> seems to be preferable to virtio-iommu which we'll never be
> able to offload to hardware.

That's not entirely true. virtio-iommu will have an extension for hardware
nesting support. It was presented in my initial proposal, and I've made
significant progress since then.

Thanks,
Jean

Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option

[Michael S. Tsirkin]

On Thu, Jul 13, 2017 at 10:29:42AM +0100, Jean-Philippe Brucker wrote:
> On 12/07/17 23:07, Michael S. Tsirkin wrote:
> [...]
> > I think using hardware support for nesting is the right final
> > solution. It will take some time though. Given this, what should
> > we do meanwhile?
> > 
> > Assuming that's the final destination, a simple quirk like this
> > seems to be preferable to virtio-iommu which we'll never be
> > able to offload to hardware.
> 
> That's not entirely true. virtio-iommu will have an extension for hardware
> nesting support. It was presented in my initial proposal, and I've made
> significant progress since then.
> 
> Thanks,
> Jean

I don't recall seeing this.

Hardware specific extensions to virtio would be interesting, the
difficulty is in finding the balance between enabling minor quirks and
each vendor going their own way.

Is this the proposal you refer to?
https://www.spinics.net/lists/kvm/msg147990.html

I couldn't find any mention of nesting, it seems to
say that map requests are relayed through host.

-- 
MST

Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option

[Jean-Philippe Brucker]

On 13/07/17 18:44, Michael S. Tsirkin wrote:
> On Thu, Jul 13, 2017 at 10:29:42AM +0100, Jean-Philippe Brucker wrote:
>> On 12/07/17 23:07, Michael S. Tsirkin wrote:
>> [...]
>>> I think using hardware support for nesting is the right final
>>> solution. It will take some time though. Given this, what should
>>> we do meanwhile?
>>>
>>> Assuming that's the final destination, a simple quirk like this
>>> seems to be preferable to virtio-iommu which we'll never be
>>> able to offload to hardware.
>>
>> That's not entirely true. virtio-iommu will have an extension for hardware
>> nesting support. It was presented in my initial proposal, and I've made
>> significant progress since then.
>>
>> Thanks,
>> Jean
> 
> I don't recall seeing this.
> 
> Hardware specific extensions to virtio would be interesting, the
> difficulty is in finding the balance between enabling minor quirks and
> each vendor going their own way.

Yes, in order to avoid too many quirks and vendor-specific formats, we'd
like the guest to only manage page tables, which have relatively stable
format, while the host kernel manages context tables (PASID tables) and
other structures, that are more volatile across implementations.

Unavoidably, a few architecture-specific details need to be described in
the API, but it seems manageable. The host tells the guest which page
table format is supported by hardware, and the guest sends back a page
directory pointer along with a few configuration parameters.

In the guest, page table operations may be abstracted in a module and used
by multiple IOMMU drivers (what ARM does in drivers/iommu/io-pgtable-arm.c
for the various vendor drivers). This abstraction is quite helpful to find
out which information needs to be exchanged between virtio-iommu device
and driver. For ARM-based IOMMUs it amounts to 6 configuration registers
and 4 quirk bits.

The other problem is forwarding TLB invalidations to the host kernel.
There is an ongoing discussion [1] about the best way to do it in VFIO,
and it seems like the format can stay mostly generic, with a few
optimization hints depending on the hardware IOMMU.

> Is this the proposal you refer to?
> https://www.spinics.net/lists/kvm/msg147990.html

Yes, I'm referring to "II. Page table sharing" in
https://www.spinics.net/lists/kvm/msg147993.html

But it is now mostly obsolete. Lots of things had to change while writing
a prototype and following public discussions. At the moment my focus is on
tidying up the base, but I will send another RFC afterwards.

Thanks,
Jean

[1] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg01117.html

Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option

[Michael S. Tsirkin]

On Thu, Jul 13, 2017 at 08:17:49PM +0100, Jean-Philippe Brucker wrote:
> But it is now mostly obsolete. Lots of things had to change while writing
> a prototype and following public discussions. At the moment my focus is on
> tidying up the base, but I will send another RFC afterwards.

Thanks!  Pls remember to copy virtio-dev at oasis (you need to subscribe
to post) as that's a requirement for anything involving host/guest APIs.

-- 
MST

Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option

[Auger Eric]

Hi Will, Michael,

On 12/07/2017 19:54, Will Deacon wrote:
> Hi Eric,
> 
> On Sun, Jul 09, 2017 at 05:15:01PM +0200, Eric Auger wrote:
>> This series adds a new tlbi-on-map option to the smmuv3 driver.
>> When set, the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk is applied for 
>> LPAE tables and the smmuv3 driver sends TLB invalidations on map.
>>
>> This mode is useful when running the driver on a guest as it allows
>> the virtualizer to trap any change to the translation structures.
>> This is similar to the Intel vtd caching mode (CM).
>>
>> This is especially needed for VFIO integration integration where
>> guest mappings must be applied to the physical IOMMU.
>>
>> At the moment the option only is available for DT probing.
> 
> I'm really not a fan of this approach. If a virtual IOMMU implementation is
> advertising itself as an SMMUv3, then it should adhere to the SMMUv3
> architecture and not require non-standard behaviour from the driver. If
> we're going to allow that, then we're better off going the extra mile and
> using a PV approach. Given that the the SMMU3 architecture does *not*
> require TLBI on map, then I don't think we should be quirking our behaviour
> in this way. The fact that you only have this implemented for DT is the
> canary in the coal mine imo.

I did not proceed with ACPI integration as I focused on the QEMU
integration POC instead. I intended to propose a new model id just like

12275bf  iommu/arm-smmu-v3, acpi: Add temporary Cavium SMMU-V3 IORT
model number definitions?

Is it really costly to maintain such a quirk (which by the way does the
same thing as Intel caching mode, unfortunately not in an HW specified
manner) compared to other quirks?

Having an smmu emulated device integrated with vfio can always be useful
for debugging purpose and also to compare perfs/functionality against
virtio-iommu solution. We said both solutions could co-exist, serving
different purposes. Now if you close the door to vfio integration,
emulated smmu just serves the purpose of guests using isolated virtio
devices which limits quite a lot its usage, somehow questioning the
efforts invested to develop/upstream it.

Besides, before any further decision, I need to complete the DPDK use
case enablement as I get a storm of page tlbi's although hugepages are
used. And this hasn't worked yet ...

Thanks

Eric


> 
> Will
>