Re: [PATCH 1/5] iommu: Enhance domain allocation code to take additional flags

[Jason Gunthorpe <jgg@ziepe.ca>]

On Fri, Aug 23, 2024 at 10:47:46AM +0800, Baolu Lu wrote:
> > Why? What exactly is the issue?
> > 
> > It is inhernetly wrong to behave differently based on DMA API or VFIO.
> > They are not different things.
> > 
> > If you have different behaviors and different properies, like AMD's
> > PASID, then they should be described and mapped to some kind of flag.
> > 
> > Otherwise the driver should always allocate a paging domain that gives
> > the highest performance.
> 
> It relates to Intel VT-d's nested translation. Intel VT-d has two types
> of page table formats for DMA translation: first level and second level.
> In nested translation, the first level page table is used for first-
> stage translation, and the second level page table is used for second-
> stage translation.
> 
> The iommu driver for vIOMMU in the guest kernel must use the first level
> page table format for kernel DMA. This page table will then be nested on
> a second level page table in the VMM host kernel.

The guest kernel must know that the IOMMU can only support the first
level format in this case! There is no other option!

> Our current design uses the first level page table for both the host and
> guest kernel for simplicity. This is why we use different page table
> formats for IOMMU_DOMAIN_DMA and IOMMU_DOMAIN_UNMANAGED.

You should just always use the first level format for every paging
domain except when IOMMU_HWPT_ALLOC_NEST_PARENT is set.

IOMMU_HWPT_ALLOC_NEST_PARENT was added specifically to solve this
problem.

Ideally the guest kernel will reject IOMMU_HWPT_ALLOC_NEST_PARENT as
not supported when allocated or attached. ARM does.

> I am not sure about other architectures, like AMD, ARM and RISC-V.
> Perhaps all of them have the similar need?

ARM/AMD/VTD all have the two page table problem.

ARM informs the guest if S1/S2 is available via IDR which is like
VT-D's ECAP. Only S1 can be used in the nested guest VM.

AMD has a global amd_iommu_pgtable, it must be set to AMD_IOMMU_V2
within a nested guest VM. The only way I saw for that to happen was
via kernel command line parameter? Was expecting some ACPI or
FEATURE_XXX thing? Vasant?

VTD should add some discoverability that the second stage is not
supported and behave as above.

Jason