RE: [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node

[Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>]

Hi Eric,

> -----Original Message-----
> From: Eric Auger [mailto:eric.auger@redhat.com]
> Sent: 30 September 2021 10:48
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> linux-arm-kernel@lists.infradead.org; linux-acpi@vger.kernel.org;
> iommu@lists.linux-foundation.org; Jean-Philippe Brucker
> <Jean-Philippe.Brucker@arm.com>
> Cc: Linuxarm <linuxarm@huawei.com>; lorenzo.pieralisi@arm.com;
> joro@8bytes.org; robin.murphy@arm.com; will@kernel.org; wanghuiqiang
> <wanghuiqiang@huawei.com>; Guohanjun (Hanjun Guo)
> <guohanjun@huawei.com>; steven.price@arm.com; Sami.Mujawar@arm.com;
> jon@solid-run.com; yangyicong <yangyicong@huawei.com>
> Subject: Re: [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node
> 
> Hi Shameer,
> 
> On 8/5/21 10:07 AM, Shameer Kolothum wrote:
> > Hi,
> >
> > The series adds support to IORT RMR nodes specified in IORT
> > Revision E.b -ARM DEN 0049E[0]. RMR nodes are used to describe
> > memory ranges that are used by endpoints and require a unity
> > mapping in SMMU.
> 
> I used your series and RMRs to force a guest iommu (vSMMUv3 nested stage
> use case) to have a flat mapping for IOVAs within [0x8000000, 0x8100000]
> (matching MSI_IOVA_BASE and MSI_IOVA_LENGTH) used by the host to map
> MSI
> physical doorbells.
> 
> That way when an assigned device protected by a vSMMUv3 implemented
> upon
> nested stage issues an MSI transaction, let's say using IOVA=0x8000000,
> we would get:
>                     S1 (guest)           S2 (host)
> 0x8000000            0x8000000            Physical DB
> 
> This method was suggested by Jean-Philippe (added in CC) and it
> simplifies the nested stage integration because we don't have to care
> about nested stage MSI bindings.
> 
> However if I understand correctly we cannot define a range of SIDs using
> the same RMR (due to the single mapping bit which must be set, Table 5
> flags format). This is a spec restriction and not an issue with your series.

Yes. The spec currently mandates single mapping bit to be set.

> 
> As VFIO devices can be hot-plugged we thus need to create as many RMR
> nodes as potential BDFs, leading to 256 * 6 = 1536 RMR nodes if you have
> 5 pcie root ports as it is usual in VMs. Then this causes some trouble
> at qemu level for instance, wrt migration. See [RFC]
> hw/arm/virt-acpi-build: Add IORT RMR regions to handle MSI nested binding.
> 
> Do you know if there is a plan to remove the single mapping limitation
> in the spec?

I would imagine so. In an earlier comment[1], Robin did mention about possible
relaxing of this in future spec revision.

Thanks,
Shameer
1. https://lore.kernel.org/linux-arm-kernel/15c7fac0-11a8-0cdb-aac3-b5d8feb8f066@arm.com/

> Thanks
> 
> Eric
> >
> > We have faced issues with 3408iMR RAID controller cards which
> > fail to boot when SMMU is enabled. This is because these
> > controllers make use of host memory for various caching related
> > purposes and when SMMU is enabled the iMR firmware fails to
> > access these memory regions as there is no mapping for them.
> > IORT RMR provides a way for UEFI to describe and report these
> > memory regions so that the kernel can make a unity mapping for
> > these in SMMU.
> >
> > Change History:
> >
> > v6 --> v7
> >
> > The only change from v6 is the fix pointed out by Steve to
> > the SMMUv2 SMR bypass install in patch #8.
> >
> > Thanks to the Tested-by tags by Laurentiu with SMMUv2 and
> > Hanjun/Huiqiang with SMMUv3 for v6. I haven't added the tags
> > yet as the series still needs more review[1].
> >
> > Feedback and tests on this series is very much appreciated.
> >
> > v5 --> v6
> > - Addressed comments from Robin & Lorenzo.
> >   : Moved iort_parse_rmr() to acpi_iort_init() from
> >     iort_init_platform_devices().
> >   : Removed use of struct iort_rmr_entry during the initial
> >     parse. Using struct iommu_resv_region instead.
> >   : Report RMR address alignment and overlap errors, but continue.
> >   : Reworked arm_smmu_init_bypass_stes() (patch # 6).
> > - Updated SMMUv2 bypass SMR code. Thanks to Jon N (patch #8).
> > - Set IOMMU protection flags(IOMMU_CACHE, IOMMU_MMIO) based
> >   on Type of RMR region. Suggested by Jon N.
> >
> > Thanks,
> > Shameer
> > [0] https://developer.arm.com/documentation/den0049/latest/
> > [1]
> https://lore.kernel.org/linux-acpi/20210716083442.1708-1-shameerali.koloth
> um.thodi@huawei.com/T/#m043c95b869973a834b2fd57f3e1ed0325c84f3b7
> > ------
> > v4 --> v5
> >  -Added a fw_data union to struct iommu_resv_region and removed
> >   struct iommu_rmr (Based on comments from Joerg/Robin).
> >  -Added iommu_put_rmrs() to release mem.
> >  -Thanks to Steve for verifying on SMMUv2, but not added the Tested-by
> >   yet because of the above changes.
> >
> > v3 -->v4
> > -Included the SMMUv2 SMR bypass install changes suggested by
> >  Steve(patch #7)
> > -As per Robin's comments, RMR reserve implementation is now
> >  more generic  (patch #8) and dropped v3 patches 8 and 10.
> > -Rebase to 5.13-rc1
> >
> > RFC v2 --> v3
> >  -Dropped RFC tag as the ACPICA header changes are now ready to be
> >   part of 5.13[0]. But this series still has a dependency on that patch.
> >  -Added IORT E.b related changes(node flags, _DSM function 5 checks for
> >   PCIe).
> >  -Changed RMR to stream id mapping from M:N to M:1 as per the spec and
> >   discussion here[1].
> >  -Last two patches add support for SMMUv2(Thanks to Jon Nettleton!)
> > ------
> >
> > Jon Nettleton (1):
> >   iommu/arm-smmu: Get associated RMR info and install bypass SMR
> >
> > Shameer Kolothum (8):
> >   iommu: Introduce a union to struct iommu_resv_region
> >   ACPI/IORT: Add support for RMR node parsing
> >   iommu/dma: Introduce generic helper to retrieve RMR info
> >   ACPI/IORT: Add a helper to retrieve RMR memory regions
> >   iommu/arm-smmu-v3: Introduce strtab init helper
> >   iommu/arm-smmu-v3: Refactor arm_smmu_init_bypass_stes() to force
> >     bypass
> >   iommu/arm-smmu-v3: Get associated RMR info and install bypass STE
> >   iommu/dma: Reserve any RMR regions associated with a dev
> >
> >  drivers/acpi/arm64/iort.c                   | 172
> +++++++++++++++++++-
> >  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c |  76 +++++++--
> >  drivers/iommu/arm/arm-smmu/arm-smmu.c       |  48 ++++++
> >  drivers/iommu/dma-iommu.c                   |  89 +++++++++-
> >  include/linux/acpi_iort.h                   |   7 +
> >  include/linux/dma-iommu.h                   |  13 ++
> >  include/linux/iommu.h                       |  11 ++
> >  7 files changed, 393 insertions(+), 23 deletions(-)
> >