Re: [PATCH v2 17/19] iommu/arm-smmu-v3: Add arm_smmu_viommu_cache_invalidate

[Jason Gunthorpe <jgg@nvidia.com>]

On Wed, Sep 11, 2024 at 06:25:16AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Friday, September 6, 2024 2:22 AM
> > 
> > On Thu, Sep 05, 2024 at 11:00:49AM -0700, Nicolin Chen wrote:
> > > On Thu, Sep 05, 2024 at 01:20:39PM -0300, Jason Gunthorpe wrote:
> > > > On Tue, Aug 27, 2024 at 09:59:54AM -0700, Nicolin Chen wrote:
> > > >
> > > > > +static int arm_smmu_viommu_cache_invalidate(struct
> > iommufd_viommu *viommu,
> > > > > +					    struct iommu_user_data_array
> > *array)
> > > > > +{
> > > > > +	struct iommu_domain *domain =
> > iommufd_viommu_to_parent_domain(viommu);
> > > > > +
> > > > > +	return __arm_smmu_cache_invalidate_user(
> > > > > +			to_smmu_domain(domain), viommu, array);
> > > >
> > > > I'd like to have the viommu struct directly hold the VMID. The nested
> > > > parent should be sharable between multiple viommus, it doesn't make
> > > > any sense that it would hold the vmid.
> > > >
> > > > This is struggling because it is trying too hard to not have the
> > > > driver allocate the viommu, and I think we should just go ahead and do
> > > > that. Store the vmid, today copied from the nesting parent in the vmid
> > > > private struct. No need for iommufd_viommu_to_parent_domain(), just
> > > > rework the APIs to pass the vmid down not a domain.
> > >
> > > OK. When I designed all this stuff, we still haven't made mind
> > > about sharing the s2 domain, i.e. moving the VMID, which might
> > > need a couple of more patches to achieve.
> > 
> > Yes, many more patches, and don't try to do it now.. But we can copy
> > the vmid from the s2 and place it in the viommu struct during
> > allocation time.
> 
> does it assume that a viommu object cannot span multiple physical
> IOMMUs so there is only one vmid per viommu?

Yes, the viommu is not intended to cross physical iommus, it is
intended to contain objects, like invalidation queues, that are tied
to single piommus only.

If someone does want to make vIOMMU that unifies multiple pIOMMUS, and
they have the feature set that would make that possible, then they
will need multiple IOMMUFD VFIOMMU objects and will have to divide up
the nested domains appropriately.

Jason