Re: [RFC PATCH 2/5] hw/arm/smmuv3: Add initial support for SMMUv3 Nested device

[Donald Dutile <ddutile@redhat.com>]

On 11/27/24 11:00 AM, Jason Gunthorpe wrote:
> On Wed, Nov 27, 2024 at 10:21:24AM +0000, Shameerali Kolothum Thodi wrote:
>>> For SMMUv3, NVIDIA-specific vCMDQ, it needs a parameter to state that
>>> specifically,
>>> since I'm concluding from reading the SMMUv3 version G.a spec, that
>>> ECMDQ was added
>>> to be able to assign an ECMDQ to a VM,
>>
>> Not sure the intention of ECMDQ as per that specification is to assign
>> it to a VM. I think the main idea behind it is to have one Command Queue
>> per host CPU to eliminate lock contention while submitting commands
>> to SMMU.
> 
> Right
> 
>> AFAIK it is not safe to assign one of the ECMDQ to guest yet. I think there is no
>> way you can associate a VMID with ECMDQ. So there is no plan to
>> support ARM ECMDQ now.
> 
> Yep
>
'Yet' ...
The association would be done via the VMM -- no different then what associates
an assigned device to a VM today -- no hw-level (VM-)ID needed; a matter of exposing
it to the VM, or not; or mapping the (virtual) CMDQ to the mapped/associated ECMDQ.
They are purposedly mapped 64K apart from each other, enabling page-level protection,
which I doubt is a per-CPU req for lock contention avoidance (large-cache-block
spaced would be sufficient, even 4k; it's 64k spaced btwn ECMDQ regs .. the largest
ARM page size.

Summary: let's not assume this can't happen, and the chosen cmdline prevents it.

>> NVIDIA VCMDQ is a completely vendor specific one. Perhaps ARM may come
>> up with an assignable CMDQ in future though.
> 
> Yes, it is easy to imagine an ECMDQ extension that provides the same HW
> features that VCMDQ has in future. I hope ARM will develop one.
> 
Right, so how to know what op is being "accel"'d wrt smmuv3.

>>> ... and all needs to be per-instance ....
>>> ... libvirt  (or any other VMM orchestrator) will need to determine
>>> compatibility for
>>>       live migration. e.g., can one live migrate an accel=nv-vcmdq-based VM to
>>> a host with
>>>       accel=ecmdq support?  only nv-vcmdq?  what if there are version diffs of
>>> nv-vcmdq over time?
>>>       -- apologies, but I don't know the minute details of nv-vcmdq to
>>> determine if that's unlikely or not.
>>
>> Yes. This require more thought. But our first aim is get the basic smmuv3-accel
>> support.
> 
> Yeah, there is no live migration support yet in the SMMU qmeu driver,
> AFAIK?
> 
> When it gets done the supported options will have to be considered
> 
> Jason
>