SMMU Stage 2 translation in QEMU

SMMU Stage 2 translation in QEMU

[shashi.mallela]

Hi All,

I am trying to understand the approach required for an emulated SMMU to
convert IPAs(from each qemu guest) to PAs(respective host addresses)
using stage 2 tables.

The questions i have are:-

1) Since SMMU stage 2 tables are expected to be created and managed by
a hypervisor,if there is no kvm support,who is responsible to create
the stage 2 tables in host memory? is it even a valid use case to
consider smmu stage 2 support with no hypervisor present?

2) with SMMU emulated by qemu:-
a) who is responsible for hosting and programming the stage 2 table
base address registers? 
b) what are the APIs available in qemu to access the stage 2 tables?
(will address_space_ API variants apply here?)

3) if each qemu instance (for a guest) emulates an SMMU,will there be a
need to protect concurrent access of stage 2 table(in host) by each of
the SMMUs?

Thanks
Shashi

Re: SMMU Stage 2 translation in QEMU

[Peter Maydell]

On Thu, 9 Sept 2021 at 21:18, <shashi.mallela@linaro.org> wrote:
> I am trying to understand the approach required for an emulated SMMU to
> convert IPAs(from each qemu guest) to PAs(respective host addresses)
> using stage 2 tables.
>
> The questions i have are:-
>
> 1) Since SMMU stage 2 tables are expected to be created and managed by
> a hypervisor,if there is no kvm support,who is responsible to create
> the stage 2 tables in host memory? is it even a valid use case to
> consider smmu stage 2 support with no hypervisor present?

So what exactly is the use case you're talking about here?
Do you mean "purely emulated QEMU, but using a host hardware
IOMMU to do device passthrough of host devices to the guest"?
Or do you mean "purely emulated QEMU with an emulated SMMU
that handles accesses to emulated devices" ?

thanks
-- PMM

Re: SMMU Stage 2 translation in QEMU

[shashi.mallela]

I am referring to the latter,"purely emulated QEMU with an emulated
SMMU that handles accesses to emulated devices"

Thanks
Shashi

On Fri, 2021-09-10 at 10:25 +0100, Peter Maydell wrote:
> On Thu, 9 Sept 2021 at 21:18, <shashi.mallela@linaro.org> wrote:
> > I am trying to understand the approach required for an emulated
> > SMMU to
> > convert IPAs(from each qemu guest) to PAs(respective host
> > addresses)
> > using stage 2 tables.
> > 
> > The questions i have are:-
> > 
> > 1) Since SMMU stage 2 tables are expected to be created and managed
> > by
> > a hypervisor,if there is no kvm support,who is responsible to
> > create
> > the stage 2 tables in host memory? is it even a valid use case to
> > consider smmu stage 2 support with no hypervisor present?
> 
> So what exactly is the use case you're talking about here?
> Do you mean "purely emulated QEMU, but using a host hardware
> IOMMU to do device passthrough of host devices to the guest"?
> Or do you mean "purely emulated QEMU with an emulated SMMU
> that handles accesses to emulated devices" ?
> 
> thanks
> -- PMM

Re: SMMU Stage 2 translation in QEMU

[Peter Maydell]

On Fri, 10 Sept 2021 at 13:39, <shashi.mallela@linaro.org> wrote:
>
> I am referring to the latter,"purely emulated QEMU with an emulated
> SMMU that handles accesses to emulated devices"

In that case, the stage 2 tables are set up by the guest
code (running at emulated EL2), just as they would be if
it were running on real hardware.

-- PMM

Re: SMMU Stage 2 translation in QEMU

[shashi.mallela]

So that would be the driver code running in guest OS because i see
tables being setup by arm-smmu driver code in linux,which is similar to
what happens with ITS(table base addresses programmed in registers by
linux driver).

On Fri, 2021-09-10 at 13:54 +0100, Peter Maydell wrote:
> On Fri, 10 Sept 2021 at 13:39, <shashi.mallela@linaro.org> wrote:
> > I am referring to the latter,"purely emulated QEMU with an emulated
> > SMMU that handles accesses to emulated devices"
> 
> In that case, the stage 2 tables are set up by the guest
> code (running at emulated EL2), just as they would be if
> it were running on real hardware.
> 
> -- PMM

Re: SMMU Stage 2 translation in QEMU

[Eric Auger]

Hi Shashi,

On 9/10/21 3:32 PM, shashi.mallela@linaro.org wrote:
> So that would be the driver code running in guest OS because i see
> tables being setup by arm-smmu driver code in linux,which is similar to
> what happens with ITS(table base addresses programmed in registers by
> linux driver).
Stage2 SMMU tables are generally not used today. Only stage 1 is used,
even with VFIO (with KVM).
SMMUv3 upstream driver does not support S1+S2 (nested). It only supports
either S1 or S2. Enabling S2 can be done through VFIO driver, if you
select the VFIO_TYPE1_NESTING_IOMMU IOMMU type. This then calls IOMMU
.enable_nesting  = arm_smmu_enable_nesting which sets smmu_domain->stage
= ARM_SMMU_DOMAIN_NESTED. But the name is misleading as it actually
forces the use of S2 instead of S1.

However if you look at QEMU VFIO code, no one uses VFIO_TYPE1_NESTING_IOMMU.

Note I have worked on 2 stage integration for years, without much
success yet:
[RFC v9 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration

[PATCH v15 00/12] SMMUv3 Nested Stage Setup (IOMMU part)
[PATCH v13 00/13] SMMUv3 Nested Stage Setup (VFIO part)


Thanks

Eric
>
> On Fri, 2021-09-10 at 13:54 +0100, Peter Maydell wrote:
>> On Fri, 10 Sept 2021 at 13:39, <shashi.mallela@linaro.org> wrote:
>>> I am referring to the latter,"purely emulated QEMU with an emulated
>>> SMMU that handles accesses to emulated devices"
>> In that case, the stage 2 tables are set up by the guest
>> code (running at emulated EL2), just as they would be if
>> it were running on real hardware.
>>
>> -- PMM

Re: SMMU Stage 2 translation in QEMU

[shashi.mallela]

Hi Eric,
On Mon, 2021-09-13 at 10:19 +0200, Eric Auger wrote:
> Hi Shashi,
> 
> On 9/10/21 3:32 PM, shashi.mallela@linaro.org wrote:
> > So that would be the driver code running in guest OS because i see
> > tables being setup by arm-smmu driver code in linux,which is
> > similar to
> > what happens with ITS(table base addresses programmed in registers
> > by
> > linux driver).
> Stage2 SMMU tables are generally not used today. Only stage 1 is
> used,
> even with VFIO (with KVM).
The stage 2 SMMU capability is required for qemu sbsa reference
platform to satisfy the sbsa level 3 acs(arm compliance suite)
requirements.

> SMMUv3 upstream driver does not support S1+S2 (nested). It only
> supports
> either S1 or S2. Enabling S2 can be done through VFIO driver, if you
> select the VFIO_TYPE1_NESTING_IOMMU IOMMU type. This then calls IOMMU
> .enable_nesting  = arm_smmu_enable_nesting which sets smmu_domain-
> >stage
> = ARM_SMMU_DOMAIN_NESTED. But the name is misleading as it actually
> forces the use of S2 instead of S1.
> 
> However if you look at QEMU VFIO code, no one uses
> VFIO_TYPE1_NESTING_IOMMU.
Since the current smmuv3 implementation in qemu advertises only stage 1
support,from the qemu device point of view is the stage 2 support
included in the integration effort you mentioned or does it need to be
taken up from scratch?
> 
> Note I have worked on 2 stage integration for years, without much
> success yet:
> [RFC v9 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration
> 
> [PATCH v15 00/12] SMMUv3 Nested Stage Setup (IOMMU part)
> [PATCH v13 00/13] SMMUv3 Nested Stage Setup (VFIO part)
> 
> 
> Thanks
> 
> Eric
> > On Fri, 2021-09-10 at 13:54 +0100, Peter Maydell wrote:
> > > On Fri, 10 Sept 2021 at 13:39, <shashi.mallela@linaro.org> wrote:
> > > > I am referring to the latter,"purely emulated QEMU with an
> > > > emulated
> > > > SMMU that handles accesses to emulated devices"
> > > In that case, the stage 2 tables are set up by the guest
> > > code (running at emulated EL2), just as they would be if
> > > it were running on real hardware.
> > > 
> > > -- PMM

Re: SMMU Stage 2 translation in QEMU

[Eric Auger]

Hi Shashi,

On 9/14/21 5:57 PM, shashi.mallela@linaro.org wrote:
> Hi Eric,
> On Mon, 2021-09-13 at 10:19 +0200, Eric Auger wrote:
>> Hi Shashi,
>>
>> On 9/10/21 3:32 PM, shashi.mallela@linaro.org wrote:
>>> So that would be the driver code running in guest OS because i see
>>> tables being setup by arm-smmu driver code in linux,which is
>>> similar to
>>> what happens with ITS(table base addresses programmed in registers
>>> by
>>> linux driver).
>> Stage2 SMMU tables are generally not used today. Only stage 1 is
>> used,
>> even with VFIO (with KVM).
> The stage 2 SMMU capability is required for qemu sbsa reference
> platform to satisfy the sbsa level 3 acs(arm compliance suite)
> requirements.
>
>> SMMUv3 upstream driver does not support S1+S2 (nested). It only
>> supports
>> either S1 or S2. Enabling S2 can be done through VFIO driver, if you
>> select the VFIO_TYPE1_NESTING_IOMMU IOMMU type. This then calls IOMMU
>> .enable_nesting  = arm_smmu_enable_nesting which sets smmu_domain-
>>> stage
>> = ARM_SMMU_DOMAIN_NESTED. But the name is misleading as it actually
>> forces the use of S2 instead of S1.
>>
>> However if you look at QEMU VFIO code, no one uses
>> VFIO_TYPE1_NESTING_IOMMU.
> Since the current smmuv3 implementation in qemu advertises only stage 1
> support,from the qemu device point of view is the stage 2 support
> included in the integration effort you mentioned or does it need to be
> taken up from scratch?

no it needs to be handled from scratch in emulation mode. My integration
efforts were dedicated to physical SMMU nested stage enablement.

Thanks

Eric
>> Note I have worked on 2 stage integration for years, without much
>> success yet:
>> [RFC v9 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration
>>
>> [PATCH v15 00/12] SMMUv3 Nested Stage Setup (IOMMU part)
>> [PATCH v13 00/13] SMMUv3 Nested Stage Setup (VFIO part)
>>
>>
>> Thanks
>>
>> Eric
>>> On Fri, 2021-09-10 at 13:54 +0100, Peter Maydell wrote:
>>>> On Fri, 10 Sept 2021 at 13:39, <shashi.mallela@linaro.org> wrote:
>>>>> I am referring to the latter,"purely emulated QEMU with an
>>>>> emulated
>>>>> SMMU that handles accesses to emulated devices"
>>>> In that case, the stage 2 tables are set up by the guest
>>>> code (running at emulated EL2), just as they would be if
>>>> it were running on real hardware.
>>>>
>>>> -- PMM