nested-smmuv3 topic for QEMU/libvirt, Nov 2024

nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Nicolin Chen]

Hi,

This is a continued discussion following previous month's:
https://lore.kernel.org/qemu-devel/Zvr%2Fbf7KgLN1cjOl@Asurada-Nvidia/

Kernel changes are getting closer to merge, as Jason's planning to
take vIOMMU series and smmuv3_nesting series into the iommufd tree:
https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
https://lore.kernel.org/all/0-v4-9e99b76f3518+3a8-smmuv3_nesting_jgg@nvidia.com/

So, I think it's probably a good time to align with each others and
talk about kicking off some QEMU upstream work in the month ahead.

@Shameer,
Do you have some update on the pluggable smmuv3 module?

Updates on my side:
1) I have kept uAPI updated to the latest version and verified too.
   There should be some polishing changes depending on how the basic
   nesting infrastructure would look like from Intel/Duan's work.
2) I got some help from NVIDIA folks for the libvirt task. And they
   have done some drafting and are now verifying the PCI topology
   with "iommu=none".

Once the pluggable smmuv3 module is ready to test, we will make some
change to libvirt for that and drop the auto-assigning patches from
the VIRT code, so as to converge for a libvirt+QEMU test.

FWIW, Robin requested a different solution for MSI mapping [1], v.s.
the RMR one that we have been using since Eric's work. I drafted a
few VFIO/IOMMUFD patches for that, yet paused for getting the vIOMMU
series merged to this kernel cycle. I plan to continue in Nov/Dec.
So, for the near term we will continue with the RMR solution, until
we have something solid later.

[1] https://lore.kernel.org/linux-iommu/ZrVN05VylFq8lK4q@Asurada-Nvidia/

Thanks
Nicolin

Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Jason Gunthorpe]

On Thu, Oct 31, 2024 at 09:09:20PM -0700, Nicolin Chen wrote:

> FWIW, Robin requested a different solution for MSI mapping [1], v.s.
> the RMR one that we have been using since Eric's work. I drafted a
> few VFIO/IOMMUFD patches for that, 

I also talked to MarcZ about this at LPC and he seems willing to
consider it. It took a bit to explain everything though. So I think we
should try in Nov/Dec

Jason

Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Zhangfei Gao]

Hi, Nico

On Fri, 1 Nov 2024 at 19:55, Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> On Thu, Oct 31, 2024 at 09:09:20PM -0700, Nicolin Chen wrote:
>
> > FWIW, Robin requested a different solution for MSI mapping [1], v.s.
> > the RMR one that we have been using since Eric's work. I drafted a
> > few VFIO/IOMMUFD patches for that,
>
> I also talked to MarcZ about this at LPC and he seems willing to
> consider it. It took a bit to explain everything though. So I think we
> should try in Nov/Dec

When boot qemu, reports this

qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI BAR?
qemu-system-aarch64: vfio_container_dma_map(0xaaaadd30f110,
0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)
qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI BAR?
qemu-system-aarch64: vfio_container_dma_map(0xaaaadd2bc310,
0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)
qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI BAR?
qemu-system-aarch64: vfio_container_dma_map(0xaaaadcf90000,
0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)

Will this also be solved in the new MSI mapping patchset?

Thanks

RE: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Zhangfei Gao <zhangfei.gao@linaro.org>
> Sent: Monday, December 2, 2024 6:05 AM
> To: Jason Gunthorpe <jgg@nvidia.com>
> Cc: Nicolin Chen <nicolinc@nvidia.com>; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; Eric Auger
> <eric.auger@redhat.com>; Mostafa Saleh <smostafa@google.com>; qemu-
> arm@nongnu.org; qemu-devel@nongnu.org; Peter Maydell
> <peter.maydell@linaro.org>; Jean-Philippe Brucker <jean-
> philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> <mshavit@google.com>; Andrea Bolognani <abologna@redhat.com>;
> Michael S. Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>;
> nathanc@nvidia.com; arighi@nvidia.com; ianm@nvidia.com;
> jan@nvidia.com; mochs@nvidia.com
> Subject: Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
> 
> Hi, Nico
> 
> On Fri, 1 Nov 2024 at 19:55, Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > On Thu, Oct 31, 2024 at 09:09:20PM -0700, Nicolin Chen wrote:
> >
> > > FWIW, Robin requested a different solution for MSI mapping [1], v.s.
> > > the RMR one that we have been using since Eric's work. I drafted a
> > > few VFIO/IOMMUFD patches for that,
> >
> > I also talked to MarcZ about this at LPC and he seems willing to
> > consider it. It took a bit to explain everything though. So I think we
> > should try in Nov/Dec
> 
> When boot qemu, reports this
> 
> qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI
> BAR?
> qemu-system-aarch64: vfio_container_dma_map(0xaaaadd30f110,
> 0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)
> qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI
> BAR?
> qemu-system-aarch64: vfio_container_dma_map(0xaaaadd2bc310,
> 0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)
> qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI
> BAR?
> qemu-system-aarch64: vfio_container_dma_map(0xaaaadcf90000,
> 0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)
> 
> Will this also be solved in the new MSI mapping patchset?

Nope. These are not related to MSIs. These mappings are required for 
P2P DMA between devices and is not supported by IOMMUFD at the moment.

See the discussion here,
https://lore.kernel.org/all/20220426103507.5693a0ca.alex.williamson@redhat.com/

Thanks,
Shameer

RE: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Zhangfei Gao <zhangfei.gao@linaro.org>
> Sent: Monday, December 2, 2024 6:05 AM
> To: Jason Gunthorpe <jgg@nvidia.com>
> Cc: Nicolin Chen <nicolinc@nvidia.com>; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; Eric Auger
> <eric.auger@redhat.com>; Mostafa Saleh <smostafa@google.com>; qemu-
> arm@nongnu.org; qemu-devel@nongnu.org; Peter Maydell
> <peter.maydell@linaro.org>; Jean-Philippe Brucker <jean-
> philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> <mshavit@google.com>; Andrea Bolognani <abologna@redhat.com>;
> Michael S. Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>;
> nathanc@nvidia.com; arighi@nvidia.com; ianm@nvidia.com;
> jan@nvidia.com; mochs@nvidia.com
> Subject: Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
> 
> Hi, Nico
> 
> On Fri, 1 Nov 2024 at 19:55, Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > On Thu, Oct 31, 2024 at 09:09:20PM -0700, Nicolin Chen wrote:
> >
> > > FWIW, Robin requested a different solution for MSI mapping [1], v.s.
> > > the RMR one that we have been using since Eric's work. I drafted a
> > > few VFIO/IOMMUFD patches for that,
> >
> > I also talked to MarcZ about this at LPC and he seems willing to
> > consider it. It took a bit to explain everything though. So I think we
> > should try in Nov/Dec
> 
> When boot qemu, reports this
> 
> qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI
> BAR?
> qemu-system-aarch64: vfio_container_dma_map(0xaaaadd30f110,
> 0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)
> qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI
> BAR?
> qemu-system-aarch64: vfio_container_dma_map(0xaaaadd2bc310,
> 0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)
> qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI
> BAR?
> qemu-system-aarch64: vfio_container_dma_map(0xaaaadcf90000,
> 0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)
> 
> Will this also be solved in the new MSI mapping patchset?

Nope. These are not related to MSIs. These mappings are required for 
P2P DMA between devices and is not supported by IOMMUFD at the moment.

See the discussion here,
https://lore.kernel.org/all/20220426103507.5693a0ca.alex.williamson@redhat.com/

Thanks,
Shameer

Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Zhangfei Gao]

On Mon, 2 Dec 2024 at 16:07, Shameerali Kolothum Thodi
<shameerali.kolothum.thodi@huawei.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Zhangfei Gao <zhangfei.gao@linaro.org>
> > Sent: Monday, December 2, 2024 6:05 AM
> > To: Jason Gunthorpe <jgg@nvidia.com>
> > Cc: Nicolin Chen <nicolinc@nvidia.com>; Shameerali Kolothum Thodi
> > <shameerali.kolothum.thodi@huawei.com>; Eric Auger
> > <eric.auger@redhat.com>; Mostafa Saleh <smostafa@google.com>; qemu-
> > arm@nongnu.org; qemu-devel@nongnu.org; Peter Maydell
> > <peter.maydell@linaro.org>; Jean-Philippe Brucker <jean-
> > philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> > <mshavit@google.com>; Andrea Bolognani <abologna@redhat.com>;
> > Michael S. Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>;
> > nathanc@nvidia.com; arighi@nvidia.com; ianm@nvidia.com;
> > jan@nvidia.com; mochs@nvidia.com
> > Subject: Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
> >
> > Hi, Nico
> >
> > On Fri, 1 Nov 2024 at 19:55, Jason Gunthorpe <jgg@nvidia.com> wrote:
> > >
> > > On Thu, Oct 31, 2024 at 09:09:20PM -0700, Nicolin Chen wrote:
> > >
> > > > FWIW, Robin requested a different solution for MSI mapping [1], v.s.
> > > > the RMR one that we have been using since Eric's work. I drafted a
> > > > few VFIO/IOMMUFD patches for that,
> > >
> > > I also talked to MarcZ about this at LPC and he seems willing to
> > > consider it. It took a bit to explain everything though. So I think we
> > > should try in Nov/Dec
> >
> > When boot qemu, reports this
> >
> > qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI
> > BAR?
> > qemu-system-aarch64: vfio_container_dma_map(0xaaaadd30f110,
> > 0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)
> > qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI
> > BAR?
> > qemu-system-aarch64: vfio_container_dma_map(0xaaaadd2bc310,
> > 0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)
> > qemu-system-aarch64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI
> > BAR?
> > qemu-system-aarch64: vfio_container_dma_map(0xaaaadcf90000,
> > 0x8000200000, 0x10000, 0xffffb8031000) = -14 (Bad address)
> >
> > Will this also be solved in the new MSI mapping patchset?
>
> Nope. These are not related to MSIs. These mappings are required for
> P2P DMA between devices and is not supported by IOMMUFD at the moment.
>
> See the discussion here,
> https://lore.kernel.org/all/20220426103507.5693a0ca.alex.williamson@redhat.com/

Thanks Shameer

RE: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Shameerali Kolothum Thodi via]

Hi Nicolin,

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Friday, November 1, 2024 4:09 AM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Eric Auger <eric.auger@redhat.com>; Mostafa Saleh
> <smostafa@google.com>; qemu-arm@nongnu.org; qemu-
> devel@nongnu.org; Peter Maydell <peter.maydell@linaro.org>; Jason
> Gunthorpe <jgg@nvidia.com>; Jean-Philippe Brucker <jean-
> philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> <mshavit@google.com>; Andrea Bolognani <abologna@redhat.com>;
> Michael S. Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>;
> Zhangfei Gao <zhangfei.gao@linaro.org>; nathanc@nvidia.com;
> arighi@nvidia.com; ianm@nvidia.com; jan@nvidia.com; mochs@nvidia.com
> Subject: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
> 
> Hi,
> 
> This is a continued discussion following previous month's:
> https://lore.kernel.org/qemu-devel/Zvr%2Fbf7KgLN1cjOl@Asurada-Nvidia/
> 
> Kernel changes are getting closer to merge, as Jason's planning to
> take vIOMMU series and smmuv3_nesting series into the iommufd tree:
> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
> https://lore.kernel.org/all/0-v4-9e99b76f3518+3a8-
> smmuv3_nesting_jgg@nvidia.com/
> 
> So, I think it's probably a good time to align with each others and
> talk about kicking off some QEMU upstream work in the month ahead.
> 
> @Shameer,
> Do you have some update on the pluggable smmuv3 module?

I have a bare minimum prototype code that works with a pluggable smmuv3.

...
-device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
-device pcie-root-port,id=pcie.port1,bus=pcie.1 \
-device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
-device vfio-pci-nohotplug,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0 \
-device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
-device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=8 \
-device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
-device vfio-pci-nohotplug,host=0000:7d:02.1,bus=pcie.port2,iommufd=iommufd0 \
...

Something like above can now boot a Guest with the latest kernel. But I am not
sure it actually works correctly. I need a bit more time to update this and carry
out some tests. Will target that in Nov.

> 
> Updates on my side:
> 1) I have kept uAPI updated to the latest version and verified too.
>    There should be some polishing changes depending on how the basic
>    nesting infrastructure would look like from Intel/Duan's work.
> 2) I got some help from NVIDIA folks for the libvirt task. And they
>    have done some drafting and are now verifying the PCI topology
>    with "iommu=none".
> 
> Once the pluggable smmuv3 module is ready to test, we will make some
> change to libvirt for that and drop the auto-assigning patches from
> the VIRT code, so as to converge for a libvirt+QEMU test.

One query I have is, do Qemu still need to check whether the VFIO devices are
assigned correctly behind the nested vSMMUv3 w.r.t the phys SMMUv3s or not?
Or we can just trust whatever the user/libvirt specifies?

(I think even if we don't explicitly check, at present  it will eventually fail in s2
HWPT attach  for the viommu if it belongs to a different phys SMMUv3).

Thanks,
Shameer

RE: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Shameerali Kolothum Thodi via]

Hi Nicolin,

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Friday, November 1, 2024 4:09 AM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Eric Auger <eric.auger@redhat.com>; Mostafa Saleh
> <smostafa@google.com>; qemu-arm@nongnu.org; qemu-
> devel@nongnu.org; Peter Maydell <peter.maydell@linaro.org>; Jason
> Gunthorpe <jgg@nvidia.com>; Jean-Philippe Brucker <jean-
> philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> <mshavit@google.com>; Andrea Bolognani <abologna@redhat.com>;
> Michael S. Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>;
> Zhangfei Gao <zhangfei.gao@linaro.org>; nathanc@nvidia.com;
> arighi@nvidia.com; ianm@nvidia.com; jan@nvidia.com; mochs@nvidia.com
> Subject: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
> 
> Hi,
> 
> This is a continued discussion following previous month's:
> https://lore.kernel.org/qemu-devel/Zvr%2Fbf7KgLN1cjOl@Asurada-Nvidia/
> 
> Kernel changes are getting closer to merge, as Jason's planning to
> take vIOMMU series and smmuv3_nesting series into the iommufd tree:
> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
> https://lore.kernel.org/all/0-v4-9e99b76f3518+3a8-
> smmuv3_nesting_jgg@nvidia.com/
> 
> So, I think it's probably a good time to align with each others and
> talk about kicking off some QEMU upstream work in the month ahead.
> 
> @Shameer,
> Do you have some update on the pluggable smmuv3 module?

I have a bare minimum prototype code that works with a pluggable smmuv3.

...
-device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
-device pcie-root-port,id=pcie.port1,bus=pcie.1 \
-device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
-device vfio-pci-nohotplug,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0 \
-device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
-device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=8 \
-device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
-device vfio-pci-nohotplug,host=0000:7d:02.1,bus=pcie.port2,iommufd=iommufd0 \
...

Something like above can now boot a Guest with the latest kernel. But I am not
sure it actually works correctly. I need a bit more time to update this and carry
out some tests. Will target that in Nov.

> 
> Updates on my side:
> 1) I have kept uAPI updated to the latest version and verified too.
>    There should be some polishing changes depending on how the basic
>    nesting infrastructure would look like from Intel/Duan's work.
> 2) I got some help from NVIDIA folks for the libvirt task. And they
>    have done some drafting and are now verifying the PCI topology
>    with "iommu=none".
> 
> Once the pluggable smmuv3 module is ready to test, we will make some
> change to libvirt for that and drop the auto-assigning patches from
> the VIRT code, so as to converge for a libvirt+QEMU test.

One query I have is, do Qemu still need to check whether the VFIO devices are
assigned correctly behind the nested vSMMUv3 w.r.t the phys SMMUv3s or not?
Or we can just trust whatever the user/libvirt specifies?

(I think even if we don't explicitly check, at present  it will eventually fail in s2
HWPT attach  for the viommu if it belongs to a different phys SMMUv3).

Thanks,
Shameer

Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Nicolin Chen]

On Fri, Nov 01, 2024 at 06:35:23PM +0000, Shameerali Kolothum Thodi wrote:
> > @Shameer,
> > Do you have some update on the pluggable smmuv3 module?
> 
> I have a bare minimum prototype code that works with a pluggable smmuv3.
> 
> ...
> -device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
> -device pcie-root-port,id=pcie.port1,bus=pcie.1 \
> -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
> -device vfio-pci-nohotplug,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0 \
> -device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
> -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=8 \
> -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
> -device vfio-pci-nohotplug,host=0000:7d:02.1,bus=pcie.port2,iommufd=iommufd0 \
> ...
> 
> Something like above can now boot a Guest with the latest kernel. But I am not
> sure it actually works correctly. I need a bit more time to update this and carry
> out some tests. Will target that in Nov.

That looks nice to me! Thanks for the update.

> > Updates on my side:
> > 1) I have kept uAPI updated to the latest version and verified too.
> >    There should be some polishing changes depending on how the basic
> >    nesting infrastructure would look like from Intel/Duan's work.
> > 2) I got some help from NVIDIA folks for the libvirt task. And they
> >    have done some drafting and are now verifying the PCI topology
> >    with "iommu=none".
> >
> > Once the pluggable smmuv3 module is ready to test, we will make some
> > change to libvirt for that and drop the auto-assigning patches from
> > the VIRT code, so as to converge for a libvirt+QEMU test.
> 
> One query I have is, do Qemu still need to check whether the VFIO devices are
> assigned correctly behind the nested vSMMUv3 w.r.t the phys SMMUv3s or not?
> Or we can just trust whatever the user/libvirt specifies?

That's a good point. I assume ideally QEMU should do something,
though that would be somewhat duplicated with libvirt.

> (I think even if we don't explicitly check, at present  it will eventually fail in s2
> HWPT attach  for the viommu if it belongs to a different phys SMMUv3).

Yes. That's what we have. Perhaps simply print something like:
"is the device assigned to the correct arm-smmuv3-nested/pxb?"
?

Thanks
Nicolin

RE: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Friday, November 1, 2024 7:29 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Eric Auger <eric.auger@redhat.com>; Mostafa Saleh
> <smostafa@google.com>; qemu-arm@nongnu.org; qemu-
> devel@nongnu.org; Peter Maydell <peter.maydell@linaro.org>; Jason
> Gunthorpe <jgg@nvidia.com>; Jean-Philippe Brucker <jean-
> philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> <mshavit@google.com>; Andrea Bolognani <abologna@redhat.com>;
> Michael S. Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>;
> Zhangfei Gao <zhangfei.gao@linaro.org>; nathanc@nvidia.com;
> arighi@nvidia.com; ianm@nvidia.com; jan@nvidia.com; mochs@nvidia.com
> Subject: Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
> 
> On Fri, Nov 01, 2024 at 06:35:23PM +0000, Shameerali Kolothum Thodi
> wrote:
> > > @Shameer,
> > > Do you have some update on the pluggable smmuv3 module?
> >
> > I have a bare minimum prototype code that works with a pluggable
> smmuv3.
> >
> > ...
> > -device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
> > -device pcie-root-port,id=pcie.port1,bus=pcie.1 \
> > -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
> > -device vfio-pci-
> nohotplug,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0 \
> > -device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
> > -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=8 \
> > -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
> > -device vfio-pci-
> nohotplug,host=0000:7d:02.1,bus=pcie.port2,iommufd=iommufd0 \
> > ...
> >
> > Something like above can now boot a Guest with the latest kernel. But I
> am not
> > sure it actually works correctly. I need a bit more time to update this and
> carry
> > out some tests. Will target that in Nov.
> 
> That looks nice to me! Thanks for the update.

I have sent out an initial RFC here(Might have missed CC few in this email. Apologies.),
https://lore.kernel.org/qemu-devel/20241108125242.60136-1-shameerali.kolothum.thodi@huawei.com/

Please take a look and let me know.

Thanks,
Shameer

RE: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Friday, November 1, 2024 7:29 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Eric Auger <eric.auger@redhat.com>; Mostafa Saleh
> <smostafa@google.com>; qemu-arm@nongnu.org; qemu-
> devel@nongnu.org; Peter Maydell <peter.maydell@linaro.org>; Jason
> Gunthorpe <jgg@nvidia.com>; Jean-Philippe Brucker <jean-
> philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> <mshavit@google.com>; Andrea Bolognani <abologna@redhat.com>;
> Michael S. Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>;
> Zhangfei Gao <zhangfei.gao@linaro.org>; nathanc@nvidia.com;
> arighi@nvidia.com; ianm@nvidia.com; jan@nvidia.com; mochs@nvidia.com
> Subject: Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
> 
> On Fri, Nov 01, 2024 at 06:35:23PM +0000, Shameerali Kolothum Thodi
> wrote:
> > > @Shameer,
> > > Do you have some update on the pluggable smmuv3 module?
> >
> > I have a bare minimum prototype code that works with a pluggable
> smmuv3.
> >
> > ...
> > -device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
> > -device pcie-root-port,id=pcie.port1,bus=pcie.1 \
> > -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
> > -device vfio-pci-
> nohotplug,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0 \
> > -device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
> > -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=8 \
> > -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
> > -device vfio-pci-
> nohotplug,host=0000:7d:02.1,bus=pcie.port2,iommufd=iommufd0 \
> > ...
> >
> > Something like above can now boot a Guest with the latest kernel. But I
> am not
> > sure it actually works correctly. I need a bit more time to update this and
> carry
> > out some tests. Will target that in Nov.
> 
> That looks nice to me! Thanks for the update.

I have sent out an initial RFC here(Might have missed CC few in this email. Apologies.),
https://lore.kernel.org/qemu-devel/20241108125242.60136-1-shameerali.kolothum.thodi@huawei.com/

Please take a look and let me know.

Thanks,
Shameer

Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Andrea Bolognani]

On Fri, Nov 01, 2024 at 06:35:23PM +0000, Shameerali Kolothum Thodi wrote:
> I have a bare minimum prototype code that works with a pluggable smmuv3.
>
> ...
> -device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
> -device pcie-root-port,id=pcie.port1,bus=pcie.1 \
> -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
> -device vfio-pci-nohotplug,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0 \
> -device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
> -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=8 \
> -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
> -device vfio-pci-nohotplug,host=0000:7d:02.1,bus=pcie.port2,iommufd=iommufd0 \

Silly bit of feedback on the interface, but why is the
arm-smmuv3-nested property called "pci-bus" instead of just "bus"?

All other devices that need to refer to an existing PCI bus use the
latter. Is there a reason for this specific one to diverge?

-- 
Andrea Bolognani / Red Hat / Virtualization

RE: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Andrea Bolognani <abologna@redhat.com>
> Sent: Wednesday, November 20, 2024 9:14 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Nicolin Chen <nicolinc@nvidia.com>; Eric Auger
> <eric.auger@redhat.com>; Mostafa Saleh <smostafa@google.com>; qemu-
> arm@nongnu.org; qemu-devel@nongnu.org; Peter Maydell
> <peter.maydell@linaro.org>; Jason Gunthorpe <jgg@nvidia.com>; Jean-
> Philippe Brucker <jean-philippe@linaro.org>; Moritz Fischer
> <mdf@kernel.org>; Michael Shavit <mshavit@google.com>; Michael S.
> Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>; Zhangfei Gao
> <zhangfei.gao@linaro.org>; nathanc@nvidia.com; arighi@nvidia.com;
> ianm@nvidia.com; jan@nvidia.com; mochs@nvidia.com
> Subject: Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
> 
> On Fri, Nov 01, 2024 at 06:35:23PM +0000, Shameerali Kolothum Thodi
> wrote:
> > I have a bare minimum prototype code that works with a pluggable
> smmuv3.
> >
> > ...
> > -device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
> > -device pcie-root-port,id=pcie.port1,bus=pcie.1 \
> > -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
> > -device vfio-pci-
> nohotplug,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0 \
> > -device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
> > -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=8 \
> > -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
> > -device vfio-pci-
> nohotplug,host=0000:7d:02.1,bus=pcie.port2,iommufd=iommufd0 \
> 
> Silly bit of feedback on the interface, but why is the
> arm-smmuv3-nested property called "pci-bus" instead of just "bus"?
> 
> All other devices that need to refer to an existing PCI bus use the
> latter. Is there a reason for this specific one to diverge?

Nope. This will be changed to "bus" in next respin.

Thanks,
Shameer

RE: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Andrea Bolognani <abologna@redhat.com>
> Sent: Wednesday, November 20, 2024 9:14 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Nicolin Chen <nicolinc@nvidia.com>; Eric Auger
> <eric.auger@redhat.com>; Mostafa Saleh <smostafa@google.com>; qemu-
> arm@nongnu.org; qemu-devel@nongnu.org; Peter Maydell
> <peter.maydell@linaro.org>; Jason Gunthorpe <jgg@nvidia.com>; Jean-
> Philippe Brucker <jean-philippe@linaro.org>; Moritz Fischer
> <mdf@kernel.org>; Michael Shavit <mshavit@google.com>; Michael S.
> Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>; Zhangfei Gao
> <zhangfei.gao@linaro.org>; nathanc@nvidia.com; arighi@nvidia.com;
> ianm@nvidia.com; jan@nvidia.com; mochs@nvidia.com
> Subject: Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
> 
> On Fri, Nov 01, 2024 at 06:35:23PM +0000, Shameerali Kolothum Thodi
> wrote:
> > I have a bare minimum prototype code that works with a pluggable
> smmuv3.
> >
> > ...
> > -device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
> > -device pcie-root-port,id=pcie.port1,bus=pcie.1 \
> > -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
> > -device vfio-pci-
> nohotplug,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0 \
> > -device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
> > -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=8 \
> > -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
> > -device vfio-pci-
> nohotplug,host=0000:7d:02.1,bus=pcie.port2,iommufd=iommufd0 \
> 
> Silly bit of feedback on the interface, but why is the
> arm-smmuv3-nested property called "pci-bus" instead of just "bus"?
> 
> All other devices that need to refer to an existing PCI bus use the
> latter. Is there a reason for this specific one to diverge?

Nope. This will be changed to "bus" in next respin.

Thanks,
Shameer

Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Eric Auger]

Hi Nicolin,

On 11/1/24 05:09, Nicolin Chen wrote:
> Hi,
>
> This is a continued discussion following previous month's:
> https://lore.kernel.org/qemu-devel/Zvr%2Fbf7KgLN1cjOl@Asurada-Nvidia/
>
> Kernel changes are getting closer to merge, as Jason's planning to
> take vIOMMU series and smmuv3_nesting series into the iommufd tree:
> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
> https://lore.kernel.org/all/0-v4-9e99b76f3518+3a8-smmuv3_nesting_jgg@nvidia.com/
>
> So, I think it's probably a good time to align with each others and
> talk about kicking off some QEMU upstream work in the month ahead.
>
> @Shameer,
> Do you have some update on the pluggable smmuv3 module?
>
> Updates on my side:
> 1) I have kept uAPI updated to the latest version and verified too.
>    There should be some polishing changes depending on how the basic
>    nesting infrastructure would look like from Intel/Duan's work.
> 2) I got some help from NVIDIA folks for the libvirt task. And they
>    have done some drafting and are now verifying the PCI topology
>    with "iommu=none".
>
> Once the pluggable smmuv3 module is ready to test, we will make some
> change to libvirt for that and drop the auto-assigning patches from
> the VIRT code, so as to converge for a libvirt+QEMU test.
>
> FWIW, Robin requested a different solution for MSI mapping [1], v.s.
> the RMR one that we have been using since Eric's work. I drafted a
> few VFIO/IOMMUFD patches for that, yet paused for getting the vIOMMU
> series merged to this kernel cycle. I plan to continue in Nov/Dec.
> So, for the near term we will continue with the RMR solution, until
> we have something solid later.
>
> [1] https://lore.kernel.org/linux-iommu/ZrVN05VylFq8lK4q@Asurada-Nvidia/

At Red Hat we may find some cycles to resume working on the QEMU
integration. Please can you sketch some tasks we could carry out in
coordination with you and Shameer? Adding Don in the loop.

Thank you in advance

Eric
>
> Thanks
> Nicolin
>

Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Nicolin Chen]

Hi Eric,

On Thu, Nov 07, 2024 at 12:11:05PM +0100, Eric Auger wrote:
> On 11/1/24 05:09, Nicolin Chen wrote:
> > Hi,
> >
> > This is a continued discussion following previous month's:
> > https://lore.kernel.org/qemu-devel/Zvr%2Fbf7KgLN1cjOl@Asurada-Nvidia/
> >
> > Kernel changes are getting closer to merge, as Jason's planning to
> > take vIOMMU series and smmuv3_nesting series into the iommufd tree:
> > https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
> > https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
> > https://lore.kernel.org/all/0-v4-9e99b76f3518+3a8-smmuv3_nesting_jgg@nvidia.com/
> >
> > So, I think it's probably a good time to align with each others and
> > talk about kicking off some QEMU upstream work in the month ahead.
> >
> > @Shameer,
> > Do you have some update on the pluggable smmuv3 module?
> >
> > Updates on my side:
> > 1) I have kept uAPI updated to the latest version and verified too.
> >    There should be some polishing changes depending on how the basic
> >    nesting infrastructure would look like from Intel/Duan's work.
> > 2) I got some help from NVIDIA folks for the libvirt task. And they
> >    have done some drafting and are now verifying the PCI topology
> >    with "iommu=none".
> >
> > Once the pluggable smmuv3 module is ready to test, we will make some
> > change to libvirt for that and drop the auto-assigning patches from
> > the VIRT code, so as to converge for a libvirt+QEMU test.
> >
> > FWIW, Robin requested a different solution for MSI mapping [1], v.s.
> > the RMR one that we have been using since Eric's work. I drafted a
> > few VFIO/IOMMUFD patches for that, yet paused for getting the vIOMMU
> > series merged to this kernel cycle. I plan to continue in Nov/Dec.
> > So, for the near term we will continue with the RMR solution, until
> > we have something solid later.
> >
> > [1] https://lore.kernel.org/linux-iommu/ZrVN05VylFq8lK4q@Asurada-Nvidia/
> 
> At Red Hat we may find some cycles to resume working on the QEMU
> integration. Please can you sketch some tasks we could carry out in
> coordination with you and Shameer? Adding Don in the loop.

That is great!

So, given that Shameer is working on pluggable module part and we
have folks working on libvirt. I think the only big thing here is
the SMMUv3 series itself. Please refer to the changes in the link:
 - cover-letter: Add HW accelerated nesting support for arm SMMUv3
 - https://github.com/nicolinc/qemu/commits/wip/for_smmuv3_nesting-v4/

I expect the IOMMU_HWPT_ALLOC (backend APIs) will go with Intel's
series once their current "emulated devices" one gets merged. And
I think I can prepare IOMMU_VIOMMU_ALLOC patches for backend APIs
aligning with HWPT's.

That being said, one thing I am not sure is how we should bridge
between these backend APIs and a virtual IOMMUs (vSMMU/intel). I
think it'd be better for you and Red Hat to provide some insight,
if calling the backend APIs directly from a viommu module isn't a
preferable way.

We also need your comments on vSMMU module patches that are still
roughly a draft requiring a rework at some small details I think.
So, if your (and Don's) bandwidth allows, perhaps you could take
over the vSMMU patches? Otherwise, we can also share the task for
reworking.

Thanks!
Nicolin

Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Donald Dutile]

On 11/7/24 3:31 PM, Nicolin Chen wrote:
> Hi Eric,
> 
> On Thu, Nov 07, 2024 at 12:11:05PM +0100, Eric Auger wrote:
>> On 11/1/24 05:09, Nicolin Chen wrote:
>>> Hi,
>>>
>>> This is a continued discussion following previous month's:
>>> https://lore.kernel.org/qemu-devel/Zvr%2Fbf7KgLN1cjOl@Asurada-Nvidia/
>>>
>>> Kernel changes are getting closer to merge, as Jason's planning to
>>> take vIOMMU series and smmuv3_nesting series into the iommufd tree:
>>> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
>>> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
>>> https://lore.kernel.org/all/0-v4-9e99b76f3518+3a8-smmuv3_nesting_jgg@nvidia.com/
>>>
>>> So, I think it's probably a good time to align with each others and
>>> talk about kicking off some QEMU upstream work in the month ahead.
>>>
>>> @Shameer,
>>> Do you have some update on the pluggable smmuv3 module?
>>>
>>> Updates on my side:
>>> 1) I have kept uAPI updated to the latest version and verified too.
>>>     There should be some polishing changes depending on how the basic
>>>     nesting infrastructure would look like from Intel/Duan's work.
>>> 2) I got some help from NVIDIA folks for the libvirt task. And they
>>>     have done some drafting and are now verifying the PCI topology
>>>     with "iommu=none".
>>>
>>> Once the pluggable smmuv3 module is ready to test, we will make some
>>> change to libvirt for that and drop the auto-assigning patches from
>>> the VIRT code, so as to converge for a libvirt+QEMU test.
>>>
>>> FWIW, Robin requested a different solution for MSI mapping [1], v.s.
>>> the RMR one that we have been using since Eric's work. I drafted a
>>> few VFIO/IOMMUFD patches for that, yet paused for getting the vIOMMU
>>> series merged to this kernel cycle. I plan to continue in Nov/Dec.
>>> So, for the near term we will continue with the RMR solution, until
>>> we have something solid later.
>>>
>>> [1] https://lore.kernel.org/linux-iommu/ZrVN05VylFq8lK4q@Asurada-Nvidia/
>>
>> At Red Hat we may find some cycles to resume working on the QEMU
>> integration. Please can you sketch some tasks we could carry out in
>> coordination with you and Shameer? Adding Don in the loop.
> 
> That is great!
> 
> So, given that Shameer is working on pluggable module part and we
> have folks working on libvirt. I think the only big thing here is
> the SMMUv3 series itself. Please refer to the changes in the link:
>   - cover-letter: Add HW accelerated nesting support for arm SMMUv3
>   - https://github.com/nicolinc/qemu/commits/wip/for_smmuv3_nesting-v4/
> 
> I expect the IOMMU_HWPT_ALLOC (backend APIs) will go with Intel's
> series once their current "emulated devices" one gets merged. And
> I think I can prepare IOMMU_VIOMMU_ALLOC patches for backend APIs
> aligning with HWPT's.
> 
> That being said, one thing I am not sure is how we should bridge
> between these backend APIs and a virtual IOMMUs (vSMMU/intel). I
> think it'd be better for you and Red Hat to provide some insight,
> if calling the backend APIs directly from a viommu module isn't a
> preferable way.
> 
> We also need your comments on vSMMU module patches that are still
> roughly a draft requiring a rework at some small details I think.
> So, if your (and Don's) bandwidth allows, perhaps you could take
> over the vSMMU patches? Otherwise, we can also share the task for
> reworking.
> 
> Thanks!
> Nicolin
> 
Nicolin,
Hi!
Sounds like a plan.  Glad to join the effort.
- Don

Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Eric Auger]

Hi Nicolin,

On 11/7/24 21:31, Nicolin Chen wrote:
> Hi Eric,
>
> On Thu, Nov 07, 2024 at 12:11:05PM +0100, Eric Auger wrote:
>> On 11/1/24 05:09, Nicolin Chen wrote:
>>> Hi,
>>>
>>> This is a continued discussion following previous month's:
>>> https://lore.kernel.org/qemu-devel/Zvr%2Fbf7KgLN1cjOl@Asurada-Nvidia/
>>>
>>> Kernel changes are getting closer to merge, as Jason's planning to
>>> take vIOMMU series and smmuv3_nesting series into the iommufd tree:
>>> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
>>> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
>>> https://lore.kernel.org/all/0-v4-9e99b76f3518+3a8-smmuv3_nesting_jgg@nvidia.com/
>>>
>>> So, I think it's probably a good time to align with each others and
>>> talk about kicking off some QEMU upstream work in the month ahead.
>>>
>>> @Shameer,
>>> Do you have some update on the pluggable smmuv3 module?
>>>
>>> Updates on my side:
>>> 1) I have kept uAPI updated to the latest version and verified too.
>>>    There should be some polishing changes depending on how the basic
>>>    nesting infrastructure would look like from Intel/Duan's work.
>>> 2) I got some help from NVIDIA folks for the libvirt task. And they
>>>    have done some drafting and are now verifying the PCI topology
>>>    with "iommu=none".
>>>
>>> Once the pluggable smmuv3 module is ready to test, we will make some
>>> change to libvirt for that and drop the auto-assigning patches from
>>> the VIRT code, so as to converge for a libvirt+QEMU test.
>>>
>>> FWIW, Robin requested a different solution for MSI mapping [1], v.s.
>>> the RMR one that we have been using since Eric's work. I drafted a
>>> few VFIO/IOMMUFD patches for that, yet paused for getting the vIOMMU
>>> series merged to this kernel cycle. I plan to continue in Nov/Dec.
>>> So, for the near term we will continue with the RMR solution, until
>>> we have something solid later.
>>>
>>> [1] https://lore.kernel.org/linux-iommu/ZrVN05VylFq8lK4q@Asurada-Nvidia/
>> At Red Hat we may find some cycles to resume working on the QEMU
>> integration. Please can you sketch some tasks we could carry out in
>> coordination with you and Shameer? Adding Don in the loop.
> That is great!
>
> So, given that Shameer is working on pluggable module part and we
> have folks working on libvirt. I think the only big thing here is
> the SMMUv3 series itself. Please refer to the changes in the link:
>  - cover-letter: Add HW accelerated nesting support for arm SMMUv3
>  - https://github.com/nicolinc/qemu/commits/wip/for_smmuv3_nesting-v4/
Looking at your branch I see the following series (marked with cover-letter)
*

  *

    cover-letter: Add RMR WAR for MSI mappings (based on former RMR flat
    mapping and not related to *[PATCH RFCv1 0/7] vfio: Allow userspace
    to specify the address for each MSI vector
    <https://lore.kernel.org/kvm/cover.1731130093.git.nicolinc@nvidia.com/#r>
    I guess)*

  *

    cover-letter: hw/arm/virt: Add multiple nested SMMUs (Nicolin ->
    Shameer)

  *

    cover-letter: Add HW accelerated nesting support for arm SMMUv3
    (Nicolin)

  *

    cover-letter: Add VIOMMU infrastructure support (Nicolin)

  *

    cover-letter: intel_iommu: Enable stage-1 translation for
    passthrough device (Zhenzhong)

  *

    cover-letter: intel_iommu: Enable stage-1 translation for emulated
    device (Zhenzhong)

The last one is covered by *[PATCH v5 00/20] intel_iommu: Enable stage-1
translation for emulated device
<https://lore.kernel.org/all/20241111083457.2090664-1-zhenzhong.duan@intel.com/#r>
*

*I see there is a reference to *"Enable stage-1 translation for
passthrough device" series but has it been posted for review? Adding
Zhenzhong in copy.

**

*Are there any posts upstream for the rest, besides Shameer's respin?
*

*
>
> I expect the IOMMU_HWPT_ALLOC (backend APIs) will go with Intel's
> series once their current "emulated devices" one gets merged. And
> I think I can prepare IOMMU_VIOMMU_ALLOC patches for backend APIs
> aligning with HWPT's.
Can you point us to the actual series including this IOMMU_HWPT_ALLOC
support? This would clarify which part you are going to work on next.

>
> That being said, one thing I am not sure is how we should bridge
> between these backend APIs and a virtual IOMMUs (vSMMU/intel). I
> think it'd be better for you and Red Hat to provide some insight,
> if calling the backend APIs directly from a viommu module isn't a
> preferable way.
can you clarify what you call backend API in that context?
>
> We also need your comments on vSMMU module patches that are still
> roughly a draft requiring a rework at some small details I think.
> So, if your (and Don's) bandwidth allows, perhaps you could take
> over the vSMMU patches? Otherwise, we can also share the task for
> reworking.
So we have started the multi smmu instantiation review and provided
feedbacks.
Which part can we work on then?

Thanks

Eric
>
> Thanks!
> Nicolin
>

RE: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Duan, Zhenzhong]

Hi Eric,

>-----Original Message-----
>From: Eric Auger <eric.auger@redhat.com>
>Sent: Tuesday, November 19, 2024 2:00 AM
>Subject: Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
>
>Hi Nicolin,
>
>On 11/7/24 21:31, Nicolin Chen wrote:
>> Hi Eric,
>>
>> On Thu, Nov 07, 2024 at 12:11:05PM +0100, Eric Auger wrote:
>>> On 11/1/24 05:09, Nicolin Chen wrote:
>>>> Hi,
>>>>
>>>> This is a continued discussion following previous month's:
>>>> https://lore.kernel.org/qemu-devel/Zvr%2Fbf7KgLN1cjOl@Asurada-Nvidia/
>>>>
>>>> Kernel changes are getting closer to merge, as Jason's planning to
>>>> take vIOMMU series and smmuv3_nesting series into the iommufd tree:
>>>> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
>>>> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
>>>> https://lore.kernel.org/all/0-v4-9e99b76f3518+3a8-
>smmuv3_nesting_jgg@nvidia.com/
>>>>
>>>> So, I think it's probably a good time to align with each others and
>>>> talk about kicking off some QEMU upstream work in the month ahead.
>>>>
>>>> @Shameer,
>>>> Do you have some update on the pluggable smmuv3 module?
>>>>
>>>> Updates on my side:
>>>> 1) I have kept uAPI updated to the latest version and verified too.
>>>>    There should be some polishing changes depending on how the basic
>>>>    nesting infrastructure would look like from Intel/Duan's work.
>>>> 2) I got some help from NVIDIA folks for the libvirt task. And they
>>>>    have done some drafting and are now verifying the PCI topology
>>>>    with "iommu=none".
>>>>
>>>> Once the pluggable smmuv3 module is ready to test, we will make some
>>>> change to libvirt for that and drop the auto-assigning patches from
>>>> the VIRT code, so as to converge for a libvirt+QEMU test.
>>>>
>>>> FWIW, Robin requested a different solution for MSI mapping [1], v.s.
>>>> the RMR one that we have been using since Eric's work. I drafted a
>>>> few VFIO/IOMMUFD patches for that, yet paused for getting the vIOMMU
>>>> series merged to this kernel cycle. I plan to continue in Nov/Dec.
>>>> So, for the near term we will continue with the RMR solution, until
>>>> we have something solid later.
>>>>
>>>> [1] https://lore.kernel.org/linux-iommu/ZrVN05VylFq8lK4q@Asurada-Nvidia/
>>> At Red Hat we may find some cycles to resume working on the QEMU
>>> integration. Please can you sketch some tasks we could carry out in
>>> coordination with you and Shameer? Adding Don in the loop.
>> That is great!
>>
>> So, given that Shameer is working on pluggable module part and we
>> have folks working on libvirt. I think the only big thing here is
>> the SMMUv3 series itself. Please refer to the changes in the link:
>>  - cover-letter: Add HW accelerated nesting support for arm SMMUv3
>>  - https://github.com/nicolinc/qemu/commits/wip/for_smmuv3_nesting-v4/
>Looking at your branch I see the following series (marked with cover-letter)
>*
>
>  *
>
>    cover-letter: Add RMR WAR for MSI mappings (based on former RMR flat
>    mapping and not related to *[PATCH RFCv1 0/7] vfio: Allow userspace
>    to specify the address for each MSI vector
>    <https://lore.kernel.org/kvm/cover.1731130093.git.nicolinc@nvidia.com/#r>
>    I guess)*
>
>  *
>
>    cover-letter: hw/arm/virt: Add multiple nested SMMUs (Nicolin ->
>    Shameer)
>
>  *
>
>    cover-letter: Add HW accelerated nesting support for arm SMMUv3
>    (Nicolin)
>
>  *
>
>    cover-letter: Add VIOMMU infrastructure support (Nicolin)
>
>  *
>
>    cover-letter: intel_iommu: Enable stage-1 translation for
>    passthrough device (Zhenzhong)
>
>  *
>
>    cover-letter: intel_iommu: Enable stage-1 translation for emulated
>    device (Zhenzhong)
>
>The last one is covered by *[PATCH v5 00/20] intel_iommu: Enable stage-1
>translation for emulated device
><https://lore.kernel.org/all/20241111083457.2090664-1-
>zhenzhong.duan@intel.com/#r>
>*
>
>*I see there is a reference to *"Enable stage-1 translation for
>passthrough device" series but has it been posted for review? Adding
>Zhenzhong in copy.

There is an RFCv1 posted upstream which is a combination of
" intel_iommu: Enable stage-1 translation for emulated device" and
" intel_iommu: Enable stage-1 translation for passthrough device".
Link: https://lists.gnu.org/archive/html/qemu-devel/2024-01/msg02740.html

If you means the split version, not sent yet. Plan it after the series about
emulated device is accepted.

Thanks
Zhenzhong

Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Eric Auger]

Hi Zhenzhong,

On 11/19/24 08:07, Duan, Zhenzhong wrote:
> Hi Eric,
>
>> -----Original Message-----
>> From: Eric Auger <eric.auger@redhat.com>
>> Sent: Tuesday, November 19, 2024 2:00 AM
>> Subject: Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
>>
>> Hi Nicolin,
>>
>> On 11/7/24 21:31, Nicolin Chen wrote:
>>> Hi Eric,
>>>
>>> On Thu, Nov 07, 2024 at 12:11:05PM +0100, Eric Auger wrote:
>>>> On 11/1/24 05:09, Nicolin Chen wrote:
>>>>> Hi,
>>>>>
>>>>> This is a continued discussion following previous month's:
>>>>> https://lore.kernel.org/qemu-devel/Zvr%2Fbf7KgLN1cjOl@Asurada-Nvidia/
>>>>>
>>>>> Kernel changes are getting closer to merge, as Jason's planning to
>>>>> take vIOMMU series and smmuv3_nesting series into the iommufd tree:
>>>>> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
>>>>> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
>>>>> https://lore.kernel.org/all/0-v4-9e99b76f3518+3a8-
>> smmuv3_nesting_jgg@nvidia.com/
>>>>> So, I think it's probably a good time to align with each others and
>>>>> talk about kicking off some QEMU upstream work in the month ahead.
>>>>>
>>>>> @Shameer,
>>>>> Do you have some update on the pluggable smmuv3 module?
>>>>>
>>>>> Updates on my side:
>>>>> 1) I have kept uAPI updated to the latest version and verified too.
>>>>>    There should be some polishing changes depending on how the basic
>>>>>    nesting infrastructure would look like from Intel/Duan's work.
>>>>> 2) I got some help from NVIDIA folks for the libvirt task. And they
>>>>>    have done some drafting and are now verifying the PCI topology
>>>>>    with "iommu=none".
>>>>>
>>>>> Once the pluggable smmuv3 module is ready to test, we will make some
>>>>> change to libvirt for that and drop the auto-assigning patches from
>>>>> the VIRT code, so as to converge for a libvirt+QEMU test.
>>>>>
>>>>> FWIW, Robin requested a different solution for MSI mapping [1], v.s.
>>>>> the RMR one that we have been using since Eric's work. I drafted a
>>>>> few VFIO/IOMMUFD patches for that, yet paused for getting the vIOMMU
>>>>> series merged to this kernel cycle. I plan to continue in Nov/Dec.
>>>>> So, for the near term we will continue with the RMR solution, until
>>>>> we have something solid later.
>>>>>
>>>>> [1] https://lore.kernel.org/linux-iommu/ZrVN05VylFq8lK4q@Asurada-Nvidia/
>>>> At Red Hat we may find some cycles to resume working on the QEMU
>>>> integration. Please can you sketch some tasks we could carry out in
>>>> coordination with you and Shameer? Adding Don in the loop.
>>> That is great!
>>>
>>> So, given that Shameer is working on pluggable module part and we
>>> have folks working on libvirt. I think the only big thing here is
>>> the SMMUv3 series itself. Please refer to the changes in the link:
>>>  - cover-letter: Add HW accelerated nesting support for arm SMMUv3
>>>  - https://github.com/nicolinc/qemu/commits/wip/for_smmuv3_nesting-v4/
>> Looking at your branch I see the following series (marked with cover-letter)
>> *
>>
>>  *
>>
>>    cover-letter: Add RMR WAR for MSI mappings (based on former RMR flat
>>    mapping and not related to *[PATCH RFCv1 0/7] vfio: Allow userspace
>>    to specify the address for each MSI vector
>>    <https://lore.kernel.org/kvm/cover.1731130093.git.nicolinc@nvidia.com/#r>
>>    I guess)*
>>
>>  *
>>
>>    cover-letter: hw/arm/virt: Add multiple nested SMMUs (Nicolin ->
>>    Shameer)
>>
>>  *
>>
>>    cover-letter: Add HW accelerated nesting support for arm SMMUv3
>>    (Nicolin)
>>
>>  *
>>
>>    cover-letter: Add VIOMMU infrastructure support (Nicolin)
>>
>>  *
>>
>>    cover-letter: intel_iommu: Enable stage-1 translation for
>>    passthrough device (Zhenzhong)
>>
>>  *
>>
>>    cover-letter: intel_iommu: Enable stage-1 translation for emulated
>>    device (Zhenzhong)
>>
>> The last one is covered by *[PATCH v5 00/20] intel_iommu: Enable stage-1
>> translation for emulated device
>> <https://lore.kernel.org/all/20241111083457.2090664-1-
>> zhenzhong.duan@intel.com/#r>
>> *
>>
>> *I see there is a reference to *"Enable stage-1 translation for
>> passthrough device" series but has it been posted for review? Adding
>> Zhenzhong in copy.
> There is an RFCv1 posted upstream which is a combination of
> " intel_iommu: Enable stage-1 translation for emulated device" and
> " intel_iommu: Enable stage-1 translation for passthrough device".
> Link: https://lists.gnu.org/archive/html/qemu-devel/2024-01/msg02740.html
>
> If you means the split version, not sent yet. Plan it after the series about
> emulated device is accepted.
OK thank you very much for the link and confirmation.

Eric
>
> Thanks
> Zhenzhong
>

Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Nicolin Chen]

On Mon, Nov 18, 2024 at 06:59:53PM +0100, Eric Auger wrote:
> Looking at your branch I see the following series (marked with cover-letter)
..
>     cover-letter: Add RMR WAR for MSI mappings (based on former RMR flat
>     mapping and not related to *[PATCH RFCv1 0/7] vfio: Allow userspace
>     to specify the address for each MSI vector
>     <https://lore.kernel.org/kvm/cover.1731130093.git.nicolinc@nvidia.com/#r>
>     I guess)*

Yes, I think we would postpone this series, by simply putting it
on top of our future shared branches while waiting for the kernel
solution gets finalized to a certain degree, to make sure an MSI
2-stage mapping could still work.

>     cover-letter: hw/arm/virt: Add multiple nested SMMUs (Nicolin ->
>     Shameer)

Yes

>     cover-letter: Add HW accelerated nesting support for arm SMMUv3
>     (Nicolin)

I think this will be moved to Don (or Don/Nic)?

>     cover-letter: Add VIOMMU infrastructure support (Nicolin)

Yes.


>     cover-letter: intel_iommu: Enable stage-1 translation for
>     passthrough device (Zhenzhong)
..
>     cover-letter: intel_iommu: Enable stage-1 translation for emulated
>     device (Zhenzhong)

Yes. We'll need the HWPT infrastructure patches in Intel's series
and I think Intel's progress is faster so Zhenzhong should be the
submitter.

> *Are there any posts upstream for the rest, besides Shameer's respin?

Not yet. I think the dependency should be
(1) HWPT infrastructure (Zhenzhong's Intel series)
(2) VIOMMU infrastructure
(3) smmuv3-acc series (2-stage translation)
(4) Multi vSMMU instance support (VIRT and IORT)
(5) RMR

> *
> >
> > I expect the IOMMU_HWPT_ALLOC (backend APIs) will go with Intel's
> > series once their current "emulated devices" one gets merged. And
> > I think I can prepare IOMMU_VIOMMU_ALLOC patches for backend APIs
> > aligning with HWPT's.
> Can you point us to the actual series including this IOMMU_HWPT_ALLOC
> support? This would clarify which part you are going to work on next.

https://github.com/yiliu1765/qemu/commits/zhenzhong/iommufd_nesting_rfcv2/
Though not planning to do so since it's unlikely the case, yet HWPT
changes could go with the vSMMU series if we're running faster than
Zhenzhong :)

> >
> > That being said, one thing I am not sure is how we should bridge
> > between these backend APIs and a virtual IOMMUs (vSMMU/intel). I
> > think it'd be better for you and Red Hat to provide some insight,
> > if calling the backend APIs directly from a viommu module isn't a
> > preferable way.
> can you clarify what you call backend API in that context?
> >
> > We also need your comments on vSMMU module patches that are still
> > roughly a draft requiring a rework at some small details I think.
> > So, if your (and Don's) bandwidth allows, perhaps you could take
> > over the vSMMU patches? Otherwise, we can also share the task for
> > reworking.
> So we have started the multi smmu instantiation review and provided
> feedbacks.
> Which part can we work on then?

This one:
     cover-letter: Add HW accelerated nesting support for arm SMMUv3
i.e. (3) that I put the list above.

Thanks
Nicolin