nested-smmuv3 topic, Sep 2024

nested-smmuv3 topic, Sep 2024

[Nicolin Chen]

Hi all,

Hope I didn't miss anybody who is related to the topic. Please,
feel free to add!

<--- Background --->
As some of you know, there is an ongoing effort for nested-smmuv3
support in QEMU on ARM, working with the kernel IOMMUFD uAPIs:
[Nesting for vSTE]
https://lore.kernel.org/linux-iommu/0-v2-621370057090+91fec-smmuv3_nesting_jgg@nvidia.com/
[Nesting for invalidations]
https://lore.kernel.org/linux-iommu/cover.1724776335.git.nicolinc@nvidia.com/

The kernel patches are still under review. Jason and I are hoping
them to get merged at next cycle for v6.13, which means the QEMU
patches might start a review process as early as Nov/Dec?

That being said, I think we are way behind the point that patches
can get reviewed: most of the QEMU patches on my branches weren't
touched very often, but merely updated to the latest kernel uAPIs
for verification. So, I feel this might be a good point to gather
folks together to discuss about the possible timeline and ask for
help. I think this would potentially help folks who are going to
attend the KVM forum (or LPC) to carry out a discussion. (Sorry,
I won't make it due to some conflict..)

<-- Task Breakdown --->
I previously sent a RFCv1 series collecting comments/suggestions,
for multi-vSMMU instance design in ARM Virt code:
https://lore.kernel.org/qemu-devel/cover.1719361174.git.nicolinc@nvidia.com/
(And thanks again for all the inputs!)

The main takeaway from the discussion is to
1) Turn the vSMMU module into a pluggable one, like intel-iommu
2) Move the per-SMMU pxb bus and device auto-assign into libvirt

Apart from the multi-vSMMU thing, there's basic nesting series:
0) Keep updating to the latest kernel uAPIs to support nesting

I was trying to do all these three, but apparently too ambitious.
The kernel side of work is still taking a lot of my bandwidth. So
far I had almost-zero progress on task (1) and completely-zero on
task (2).

<-- Help Needed --->
So, I'm wondering if anyone(s) might have some extra bandwidth in
the following months helping these two tasks, either of which can
be a standalone project I think.

For task (0), I think I can keep updating the uAPI part, although
it'd need some help for reviews, which I was hoping to occur after
Intel sends the QEMU nesting backend patches. Once we know how big
the rework is going to be, we may need to borrow some help at that
point once again..

Thank you
Nicolin

RE: nested-smmuv3 topic, Sep 2024

[Shameerali Kolothum Thodi via]

Hi Nicolin,

Thanks for the write up and task progress status.

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Thursday, September 5, 2024 9:26 AM
> To: Eric Auger <eric.auger@redhat.com>; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; Mostafa Saleh
> <smostafa@google.com>
> Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org; Peter Maydell
> <peter.maydell@linaro.org>; Jason Gunthorpe <jgg@nvidia.com>; Jean-
> Philippe Brucker <jean-philippe@linaro.org>; Moritz Fischer
> <mdf@kernel.org>; Michael Shavit <mshavit@google.com>; Andrea
> Bolognani <abologna@redhat.com>; Michael S. Tsirkin <mst@redhat.com>;
> Peter Xu <peterx@redhat.com>
> Subject: nested-smmuv3 topic, Sep 2024
> 
> Hi all,
> 
> Hope I didn't miss anybody who is related to the topic. Please,
> feel free to add!
> 
> <--- Background --->
> As some of you know, there is an ongoing effort for nested-smmuv3
> support in QEMU on ARM, working with the kernel IOMMUFD uAPIs:
> [Nesting for vSTE]
> https://lore.kernel.org/linux-iommu/0-v2-621370057090+91fec-
> smmuv3_nesting_jgg@nvidia.com/
> [Nesting for invalidations]
> https://lore.kernel.org/linux-
> iommu/cover.1724776335.git.nicolinc@nvidia.com/
> 
> The kernel patches are still under review. Jason and I are hoping
> them to get merged at next cycle for v6.13, which means the QEMU
> patches might start a review process as early as Nov/Dec?
> 
> That being said, I think we are way behind the point that patches
> can get reviewed: most of the QEMU patches on my branches weren't
> touched very often, but merely updated to the latest kernel uAPIs
> for verification. So, I feel this might be a good point to gather
> folks together to discuss about the possible timeline and ask for
> help. I think this would potentially help folks who are going to
> attend the KVM forum (or LPC) to carry out a discussion. (Sorry,
> I won't make it due to some conflict..)
> 
> <-- Task Breakdown --->
> I previously sent a RFCv1 series collecting comments/suggestions,
> for multi-vSMMU instance design in ARM Virt code:
> https://lore.kernel.org/qemu-
> devel/cover.1719361174.git.nicolinc@nvidia.com/
> (And thanks again for all the inputs!)
> 
> The main takeaway from the discussion is to
> 1) Turn the vSMMU module into a pluggable one, like intel-iommu
> 2) Move the per-SMMU pxb bus and device auto-assign into libvirt
> 
> Apart from the multi-vSMMU thing, there's basic nesting series:
> 0) Keep updating to the latest kernel uAPIs to support nesting

By this you mean the old HWPT based nested-smmuv3 support?

> 
> I was trying to do all these three, but apparently too ambitious.
> The kernel side of work is still taking a lot of my bandwidth. So
> far I had almost-zero progress on task (1) and completely-zero on
> task (2).
> 
> <-- Help Needed --->
> So, I'm wondering if anyone(s) might have some extra bandwidth in
> the following months helping these two tasks, either of which can
> be a standalone project I think.
> 
> For task (0), I think I can keep updating the uAPI part, although
> it'd need some help for reviews, which I was hoping to occur after
> Intel sends the QEMU nesting backend patches. Once we know how big
> the rework is going to be, we may need to borrow some help at that
> point once again..

I might have some bandwidth starting October and can take a look at 
task 1 above. I haven't gone through the VIOMMU API model completely
yet and plan to do that soon.

Also I am planning to attend KVM forum, so if there are anyone interested
to have  a chat on this, please let me know.

Thanks,
Shameer

Re: nested-smmuv3 topic, Sep 2024

[Nicolin Chen]

Hi Shameer,

Thanks for the reply!

On Thu, Sep 05, 2024 at 12:55:52PM +0000, Shameerali Kolothum Thodi wrote:
> > The main takeaway from the discussion is to
> > 1) Turn the vSMMU module into a pluggable one, like intel-iommu
> > 2) Move the per-SMMU pxb bus and device auto-assign into libvirt
> >
> > Apart from the multi-vSMMU thing, there's basic nesting series:
> > 0) Keep updating to the latest kernel uAPIs to support nesting
> 
> By this you mean the old HWPT based nested-smmuv3 support?

HWPT + vIOMMU. The for-viommu/virq branches that I shared in my
kernel series have those changes. Invalidations is done via the
vIOMMU infrastructure.

> >
> > I was trying to do all these three, but apparently too ambitious.
> > The kernel side of work is still taking a lot of my bandwidth. So
> > far I had almost-zero progress on task (1) and completely-zero on
> > task (2).
> >
> > <-- Help Needed --->
> > So, I'm wondering if anyone(s) might have some extra bandwidth in
> > the following months helping these two tasks, either of which can
> > be a standalone project I think.
> >
> > For task (0), I think I can keep updating the uAPI part, although
> > it'd need some help for reviews, which I was hoping to occur after
> > Intel sends the QEMU nesting backend patches. Once we know how big
> > the rework is going to be, we may need to borrow some help at that
> > point once again..
> 
> I might have some bandwidth starting October and can take a look at
> task 1 above. I haven't gone through the VIOMMU API model completely
> yet and plan to do that soon.

Thank you! That'd be helpful! The major effort I think is in the
VIRT code also, where "-device nested-smmuv3" must tell the info
to build Device Tree or IORT.

The vIOMMU uAPI is not that complicated. That being said, I am
trying to add some kernel documentation for nested translation,
so hopefully this would be helpful in the near future.

> Also I am planning to attend KVM forum, so if there are anyone interested
> to have  a chat on this, please let me know.

Wish I could make it to that. I think we will need another Oct
thread to get all of us aligned once again.

Regards
Nicolin

RE: nested-smmuv3 topic, Sep 2024

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Thursday, September 5, 2024 9:37 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Eric Auger <eric.auger@redhat.com>; Mostafa Saleh
> <smostafa@google.com>; qemu-arm@nongnu.org; qemu-
> devel@nongnu.org; Peter Maydell <peter.maydell@linaro.org>; Jason
> Gunthorpe <jgg@nvidia.com>; Jean-Philippe Brucker <jean-
> philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> <mshavit@google.com>; Andrea Bolognani <abologna@redhat.com>;
> Michael S. Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>
> Subject: Re: nested-smmuv3 topic, Sep 2024
> 
> Hi Shameer,
> 
> Thanks for the reply!
> 
> On Thu, Sep 05, 2024 at 12:55:52PM +0000, Shameerali Kolothum Thodi
> wrote:
> > > The main takeaway from the discussion is to
> > > 1) Turn the vSMMU module into a pluggable one, like intel-iommu
> > > 2) Move the per-SMMU pxb bus and device auto-assign into libvirt
> > >
> > > Apart from the multi-vSMMU thing, there's basic nesting series:
> > > 0) Keep updating to the latest kernel uAPIs to support nesting
> >
> > By this you mean the old HWPT based nested-smmuv3 support?
> 
> HWPT + vIOMMU. The for-viommu/virq branches that I shared in my
> kernel series have those changes. Invalidations is done via the
> vIOMMU infrastructure.
> 
> > >
> > > I was trying to do all these three, but apparently too ambitious.
> > > The kernel side of work is still taking a lot of my bandwidth. So
> > > far I had almost-zero progress on task (1) and completely-zero on
> > > task (2).
> > >
> > > <-- Help Needed --->
> > > So, I'm wondering if anyone(s) might have some extra bandwidth in
> > > the following months helping these two tasks, either of which can
> > > be a standalone project I think.
> > >
> > > For task (0), I think I can keep updating the uAPI part, although
> > > it'd need some help for reviews, which I was hoping to occur after
> > > Intel sends the QEMU nesting backend patches. Once we know how big
> > > the rework is going to be, we may need to borrow some help at that
> > > point once again..
> >
> > I might have some bandwidth starting October and can take a look at
> > task 1 above. I haven't gone through the VIOMMU API model completely
> > yet and plan to do that soon.
>

I had  an initial look at this and also had some discussions with Eric at KVM
Forum(Thanks Eric!).

Going through the code, is it ok to introduce a "pci-bus" for the proposed
nested SMMUv3 device which will create the link between the SMMUv3 dev 
and the associated root complex(pxb-pcie).

Something like below,

-device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
-device arm-nested-smmuv3,pci-bus=pcie.1 \
-device pcie-root-port,id=pcie.port1,bus=pcie.1 \
-device vfio-pci,host=0000:75:00.1, bus=pcie.port1 \
...
-device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
-device arm-nested-smmuv3,pci-bus=pcie.2 \
-device pcie-root-port,id=pcie.port2,bus=pcie.2 \
-device vfio-pci,host=0000:75:00.2, bus=pcie.port2 \

This way we can invoke the pci_setup_iommu() with the 
right PCIBus during the nested SMMUv3 device realize fn.

Please let me know, if this works/scales with all the use cases we have.

Also Eric mentioned that when he initially added the support for SMMUv3,
the initial approach was -device based solution, but later changed to machine
option instead based on review comments. I managed to find the link where
this change was proposed(by Peter),

https://lore.kernel.org/all/CAFEAcA_H+sraWNVhEZc48eS11n6dC9CyEwTL44tPERiPBO+hbw@mail.gmail.com/

I hope the use cases we now have make it reasonable to introduce a "-device arm-nested-smmuv3" model.
Please let me know if there are still objections to going this way.

Thanks,
Shameer

Re: nested-smmuv3 topic, Sep 2024

[Nicolin Chen]

On Mon, Sep 30, 2024 at 10:45:31AM +0000, Shameerali Kolothum Thodi wrote:
> > -----Original Message-----
> > From: Nicolin Chen <nicolinc@nvidia.com>
> > Sent: Thursday, September 5, 2024 9:37 PM
> > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> > Cc: Eric Auger <eric.auger@redhat.com>; Mostafa Saleh
> > <smostafa@google.com>; qemu-arm@nongnu.org; qemu-
> > devel@nongnu.org; Peter Maydell <peter.maydell@linaro.org>; Jason
> > Gunthorpe <jgg@nvidia.com>; Jean-Philippe Brucker <jean-
> > philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> > <mshavit@google.com>; Andrea Bolognani <abologna@redhat.com>;
> > Michael S. Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>
> > Subject: Re: nested-smmuv3 topic, Sep 2024
> >
> > Hi Shameer,
> >
> > Thanks for the reply!
> >
> > On Thu, Sep 05, 2024 at 12:55:52PM +0000, Shameerali Kolothum Thodi
> > wrote:
> > > > The main takeaway from the discussion is to
> > > > 1) Turn the vSMMU module into a pluggable one, like intel-iommu
> > > > 2) Move the per-SMMU pxb bus and device auto-assign into libvirt
> > > >
> > > > Apart from the multi-vSMMU thing, there's basic nesting series:
> > > > 0) Keep updating to the latest kernel uAPIs to support nesting
> > >
> > > By this you mean the old HWPT based nested-smmuv3 support?
> >
> > HWPT + vIOMMU. The for-viommu/virq branches that I shared in my
> > kernel series have those changes. Invalidations is done via the
> > vIOMMU infrastructure.
> >
> > > >
> > > > I was trying to do all these three, but apparently too ambitious.
> > > > The kernel side of work is still taking a lot of my bandwidth. So
> > > > far I had almost-zero progress on task (1) and completely-zero on
> > > > task (2).
> > > >
> > > > <-- Help Needed --->
> > > > So, I'm wondering if anyone(s) might have some extra bandwidth in
> > > > the following months helping these two tasks, either of which can
> > > > be a standalone project I think.
> > > >
> > > > For task (0), I think I can keep updating the uAPI part, although
> > > > it'd need some help for reviews, which I was hoping to occur after
> > > > Intel sends the QEMU nesting backend patches. Once we know how big
> > > > the rework is going to be, we may need to borrow some help at that
> > > > point once again..
> > >
> > > I might have some bandwidth starting October and can take a look at
> > > task 1 above. I haven't gone through the VIOMMU API model completely
> > > yet and plan to do that soon.
> >
> 
> I had  an initial look at this and also had some discussions with Eric at KVM
> Forum(Thanks Eric!).

Wow, thank both of you!

> Going through the code, is it ok to introduce a "pci-bus" for the proposed
> nested SMMUv3 device which will create the link between the SMMUv3 dev
> and the associated root complex(pxb-pcie).
> 
> Something like below,
> 
> -device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
> -device arm-nested-smmuv3,pci-bus=pcie.1 \
> -device pcie-root-port,id=pcie.port1,bus=pcie.1 \
> -device vfio-pci,host=0000:75:00.1, bus=pcie.port1 \
> ...
> -device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
> -device arm-nested-smmuv3,pci-bus=pcie.2 \
> -device pcie-root-port,id=pcie.port2,bus=pcie.2 \
> -device vfio-pci,host=0000:75:00.2, bus=pcie.port2 \
> 
> This way we can invoke the pci_setup_iommu() with the
> right PCIBus during the nested SMMUv3 device realize fn.
> 
> Please let me know, if this works/scales with all the use cases we have.

That looks nice to me. Hopefully, IORT or Device Tree would be
easy to tie to the corresponding pci-bus as well..

> Also Eric mentioned that when he initially added the support for SMMUv3,
> the initial approach was -device based solution, but later changed to machine
> option instead based on review comments. I managed to find the link where
> this change was proposed(by Peter),
> 
> https://lore.kernel.org/all/CAFEAcA_H+sraWNVhEZc48eS11n6dC9CyEwTL44tPERiPBO+hbw@mail.gmail.com/
> 
> I hope the use cases we now have make it reasonable to introduce a "-device arm-nested-smmuv3" model.
> Please let me know if there are still objections to going this way.

I assume so. With multiple smmuv3 devices in the VM, we would need
this kinda flexibility to create them.

And FYI, I also found some resource in NVIDIA who will help me on
the QEMU workload, including our remaining task -- libvirt. I'll
align with them in the days ahead, and will keep all of us updated
after.

Thanks!
Nicolin

Re: nested-smmuv3 topic, Sep 2024

[Mostafa Saleh]

Hi Nicolin,

On Thu, Sep 05, 2024 at 01:26:20AM -0700, Nicolin Chen wrote:
> Hi all,
> 
> Hope I didn't miss anybody who is related to the topic. Please,
> feel free to add!
> 
> <--- Background --->
> As some of you know, there is an ongoing effort for nested-smmuv3
> support in QEMU on ARM, working with the kernel IOMMUFD uAPIs:
> [Nesting for vSTE]
> https://lore.kernel.org/linux-iommu/0-v2-621370057090+91fec-smmuv3_nesting_jgg@nvidia.com/
> [Nesting for invalidations]
> https://lore.kernel.org/linux-iommu/cover.1724776335.git.nicolinc@nvidia.com/
> 
> The kernel patches are still under review. Jason and I are hoping
> them to get merged at next cycle for v6.13, which means the QEMU
> patches might start a review process as early as Nov/Dec?
> 
> That being said, I think we are way behind the point that patches
> can get reviewed: most of the QEMU patches on my branches weren't
> touched very often, but merely updated to the latest kernel uAPIs
> for verification. So, I feel this might be a good point to gather
> folks together to discuss about the possible timeline and ask for
> help. I think this would potentially help folks who are going to
> attend the KVM forum (or LPC) to carry out a discussion. (Sorry,
> I won't make it due to some conflict..)
> 
> <-- Task Breakdown --->
> I previously sent a RFCv1 series collecting comments/suggestions,
> for multi-vSMMU instance design in ARM Virt code:
> https://lore.kernel.org/qemu-devel/cover.1719361174.git.nicolinc@nvidia.com/
> (And thanks again for all the inputs!)
> 
> The main takeaway from the discussion is to
> 1) Turn the vSMMU module into a pluggable one, like intel-iommu
> 2) Move the per-SMMU pxb bus and device auto-assign into libvirt
> 
> Apart from the multi-vSMMU thing, there's basic nesting series:
> 0) Keep updating to the latest kernel uAPIs to support nesting
> 
> I was trying to do all these three, but apparently too ambitious.
> The kernel side of work is still taking a lot of my bandwidth. So
> far I had almost-zero progress on task (1) and completely-zero on
> task (2).
> 
> <-- Help Needed --->
> So, I'm wondering if anyone(s) might have some extra bandwidth in
> the following months helping these two tasks, either of which can
> be a standalone project I think.

I don’t have plans to work on qemu in the next months, most of my
upstream focus will be on pKVM SMMUv3 support[1] in Linux which might
overlap with some of the vSMMU work but in the kernel side.

Otherwise, I’d be happy to review patches.

[1] https://lore.kernel.org/kvmarm/20230201125328.2186498-1-jean-philippe@linaro.org/

Thanks,
Mostafa

> 
> For task (0), I think I can keep updating the uAPI part, although
> it'd need some help for reviews, which I was hoping to occur after
> Intel sends the QEMU nesting backend patches. Once we know how big
> the rework is going to be, we may need to borrow some help at that
> point once again..
> 
> Thank you
> Nicolin

Re: nested-smmuv3 topic, Sep 2024

[Nicolin Chen]

Hi Mostafa,

On Fri, Sep 06, 2024 at 11:50:38AM +0000, Mostafa Saleh wrote:

> > <-- Help Needed --->
> > So, I'm wondering if anyone(s) might have some extra bandwidth in
> > the following months helping these two tasks, either of which can
> > be a standalone project I think.
> 
> I don’t have plans to work on qemu in the next months, most of my
> upstream focus will be on pKVM SMMUv3 support[1] in Linux which might
> overlap with some of the vSMMU work but in the kernel side.

Oh, that's a big work. I'll keep that under my radar.

> Otherwise, I’d be happy to review patches.

That'd be helpful indeed!

Thanks
Nicolin