From: Andrew Jones <andrew.jones@linux.dev>
To: Gong Shuai <gong.shuai@sanechips.com.cn>
Cc: andrew.jones@oss.qualcomm.com, anup@brainfault.org,
atish.patra@linux.dev, fangyu.yu@linux.alibaba.com,
iommu@lists.linux.dev, jgg@nvidia.com, jgg@ziepe.ca,
joro@8bytes.org, kvm-riscv@lists.infradead.org,
kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-riscv@lists.infradead.org, palmer@dabbelt.com,
pjw@kernel.org, robin.murphy@arm.com, tglx@kernel.org,
tomasz.jeznach@linux.dev, will@kernel.org,
zhangzhanpeng.jasper@bytedance.com, zong.li@sifive.com,
gsh517025@gmail.com
Subject: Re: [RFC PATCH v3 04/14] iommu/riscv: Reject live S2 replacement with forwarded IRQs
Date: Sat, 21 Nov 2026 18:01:46 +0100 [thread overview]
Message-ID: <awHLO6bWYw5V0k4p@orel> (raw)
In-Reply-To: <202610090914.6999EfSj006317@mse-fl1.zte.com.cn>
On Fri, Oct 09, 2026 at 05:05:30PM +0800, Gong Shuai wrote:
> Hi Drew,
>
> > MSI forwarding state is tied to the S2 domain MSI table. Replacing a
> > device's S2 domain while one of its IRQs is forwarded would leave the
> > IRQ state referring to the detached table.
> >
> > Reject direct S2-to-S2 replacement when the moving device has forwarded
> > IRQs. Introduce a per-device nr_forwarded_irqs counter, rather than
> > domain-wide accounting, so other devices in the same IOMMU group can
> > still move and can be rolled back if a later device fails.
> >
> > Later patches which introduce interrupt remapping support will manage
> > the newly introduced nr_forwarded_irqs counter.
> >
> > Signed-off-by: Andrew Jones <andrew.jones@oss.qualcomm.com>
> > ---
> > drivers/iommu/riscv/iommu.c | 36 ++++++++++++++++++++++++++++++++++++
> > 1 file changed, 36 insertions(+)
> >
> > diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
> > index 57f2884dec42..d48667112cd9 100644
> > --- a/drivers/iommu/riscv/iommu.c
> > +++ b/drivers/iommu/riscv/iommu.c
> > @@ -872,6 +872,7 @@ PT_IOMMU_CHECK_DOMAIN(struct riscv_iommu_domain, riscvpt.iommu, domain);
> > /* Private IOMMU data for managed devices, dev_iommu_priv_* */
> > struct riscv_iommu_info {
> > struct riscv_iommu_domain *domain;
> > + unsigned int nr_forwarded_irqs;
> > };
> >
> > static struct riscv_iommu_msi_table *riscv_iommu_domain_msi_table(struct iommu_domain *iommu_domain)
> > @@ -1432,6 +1433,36 @@ static int riscv_iommu_msi_table_alloc(struct riscv_iommu_domain *domain,
> > return 0;
> > }
> >
> > +static bool riscv_iommu_can_attach_paging_domain(struct iommu_domain *iommu_domain,
> > + struct device *dev,
> > + struct iommu_domain *old)
> > +{
> > + struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
> > + struct riscv_iommu_info *info = dev_iommu_priv_get(dev);
> > + bool new_is_s2 = domain->gscid;
> > + struct riscv_iommu_msi_table *new_msi_table, *old_msi_table;
> > +
> > + if (iommu_domain == old)
> > + return true;
> > +
> > + new_msi_table = riscv_iommu_domain_msi_table(iommu_domain);
> > + old_msi_table = riscv_iommu_domain_msi_table(old);
> > +
> > + if (new_msi_table)
> > + lockdep_assert_held(&new_msi_table->lock);
> > + if (old_msi_table)
> > + lockdep_assert_held(&old_msi_table->lock);
> > +
> > + /*
> > + * Per-device accounting allows other devices in the same IOMMU group
> > + * to move or roll back while this device has forwarded interrupts.
> > + */
> > + if (new_is_s2 && old_msi_table && info->nr_forwarded_irqs)
>
> This check might miss one case. old_msi_table comes from this
> attach's old domain, so only a direct S2 to S2 swap is caught.
> A device with live forwarded IRQs can still be moved to blocking
> and then to a different S2 domain, and on that second attach old
> is the blocking domain, so old_msi_table is NULL and the check
> passes. This is only from reading the code, I have no test program
> that triggers it.
>
> If you think this is a real gap, one idea would be to record the
> table the forwarding was set up on in riscv_iommu_info(a pointer to
> riscv_iommu_msi_table), set when the first IRQ of the device is
> forwarded and cleared when the last one is unforwarded. The pointer
> survives the blocking hop, since attaching to blocking domain only
> clears info->domain. Then can_attach here will reject when the new
> domain's table differs from it. That covers both the direct and the
> blocking case, and the PCI reset path still works because the device
> returns to the same S2 domain. The unforward path can use the pointer
> instead of info->domain. Note that the S2 domain owning the table
> must not be freed while a device still points at that table.
>
Hi Shuai,
Yes, this is a gap and I agree we should track and check the association
for the fix. For v4, I'm looking at moving the MSI table configuration
into a VM-wide object which S2 domains reference, similar to GICv4's
its_vm. The recorded pointer would then be to that object, which the
hypervisor keeps alive until every forward is undone.
Thanks,
drew
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
WARNING: multiple messages have this Message-ID (diff)
From: Andrew Jones <andrew.jones@linux.dev>
To: Gong Shuai <gong.shuai@sanechips.com.cn>
Cc: andrew.jones@oss.qualcomm.com, anup@brainfault.org,
atish.patra@linux.dev, fangyu.yu@linux.alibaba.com,
iommu@lists.linux.dev, jgg@nvidia.com, jgg@ziepe.ca,
joro@8bytes.org, kvm-riscv@lists.infradead.org,
kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-riscv@lists.infradead.org, palmer@dabbelt.com,
pjw@kernel.org, robin.murphy@arm.com, tglx@kernel.org,
tomasz.jeznach@linux.dev, will@kernel.org,
zhangzhanpeng.jasper@bytedance.com, zong.li@sifive.com,
gsh517025@gmail.com
Subject: Re: [RFC PATCH v3 04/14] iommu/riscv: Reject live S2 replacement with forwarded IRQs
Date: Sat, 21 Nov 2026 18:01:46 +0100 [thread overview]
Message-ID: <awHLO6bWYw5V0k4p@orel> (raw)
In-Reply-To: <202610090914.6999EfSj006317@mse-fl1.zte.com.cn>
On Fri, Oct 09, 2026 at 05:05:30PM +0800, Gong Shuai wrote:
> Hi Drew,
>
> > MSI forwarding state is tied to the S2 domain MSI table. Replacing a
> > device's S2 domain while one of its IRQs is forwarded would leave the
> > IRQ state referring to the detached table.
> >
> > Reject direct S2-to-S2 replacement when the moving device has forwarded
> > IRQs. Introduce a per-device nr_forwarded_irqs counter, rather than
> > domain-wide accounting, so other devices in the same IOMMU group can
> > still move and can be rolled back if a later device fails.
> >
> > Later patches which introduce interrupt remapping support will manage
> > the newly introduced nr_forwarded_irqs counter.
> >
> > Signed-off-by: Andrew Jones <andrew.jones@oss.qualcomm.com>
> > ---
> > drivers/iommu/riscv/iommu.c | 36 ++++++++++++++++++++++++++++++++++++
> > 1 file changed, 36 insertions(+)
> >
> > diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
> > index 57f2884dec42..d48667112cd9 100644
> > --- a/drivers/iommu/riscv/iommu.c
> > +++ b/drivers/iommu/riscv/iommu.c
> > @@ -872,6 +872,7 @@ PT_IOMMU_CHECK_DOMAIN(struct riscv_iommu_domain, riscvpt.iommu, domain);
> > /* Private IOMMU data for managed devices, dev_iommu_priv_* */
> > struct riscv_iommu_info {
> > struct riscv_iommu_domain *domain;
> > + unsigned int nr_forwarded_irqs;
> > };
> >
> > static struct riscv_iommu_msi_table *riscv_iommu_domain_msi_table(struct iommu_domain *iommu_domain)
> > @@ -1432,6 +1433,36 @@ static int riscv_iommu_msi_table_alloc(struct riscv_iommu_domain *domain,
> > return 0;
> > }
> >
> > +static bool riscv_iommu_can_attach_paging_domain(struct iommu_domain *iommu_domain,
> > + struct device *dev,
> > + struct iommu_domain *old)
> > +{
> > + struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
> > + struct riscv_iommu_info *info = dev_iommu_priv_get(dev);
> > + bool new_is_s2 = domain->gscid;
> > + struct riscv_iommu_msi_table *new_msi_table, *old_msi_table;
> > +
> > + if (iommu_domain == old)
> > + return true;
> > +
> > + new_msi_table = riscv_iommu_domain_msi_table(iommu_domain);
> > + old_msi_table = riscv_iommu_domain_msi_table(old);
> > +
> > + if (new_msi_table)
> > + lockdep_assert_held(&new_msi_table->lock);
> > + if (old_msi_table)
> > + lockdep_assert_held(&old_msi_table->lock);
> > +
> > + /*
> > + * Per-device accounting allows other devices in the same IOMMU group
> > + * to move or roll back while this device has forwarded interrupts.
> > + */
> > + if (new_is_s2 && old_msi_table && info->nr_forwarded_irqs)
>
> This check might miss one case. old_msi_table comes from this
> attach's old domain, so only a direct S2 to S2 swap is caught.
> A device with live forwarded IRQs can still be moved to blocking
> and then to a different S2 domain, and on that second attach old
> is the blocking domain, so old_msi_table is NULL and the check
> passes. This is only from reading the code, I have no test program
> that triggers it.
>
> If you think this is a real gap, one idea would be to record the
> table the forwarding was set up on in riscv_iommu_info(a pointer to
> riscv_iommu_msi_table), set when the first IRQ of the device is
> forwarded and cleared when the last one is unforwarded. The pointer
> survives the blocking hop, since attaching to blocking domain only
> clears info->domain. Then can_attach here will reject when the new
> domain's table differs from it. That covers both the direct and the
> blocking case, and the PCI reset path still works because the device
> returns to the same S2 domain. The unforward path can use the pointer
> instead of info->domain. Note that the S2 domain owning the table
> must not be freed while a device still points at that table.
>
Hi Shuai,
Yes, this is a gap and I agree we should track and check the association
for the fix. For v4, I'm looking at moving the MSI table configuration
into a VM-wide object which S2 domains reference, similar to GICv4's
its_vm. The recorded pointer would then be to that object, which the
hypervisor keeps alive until every forward is undone.
Thanks,
drew
WARNING: multiple messages have this Message-ID (diff)
From: Andrew Jones <andrew.jones@linux.dev>
To: Gong Shuai <gong.shuai@sanechips.com.cn>
Cc: andrew.jones@oss.qualcomm.com, anup@brainfault.org,
atish.patra@linux.dev, fangyu.yu@linux.alibaba.com,
iommu@lists.linux.dev, jgg@nvidia.com, jgg@ziepe.ca,
joro@8bytes.org, kvm-riscv@lists.infradead.org,
kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-riscv@lists.infradead.org, palmer@dabbelt.com,
pjw@kernel.org, robin.murphy@arm.com, tglx@kernel.org,
tomasz.jeznach@linux.dev, will@kernel.org,
zhangzhanpeng.jasper@bytedance.com, zong.li@sifive.com,
gsh517025@gmail.com
Subject: Re: [RFC PATCH v3 04/14] iommu/riscv: Reject live S2 replacement with forwarded IRQs
Date: Sat, 21 Nov 2026 18:01:46 +0100 [thread overview]
Message-ID: <awHLO6bWYw5V0k4p@orel> (raw)
In-Reply-To: <202610090914.6999EfSj006317@mse-fl1.zte.com.cn>
On Fri, Oct 09, 2026 at 05:05:30PM +0800, Gong Shuai wrote:
> Hi Drew,
>
> > MSI forwarding state is tied to the S2 domain MSI table. Replacing a
> > device's S2 domain while one of its IRQs is forwarded would leave the
> > IRQ state referring to the detached table.
> >
> > Reject direct S2-to-S2 replacement when the moving device has forwarded
> > IRQs. Introduce a per-device nr_forwarded_irqs counter, rather than
> > domain-wide accounting, so other devices in the same IOMMU group can
> > still move and can be rolled back if a later device fails.
> >
> > Later patches which introduce interrupt remapping support will manage
> > the newly introduced nr_forwarded_irqs counter.
> >
> > Signed-off-by: Andrew Jones <andrew.jones@oss.qualcomm.com>
> > ---
> > drivers/iommu/riscv/iommu.c | 36 ++++++++++++++++++++++++++++++++++++
> > 1 file changed, 36 insertions(+)
> >
> > diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
> > index 57f2884dec42..d48667112cd9 100644
> > --- a/drivers/iommu/riscv/iommu.c
> > +++ b/drivers/iommu/riscv/iommu.c
> > @@ -872,6 +872,7 @@ PT_IOMMU_CHECK_DOMAIN(struct riscv_iommu_domain, riscvpt.iommu, domain);
> > /* Private IOMMU data for managed devices, dev_iommu_priv_* */
> > struct riscv_iommu_info {
> > struct riscv_iommu_domain *domain;
> > + unsigned int nr_forwarded_irqs;
> > };
> >
> > static struct riscv_iommu_msi_table *riscv_iommu_domain_msi_table(struct iommu_domain *iommu_domain)
> > @@ -1432,6 +1433,36 @@ static int riscv_iommu_msi_table_alloc(struct riscv_iommu_domain *domain,
> > return 0;
> > }
> >
> > +static bool riscv_iommu_can_attach_paging_domain(struct iommu_domain *iommu_domain,
> > + struct device *dev,
> > + struct iommu_domain *old)
> > +{
> > + struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
> > + struct riscv_iommu_info *info = dev_iommu_priv_get(dev);
> > + bool new_is_s2 = domain->gscid;
> > + struct riscv_iommu_msi_table *new_msi_table, *old_msi_table;
> > +
> > + if (iommu_domain == old)
> > + return true;
> > +
> > + new_msi_table = riscv_iommu_domain_msi_table(iommu_domain);
> > + old_msi_table = riscv_iommu_domain_msi_table(old);
> > +
> > + if (new_msi_table)
> > + lockdep_assert_held(&new_msi_table->lock);
> > + if (old_msi_table)
> > + lockdep_assert_held(&old_msi_table->lock);
> > +
> > + /*
> > + * Per-device accounting allows other devices in the same IOMMU group
> > + * to move or roll back while this device has forwarded interrupts.
> > + */
> > + if (new_is_s2 && old_msi_table && info->nr_forwarded_irqs)
>
> This check might miss one case. old_msi_table comes from this
> attach's old domain, so only a direct S2 to S2 swap is caught.
> A device with live forwarded IRQs can still be moved to blocking
> and then to a different S2 domain, and on that second attach old
> is the blocking domain, so old_msi_table is NULL and the check
> passes. This is only from reading the code, I have no test program
> that triggers it.
>
> If you think this is a real gap, one idea would be to record the
> table the forwarding was set up on in riscv_iommu_info(a pointer to
> riscv_iommu_msi_table), set when the first IRQ of the device is
> forwarded and cleared when the last one is unforwarded. The pointer
> survives the blocking hop, since attaching to blocking domain only
> clears info->domain. Then can_attach here will reject when the new
> domain's table differs from it. That covers both the direct and the
> blocking case, and the PCI reset path still works because the device
> returns to the same S2 domain. The unforward path can use the pointer
> instead of info->domain. Note that the S2 domain owning the table
> must not be freed while a device still points at that table.
>
Hi Shuai,
Yes, this is a gap and I agree we should track and check the association
for the fix. For v4, I'm looking at moving the MSI table configuration
into a VM-wide object which S2 domains reference, similar to GICv4's
its_vm. The recorded pointer would then be to that object, which the
hypervisor keeps alive until every forward is undone.
Thanks,
drew
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
next prev parent reply other threads:[~2026-11-21 17:02 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-28 14:30 [RFC PATCH v3 00/14] iommu/riscv: Add irqbypass support Andrew Jones
2026-09-28 14:30 ` Andrew Jones
2026-09-28 14:30 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 01/14] iommu/riscv: Allocate MSI tables for second-stage domains Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-10-05 13:25 ` Andrew Jones
2026-10-05 13:25 ` Andrew Jones
2026-10-05 13:25 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 02/14] iommu/riscv: Prepare domain bonds for outer locking Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 03/14] iommu/riscv: Serialize MSI table publication with domain attachment Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 04/14] iommu/riscv: Reject live S2 replacement with forwarded IRQs Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-10-09 9:05 ` Gong Shuai
2026-10-09 9:05 ` Gong Shuai
2026-10-09 9:05 ` Gong Shuai
2026-11-21 17:01 ` Andrew Jones [this message]
2026-11-21 17:01 ` Andrew Jones
2026-11-21 17:01 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 05/14] iommu/riscv: Derive the IOMMU from the device in IODIR updates Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 06/14] iommu/riscv: Cache the programmed device context Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 07/14] iommu/riscv: Prepare MSI table updates for interrupt remapping Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 08/14] irqchip/riscv-imsic: Define IOMMU IRQ bypass protocol Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 09/14] genirq/msi: Provide DOMAIN_BUS_MSI_REMAP Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 10/14] iommu/riscv: Add IRQ domain for interrupt remapping Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 11/14] iommu/riscv: Prepare info->domain for concurrent RCU access Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 12/14] iommu/riscv: Prepare interrupt remapping for IRQ bypass Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 13/14] iommu/riscv: Validate IRQ forwarding requests Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 14/14] iommu/riscv: Implement IRQ forwarding Andrew Jones
2026-09-28 14:31 ` Andrew Jones
2026-09-28 14:31 ` Andrew Jones
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=awHLO6bWYw5V0k4p@orel \
--to=andrew.jones@linux.dev \
--cc=andrew.jones@oss.qualcomm.com \
--cc=anup@brainfault.org \
--cc=atish.patra@linux.dev \
--cc=fangyu.yu@linux.alibaba.com \
--cc=gong.shuai@sanechips.com.cn \
--cc=gsh517025@gmail.com \
--cc=iommu@lists.linux.dev \
--cc=jgg@nvidia.com \
--cc=jgg@ziepe.ca \
--cc=joro@8bytes.org \
--cc=kvm-riscv@lists.infradead.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=palmer@dabbelt.com \
--cc=pjw@kernel.org \
--cc=robin.murphy@arm.com \
--cc=tglx@kernel.org \
--cc=tomasz.jeznach@linux.dev \
--cc=will@kernel.org \
--cc=zhangzhanpeng.jasper@bytedance.com \
--cc=zong.li@sifive.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.