Kernel KVM virtualization development
 help / color / mirror / Atom feed
From: Andrew Jones <andrew.jones@linux.dev>
To: Gong Shuai <gong.shuai@sanechips.com.cn>
Cc: andrew.jones@oss.qualcomm.com, anup@brainfault.org,
	 atish.patra@linux.dev, fangyu.yu@linux.alibaba.com,
	iommu@lists.linux.dev,  jgg@nvidia.com, jgg@ziepe.ca,
	joro@8bytes.org, kvm-riscv@lists.infradead.org,
	 kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-riscv@lists.infradead.org,  palmer@dabbelt.com,
	pjw@kernel.org, robin.murphy@arm.com, tglx@kernel.org,
	 tomasz.jeznach@linux.dev, will@kernel.org,
	zhangzhanpeng.jasper@bytedance.com,  zong.li@sifive.com,
	gsh517025@gmail.com
Subject: Re: [RFC PATCH v3 04/14] iommu/riscv: Reject live S2 replacement with forwarded IRQs
Date: Sat, 21 Nov 2026 18:01:46 +0100	[thread overview]
Message-ID: <awHLO6bWYw5V0k4p@orel> (raw)
In-Reply-To: <202610090914.6999EfSj006317@mse-fl1.zte.com.cn>

On Fri, Oct 09, 2026 at 05:05:30PM +0800, Gong Shuai wrote:
> Hi Drew,
> 
> > MSI forwarding state is tied to the S2 domain MSI table. Replacing a
> > device's S2 domain while one of its IRQs is forwarded would leave the
> > IRQ state referring to the detached table.
> > 
> > Reject direct S2-to-S2 replacement when the moving device has forwarded
> > IRQs. Introduce a per-device nr_forwarded_irqs counter, rather than
> > domain-wide accounting, so other devices in the same IOMMU group can
> > still move and can be rolled back if a later device fails.
> > 
> > Later patches which introduce interrupt remapping support will manage
> > the newly introduced nr_forwarded_irqs counter.
> > 
> > Signed-off-by: Andrew Jones <andrew.jones@oss.qualcomm.com>
> > ---
> >  drivers/iommu/riscv/iommu.c | 36 ++++++++++++++++++++++++++++++++++++
> >  1 file changed, 36 insertions(+)
> > 
> > diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
> > index 57f2884dec42..d48667112cd9 100644
> > --- a/drivers/iommu/riscv/iommu.c
> > +++ b/drivers/iommu/riscv/iommu.c
> > @@ -872,6 +872,7 @@ PT_IOMMU_CHECK_DOMAIN(struct riscv_iommu_domain, riscvpt.iommu, domain);
> >  /* Private IOMMU data for managed devices, dev_iommu_priv_* */
> >  struct riscv_iommu_info {
> >  	struct riscv_iommu_domain *domain;
> > +	unsigned int nr_forwarded_irqs;
> >  };
> >  
> >  static struct riscv_iommu_msi_table *riscv_iommu_domain_msi_table(struct iommu_domain *iommu_domain)
> > @@ -1432,6 +1433,36 @@ static int riscv_iommu_msi_table_alloc(struct riscv_iommu_domain *domain,
> >  	return 0;
> >  }
> >  
> > +static bool riscv_iommu_can_attach_paging_domain(struct iommu_domain *iommu_domain,
> > +						 struct device *dev,
> > +						 struct iommu_domain *old)
> > +{
> > +	struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
> > +	struct riscv_iommu_info *info = dev_iommu_priv_get(dev);
> > +	bool new_is_s2 = domain->gscid;
> > +	struct riscv_iommu_msi_table *new_msi_table, *old_msi_table;
> > +
> > +	if (iommu_domain == old)
> > +		return true;
> > +
> > +	new_msi_table = riscv_iommu_domain_msi_table(iommu_domain);
> > +	old_msi_table = riscv_iommu_domain_msi_table(old);
> > +
> > +	if (new_msi_table)
> > +		lockdep_assert_held(&new_msi_table->lock);
> > +	if (old_msi_table)
> > +		lockdep_assert_held(&old_msi_table->lock);
> > +
> > +	/*
> > +	 * Per-device accounting allows other devices in the same IOMMU group
> > +	 * to move or roll back while this device has forwarded interrupts.
> > +	 */
> > +	if (new_is_s2 && old_msi_table && info->nr_forwarded_irqs)
> 
> This check might miss one case. old_msi_table comes from this
> attach's old domain, so only a direct S2 to S2 swap is caught.
> A device with live forwarded IRQs can still be moved to blocking
> and then to a different S2 domain, and on that second attach old
> is the blocking domain, so old_msi_table is NULL and the check
> passes. This is only from reading the code, I have no test program
> that triggers it.
> 
> If you think this is a real gap, one idea would be to record the
> table the forwarding was set up on in riscv_iommu_info(a pointer to
> riscv_iommu_msi_table), set when the first IRQ of the device is
> forwarded and cleared when the last one is unforwarded. The pointer
> survives the blocking hop, since attaching to blocking domain only
> clears info->domain. Then can_attach here will reject when the new
> domain's table differs from it. That covers both the direct and the
> blocking case, and the PCI reset path still works because the device
> returns to the same S2 domain. The unforward path can use the pointer
> instead of info->domain. Note that the S2 domain owning the table
> must not be freed while a device still points at that table.
> 

Hi Shuai,

Yes, this is a gap and I agree we should track and check the association
for the fix. For v4, I'm looking at moving the MSI table configuration
into a VM-wide object which S2 domains reference, similar to GICv4's
its_vm. The recorded pointer would then be to that object, which the
hypervisor keeps alive until every forward is undone.

Thanks,
drew

  reply	other threads:[~2026-11-21 17:02 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-09-28 14:30 [RFC PATCH v3 00/14] iommu/riscv: Add irqbypass support Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 01/14] iommu/riscv: Allocate MSI tables for second-stage domains Andrew Jones
2026-10-05 13:25   ` Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 02/14] iommu/riscv: Prepare domain bonds for outer locking Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 03/14] iommu/riscv: Serialize MSI table publication with domain attachment Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 04/14] iommu/riscv: Reject live S2 replacement with forwarded IRQs Andrew Jones
2026-10-09  9:05   ` Gong Shuai
2026-11-21 17:01     ` Andrew Jones [this message]
2026-09-28 14:31 ` [RFC PATCH v3 05/14] iommu/riscv: Derive the IOMMU from the device in IODIR updates Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 06/14] iommu/riscv: Cache the programmed device context Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 07/14] iommu/riscv: Prepare MSI table updates for interrupt remapping Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 08/14] irqchip/riscv-imsic: Define IOMMU IRQ bypass protocol Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 09/14] genirq/msi: Provide DOMAIN_BUS_MSI_REMAP Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 10/14] iommu/riscv: Add IRQ domain for interrupt remapping Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 11/14] iommu/riscv: Prepare info->domain for concurrent RCU access Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 12/14] iommu/riscv: Prepare interrupt remapping for IRQ bypass Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 13/14] iommu/riscv: Validate IRQ forwarding requests Andrew Jones
2026-09-28 14:31 ` [RFC PATCH v3 14/14] iommu/riscv: Implement IRQ forwarding Andrew Jones

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=awHLO6bWYw5V0k4p@orel \
    --to=andrew.jones@linux.dev \
    --cc=andrew.jones@oss.qualcomm.com \
    --cc=anup@brainfault.org \
    --cc=atish.patra@linux.dev \
    --cc=fangyu.yu@linux.alibaba.com \
    --cc=gong.shuai@sanechips.com.cn \
    --cc=gsh517025@gmail.com \
    --cc=iommu@lists.linux.dev \
    --cc=jgg@nvidia.com \
    --cc=jgg@ziepe.ca \
    --cc=joro@8bytes.org \
    --cc=kvm-riscv@lists.infradead.org \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-riscv@lists.infradead.org \
    --cc=palmer@dabbelt.com \
    --cc=pjw@kernel.org \
    --cc=robin.murphy@arm.com \
    --cc=tglx@kernel.org \
    --cc=tomasz.jeznach@linux.dev \
    --cc=will@kernel.org \
    --cc=zhangzhanpeng.jasper@bytedance.com \
    --cc=zong.li@sifive.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox