From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5189223EAB7 for ; Sun, 21 Dec 2025 11:55:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766318142; cv=none; b=DS0rleklVayZ5S/lr7SdxmaGRe33GI9TmdzIkzXJXQjPgSYkuRphndOBVFw9X8omjjZ3apSvJ4NHVy6CcvJ3PRZa8ITxoWV+BcC4n72qn9m7agPSknFXcWxSj4rRxEOGCnliGpY9GUsYGRAKTbIZENblu0ZLWIJEt83UFeJda18= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766318142; c=relaxed/simple; bh=xFCxaQ4sMgV9aEF9w0pd8Mpg1gknq0Yw3Ka/qdvZOsI=; h=Date:Message-ID:From:To:Cc:Subject:In-Reply-To:References: MIME-Version:Content-Type; b=mr0jiXKmzKi0/ndZiTvTn0ynGgIgDSTYfd4hbRSDK0MZjpgQqDT3HOxOg1DBgF0Y20nxbCt4vU8DrcqfT5ATQEYWYPeVmW7Tc9nUbt0YkNmZqMsrjVWOYOFamEFhCOP3PIMa/0WhOZaCXPVyr1iHVBbFEugPuvnmiOHLybGCgHM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CfYiiMsA; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CfYiiMsA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 01D46C116C6; Sun, 21 Dec 2025 11:55:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1766318142; bh=xFCxaQ4sMgV9aEF9w0pd8Mpg1gknq0Yw3Ka/qdvZOsI=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=CfYiiMsAsw32cEDnurYndGAHkKrI5qm2rrx7MfTa8JauPeHICa9D+LV9cmag0wyqH biwEqeiFxtDyo0EpdPE3VypBgjvrqCYUjBz6acXt9YXZjVAbiDbOD4tjMmf8FXrwVs Qi4tmq7NQu7eLhVM24RMM7atHUg3g8yBYt1ofLl9e8oxDZabjCqVMbXBDeQmXaI/lf x8JwNzcSFBRV1cAIbIII7X8cZet2XQk24rRmFC/v8r/ezyCFG/blleMA6/3CFj7ckK iCfC8trdf/V8QkXpKFSfus3/nzw/38aksO81XXPy+lJpBgem1UvhfmVsptwc86fT0x MWhJrkUqX7OfA== Received: from sofa.misterjones.org ([185.219.108.64] helo=lobster-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1vXI2F-0000000EL7H-3LVk; Sun, 21 Dec 2025 11:55:39 +0000 Date: Sun, 21 Dec 2025 11:55:39 +0000 Message-ID: <87tsxkdp6s.wl-maz@kernel.org> From: Marc Zyngier To: Luigi Rizzo Cc: tglx@linutronix.de, bhelgaas@google.com, linux-kernel@vger.kernel.org Subject: Re: [patch 1/2] irqchip/msi-lib: Honor the MSI_FLAG_PCI_MSI_MASK_PARENT flag In-Reply-To: <20251220193120.3339162-1-lrizzo@google.com> References: <20250903135433.380783272@linutronix.de> <20251220193120.3339162-1-lrizzo@google.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: lrizzo@google.com, tglx@linutronix.de, bhelgaas@google.com, linux-kernel@vger.kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Sat, 20 Dec 2025 19:31:19 +0000, Luigi Rizzo wrote: > > There are platforms (including some ARM SoC) where the MSIx > writes are a performance killer, because they are exceedingly > serializing on the PCIe root port. > > These platforms are the key motivation for Global Software > Interrupt Moderation (GSIM) which relies on actually masking > device interrupts so the MSIx writes are not generated. > https://lore.kernel.org/all/20251217112128.1401896-1-lrizzo@google.com/ > > Overriding mask/unmask with irq_chip_mask_parent() makes software > moderation ineffective. GSIM works great on ARM platforms before > this patch, but becomes ineffective afterwards, e.g. on linux 6.18. You do realise that "ARM platforms" means nothing at all, right? What you actually mean is "the ARM machines I have access to exhibit some platform-specific behaviour that may or may not be a general behaviour". Your particular circumstances are not in any way something you can generalise, unless you demonstrate this is caused by an architectural requirement rather than an implementation defect. > The round trip through the PCI endpoint for mask_irq(), caused by the > readback to make sure the PCI write has been sent, is almost always > (or really always) unnecessary. Masking is inherently racy; waiting > that the PCIe write has arrived at the device won't guarantee that an > interrupt has arrived in the meantime, so there is really no benefit > in the readback (which, for instance, can be conditionally removed with > code like the one below). > > I measured the cost of pci_irq_mask_msix() and it goes from 1000-1500ns > with the readl(), down to 40-50ns without it. > > Once we remove the costly readback, is there any remaining reason > to overwrite [un]mask_irq() with irq_chip_[un]mask_parent() ? So you are effectively not masking at all and just rely on hope instead. I have the utmost confidence in this sort of stuff. Totally. What you missing is that hitting the config space is causing pretty high overhead in KVM guests, where the accesses (write and read to the MSI masks) are trapped all the way to userspace (and back into VFIO), while the masking at the ITS level is much cheaper. Masking at the ITS level (and only there) also means that the VM can be migrated without having to worry about the PBA in each device, because the pending state is already part of the VM's memory, nicely tucked away in the RD tables. Finally, it aligns PCI devices with non-PCI device behaviour, something that is highly desirable. For me, that totally beats your interrupt mitigation thing. Thanks, M. -- Jazz isn't dead. It just smells funny.