From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 155901E2009; Tue, 3 Dec 2024 22:59:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733266761; cv=none; b=ZRx0C/o7oQ8GMlCKeUtX8sECejFZ/YiA719jPu9SKNZ+Q5JHFqt9GuK5qGcGZFyGrSwkEoka4RmARMyJNPcgKYS+z7lRAjY8MxHOhN5B9s2yjZvW6CHpI/Li2Tu/Ai9iV4fiiMABbitMVf0eodMHEvq57afxRmLFOWcYuvQIs+s= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733266761; c=relaxed/simple; bh=Ts5twGI5WLXGCgWxTjs5dJLKgs4RCUcQq24XWrHPHPg=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=oJrvl76CmSe4J9z06UtVmJKCRMIJuyGqRJC8+yx0KU9dEaFF71GbBzLju9DHhw4Zq4pDUPifvKZBBjaYobOvClplEO7t3GRFTblWlTxUMr2l6PNkJkxb0ph+m1Sjy3XXDW6NdwObfqPOT5xnGMhyhE3NDihUTf+iqLas3q4Ps5I= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=uUpHwAst; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=c+FJ40Es; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="uUpHwAst"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="c+FJ40Es" From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1733266758; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/xkyLKWpUVdm6mfEZQQyS1yp6VoP0OWawZ5KoEQBw98=; b=uUpHwAstAvPIPqurwVOkG4Z+xsuHUFtZ1THMwPD1xXJxmU3itDeHHaJi+jhl8lLBJ7eZ/4 rikeyL9y6/r0ZR071XOgksOdUYZ/PQCe+hGK2BvzyddYsmst8WpxFVe+UJVJzQGrdhxgcV HLNq7f9LeqLepzH7RGzXJqq5B5jJnr9J9t79clawHMUjQb2mmGKapYpb+SK2DhA2lakxq8 iBadArbE9YJ0ZSq/71hfYpbAAi3BpT0+6GSmKjPazSH/yyLgWTtVB1F0ADpZ6/xF9oAxLK SGrYIEc+XE/odY9cufIHKNqH/CkVZlI3It65iNeu/FplcxvHLpJdwfka9aOq7Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1733266758; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/xkyLKWpUVdm6mfEZQQyS1yp6VoP0OWawZ5KoEQBw98=; b=c+FJ40EsursGL5c6ZmSdluV0Hszl6xqc8ugVFc+JqAg5TyAq0eyEU7V2yrjQR13ZoVbsYq CKD1ARZ8ojo9HDCA== To: Anup Patel Cc: Andrew Jones , iommu@lists.linux.dev, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, tjeznach@rivosinc.com, zong.li@sifive.com, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, atishp@atishpatra.org, alex.williamson@redhat.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu Subject: Re: [RFC PATCH 01/15] irqchip/riscv-imsic: Use hierarchy to reach irq_set_affinity In-Reply-To: <874j3ktrjv.ffs@tglx> References: <20241114161845.502027-17-ajones@ventanamicro.com> <20241114161845.502027-18-ajones@ventanamicro.com> <87mshcub2u.ffs@tglx> <874j3ktrjv.ffs@tglx> Date: Tue, 03 Dec 2024 23:59:17 +0100 Message-ID: <87ser4s796.ffs@tglx> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Tue, Dec 03 2024 at 21:55, Thomas Gleixner wrote: > On Tue, Dec 03 2024 at 22:07, Anup Patel wrote: >> On Tue, Dec 3, 2024 at 7:23=E2=80=AFPM Thomas Gleixner wrote: >>> Sorry, I missed that when reviewing the original IMSIC MSI support. >>> >>> The whole IMSIC MSI support can be moved over to MSI LIB which makes all >>> of this indirection go away and your intermediate domain will just fit >>> in. >>> >>> Uncompiled patch below. If that works, it needs to be split up properly. >>> >>> Note, this removes the setup of the irq_retrigger callback, but that's >>> fine because on hierarchical domains irq_chip_retrigger_hierarchy() is >>> invoked anyway. See try_retrigger(). >> >> The IMSIC driver was merged one kernel release before common >> MSI LIB was merged. > > Ah indeed. > >> We should definitely update the IMSIC driver to use MSI LIB, I will >> try your suggested changes (below) and post a separate series. > > Pick up the delta patch I gave Andrew... As I was looking at something else MSI related I had a look at imsic_irq_set_affinity() again. It's actually required to have the message write in that function and not afterwards as you invoke imsic_vector_move() from that function. That's obviously not true for the remap case as that will not change the message address/data pair because the remap table entry is immutable - at least I assume so for my mental sanity sake :) But that brings me to a related question. How is this supposed to work with non-atomic message updates? PCI/MSI does not necessarily provide masking, and the write of the address/data pair is done in bits and pieces. So you can end up with an intermediate state seen by the device which ends up somewhere in interrupt nirvana space. See the dance in msi_set_affinity() and commit 6f1a4891a592 ("x86/apic/msi: Plug non-maskable MSI affinity race") for further explanation. The way how the IMSIC driver works seems to be pretty much the same as the x86 APIC mess: @address is the physical address of the per CPU MSI target address and @data is the vector ID on that CPU. So the non-atomic update in case of non-maskable MSI suffers from the same problem. It works most of the time, but if it doesn't you might stare at the occasionally lost interrupt and the stale device in disbelief for quite a while :) I might be missing something which magically prevent that though :) Thanks, tglx From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 47220E77160 for ; Tue, 3 Dec 2024 22:59:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:References :In-Reply-To:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vxmEfyeISi6g91yUHYfe0znjF7HSTupEEvsFfCuL/JY=; b=sHq96SkG408uil CPsakH68g2du1EHTPuOqiARgQ4S9sY7d4GIC7xX65mAvqtfh0S3J9wHLOzHL3wBKZahzpfibrMe6W YfCKALdAYZr60J+GOQ0FiELqytFPxcEzNn1Rom6VG6ikGm9pW0s+bTkFw+NUjVeIOyAhn2b7miekc DlPIOw68C5/wy5pQYB2zIcYBs/l//fPUBCG1QzAnB4+cvJLaMNJUHMOqLn4hL+n+9FPc4A8VCiNNM QuLLJgVe4sdLaq7zK9cBDpSC2Q+246PfuskJFSwYskWVmtB7inNh3vAW1EGIdgvjhgjTU679EDQur uhnWGABLIZJLwiU0vqbg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tIbrY-0000000AynU-1qOG; Tue, 03 Dec 2024 22:59:24 +0000 Received: from galois.linutronix.de ([2a0a:51c0:0:12e:550::1]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tIbrU-0000000Aym0-475D; Tue, 03 Dec 2024 22:59:22 +0000 From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1733266758; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/xkyLKWpUVdm6mfEZQQyS1yp6VoP0OWawZ5KoEQBw98=; b=uUpHwAstAvPIPqurwVOkG4Z+xsuHUFtZ1THMwPD1xXJxmU3itDeHHaJi+jhl8lLBJ7eZ/4 rikeyL9y6/r0ZR071XOgksOdUYZ/PQCe+hGK2BvzyddYsmst8WpxFVe+UJVJzQGrdhxgcV HLNq7f9LeqLepzH7RGzXJqq5B5jJnr9J9t79clawHMUjQb2mmGKapYpb+SK2DhA2lakxq8 iBadArbE9YJ0ZSq/71hfYpbAAi3BpT0+6GSmKjPazSH/yyLgWTtVB1F0ADpZ6/xF9oAxLK SGrYIEc+XE/odY9cufIHKNqH/CkVZlI3It65iNeu/FplcxvHLpJdwfka9aOq7Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1733266758; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/xkyLKWpUVdm6mfEZQQyS1yp6VoP0OWawZ5KoEQBw98=; b=c+FJ40EsursGL5c6ZmSdluV0Hszl6xqc8ugVFc+JqAg5TyAq0eyEU7V2yrjQR13ZoVbsYq CKD1ARZ8ojo9HDCA== To: Anup Patel Cc: Andrew Jones , iommu@lists.linux.dev, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, tjeznach@rivosinc.com, zong.li@sifive.com, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, atishp@atishpatra.org, alex.williamson@redhat.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu Subject: Re: [RFC PATCH 01/15] irqchip/riscv-imsic: Use hierarchy to reach irq_set_affinity In-Reply-To: <874j3ktrjv.ffs@tglx> References: <20241114161845.502027-17-ajones@ventanamicro.com> <20241114161845.502027-18-ajones@ventanamicro.com> <87mshcub2u.ffs@tglx> <874j3ktrjv.ffs@tglx> Date: Tue, 03 Dec 2024 23:59:17 +0100 Message-ID: <87ser4s796.ffs@tglx> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241203_145921_160963_0ECBC1DF X-CRM114-Status: GOOD ( 16.67 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gVHVlLCBEZWMgMDMgMjAyNCBhdCAyMTo1NSwgVGhvbWFzIEdsZWl4bmVyIHdyb3RlOgo+IE9u IFR1ZSwgRGVjIDAzIDIwMjQgYXQgMjI6MDcsIEFudXAgUGF0ZWwgd3JvdGU6Cj4+IE9uIFR1ZSwg RGVjIDMsIDIwMjQgYXQgNzoyM+KAr1BNIFRob21hcyBHbGVpeG5lciA8dGdseEBsaW51dHJvbml4 LmRlPiB3cm90ZToKPj4+IFNvcnJ5LCBJIG1pc3NlZCB0aGF0IHdoZW4gcmV2aWV3aW5nIHRoZSBv cmlnaW5hbCBJTVNJQyBNU0kgc3VwcG9ydC4KPj4+Cj4+PiBUaGUgd2hvbGUgSU1TSUMgTVNJIHN1 cHBvcnQgY2FuIGJlIG1vdmVkIG92ZXIgdG8gTVNJIExJQiB3aGljaCBtYWtlcyBhbGwKPj4+IG9m IHRoaXMgaW5kaXJlY3Rpb24gZ28gYXdheSBhbmQgeW91ciBpbnRlcm1lZGlhdGUgZG9tYWluIHdp bGwganVzdCBmaXQKPj4+IGluLgo+Pj4KPj4+IFVuY29tcGlsZWQgcGF0Y2ggYmVsb3cuIElmIHRo YXQgd29ya3MsIGl0IG5lZWRzIHRvIGJlIHNwbGl0IHVwIHByb3Blcmx5Lgo+Pj4KPj4+IE5vdGUs IHRoaXMgcmVtb3ZlcyB0aGUgc2V0dXAgb2YgdGhlIGlycV9yZXRyaWdnZXIgY2FsbGJhY2ssIGJ1 dCB0aGF0J3MKPj4+IGZpbmUgYmVjYXVzZSBvbiBoaWVyYXJjaGljYWwgZG9tYWlucyBpcnFfY2hp cF9yZXRyaWdnZXJfaGllcmFyY2h5KCkgaXMKPj4+IGludm9rZWQgYW55d2F5LiBTZWUgdHJ5X3Jl dHJpZ2dlcigpLgo+Pgo+PiBUaGUgSU1TSUMgZHJpdmVyIHdhcyBtZXJnZWQgb25lIGtlcm5lbCBy ZWxlYXNlIGJlZm9yZSBjb21tb24KPj4gTVNJIExJQiB3YXMgbWVyZ2VkLgo+Cj4gQWggaW5kZWVk Lgo+Cj4+IFdlIHNob3VsZCBkZWZpbml0ZWx5IHVwZGF0ZSB0aGUgSU1TSUMgZHJpdmVyIHRvIHVz ZSBNU0kgTElCLCBJIHdpbGwKPj4gdHJ5IHlvdXIgc3VnZ2VzdGVkIGNoYW5nZXMgKGJlbG93KSBh bmQgcG9zdCBhIHNlcGFyYXRlIHNlcmllcy4KPgo+IFBpY2sgdXAgdGhlIGRlbHRhIHBhdGNoIEkg Z2F2ZSBBbmRyZXcuLi4KCkFzIEkgd2FzIGxvb2tpbmcgYXQgc29tZXRoaW5nIGVsc2UgTVNJIHJl bGF0ZWQgSSBoYWQgYSBsb29rIGF0Cmltc2ljX2lycV9zZXRfYWZmaW5pdHkoKSBhZ2Fpbi4KCkl0 J3MgYWN0dWFsbHkgcmVxdWlyZWQgdG8gaGF2ZSB0aGUgbWVzc2FnZSB3cml0ZSBpbiB0aGF0IGZ1 bmN0aW9uIGFuZApub3QgYWZ0ZXJ3YXJkcyBhcyB5b3UgaW52b2tlIGltc2ljX3ZlY3Rvcl9tb3Zl KCkgZnJvbSB0aGF0IGZ1bmN0aW9uLgoKVGhhdCdzIG9idmlvdXNseSBub3QgdHJ1ZSBmb3IgdGhl IHJlbWFwIGNhc2UgYXMgdGhhdCB3aWxsIG5vdCBjaGFuZ2UgdGhlCm1lc3NhZ2UgYWRkcmVzcy9k YXRhIHBhaXIgYmVjYXVzZSB0aGUgcmVtYXAgdGFibGUgZW50cnkgaXMgaW1tdXRhYmxlIC0KYXQg bGVhc3QgSSBhc3N1bWUgc28gZm9yIG15IG1lbnRhbCBzYW5pdHkgc2FrZSA6KQoKQnV0IHRoYXQg YnJpbmdzIG1lIHRvIGEgcmVsYXRlZCBxdWVzdGlvbi4gSG93IGlzIHRoaXMgc3VwcG9zZWQgdG8g d29yawp3aXRoIG5vbi1hdG9taWMgbWVzc2FnZSB1cGRhdGVzPyBQQ0kvTVNJIGRvZXMgbm90IG5l Y2Vzc2FyaWx5IHByb3ZpZGUKbWFza2luZywgYW5kIHRoZSB3cml0ZSBvZiB0aGUgYWRkcmVzcy9k YXRhIHBhaXIgaXMgZG9uZSBpbiBiaXRzIGFuZApwaWVjZXMuIFNvIHlvdSBjYW4gZW5kIHVwIHdp dGggYW4gaW50ZXJtZWRpYXRlIHN0YXRlIHNlZW4gYnkgdGhlIGRldmljZQp3aGljaCBlbmRzIHVw IHNvbWV3aGVyZSBpbiBpbnRlcnJ1cHQgbmlydmFuYSBzcGFjZS4KClNlZSB0aGUgZGFuY2UgaW4g bXNpX3NldF9hZmZpbml0eSgpIGFuZCBjb21taXQgNmYxYTQ4OTFhNTkyCigieDg2L2FwaWMvbXNp OiBQbHVnIG5vbi1tYXNrYWJsZSBNU0kgYWZmaW5pdHkgcmFjZSIpIGZvciBmdXJ0aGVyCmV4cGxh bmF0aW9uLgoKVGhlIHdheSBob3cgdGhlIElNU0lDIGRyaXZlciB3b3JrcyBzZWVtcyB0byBiZSBw cmV0dHkgbXVjaCB0aGUgc2FtZSBhcwp0aGUgeDg2IEFQSUMgbWVzczoKCiAgICAgICAgQGFkZHJl c3MgaXMgdGhlIHBoeXNpY2FsIGFkZHJlc3Mgb2YgdGhlIHBlciBDUFUgTVNJIHRhcmdldAogICAg ICAgIGFkZHJlc3MgYW5kIEBkYXRhIGlzIHRoZSB2ZWN0b3IgSUQgb24gdGhhdCBDUFUuCgpTbyB0 aGUgbm9uLWF0b21pYyB1cGRhdGUgaW4gY2FzZSBvZiBub24tbWFza2FibGUgTVNJIHN1ZmZlcnMg ZnJvbSB0aGUKc2FtZSBwcm9ibGVtLiBJdCB3b3JrcyBtb3N0IG9mIHRoZSB0aW1lLCBidXQgaWYg aXQgZG9lc24ndCB5b3UgbWlnaHQKc3RhcmUgYXQgdGhlIG9jY2FzaW9uYWxseSBsb3N0IGludGVy cnVwdCBhbmQgdGhlIHN0YWxlIGRldmljZSBpbgpkaXNiZWxpZWYgZm9yIHF1aXRlIGEgd2hpbGUg OikKCkkgbWlnaHQgYmUgbWlzc2luZyBzb21ldGhpbmcgd2hpY2ggbWFnaWNhbGx5IHByZXZlbnQg dGhhdCB0aG91Z2ggOikKClRoYW5rcywKCiAgICAgICAgdGdseAoKX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtcmlzY3YgbWFpbGluZyBsaXN0Cmxp bnV4LXJpc2N2QGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcv bWFpbG1hbi9saXN0aW5mby9saW51eC1yaXNjdgo=