From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AAAFCC001DF for ; Fri, 20 Oct 2023 14:40:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:References :In-Reply-To:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=tLWhzp2kdb+1A7fo+KVMAOj4EZ3gIAQQVRfiafzFqhE=; b=efuqIh5LxfIal7 8phZveMK/5RcfGAcfSY81h+wyPy42GPfDX38XYEhsJH8chju/1jyUyxLbHLTtvdrmJHAcwDvimq+D xNoQ/akXJwTa0Nr2vfa6HWofno8uyJ/zPE/hVf0XLvSkvmswfFLprQ8/yQE2YLZQvEJLB1Sy42dFj nPRtt760JdH5s+AjL96RsMs4gAcp+mADrtfx1SJkhzukvt3glxLTfKXVhk9vhnlmUPC+e+Dw0S2f3 8G6ZAXY+edv3CY9WaAlUJGCqtW6SF7ZGxuWHbNkrvynvqwvASLvceuoGEklFljLea/+7U4+p0WO3o Wqd6WTdt5nFeAuCaEpNQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qtqg3-002W9M-0R; Fri, 20 Oct 2023 14:40:39 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qtqfz-002W8s-2S for linux-riscv@lists.infradead.org; Fri, 20 Oct 2023 14:40:37 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 15AB761F28; Fri, 20 Oct 2023 14:40:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 17F68C433C7; Fri, 20 Oct 2023 14:40:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1697812834; bh=u/U9nnqgPeREKVvmxW1xeTootMKuuSYkI2HYOXzF4ME=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=VGAFqtx232BDmzSrStY1SaghPb5zcaRYo3f/dxKRFyLl3llsKJcdYXH9wr991Xq+/ nEq3pvV+cStQODCp6NgWsebTukdU7HNoTQ7sJ4z852R6HH2QCIK4pQqFLvIrx0Zrrn Rbh25+A6On9EuNIutk3155soPgu1uh/CQ3WdHFmfxmJejPK7F44TE71SZIjQLGJirS j5qJSEhRMiZefZ1KHS3sJ/hCPPje9nkjLEKx41TTXjVA0V6j/bw1S6VSKnCbidTD/i mg80tpERyDcMBEWRNmYyP0EDZbFgVXkKkbATNeBJ+AGV0gBrF6yrGSkvtBR/8VoOc3 N+EYyHDE+3fXQ== From: =?utf-8?B?QmrDtnJuIFTDtnBlbA==?= To: Anup Patel Subject: Re: [PATCH v10 00/15] Linux RISC-V AIA Support In-Reply-To: References: <20231003044403.1974628-1-apatel@ventanamicro.com> <87o7gu7mo9.fsf@all.your.base.are.belong.to.us> <87h6mlbryy.fsf@all.your.base.are.belong.to.us> Date: Fri, 20 Oct 2023 16:40:31 +0200 Message-ID: <87o7gtpdb4.fsf@all.your.base.are.belong.to.us> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231020_074035_894562_9698C9C0 X-CRM114-Status: GOOD ( 38.79 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, Conor Dooley , Saravana Kannan , Marc Zyngier , Anup Patel , Atish Patra , linux-kernel@vger.kernel.org, Rob Herring , Palmer Dabbelt , Krzysztof Kozlowski , Paul Walmsley , Thomas Gleixner , Frank Rowand , linux-riscv@lists.infradead.org, Andrew Jones Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org QW51cCBQYXRlbCA8YXBhdGVsQHZlbnRhbmFtaWNyby5jb20+IHdyaXRlczoKCj4gT24gRnJpLCBP Y3QgMjAsIDIwMjMgYXQgMjoxN+KAr1BNIEJqw7ZybiBUw7ZwZWwgPGJqb3JuQGtlcm5lbC5vcmc+ IHdyb3RlOgo+Pgo+PiBUaGFua3MgZm9yIHRoZSBxdWljayByZXBseSEKPj4KPj4gQW51cCBQYXRl bCA8YXBhdGVsQHZlbnRhbmFtaWNyby5jb20+IHdyaXRlczoKPj4KPj4gPiBPbiBUaHUsIE9jdCAx OSwgMjAyMyBhdCA3OjEz4oCvUE0gQmrDtnJuIFTDtnBlbCA8Ympvcm5Aa2VybmVsLm9yZz4gd3Jv dGU6Cj4+ID4+Cj4+ID4+IEhpIEFudXAsCj4+ID4+Cj4+ID4+IEFudXAgUGF0ZWwgPGFwYXRlbEB2 ZW50YW5hbWljcm8uY29tPiB3cml0ZXM6Cj4+ID4+Cj4+ID4+ID4gVGhlIFJJU0MtViBBSUEgc3Bl Y2lmaWNhdGlvbiBpcyByYXRpZmllZCBhcy1wZXIgdGhlIFJJU0MtViBpbnRlcm5hdGlvbmFsCj4+ ID4+ID4gcHJvY2Vzcy4gVGhlIGxhdGVzdCByYXRpZmllZCBBSUEgc3BlY2lmY2F0aW9uIGNhbiBi ZSBmb3VuZCBhdDoKPj4gPj4gPiBodHRwczovL2dpdGh1Yi5jb20vcmlzY3YvcmlzY3YtYWlhL3Jl bGVhc2VzL2Rvd25sb2FkLzEuMC9yaXNjdi1pbnRlcnJ1cHRzLTEuMC5wZGYKPj4gPj4gPgo+PiA+ PiA+IEF0IGEgaGlnaC1sZXZlbCwgdGhlIEFJQSBzcGVjaWZpY2F0aW9uIGFkZHMgdGhyZWUgdGhp bmdzOgo+PiA+PiA+IDEpIEFJQSBDU1JzCj4+ID4+ID4gICAgLSBJbXByb3ZlZCBsb2NhbCBpbnRl cnJ1cHQgc3VwcG9ydAo+PiA+PiA+IDIpIEluY29taW5nIE1lc3NhZ2UgU2lnbmFsZWQgSW50ZXJy dXB0IENvbnRyb2xsZXIgKElNU0lDKQo+PiA+PiA+ICAgIC0gUGVyLUhBUlQgTVNJIGNvbnRyb2xs ZXIKPj4gPj4gPiAgICAtIFN1cHBvcnQgTVNJIHZpcnR1YWxpemF0aW9uCj4+ID4+ID4gICAgLSBT dXBwb3J0IElQSSBhbG9uZyB3aXRoIHZpcnR1YWxpemF0aW9uCj4+ID4+ID4gMykgQWR2YW5jZWQg UGxhdGZvcm0tTGV2ZWwgSW50ZXJydXB0IENvbnRyb2xsZXIgKEFQTElDKQo+PiA+PiA+ICAgIC0g V2lyZWQgaW50ZXJydXB0IGNvbnRyb2xsZXIKPj4gPj4gPiAgICAtIEluIE1TSS1tb2RlLCBjb252 ZXJ0cyB3aXJlZCBpbnRlcnJ1cHQgaW50byBNU0lzIChpLmUuIE1TSSBnZW5lcmF0b3IpCj4+ID4+ ID4gICAgLSBJbiBEaXJlY3QtbW9kZSwgaW5qZWN0cyBleHRlcm5hbCBpbnRlcnJ1cHRzIGRpcmVj dGx5IGludG8gSEFSVHMKPj4gPj4KPj4gPj4gVGhhbmtzIGZvciB3b3JraW5nIG9uIHRoZSBBSUEg c3VwcG9ydCEgSSBoYWQgYSBsb29rIGF0IHRoZSBzZXJpZXMsIGFuZAo+PiA+PiBoYXZlIHNvbWUg Y29uY2VybnMgYWJvdXQgaW50ZXJydXB0IElEIGFic3RyYWN0aW9uLgo+PiA+Pgo+PiA+PiBBIGJp dCBvZiBiYWNrZ3JvdW5kLCBmb3IgcmVhZGVycyBub3QgZmFtaWxpYXIgd2l0aCB0aGUgQUlBIGRl dGFpbHMuCj4+ID4+Cj4+ID4+IElNU0lDIGFsbG93cyBmb3IgMjA0NyB1bmlxdWUgTVNJICgibXNp LWlycSIpIHNvdXJjZXMgcGVyIGhhcnQsIGFuZAo+PiA+PiBlYWNoIE1TSSBpcyBkZWRpY2F0ZWQg dG8gYSBjZXJ0YWluIGhhcnQuIFRoZSBzZXJpZXMgdGFrZXMgdGhlIGFwcHJvYWNoCj4+ID4+IHRv IHNheSB0aGF0IHRoZXJlIGFyZSwgZS5nLiwgMjA0NyBpbnRlcnJ1cHRzICgibG54LWlycSIpIGds b2JhbGx5Lgo+PiA+PiBFYWNoIGxueC1pcnEgY29uc2lzdHMgb2YgI2hhcnRzICogbXNpLWlycSAt LSBhIHNsaWNlIC0tIGFuZCBpbiB0aGUKPj4gPj4gc2xpY2Ugb25seSAqb25lKiBtc2ktaXJxIGlz IGFjdXRhbGx5IHVzZWQuCj4+ID4+Cj4+ID4+IFRoaXMgc2NoZW1lIG1ha2VzIGFmZmluaXR5IGNo YW5nZXMgbW9yZSByb2J1c3QsIGJlY2F1c2UgdGhlIGludGVycnVwdAo+PiA+PiBzb3VyY2VzIG9u ICJvdGhlciIgaGFydHMgYXJlIHByZS1hbGxvY2F0ZWQuIE9uIHRoZSBvdGhlciBoYW5kIGl0Cj4+ ID4+IHJlcXVpcmVzIHRvIHByb3BhZ2F0ZSBpcnEgbWFza2luZyB0byBvdGhlciBoYXJ0cyB2aWEg SVBJcyAodGhpcyBpcwo+PiA+PiBtb3N0bHkgZG9uZSB1cCBzZXR1cC90ZWFyIGRvd24pLiBJdCdz IGFsc28gd2FzdGVmdWwsIGJlY2F1c2UgbXNpLWlycXMKPj4gPj4gYXJlIGhvZ2dlZCwgYW5kIGNh bm5vdCBiZSB1c2VkLgo+PiA+Pgo+PiA+PiBDb250ZW1wb3Jhcnkgc3RvcmFnZS9uZXR3b3JraW5n IGRyaXZlcnMgdXN1YWxseSB1c2VzIHF1ZXVlcyBwZXIgY29yZQo+PiA+PiAob3IgYSBzdWItc2V0 IG9mIGNvcmVzKS4gVGhlIGN1cnJlbnQgc2NoZW1lIHdhc3RlcyBhIGxvdCBvZiBtc2ktaXJxcy4K Pj4gPj4gSWYgd2UgaW5zdGVhZCB1c2VkIGEgc2NoZW1lIHdoZXJlICJtc2ktaXJxID09IGxueC1p cnEiLCBpbnN0ZWFkIG9mCj4+ID4+ICJsbnEtaXJxID0ge2hhcnQgMDttc2ktaXJxIHggLCAuLi4g aGFydCBOO21zaS1pcnEgeH0iLCB0aGVyZSB3b3VsZCBiZQo+PiA+PiBhIGxvdCBNU0lzIGZvciBv dGhlciB1c2Vycy4gMS0xIHZzIDEtTi4gRS5nLiwgaWYgYSBzdG9yYWdlIGRldmljZQo+PiA+PiB3 b3VsZCBsaWtlIHRvIHVzZSA1IHF1ZXVlcyAoNSBjb3Jlcykgb24gYSAxMjggY29yZSBzeXN0ZW0s IHRoZSBjdXJyZW50Cj4+ID4+IHNjaGVtZSB3b3VsZCBjb25zdW1lIDUgKiAxMjggTVNJcywgaW5z dGVhZCBvZiBqdXN0IDUuCj4+ID4+Cj4+ID4+IE9uIHRoZSBwbHVzIHNpZGU6Cj4+ID4+ICogQ2hh bmdpbmcgaW50ZXJydXB0cyBhZmZpbml0eSB3aWxsIG5ldmVyIGZhaWwsIGJlY2F1c2UgdGhlIGlu dGVycnVwdHMKPj4gPj4gICBvbiBlYWNoIGhhcnQgaXMgcHJlLWFsbG9jYXRlZC4KPj4gPj4KPj4g Pj4gT24gdGhlIG5lZ2F0aXZlIHNpZGU6Cj4+ID4+ICogV2FzdGVmdWwgaW50ZXJydXB0IHVzYWdl LCBhbmQgYSBzeXN0ZW0gY2FuIHBvdGllbnRpYWxseSAicnVuIG91dCIgb2YKPj4gPj4gICBpbnRl cnJ1cHRzLiBFc3BlY2lhbGx5IGZvciBtYW55IGNvcmUgc3lzdGVtcy4KPj4gPj4gKiBJbnRlcnJ1 cHQgbWFza2luZyBuZWVkIHRvIHByb2FnYXRlIHRvIGhhcnRzIHZpYSBJUElzICh0aGVyZSdzIG5v Cj4+ID4+ICAgYnJvYWRjYXN0IGNzciBpbiBJTVNJQyksIGFuZCBhIG1vcmUgY29tcGxleCBsb2Nr aW5nIHNjaGVtZSBJTVNJQwo+PiA+Pgo+PiA+PiBTdW1tYXJ5Ogo+PiA+PiBUaGUgY3VycmVudCBz ZXJpZXMgY2FwcyB0aGUgbnVtYmVyIG9mIGdsb2JhbCBpbnRlcnJ1cHRzIHRvIG1heGltdW0KPj4g Pj4gMjA0NyBNU0lzIGZvciBhbGwgY29yZXMgKHdob2xlIHN5c3RlbSkuIEEgYmV0dGVyIHNjaGVt ZSwgSU1PLCB3b3VsZCBiZQo+PiA+PiB0byBleHBvc2UgMjA0NyAqICNoYXJ0cyB1bmlxdWUgTVNJ cy4KPj4gPj4KPj4gPj4gSSB0aGluayB0aGlzIGNvdWxkIHNpbXBsaWZ5L3JlbW92ZSg/KSB0aGUg bG9ja2luZyBhcyB3ZWxsLgo+PiA+Cj4+ID4gRXhwb3NpbmcgMjA0NyAqICNoYXJ0cyB1bmlxdWUg TVNJcyBoYXMgbXVsdGlwbGUgaXNzdWVzOgo+PiA+IDEpIFRoZSBpcnFfc2V0X2FmZmluaXR5KCkg ZG9lcyBub3Qgd29yayBmb3IgTVNJcyBiZWNhdXNlIGVhY2gKPj4gPiAgICAgIElSUSBpcyBub3Qg dGllZCB0byBhIHBhcnRpY3VsYXIgSEFSVC4gVGhpcyBtZWFucyB3ZSBjYW4ndAo+PiA+ICAgICAg YmFsYW5jZSB0aGUgSVJRIHByb2Nlc3NpbmcgbG9hZCBhbW9uZyBIQVJUcy4KPj4KPj4gWWVzLCB5 b3UgY2FuIGJhbGFuY2UuIEluIHlvdXIgY29kZSwgZWFjaCAqYWN0aXZlKiBNU0kgaXMgc3RpbGwK Pj4gYm91bmQvYWN0aXZlIHRvIGEgc3BlY2lmaWMgaGFyZCB0b2dldGhlciB3aXRoIHRoZSBhZmZp bml0eSBtYXNrLiBJbiBhbgo+PiAxLTEgbW9kZWwgeW91IHdvdWxkIHN0aWxsIG5lZWQgdG8gdHJh Y2sgdGhlIGFmZmluaXR5IG1hc2ssIGJ1dCB0aGUKPj4gaXJxX3NldF9hZmZpbml0eSgpIHdvdWxk IGJlIGRpZmZlcmVudC4gSXQgd291bGQgdHJ5IHRvIGFsbG9jYXRlIGEgbmV3Cj4+IE1TSSBmcm9t IHRoZSB0YXJnZXQgQ1BVLCBhbmQgdGhlbiBzd2l0Y2ggdG8gaGF2aW5nIHRoYXQgTVNJIGFjdGl2 ZS4KPj4KPj4gVGhhdCdzIHdoYXQgeDg2IGRvZXMgQUZBSVUsIHdoaWNoIGlzIGFsc28gY29uc3Ry YWluZWQgYnkgdGhlICMgb2YKPj4gYXZhaWxhYmxlIE1TSXMuCj4+Cj4+IFRoZSBkb3duc2lkZSwg YXMgSSBwb2ludGVkIG91dCwgaXMgdGhhdCB0aGUgc2V0IGFmZmluaXR5IGFjdGlvbiBjYW4KPj4g ZmFpbCBmb3IgYSBjZXJ0YWluIHRhcmdldCBDUFUuCj4KPiBZZXMsIGlycV9zZXRfYWZmaW5pdHko KSBjYW4gZmFpbCBmb3IgdGhlIHN1Z2dlc3RlZCBhcHByb2FjaCBwbHVzIGZvcgo+IFJJU0MtViBB SUEsIG9uZSBIQVJUIGRvZXMgbm90IGhhdmUgYWNjZXNzIHRvIG90aGVyIEhBUlRzCj4gTVNJIGVu YWJsZS9kaXNhYmxlIGJpdHMgc28gdGhlIGFwcHJvYWNoIHdpbGwgYWxzbyBpbnZvbHZlIElQSS4K CkNvcnJlY3QsIGJ1dCB0aGUgY3VycmVudCBzZXJpZXMgZG9lcyBhIGJyb2FkY2FzdCB0byBhbGwg Y29yZXMsIHdoZXJlIHRoZQoxLTEgYXBwcm9hY2ggaXMgYXQgbW9zdCBhbiBJUEkgdG8gYSBzaW5n bGUgY29yZS4KCjEyOCtjIG1hY2hpbmVzIGFyZSBnZXR0aW5nIG1vcmUgY29tbW9uLCBhbmQgeW91 IGhhdmUgZGV2aWNlcyB0aGF0IHlvdQpicmluZyB1cC9kb3duIG9uIGEgcGVyLWNvcmUgYmFzaXMu IEJyb2FkY2FzdGluZyBJUElzIHRvIGFsbCBjb3Jlcywgd2hlbgpkZWFsaW5nIHdpdGggYSBwZXIt Y29yZSBhY3Rpdml0eSBpcyBhIHByZXR0eSBub2lzeSBuZWlnaGJvci4KClRoaXMgY291bGQgYmUg Zml4ZWQgaW4gdGhlIGV4aXN0aW5nIDEtbiBhcHByb2FjaCwgYnkgbm90IHJlcXVpcmUgdG8gc3lu Ywp0aGUgY29yZXMgdGhhdCBhcmUgbm90IGhhbmRsaW5nIHRoZSBNU0kgaW4gcXVlc3Rpb24uICJM YXp5IGRpc2FibGUiCgo+PiA+IDIpIEFsbCB3aXJlZCBJUlFzIGZvciBBUExJQyBNU0ktbW9kZSB3 aWxsIGFsc28gdGFyZ2V0IGEKPj4gPiAgICAgZml4ZWQgSEFSVCBoZW5jZSBpcnFfc2V0X2FmZmlu aXR5KCkgd29uJ3Qgd29yayBmb3Igd2lyZWQKPj4gPiAgICAgSVJRcyBhcyB3ZWxsLgo+Pgo+PiBJ J20gbm90IGZvbGxvd2luZyBoZXJlLiBXaHkgd291bGQgQVBMSUMgcHV0IGEgY29uc3RyYWludCBo ZXJlPyBJIGhhZCBhCj4+IGxvb2sgYXQgdGhlIHNwZWNzLCBhbmQgSSBkaWRuJ3Qgc2VlIGFueXRo aW5nIHN1cHBvcnRpbmcgdGhlIGN1cnJlbnQKPj4gc2NoZW1lIGV4cGxpY2l0bHkuCj4KPiBMZXRz IHNheSB0aGUgbnVtYmVyIG9mIEFQTElDIHdpcmVkIGludGVycnVwdHMgIGFyZSBncmVhdGVyIHRo YW4gdGhlCj4gbnVtYmVyIG9mIHBlci1DUFUgSU1TSUMgSURzLiBJbiB0aGlzIGNhc2UsIGlmIGFs bCB3aXJlZCBpbnRlcnJ1cHRzIGFyZQo+IG1vdmVkIHRvIGEgcGFydGljdWxhciBDUFUgdGhlbiBp cnFfc2V0X2FmZmluaXR5KCkgd2lsbCBmYWlsIGZvciBzb21lIG9mCj4gdGhlIHdpcmVkIGludGVy cnVwdHMuCgpSaWdodCwgaXQncyB0aGUgY2FzZSBvZiAiZnVsbCByZW1vdGUgQ1BVIiBhZ2Fpbi4g VGhhbmtzIGZvciBjbGVhcmluZwp0aGF0IHVwLgoKPj4gPiBUaGUgaWRlYSBvZiB0cmVhdGluZyBw ZXItSEFSVCBNU0lzIGFzIHNlcGFyYXRlIElSUXMgaGFzCj4+ID4gYmVlbiBkaXNjdXNzZWQgaW4g dGhlIHBhc3QuCj4+Cj4+IEFoYSEgSSB0cmllZCB0byBsb29rIGZvciBpdCBpbiBsb3JlLCBidXQg ZGlkbid0IGZpbmQgYW55LiBDb3VsZCB5b3UKPj4gcG9pbnQgbWUgdG8gdGhvc2UgZGlzY3Vzc2lv bnM/Cj4KPiBUaGlzIHdhcyBkb25lIDIgeWVhcnMgYmFjayBpbiB0aGUgQUlBIFRHIG1lZXRpbmcg d2hlbiB3ZSB3ZXJlCj4gZG9pbmcgdGhlIFBvQyBmb3IgQUlBIHNwZWMuCgpBaCwgdG9vIGJhZC4g VGhhbmtzIHJlZ2FyZGxlc3MuCgo+PiBNeSBjb25jZXJuIGlzIGludGVycnVwdHMgYmVjb21lIGEg c2NhcmNlIHJlc291cmNlIHdpdGggdGhpcwo+PiBpbXBsZW1lbnRhdGlvbiwgYnV0IG1heWJlIG15 IHZpZXcgaXMgaW5jb3JyZWN0LiBJJ3ZlIHNlZW4gYmFyZS1tZXRhbAo+PiB4ODYgc3lzdGVtcyAo bm8gVk1zKSB3aXRoIH4yMDAgY29yZXMsIGFuZCB+MjAwMCBpbnRlcnJ1cHRzLCBidXQgbWF5YmUK Pj4gdGhhdCBpcyBjb25zaWRlcmVkICJhIGxvdCBvZiBpbnRlcnJ1cHRzIi4KPj4KPj4gQXMgbG9u ZyBhcyB3ZSBkb24ndCBnZXQgaW50byBzY2VuYXJpb3Mgd2hlcmUgd2UncmUgcnVubmluZyBvdXQg b2YKPj4gaW50ZXJydXB0cywgZHVlIHRvIHRoZSBzb2Z0d2FyZSBkZXNpZ24uCj4+Cj4KPiBUaGUg Y3VycmVudCBhcHByb2FjaCBpcyBzaW1wbGVyIGFuZCBlbnN1cmVzIGlycV9zZXRfYWZmaW5pdHkK PiBhbHdheXMgd29ya3MuIFRoZSBsaW1pdCBvZiBtYXggMjA0NyBJRHMgaXMgc3VmZmljaWVudCBm b3IgbWFueQo+IHN5c3RlbXMgKGlmIG5vdCBhbGwpLgoKTGV0IG1lIGdpdmUgeW91IGFub3RoZXIg dmlldy4gT24gYSAxMjhjIHN5c3RlbSBlYWNoIGNvcmUgaGFzIH4xNiB1bmlxdWUKaW50ZXJydXB0 cyBmb3IgZGlzcG9zYWwuIEUuZy4gdGhlIEludGVsIEU4MDAgTklDIGhhcyBtb3JlIHRoYW4gMjA0 OApuZXR3b3JrIHF1ZXVlIHBhaXJzIGZvciBlYWNoIFBGLgoKPiBXaGVuIHdlIGVuY291bnRlciBh IHN5c3RlbSByZXF1aXJpbmcgYSBsYXJnZSBudW1iZXIgb2YgTVNJcywKPiB3ZSBjYW4gZWl0aGVy Ogo+IDEpIEV4dGVuZCB0aGUgQUlBIHNwZWMgdG8gc3VwcG9ydCBncmVhdGVyIHRoYW4gMjA0NyBJ RHMKPiAyKSBSZS10aGluayB0aGUgYXBwcm9hY2ggaW4gdGhlIElNU0lDIGRyaXZlcgo+Cj4gVGhl IGNob2ljZSBiZXR3ZWVuICMxIGFuZCAjMiBhYm92ZSBkZXBlbmRzIG9uIHRoZQo+IGd1YXJhbnRl ZXMgd2Ugd2FudCBmb3IgaXJxX3NldF9hZmZpbml0eSgpLgoKVGhlIGlycV9zZXRfYWZmaW5pdHko KSBiZWhhdmlvciBpcyBiZXR0ZXIgd2l0aCB0aGlzIHNlcmllcywgYnV0IEkgdGhpbmsKdGhlIG90 aGVyIGRvd25zaWRlczogbnVtYmVyIG9mIGF2YWlsYWJsZSBpbnRlcnJ1cHQgc291cmNlcywgYW5k IElQSQpicm9hZGNhc3QgYXJlIHdvcnNlLgoKCkJqw7ZybgoKX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtcmlzY3YgbWFpbGluZyBsaXN0CmxpbnV4 LXJpc2N2QGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFp bG1hbi9saXN0aW5mby9saW51eC1yaXNjdgo= From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 469161DA20 for ; Fri, 20 Oct 2023 14:40:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VGAFqtx2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 17F68C433C7; Fri, 20 Oct 2023 14:40:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1697812834; bh=u/U9nnqgPeREKVvmxW1xeTootMKuuSYkI2HYOXzF4ME=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=VGAFqtx232BDmzSrStY1SaghPb5zcaRYo3f/dxKRFyLl3llsKJcdYXH9wr991Xq+/ nEq3pvV+cStQODCp6NgWsebTukdU7HNoTQ7sJ4z852R6HH2QCIK4pQqFLvIrx0Zrrn Rbh25+A6On9EuNIutk3155soPgu1uh/CQ3WdHFmfxmJejPK7F44TE71SZIjQLGJirS j5qJSEhRMiZefZ1KHS3sJ/hCPPje9nkjLEKx41TTXjVA0V6j/bw1S6VSKnCbidTD/i mg80tpERyDcMBEWRNmYyP0EDZbFgVXkKkbATNeBJ+AGV0gBrF6yrGSkvtBR/8VoOc3 N+EYyHDE+3fXQ== From: =?utf-8?B?QmrDtnJuIFTDtnBlbA==?= To: Anup Patel Cc: Palmer Dabbelt , Paul Walmsley , Thomas Gleixner , Marc Zyngier , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Conor Dooley , Atish Patra , Andrew Jones , Sunil V L , Saravana Kannan , Anup Patel , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org Subject: Re: [PATCH v10 00/15] Linux RISC-V AIA Support In-Reply-To: References: <20231003044403.1974628-1-apatel@ventanamicro.com> <87o7gu7mo9.fsf@all.your.base.are.belong.to.us> <87h6mlbryy.fsf@all.your.base.are.belong.to.us> Date: Fri, 20 Oct 2023 16:40:31 +0200 Message-ID: <87o7gtpdb4.fsf@all.your.base.are.belong.to.us> Precedence: bulk X-Mailing-List: devicetree@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Anup Patel writes: > On Fri, Oct 20, 2023 at 2:17=E2=80=AFPM Bj=C3=B6rn T=C3=B6pel wrote: >> >> Thanks for the quick reply! >> >> Anup Patel writes: >> >> > On Thu, Oct 19, 2023 at 7:13=E2=80=AFPM Bj=C3=B6rn T=C3=B6pel wrote: >> >> >> >> Hi Anup, >> >> >> >> Anup Patel writes: >> >> >> >> > The RISC-V AIA specification is ratified as-per the RISC-V internat= ional >> >> > process. The latest ratified AIA specifcation can be found at: >> >> > https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-inte= rrupts-1.0.pdf >> >> > >> >> > At a high-level, the AIA specification adds three things: >> >> > 1) AIA CSRs >> >> > - Improved local interrupt support >> >> > 2) Incoming Message Signaled Interrupt Controller (IMSIC) >> >> > - Per-HART MSI controller >> >> > - Support MSI virtualization >> >> > - Support IPI along with virtualization >> >> > 3) Advanced Platform-Level Interrupt Controller (APLIC) >> >> > - Wired interrupt controller >> >> > - In MSI-mode, converts wired interrupt into MSIs (i.e. MSI gene= rator) >> >> > - In Direct-mode, injects external interrupts directly into HARTs >> >> >> >> Thanks for working on the AIA support! I had a look at the series, and >> >> have some concerns about interrupt ID abstraction. >> >> >> >> A bit of background, for readers not familiar with the AIA details. >> >> >> >> IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and >> >> each MSI is dedicated to a certain hart. The series takes the approach >> >> to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. >> >> Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the >> >> slice only *one* msi-irq is acutally used. >> >> >> >> This scheme makes affinity changes more robust, because the interrupt >> >> sources on "other" harts are pre-allocated. On the other hand it >> >> requires to propagate irq masking to other harts via IPIs (this is >> >> mostly done up setup/tear down). It's also wasteful, because msi-irqs >> >> are hogged, and cannot be used. >> >> >> >> Contemporary storage/networking drivers usually uses queues per core >> >> (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. >> >> If we instead used a scheme where "msi-irq =3D=3D lnx-irq", instead of >> >> "lnq-irq =3D {hart 0;msi-irq x , ... hart N;msi-irq x}", there would = be >> >> a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device >> >> would like to use 5 queues (5 cores) on a 128 core system, the current >> >> scheme would consume 5 * 128 MSIs, instead of just 5. >> >> >> >> On the plus side: >> >> * Changing interrupts affinity will never fail, because the interrupts >> >> on each hart is pre-allocated. >> >> >> >> On the negative side: >> >> * Wasteful interrupt usage, and a system can potientially "run out" of >> >> interrupts. Especially for many core systems. >> >> * Interrupt masking need to proagate to harts via IPIs (there's no >> >> broadcast csr in IMSIC), and a more complex locking scheme IMSIC >> >> >> >> Summary: >> >> The current series caps the number of global interrupts to maximum >> >> 2047 MSIs for all cores (whole system). A better scheme, IMO, would be >> >> to expose 2047 * #harts unique MSIs. >> >> >> >> I think this could simplify/remove(?) the locking as well. >> > >> > Exposing 2047 * #harts unique MSIs has multiple issues: >> > 1) The irq_set_affinity() does not work for MSIs because each >> > IRQ is not tied to a particular HART. This means we can't >> > balance the IRQ processing load among HARTs. >> >> Yes, you can balance. In your code, each *active* MSI is still >> bound/active to a specific hard together with the affinity mask. In an >> 1-1 model you would still need to track the affinity mask, but the >> irq_set_affinity() would be different. It would try to allocate a new >> MSI from the target CPU, and then switch to having that MSI active. >> >> That's what x86 does AFAIU, which is also constrained by the # of >> available MSIs. >> >> The downside, as I pointed out, is that the set affinity action can >> fail for a certain target CPU. > > Yes, irq_set_affinity() can fail for the suggested approach plus for > RISC-V AIA, one HART does not have access to other HARTs > MSI enable/disable bits so the approach will also involve IPI. Correct, but the current series does a broadcast to all cores, where the 1-1 approach is at most an IPI to a single core. 128+c machines are getting more common, and you have devices that you bring up/down on a per-core basis. Broadcasting IPIs to all cores, when dealing with a per-core activity is a pretty noisy neighbor. This could be fixed in the existing 1-n approach, by not require to sync the cores that are not handling the MSI in question. "Lazy disable" >> > 2) All wired IRQs for APLIC MSI-mode will also target a >> > fixed HART hence irq_set_affinity() won't work for wired >> > IRQs as well. >> >> I'm not following here. Why would APLIC put a constraint here? I had a >> look at the specs, and I didn't see anything supporting the current >> scheme explicitly. > > Lets say the number of APLIC wired interrupts are greater than the > number of per-CPU IMSIC IDs. In this case, if all wired interrupts are > moved to a particular CPU then irq_set_affinity() will fail for some of > the wired interrupts. Right, it's the case of "full remote CPU" again. Thanks for clearing that up. >> > The idea of treating per-HART MSIs as separate IRQs has >> > been discussed in the past. >> >> Aha! I tried to look for it in lore, but didn't find any. Could you >> point me to those discussions? > > This was done 2 years back in the AIA TG meeting when we were > doing the PoC for AIA spec. Ah, too bad. Thanks regardless. >> My concern is interrupts become a scarce resource with this >> implementation, but maybe my view is incorrect. I've seen bare-metal >> x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe >> that is considered "a lot of interrupts". >> >> As long as we don't get into scenarios where we're running out of >> interrupts, due to the software design. >> > > The current approach is simpler and ensures irq_set_affinity > always works. The limit of max 2047 IDs is sufficient for many > systems (if not all). Let me give you another view. On a 128c system each core has ~16 unique interrupts for disposal. E.g. the Intel E800 NIC has more than 2048 network queue pairs for each PF. > When we encounter a system requiring a large number of MSIs, > we can either: > 1) Extend the AIA spec to support greater than 2047 IDs > 2) Re-think the approach in the IMSIC driver > > The choice between #1 and #2 above depends on the > guarantees we want for irq_set_affinity(). The irq_set_affinity() behavior is better with this series, but I think the other downsides: number of available interrupt sources, and IPI broadcast are worse. Bj=C3=B6rn