From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D6482C54EE9 for ; Thu, 8 Sep 2022 14:20:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=j71XLyH93HdzXJjNgLKeHsYWTCOtPFaSDjYwmgZlPoM=; b=gnoFCiUsmWwlBc 4J15VGGwAS1aIvo2/FNXJufbf0hVFpSs8AMup05TfxaXcikalir8R6ye70KRsJj2r5gR0ZYKaOV8N XjSLhP2yIOilxMvc0nuhJgHm8BSMSTQG9/XdfLH1C2BCljlI0vr0CGzJN3ERaCQ8ZMziDdBH/b1cE s2h6vL1GPNRAsx3eoCZ0o6AMiopZgibmqjqrym46PNmevaITai7FI54id5SQfLwS77AloSoic2INm /1jFznMGyJsa43TqLZsVnwstkjsRJIL8ANGIAVv7qhXyfO+w1yiOnFRjBrrBx1kM1QjiauSwpr4LX TUerLj+HsLFLgyvU4N+w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oWIMy-004Y26-DW; Thu, 08 Sep 2022 14:19:04 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oWIMu-004Xxi-H5 for linux-arm-kernel@lists.infradead.org; Thu, 08 Sep 2022 14:19:02 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 69CD1B820FE; Thu, 8 Sep 2022 14:18:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C9E41C433C1; Thu, 8 Sep 2022 14:18:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1662646737; bh=fYsnf4LrrFtf7OEtEe7aCauVRkUDQxTiLN21Bt65o30=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=LE9p6CC6sFqVUCepSNaAT6Ni1pdcdb70MLWNi3jpPtU1ZKJ3j5cZMwIaebxCeUiuS bzYxKqHW7AtFyepnitIrl1MOTGt3DKxG5isW6VJFVL2z4tNCShuGNldtYxZBnb5C5O TjbeKyasNhRP0c2gJvWxMTgpS66gIn5skGjjmCwEGPGbuDl33RXQHWN31c07H7P0fl yKp8qCPcoemnRXBkYeOSqiAGR0jXA1qHNQbBHagEh5kPlRnzUvLPD7xuTN/BHoY8za 6QWQxCuC8kXmQxUgIM3MMeYrjJahE2PbuBNLcLL816bZdffRt2diuLPxZlL4sp8scH 9J6IF1GLyz4wg== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1oWIMo-008wGz-Ii; Thu, 08 Sep 2022 15:18:54 +0100 Date: Thu, 08 Sep 2022 15:18:53 +0100 Message-ID: <87bkrqueya.wl-maz@kernel.org> From: Marc Zyngier To: "Radovanovic, Aleksandar" Cc: Jason Gunthorpe , "Gupta, Nipun" , "robh+dt@kernel.org" , "krzysztof.kozlowski+dt@linaro.org" , "gregkh@linuxfoundation.org" , "rafael@kernel.org" , "eric.auger@redhat.com" , "alex.williamson@redhat.com" , "cohuck@redhat.com" , "Gupta, Puneet (DCG-ENG)" , "song.bao.hua@hisilicon.com" , "mchehab+huawei@kernel.org" , "f.fainelli@gmail.com" , "jeffrey.l.hugo@gmail.com" , "saravanak@google.com" , "Michael.Srba@seznam.cz" , "mani@kernel.org" , "yishaih@nvidia.com" , "robin.murphy@arm.com" , "will@kernel.org" , "joro@8bytes.org" , "masahiroy@kernel.org" , "ndesaulniers@google.com" , "linux-arm-kernel@lists.infradead.org" , "linux-kbuild@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "devicetree@vger.kernel.org" , "kvm@vger.kernel.org" , "okaya@kernel.org" , "Anand, Harpreet" , "Agarwal, Nikhil" , "Simek,\ Michal" , "git (AMD-Xilinx)" Subject: Re: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its domain as parent In-Reply-To: References: <20220803122655.100254-1-nipun.gupta@amd.com> <20220906134801.4079497-1-nipun.gupta@amd.com> <20220906134801.4079497-5-nipun.gupta@amd.com> <87leqvv3g7.wl-maz@kernel.org> <87illzuzyw.wl-maz@kernel.org> <87edwmuw4f.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: aleksandar.radovanovic@amd.com, jgg@nvidia.com, Nipun.Gupta@amd.com, robh+dt@kernel.org, krzysztof.kozlowski+dt@linaro.org, gregkh@linuxfoundation.org, rafael@kernel.org, eric.auger@redhat.com, alex.williamson@redhat.com, cohuck@redhat.com, puneet.gupta@amd.com, song.bao.hua@hisilicon.com, mchehab+huawei@kernel.org, f.fainelli@gmail.com, jeffrey.l.hugo@gmail.com, saravanak@google.com, Michael.Srba@seznam.cz, mani@kernel.org, yishaih@nvidia.com, robin.murphy@arm.com, will@kernel.org, joro@8bytes.org, masahiroy@kernel.org, ndesaulniers@google.com, linux-arm-kernel@lists.infradead.org, linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, kvm@vger.kernel.org, okaya@kernel.org, harpreet.anand@amd.com, nikhil.agarwal@amd.com, michal.simek@amd.com, git@amd.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220908_071900_891494_FA091285 X-CRM114-Status: GOOD ( 48.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, 08 Sep 2022 10:51:01 +0100, "Radovanovic, Aleksandar" wrote: > > [AMD Official Use Only - General] > > > > > -----Original Message----- > > From: Marc Zyngier > > Sent: 08 September 2022 09:08 > > To: Radovanovic, Aleksandar > > Cc: Jason Gunthorpe ; Gupta, Nipun > > ; robh+dt@kernel.org; > > krzysztof.kozlowski+dt@linaro.org; gregkh@linuxfoundation.org; > > rafael@kernel.org; eric.auger@redhat.com; alex.williamson@redhat.com; > > cohuck@redhat.com; Gupta, Puneet (DCG-ENG) > > ; song.bao.hua@hisilicon.com; > > mchehab+huawei@kernel.org; f.fainelli@gmail.com; > > jeffrey.l.hugo@gmail.com; saravanak@google.com; > > Michael.Srba@seznam.cz; mani@kernel.org; yishaih@nvidia.com; > > robin.murphy@arm.com; will@kernel.org; joro@8bytes.org; > > masahiroy@kernel.org; ndesaulniers@google.com; linux-arm- > > kernel@lists.infradead.org; linux-kbuild@vger.kernel.org; linux- > > kernel@vger.kernel.org; devicetree@vger.kernel.org; kvm@vger.kernel.org; > > okaya@kernel.org; Anand, Harpreet ; Agarwal, > > Nikhil ; Simek, Michal ; > > git (AMD-Xilinx) > > Subject: Re: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its > > domain as parent > > > > [CAUTION: External Email] > > > > OK, so you definitely need a mapping, but it cannot be a translation, and it > > needs to be in all the possible address spaces. OMG. > > Could you elaborate why it needs to be in all the possible address > spaces? I'm in no way familiar with kernel IOVA allocation, so not > sure I understand this requirement. Note that each CDX device will > have its own unique StreamID (in general case, equal to DeviceID > sent to the GIC), so, from a SMMU perspective, the mapping can be > specific to that device. As long as that IOVA is not allocated to > any DMA region for _that_ device, things should be OK? But, I > appreciate it might not be that simple from a kernel perspective. Robin answered that one. This also has direct impacts on vfio and virtualisation, as your userspace/VM cannot have any memory at the address of the ITS (because you cannot DMA to it). > > > > > > As for the data part (EventID in GIC parlance), this is always > > > > > going to be the CDX device-relative vector number - I believe this > > > > > can't be changed, it is a hardware limitation (but I need to double- > > check). > > > > > That should be OK, though, as I believe this is exactly what Linux > > > > > would write anyway, as each CDX device should be in its own IRQ > > > > > domain (i.e. have its own ITS device table). > > > > > > > > But that's really the worse part. You have hardcoded what is the > > > > *current* Linux behaviour. Things change. And baking SW behaviour > > > > into a piece of HW looks incredibly shortsighted... > > > > > > For posterity, I'm not an RTL designer/architect, so share your > > > sentiment to a certain extent. That said, I expect the decision was > > > not based on Linux or any other SW behaviour, but because it is the > > > most straightforward and least expensive way to do it. Having an > > > EventID register for each and every MSI source just so you can program > > > it in any random order costs flops and all the associated complexity > > > of extra RTL logic (think timing closure, etc.), so trade-offs are > > > made. The fact that it matches current Linux behaviour means we just > > > got lucky... > > > > Yeah, but that's not the only problem: there is no guarantee that we have > > enough LPIs to allocate for the device, so we'll perform a partial allocation (8 > > instead of 32 LPIs, for example). > > Why should that be a problem? The driver will know in advance the > number of vectors required by the device. I expect it will need to > call some equivalent of platform_msi_domain_alloc_irqs(), shouldn't > that fail if not enough IRQs are allocated (and ultimately fail the > probe)? No, that's not how MSIs work in Linux: this is a best effort allocation, where the allocator is free to dish out the number of MSIs it can allocate at this point in time (the ITS will divide the allocation by two until it succeeds). If the end-point driver doesn't like it, it can of course decide to fail its own probe. > Even if not, we can still inform the firmware in write_msg, > which will serve as an indication that a particular vector is > enabled. The firmware can decide what to do with the device if not > all of the vectors are enabled. Do you anticipate that the FW will be involved at the point where the driver is finalising the device configuration? It seems that you need to come up with some sort of spec for this if there isn't one yet. M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel