From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE0DDC4332F for ; Fri, 14 Oct 2022 14:00:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230202AbiJNOAf (ORCPT ); Fri, 14 Oct 2022 10:00:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230216AbiJNOAI (ORCPT ); Fri, 14 Oct 2022 10:00:08 -0400 Received: from mail-qk1-x72a.google.com (mail-qk1-x72a.google.com [IPv6:2607:f8b0:4864:20::72a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3630838A38 for ; Fri, 14 Oct 2022 06:59:56 -0700 (PDT) Received: by mail-qk1-x72a.google.com with SMTP id j21so2461615qkk.9 for ; Fri, 14 Oct 2022 06:59:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=qcbPZWPJBvSlgI0JaT98JkQa2BdMkjvTQHHNjJDLWR8=; b=cWHaCe+7NiLWV3T+fjLF8EZtlhexGbCg+WEq/dCVkwZlV9OSmlvmQvAf8tHOWxlB15 R1drmhQ65EsQ2EK/mhECj9QEh2CQLLNUzqJ9oqoPSSrkne89KFSq4YWdvfq9IP7Uota4 7XZtiYtVxdCKWb4baqEwWC8JB6Fdyr6kl2BDZy+wPH+ZpnDQcjanaP3WyBwV3Z1PZzGD CZWPDEfUXKdIW6mrakAhQPVrXN8JnsQ5WVKDAxqsL5jcUEWUMlF+wYwCnWrfob8HgcIT Gi3IrihaKlsefDNPRlWX3Hqdb8YfdREQACdRk26hiLm8LTNeDPPdkOwUpLkPKgwwv3Yz 5VcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=qcbPZWPJBvSlgI0JaT98JkQa2BdMkjvTQHHNjJDLWR8=; b=OIEFoGwgEoE0TnyBdbo4amCJ+wGVdxduoV524hv++gxiPkyoRL8grl+waxRiYY9ghL FcZ6eiJ4Hhpjbi4JTEXd9B+vmJbz5zHVv3ohzh0djOOQsjhA7OS/dEbOIGY5Z9dOHvjn uL1mH+EbbZicrYOGO84WqNxQr2tyYeqmZR4YRKA6ARe/ncvqvCJtrPRKDby6LjcHbBrE D4jTMLovj1meibeB7dSFxGWbgrJk0ZlWOARRmOswLMHAC9/CgePfMrsa8c4COSBjpoKn 3QlsVrXC44tJy9iiZBZbUKoNo2w05RSqZvQeAICTIIGJ4HBFUa13jVNs8OHzzBAQk+c2 eXbg== X-Gm-Message-State: ACrzQf3HaYohJuhNYTgRGupbPBA4yh+zg11oG3qv5bmMIPGXu+6IxlLy nL9n3GI0XMc3WV13ZNhnNX19og== X-Google-Smtp-Source: AMsMyM6IwCN+c1z62UhOHVbSClec5ccka72g2QriRCYRsTQFGoKqENCt7u1RwIvsCpE32fxIp3ANJA== X-Received: by 2002:a05:620a:4107:b0:6ee:ce95:1d15 with SMTP id j7-20020a05620a410700b006eece951d15mr1769931qko.266.1665755892583; Fri, 14 Oct 2022 06:58:12 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-47-55-122-23.dhcp-dynamic.fibreop.ns.bellaliant.net. [47.55.122.23]) by smtp.gmail.com with ESMTPSA id k15-20020a05620a414f00b006ce40fbb8f6sm2605722qko.21.2022.10.14.06.58.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 06:58:11 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1ojLCU-0033hQ-T0; Fri, 14 Oct 2022 10:58:10 -0300 Date: Fri, 14 Oct 2022 10:58:10 -0300 From: Jason Gunthorpe To: "Radovanovic, Aleksandar" Cc: "Gupta, Nipun" , Marc Zyngier , Robin Murphy , "robh+dt@kernel.org" , "krzysztof.kozlowski+dt@linaro.org" , "gregkh@linuxfoundation.org" , "rafael@kernel.org" , "eric.auger@redhat.com" , "alex.williamson@redhat.com" , "cohuck@redhat.com" , "Gupta, Puneet (DCG-ENG)" , "song.bao.hua@hisilicon.com" , "mchehab+huawei@kernel.org" , "f.fainelli@gmail.com" , "jeffrey.l.hugo@gmail.com" , "saravanak@google.com" , "Michael.Srba@seznam.cz" , "mani@kernel.org" , "yishaih@nvidia.com" , "will@kernel.org" , "joro@8bytes.org" , "masahiroy@kernel.org" , "ndesaulniers@google.com" , "linux-arm-kernel@lists.infradead.org" , "linux-kbuild@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "devicetree@vger.kernel.org" , "kvm@vger.kernel.org" , "okaya@kernel.org" , "Anand, Harpreet" , "Agarwal, Nikhil" , "Simek, Michal" , "git (AMD-Xilinx)" Subject: Re: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its domain as parent Message-ID: References: <20220906134801.4079497-5-nipun.gupta@amd.com> <87h71juxuk.wl-maz@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org On Fri, Oct 14, 2022 at 11:18:36AM +0000, Radovanovic, Aleksandar wrote: > And that still does not imply lack of ordering or sharing of MSI > target addresses between devices. Either the end point generates the MSI, and maybe the bridge mangles it, or it raises a lot of suspicion that this is not right. If the end point generates the MSI then it raises the question why do we need to tolerate these limits? > This is a highly programmable IP block, at the core of which is an > interconnect interfacing to programmable logic (PL), a number of > PCIe controllers (either endpoint or root-port), DMA engines, > offload engines, the embedded processor subsystem (PSX), etc. DMA > and interrupts can be routed across it in almost any (meaningful) > direction. The datapath 'endpoints' request DMA and interrupts, but > don't concern themselves with the mechanics of delivering that in > the target domain. It is the responsibility of the egress bridges to > the target domains to convert the interconnect interrupt > transactions to whatever the interrupt delivery mechanism for that > domain is. E.g. for PCIe controllers in endpoint mode, that would be > through PCIe MSI-X tables internal to the controller (and managed by > the PCIe host), for PSX that would be the PSX bridge (partially > managed by the PSX OS, mediated through firmware, i.e. through CDX > bus driver) and so on. It is the responsibility of the interconnect > to maintain transaction ordering (including DMA vs. interrupts). It > is the responsibility of the firmware to manage the bridges > according to the implemented use-case, so everything works as > expected. Again, this all just seems wrongly designed. MSI should not be part of an interconnect bridge. We did that already 20 years ago, it was called IOAPICs on x86 and I think everyone is happy to see it gone. If you want to build IOAPICs again, I guess you can, but that is a slightly different SW setup than the MSI you are trying to use here, and even that didn't have the same limitations you are proposing. > So, yes, the hardware that translates interrupt transactions to GIC > AXI writes is shared between endpoints, but what I said above still > applies. And that doesn't necessarily make it weird/wrong, it's just > more complex than you might think. If it doesn't fit the architecture, then I think it must be considered wrong. Mis-using platform architected components like MSI in HW is problematic. You should design the HW properly so you don't have these problems. Involving FW in the MSI setup is also a bad idea, POWER did this and it made a big mess of their arch code :( Jason