From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18681C4332F for ; Wed, 12 Oct 2022 14:38:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229712AbiJLOih (ORCPT ); Wed, 12 Oct 2022 10:38:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229603AbiJLOie (ORCPT ); Wed, 12 Oct 2022 10:38:34 -0400 Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 809EF43631 for ; Wed, 12 Oct 2022 07:38:33 -0700 (PDT) Received: by mail-qk1-x733.google.com with SMTP id x13so8017457qkg.11 for ; Wed, 12 Oct 2022 07:38:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=u0eqSs8D8Zyq93HxhcB9uauCbxkxsD79LqnsXThBZU4=; b=nBlGzmppneJVftZVAdRUc8SDMtu28+R52PjX0hqulIEfS4QZ9X3ZxOsFb+wI9VyrWn L/LUAF4zAGuwresjaJtPbsUQzAbxfPQ0W2hprNo7mjdNFp4BPvQT9n9LBcPQkREsO7Af 6b9iZUi3QgZnvK6cGojAjMQUAd6i+J3tln76arHaaeh3g/AUeFxxwDOr6ngw2zlZt5LZ JI28ohC7AeIA6B2gAXCA3RPFGhJ/s3wMBSxnkdNKGiphQAntDwcnxELsJvGLiKgoJ22n PfjHbfxSykYX0msc93QSe97LiewYxQf7oVnf6O3oXEw2GKmjUiydwB8oUe6E42DMK7Y0 zXIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=u0eqSs8D8Zyq93HxhcB9uauCbxkxsD79LqnsXThBZU4=; b=QQaly70G3lXhkltMgCXTS5aaUzNL91Z06gn/6cGltjf3GM8P+hUB3+aW4jm1cmp2fH gqors4JOq5kDliKquCpCcTR6OkOYwe5BUSg+nLWlvnFPk6wi0B54d0+AcboNa4CoRWRL YmL1Fe/II+f0Cmhv1lQ47uXRMsL/rTZvxDnN4w5ZvG1brwGxJsaP3oLN4G1t24oBQpGZ FaVv+3KxCXRMIRODQq/nO0y6n9wBIkOtjU3tkzkE7C+B6dE+GeaGQMFt0TKftAhK+E45 2wVOzDwLKoA9YyAGtMZpPKPI/6SsEV0gxmjjQOQue36J/PmuNSHFGDzMdMPAjIujYN5V OdFw== X-Gm-Message-State: ACrzQf0lC6hNHD1irNUZ6D9+PgyKiAyw9c5IzWbeAW+D+Mxf1wJtV6hJ EYHdtz4W8rX7POKin4lt8SJbZg== X-Google-Smtp-Source: AMsMyM4+AIWJQxWBA4Jv0JHMqvqHYDWJctiOTgcBHB44Ww27YolyE0l6UFRUO8aTgAryoalNdC720g== X-Received: by 2002:a37:34e:0:b0:6ea:2cef:131f with SMTP id 75-20020a37034e000000b006ea2cef131fmr16541977qkd.554.1665585512620; Wed, 12 Oct 2022 07:38:32 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-47-55-122-23.dhcp-dynamic.fibreop.ns.bellaliant.net. [47.55.122.23]) by smtp.gmail.com with ESMTPSA id i8-20020a05620a404800b006bc192d277csm16352660qko.10.2022.10.12.07.38.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 07:38:31 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1oicsP-001y84-7s; Wed, 12 Oct 2022 11:38:29 -0300 Date: Wed, 12 Oct 2022 11:38:29 -0300 From: Jason Gunthorpe To: "Radovanovic, Aleksandar" Cc: "Gupta, Nipun" , Marc Zyngier , Robin Murphy , "robh+dt@kernel.org" , "krzysztof.kozlowski+dt@linaro.org" , "gregkh@linuxfoundation.org" , "rafael@kernel.org" , "eric.auger@redhat.com" , "alex.williamson@redhat.com" , "cohuck@redhat.com" , "Gupta, Puneet (DCG-ENG)" , "song.bao.hua@hisilicon.com" , "mchehab+huawei@kernel.org" , "f.fainelli@gmail.com" , "jeffrey.l.hugo@gmail.com" , "saravanak@google.com" , "Michael.Srba@seznam.cz" , "mani@kernel.org" , "yishaih@nvidia.com" , "will@kernel.org" , "joro@8bytes.org" , "masahiroy@kernel.org" , "ndesaulniers@google.com" , "linux-arm-kernel@lists.infradead.org" , "linux-kbuild@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "devicetree@vger.kernel.org" , "kvm@vger.kernel.org" , "okaya@kernel.org" , "Anand, Harpreet" , "Agarwal, Nikhil" , "Simek, Michal" , "git (AMD-Xilinx)" Subject: Re: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its domain as parent Message-ID: References: <20220803122655.100254-1-nipun.gupta@amd.com> <20220906134801.4079497-1-nipun.gupta@amd.com> <20220906134801.4079497-5-nipun.gupta@amd.com> <87h71juxuk.wl-maz@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org On Wed, Oct 12, 2022 at 01:37:54PM +0000, Radovanovic, Aleksandar wrote: > > On Wed, Oct 12, 2022 at 10:34:23AM +0000, Radovanovic, Aleksandar wrote: > > > > > > > As for GITS_TRANSLATER, we can take up to 4 different IOVAs, which > > > limits us to 4 CDX devices (should be sufficient for current HW > > > use-cases). Also, it means that the address part must be the same for > > > all vectors within a single CDX device. I'm assuming this is OK as it > > > is going to be a single interrupt and IOMMU domain anyway. > > > > This is not at all how MSI is supposed to work. > > In the general case, no, they're not. I don't mean that you can hack this to work - I mean that in MSI the addr/data is supposed to come from the end point itself, not from some kind of shared structure. This is important because the actual act of generating the write has to be coherent with the DMA the device is doing, as the MSI write must push any DMA data to visibility to meet the "producer / consumer" model. So it is really weird/wrong to have a HW design where the MSI infrastructure is shared across many devices. Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4D3B4C4332F for ; Wed, 12 Oct 2022 14:39:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Yt7bt1az0tV2Mk1Nz9K0CD13MT6Hm+R5SEpg5Y8vcOA=; b=S3VmpCKYUfxhYR JmnfoueaAm6Gw/xd8+9hPMrxQTekhYE7FgUUhCkInKWARanw0fG8Ga8fJd85Z3GixhcnDTFmFq6R6 UecxbxBCi2KZkDzh4HqgjbcrSuCwPbX5iqDwhVXMOs/krigTiA5C/T+Ciqj8h9xu5SUTkxVj81h2C Cbq/jPrTZtziO1HbxW1+Ppgg7Nw61c3X0cQcpDqnAj5VksmL6VP7hNOkuyQIlv8q2hTsKwmAFx3HC A48F3B3Y6MMQB2gFD3RkE2sRWgdy/q2CoPWuBVrkPIGJf/gfgYJksFIBlSB236RYfYqPqPrnmBZxQ c7Rub53QWO28+I5iMpLw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oicsY-008Pzt-AV; Wed, 12 Oct 2022 14:38:38 +0000 Received: from mail-qk1-x733.google.com ([2607:f8b0:4864:20::733]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oicsV-008Pyk-OO for linux-arm-kernel@lists.infradead.org; Wed, 12 Oct 2022 14:38:37 +0000 Received: by mail-qk1-x733.google.com with SMTP id o22so10152533qkl.8 for ; Wed, 12 Oct 2022 07:38:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=u0eqSs8D8Zyq93HxhcB9uauCbxkxsD79LqnsXThBZU4=; b=nBlGzmppneJVftZVAdRUc8SDMtu28+R52PjX0hqulIEfS4QZ9X3ZxOsFb+wI9VyrWn L/LUAF4zAGuwresjaJtPbsUQzAbxfPQ0W2hprNo7mjdNFp4BPvQT9n9LBcPQkREsO7Af 6b9iZUi3QgZnvK6cGojAjMQUAd6i+J3tln76arHaaeh3g/AUeFxxwDOr6ngw2zlZt5LZ JI28ohC7AeIA6B2gAXCA3RPFGhJ/s3wMBSxnkdNKGiphQAntDwcnxELsJvGLiKgoJ22n PfjHbfxSykYX0msc93QSe97LiewYxQf7oVnf6O3oXEw2GKmjUiydwB8oUe6E42DMK7Y0 zXIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=u0eqSs8D8Zyq93HxhcB9uauCbxkxsD79LqnsXThBZU4=; b=YofmX6U93p53Lp5Lf2e2d7TXGmpy/qpuFs6NdYDMN1ii8jECVzd/jUwVsKm834scO7 XXYbXmsXGIwNlRVLV5YhkkhCG7bOfkKExtF+rimCPs9OlBV8Xa/YCD8I3rcAeEiG1MPP v5yXoi6dQOhdJuHZVbeYTQXjVCCVdrnhMXRgE07RfyKNGgW+U+WaoEBVqTLrkcXCJMVF SWqju05KDkVV33qHiPYrwULaHkH6QxSdY6iO7L02n+5UBwUViFummAUTdnwpgXKarAP5 yF3yat1i56FEqZYOFdluKKuivdT88Wlk63yoOgpcXbvHci7JuricHgK5qQBXxlADyGQ+ mhkQ== X-Gm-Message-State: ACrzQf1q/p4KGF2bRYsiZ62vncD06/2eEtHDE3Ot1D+re+9mI96XC0tX PZ44LZyflafMiQAphUHmiGzTqQ== X-Google-Smtp-Source: AMsMyM4+AIWJQxWBA4Jv0JHMqvqHYDWJctiOTgcBHB44Ww27YolyE0l6UFRUO8aTgAryoalNdC720g== X-Received: by 2002:a37:34e:0:b0:6ea:2cef:131f with SMTP id 75-20020a37034e000000b006ea2cef131fmr16541977qkd.554.1665585512620; Wed, 12 Oct 2022 07:38:32 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-47-55-122-23.dhcp-dynamic.fibreop.ns.bellaliant.net. [47.55.122.23]) by smtp.gmail.com with ESMTPSA id i8-20020a05620a404800b006bc192d277csm16352660qko.10.2022.10.12.07.38.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 07:38:31 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1oicsP-001y84-7s; Wed, 12 Oct 2022 11:38:29 -0300 Date: Wed, 12 Oct 2022 11:38:29 -0300 From: Jason Gunthorpe To: "Radovanovic, Aleksandar" Cc: "Gupta, Nipun" , Marc Zyngier , Robin Murphy , "robh+dt@kernel.org" , "krzysztof.kozlowski+dt@linaro.org" , "gregkh@linuxfoundation.org" , "rafael@kernel.org" , "eric.auger@redhat.com" , "alex.williamson@redhat.com" , "cohuck@redhat.com" , "Gupta, Puneet (DCG-ENG)" , "song.bao.hua@hisilicon.com" , "mchehab+huawei@kernel.org" , "f.fainelli@gmail.com" , "jeffrey.l.hugo@gmail.com" , "saravanak@google.com" , "Michael.Srba@seznam.cz" , "mani@kernel.org" , "yishaih@nvidia.com" , "will@kernel.org" , "joro@8bytes.org" , "masahiroy@kernel.org" , "ndesaulniers@google.com" , "linux-arm-kernel@lists.infradead.org" , "linux-kbuild@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "devicetree@vger.kernel.org" , "kvm@vger.kernel.org" , "okaya@kernel.org" , "Anand, Harpreet" , "Agarwal, Nikhil" , "Simek, Michal" , "git (AMD-Xilinx)" Subject: Re: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its domain as parent Message-ID: References: <20220803122655.100254-1-nipun.gupta@amd.com> <20220906134801.4079497-1-nipun.gupta@amd.com> <20220906134801.4079497-5-nipun.gupta@amd.com> <87h71juxuk.wl-maz@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221012_073836_075751_8C6125CE X-CRM114-Status: GOOD ( 19.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Oct 12, 2022 at 01:37:54PM +0000, Radovanovic, Aleksandar wrote: > > On Wed, Oct 12, 2022 at 10:34:23AM +0000, Radovanovic, Aleksandar wrote: > > > > > > > As for GITS_TRANSLATER, we can take up to 4 different IOVAs, which > > > limits us to 4 CDX devices (should be sufficient for current HW > > > use-cases). Also, it means that the address part must be the same for > > > all vectors within a single CDX device. I'm assuming this is OK as it > > > is going to be a single interrupt and IOMMU domain anyway. > > > > This is not at all how MSI is supposed to work. > > In the general case, no, they're not. I don't mean that you can hack this to work - I mean that in MSI the addr/data is supposed to come from the end point itself, not from some kind of shared structure. This is important because the actual act of generating the write has to be coherent with the DMA the device is doing, as the MSI write must push any DMA data to visibility to meet the "producer / consumer" model. So it is really weird/wrong to have a HW design where the MSI infrastructure is shared across many devices. Jason _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel