From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A55D8C433EF for ; Mon, 20 Dec 2021 18:44:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:To:Subject:MIME-Version: Date:Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=PwlWneawfSiecthMKP7/36aCB14VKuyLsSRHDeZxv7w=; b=QEnIU8sahNpgSfG+WIEabxhpA7 hwk78p63Qw3aC2Z4h5DLRA8w4vqgXDqkyP/cmPOcaUCut+b+OUoSBZX1RPo1gmHqMeyL6r/fVGQZl 2m/ju2F6Tm97SCN6A1j6uQAdEYDC0GSfGqg061+I2vuuASinnrdVSDEZd8ZCwZp3Gpt/jfjhuDa8e KGO3kQcnyUH28p92AbKTGHENjM4mSQejgM3slhK/jzWhRdjrMsB2LC0uvuidswbv5fvqGLuT8kDow daiLrsrYTr4DP1qxF73l8UbnBgOgCnneKl3P4Kmcs73opQWEIyjtFYRNZgSynpv4Ay+v1gbTXAqG1 1Jp4TU/w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mzNcP-003r6R-Ef; Mon, 20 Dec 2021 18:42:41 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mzNcL-003r4v-FJ for linux-arm-kernel@lists.infradead.org; Mon, 20 Dec 2021 18:42:39 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6C2BC6D; Mon, 20 Dec 2021 10:42:35 -0800 (PST) Received: from [10.57.34.58] (unknown [10.57.34.58]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 904723F774; Mon, 20 Dec 2021 10:42:30 -0800 (PST) Message-ID: Date: Mon, 20 Dec 2021 18:42:26 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Thunderbird/91.4.0 Subject: Re: [PATCH v3 4/5] iommu/arm-smmu-v3: Add host support for NVIDIA Grace CMDQ-V Content-Language: en-GB To: Nicolin Chen , joro@8bytes.org, will@kernel.org References: <20211119071959.16706-1-nicolinc@nvidia.com> <20211119071959.16706-5-nicolinc@nvidia.com> From: Robin Murphy In-Reply-To: <20211119071959.16706-5-nicolinc@nvidia.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211220_104237_590648_88527D3A X-CRM114-Status: GOOD ( 22.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: jean-philippe@linaro.org, nwatterson@nvidia.com, chenxiang66@hisilicon.com, Jonathan.Cameron@huawei.com, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, nicoleotsuka@gmail.com, linux-tegra@vger.kernel.org, thierry.reding@gmail.com, jgg@nvidia.com, thunder.leizhen@huawei.com, yuzenghui@huawei.com, linux-arm-kernel@lists.infradead.org Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 2021-11-19 07:19, Nicolin Chen wrote: > From: Nate Watterson > > NVIDIA's Grace Soc has a CMDQ-Virtualization (CMDQV) hardware, > which extends the standard ARM SMMU v3 IP to support multiple > VCMDQs with virtualization capabilities. In-kernel of host OS, > they're used to reduce contention on a single queue. In terms > of command queue, they are very like the standard CMDQ/ECMDQs, > but only support CS_NONE in the CS field of CMD_SYNC command. > > This patch adds a new nvidia-grace-cmdqv file and inserts its > structure pointer into the existing arm_smmu_device, and then > adds related function calls in the arm-smmu-v3 driver. > > In the CMDQV driver itself, this patch only adds minimal part > for host kernel support. Upon probe(), VINTF0 is reserved for > in-kernel use. And some of the VCMDQs are assigned to VINTF0. > Then the driver will select one of VCMDQs in the VINTF0 based > on the CPU currently executing, to issue commands. Is there a tangible difference to DMA API or VFIO performance? [...] > +struct arm_smmu_cmdq *nvidia_grace_cmdqv_get_cmdq(struct arm_smmu_device *smmu) > +{ > + struct nvidia_grace_cmdqv *cmdqv = smmu->nvidia_grace_cmdqv; > + struct nvidia_grace_cmdqv_vintf *vintf0 = &cmdqv->vintf0; > + u16 qidx; > + > + /* Check error status of vintf0 */ > + if (!FIELD_GET(VINTF_STATUS, vintf0->status)) > + return &smmu->cmdq; > + > + /* > + * Select a vcmdq to use. Here we use a temporal solution to > + * balance out traffic on cmdq issuing: each cmdq has its own > + * lock, if all cpus issue cmdlist using the same cmdq, only > + * one CPU at a time can enter the process, while the others > + * will be spinning at the same lock. > + */ > + qidx = smp_processor_id() % cmdqv->num_vcmdqs_per_vintf; How does ordering work between queues? Do they follow a global order such that a sync on any queue is guaranteed to complete all prior commands on all queues? The challenge to make ECMDQ useful to Linux is how to make sure that all the commands expected to be within scope of a future CMND_SYNC plus that sync itself all get issued on the same queue, so I'd be mildly surprised if you didn't have the same problem. Robin. > + return &vintf0->vcmdqs[qidx]; > +} _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel