From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60665C636D3 for ; Mon, 6 Feb 2023 19:46:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=BOqAXPLtDr+HD4E4hp79tXLiaZWDxf9YFZqohz3pA78=; b=yxFzvb9pcuB3x2 oErf1s0QVLj+tJiEE4O9ufi7FK2buPB4rp4oCNq7UiUAMSoPiOIqe4EZAiKMVW9l+HbCl2JhgPogz pExNyy1u0dXiHtHbDuwSduQO5Z7j6zfZ1x3YaVuhXeHUB35oUPiN5qDB3bS6Jg40+F+odKFBvuphE ZYpkZfVPbhUQ4luJ+MvHPwGFts9O97dk4lU8HdkdQVfNfi7nCs2CB9stSl+DfQHh61clluU7xseIk xFUyW5c+vPE7zg3uSPjwf89xfHjppWe56n8/p6pHaq55DoMWtQy30FLVz6B7Azgq3kf1zahKB+bZQ AjbZYzsLf8bVD1ruJd8g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pP7QQ-009jsK-Vv; Mon, 06 Feb 2023 19:45:15 +0000 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pP7QN-009jrV-Pu for linux-arm-kernel@lists.infradead.org; Mon, 06 Feb 2023 19:45:13 +0000 Received: by mail-wm1-x334.google.com with SMTP id az4-20020a05600c600400b003dff767a1f1so4491647wmb.2 for ; Mon, 06 Feb 2023 11:45:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=APDyx3z4IV3rW3QxZlA7mfU0mjO7Ox32BtkbuWjdnhQ=; b=BA9E+Wxq5Z1bboYNGKdE7TzAScXqwKruYCCehgI70/13ingHOhZ6Y5lGoiIW56O4oD iPyb2vXEu6NdFnIsOqjcMM+aTduZbnWf9PLpnpM6swoMBvfb8WtbdsuCBWcMA67TrDDq Eg2djz5q+d9V14je6D2HYO8GqM6bxlcDxePdjqAk9G4NrYZ4wgFRJ8/Nav7JHwNxaABP 42QiftW1N6IgjvDmEqAWK2U/CB9OYPN0KhibLpI+IylVivMMA4ymm8RHisqUCezi9lMz J6BlkASoQUWyqBX+Q15WD5ygF0797iMyOHWin84BkroAw5L0qb6NZp+uIG7DXIZ8uBVd Mp9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=APDyx3z4IV3rW3QxZlA7mfU0mjO7Ox32BtkbuWjdnhQ=; b=SEM2W8QUqFnNw5bDH78jMRCmXT9xW37F1uZyBCB1yIQgRTPlHxQdwCMHKZvPwJrPc6 ipV1XlUoxY9DvO5CTkqQG+pVcWXZrnNqmSlqVhIFmCxI8OVXxv4tEKE3DgkP9mMbap9p 5S9Z9DuUx0gB1FlX7cggEs36AjhYSSVdw0BOkeld8Ds064luQfw/MBaITydavlP1CIpd GGHmJeYw35/vGT8mFYR66isYkzI4sLhjSIHSTcbzqxge96a7GZg1g9HCqRPL7hbfJSrp QT4uLC7lCxpJVbFe7BEPPMkW4NdXr4CNlVGjEom3O/6KhgtqELAfn+kNbtMJB/tv41jB Uwpw== X-Gm-Message-State: AO0yUKUnI/bNuLwoTngBVaD2/XsgZvD2thD8CzdncEdlm7UsfQBXF7cT i6cwUlmCf1AP/ZcAzK1B3b8JoA== X-Google-Smtp-Source: AK7set8XgVcldAJlafsoSXSh5Q33TEccODw28mHgBONK8/sMpsHldbDSv7P8EojCeImLsUHJJWeYsQ== X-Received: by 2002:a05:600c:43ca:b0:3d9:a145:4d1a with SMTP id f10-20020a05600c43ca00b003d9a1454d1amr764913wmn.34.1675712710077; Mon, 06 Feb 2023 11:45:10 -0800 (PST) Received: from myrica (054592b0.skybroadband.com. [5.69.146.176]) by smtp.gmail.com with ESMTPSA id ip20-20020a05600ca69400b003dd19baf45asm11232511wmb.40.2023.02.06.11.45.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Feb 2023 11:45:09 -0800 (PST) Date: Mon, 6 Feb 2023 19:45:05 +0000 From: Jean-Philippe Brucker To: Ganapatrao Kulkarni Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, robin.murphy@arm.com, will@kernel.org, joro@8bytes.org, darren@os.amperecomputing.com, scott@os.amperecomputing.com Subject: Re: [PATCH] iommu/arm-smmu-v3: Enable PCI ATS in passthrough mode as well Message-ID: References: <20230202124053.848792-1-gankulkarni@os.amperecomputing.com> <3a681b82-d840-6ef1-40dd-358f34c8be9c@os.amperecomputing.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <3a681b82-d840-6ef1-40dd-358f34c8be9c@os.amperecomputing.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230206_114511_866607_9A0AA00A X-CRM114-Status: GOOD ( 36.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Feb 06, 2023 at 10:50:00PM +0530, Ganapatrao Kulkarni wrote: > > > On 03-02-2023 05:30 pm, Jean-Philippe Brucker wrote: > > On Thu, Feb 02, 2023 at 04:40:53AM -0800, Ganapatrao Kulkarni wrote: > > > The current smmu-v3 driver does not enable PCI ATS for physical functions > > > of ATS capable End Points when booted in smmu bypass mode > > > (iommu.passthrough=1). This will not allow virtual functions to enable > > > ATS(even though EP supports it) while they are attached to a VM using > > > VFIO driver. > > > > > > This patch adds changes to enable ATS support for physical functions > > > in passthrough/bypass mode as well. > > [...] > > > @@ -2453,8 +2458,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > > > master->domain = smmu_domain; > > > - if (smmu_domain->stage != ARM_SMMU_DOMAIN_BYPASS) > > > - master->ats_enabled = arm_smmu_ats_supported(master); > > > + master->ats_enabled = arm_smmu_ats_supported(master); > > > > I should have added a comment for this. Only found the reason in an old > > cover letter [1]: > > > > "When no translation stages are enabled (0b100), ATS Translation Requests > > (and Translated traffic, if SMMU_CR0.ATSCHK == 1) are denied as though > > EATS == 0b00; the actual value of the EATS field is IGNORED. Such a > > Translation Request causes F_BAD_ATS_TREQ and Translated traffic causes > > F_TRANSL_FORBIDDEN." > > > > (See 3.9.1.1 "Responses to ATS Translation Requests and Translated > > transactions" and 5.2 "Stream Table Entry") > > > > So I don't think we can enable ATS for bypass domains :/ The PF needs to > > be in translated mode in that case. > > Are you intending to say smmu-v3 driver/spec will not support ATS to a VF, > if it's PF is in bypass? What I meant was that if, in order to enable the VF ATS capability, the PCIe device requires to first enable the PF ATS capability, then given that the SMMU does not allow enabling ATS for streams in bypass, we need to enable translation on both PF and VF SMMU configurations. But reading the PCIe spec, it looks like the capabilities are not necessarily tied together: "The ATS Capabilities in VFs and their associated PFs are permitted to be enabled independently." 10.5.1 ATS Extended Capability That wording doesn't indicate that all implementations must allow enabling the caps independently, only that they are allowed to do so. If a device provides independent capabilities, then I don't think you need to enable ATS in the PF. Then the ATS Control Register seems to contradict this with the STU field: "For VFs, this field must be hardwired to Zero. The associated PF's value applies." 10.5.1.3 ATS Control Register (Offset 06h) So pci_enable_ats() requires that pdev->ats_stu is set in order to enable ATS on the VF: /* * Note that enabling ATS on a VF fails unless it's already enabled * with the same STU on the PF. */ ... if (dev->is_virtfn) { pdev = pci_physfn(dev); if (pdev->ats_stu != ps) return -EINVAL; But I suspect it's done this way only because it's easier to implement. The spec doesn't require the PF ATS capability to be enabled, it just says that the PF's STU value counts as the VF's one. It looks like we're allowed to enable ATS on the VF without enabling it on the PF, right? If you rework the PCI driver to only write the PF's STU without enabling the capability, then you could enable it in the VF. Thanks, Jean > > > > > I can send a patch adding the comment next cycle. > > I am more keen to know, how we enable ATS to a VF of ATS capable EP when > it's PF is in bypass? > or it is mandatory to have a PF also translated? then that should be > captured somewhere in documentation. > > > > > > Thanks, > > Jean > > > > [1] https://lore.kernel.org/linux-iommu/20190409165245.26500-1-jean-philippe.brucker@arm.com/ > > > > Thanks, > Ganapat _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel