From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EED40C43219 for ; Fri, 25 Nov 2022 14:26:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229720AbiKYO0h (ORCPT ); Fri, 25 Nov 2022 09:26:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229553AbiKYO0f (ORCPT ); Fri, 25 Nov 2022 09:26:35 -0500 Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 539172AC6D for ; Fri, 25 Nov 2022 06:26:34 -0800 (PST) Received: by mail-pg1-x533.google.com with SMTP id 62so4060292pgb.13 for ; Fri, 25 Nov 2022 06:26:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=pr9EPBHz4BamKgB8yWADZkXZ2+tlYR/qk7DwwWPntys=; b=I22Ui3Ws2mdvy3i8n4OvfE7B4UBhCYea7j3aLWfiqH6e04pBvYdBesUghp62dWLwlP nUvmU/k0g1An/0OlJ34HaFJQiRBzF0H4W6T/bVUx1zm9Bl2bgWzoL3GpF4JfgONWkVbE EwQpaRMKtu1ensgpA7mEiE9oiRjstNpI6/teS/2EmsCih9G1zkFWu57x03IKOctYmO6b 8uIFnuHSjhfW3vcW332yGJfmlCNj0hM2e3UlZ2FvRnUWb5xA0fGO9KsZPNX63gbTbsLc Avx33IJSJQXBDcMj1TDY9VR9YSk/OF8WhfUPLTuG21X7mpO1vVjJj47paR3SHE3BHjTr ROiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pr9EPBHz4BamKgB8yWADZkXZ2+tlYR/qk7DwwWPntys=; b=XDcQhGkS6orZrUC9VFPAiFISigijgZfLmnb5c3QJcn9O5kV/v6a7kDYRLp3bt/VEcP wRO5oUeQ3sirKT4GVqKRGwLCOnE8ji4bAz52qMh7d5+N5zgpCXAN+ud8sN+ku97QXrHR MiCU5ZyyQ84V7A0PJN1ylPHWHzQ8wazdzL80uQDIDSsGEp5mZ9isajD6TeGRjlVTSkNu aK5nYmSPtMeuTVA2vNYo62NZ35J/Rc82e5RJN473mZs8mmEpWLtHuBo0vYZS92upS3hO qKz4shv3UQS/gkZWwJ26KfSfkoJkrlamVtBTUtw7gaNkXeaCLJIRf67XAnB6bjbsUxVj VDGQ== X-Gm-Message-State: ANoB5plhGnIq+lw8nK37ELxx0p88Q3IJqKV0+MnYZQ/fSgQPZHGuagcT N86T20//97rJeINCnTo0UmIg X-Google-Smtp-Source: AA0mqf49zCazD+UtCRa4BNYXgoE42BG1LqZ3W0LypZAXCz7fviyny0CBtnZkM2HGDAm3DL59CDuHnQ== X-Received: by 2002:a63:1e62:0:b0:46b:3acb:77b2 with SMTP id p34-20020a631e62000000b0046b3acb77b2mr15395586pgm.560.1669386393687; Fri, 25 Nov 2022 06:26:33 -0800 (PST) Received: from thinkpad ([117.202.190.212]) by smtp.gmail.com with ESMTPSA id n7-20020a170902d2c700b00186ffe62502sm3448694plc.254.2022.11.25.06.26.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Nov 2022 06:26:32 -0800 (PST) Date: Fri, 25 Nov 2022 19:56:25 +0530 From: Manivannan Sadhasivam To: Johan Hovold Cc: Bjorn Andersson , Andy Gross , Konrad Dybcio , Rob Herring , Krzysztof Kozlowski , Will Deacon , Robin Murphy , Christoph Hellwig , Ard Biesheuvel , Catalin Marinas , linux-arm-msm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH] arm64: dts: qcom: sc8280xp: fix PCIe DMA coherency Message-ID: <20221125142625.GA9892@thinkpad> References: <20221124142501.29314-1-johan+linaro@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20221124142501.29314-1-johan+linaro@kernel.org> Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org On Thu, Nov 24, 2022 at 03:25:01PM +0100, Johan Hovold wrote: > The devices on the SC8280XP PCIe buses are cache coherent and must be > marked as such to avoid data corruption. > > A coherent device can, for example, end up snooping stale data from the > caches instead of using data written by the CPU through the > non-cacheable mapping which is used for consistent DMA buffers for > non-coherent devices. > Also, the device may write into the L2 cache (or whatever cache that is accessible) if there is an entry and the CPU may invalidate it before reading from the DMA buffer. This will end up in a data loss. > Note that this is much more likely to happen since commit c44094eee32f > ("arm64: dma: Drop cache invalidation from arch_dma_prep_coherent()") > that was added in 6.1 and which removed the cache invalidation when > setting up the non-cacheable mapping. > > Marking the PCIe devices as coherent specifically fixes the intermittent > NVMe probe failures observed on the Thinkpad X13s, which was due to > corruption of the submission and completion queues. This was typically > observed as corruption of the admin submission queue (with well-formed > completion): > > could not locate request for tag 0x0 > nvme nvme0: invalid id 0 completed on queue 0 > > or corruption of the admin or I/O completion queues (malformed > completion): > > could not locate request for tag 0x45f > nvme nvme0: invalid id 25695 completed on queue 25965 > > presumably as these queues are small enough to not be allocated using > CMA which in turn make them more likely to be cached (e.g. due to > accesses to nearby pages through the cacheable linear map). Increasing > the buffer sizes to two pages to force CMA allocation also appears to > make the problem go away. > I don't think the problem will go away if the allocation happens from CMA region. It may just decrease the chances of cache hit but it could always happen due to the existence of linear mapping with cacheable attribute. > Fixes: 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes") > Signed-off-by: Johan Hovold Anyway, this is a really good find! Reviewed-by: Manivannan Sadhasivam Thanks, Mani > --- > arch/arm64/boot/dts/qcom/sc8280xp.dtsi | 10 ++++++++++ > 1 file changed, 10 insertions(+) > > diff --git a/arch/arm64/boot/dts/qcom/sc8280xp.dtsi b/arch/arm64/boot/dts/qcom/sc8280xp.dtsi > index 27f5c2f82338..7748cd29276d 100644 > --- a/arch/arm64/boot/dts/qcom/sc8280xp.dtsi > +++ b/arch/arm64/boot/dts/qcom/sc8280xp.dtsi > @@ -854,6 +854,8 @@ pcie4: pcie@1c00000 { > <0x02000000 0x0 0x30300000 0x0 0x30300000 0x0 0x1d00000>; > bus-range = <0x00 0xff>; > > + dma-coherent; > + > linux,pci-domain = <6>; > num-lanes = <1>; > > @@ -951,6 +953,8 @@ pcie3b: pcie@1c08000 { > <0x02000000 0x0 0x32300000 0x0 0x32300000 0x0 0x1d00000>; > bus-range = <0x00 0xff>; > > + dma-coherent; > + > linux,pci-domain = <5>; > num-lanes = <2>; > > @@ -1046,6 +1050,8 @@ pcie3a: pcie@1c10000 { > <0x02000000 0x0 0x34300000 0x0 0x34300000 0x0 0x1d00000>; > bus-range = <0x00 0xff>; > > + dma-coherent; > + > linux,pci-domain = <4>; > num-lanes = <4>; > > @@ -1144,6 +1150,8 @@ pcie2b: pcie@1c18000 { > <0x02000000 0x0 0x38300000 0x0 0x38300000 0x0 0x1d00000>; > bus-range = <0x00 0xff>; > > + dma-coherent; > + > linux,pci-domain = <3>; > num-lanes = <2>; > > @@ -1239,6 +1247,8 @@ pcie2a: pcie@1c20000 { > <0x02000000 0x0 0x3c300000 0x0 0x3c300000 0x0 0x1d00000>; > bus-range = <0x00 0xff>; > > + dma-coherent; > + > linux,pci-domain = <2>; > num-lanes = <4>; > > -- > 2.37.4 > -- மணிவண்ணன் சதாசிவம்