From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7710C48BC3 for ; Wed, 21 Feb 2024 15:27:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=r5Yvk+FAqsoQ9Tk2LJwGlKiUwdQPo41u4D4Y4dtNHzk=; b=vMmdm+Y+A/ioen AuMyu7XDqTpsWQ0mhVL8t046aTc+6p9578hVPQbKJnf6xpQVfTdjmrU0fKbJ3oiV8frxI9fAt6bd9 04Q8NHFH02xBXqwCITY/03Jq06nhbY19ceX7gyUHvOUPw0tNMoSFL+kZz91pKjSbfacW5AP0YJiSD RtdPh27bW3ftWNMmc5i+FscwvNH4Q5RalgJ5x3Tgr8c3ZwnhQApMoCmXCxX39jivokivcDU5Mel6G a2fpDi1ruj7AQwsNUDmXy0YtTjzfDv1ObcfAxVHy4y8qug45CaT+5VHPXaMdnP8U9eQKLUdfP5dd+ yyYLEu/fXC2h0tSqimoQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rcoV5-00000001Ttk-0N3K; Wed, 21 Feb 2024 15:27:11 +0000 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rcoUY-00000001Tkz-1BUt for linux-arm-kernel@lists.infradead.org; Wed, 21 Feb 2024 15:26:39 +0000 Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-6e48eef8be5so557388b3a.1 for ; Wed, 21 Feb 2024 07:26:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708529197; x=1709133997; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wW/wH8Bxvy6vIw9lSkZQNzRLcUxp6rWTmo8sEe8KFwU=; b=Pg+EicFMwcoT6rFUawkVL/7FsGyhvv1P8BCgvuT0YrdNOO3Fe7ggpifIH9qavFcxD3 Z1Z23wrCtekW4afKTEvxKGB8j3gZw+hI2UkWmaUHL1Xm19sZpNTPuGqc+PMSTRRDQFah O3s7uRSalJqg7KKnkwISXAQVQUW5x/plVnCWf4CBSqcw9/FqnzAROt/DlxoqrAUarUZn n1anCGCjz2emZLFxY8lfxXQLuyyacVyQxdGAZdKshREr+gwAy6GMsRJ/Bb7Dg4HUS+IS 4SpNTGxYx5ZyboHWT2hBO60xIz39jGCUoX3dAbLsd2Os/qQNbS6IzyylwzjvJ7oyY8lJ xlvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708529197; x=1709133997; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wW/wH8Bxvy6vIw9lSkZQNzRLcUxp6rWTmo8sEe8KFwU=; b=Sx4ioQcWbKdNyU/EZu09N94sGa+Nec6B3vtJAjyHIT8QVEp1Xxw2QybShMql+YIZR/ 70tK/lcHDnEydtwxh8eRd06YS7faBcuKeCiUCKicbjnfnlgqKK0ZWiLoOTLCLErLNHTs 1fXGmO47PbitIn3SB4/TWBjuIHfsL3WtAPKirqwI53/zaEeKizGciF8Wdsd7WLkyTBR7 SWSIgyqWzuoDNLZK9+YvpK+aANuIcWY6NmsO+CGpe9RpJHwnRPACBt5ltfkYA0McylEV AlCJM5obAoYgNS8tVQ6igl7d5M0ItljZGiINPnY451roLIOcx8A38g29cEJr3vFC/Sck NjTw== X-Forwarded-Encrypted: i=1; AJvYcCWI0rclNVObM+Sz48kN2bsSJQNLIR5Wek0irycSHM8pDSJG0yuFsD1pLQFzgNiRbZKEC2bdIOgjIuGWOJ8zxFBkvBPac0bYXkVZJZ0m2MThoT4AN5E= X-Gm-Message-State: AOJu0YxEQrdIpzjzYuq+f0CvxAhduDXAX7aWr5jl1HEx9PE1CseOwDHD O5gW5574UknvBbHqMLlUfB4UiasuWUjQ3nGZwoYAW0C6+ysmlAzE X-Google-Smtp-Source: AGHT+IEt3BD9AeWE2Q2VZ6pJE2NucbpYuT5K2Ta/OzpXuZ/4vRBzk1AoIfF/xKdkIbIGN436T6HTyQ== X-Received: by 2002:aa7:8652:0:b0:6e4:6163:7e05 with SMTP id a18-20020aa78652000000b006e461637e05mr7953591pfo.6.1708529196847; Wed, 21 Feb 2024 07:26:36 -0800 (PST) Received: from nilq-virtual-machine.localdomain ([60.24.211.11]) by smtp.gmail.com with ESMTPSA id ka36-20020a056a0093a400b006e48e0499dfsm1619085pfb.39.2024.02.21.07.26.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 07:26:36 -0800 (PST) From: "ni.liqiang" To: will@kernel.org, danielmentz@google.com Cc: iommu@lists.linux.dev, jin.qi@zte.com.cn, joro@8bytes.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, robin.murphy@arm.com, ni.liqiang@zte.com.cn Subject: Re: [PATCH] drivers/iommu: Ensure that the queue base address is successfully written during SMMU initialization. Date: Wed, 21 Feb 2024 23:26:29 +0800 Message-Id: <20240221152629.3656-1-niliqiang.io@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240219091709.GA4105@willie-the-truck> References: <20240219091709.GA4105@willie-the-truck> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240221_072638_397895_AB7AD758 X-CRM114-Status: GOOD ( 15.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org >> The SMMU registers are accessed using Device-nGnRE attributes. It is >> my understanding that, for Device-nGnRE, the Arm architecture requires >> that writes to the same peripheral arrive at the endpoint in program >> order. > > Yup, that's correct. The "nR" part means "non-Reordering", so something > else is going on here. Yes, the SMMU registers are accessed using Device-nGnRE attributes. One additional point to note is: in cases where there is a failure writing to the CMDQ base address register, the testing environment was a multi-die, multi-socket server. This issue has not been observed on a single-die server. I apologize for omitting this information in my initial patch submission. Over the past few days, I have referenced the kernel source code and ported the SMMU register initialization process. Through multiple stress tests, I have attempted to reproduce the CMDQ base address register write failure issue. The summarized results of my experiments are as follows: 1. When testing with one CPU core bound using taskset, the initialization process was executed 300,000 times without encountering the CMDQ base address register write failure issue. However, when not binding CPU using taskset, the issue was reproduced around 1,000 iterations into the test. 2. Without CPU binding, I inserted a memory barrier between accesses to the CMDQ_BASE register and CMDQEN register, similar to the modification made in the patch. After executing the initialization process 300,000 times, the CMDQ base address register write failure issue did not occur. Based on these observations and joint analysis with CMN colleagues, we speculate that in the SMMU register initialization process, if the CPU core changes, and these CPUs are located on different dies, the underlying 4 CCG ports are utilized to perform die-to-die accesses. However, in our current strategy, these 4 CCG ports cannot guarantee ordering, resulting in the completion of CMDQEN writing before the completion of CMDQ base address writing. >From the analysis above, it seems that modifying the die-to-die access strategy to achieve ordering of Device-nGnRE memory might be a better solution compared to adding a memory barrier? _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel