From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B57DC433DB for ; Wed, 6 Jan 2021 11:55:57 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D8EC223110 for ; Wed, 6 Jan 2021 11:55:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D8EC223110 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WOE/cKC+M67ngNwxqWpEpnH5D6GbkTtoIW6X4PtHpZ4=; b=rBwoaZ/iD5e2CJzdWOcsZedVe mlmpCF97uUeIKkBr39nVUHb4Lr6C69KoT7dL0kIG+hXS8cah0M4YEnjn5Lkom347pgbnSrVVhaOSw 5tXNVKzrZiAuwHs5r8WymT/PnOGwrb71QQEHc55Rg4yhKjZzPIhqpWVYPwQXNrZZuydpCmlATaZiI pVkk7nhMhHKHBBYR8MnvNY+aIgtLsSOSzXnjjiR3kLcPX2HfNYGqfFNs/o7QGcg52hJuuR09kmoqD 4Rd80AkgGAL7YFWFLX+NZcusXVXLYd78ENO6dCkgyavzIVdIGVx3oIH0E1kJBz1STRGfi0KRVMyUU FHsMHXwmQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kx7OD-00037D-Lw; Wed, 06 Jan 2021 11:54:09 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kx7OA-00036o-Fe for linux-arm-kernel@lists.infradead.org; Wed, 06 Jan 2021 11:54:07 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5546BD6E; Wed, 6 Jan 2021 03:54:04 -0800 (PST) Received: from C02TD0UTHF1T.local (unknown [10.57.36.216]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 75E0C3F70D; Wed, 6 Jan 2021 03:54:02 -0800 (PST) Date: Wed, 6 Jan 2021 11:53:59 +0000 From: Mark Rutland To: Russell King - ARM Linux admin Subject: Re: Aarch64 EXT4FS inode checksum failures - seems to be weak memory ordering issues Message-ID: <20210106115359.GB26994@C02TD0UTHF1T.local> References: <20210105154726.GD1551@shell.armlinux.org.uk> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20210105154726.GD1551@shell.armlinux.org.uk> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210106_065406_587966_1F45DA38 X-CRM114-Status: GOOD ( 21.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Theodore Ts'o , linux-kernel@vger.kernel.org, Andreas Dilger , linux-ext4@vger.kernel.org, Will Deacon , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Jan 05, 2021 at 03:47:26PM +0000, Russell King - ARM Linux admin wrote: > Hi, Hi Russell, > This is an update on where I am with this long standing issue at the > current time. > > Since 5.4, I have been struggling with several of my ARM64 systems, of > different SoC vendors and differing filesystem media, were sporadically > reporting inode checksum failures on their root filesystems. The time > taken to report this has been anything between a few hours and three > months of uptime, making the problem unrealistic to bisect. > However, over the last couple of days, a way to reproduce it has been > found, at least for the LX2160A based system. Power down, leave the > machine powered off for some time. Power up, log in and run: > > while :; do sleep 5; find /var /usr /bin /sbin -type f -print0 | \ > xargs -0 md5sum >/dev/null; done I've just set this off on an Raspberry Pi 4 running a locally-built arm64 v5.10 defconfig. I'm using a SATA SSD mounted via a USB-SATA adapter. I'll try to give that a few reboots see if I can reproduce the issue. Just to check -- how is your ext4 fs mounted? Mine is mounted as: /dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro) ... and are you using defconfig or something else? [...] > It is possible that it could be compiler related, but I don't see that; > if the "dmb oshld" were strong enough, then it should mean that the > subsequent reads to checksum the inode data after the inode data has > been DMA'd into memory should be reading the correct values from memory > already - but they aren't. And if changing "dmb oshld" to "dsb ld" means > that the code can then read the right values, that to me points fairly > definitively to a hardware problem. Just in case the compiler has some impact, can you way which compilers you've been using? On the rpi4 I'm using GCC 8.3.0 as packaged by Debian Buster (10.7). Thanks, Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel