From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sandeen.net ([63.231.237.45]:59570 "EHLO sandeen.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727478AbeGaBTx (ORCPT ); Mon, 30 Jul 2018 21:19:53 -0400 Subject: Re: Recently-formatted XFS filesystems reporting negative used space References: <3895eebe-3fa2-a977-4021-3bd6ef645fdd@sandeen.net> <20180717092626.pxb6wzhcswf2f77p@odin.usersys.redhat.com> <830a7de7-34ca-7d79-d94a-ff16c98381a3@sandeen.net> From: Eric Sandeen Message-ID: <3a077653-8dc5-b38c-6ecf-6ed607a54d2b@sandeen.net> Date: Mon, 30 Jul 2018 18:42:30 -0500 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: linux-xfs-owner@vger.kernel.org List-ID: List-Id: xfs To: Filippo Giunchedi Cc: linux-xfs@vger.kernel.org On 7/30/18 5:02 AM, Filippo Giunchedi wrote: > On Sun, Jul 22, 2018 at 2:03 AM Eric Sandeen wrote: >> On 7/20/18 3:20 AM, Filippo Giunchedi wrote: >>> To recap what we've seen, hardware bit flipping is extremely unlikely: >>> the same type of sb_fdblocks corruption has appeared on four different >>> hosts affecting at most one third of xfs filesystems per host. Also >>> the corruption looks always the same, namely the 33rd bit flipped >>> which also seems suspicious. >> >> Running a debug kernel with memory poisoning, KASAN, or something similar might >> help catch it if it's a stray memory write of some sort... > > Thanks! BTW we've experienced this again on a FS at around 77% usage > and xfs_repair reports a flip in the 32nd bit (output below). We'll > enable memory poisoning on said host and see if other filesystems on > that host experience the same. > I see in the patch thread it has been mentioned this particular > condition will be checked and fixed/prevented in 4.19 though the root > cause isn't know (?) Yeah, the validation should happen in 4.19, but no idea what the root cause is. Catching it at write time may offer some clues, I hope, if the debug kernel doesn't help. -Eric