From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Kara Subject: Re: [Bug-tar] --sparse is broken on filesystems where small files may have zero blocks Date: Tue, 29 Oct 2013 18:37:13 +0100 Message-ID: <20131029173713.GB6087@quack.suse.cz> References: <20131028211739.GA26741@ti119.telemetry-investments.com> <526F376E.5030307@cs.ucla.edu> <10978717.9ApR7zKiv0@nb.usersys.redhat.com> <6917534.P50uorHOUu@nb.usersys.redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: bug-tar@gnu.org, "Andrew J. Schorr" , Paul Eggert , linux-fsdevel@vger.kernel.org, Andreas Dilger To: Pavel Raiskup Return-path: Received: from cantor2.suse.de ([195.135.220.15]:40656 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751243Ab3J2RhR (ORCPT ); Tue, 29 Oct 2013 13:37:17 -0400 Content-Disposition: inline In-Reply-To: <6917534.P50uorHOUu@nb.usersys.redhat.com> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On Tue 29-10-13 16:27:02, Pavel Raiskup wrote: > On Tuesday, October 29, 2013 09:59:56 Pavel Raiskup wrote: > > > #define ST_IS_SPARSE(st) \ > > > (ST_NBLOCKS (st) \ > > > - < ((st).st_size / ST_NBLOCKSIZE + ((st).st_size % ST_NBLOCKS= IZE !=3D 0))) > > > + < ((st).st_size / ST_NBLOCKSIZE \ > > > + + ((st).st_size % ST_NBLOCKSIZE !=3D 0 \ > > > + && (st).st_size / ST_NBLOCKSIZE !=3D 0))) > >=20 > > May the st.st_size / ST_NBLOCKSIZE be greater than 1 and data still= stored > > in inode directly? Seems like on ext4 filesystem it is not possibl= e [1] > > but does anybody know about exception? > >=20 > > [1] https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Inline_= Data >=20 > Well, I now recalled somehow relevant Red Hat bug, sorry I have not > mentioned it before: > https://bugzilla.redhat.com/show_bug.cgi?id=3D757557 >=20 > CC'ing fs-devel: The question is whether that ^^^^ is not a bug in > filesystem =E2=80=94 whether filesystem should not _always_ return to= fstat() > block count at least 1 if there are at least some data (even if these= data > are inlined in inode)? Just for catching the context, this thread st= arts > here: http://lists.gnu.org/archive/html/bug-tar/2013-10/msg00030.html So 'st_blocks' should be "the number of blocks allocated to the file, 512-byte units". If we are able to store the whole file within inode, w= e have no blocks allocated and thus setting st_blocks to 0 looks as a dec= ent thing to do. Looking into filesystems where this is possible (ext4, btr= fs, reiserfs) they will all set st_blocks to 0 if the file body is inlined = and the size is smaller than 512 bytes. But OTOH btrfs and reiserfs *do* in fact account the amount of space consumed by the file. It is just that stat(2) syscall reports the space= in 512-byte units and for historical reasons we ended up just truncating t= he byte-precision space counter instead of rounding it up (that is a mista= ke I made like 10 years ago :(). It is easy enough to start reporting the va= lue rounded up but I'm not sure if it won't break some userspace which alre= ady developped some dependency on this buggy kernel behavior. ext4 is yet a different matter. It does really report the number of allocated blocks in st_blocks so it will report 0 while data can fit in= to the inode (whose size is configurable during fs creation, default is 25= 6). In practice that will result in reporting non-zero st_blocks for 512-by= te and larger files anyways so there won't be an observable difference bet= ween what ext4 and btrfs / reiserfs do. But we might still want to fix up ex= t4 to be consistent with btrfs and reiserfs so that things are more future-proof. > If that is not a bug in fs, is there possible to detect that particul= ar > file is completely sparse? As Joerg wrote, SEEK_DATA / SEEK_HOLE is a proper interface for this = at least for systems that support it. Once you have called stat(2) on the file, inode will be in cache anyways so the additional cost of open(2), lseek(2), close(2) won't be that big. For systems that don't support SEEK_DATA / SEEK_HOLE, you can use some heuristic like: if (st.st_size < st.st_blksize || st.st_blocks > 0) /* Bite the bullet and scan the data for non-zero bytes */ else /* Assume the file is sparse */ Honza --=20 Jan Kara SUSE Labs, CR -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel= " in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html