[PATCH 00/39] e2fsprogs October 2014 patchbomb, part 6.1

[PATCH 00/39] e2fsprogs October 2014 patchbomb, part 6.1

[Darrick J. Wong]

Hi all,

This is a revised edition of part 6 of the 2014 e2fsprogs patchset.
I've fixed a number of bugs discovered since the last patchbomb in
September, added a couple more e2fsck peformance improvements, and
revised all the new features for 1.43 per our discussions here.

The first two patches fix some minor compiler warnings and make e2fuzz
a bit more aggressive in what it fuzzes.  Patches 3-6 fix three
complaints by Sami Liedes, and another silly bug I found in the
plausible.c's libmagic integration.

Patches 7-8 fix a couple of problems with metadata checksum support in
tune2fs -- the first greatly reduces the amount of work done to add
checksums to the extent tree, and the second makes it so that one
cannot change the metadata_csum feature bit when the filesystem is
mounted.

Patches 9-13 fix some bugs I found in resize2fs: various bookkeeping
errors with sparse_super2 support; a bug where internal extent tree
blocks would not have their checksums rewritten during an inode number
migration; the "insane RAID stride" bug that TR Reardon sent in; and a
patch to fix up the flags and unused inode count of the former last
block group when growing a filesystem.  (More on this in patch 18.)

Patch 14 teaches mke2fs not to zero inode table blocks that have been
discarded on a discard-zeroes-data device.

Patches 15-16 clean up the dumpe2fs output.

Patches 17-21 deal with e2fsck.  The first one fixes a Valgrind
complaint sent in by Sami Liedes.  The two patches after that (18-19)
teach e2fsck how to update a block group's INODE_UNINIT and
BLOCK_UNINIT flags for all groups that have no inodes/blocks
allocated, respectively; and it recalculates the unused inode count so
that future invocations of e2fsck can save time by not processing runs
of unused inodes.  This can improve e2fsck runtimes by up to 10% on
aged filesystems that have experienced massive inode die-offs in the
past.

The last two patches of this chunk are the e2fsck metadata readahead
patches, unchanged from September.

Patch 22 teaches ext2fs_new_block2() to call the alloc_block hook to
fix a bug in e2fsck where new blocks would be allocated from an
unverified block bitmap.  This is also left over from September.

Patches 23-24 hook up ext2fs_zero_blocks2 to the BLKZEROOUT blockdev
ioctl or the FALLOC_FL_PUNCH_HOLE feature of fallocate() to zero out
data blocks if possible.  This will be useful for zeroing inode
tables, clearing the journal, and the future ext2fs_fallocate API.
There's also a cleanup patch that ensure that the zero_blocks2 static
buffer gets cleaned up when the FS exits and converts each area that
was writing zero blocks to use the zero_blocks2 call instead.  These
two patches have been altered considerably to work around page cache
bugs related to BLK* ioctls, and to fix Ted's criticisms.

Patches 25-26 enhance ext2fs_bmap2() to allow the creation of
uninitialized extents.  The functionality is already there; this
simply adds a flag for clients to create uninitialized mappings.
There's also a patch to the fileio routines to handle uninitialized
extents.  These patches are unchanged from December 2013, aside from
having grown some more test cases.

Patches 27-28 add to resize2fs the ability to convert a filesystem to
and from 64bit mode.  These patches are unchanged from December 2013,
aside from having grown some more test cases.

Patches 29-33 implement fallocate for e2fsprogs, and modifies Ted's
mk_hugefiles functionality to use it.  The general fallocate API call
is (regrettably) much more complex than what hugefiles did, since it
must grapple with the possibility that the file already has mapped
blocks.  There were also a lot of bigalloc related subtleties; at some
point it might behoove someone to write a extent tree compressor.
The API call has been plumbed into debugfs, with accompanying tests of
both the fallocate and punch calls.

Patches 34-35 implement an interface in debugfs to fallocate and punch
so that we can provide some basic regression testing.

Patches 36-39 implement fuse2fs, a FUSE server based on libext2fs.
Primarily I've been using it to shake out bugs in the library via
xfstests and the metadata checksumming test program.  It can also be
used to mount ext4 on any OS supporting FUSE, and it can also mount
64k-block filesystems on x86, though I'd be wary of using rw mode.
fuse2fs depends on these new APIs: xattr editing, uninit extent
handling, and the new fallocate call.

I've tested these e2fsprogs changes against the -next branch as of
10/10.  The patches have been tested against the 'make check' suite
and some amount of e2fuzz testing on x86_64, i686, ppc64, and aarch64.

Comments and questions are, as always, welcome.

--D

[PATCH 01/39] misc: fix compiler warnings

[Darrick J. Wong]

Fix some gcc-4.8 warnings.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass5.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


diff --git a/e2fsck/pass5.c b/e2fsck/pass5.c
index 8d6032b..64fb7fe 100644
--- a/e2fsck/pass5.c
+++ b/e2fsck/pass5.c
@@ -84,7 +84,7 @@ void e2fsck_pass5(e2fsck_t ctx)
 static void check_inode_bitmap_checksum(e2fsck_t ctx)
 {
 	struct problem_context	pctx;
-	char		*buf;
+	char		*buf = NULL;
 	dgrp_t		i;
 	int		nbytes;
 	ext2_ino_t	ino_itr;
@@ -139,7 +139,7 @@ static void check_inode_bitmap_checksum(e2fsck_t ctx)
 static void check_block_bitmap_checksum(e2fsck_t ctx)
 {
 	struct problem_context	pctx;
-	char		*buf;
+	char		*buf = NULL;
 	dgrp_t		i;
 	int		nbytes;
 	blk64_t		blk_itr;

[PATCH 02/39] e2fuzz: exercise fuzzed blocks more aggressively

[Darrick J. Wong]

Exercise fuzzed metadata blocks more aggressively by expanding up to
50000 files (instead of just test.1, which might not hit anything).
Fix a typo while we're at it.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/e2fuzz.sh |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)


diff --git a/misc/e2fuzz.sh b/misc/e2fuzz.sh
index a261d5a..4cb7b61 100755
--- a/misc/e2fuzz.sh
+++ b/misc/e2fuzz.sh
@@ -34,7 +34,7 @@ print_help() {
 	echo "-p:	Use system's mke2fs/e2fsck/tune2fs tools."
 	echo "-s:	Create FS images of this size. (${SZ})"
 	echo "-S:	Copy files from this dir. (${SRCDIR})"
-	echo "-x:	Run e2fck at most this many times. (${MAX_FSCK})"
+	echo "-x:	Run e2fsck at most this many times. (${MAX_FSCK})"
 	test "${HAS_FUSE2FS}" -gt 0 && echo "-u:	Use fuse2fs instead of the kernel."
 	exit 0
 }
@@ -192,12 +192,12 @@ seq 1 "${PASSES}" | while read pass; do
 		find "${TESTMNT}/test.1/" -type f -size -1048576k -print0 | xargs -0 cat > /dev/null 2>> "${OPS_LOG}"
 
 		echo "+++ expand"
-		find "${TESTMNT}/test.1/" -type f 2> /dev/null | while read f; do
+		find "${TESTMNT}/" -type f 2> /dev/null | head -n 50000 | while read f; do
 			attr -l "$f" > /dev/null 2>> "${OPS_LOG}"
-			mv "$f" "$f.longer" > /dev/null 2>> "${OPS_LOG}"
 			if [ -f "$f" -a -w "$f" ]; then
 				dd if=/dev/zero bs="${BLK_SZ}" count=1 >> "$f" 2>> "${OPS_LOG}"
 			fi
+			mv "$f" "$f.longer" > /dev/null 2>> "${OPS_LOG}"
 		done
 		sync
 

[PATCH 03/39] libext2fs: directory iteration mustn't walk off the buffer end

[Darrick J. Wong]

When we're iterating a directory, the loop control code reads the
length of the next directory record, failing to account for the fact
that there must be at least 8 bytes (the minimum size of a directory
entry) left in the buffer to read the next directory record.  Fix the
loop conditional so that we don't read off the end of the buffer.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reported-by: Sami Liedes <sami.liedes@iki.fi>
---
 lib/ext2fs/dir_iterate.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/lib/ext2fs/dir_iterate.c b/lib/ext2fs/dir_iterate.c
index 67152cc..9063c17 100644
--- a/lib/ext2fs/dir_iterate.c
+++ b/lib/ext2fs/dir_iterate.c
@@ -222,7 +222,7 @@ int ext2fs_process_dir_block(ext2_filsys fs,
 					EXT4_FEATURE_RO_COMPAT_METADATA_CSUM))
 		csum_size = sizeof(struct ext2_dir_entry_tail);
 
-	while (offset < buflen) {
+	while (offset < buflen - 8) {
 		dirent = (struct ext2_dir_entry *) (ctx->buf + offset);
 		if (ext2fs_get_rec_len(fs, dirent, &rec_len))
 			return BLOCK_ABORT;

[PATCH 04/39] libext2fs: zero the EA block buffer before filling it

[Darrick J. Wong]

When writing an extended attribute (EA) block, it's quite possible
that the EA formatting code will not write the entire buffer.
Therefore, we must zero the buffer beforehand to avoid writing random
heap contents to disk.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reported-by: Sami Liedes <sami.liedes@iki.fi>
---
 lib/ext2fs/ext_attr.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/lib/ext2fs/ext_attr.c b/lib/ext2fs/ext_attr.c
index b52abb5..c397e00 100644
--- a/lib/ext2fs/ext_attr.c
+++ b/lib/ext2fs/ext_attr.c
@@ -550,7 +550,7 @@ errcode_t ext2fs_xattrs_write(struct ext2_xattr_handle *handle)
 
 write_ea_block:
 	/* Write the EA block */
-	err = ext2fs_get_mem(handle->fs->blocksize, &block_buf);
+	err = ext2fs_get_memzero(handle->fs->blocksize, &block_buf);
 	if (err)
 		goto out;
 

[PATCH 05/39] libext2fs: don't memcpy identical pointers when writing a cache block

[Darrick J. Wong]

Sami Liedes found a scenario where we could memcpy incorrectly:

If a block read fails during an e2fsck run, the UNIX IO manager will
call the io->read_error routine with a pointer to the internal block
cache.  The e2fsck read error handler immediately tries to write the
buffer back out to disk(!), at which point the block write code will
try to copy the buffer contents back into the block cache.  Normally
this is fine, but not when the write buffer is the cache itself!

So, plumb in a trivial check for this condition.  A more thorough
solution would pass a duplicated buffer to the IO error handlers, but
I don't know if that happens frequently enough to be worth the extra
point of failure.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reported-by: Sami Liedes <sami.liedes@iki.fi>
---
 lib/ext2fs/unix_io.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


diff --git a/lib/ext2fs/unix_io.c b/lib/ext2fs/unix_io.c
index 189adce..c3a8ea5 100644
--- a/lib/ext2fs/unix_io.c
+++ b/lib/ext2fs/unix_io.c
@@ -823,7 +823,8 @@ static errcode_t unix_write_blk64(io_channel channel, unsigned long long block,
 			cache = reuse;
 			reuse_cache(channel, data, cache, block);
 		}
-		memcpy(cache->buf, cp, channel->block_size);
+		if (cache->buf != cp)
+			memcpy(cache->buf, cp, channel->block_size);
 		cache->dirty = !writethrough;
 		count--;
 		block++;

[PATCH 06/39] misc: fix broken libmagic interaction with plausibility check

[Darrick J. Wong]

If we get as far as calling libmagic, return the correct error code so
that mkfs asks for confirmation if libmagic finds something and
doesn't ask if nothing is found.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/plausible.c                     |    9 ++++++---
 tests/f_detect_junk/expect           |    6 +++---
 tests/f_detect_junk/expect.nodebugfs |    4 ++--
 tests/f_detect_junk/script           |    2 +-
 4 files changed, 12 insertions(+), 9 deletions(-)


diff --git a/misc/plausible.c b/misc/plausible.c
index 00bc6e2..1848a26 100644
--- a/misc/plausible.c
+++ b/misc/plausible.c
@@ -243,6 +243,7 @@ int check_plausibility(const char *device, int flags, int *ret_is_dev)
 	if ((flags & CHECK_FS_EXIST) && magic_library_available()) {
 		const char *msg;
 		magic_t mag;
+		int has_magic = 0;
 
 		mag = dl_magic_open(MAGIC_RAW | MAGIC_SYMLINK | MAGIC_DEVICES |
 				    MAGIC_ERROR | MAGIC_NO_CHECK_ELF |
@@ -250,11 +251,13 @@ int check_plausibility(const char *device, int flags, int *ret_is_dev)
 		dl_magic_load(mag, NULL);
 
 		msg = dl_magic_file(mag, device);
-		if (msg && strcmp(msg, "data") && strcmp(msg, "empty"))
-			printf(_("%s contains a `%s'\n"), device, msg);
+		if (msg && strcmp(msg, "data") && strcmp(msg, "empty")) {
+			printf(_("%s contains `%s' data\n"), device, msg);
+			has_magic = 1;
+		}
 
 		dl_magic_close(mag);
-		return 0;
+		return !has_magic;
 	}
 #endif
 
diff --git a/tests/f_detect_junk/expect b/tests/f_detect_junk/expect
index 57f7f89..f3300de 100644
--- a/tests/f_detect_junk/expect
+++ b/tests/f_detect_junk/expect
@@ -11,13 +11,13 @@ is corrupt, and you might try running e2fsck with an alternate superblock:
  or
     e2fsck -b 32768 <device>
 
-test.img contains a `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced'
+test.img contains `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced' data
 *** debugfs
 test.img: Bad magic number in super-block while opening filesystem
-test.img contains a `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced'
+test.img contains `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced' data
 *** tune2fs
 ../misc/tune2fs: Bad magic number in super-block while trying to open test.img
-test.img contains a `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced'
+test.img contains `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced' data
 *** mke2fs
 Creating filesystem with 16384 1k blocks and 4096 inodes
 Superblock backups stored on blocks: 
diff --git a/tests/f_detect_junk/expect.nodebugfs b/tests/f_detect_junk/expect.nodebugfs
index d9281a0..0d4ba54 100644
--- a/tests/f_detect_junk/expect.nodebugfs
+++ b/tests/f_detect_junk/expect.nodebugfs
@@ -11,11 +11,11 @@ is corrupt, and you might try running e2fsck with an alternate superblock:
  or
     e2fsck -b 32768 <device>
 
-test.img contains a `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced'
+test.img contains `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced' data
 *** debugfs
 *** tune2fs
 ../misc/tune2fs: Bad magic number in super-block while trying to open test.img
-test.img contains a `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced'
+test.img contains `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced' data
 *** mke2fs
 Creating filesystem with 16384 1k blocks and 4096 inodes
 Superblock backups stored on blocks: 
diff --git a/tests/f_detect_junk/script b/tests/f_detect_junk/script
index 8409fdd..3aebcd0 100644
--- a/tests/f_detect_junk/script
+++ b/tests/f_detect_junk/script
@@ -1,6 +1,6 @@
 #!/bin/bash
 
-if [ "$(grep -c 'define HAVE_MAGIC_H' $test_dir/../../lib/config.h)" -gt 0 ]; then
+if [ "$(grep -c 'define HAVE_MAGIC_H' ../lib/config.h)" -gt 0 ]; then
 
 FSCK_OPT=-fn
 IMAGE=$test_dir/image.bz2

[PATCH 07/39] tune2fs: speed up rewriting extent tree when enabling metadata_csum

[Darrick J. Wong]

When enabling checksums, tune2fs naively rewrites every extent in the
entire tree!  This is unnecessary since we only need to rewrite each
extent tree block; therefore, only rewrite the extent if it's the
first one in an internal extent tree block.

Also, don't bother iterating the extent tree when clearing checksums.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/tune2fs.c |   49 +++++++++++++++++++++++++++++++++----------------
 1 file changed, 33 insertions(+), 16 deletions(-)


diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index d17c8de..ecbdea0 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -442,39 +442,56 @@ static errcode_t rewrite_extents(ext2_filsys fs, ext2_ino_t ino,
 {
 	ext2_extent_handle_t	handle;
 	struct ext2fs_extent	extent;
-	int			op = EXT2_EXTENT_ROOT;
 	errcode_t		errcode;
+	struct ext2_extent_info	info;
 
-	if (!(inode->i_flags & EXT4_EXTENTS_FL))
+	if (!(inode->i_flags & EXT4_EXTENTS_FL) ||
+	    !EXT2_HAS_RO_COMPAT_FEATURE(fs->super,
+					EXT4_FEATURE_RO_COMPAT_METADATA_CSUM))
 		return 0;
 
 	errcode = ext2fs_extent_open(fs, ino, &handle);
 	if (errcode)
 		return errcode;
 
-	while (1) {
-		errcode = ext2fs_extent_get(handle, op, &extent);
+	errcode = ext2fs_extent_get(handle, EXT2_EXTENT_ROOT, &extent);
+	if (errcode)
+		goto out;
+
+	do {
+		errcode = ext2fs_extent_get_info(handle, &info);
 		if (errcode)
 			break;
 
-		/* Root node is in the separately checksummed inode */
-		if (op == EXT2_EXTENT_ROOT) {
-			op = EXT2_EXTENT_NEXT;
-			continue;
+		/*
+		 * If this is the first extent in an extent block that we
+		 * haven't visited, rewrite the extent to force the ETB
+		 * checksum to be rewritten.
+		 */
+		if (info.curr_entry == 1 && info.curr_level != 0 &&
+		    !(extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT)) {
+			errcode = ext2fs_extent_replace(handle, 0, &extent);
+			if (errcode)
+				break;
 		}
-		op = EXT2_EXTENT_NEXT;
 
-		/* Only visit the first extent in each extent block */
-		if (extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT)
-			continue;
-		errcode = ext2fs_extent_replace(handle, 0, &extent);
-		if (errcode)
-			break;
-	}
+		/* Skip to the end of a block of leaf nodes */
+		if (extent.e_flags & EXT2_EXTENT_FLAGS_LEAF) {
+			errcode = ext2fs_extent_get(handle,
+						    EXT2_EXTENT_LAST_SIB,
+						    &extent);
+			if (errcode)
+				break;
+		}
 
+		errcode = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT, &extent);
+	} while (errcode == 0);
+
+out:
 	/* Ok if we run off the end */
 	if (errcode == EXT2_ET_EXTENT_NO_NEXT)
 		errcode = 0;
+	ext2fs_extent_free(handle);
 	return errcode;
 }
 

[PATCH 08/39] tune2fs: don't change metadata_csum on a mounted fs

[Darrick J. Wong]

Don't let users change metadata_csum on a mounted filesystem because
there's no way to tell the kernel to turn on the feature; there's no
way to prevent the kernel from rewriting on-disk structures while
tune2fs is also rewriting them; and there's no way to tell the kernel
to reload them after tune2fs is finished.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/tune2fs.c |    6 ++++++
 1 file changed, 6 insertions(+)


diff --git a/misc/tune2fs.c b/misc/tune2fs.c
index ecbdea0..7fee870 100644
--- a/misc/tune2fs.c
+++ b/misc/tune2fs.c
@@ -1110,6 +1110,9 @@ mmp_error:
 		       EXT4_FEATURE_RO_COMPAT_METADATA_CSUM)) {
 		if (check_fsck_needed(fs))
 			exit(1);
+		if (mount_flags & EXT2_MF_MOUNTED)
+			fputs(_("Cannot enable metadata_csum on a mounted "
+				"filesystem!\n"), stderr);
 		rewrite_checksums = 1;
 		/* metadata_csum supersedes uninit_bg */
 		fs->super->s_feature_ro_compat &=
@@ -1132,6 +1135,9 @@ mmp_error:
 			EXT4_FEATURE_RO_COMPAT_METADATA_CSUM)) {
 		if (check_fsck_needed(fs))
 			exit(1);
+		if (mount_flags & EXT2_MF_MOUNTED)
+			fputs(_("Cannot disable metadata_csum on a mounted "
+				"filesystem!\n"), stderr);
 		rewrite_checksums = 1;
 		/*
 		 * If we're turning off metadata_csum and not turning on

[PATCH 09/39] resize2fs: don't exit if shrinking sparse_super2 fs to one bg

[Darrick J. Wong]

If we're shrinking a sparse_super2 filesystem to a single block group,
the superblock will be in block 0.  This is perfectly valid (for block
group 0 with a blocksize > 1024) so don't exit.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 resize/resize2fs.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index b59f482..ff18817 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -2126,7 +2126,7 @@ static errcode_t reserve_sparse_super2_last_group(ext2_resize_t rfs,
 	if (retval)
 		return retval;
 
-	if (!sb) {
+	if (last_bg && !sb) {
 		fputs(_("Should never happen!  No sb in last super_sparse bg?\n"),
 		      stderr);
 		exit(1);

[PATCH 10/39] resize2fs: quickly rewrite extent blocks when moving an inode w/ metadata_csum

[Darrick J. Wong]

When we're moving an inode on a metadata_csum filesystem, we need to
rewrite the checksum of all interior nodes of the extent tree.  The
current code does this inefficiently via set_bmap, but we can do this
more efficiently through direct iteration of the extent tree.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 resize/resize2fs.c |   74 +++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 64 insertions(+), 10 deletions(-)


diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index ff18817..212fee2 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -1492,16 +1492,6 @@ static int process_block(ext2_filsys fs, blk64_t	*block_nr,
 		}
 	}
 
-	/*
-	 * If we moved inodes and metadata_csum is enabled, we must force the
-	 * extent block to be rewritten with new checksum.
-	 */
-	if (EXT2_HAS_RO_COMPAT_FEATURE(fs->super,
-				       EXT4_FEATURE_RO_COMPAT_METADATA_CSUM) &&
-	    pb->has_extents &&
-	    pb->old_ino != pb->ino)
-		ret |= BLOCK_CHANGED;
-
 	if (pb->is_dir) {
 		retval = ext2fs_add_dir_block2(fs->dblist, pb->ino,
 					       block, (int) blockcnt);
@@ -1581,6 +1571,59 @@ out:
 	return err;
 }
 
+/* Rewrite extents */
+static errcode_t rewrite_extents(ext2_filsys fs, ext2_ino_t ino)
+{
+	ext2_extent_handle_t	handle;
+	struct ext2fs_extent	extent;
+	errcode_t		errcode;
+	struct ext2_extent_info	info;
+
+	errcode = ext2fs_extent_open(fs, ino, &handle);
+	if (errcode)
+		return errcode;
+
+	errcode = ext2fs_extent_get(handle, EXT2_EXTENT_ROOT, &extent);
+	if (errcode)
+		goto out;
+
+	do {
+		errcode = ext2fs_extent_get_info(handle, &info);
+		if (errcode)
+			break;
+
+		/*
+		 * If this is the first extent in an extent block that we
+		 * haven't visited, rewrite the extent to force the ETB
+		 * checksum to be rewritten.
+		 */
+		if (info.curr_entry == 1 && info.curr_level != 0 &&
+		    !(extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT)) {
+			errcode = ext2fs_extent_replace(handle, 0, &extent);
+			if (errcode)
+				break;
+		}
+
+		/* Skip to the end of a block of leaf nodes */
+		if (extent.e_flags & EXT2_EXTENT_FLAGS_LEAF) {
+			errcode = ext2fs_extent_get(handle,
+						    EXT2_EXTENT_LAST_SIB,
+						    &extent);
+			if (errcode)
+				break;
+		}
+
+		errcode = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT, &extent);
+	} while (errcode == 0);
+
+out:
+	/* Ok if we run off the end */
+	if (errcode == EXT2_ET_EXTENT_NO_NEXT)
+		errcode = 0;
+	ext2fs_extent_free(handle);
+	return errcode;
+}
+
 static errcode_t inode_scan_and_fix(ext2_resize_t rfs)
 {
 	struct process_block_struct	pb;
@@ -1699,6 +1742,17 @@ remap_blocks:
 		if (retval)
 			goto errout;
 
+		/* Rewrite extent block checksums with new inode number */
+		if (EXT2_HAS_RO_COMPAT_FEATURE(rfs->old_fs->super,
+				       EXT4_FEATURE_RO_COMPAT_METADATA_CSUM) &&
+		    (inode->i_flags & EXT4_EXTENTS_FL)) {
+			rfs->old_fs->flags |= EXT2_FLAG_IGNORE_CSUM_ERRORS;
+			retval = rewrite_extents(rfs->old_fs, new_inode);
+			rfs->old_fs->flags &= ~EXT2_FLAG_IGNORE_CSUM_ERRORS;
+			if (retval)
+				goto errout;
+		}
+
 		/*
 		 * Update inodes to point to new blocks; schedule directory
 		 * blocks for inode remapping.  Need to write out dir blocks

[PATCH 11/39] resize2fs: use old_fs to detect per-bg metadata blocks to free

[Darrick J. Wong]

When shrinking a filesystem, resize2fs wants to free per-bg metadata
blocks that are no longer needed.  This behavior is gated on whether
there's a superblock in the group as told by new_fs.  The check really
should be against old_fs, since we're effectively freeing blocks out
of old_fs in the transition to new_fs, but prior to sparse_super2 this
didn't matter since superblocks didn't move, so it didn't matter.

Under sparse_super2, however, there's a superblock in the last group,
so now we need to change the test to use old_fs as it should.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 resize/resize2fs.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 212fee2..a8a6850 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -1042,7 +1042,7 @@ static errcode_t blocks_to_move(ext2_resize_t rfs)
 		}
 
 		for (i = 0; i < max_groups; i++) {
-			if (!ext2fs_bg_has_super(fs, i)) {
+			if (!ext2fs_bg_has_super(old_fs, i)) {
 				group_blk += fs->super->s_blocks_per_group;
 				continue;
 			}

[PATCH 12/39] resize2fs: don't interpret bitmap shift while crossing flexbg as raid stride

[Darrick J. Wong]

resize2fs tries to infer the RAID stride by observing differences between the
locations of adjacent block groups' block and inode bitmaps within the block
group.  If the two block groups being compared belong to different flexbgs,
however, it'll be fooled by the large offset into thinking that the FS has an
abnormally large RAID stride.

Therefore, teach it not to get confused by crossing a flexbg.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reported-by: TR Reardon <thomas_reardon@hotmail.com>
---
 resize/main.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)


diff --git a/resize/main.c b/resize/main.c
index 983d8c2..e017792 100644
--- a/resize/main.c
+++ b/resize/main.c
@@ -105,6 +105,7 @@ static void determine_fs_stride(ext2_filsys fs)
 	unsigned long long sum;
 	unsigned int	has_sb, prev_has_sb = 0, num;
 	int		i_stride, b_stride;
+	int		flexbg_size = 1 << fs->super->s_log_groups_per_flex;
 
 	if (fs->stride)
 		return;
@@ -120,7 +121,8 @@ static void determine_fs_stride(ext2_filsys fs)
 			ext2fs_inode_bitmap_loc(fs, group - 1) -
 			fs->super->s_blocks_per_group;
 		if (b_stride != i_stride ||
-		    b_stride < 0)
+		    b_stride < 0 ||
+		    (flexbg_size > 1 && (group % flexbg_size == 0)))
 			goto next;
 
 		/* printf("group %d has stride %d\n", group, b_stride); */

[PATCH 13/39] resize2fs: set block_uninit in former last bg when expanding fs

[Darrick J. Wong]

If expanding a filesystem and the last bg of the old fs has no blocks
in use, sets the block_uninit flag.  This might save a small amount of
time at mount, but will be particularly useful if e2fsck ever starts
to verify that block_uninit is set for empty block groups.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 resize/resize2fs.c              |   54 +++++++++++++++++++++++++++++++++++++++
 tests/r_fixup_lastbg/expect     |   36 ++++++++++++++++++++++++++
 tests/r_fixup_lastbg/script     |   37 +++++++++++++++++++++++++++
 tests/r_fixup_lastbg_big/expect |   39 ++++++++++++++++++++++++++++
 tests/r_fixup_lastbg_big/script |   37 +++++++++++++++++++++++++++
 5 files changed, 203 insertions(+)
 create mode 100644 tests/r_fixup_lastbg/expect
 create mode 100755 tests/r_fixup_lastbg/script
 create mode 100644 tests/r_fixup_lastbg_big/expect
 create mode 100755 tests/r_fixup_lastbg_big/script


diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index a8a6850..e71acf3 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -50,6 +50,7 @@ static errcode_t inode_ref_fix(ext2_resize_t rfs);
 static errcode_t move_itables(ext2_resize_t rfs);
 static errcode_t fix_resize_inode(ext2_filsys fs);
 static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs);
+static errcode_t fixup_last_bg(ext2_resize_t rfs);
 static errcode_t fix_sb_journal_backup(ext2_filsys fs);
 static errcode_t mark_table_blocks(ext2_filsys fs,
 				   ext2fs_block_bitmap bmap);
@@ -180,6 +181,9 @@ errcode_t resize_fs(ext2_filsys fs, blk64_t *new_size, int flags,
 	retval = ext2fs_calculate_summary_stats(rfs->new_fs);
 	if (retval)
 		goto errout;
+	retval = fixup_last_bg(rfs);
+	if (retval)
+		goto errout;
 	print_resource_track(rfs, &rtrack, fs->io);
 
 	init_resource_track(&rtrack, "fix_resize_inode", fs->io);
@@ -2388,6 +2392,56 @@ static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs)
 }
 
 /*
+ * Fix up the bg flags + ununsed inode count in the last group of the fs if
+ * we're expanding.
+ */
+static errcode_t fixup_last_bg(ext2_resize_t rfs)
+{
+	dgrp_t		group;
+	ext2_ino_t	start, end, j, new_unused;
+	int		dirty = 0;
+	ext2_filsys	fs = rfs->new_fs;
+
+	if (rfs->old_fs->group_desc_count > rfs->new_fs->group_desc_count ||
+	    !ext2fs_has_group_desc_csum(fs))
+		return 0;
+
+	group = rfs->old_fs->group_desc_count - 1;
+	if (fs->super->s_blocks_per_group ==
+	    ext2fs_bg_free_blocks_count(fs, group)) {
+		ext2fs_bg_flags_set(fs, group, EXT2_BG_BLOCK_UNINIT);
+		dirty = 1;
+	}
+
+	if (fs->super->s_inodes_per_group ==
+	    ext2fs_bg_free_inodes_count(fs, group)) {
+		ext2fs_bg_itable_unused_set(fs, group,
+					    fs->super->s_inodes_per_group);
+		ext2fs_bg_flags_set(fs, group, EXT2_BG_INODE_UNINIT);
+		dirty = 1;
+	} else {
+		start = (fs->super->s_inodes_per_group * group) + 1;
+		end = (fs->super->s_inodes_per_group * (group + 1));
+		end -= ext2fs_bg_itable_unused(fs, group);
+		for (j = end; j >= start; j--)
+			if (ext2fs_test_inode_bitmap2(fs->inode_map, j))
+				break;
+
+		new_unused = fs->super->s_inodes_per_group - (j - start + 1);
+		if (ext2fs_bg_itable_unused(fs, group) != new_unused) {
+			ext2fs_bg_itable_unused_set(fs, group, new_unused);
+			dirty = 1;
+		}
+	}
+
+	if (dirty) {
+		ext2fs_group_desc_csum_set(fs, group);
+		ext2fs_mark_super_dirty(fs);
+	}
+	return 0;
+}
+
+/*
  *  Journal may have been relocated; update the backup journal blocks
  *  in the superblock.
  */
diff --git a/tests/r_fixup_lastbg/expect b/tests/r_fixup_lastbg/expect
new file mode 100644
index 0000000..f9945b3
--- /dev/null
+++ b/tests/r_fixup_lastbg/expect
@@ -0,0 +1,36 @@
+Creating filesystem with 20000 1k blocks and 1248 inodes
+Superblock backups stored on blocks: 
+	8193
+
+Allocating group tables:    \b\b\bdone                            
+Writing inode tables:    \b\b\bdone                            
+Creating journal (1024 blocks): done
+Writing superblocks and filesystem accounting information:    \b\b\bdone
+
+Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Group 2: (Blocks 16385-19999)
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Resizing the filesystem on test.img to 20004 (1k) blocks.
+The filesystem on test.img is now 20004 (1k) blocks long.
+
+Group 2: (Blocks 16385-20003) [INODE_UNINIT]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3619 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-20003
+  Free inodes: 833-1248
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 11/1248 files (0.0% non-contiguous), 1517/20004 blocks
diff --git a/tests/r_fixup_lastbg/script b/tests/r_fixup_lastbg/script
new file mode 100755
index 0000000..ad223d4
--- /dev/null
+++ b/tests/r_fixup_lastbg/script
@@ -0,0 +1,37 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+test_description="fix up last bg when expanding within the last bg"
+
+EXP=$test_dir/expect
+OUT=$test_name.out
+LOG=$test_name.log
+E2FSCK=../e2fsck/e2fsck
+
+$MKE2FS -T ext4 -b 1024 -F -U 56d3ee50-8532-4f29-8181-d7c6ea4a94a6 $TMPFILE 20000 > $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$DEBUGFS -R "set_bg 2 itable_unused 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 flags 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 checksum 0xd318" -w $TMPFILE > /dev/null 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+dd if=/dev/zero of=$TMPFILE bs=1 count=1 seek=$((1024 * 20004)) conv=notrunc >> $OUT 2> /dev/null
+$RESIZE2FS_EXE -f -p $TMPFILE >> $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$E2FSCK -fy $TMPFILE >> $OUT 2>&1
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" < $OUT > $LOG
+rm -rf $OUT
+
+cmp -s $LOG $EXP
+RC=$?
+if [ $RC -eq 0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff -u $EXP $LOG > $test_name.failed
+fi
+
+unset EXP LOG OUT E2FSCK
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
diff --git a/tests/r_fixup_lastbg_big/expect b/tests/r_fixup_lastbg_big/expect
new file mode 100644
index 0000000..1aa36fc
--- /dev/null
+++ b/tests/r_fixup_lastbg_big/expect
@@ -0,0 +1,39 @@
+Creating filesystem with 20000 1k blocks and 1248 inodes
+Superblock backups stored on blocks: 
+	8193
+
+Allocating group tables:    \b\b\bdone                            
+Writing inode tables:    \b\b\bdone                            
+Creating journal (1024 blocks): done
+Writing superblocks and filesystem accounting information:    \b\b\bdone
+
+Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Group 2: (Blocks 16385-19999)
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Resizing the filesystem on test.img to 40000 (1k) blocks.
+The filesystem on test.img is now 40000 (1k) blocks long.
+
+Group 2: (Blocks 16385-24576) [INODE_UNINIT, BLOCK_UNINIT]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  8192 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-24576
+  Free inodes: 833-1248
+Group 3: (Blocks 24577-32768) [INODE_UNINIT]
+  Backup superblock at 24577, Group descriptors at 24578-24578
+  Reserved GDT blocks at 24579-24656
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 11/2080 files (0.0% non-contiguous), 1809/40000 blocks
diff --git a/tests/r_fixup_lastbg_big/script b/tests/r_fixup_lastbg_big/script
new file mode 100755
index 0000000..4c4a351
--- /dev/null
+++ b/tests/r_fixup_lastbg_big/script
@@ -0,0 +1,37 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+test_description="fix up last bg when expanding beyond the last bg"
+
+EXP=$test_dir/expect
+OUT=$test_name.out
+LOG=$test_name.log
+E2FSCK=../e2fsck/e2fsck
+
+$MKE2FS -T ext4 -b 1024 -F -U 56d3ee50-8532-4f29-8181-d7c6ea4a94a6 $TMPFILE 20000 > $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$DEBUGFS -R "set_bg 2 itable_unused 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 flags 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 checksum 0xd318" -w $TMPFILE > /dev/null 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+dd if=/dev/zero of=$TMPFILE bs=1 count=1 seek=$((1024 * 40000)) conv=notrunc >> $OUT 2> /dev/null
+$RESIZE2FS_EXE -f -p $TMPFILE >> $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$E2FSCK -fy $TMPFILE >> $OUT 2>&1
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" < $OUT > $LOG
+rm -rf $OUT
+
+cmp -s $LOG $EXP
+RC=$?
+if [ $RC -eq 0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff -u $EXP $LOG > $test_name.failed
+fi
+
+unset EXP LOG OUT E2FSCK
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi

[PATCH 14/39] mke2fs: don't zero inode table blocks that are already zeroed

[Darrick J. Wong]

At mke2fs time, if we discard the device and discard zeroes data,
don't bother zeroing the inode table blocks a second time.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/mke2fs.c |   14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)


diff --git a/misc/mke2fs.c b/misc/mke2fs.c
index a7bd35a..c6a2c2f 100644
--- a/misc/mke2fs.c
+++ b/misc/mke2fs.c
@@ -421,12 +421,14 @@ static void write_inode_tables(ext2_filsys fs, int lazy_flag, int itable_zeroed)
 			ext2fs_bg_flags_set(fs, i, EXT2_BG_INODE_ZEROED);
 			ext2fs_group_desc_csum_set(fs, i);
 		}
-		retval = ext2fs_zero_blocks2(fs, blk, num, &blk, &num);
-		if (retval) {
-			fprintf(stderr, _("\nCould not write %d "
-				  "blocks in inode table starting at %llu: %s\n"),
-				num, blk, error_message(retval));
-			exit(1);
+		if (!itable_zeroed) {
+			retval = ext2fs_zero_blocks2(fs, blk, num, &blk, &num);
+			if (retval) {
+				fprintf(stderr, _("\nCould not write %d "
+					  "blocks in inode table starting at %llu: %s\n"),
+					num, blk, error_message(retval));
+				exit(1);
+			}
 		}
 		if (sync_kludge) {
 			if (sync_kludge == 1)

[PATCH 15/39] dumpe2fs: 80 column outputs, please

[Darrick J. Wong]

Try to reduce dumpe2fs output to 80 columns or less.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/dumpe2fs.c                                    |    6 +++++-
 tests/f_jnl_errno/expect.0                         |    3 ++-
 tests/j_short_revoke_trans/script                  |    2 +-
 tests/j_short_revoke_trans_mcsum_64bit/script      |    2 +-
 tests/j_short_trans/script                         |    2 +-
 tests/j_short_trans_64bit/script                   |    2 +-
 tests/j_short_trans_mcsum_64bit/script             |    2 +-
 tests/j_short_trans_old_csum/script                |    2 +-
 tests/j_short_trans_open_recover/script            |    2 +-
 tests/j_short_trans_recover/script                 |    2 +-
 tests/j_short_trans_recover_mcsum_64bit/script     |    2 +-
 tests/j_short_uncommitted_trans/script             |    2 +-
 tests/j_short_uncommitted_trans_mcsum_64bit/script |    2 +-
 tests/m_64bit_flexbg/expect.1                      |    3 ++-
 tests/m_bigjournal/script                          |    3 +++
 tests/m_dasd_bs/expect.1                           |    6 ++++--
 tests/m_desc_size_128/script                       |    3 +++
 tests/m_extent_journal/script                      |    3 +++
 tests/m_large_file/expect.1                        |    3 ++-
 tests/m_meta_bg/script                             |    3 +++
 tests/m_mmp/expect.1                               |    6 ++++--
 tests/m_no_opt/script                              |    3 +++
 tests/m_quota/script                               |    3 +++
 tests/m_raid_opt/script                            |    3 +++
 tests/m_root_owner/expect.1                        |    3 ++-
 tests/m_std/script                                 |    3 +++
 tests/m_uninit/script                              |    3 +++
 tests/r_fixup_lastbg/expect                        |    9 ++++++---
 tests/r_fixup_lastbg_big/expect                    |   10 ++++++----
 tests/r_move_itable/script                         |    8 ++++----
 tests/r_resize_inode/script                        |    6 +++---
 31 files changed, 78 insertions(+), 34 deletions(-)


diff --git a/misc/dumpe2fs.c b/misc/dumpe2fs.c
index e565f2d..f39cb3c 100644
--- a/misc/dumpe2fs.c
+++ b/misc/dumpe2fs.c
@@ -264,7 +264,11 @@ static void list_desc(ext2_filsys fs, int grp_only)
 		    EXT4_FEATURE_RO_COMPAT_METADATA_CSUM)
 			printf(_(", csum 0x%08x"),
 			       ext2fs_block_bitmap_checksum(fs, i));
-		fputs(_(", Inode bitmap at "), stdout);
+		if (getenv("DUMPE2FS_IGNORE_80COL"))
+			fputs(_(","), stdout);
+		else
+			fputs(_("\n "), stdout);
+		fputs(_(" Inode bitmap at "), stdout);
 		print_number(ext2fs_inode_bitmap_loc(fs, i));
 		print_bg_rel_offset(fs, ext2fs_inode_bitmap_loc(fs, i), 0,
 				    first_block, last_block);
diff --git a/tests/f_jnl_errno/expect.0 b/tests/f_jnl_errno/expect.0
index 6dad72a..2a9426d 100644
--- a/tests/f_jnl_errno/expect.0
+++ b/tests/f_jnl_errno/expect.0
@@ -40,7 +40,8 @@ Journal start:            0
 Group 0: (Blocks 1-8191) [ITABLE_ZEROED]
   Primary superblock at 1, Group descriptors at 2-2
   Reserved GDT blocks at 3-33
-  Block bitmap at 34 (+33), Inode bitmap at 50 (+49)
+  Block bitmap at 34 (+33)
+  Inode bitmap at 50 (+49)
   Inode table at 66-321 (+65)
   6862 free blocks, 2037 free inodes, 2 directories, 2037 unused inodes
   Free blocks: 1330-8191
diff --git a/tests/j_short_revoke_trans/script b/tests/j_short_revoke_trans/script
index edf28f9..4c8a7a9 100644
--- a/tests/j_short_revoke_trans/script
+++ b/tests/j_short_revoke_trans/script
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 echo "debugfs write journal" >> $OUT
 echo "jo" > $TMPFILE.cmd
diff --git a/tests/j_short_revoke_trans_mcsum_64bit/script b/tests/j_short_revoke_trans_mcsum_64bit/script
index 8bd6e08..4c735dc 100644
--- a/tests/j_short_revoke_trans_mcsum_64bit/script
+++ b/tests/j_short_revoke_trans_mcsum_64bit/script
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT
 
diff --git a/tests/j_short_trans/script b/tests/j_short_trans/script
index 00f4d6e..d03a548 100644
--- a/tests/j_short_trans/script
+++ b/tests/j_short_trans/script
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 echo "debugfs write journal" >> $OUT
 echo "jo" > $TMPFILE.cmd
diff --git a/tests/j_short_trans_64bit/script b/tests/j_short_trans_64bit/script
index daf0866..cb4af22 100644
--- a/tests/j_short_trans_64bit/script
+++ b/tests/j_short_trans_64bit/script
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT
 
diff --git a/tests/j_short_trans_mcsum_64bit/script b/tests/j_short_trans_mcsum_64bit/script
index 0534941..eab3c17 100644
--- a/tests/j_short_trans_mcsum_64bit/script
+++ b/tests/j_short_trans_mcsum_64bit/script
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT
 
diff --git a/tests/j_short_trans_old_csum/script b/tests/j_short_trans_old_csum/script
index b968bc1..8d1d036 100644
--- a/tests/j_short_trans_old_csum/script
+++ b/tests/j_short_trans_old_csum/script
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT
 
diff --git a/tests/j_short_trans_open_recover/script b/tests/j_short_trans_open_recover/script
index fbbb714..7a7fa73 100644
--- a/tests/j_short_trans_open_recover/script
+++ b/tests/j_short_trans_open_recover/script
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 echo "debugfs write journal" >> $OUT
 echo "jo" > $TMPFILE.cmd
diff --git a/tests/j_short_trans_recover/script b/tests/j_short_trans_recover/script
index 7fa6158..9d30f97 100644
--- a/tests/j_short_trans_recover/script
+++ b/tests/j_short_trans_recover/script
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 echo "debugfs write journal" >> $OUT
 echo "jo" > $TMPFILE.cmd
diff --git a/tests/j_short_trans_recover_mcsum_64bit/script b/tests/j_short_trans_recover_mcsum_64bit/script
index c283fe2..5b0246e 100644
--- a/tests/j_short_trans_recover_mcsum_64bit/script
+++ b/tests/j_short_trans_recover_mcsum_64bit/script
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT
 
diff --git a/tests/j_short_uncommitted_trans/script b/tests/j_short_uncommitted_trans/script
index 207d746..583c19d 100644
--- a/tests/j_short_uncommitted_trans/script
+++ b/tests/j_short_uncommitted_trans/script
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 echo "debugfs write journal" >> $OUT
 echo "jo" > $TMPFILE.cmd
diff --git a/tests/j_short_uncommitted_trans_mcsum_64bit/script b/tests/j_short_uncommitted_trans_mcsum_64bit/script
index de506be..969372e 100644
--- a/tests/j_short_uncommitted_trans_mcsum_64bit/script
+++ b/tests/j_short_uncommitted_trans_mcsum_64bit/script
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT
 
diff --git a/tests/m_64bit_flexbg/expect.1 b/tests/m_64bit_flexbg/expect.1
index 3635318..cfa3bc9 100644
--- a/tests/m_64bit_flexbg/expect.1
+++ b/tests/m_64bit_flexbg/expect.1
@@ -48,7 +48,8 @@ Default directory hash:   half_md4
 Group 0: (Blocks 1-1023)
   Primary superblock at 1, Group descriptors at 2-2
   Reserved GDT blocks at 3-9
-  Block bitmap at 10 (+9), Inode bitmap at 26 (+25)
+  Block bitmap at 10 (+9)
+  Inode bitmap at 26 (+25)
   Inode table at 42-57 (+41)
   982 free blocks, 117 free inodes, 2 directories
   Free blocks: 24-25, 28-41, 58-1023
diff --git a/tests/m_bigjournal/script b/tests/m_bigjournal/script
index 4c1ed9a..576d99e 100644
--- a/tests/m_bigjournal/script
+++ b/tests/m_bigjournal/script
@@ -1,8 +1,11 @@
 DESCRIPTION="journal over 4GB in size"
 FS_SIZE=11000000
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 MKE2FS_OPTS="-t ext4 -G 512 -N 1280 -J size=5000 -q -E lazy_journal_init,lazy_itable_init,nodiscard"
 if [ $(uname -s) = "Darwin" ]; then
 	echo "$test_name: $DESCRIPTION: skipped for HFS+ (no sparse files)"
 	return 0
 fi
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL
diff --git a/tests/m_dasd_bs/expect.1 b/tests/m_dasd_bs/expect.1
index 75a401d..0e55e8f 100644
--- a/tests/m_dasd_bs/expect.1
+++ b/tests/m_dasd_bs/expect.1
@@ -48,7 +48,8 @@ Default directory hash:   half_md4
 Group 0: (Blocks 0-16383)
   Primary superblock at 0, Group descriptors at 1-1
   Reserved GDT blocks at 2-32
-  Block bitmap at 33 (+33), Inode bitmap at 34 (+34)
+  Block bitmap at 33 (+33)
+  Inode bitmap at 34 (+34)
   Inode table at 35-546 (+35)
   15827 free blocks, 8181 free inodes, 2 directories
   Free blocks: 557-16383
@@ -56,7 +57,8 @@ Group 0: (Blocks 0-16383)
 Group 1: (Blocks 16384-32767)
   Backup superblock at 16384, Group descriptors at 16385-16385
   Reserved GDT blocks at 16386-16416
-  Block bitmap at 16417 (+33), Inode bitmap at 16418 (+34)
+  Block bitmap at 16417 (+33)
+  Inode bitmap at 16418 (+34)
   Inode table at 16419-16930 (+35)
   15837 free blocks, 8192 free inodes, 0 directories
   Free blocks: 16931-32767
diff --git a/tests/m_desc_size_128/script b/tests/m_desc_size_128/script
index de3def9..a4def21 100644
--- a/tests/m_desc_size_128/script
+++ b/tests/m_desc_size_128/script
@@ -1,4 +1,7 @@
 DESCRIPTION="enable 128-byte group descriptor on mkfs"
 FS_SIZE=131072
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 MKE2FS_OPTS="-b 1024 -O 64bit,extents -g 1024 -N 8192 -E desc_size=128"
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL
diff --git a/tests/m_extent_journal/script b/tests/m_extent_journal/script
index 5e0cac2..efade21 100644
--- a/tests/m_extent_journal/script
+++ b/tests/m_extent_journal/script
@@ -1,4 +1,7 @@
 DESCRIPTION="extent-mapped journal"
 FS_SIZE=65536
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 MKE2FS_OPTS="-O extents -j"
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL
diff --git a/tests/m_large_file/expect.1 b/tests/m_large_file/expect.1
index 8ab2711..4acca41 100644
--- a/tests/m_large_file/expect.1
+++ b/tests/m_large_file/expect.1
@@ -48,7 +48,8 @@ Default directory hash:   half_md4
 Group 0: (Blocks 0-16383)
   Primary superblock at 0, Group descriptors at 1-1
   Reserved GDT blocks at 2-4
-  Block bitmap at 5 (+5), Inode bitmap at 6 (+6)
+  Block bitmap at 5 (+5)
+  Inode bitmap at 6 (+6)
   Inode table at 7-10 (+7)
   16367 free blocks, 53 free inodes, 2 directories
   Free blocks: 17-16383
diff --git a/tests/m_meta_bg/script b/tests/m_meta_bg/script
index 7ca2bea..5e285b4 100644
--- a/tests/m_meta_bg/script
+++ b/tests/m_meta_bg/script
@@ -1,4 +1,7 @@
 DESCRIPTION="meta blockgroup feature"
 FS_SIZE=131072
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 MKE2FS_OPTS="-O meta_bg,sparse_super,^resize_inode -g 1024"
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL
diff --git a/tests/m_mmp/expect.1 b/tests/m_mmp/expect.1
index aec0a84..a1452e6 100644
--- a/tests/m_mmp/expect.1
+++ b/tests/m_mmp/expect.1
@@ -51,7 +51,8 @@ MMP update interval:      5
 Group 0: (Blocks 0-32767)
   Primary superblock at 0, Group descriptors at 1-1
   Reserved GDT blocks at 2-16
-  Block bitmap at 17 (+17), Inode bitmap at 18 (+18)
+  Block bitmap at 17 (+17)
+  Inode bitmap at 18 (+18)
   Inode table at 19-1042 (+19)
   31718 free blocks, 32757 free inodes, 2 directories
   Free blocks: 1050-32767
@@ -59,7 +60,8 @@ Group 0: (Blocks 0-32767)
 Group 1: (Blocks 32768-65535)
   Backup superblock at 32768, Group descriptors at 32769-32769
   Reserved GDT blocks at 32770-32784
-  Block bitmap at 32785 (+17), Inode bitmap at 32786 (+18)
+  Block bitmap at 32785 (+17)
+  Inode bitmap at 32786 (+18)
   Inode table at 32787-33810 (+19)
   31725 free blocks, 32768 free inodes, 0 directories
   Free blocks: 33811-65535
diff --git a/tests/m_no_opt/script b/tests/m_no_opt/script
index 6d1d0eb..223f117 100644
--- a/tests/m_no_opt/script
+++ b/tests/m_no_opt/script
@@ -1,4 +1,7 @@
 DESCRIPTION="no filesystem extensions"
 FS_SIZE=65536
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 MKE2FS_OPTS="-O ^sparse_super,^filetype,^resize_inode,^dir_index,^ext_attr"
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL
diff --git a/tests/m_quota/script b/tests/m_quota/script
index 36ab630..fe63939 100644
--- a/tests/m_quota/script
+++ b/tests/m_quota/script
@@ -5,4 +5,7 @@ if [ "$QUOTA" != "y" ]; then
 	echo "$test_name: $DESCRIPTION: skipped"
 	return 0
 fi
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL
diff --git a/tests/m_raid_opt/script b/tests/m_raid_opt/script
index 296fe94..8c859e0 100644
--- a/tests/m_raid_opt/script
+++ b/tests/m_raid_opt/script
@@ -1,4 +1,7 @@
 DESCRIPTION="raid options"
 FS_SIZE=131072
 MKE2FS_OPTS="-E stride=13 -O sparse_super -g 1024"
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL
diff --git a/tests/m_root_owner/expect.1 b/tests/m_root_owner/expect.1
index cd94993..30d119e 100644
--- a/tests/m_root_owner/expect.1
+++ b/tests/m_root_owner/expect.1
@@ -46,7 +46,8 @@ Default directory hash:   half_md4
 Group 0: (Blocks 1-1023)
   Primary superblock at 1, Group descriptors at 2-2
   Reserved GDT blocks at 3-5
-  Block bitmap at 6 (+5), Inode bitmap at 7 (+6)
+  Block bitmap at 6 (+5)
+  Inode bitmap at 7 (+6)
   Inode table at 8-23 (+7)
   986 free blocks, 117 free inodes, 2 directories
   Free blocks: 38-1023
diff --git a/tests/m_std/script b/tests/m_std/script
index a2f2cda..61c50c7 100644
--- a/tests/m_std/script
+++ b/tests/m_std/script
@@ -1,3 +1,6 @@
 DESCRIPTION="standard filesystem options"
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 FS_SIZE=65536
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL
diff --git a/tests/m_uninit/script b/tests/m_uninit/script
index 0b565ac..d4d373a 100644
--- a/tests/m_uninit/script
+++ b/tests/m_uninit/script
@@ -1,4 +1,7 @@
 DESCRIPTION="uninitialized group feature"
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 FS_SIZE=131072
 MKE2FS_OPTS="-O uninit_bg"
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL
diff --git a/tests/r_fixup_lastbg/expect b/tests/r_fixup_lastbg/expect
index f9945b3..96b154a 100644
--- a/tests/r_fixup_lastbg/expect
+++ b/tests/r_fixup_lastbg/expect
@@ -8,13 +8,15 @@ Creating journal (1024 blocks): done
 Writing superblocks and filesystem accounting information:    \b\b\bdone
 
 Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
   Free blocks: 16385-19999
   Free inodes: 833-1248
 Group 2: (Blocks 16385-19999)
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   3615 free blocks, 416 free inodes, 0 directories
   Free blocks: 16385-19999
@@ -23,7 +25,8 @@ Resizing the filesystem on test.img to 20004 (1k) blocks.
 The filesystem on test.img is now 20004 (1k) blocks long.
 
 Group 2: (Blocks 16385-20003) [INODE_UNINIT]
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   3619 free blocks, 416 free inodes, 0 directories, 416 unused inodes
   Free blocks: 16385-20003
diff --git a/tests/r_fixup_lastbg_big/expect b/tests/r_fixup_lastbg_big/expect
index 1aa36fc..badf6cf 100644
--- a/tests/r_fixup_lastbg_big/expect
+++ b/tests/r_fixup_lastbg_big/expect
@@ -8,13 +8,15 @@ Creating journal (1024 blocks): done
 Writing superblocks and filesystem accounting information:    \b\b\bdone
 
 Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
   Free blocks: 16385-19999
   Free inodes: 833-1248
 Group 2: (Blocks 16385-19999)
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   3615 free blocks, 416 free inodes, 0 directories
   Free blocks: 16385-19999
@@ -23,14 +25,14 @@ Resizing the filesystem on test.img to 40000 (1k) blocks.
 The filesystem on test.img is now 40000 (1k) blocks long.
 
 Group 2: (Blocks 16385-24576) [INODE_UNINIT, BLOCK_UNINIT]
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   8192 free blocks, 416 free inodes, 0 directories, 416 unused inodes
   Free blocks: 16385-24576
   Free inodes: 833-1248
 Group 3: (Blocks 24577-32768) [INODE_UNINIT]
   Backup superblock at 24577, Group descriptors at 24578-24578
-  Reserved GDT blocks at 24579-24656
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
diff --git a/tests/r_move_itable/script b/tests/r_move_itable/script
index e567888..3f02a79 100644
--- a/tests/r_move_itable/script
+++ b/tests/r_move_itable/script
@@ -33,7 +33,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 echo "--------------------------------" >> $OUT
 
@@ -53,7 +53,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 echo "--------------------------------" >> $OUT
 
@@ -73,7 +73,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 echo "--------------------------------" >> $OUT
 
@@ -93,7 +93,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 rm -f $TMPFILE
 
diff --git a/tests/r_resize_inode/script b/tests/r_resize_inode/script
index 0f12138..4e3eb19 100644
--- a/tests/r_resize_inode/script
+++ b/tests/r_resize_inode/script
@@ -33,7 +33,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 echo "--------------------------------" >> $OUT
 
@@ -59,7 +59,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 echo "--------------------------------" >> $OUT
 
@@ -79,7 +79,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 rm -f $TMPFILE
 cmp -s $OUT $EXP

[PATCH 16/39] dumpe2fs: output cleanup

[Darrick J. Wong]

Don't display unused inodes twice, and make it clear that we're
printing a descriptor checksum.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Cc: TR Reardon <thomas_reardon@hotmail.com>
---
 misc/dumpe2fs.c                 |    8 +++-----
 tests/filter.sed                |    3 ++-
 tests/r_fixup_lastbg_big/expect |    2 ++
 3 files changed, 7 insertions(+), 6 deletions(-)


diff --git a/misc/dumpe2fs.c b/misc/dumpe2fs.c
index f39cb3c..6751bff 100644
--- a/misc/dumpe2fs.c
+++ b/misc/dumpe2fs.c
@@ -218,20 +218,18 @@ static void list_desc(ext2_filsys fs, int grp_only)
 			continue;
 		}
 
-		printf (_("Group %lu: (Blocks "), i);
+		printf(_("Group %lu: (Blocks "), i);
 		print_range(first_block, last_block);
 		fputs(")", stdout);
-		print_bg_opts(fs, i);
 		if (ext2fs_has_group_desc_csum(fs)) {
 			unsigned csum = ext2fs_bg_checksum(fs, i);
 			unsigned exp_csum = ext2fs_group_desc_csum(fs, i);
 
-			printf(_("  Checksum 0x%04x"), csum);
+			printf(_(" csum 0x%04x"), csum);
 			if (csum != exp_csum)
 				printf(_(" (EXPECTED 0x%04x)"), exp_csum);
-			printf(_(", unused inodes %u\n"),
-			       ext2fs_bg_itable_unused(fs, i));
 		}
+		print_bg_opts(fs, i);
 		has_super = ((i==0) || super_blk);
 		if (has_super) {
 			printf (_("  %s superblock at "),
diff --git a/tests/filter.sed b/tests/filter.sed
index 59fad4e..d9a336c 100644
--- a/tests/filter.sed
+++ b/tests/filter.sed
@@ -21,4 +21,5 @@ s/\\015//g
 /Reserved blocks uid:/s/ (user .*)//
 /Reserved blocks gid:/s/ (group .*)//
 /whichever comes first/d
-/^  Checksum /d
+s/, csum 0x\([0-9a-f]*\)//g
+s/ csum 0x\([0-9a-f]*\)//g
diff --git a/tests/r_fixup_lastbg_big/expect b/tests/r_fixup_lastbg_big/expect
index badf6cf..8f302a3 100644
--- a/tests/r_fixup_lastbg_big/expect
+++ b/tests/r_fixup_lastbg_big/expect
@@ -33,6 +33,8 @@ Group 2: (Blocks 16385-24576) [INODE_UNINIT, BLOCK_UNINIT]
   Free inodes: 833-1248
 Group 3: (Blocks 24577-32768) [INODE_UNINIT]
   Backup superblock at 24577, Group descriptors at 24578-24578
+  Reserved GDT blocks at 24579-24656
+  Block bitmap at 413 (bg #0 + 412)
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity

[PATCH 17/39] e2fsck: fix dangling pointer when dir_info array is resized

[Darrick J. Wong]

e2fsck uses an array to store directory usage information during pass
3; the usage context also contains a pointer to the last directory
looked up.  When expanding the dir_info array, this cache pointer
needs to be cleared if the array resize changed the pointer location,
or else we'll later walk off the end of this dead pointer.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reported-by: Sami Liedes <sami.liedes@iki.fi>
---
 e2fsck/dirinfo.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)


diff --git a/e2fsck/dirinfo.c b/e2fsck/dirinfo.c
index 4a9019b..dab5a13 100644
--- a/e2fsck/dirinfo.c
+++ b/e2fsck/dirinfo.c
@@ -121,7 +121,7 @@ static void setup_db(e2fsck_t ctx)
 void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent)
 {
 	struct dir_info_db 	*db;
-	struct dir_info 	*dir, ent;
+	struct dir_info		*dir, ent, *old_array;
 	int			i, j;
 	errcode_t		retval;
 	unsigned long		old_size;
@@ -136,6 +136,7 @@ void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent)
 	if (ctx->dir_info->count >= ctx->dir_info->size) {
 		old_size = ctx->dir_info->size * sizeof(struct dir_info);
 		ctx->dir_info->size += 10;
+		old_array = ctx->dir_info->array;
 		retval = ext2fs_resize_mem(old_size, ctx->dir_info->size *
 					   sizeof(struct dir_info),
 					   &ctx->dir_info->array);
@@ -147,6 +148,8 @@ void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent)
 			ctx->dir_info->size -= 10;
 			return;
 		}
+		if (old_array != ctx->dir_info->array)
+			ctx->dir_info->last_lookup = NULL;
 	}
 
 	ent.ino = ino;

[PATCH 18/39] e2fsck: opportunistically recalculate unused inode count and inode_uninit after pass 5

[Darrick J. Wong]

At the end of pass5, update the unused inode count to cover any newly
unused inodes and set inode_uninit if none of the inodes in the group
are used.  This will save us time on future fsck runs since we don't
need to scan those inodes any more.

Since it's not an error for the unused inode count to be smaller than
necessary, don't bother if the fs is mounted and we're not fixing
anything else.

On an aged filesystem that has experienced at least one massive inode
die-off, this has been shown to reduce future e2fsck times by around
10%.  The effect is diluted on less heavily used filesystems.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass5.c                          |   38 +++++++++++++++++++++++++++++++
 e2fsck/problem.c                        |    5 ++++
 e2fsck/problem.h                        |    3 ++
 tests/f_bad_bbitmap/expect.1            |    3 ++
 tests/f_bad_bmap_csum/expect.1          |    3 ++
 tests/f_bad_gdt_csum/expect.1           |    7 +++++-
 tests/f_bad_ibitmap/expect.1            |    3 ++
 tests/f_bad_inode_csum/expect.1         |    3 ++
 tests/f_idata_and_extents/expect.1      |    3 ++
 tests/f_illitable_flexbg/expect.1       |   12 ++++++++++
 tests/f_inlinedata_repair/expect.1      |    3 ++
 tests/f_invalid_extent_symlink/expect.1 |    3 ++
 tests/f_jnl_64bit/expect.1              |    6 +++++
 tests/f_super_bad_csum/expect.1         |    8 +++++--
 tests/f_unused_itable/expect.1          |    6 +++++
 15 files changed, 103 insertions(+), 3 deletions(-)


diff --git a/e2fsck/pass5.c b/e2fsck/pass5.c
index 64fb7fe..1a225fb 100644
--- a/e2fsck/pass5.c
+++ b/e2fsck/pass5.c
@@ -821,6 +821,44 @@ do_counts:
 			} else
 				ext2fs_unmark_valid(fs);
 		}
+
+		/*
+		 * Opportunistically update the unused inodes count
+		 * and set inode_uninit so we can skip scanning unused
+		 * inodes during future fsck runs.  However, it's not
+		 * an error if the unused inode count is smaller
+		 * than necessary, so don't bother the user if FS is
+		 * mounted and we haven't fixed anything else, to
+		 * minimize unnecessary reboots.
+		 */
+		if (((ctx->options & E2F_OPT_NO) ||
+		     !(ctx->mount_flags & EXT2_MF_MOUNTED) ||
+		     ext2fs_test_changed(fs)) &&
+		    ext2fs_has_group_desc_csum(fs) &&
+		    !ext2fs_bg_flags_test(fs, i, EXT2_BG_INODE_UNINIT)) {
+			ext2_ino_t start, end, j;
+
+			start = (fs->super->s_inodes_per_group * i) + 1;
+			end = (fs->super->s_inodes_per_group * (i + 1));
+			pctx.group = i;
+			pctx.ino = ext2fs_bg_itable_unused(fs, i);
+			end -= pctx.ino;
+			for (j = end; j >= start; j--)
+				if (ext2fs_test_inode_bitmap2(ctx->inode_used_map, j))
+					break;
+
+			pctx.ino2 = fs->super->s_inodes_per_group -
+						(j - start + 1);
+			if (pctx.ino != pctx.ino2 &&
+			    fix_problem(ctx, PR_5_UNUSED_INODES_COUNT_GROUP,
+					&pctx)) {
+				if (pctx.ino2 == fs->super->s_inodes_per_group)
+					ext2fs_bg_flags_set(fs, i,
+							EXT2_BG_INODE_UNINIT);
+				ext2fs_bg_itable_unused_set(fs, i, pctx.ino2);
+				ext2fs_mark_super_dirty(fs);
+			}
+		}
 	}
 	if (free_inodes != fs->super->s_free_inodes_count) {
 		pctx.group = -1;
diff --git a/e2fsck/problem.c b/e2fsck/problem.c
index a4da64b..9c2b2e6 100644
--- a/e2fsck/problem.c
+++ b/e2fsck/problem.c
@@ -1899,6 +1899,11 @@ static struct e2fsck_problem problem_table[] = {
 	  N_("@g %g @b @B does not match checksum.\n"),
 	  PROMPT_FIX, PR_LATCH_BBITMAP | PR_PREEN_OK },
 
+	/* Unused inode count for group wrong */
+	{ PR_5_UNUSED_INODES_COUNT_GROUP,
+	  N_("Unused @i count wrong for @g #%g (%i, counted=%j).\n"),
+	  PROMPT_FIX, PR_PREEN_OK | PR_NO_OK },
+
 	/* Post-Pass 5 errors */
 
 	/* Recreate journal if E2F_FLAG_JOURNAL_INODE flag is set */
diff --git a/e2fsck/problem.h b/e2fsck/problem.h
index 3c28166..6a71fc8 100644
--- a/e2fsck/problem.h
+++ b/e2fsck/problem.h
@@ -1139,6 +1139,9 @@ struct problem_context {
 /* Block bitmap checksum does not match */
 #define PR_5_BLOCK_BITMAP_CSUM_INVALID	0x05001B
 
+/* Unused inode count for group wrong */
+#define PR_5_UNUSED_INODES_COUNT_GROUP	0x05001C
+
 /*
  * Post-Pass 5 errors
  */
diff --git a/tests/f_bad_bbitmap/expect.1 b/tests/f_bad_bbitmap/expect.1
index 71ad1bb..af3f632 100644
--- a/tests/f_bad_bbitmap/expect.1
+++ b/tests/f_bad_bbitmap/expect.1
@@ -9,6 +9,9 @@ Pass 5: Checking group summary information
 Block bitmap differences:  -(8--10) -(12--17) -(19--31)
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks
diff --git a/tests/f_bad_bmap_csum/expect.1 b/tests/f_bad_bmap_csum/expect.1
index ca8f77f..7616751 100644
--- a/tests/f_bad_bmap_csum/expect.1
+++ b/tests/f_bad_bmap_csum/expect.1
@@ -6,6 +6,9 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
 Inode bitmap differences: Group 0 inode bitmap does not match checksum.
 FIXED.
 Block bitmap differences: Group 0 block bitmap does not match checksum.
diff --git a/tests/f_bad_gdt_csum/expect.1 b/tests/f_bad_gdt_csum/expect.1
index e14c897..0f496e0 100644
--- a/tests/f_bad_gdt_csum/expect.1
+++ b/tests/f_bad_gdt_csum/expect.1
@@ -6,5 +6,10 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks
-Exit status is 0
+Exit status is 1
diff --git a/tests/f_bad_ibitmap/expect.1 b/tests/f_bad_ibitmap/expect.1
index ea17523..81689e0 100644
--- a/tests/f_bad_ibitmap/expect.1
+++ b/tests/f_bad_ibitmap/expect.1
@@ -9,6 +9,9 @@ Pass 5: Checking group summary information
 Inode bitmap differences:  -(12--32)
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks
diff --git a/tests/f_bad_inode_csum/expect.1 b/tests/f_bad_inode_csum/expect.1
index b3c628d..b28a349 100644
--- a/tests/f_bad_inode_csum/expect.1
+++ b/tests/f_bad_inode_csum/expect.1
@@ -117,6 +117,9 @@ Fix? yes
 Free inodes count wrong for group #0 (0, counted=32).
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=32).
+Fix? yes
+
 Free inodes count wrong (0, counted=32).
 Fix? yes
 
diff --git a/tests/f_idata_and_extents/expect.1 b/tests/f_idata_and_extents/expect.1
index 7f7fbf3..aba063a 100644
--- a/tests/f_idata_and_extents/expect.1
+++ b/tests/f_idata_and_extents/expect.1
@@ -26,6 +26,9 @@ Fix? yes
 Free inodes count wrong for group #0 (105, counted=106).
 Fix? yes
 
+Unused inode count wrong for group #0 (103, counted=105).
+Fix? yes
+
 Free inodes count wrong (105, counted=106).
 Fix? yes
 
diff --git a/tests/f_illitable_flexbg/expect.1 b/tests/f_illitable_flexbg/expect.1
index fa42a0f..04ce65b 100644
--- a/tests/f_illitable_flexbg/expect.1
+++ b/tests/f_illitable_flexbg/expect.1
@@ -18,6 +18,18 @@ Pass 5: Checking group summary information
 Inode bitmap differences:  -(65--128)
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=52).
+Fix? yes
+
+Unused inode count wrong for group #1 (0, counted=64).
+Fix? yes
+
+Unused inode count wrong for group #2 (0, counted=64).
+Fix? yes
+
+Unused inode count wrong for group #3 (0, counted=64).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 12/256 files (0.0% non-contiguous), 31163/32768 blocks
diff --git a/tests/f_inlinedata_repair/expect.1 b/tests/f_inlinedata_repair/expect.1
index faba192..f7eebd3 100644
--- a/tests/f_inlinedata_repair/expect.1
+++ b/tests/f_inlinedata_repair/expect.1
@@ -69,6 +69,9 @@ Pass 5: Checking group summary information
 Directories count wrong for group #0 (7, counted=8).
 Fix? yes
 
+Unused inode count wrong for group #0 (90, counted=91).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 28/128 files (0.0% non-contiguous), 18/512 blocks
diff --git a/tests/f_invalid_extent_symlink/expect.1 b/tests/f_invalid_extent_symlink/expect.1
index 7bda0b7..cbb2ac5 100644
--- a/tests/f_invalid_extent_symlink/expect.1
+++ b/tests/f_invalid_extent_symlink/expect.1
@@ -6,6 +6,9 @@ Clear? yes
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
+Unused inode count wrong for group #0 (4, counted=5).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/16 files (9.1% non-contiguous), 21/100 blocks
diff --git a/tests/f_jnl_64bit/expect.1 b/tests/f_jnl_64bit/expect.1
index e360e2f..915076b 100644
--- a/tests/f_jnl_64bit/expect.1
+++ b/tests/f_jnl_64bit/expect.1
@@ -7,6 +7,12 @@ Pass 5: Checking group summary information
 Free blocks count wrong (14059, counted=12712).
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=2876).
+Fix? yes
+
+Unused inode count wrong for group #1 (1395, counted=3958).
+Fix? yes
+
 Free inodes count wrong (8181, counted=6834).
 Fix? yes
 
diff --git a/tests/f_super_bad_csum/expect.1 b/tests/f_super_bad_csum/expect.1
index 25ced5c..a606fab 100644
--- a/tests/f_super_bad_csum/expect.1
+++ b/tests/f_super_bad_csum/expect.1
@@ -5,8 +5,12 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
-Inode bitmap differences: Group 1 inode bitmap does not match checksum.
-FIXED.
+Unused inode count wrong for group #0 (0, counted=501).
+Fix? yes
+
+Unused inode count wrong for group #1 (0, counted=512).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/1024 files (0.0% non-contiguous), 1557/16384 blocks
diff --git a/tests/f_unused_itable/expect.1 b/tests/f_unused_itable/expect.1
index a4da987..fbc6538 100644
--- a/tests/f_unused_itable/expect.1
+++ b/tests/f_unused_itable/expect.1
@@ -19,9 +19,15 @@ Pass 5: Checking group summary information
 Free inodes count wrong for group #0 (53, counted=51).
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=51).
+Fix? yes
+
 Free inodes count wrong for group #1 (64, counted=58).
 Fix? yes
 
+Unused inode count wrong for group #1 (0, counted=58).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 19/128 files (0.0% non-contiguous), 165/1000 blocks

[PATCH 19/39] e2fsck: opportunistically set block_uninit after pass 5

[Darrick J. Wong]

At the end of pass5, set block_uninit on any group that's empty.  For
meta_bg filesystems this might save a small amount of disk accesses at
load time.

Since it's not an error for the flag not to be set even when it could,
don't bother if the fs is mounted and we're not fixing anything else.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass5.c                        |   22 ++++++++++++++++++++++
 e2fsck/problem.c                      |    5 +++++
 e2fsck/problem.h                      |    3 +++
 tests/j_long_trans/expect             |    3 +++
 tests/j_long_trans_mcsum_32bit/expect |    6 ++++++
 tests/j_long_trans_mcsum_64bit/expect |    6 ++++++
 6 files changed, 45 insertions(+)


diff --git a/e2fsck/pass5.c b/e2fsck/pass5.c
index 1a225fb..92ab633 100644
--- a/e2fsck/pass5.c
+++ b/e2fsck/pass5.c
@@ -544,6 +544,28 @@ redo_counts:
 			} else
 				ext2fs_unmark_valid(fs);
 		}
+
+		/*
+		 * Opportunistically set block_uninit if none of the
+		 * blocks are in use.  However, it's not an error if
+		 * the flag could be set but isn't, so don't bother
+		 * the user if the FS is mounted and we haven't fixed
+		 * anything else, to minimize unnecessary reboots.
+		 */
+		if (((ctx->options & E2F_OPT_NO) ||
+		     !(ctx->mount_flags & EXT2_MF_MOUNTED) ||
+		     ext2fs_test_changed(fs)) &&
+		    ext2fs_has_group_desc_csum(fs) &&
+		    !ext2fs_bg_flags_test(fs, g, EXT2_BG_BLOCK_UNINIT) &&
+		    (g != fs->group_desc_count - 1) &&
+		    free_array[g] == fs->super->s_blocks_per_group) {
+			pctx.group = g;
+			if (fix_problem(ctx, PR_5_BLOCK_UNINIT_UNSET, &pctx)) {
+				ext2fs_bg_flags_set(fs, g,
+							EXT2_BG_BLOCK_UNINIT);
+				ext2fs_mark_super_dirty(fs);
+			}
+		}
 	}
 	free_blocks = EXT2FS_C2B(fs, free_blocks);
 	if (free_blocks != ext2fs_free_blocks_count(fs->super)) {
diff --git a/e2fsck/problem.c b/e2fsck/problem.c
index 9c2b2e6..6a5f814 100644
--- a/e2fsck/problem.c
+++ b/e2fsck/problem.c
@@ -1904,6 +1904,11 @@ static struct e2fsck_problem problem_table[] = {
 	  N_("Unused @i count wrong for @g #%g (%i, counted=%j).\n"),
 	  PROMPT_FIX, PR_PREEN_OK | PR_NO_OK },
 
+	/* block_uninit flag can be set for group */
+	{ PR_5_BLOCK_UNINIT_UNSET,
+	  N_("BLOCK_UNINIT flag can be set for @g #%g.\n"),
+	  PROMPT_FIX, PR_PREEN_OK | PR_NO_OK },
+
 	/* Post-Pass 5 errors */
 
 	/* Recreate journal if E2F_FLAG_JOURNAL_INODE flag is set */
diff --git a/e2fsck/problem.h b/e2fsck/problem.h
index 6a71fc8..345dd0a 100644
--- a/e2fsck/problem.h
+++ b/e2fsck/problem.h
@@ -1142,6 +1142,9 @@ struct problem_context {
 /* Unused inode count for group wrong */
 #define PR_5_UNUSED_INODES_COUNT_GROUP	0x05001C
 
+/* Unused inode count for group wrong */
+#define PR_5_BLOCK_UNINIT_UNSET		0x05001D
+
 /*
  * Post-Pass 5 errors
  */
diff --git a/tests/j_long_trans/expect b/tests/j_long_trans/expect
index 7638ef1..2cc1dab 100644
--- a/tests/j_long_trans/expect
+++ b/tests/j_long_trans/expect
@@ -81,6 +81,9 @@ Fix? yes
 Free blocks count wrong for group #14 (0, counted=8192).
 Fix? yes
 
+BLOCK_UNINIT flag can be set for group #14.
+Fix? yes
+
 Free blocks count wrong (247711, counted=255916).
 Fix? yes
 
diff --git a/tests/j_long_trans_mcsum_32bit/expect b/tests/j_long_trans_mcsum_32bit/expect
index 0d141c1..39b6e6d 100644
--- a/tests/j_long_trans_mcsum_32bit/expect
+++ b/tests/j_long_trans_mcsum_32bit/expect
@@ -117,9 +117,15 @@ Fix? yes
 Free blocks count wrong for group #17 (0, counted=8192).
 Fix? yes
 
+BLOCK_UNINIT flag can be set for group #17.
+Fix? yes
+
 Free blocks count wrong for group #18 (0, counted=8192).
 Fix? yes
 
+BLOCK_UNINIT flag can be set for group #18.
+Fix? yes
+
 Free blocks count wrong (497236, counted=513633).
 Fix? yes
 
diff --git a/tests/j_long_trans_mcsum_64bit/expect b/tests/j_long_trans_mcsum_64bit/expect
index 94e9925..edb7776 100644
--- a/tests/j_long_trans_mcsum_64bit/expect
+++ b/tests/j_long_trans_mcsum_64bit/expect
@@ -116,9 +116,15 @@ Fix? yes
 Free blocks count wrong for group #17 (0, counted=8192).
 Fix? yes
 
+BLOCK_UNINIT flag can be set for group #17.
+Fix? yes
+
 Free blocks count wrong for group #18 (0, counted=8192).
 Fix? yes
 
+BLOCK_UNINIT flag can be set for group #18.
+Fix? yes
+
 Free blocks count wrong (497218, counted=513615).
 Fix? yes
 

[PATCH 20/39] libext2fs/e2fsck: provide routines to read-ahead metadata

[Darrick J. Wong]

This patch adds to e2fsck the ability to pre-fetch metadata into the
page cache in the hopes of speeding up fsck runs.  There are two new
functions -- the first allows a caller to readahead a list of blocks,
and the second is a helper function that uses that first mechanism to
load group data (bitmaps, inode tables).

These new e2fsck routines require the addition of a dblist API to
allow us to iterate a subset of a dblist.  This will enable
incremental directory block readahead in e2fsck pass 2.

There's also a function to estimate the readahead given a FS.

v2: Add an API to create a dblist with a given number of list elements
pre-allocated.  This enables us to save ~2ms per call to
e2fsck_readahead() (assuming a 2MB RA buffer) by not having to
repeatedly call ext2_resize_mem as we add blocks to the list.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 configure           |    2 
 configure.in        |    1 
 e2fsck/Makefile.in  |    9 +-
 e2fsck/e2fsck.h     |   18 ++++
 e2fsck/readahead.c  |  252 +++++++++++++++++++++++++++++++++++++++++++++++++++
 e2fsck/util.c       |   51 ++++++++++
 lib/config.h.in     |    3 +
 lib/ext2fs/dblist.c |   21 ++++
 lib/ext2fs/ext2fs.h |   10 ++
 9 files changed, 359 insertions(+), 8 deletions(-)
 create mode 100644 e2fsck/readahead.c


diff --git a/configure b/configure
index 2bc1275..a65d3e6 100755
--- a/configure
+++ b/configure
@@ -12414,7 +12414,7 @@ fi
 done
 
 fi
-for ac_header in  	dirent.h 	errno.h 	execinfo.h 	getopt.h 	malloc.h 	mntent.h 	paths.h 	semaphore.h 	setjmp.h 	signal.h 	stdarg.h 	stdint.h 	stdlib.h 	termios.h 	termio.h 	unistd.h 	utime.h 	attr/xattr.h 	linux/falloc.h 	linux/fd.h 	linux/major.h 	linux/loop.h 	net/if_dl.h 	netinet/in.h 	sys/disklabel.h 	sys/disk.h 	sys/file.h 	sys/ioctl.h 	sys/mkdev.h 	sys/mman.h 	sys/mount.h 	sys/prctl.h 	sys/resource.h 	sys/select.h 	sys/socket.h 	sys/sockio.h 	sys/stat.h 	sys/syscall.h 	sys/sysmacros.h 	sys/time.h 	sys/types.h 	sys/un.h 	sys/wait.h
+for ac_header in  	dirent.h 	errno.h 	execinfo.h 	getopt.h 	malloc.h 	mntent.h 	paths.h 	semaphore.h 	setjmp.h 	signal.h 	stdarg.h 	stdint.h 	stdlib.h 	termios.h 	termio.h 	unistd.h 	utime.h 	attr/xattr.h 	linux/falloc.h 	linux/fd.h 	linux/major.h 	linux/loop.h 	net/if_dl.h 	netinet/in.h 	sys/disklabel.h 	sys/disk.h 	sys/file.h 	sys/ioctl.h 	sys/mkdev.h 	sys/mman.h 	sys/mount.h 	sys/prctl.h 	sys/resource.h 	sys/select.h 	sys/socket.h 	sys/sockio.h 	sys/stat.h 	sys/syscall.h 	sys/sysctl.h 	sys/sysmacros.h 	sys/time.h 	sys/types.h 	sys/un.h 	sys/wait.h
 do :
   as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
 ac_fn_c_check_header_mongrel "$LINENO" "$ac_header" "$as_ac_Header" "$ac_includes_default"
diff --git a/configure.in b/configure.in
index e809324..d1be5b9 100644
--- a/configure.in
+++ b/configure.in
@@ -949,6 +949,7 @@ AC_CHECK_HEADERS(m4_flatten([
 	sys/sockio.h
 	sys/stat.h
 	sys/syscall.h
+	sys/sysctl.h
 	sys/sysmacros.h
 	sys/time.h
 	sys/types.h
diff --git a/e2fsck/Makefile.in b/e2fsck/Makefile.in
index d0e64eb..e40e51b 100644
--- a/e2fsck/Makefile.in
+++ b/e2fsck/Makefile.in
@@ -62,7 +62,7 @@ OBJS= dict.o unix.o e2fsck.o super.o pass1.o pass1b.o pass2.o \
 	pass3.o pass4.o pass5.o journal.o badblocks.o util.o dirinfo.o \
 	dx_dirinfo.o ehandler.o problem.o message.o quota.o recovery.o \
 	region.o revoke.o ea_refcount.o rehash.o profile.o prof_err.o \
-	logfile.o sigcatcher.o $(MTRACE_OBJ) plausible.o
+	logfile.o sigcatcher.o $(MTRACE_OBJ) plausible.o readahead.o
 
 PROFILED_OBJS= profiled/dict.o profiled/unix.o profiled/e2fsck.o \
 	profiled/super.o profiled/pass1.o profiled/pass1b.o \
@@ -73,7 +73,8 @@ PROFILED_OBJS= profiled/dict.o profiled/unix.o profiled/e2fsck.o \
 	profiled/recovery.o profiled/region.o profiled/revoke.o \
 	profiled/ea_refcount.o profiled/rehash.o profiled/profile.o \
 	profiled/prof_err.o profiled/logfile.o \
-	profiled/sigcatcher.o profiled/plausible.o
+	profiled/sigcatcher.o profiled/plausible.o \
+	profiled/sigcatcher.o profiled/readahead.o
 
 SRCS= $(srcdir)/e2fsck.c \
 	$(srcdir)/dict.c \
@@ -97,6 +98,7 @@ SRCS= $(srcdir)/e2fsck.c \
 	$(srcdir)/message.c \
 	$(srcdir)/ea_refcount.c \
 	$(srcdir)/rehash.c \
+	$(srcdir)/readahead.c \
 	$(srcdir)/region.c \
 	$(srcdir)/profile.c \
 	$(srcdir)/sigcatcher.c \
@@ -541,3 +543,6 @@ plausible.o: $(srcdir)/../misc/plausible.c $(top_builddir)/lib/config.h \
  $(top_builddir)/lib/ext2fs/ext2_err.h \
  $(top_srcdir)/lib/ext2fs/ext2_ext_attr.h $(top_srcdir)/lib/ext2fs/bitops.h \
  $(srcdir)/../misc/nls-enable.h $(srcdir)/../misc/plausible.h
+readahead.o: $(srcdir)/readahead.c $(top_builddir)/lib/config.h \
+ $(top_srcdir)/lib/ext2fs/ext2fs.h $(top_srcdir)/lib/ext2fs/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/e2fsck.h prof_err.h
diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index 6ca3a6a..8837af9 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -490,6 +490,23 @@ extern ext2_ino_t e2fsck_get_lost_and_found(e2fsck_t ctx, int fix);
 extern errcode_t e2fsck_adjust_inode_count(e2fsck_t ctx, ext2_ino_t ino,
 					   int adj);
 
+/* readahead.c */
+#define E2FSCK_READA_SUPER	(0x01)
+#define E2FSCK_READA_GDT	(0x02)
+#define E2FSCK_READA_BBITMAP	(0x04)
+#define E2FSCK_READA_IBITMAP	(0x08)
+#define E2FSCK_READA_ITABLE	(0x10)
+#define E2FSCK_READA_ALL_FLAGS	(0x1F)
+errcode_t e2fsck_readahead(ext2_filsys fs, int flags, dgrp_t start,
+			   dgrp_t ngroups);
+#define E2FSCK_RA_DBLIST_IGNORE_BLOCKCNT	(0x01)
+#define E2FSCK_RA_DBLIST_ALL_FLAGS		(0x01)
+errcode_t e2fsck_readahead_dblist(ext2_filsys fs, int flags,
+				  ext2_dblist dblist,
+				  unsigned long long start,
+				  unsigned long long count);
+int e2fsck_can_readahead(ext2_filsys fs);
+unsigned long long e2fsck_guess_readahead(ext2_filsys fs);
 
 /* region.c */
 extern region_t region_create(region_addr_t min, region_addr_t max);
@@ -579,6 +596,7 @@ extern errcode_t e2fsck_allocate_subcluster_bitmap(ext2_filsys fs,
 						   int default_type,
 						   const char *profile_name,
 						   ext2fs_block_bitmap *ret);
+unsigned long long get_memory_size(void);
 
 /* unix.c */
 extern void e2fsck_clear_progbar(e2fsck_t ctx);
diff --git a/e2fsck/readahead.c b/e2fsck/readahead.c
new file mode 100644
index 0000000..a35f9f8
--- /dev/null
+++ b/e2fsck/readahead.c
@@ -0,0 +1,252 @@
+/*
+ * readahead.c -- Prefetch filesystem metadata to speed up fsck.
+ *
+ * Copyright (C) 2014 Oracle.
+ *
+ * %Begin-Header%
+ * This file may be redistributed under the terms of the GNU Library
+ * General Public License, version 2.
+ * %End-Header%
+ */
+
+#include "config.h"
+#include <string.h>
+
+#include "e2fsck.h"
+
+#undef DEBUG
+
+#ifdef DEBUG
+# define dbg_printf(f, a...)  do {printf(f, ## a); fflush(stdout); } while (0)
+#else
+# define dbg_printf(f, a...)
+#endif
+
+struct read_dblist {
+	errcode_t err;
+	blk64_t run_start;
+	blk64_t run_len;
+	int flags;
+};
+
+static int readahead_dir_block(ext2_filsys fs, struct ext2_db_entry2 *db,
+			       void *priv_data)
+{
+	struct read_dblist *pr = priv_data;
+	e2_blkcnt_t count = (pr->flags & E2FSCK_RA_DBLIST_IGNORE_BLOCKCNT ?
+			     1 : db->blockcnt);
+
+	if (!pr->run_len || db->blk != pr->run_start + pr->run_len) {
+		if (pr->run_len) {
+			pr->err = io_channel_cache_readahead(fs->io,
+							     pr->run_start,
+							     pr->run_len);
+			dbg_printf("readahead start=%llu len=%llu err=%d\n",
+				   pr->run_start, pr->run_len,
+				   (int)pr->err);
+		}
+		pr->run_start = db->blk;
+		pr->run_len = 0;
+	}
+	pr->run_len += count;
+
+	return pr->err ? DBLIST_ABORT : 0;
+}
+
+errcode_t e2fsck_readahead_dblist(ext2_filsys fs, int flags,
+				  ext2_dblist dblist,
+				  unsigned long long start,
+				  unsigned long long count)
+{
+	errcode_t err;
+	struct read_dblist pr;
+
+	dbg_printf("%s: flags=0x%x\n", __func__, flags);
+	if (flags & ~E2FSCK_RA_DBLIST_ALL_FLAGS)
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	memset(&pr, 0, sizeof(pr));
+	pr.flags = flags;
+	err = ext2fs_dblist_iterate3(dblist, readahead_dir_block, start,
+				     count, &pr);
+	if (pr.err)
+		return pr.err;
+	if (err)
+		return err;
+
+	if (pr.run_len)
+		err = io_channel_cache_readahead(fs->io, pr.run_start,
+						 pr.run_len);
+
+	return err;
+}
+
+static errcode_t e2fsck_readahead_bitmap(ext2_filsys fs,
+					 ext2fs_block_bitmap ra_map)
+{
+	blk64_t start, end, out;
+	errcode_t err;
+
+	start = 1;
+	end = ext2fs_blocks_count(fs->super) - 1;
+
+	err = ext2fs_find_first_set_block_bitmap2(ra_map, start, end, &out);
+	while (err == 0) {
+		start = out;
+		err = ext2fs_find_first_zero_block_bitmap2(ra_map, start, end,
+							   &out);
+		if (err == ENOENT) {
+			out = end;
+			err = 0;
+		} else if (err)
+			break;
+
+		err = io_channel_cache_readahead(fs->io, start, out - start);
+		if (err)
+			break;
+		start = out;
+		err = ext2fs_find_first_set_block_bitmap2(ra_map, start, end,
+							  &out);
+	}
+
+	if (err == ENOENT)
+		err = 0;
+
+	return err;
+}
+
+/* Try not to spew bitmap range errors for readahead */
+static errcode_t mark_bmap_range(ext2_filsys fs, ext2fs_block_bitmap map,
+				 blk64_t blk, unsigned int num)
+{
+	if (blk >= ext2fs_get_generic_bmap_start(map) &&
+	    blk + num <= ext2fs_get_generic_bmap_end(map))
+		ext2fs_mark_block_bitmap_range2(map, blk, num);
+	else
+		return EXT2_ET_INVALID_ARGUMENT;
+	return 0;
+}
+
+static errcode_t mark_bmap(ext2_filsys fs, ext2fs_block_bitmap map, blk64_t blk)
+{
+	if (blk >= ext2fs_get_generic_bmap_start(map) &&
+	    blk <= ext2fs_get_generic_bmap_end(map))
+		ext2fs_mark_block_bitmap2(map, blk);
+	else
+		return EXT2_ET_INVALID_ARGUMENT;
+	return 0;
+}
+
+errcode_t e2fsck_readahead(ext2_filsys fs, int flags, dgrp_t start,
+			   dgrp_t ngroups)
+{
+	blk64_t		super, old_gdt, new_gdt;
+	blk_t		blocks;
+	dgrp_t		i;
+	ext2fs_block_bitmap		ra_map = NULL;
+	dgrp_t		end = start + ngroups;
+	errcode_t	err = 0;
+
+	dbg_printf("%s: flags=0x%x start=%d groups=%d\n", __func__, flags,
+		   start, ngroups);
+	if (flags & ~E2FSCK_READA_ALL_FLAGS)
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	if (end > fs->group_desc_count)
+		end = fs->group_desc_count;
+
+	if (flags == 0)
+		return 0;
+
+	err = ext2fs_allocate_block_bitmap(fs, "readahead bitmap",
+					   &ra_map);
+	if (err)
+		return err;
+
+	for (i = start; i < end; i++) {
+		err = ext2fs_super_and_bgd_loc2(fs, i, &super, &old_gdt,
+						&new_gdt, &blocks);
+		if (err)
+			break;
+
+		if (flags & E2FSCK_READA_SUPER) {
+			err = mark_bmap(fs, ra_map, super);
+			if (err)
+				break;
+		}
+
+		if (flags & E2FSCK_READA_GDT) {
+			err = mark_bmap_range(fs, ra_map,
+					      old_gdt ? old_gdt : new_gdt,
+					      blocks);
+			if (err)
+				break;
+		}
+
+		if ((flags & E2FSCK_READA_BBITMAP) &&
+		    !ext2fs_bg_flags_test(fs, i, EXT2_BG_BLOCK_UNINIT) &&
+		    ext2fs_bg_free_blocks_count(fs, i) <
+				fs->super->s_blocks_per_group) {
+			super = ext2fs_block_bitmap_loc(fs, i);
+			err = mark_bmap(fs, ra_map, super);
+			if (err)
+				break;
+		}
+
+		if ((flags & E2FSCK_READA_IBITMAP) &&
+		    !ext2fs_bg_flags_test(fs, i, EXT2_BG_INODE_UNINIT) &&
+		    ext2fs_bg_free_inodes_count(fs, i) <
+				fs->super->s_inodes_per_group) {
+			super = ext2fs_inode_bitmap_loc(fs, i);
+			err = mark_bmap(fs, ra_map, super);
+			if (err)
+				break;
+		}
+
+		if ((flags & E2FSCK_READA_ITABLE) &&
+		    ext2fs_bg_free_inodes_count(fs, i) <
+				fs->super->s_inodes_per_group) {
+			super = ext2fs_inode_table_loc(fs, i);
+			blocks = fs->inode_blocks_per_group -
+				 (ext2fs_bg_itable_unused(fs, i) *
+				  EXT2_INODE_SIZE(fs->super) / fs->blocksize);
+			err = mark_bmap_range(fs, ra_map, super, blocks);
+			if (err)
+				break;
+		}
+	}
+
+	if (!err)
+		err = e2fsck_readahead_bitmap(fs, ra_map);
+
+	ext2fs_free_block_bitmap(ra_map);
+	return err;
+}
+
+int e2fsck_can_readahead(ext2_filsys fs)
+{
+	errcode_t err;
+
+	err = io_channel_cache_readahead(fs->io, 0, 1);
+	dbg_printf("%s: supp=%d\n", __func__, err != EXT2_ET_OP_NOT_SUPPORTED);
+	return err != EXT2_ET_OP_NOT_SUPPORTED;
+}
+
+unsigned long long e2fsck_guess_readahead(ext2_filsys fs)
+{
+	unsigned long long guess;
+
+	/*
+	 * The optimal readahead sizes were experimentally determined by
+	 * djwong in August 2014.  Setting the RA size to one block group's
+	 * worth of inode table blocks seems to yield the largest reductions
+	 * in e2fsck runtime.
+	 */
+	guess = fs->blocksize * fs->inode_blocks_per_group;
+
+	/* Disable RA if it'd use more 1/100th of RAM. */
+	if (get_memory_size() > (guess * 100))
+		return guess / 1024;
+
+	return 0;
+}
diff --git a/e2fsck/util.c b/e2fsck/util.c
index 8237328..74f20062 100644
--- a/e2fsck/util.c
+++ b/e2fsck/util.c
@@ -37,6 +37,10 @@
 #include <errno.h>
 #endif
 
+#ifdef HAVE_SYS_SYSCTL_H
+#include <sys/sysctl.h>
+#endif
+
 #include "e2fsck.h"
 
 extern e2fsck_t e2fsck_global_ctx;   /* Try your very best not to use this! */
@@ -848,3 +852,50 @@ errcode_t e2fsck_allocate_subcluster_bitmap(ext2_filsys fs, const char *descr,
 	fs->default_bitmap_type = save_type;
 	return retval;
 }
+
+/* Return memory size in bytes */
+unsigned long long get_memory_size(void)
+{
+#if defined(_SC_PHYS_PAGES)
+# if defined(_SC_PAGESIZE)
+	return (unsigned long long)sysconf(_SC_PHYS_PAGES) *
+	       (unsigned long long)sysconf(_SC_PAGESIZE);
+# elif defined(_SC_PAGE_SIZE)
+	return (unsigned long long)sysconf(_SC_PHYS_PAGES) *
+	       (unsigned long long)sysconf(_SC_PAGE_SIZE);
+# endif
+#elif defined(CTL_HW)
+# if (defined(HW_MEMSIZE) || defined(HW_PHYSMEM64))
+#  define CTL_HW_INT64
+# elif (defined(HW_PHYSMEM) || defined(HW_REALMEM))
+#  define CTL_HW_UINT
+# endif
+	int mib[2];
+
+	mib[0] = CTL_HW;
+# if defined(HW_MEMSIZE)
+	mib[1] = HW_MEMSIZE;
+# elif defined(HW_PHYSMEM64)
+	mib[1] = HW_PHYSMEM64;
+# elif defined(HW_REALMEM)
+	mib[1] = HW_REALMEM;
+# elif defined(HW_PYSMEM)
+	mib[1] = HW_PHYSMEM;
+# endif
+# if defined(CTL_HW_INT64)
+	unsigned long long size = 0;
+# elif defined(CTL_HW_UINT)
+	unsigned int size = 0;
+# endif
+# if defined(CTL_HW_INT64) || defined(CTL_HW_UINT)
+	size_t len = sizeof(size);
+
+	if (sysctl(mib, 2, &size, &len, NULL, 0) == 0)
+		return (unsigned long long)size;
+# endif
+	return 0;
+#else
+# warning "Don't know how to detect memory on your platform?"
+	return 0;
+#endif
+}
diff --git a/lib/config.h.in b/lib/config.h.in
index e318969..92fca3e 100644
--- a/lib/config.h.in
+++ b/lib/config.h.in
@@ -509,6 +509,9 @@
 /* Define to 1 if you have the <sys/syscall.h> header file. */
 #undef HAVE_SYS_SYSCALL_H
 
+/* Define to 1 if you have the <sys/sysctl.h> header file. */
+#undef HAVE_SYS_SYSCTL_H
+
 /* Define to 1 if you have the <sys/sysmacros.h> header file. */
 #undef HAVE_SYS_SYSMACROS_H
 
diff --git a/lib/ext2fs/dblist.c b/lib/ext2fs/dblist.c
index 942c4f0..bbdb221 100644
--- a/lib/ext2fs/dblist.c
+++ b/lib/ext2fs/dblist.c
@@ -194,20 +194,25 @@ void ext2fs_dblist_sort2(ext2_dblist dblist,
 /*
  * This function iterates over the directory block list
  */
-errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
+errcode_t ext2fs_dblist_iterate3(ext2_dblist dblist,
 				 int (*func)(ext2_filsys fs,
 					     struct ext2_db_entry2 *db_info,
 					     void	*priv_data),
+				 unsigned long long start,
+				 unsigned long long count,
 				 void *priv_data)
 {
-	unsigned long long	i;
+	unsigned long long	i, end;
 	int		ret;
 
 	EXT2_CHECK_MAGIC(dblist, EXT2_ET_MAGIC_DBLIST);
 
+	end = start + count;
 	if (!dblist->sorted)
 		ext2fs_dblist_sort2(dblist, 0);
-	for (i=0; i < dblist->count; i++) {
+	if (end > dblist->count)
+		end = dblist->count;
+	for (i = start; i < end; i++) {
 		ret = (*func)(dblist->fs, &dblist->list[i], priv_data);
 		if (ret & DBLIST_ABORT)
 			return 0;
@@ -215,6 +220,16 @@ errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
 	return 0;
 }
 
+errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
+				 int (*func)(ext2_filsys fs,
+					     struct ext2_db_entry2 *db_info,
+					     void	*priv_data),
+				 void *priv_data)
+{
+	return ext2fs_dblist_iterate3(dblist, func, 0, dblist->count,
+				      priv_data);
+}
+
 static EXT2_QSORT_TYPE dir_block_cmp2(const void *a, const void *b)
 {
 	const struct ext2_db_entry2 *db_a =
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index fef6910..21fe45c 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1055,11 +1055,17 @@ extern void ext2fs_dblist_sort2(ext2_dblist dblist,
 extern errcode_t ext2fs_dblist_iterate(ext2_dblist dblist,
 	int (*func)(ext2_filsys fs, struct ext2_db_entry *db_info,
 		    void	*priv_data),
-       void *priv_data);
+	void *priv_data);
 extern errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
 	int (*func)(ext2_filsys fs, struct ext2_db_entry2 *db_info,
 		    void	*priv_data),
-       void *priv_data);
+	void *priv_data);
+extern errcode_t ext2fs_dblist_iterate3(ext2_dblist dblist,
+	int (*func)(ext2_filsys fs, struct ext2_db_entry2 *db_info,
+		    void	*priv_data),
+	unsigned long long start,
+	unsigned long long count,
+	void *priv_data);
 extern errcode_t ext2fs_set_dir_block(ext2_dblist dblist, ext2_ino_t ino,
 				      blk_t blk, int blockcnt);
 extern errcode_t ext2fs_set_dir_block2(ext2_dblist dblist, ext2_ino_t ino,

[PATCH 21/39] e2fsck: read-ahead metadata during passes 1, 2, and 4

[Darrick J. Wong]

e2fsck pass1 is modified to use the block group data prefetch function
to try to fetch the inode tables into the pagecache before it is
needed.  We iterate through the blockgroups until we have enough inode
tables that need reading such that we can issue readahead; then we sit
and wait until the last inode table block read of the last group to
start fetching the next bunch.

pass2 is modified to use the dirblock prefetching function to prefetch
the list of directory blocks that are assembled in pass1.  We use the
"iterate a subset of a dblist" and avoid copying the dblist.  Directory
blocks are fetched incrementally as we walk through the directory
block list.  In previous iterations of this patch we would free the
directory blocks after processing, but the performance hit to e2fsck
itself wasn't worth it.  Furthermore, it is anticipated that most
users will then mount the FS and start using the directories, so they
may as well remain in the page cache.

pass4 is modified to prefetch the block and inode bitmaps in
anticipation of pass 5, because pass4 is entirely CPU bound.

In general, these mechanisms can decrease fsck time by 10-40%, if the
host system has sufficient memory and the storage system can provide a
lot of IOPs.  Pretty much any storage system capable of handling
multiple IOs in-flight at any time will see a fairly large performance
boost.  (Single-issue USB mass storage disks seem to suffer badly.)

By default, the readahead buffer size will be set to the size of a block
group's inode table (which is 2MiB for a regular ext4 FS).  The -E
readahead_kb= option can be given to specify the amount of memory to
use for readahead or zero to disable it entirely; or an option can be
given in e2fsck.conf.

v2: Fix an off-by-one error in the pass1 readahead which made the
readahead trigger one inode too late if the block groups are full.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/e2fsck.8.in      |    7 +++++
 e2fsck/e2fsck.conf.5.in |   15 +++++++++++
 e2fsck/e2fsck.h         |    3 ++
 e2fsck/pass1.c          |   65 +++++++++++++++++++++++++++++++++++++++++++++++
 e2fsck/pass2.c          |   38 +++++++++++++++++++++++++++
 e2fsck/pass4.c          |    9 +++++++
 e2fsck/unix.c           |   28 ++++++++++++++++++++
 lib/ext2fs/ext2fs.h     |    1 +
 lib/ext2fs/inode.c      |    3 +-
 9 files changed, 167 insertions(+), 2 deletions(-)


diff --git a/e2fsck/e2fsck.8.in b/e2fsck/e2fsck.8.in
index f5ed758..84ae50f 100644
--- a/e2fsck/e2fsck.8.in
+++ b/e2fsck/e2fsck.8.in
@@ -207,6 +207,13 @@ option may prevent you from further manual data recovery.
 .BI nodiscard
 Do not attempt to discard free blocks and unused inode blocks. This option is
 exactly the opposite of discard option. This is set as default.
+.TP
+.BI readahead_kb
+Use this many KiB of memory to pre-fetch metadata in the hopes of reducing
+e2fsck runtime.  By default, this is set to the size of a block group's inode
+table (typically 2MiB on a regular ext4 filesystem); if this amount is more
+than 1/100 of total physical memory, readahead is disabled.  Set this to zero
+to disable readahead entirely.
 .RE
 .TP
 .B \-f
diff --git a/e2fsck/e2fsck.conf.5.in b/e2fsck/e2fsck.conf.5.in
index 9ebfbbf..e1d0518 100644
--- a/e2fsck/e2fsck.conf.5.in
+++ b/e2fsck/e2fsck.conf.5.in
@@ -205,6 +205,21 @@ of that type are squelched.  This can be useful if the console is slow
 (i.e., connected to a serial port) and so a large amount of output could
 end up delaying the boot process for a long time (potentially hours).
 .TP
+.I readahead_mem_pct
+Use this percentage of memory to try to read in metadata blocks ahead of the
+main e2fsck thread.  This should reduce run times, depending on the speed of
+the underlying storage and the amount of free memory.  There is no default, but
+see
+.B readahead_mem_pct
+for more details.
+.TP
+.I readahead_kb
+Use this amount of memory to read in metadata blocks ahead of the main checking
+thread.  Setting this value to zero disables readahead entirely.  By default,
+this is set the size of one block group's inode table (typically 2MiB on a
+regular ext4 filesystem); if this amount is more than 1/100th of total physical
+memory, readahead is disabled.
+.TP
 .I report_features
 If this boolean relation is true, e2fsck will print the file system
 features as part of its verbose reporting (i.e., if the
diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index 8837af9..b2654ef 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -378,6 +378,9 @@ struct e2fsck_struct {
 	 */
 	void *priv_data;
 	ext2fs_block_bitmap block_metadata_map; /* Metadata blocks */
+
+	/* How much are we allowed to readahead? */
+	unsigned long long readahead_kb;
 };
 
 /* Used by the region allocation code */
diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index db5273e..36eea16 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -868,6 +868,60 @@ out:
 	return 0;
 }
 
+static void pass1_readahead(e2fsck_t ctx, dgrp_t *group, ext2_ino_t *next_ino)
+{
+	ext2_ino_t inodes_in_group = 0, inodes_per_block, inodes_per_buffer;
+	dgrp_t start = *group, grp;
+	blk64_t blocks_to_read = 0;
+	errcode_t err = EXT2_ET_INVALID_ARGUMENT;
+
+	if (ctx->readahead_kb == 0)
+		goto out;
+
+	/* Keep iterating groups until we have enough to readahead */
+	inodes_per_block = EXT2_INODES_PER_BLOCK(ctx->fs->super);
+	for (grp = start; grp < ctx->fs->group_desc_count; grp++) {
+		if (ext2fs_bg_flags_test(ctx->fs, grp, EXT2_BG_INODE_UNINIT))
+			continue;
+		inodes_in_group = ctx->fs->super->s_inodes_per_group -
+					ext2fs_bg_itable_unused(ctx->fs, grp);
+		blocks_to_read += (inodes_in_group + inodes_per_block - 1) /
+					inodes_per_block;
+		if (blocks_to_read * ctx->fs->blocksize >
+		    ctx->readahead_kb * 1024)
+			break;
+	}
+
+	err = e2fsck_readahead(ctx->fs, E2FSCK_READA_ITABLE, start,
+			       grp - start + 1);
+	if (err == EAGAIN) {
+		ctx->readahead_kb /= 2;
+		err = 0;
+	}
+
+out:
+	if (err) {
+		/* Error; disable itable readahead */
+		*group = ctx->fs->group_desc_count;
+		*next_ino = ctx->fs->super->s_inodes_count;
+	} else {
+		/*
+		 * Don't do more readahead until we've reached the first inode
+		 * of the last inode scan buffer block for the last group.
+		 */
+		*group = grp + 1;
+		inodes_per_buffer = (ctx->inode_buffer_blocks ?
+				     ctx->inode_buffer_blocks :
+				     EXT2_INODE_SCAN_DEFAULT_BUFFER_BLOCKS) *
+				    ctx->fs->blocksize /
+				    EXT2_INODE_SIZE(ctx->fs->super);
+		inodes_in_group--;
+		*next_ino = inodes_in_group -
+			    (inodes_in_group % inodes_per_buffer) + 1 +
+			    (grp * ctx->fs->super->s_inodes_per_group);
+	}
+}
+
 void e2fsck_pass1(e2fsck_t ctx)
 {
 	int	i;
@@ -890,10 +944,19 @@ void e2fsck_pass1(e2fsck_t ctx)
 	int		low_dtime_check = 1;
 	int		inode_size;
 	int		failed_csum = 0;
+	ext2_ino_t	ino_threshold = 0;
+	dgrp_t		ra_group = 0;
 
 	init_resource_track(&rtrack, ctx->fs->io);
 	clear_problem_context(&pctx);
 
+	/* If we can do readahead, figure out how many groups to pull in. */
+	if (!e2fsck_can_readahead(ctx->fs))
+		ctx->readahead_kb = 0;
+	else if (ctx->readahead_kb == ~0ULL)
+		ctx->readahead_kb = e2fsck_guess_readahead(ctx->fs);
+	pass1_readahead(ctx, &ra_group, &ino_threshold);
+
 	if (!(ctx->options & E2F_OPT_PREEN))
 		fix_problem(ctx, PR_1_PASS_HEADER, &pctx);
 
@@ -1073,6 +1136,8 @@ void e2fsck_pass1(e2fsck_t ctx)
 		old_op = ehandler_operation(_("getting next inode from scan"));
 		pctx.errcode = ext2fs_get_next_inode_full(scan, &ino,
 							  inode, inode_size);
+		if (ino > ino_threshold)
+			pass1_readahead(ctx, &ra_group, &ino_threshold);
 		ehandler_operation(old_op);
 		if (ctx->flags & E2F_FLAG_SIGNAL_MASK)
 			return;
diff --git a/e2fsck/pass2.c b/e2fsck/pass2.c
index 0b9c5c5..2060ed2 100644
--- a/e2fsck/pass2.c
+++ b/e2fsck/pass2.c
@@ -61,6 +61,9 @@
  * Keeps track of how many times an inode is referenced.
  */
 static void deallocate_inode(e2fsck_t ctx, ext2_ino_t ino, char* block_buf);
+static int check_dir_block2(ext2_filsys fs,
+			   struct ext2_db_entry2 *dir_blocks_info,
+			   void *priv_data);
 static int check_dir_block(ext2_filsys fs,
 			   struct ext2_db_entry2 *dir_blocks_info,
 			   void *priv_data);
@@ -77,6 +80,9 @@ struct check_dir_struct {
 	struct problem_context	pctx;
 	int	count, max;
 	e2fsck_t ctx;
+	unsigned long long list_offset;
+	unsigned long long ra_entries;
+	unsigned long long next_ra_off;
 };
 
 void e2fsck_pass2(e2fsck_t ctx)
@@ -96,6 +102,9 @@ void e2fsck_pass2(e2fsck_t ctx)
 	int			i, depth;
 	problem_t		code;
 	int			bad_dir;
+	int (*check_dir_func)(ext2_filsys fs,
+			      struct ext2_db_entry2 *dir_blocks_info,
+			      void *priv_data);
 
 	init_resource_track(&rtrack, ctx->fs->io);
 	clear_problem_context(&cd.pctx);
@@ -139,6 +148,9 @@ void e2fsck_pass2(e2fsck_t ctx)
 	cd.ctx = ctx;
 	cd.count = 1;
 	cd.max = ext2fs_dblist_count2(fs->dblist);
+	cd.list_offset = 0;
+	cd.ra_entries = ctx->readahead_kb * 1024 / ctx->fs->blocksize;
+	cd.next_ra_off = 0;
 
 	if (ctx->progress)
 		(void) (ctx->progress)(ctx, 2, 0, cd.max);
@@ -146,7 +158,8 @@ void e2fsck_pass2(e2fsck_t ctx)
 	if (fs->super->s_feature_compat & EXT2_FEATURE_COMPAT_DIR_INDEX)
 		ext2fs_dblist_sort2(fs->dblist, special_dir_block_cmp);
 
-	cd.pctx.errcode = ext2fs_dblist_iterate2(fs->dblist, check_dir_block,
+	check_dir_func = cd.ra_entries ? check_dir_block2 : check_dir_block;
+	cd.pctx.errcode = ext2fs_dblist_iterate2(fs->dblist, check_dir_func,
 						 &cd);
 	if (ctx->flags & E2F_FLAG_SIGNAL_MASK || ctx->flags & E2F_FLAG_RESTART)
 		return;
@@ -824,6 +837,29 @@ err:
 	return retval;
 }
 
+static int check_dir_block2(ext2_filsys fs,
+			   struct ext2_db_entry2 *db,
+			   void *priv_data)
+{
+	int err;
+	struct check_dir_struct *cd = priv_data;
+
+	if (cd->ra_entries && cd->list_offset >= cd->next_ra_off) {
+		err = e2fsck_readahead_dblist(fs,
+					E2FSCK_RA_DBLIST_IGNORE_BLOCKCNT,
+					fs->dblist,
+					cd->list_offset + cd->ra_entries / 8,
+					cd->ra_entries);
+		if (err)
+			cd->ra_entries = 0;
+		cd->next_ra_off = cd->list_offset + (cd->ra_entries * 7 / 8);
+	}
+
+	err = check_dir_block(fs, db, priv_data);
+	cd->list_offset++;
+	return err;
+}
+
 static int check_dir_block(ext2_filsys fs,
 			   struct ext2_db_entry2 *db,
 			   void *priv_data)
diff --git a/e2fsck/pass4.c b/e2fsck/pass4.c
index 21d93f0..bc9a2c4 100644
--- a/e2fsck/pass4.c
+++ b/e2fsck/pass4.c
@@ -106,6 +106,15 @@ void e2fsck_pass4(e2fsck_t ctx)
 #ifdef MTRACE
 	mtrace_print("Pass 4");
 #endif
+	/*
+	 * Since pass4 is mostly CPU bound, start readahead of bitmaps
+	 * ahead of pass 5 if we haven't already loaded them.
+	 */
+	if (ctx->readahead_kb &&
+	    (fs->block_map == NULL || fs->inode_map == NULL))
+		e2fsck_readahead(fs, E2FSCK_READA_BBITMAP |
+				     E2FSCK_READA_IBITMAP,
+				 0, fs->group_desc_count);
 
 	clear_problem_context(&pctx);
 
diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index 615d690..f3672c0 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -650,6 +650,7 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 	char	*buf, *token, *next, *p, *arg;
 	int	ea_ver;
 	int	extended_usage = 0;
+	unsigned long long reada_kb;
 
 	buf = string_copy(ctx, opts, 0);
 	for (token = buf; token && *token; token = next) {
@@ -678,6 +679,15 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 				continue;
 			}
 			ctx->ext_attr_ver = ea_ver;
+		} else if (strcmp(token, "readahead_kb") == 0) {
+			reada_kb = strtoull(arg, &p, 0);
+			if (*p) {
+				fprintf(stderr, "%s",
+					_("Invalid readahead buffer size.\n"));
+				extended_usage++;
+				continue;
+			}
+			ctx->readahead_kb = reada_kb;
 		} else if (strcmp(token, "fragcheck") == 0) {
 			ctx->options |= E2F_OPT_FRAGCHECK;
 			continue;
@@ -717,6 +727,7 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 		fputs(("\tjournal_only\n"), stderr);
 		fputs(("\tdiscard\n"), stderr);
 		fputs(("\tnodiscard\n"), stderr);
+		fputs(("\treadahead_kb=<buffer size>\n"), stderr);
 		fputc('\n', stderr);
 		exit(1);
 	}
@@ -750,6 +761,7 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 #ifdef CONFIG_JBD_DEBUG
 	char 		*jbd_debug;
 #endif
+	unsigned long long phys_mem_kb;
 
 	retval = e2fsck_allocate_context(&ctx);
 	if (retval)
@@ -777,6 +789,8 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 	else
 		ctx->program_name = "e2fsck";
 
+	phys_mem_kb = get_memory_size() / 1024;
+	ctx->readahead_kb = ~0ULL;
 	while ((c = getopt (argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDk")) != EOF)
 		switch (c) {
 		case 'C':
@@ -961,6 +975,20 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 	if (c)
 		verbose = 1;
 
+	if (ctx->readahead_kb == ~0ULL) {
+		profile_get_integer(ctx->profile, "options",
+				    "readahead_mem_pct", 0, -1, &c);
+		if (c >= 0 && c <= 100)
+			ctx->readahead_kb = phys_mem_kb * c / 100;
+		profile_get_integer(ctx->profile, "options",
+				    "readahead_kb", 0, -1, &c);
+		if (c >= 0)
+			ctx->readahead_kb = c;
+		if (ctx->readahead_kb != ~0ULL &&
+		    ctx->readahead_kb > phys_mem_kb)
+			ctx->readahead_kb = phys_mem_kb;
+	}
+
 	/* Turn off discard in read-only mode */
 	if ((ctx->options & E2F_OPT_NO) &&
 	    (ctx->options & E2F_OPT_DISCARD))
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 21fe45c..689f6a6 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1421,6 +1421,7 @@ extern errcode_t ext2fs_get_next_inode_full(ext2_inode_scan scan,
 					    ext2_ino_t *ino,
 					    struct ext2_inode *inode,
 					    int bufsize);
+#define EXT2_INODE_SCAN_DEFAULT_BUFFER_BLOCKS	8
 extern errcode_t ext2fs_open_inode_scan(ext2_filsys fs, int buffer_blocks,
 				  ext2_inode_scan *ret_scan);
 extern void ext2fs_close_inode_scan(ext2_inode_scan scan);
diff --git a/lib/ext2fs/inode.c b/lib/ext2fs/inode.c
index 4310b82..4b3e14e 100644
--- a/lib/ext2fs/inode.c
+++ b/lib/ext2fs/inode.c
@@ -175,7 +175,8 @@ errcode_t ext2fs_open_inode_scan(ext2_filsys fs, int buffer_blocks,
 	scan->bytes_left = 0;
 	scan->current_group = 0;
 	scan->groups_left = fs->group_desc_count - 1;
-	scan->inode_buffer_blocks = buffer_blocks ? buffer_blocks : 8;
+	scan->inode_buffer_blocks = buffer_blocks ? buffer_blocks :
+				    EXT2_INODE_SCAN_DEFAULT_BUFFER_BLOCKS;
 	scan->current_block = ext2fs_inode_table_loc(scan->fs,
 						     scan->current_group);
 	scan->inodes_left = EXT2_INODES_PER_GROUP(scan->fs->super);

[PATCH 22/39] libext2fs: ext2fs_new_block2() should call alloc_block hook

[Darrick J. Wong]

If ext2fs_new_block2() is called without a specific block map, we
should call the alloc_block hook before checking fs->block_map.  This
helps us to avoid a bug in e2fsck where we need to allocate a block
but instead of consulting block_found_map, we use the FS bitmaps,
which (prior to pass 5) could be wrong.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1.c     |    2 +-
 lib/ext2fs/alloc.c |   15 +++++++++++++++
 2 files changed, 16 insertions(+), 1 deletion(-)


diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index 36eea16..d4760ef 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -3790,7 +3790,7 @@ static errcode_t e2fsck_get_alloc_block(ext2_filsys fs, blk64_t goal,
 				return retval;
 		}
 
-		retval = ext2fs_new_block2(fs, goal, 0, &new_block);
+		retval = ext2fs_new_block2(fs, goal, fs->block_map, &new_block);
 		if (retval)
 			return retval;
 	}
diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
index 578fd7f..d1c1a84 100644
--- a/lib/ext2fs/alloc.c
+++ b/lib/ext2fs/alloc.c
@@ -137,9 +137,23 @@ errcode_t ext2fs_new_block2(ext2_filsys fs, blk64_t goal,
 {
 	errcode_t retval;
 	blk64_t	b = 0;
+	errcode_t (*gab)(ext2_filsys fs, blk64_t goal, blk64_t *ret);
 
 	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
 
+	if (!map && fs->get_alloc_block) {
+		/*
+		 * In case there are clients out there whose get_alloc_block
+		 * handlers call ext2fs_new_block2 with a NULL block map,
+		 * temporarily swap out the function pointer so that we don't
+		 * end up in an infinite loop.
+		 */
+		gab = fs->get_alloc_block;
+		fs->get_alloc_block = NULL;
+		retval = gab(fs, goal, &b);
+		fs->get_alloc_block = gab;
+		goto allocated;
+	}
 	if (!map)
 		map = fs->block_map;
 	if (!map)
@@ -153,6 +167,7 @@ errcode_t ext2fs_new_block2(ext2_filsys fs, blk64_t goal,
 	if ((retval == ENOENT) && (goal != fs->super->s_first_data_block))
 		retval = ext2fs_find_first_zero_block_bitmap2(map,
 			fs->super->s_first_data_block, goal - 1, &b);
+allocated:
 	if (retval == ENOENT)
 		return EXT2_ET_BLOCK_ALLOC_FAIL;
 	if (retval)

[PATCH 23/39] libext2fs: support BLKZEROOUT/FALLOC_FL_ZERO_RANGE in ext2fs_zero_blocks

[Darrick J. Wong]

Plumb a new call into the IO manager to support translating
ext2fs_zero_blocks calls into the equivalent kernel-level BLKZEROOUT
ioctl or FALLOC_FL_ZERO_RANGE fallocate flag primitives when possible.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/ext2_io.h    |    7 ++-
 lib/ext2fs/io_manager.c |   11 ++++
 lib/ext2fs/mkjournal.c  |    6 ++
 lib/ext2fs/test_io.c    |   21 ++++++++
 lib/ext2fs/unix_io.c    |  125 +++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 169 insertions(+), 1 deletion(-)


diff --git a/lib/ext2fs/ext2_io.h b/lib/ext2fs/ext2_io.h
index 4c5a5c5..1faa720 100644
--- a/lib/ext2fs/ext2_io.h
+++ b/lib/ext2fs/ext2_io.h
@@ -93,7 +93,9 @@ struct struct_io_manager {
 	errcode_t (*cache_readahead)(io_channel channel,
 				     unsigned long long block,
 				     unsigned long long count);
-	long	reserved[15];
+	errcode_t (*zeroout)(io_channel channel, unsigned long long block,
+			     unsigned long long count);
+	long	reserved[14];
 };
 
 #define IO_FLAG_RW		0x0001
@@ -125,6 +127,9 @@ extern errcode_t io_channel_write_blk64(io_channel channel,
 extern errcode_t io_channel_discard(io_channel channel,
 				    unsigned long long block,
 				    unsigned long long count);
+extern errcode_t io_channel_zeroout(io_channel channel,
+				    unsigned long long block,
+				    unsigned long long count);
 extern errcode_t io_channel_alloc_buf(io_channel channel,
 				      int count, void *ptr);
 extern errcode_t io_channel_cache_readahead(io_channel io,
diff --git a/lib/ext2fs/io_manager.c b/lib/ext2fs/io_manager.c
index dc5888d..c395d61 100644
--- a/lib/ext2fs/io_manager.c
+++ b/lib/ext2fs/io_manager.c
@@ -112,6 +112,17 @@ errcode_t io_channel_discard(io_channel channel, unsigned long long block,
 	return EXT2_ET_UNIMPLEMENTED;
 }
 
+errcode_t io_channel_zeroout(io_channel channel, unsigned long long block,
+			     unsigned long long count)
+{
+	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+
+	if (channel->manager->zeroout)
+		return (channel->manager->zeroout)(channel, block, count);
+
+	return EXT2_ET_UNIMPLEMENTED;
+}
+
 errcode_t io_channel_alloc_buf(io_channel io, int count, void *ptr)
 {
 	size_t	size;
diff --git a/lib/ext2fs/mkjournal.c b/lib/ext2fs/mkjournal.c
index 6f3a862..5be425c 100644
--- a/lib/ext2fs/mkjournal.c
+++ b/lib/ext2fs/mkjournal.c
@@ -164,6 +164,12 @@ errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
 		}
 		return 0;
 	}
+
+	/* Try a zero out command, if supported */
+	retval = io_channel_zeroout(fs->io, blk, num);
+	if (retval == 0)
+		return 0;
+
 	/* Allocate the zeroizing buffer if necessary */
 	if (!buf) {
 		buf = malloc(fs->blocksize * STRIDE_LENGTH);
diff --git a/lib/ext2fs/test_io.c b/lib/ext2fs/test_io.c
index b03a939..f7c50d1 100644
--- a/lib/ext2fs/test_io.c
+++ b/lib/ext2fs/test_io.c
@@ -86,6 +86,7 @@ void (*test_io_cb_write_byte)
 #define TEST_FLAG_SET_OPTION		0x20
 #define TEST_FLAG_DISCARD		0x40
 #define TEST_FLAG_READAHEAD		0x80
+#define TEST_FLAG_ZEROOUT		0x100
 
 static void test_dump_block(io_channel channel,
 			    struct test_private_data *data,
@@ -507,6 +508,25 @@ static errcode_t test_cache_readahead(io_channel channel,
 	return retval;
 }
 
+static errcode_t test_zeroout(io_channel channel, unsigned long long block,
+			      unsigned long long count)
+{
+	struct test_private_data *data;
+	errcode_t	retval = 0;
+
+	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+	data = (struct test_private_data *) channel->private_data;
+	EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_TEST_IO_CHANNEL);
+
+	if (data->real)
+		retval = io_channel_zeroout(data->real, block, count);
+	if (data->flags & TEST_FLAG_ZEROOUT)
+		fprintf(data->outfile,
+			"Test_io: zeroout(%llu, %llu) returned %s\n",
+			block, count, retval ? error_message(retval) : "OK");
+	return retval;
+}
+
 static struct struct_io_manager struct_test_manager = {
 	.magic		= EXT2_ET_MAGIC_IO_MANAGER,
 	.name		= "Test I/O Manager",
@@ -523,6 +543,7 @@ static struct struct_io_manager struct_test_manager = {
 	.write_blk64	= test_write_blk64,
 	.discard	= test_discard,
 	.cache_readahead	= test_cache_readahead,
+	.zeroout	= test_zeroout,
 };
 
 io_manager test_io_manager = &struct_test_manager;
diff --git a/lib/ext2fs/unix_io.c b/lib/ext2fs/unix_io.c
index c3a8ea5..6bc543b 100644
--- a/lib/ext2fs/unix_io.c
+++ b/lib/ext2fs/unix_io.c
@@ -987,6 +987,130 @@ unimplemented:
 	return EXT2_ET_UNIMPLEMENTED;
 }
 
+#if defined(__linux__) && !defined(BLKZEROOUT)
+#define BLKZEROOUT		_IO(0x12, 127)
+#endif
+
+#if defined(__linux__) && !defined(BLKFLSBUF)
+#define BLKFLSBUF		_IO(0x12, 97)
+#endif
+
+#if defined(__linux__) && !defined(FALLOC_FL_ZERO_RANGE)
+#define FALLOC_FL_ZERO_RANGE    0x10
+#endif
+
+static errcode_t unix_zeroout(io_channel channel, unsigned long long block,
+			      unsigned long long count)
+{
+	struct unix_private_data *data;
+	int		ret;
+
+	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+	data = (struct unix_private_data *) channel->private_data;
+	EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL);
+
+	if (getenv("UNIX_IO_NOZEROOUT"))
+		goto unimplemented;
+
+	if (channel->flags & CHANNEL_FLAGS_BLOCK_DEVICE) {
+		__u64 range[2];
+
+		if (count == 0)
+			return 0;
+		/*
+		 * This is a heavyweight operation.  Refuse to
+		 * zero-out a single block, and hope that the
+		 * caller will fall back to io_channel_write().
+		 */
+		else if (count == 1)
+			goto unimplemented;
+
+		range[0] = (__u64)(block) * channel->block_size;
+		range[1] = (__u64)(count) * channel->block_size;
+
+		errno = EOPNOTSUPP;
+		ret = -1;
+#if defined(BLKZEROOUT) && (defined(POSIX_FADV_DONTNEED) || defined(BLKFLSBUF))
+		/*
+		 * BLKZEROOUT doesn't invalidate the page cache.  Therefore,
+		 * we need to flush dirty pages to disk prior to calling the
+		 * ioctl and invalidate at least those same pages from the page
+		 * cache afterwards.  If we're using direct IO we can skip all
+		 * that wrangling.
+		 */
+		if (!(data->flags & IO_FLAG_DIRECT_IO)) {
+			ret = fsync(data->dev);
+			if (ret)
+				goto err;
+		}
+		ret = ioctl(data->dev, BLKZEROOUT, range);
+		if (ret)
+			goto err;
+		if (!(data->flags & IO_FLAG_DIRECT_IO)) {
+#ifdef POSIX_FADV_DONTNEED
+			ret = posix_fadvise(data->dev, range[0], range[1],
+					    POSIX_FADV_DONTNEED);
+			if (ret == 0)
+				goto err;
+#endif
+#ifdef BLKFLSBUF
+			ret = ioctl(data->dev, BLKFLSBUF, 0);
+#endif
+		}
+#elif defined(BLKZEROOUT)
+		if (data->flags & IO_FLAG_DIRECT_IO)
+			ret = ioctl(data->dev, BLKZEROOUT, range);
+#endif
+	} else {
+		/* Regular file, try to use truncate/punch/zero. */
+#if defined(HAVE_FALLOCATE) && (defined(FALLOC_FL_ZERO_RANGE) || \
+	(defined(FALLOC_FL_PUNCH_HOLE) && defined(FALLOC_FL_KEEP_SIZE)))
+		struct stat statbuf;
+
+		if (count == 0)
+			return 0;
+		/*
+		 * If we're trying to zero a range past the end of the file,
+		 * extend the file size, then punch (or zero_range) everything.
+		 */
+		ret = fstat(data->dev, &statbuf);
+		if (ret)
+			goto err;
+		if (statbuf.st_size < (block + count) * channel->block_size) {
+			ret = ftruncate(data->dev,
+					(block + count) * channel->block_size);
+			if (ret)
+				goto err;
+		}
+#if defined(FALLOC_FL_PUNCH_HOLE) && defined(FALLOC_FL_KEEP_SIZE)
+		ret = fallocate(data->dev,
+				FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
+				(off_t)(block) * channel->block_size,
+				(off_t)(count) * channel->block_size);
+		if (ret == 0)
+			goto err;
+#endif
+#ifdef FALLOC_FL_ZERO_RANGE
+		ret = fallocate(data->dev,
+				FALLOC_FL_ZERO_RANGE,
+				(off_t)(block) * channel->block_size,
+				(off_t)(count) * channel->block_size);
+#endif
+#else
+		goto unimplemented;
+#endif /* HAVE_FALLOCATE && (ZERO_RANGE || (PUNCH_HOLE && KEEP_SIZE)) */
+	}
+err:
+	if (ret < 0) {
+		if (errno == EOPNOTSUPP)
+			goto unimplemented;
+		return errno;
+	}
+	return 0;
+unimplemented:
+	return EXT2_ET_UNIMPLEMENTED;
+}
+
 static struct struct_io_manager struct_unix_manager = {
 	.magic		= EXT2_ET_MAGIC_IO_MANAGER,
 	.name		= "Unix I/O Manager",
@@ -1003,6 +1127,7 @@ static struct struct_io_manager struct_unix_manager = {
 	.write_blk64	= unix_write_blk64,
 	.discard	= unix_discard,
 	.cache_readahead	= unix_cache_readahead,
+	.zeroout	= unix_zeroout,
 };
 
 io_manager unix_io_manager = &struct_unix_manager;

[PATCH 24/39] libext2fs/e2fsck: refactor everyone who writes zero blocks to disk

[Darrick J. Wong]

Convert all call sites that write zero blocks to disk to use
ext2fs_zero_blocks2() since it can use Linux's zero out feature to do
the writes more quickly.  Reclaim the zero buffer at freefs time and
make the write-zeroes fallback use a larger buffer.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/e2fsck.h        |    2 -
 e2fsck/pass1.c         |   13 +++++---
 e2fsck/pass3.c         |   13 ++------
 e2fsck/util.c          |   53 ----------------------------------
 lib/ext2fs/alloc.c     |   14 ++-------
 lib/ext2fs/expanddir.c |   13 ++------
 lib/ext2fs/ext2fs.h    |    3 ++
 lib/ext2fs/freefs.c    |    1 +
 lib/ext2fs/mkjournal.c |   75 +++++++++++++++++++++++++++++++++++-------------
 misc/mke2fs.c          |    2 -
 resize/resize2fs.c     |   25 ++++++----------
 11 files changed, 84 insertions(+), 130 deletions(-)


diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h
index b2654ef..e359515 100644
--- a/e2fsck/e2fsck.h
+++ b/e2fsck/e2fsck.h
@@ -544,8 +544,6 @@ extern void e2fsck_read_bitmaps(e2fsck_t ctx);
 extern void e2fsck_write_bitmaps(e2fsck_t ctx);
 extern void preenhalt(e2fsck_t ctx);
 extern char *string_copy(e2fsck_t ctx, const char *str, int len);
-extern errcode_t e2fsck_zero_blocks(ext2_filsys fs, blk_t blk, int num,
-				    blk_t *ret_blk, int *ret_count);
 extern int fs_proc_check(const char *fs_name);
 extern int check_for_modules(const char *fs_name);
 #ifdef RESOURCE_TRACK
diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index d4760ef..a963849 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -3576,12 +3576,15 @@ static void new_table_block(e2fsck_t ctx, blk64_t first_block, dgrp_t group,
 				   old_block + i, 1, buf);
 			if (pctx.errcode)
 				fix_problem(ctx, PR_1_RELOC_READ_ERR, &pctx);
-		} else
-			memset(buf, 0, fs->blocksize);
+			pctx.blk = (*new_block) + i;
+			pctx.errcode = io_channel_write_blk64(fs->io, pctx.blk,
+							      1, buf);
+		} else {
+			pctx.blk = (*new_block) + i;
+			pctx.errcode = ext2fs_zero_blocks2(fs, pctx.blk, 1,
+							   NULL, NULL);
+		}
 
-		pctx.blk = (*new_block) + i;
-		pctx.errcode = io_channel_write_blk64(fs->io, pctx.blk,
-					      1, buf);
 		if (pctx.errcode)
 			fix_problem(ctx, PR_1_RELOC_WRITE_ERR, &pctx);
 	}
diff --git a/e2fsck/pass3.c b/e2fsck/pass3.c
index f03c7ae..2d94ece 100644
--- a/e2fsck/pass3.c
+++ b/e2fsck/pass3.c
@@ -809,20 +809,13 @@ static int expand_dir_proc(ext2_filsys fs,
 		es->num--;
 		retval = ext2fs_write_dir_block4(fs, new_blk, block, 0,
 						 es->dir);
-	} else {
-		retval = ext2fs_get_mem(fs->blocksize, &block);
-		if (retval) {
-			es->err = retval;
-			return BLOCK_ABORT;
-		}
-		memset(block, 0, fs->blocksize);
-		retval = io_channel_write_blk64(fs->io, new_blk, 1, block);
-	}
+		ext2fs_free_mem(&block);
+	} else
+		retval = ext2fs_zero_blocks2(fs, new_blk, 1, NULL, NULL);
 	if (retval) {
 		es->err = retval;
 		return BLOCK_ABORT;
 	}
-	ext2fs_free_mem(&block);
 	*blocknr = new_blk;
 	ext2fs_mark_block_bitmap2(ctx->block_found_map, new_blk);
 
diff --git a/e2fsck/util.c b/e2fsck/util.c
index 74f20062..723dafb 100644
--- a/e2fsck/util.c
+++ b/e2fsck/util.c
@@ -612,59 +612,6 @@ int ext2_file_type(unsigned int mode)
 	return 0;
 }
 
-#define STRIDE_LENGTH 8
-/*
- * Helper function which zeros out _num_ blocks starting at _blk_.  In
- * case of an error, the details of the error is returned via _ret_blk_
- * and _ret_count_ if they are non-NULL pointers.  Returns 0 on
- * success, and an error code on an error.
- *
- * As a special case, if the first argument is NULL, then it will
- * attempt to free the static zeroizing buffer.  (This is to keep
- * programs that check for memory leaks happy.)
- */
-errcode_t e2fsck_zero_blocks(ext2_filsys fs, blk_t blk, int num,
-			     blk_t *ret_blk, int *ret_count)
-{
-	int		j, count;
-	static char	*buf;
-	errcode_t	retval;
-
-	/* If fs is null, clean up the static buffer and return */
-	if (!fs) {
-		if (buf) {
-			free(buf);
-			buf = 0;
-		}
-		return 0;
-	}
-	/* Allocate the zeroizing buffer if necessary */
-	if (!buf) {
-		buf = malloc(fs->blocksize * STRIDE_LENGTH);
-		if (!buf) {
-			com_err("malloc", ENOMEM, "%s",
-				_("while allocating zeroizing buffer"));
-			exit(1);
-		}
-		memset(buf, 0, fs->blocksize * STRIDE_LENGTH);
-	}
-	/* OK, do the write loop */
-	for (j = 0; j < num; j += STRIDE_LENGTH, blk += STRIDE_LENGTH) {
-		count = num - j;
-		if (count > STRIDE_LENGTH)
-			count = STRIDE_LENGTH;
-		retval = io_channel_write_blk64(fs->io, blk, count, buf);
-		if (retval) {
-			if (ret_count)
-				*ret_count = count;
-			if (ret_blk)
-				*ret_blk = blk;
-			return retval;
-		}
-	}
-	return 0;
-}
-
 /*
  * Check to see if a filesystem is in /proc/filesystems.
  * Returns 1 if found, 0 if not
diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
index d1c1a84..82af18e 100644
--- a/lib/ext2fs/alloc.c
+++ b/lib/ext2fs/alloc.c
@@ -198,15 +198,9 @@ errcode_t ext2fs_alloc_block2(ext2_filsys fs, blk64_t goal,
 {
 	errcode_t	retval;
 	blk64_t		block;
-	char		*buf = 0;
 
-	if (!block_buf) {
-		retval = ext2fs_get_mem(fs->blocksize, &buf);
-		if (retval)
-			return retval;
-		block_buf = buf;
-	}
-	memset(block_buf, 0, fs->blocksize);
+	if (block_buf)
+		memset(block_buf, 0, fs->blocksize);
 
 	if (fs->get_alloc_block) {
 		retval = (fs->get_alloc_block)(fs, goal, &block);
@@ -224,7 +218,7 @@ errcode_t ext2fs_alloc_block2(ext2_filsys fs, blk64_t goal,
 			goto fail;
 	}
 
-	retval = io_channel_write_blk64(fs->io, block, 1, block_buf);
+	retval = ext2fs_zero_blocks3(fs, block, 1, NULL, NULL, block_buf);
 	if (retval)
 		goto fail;
 
@@ -232,8 +226,6 @@ errcode_t ext2fs_alloc_block2(ext2_filsys fs, blk64_t goal,
 	*ret = block;
 
 fail:
-	if (buf)
-		ext2fs_free_mem(&buf);
 	return retval;
 }
 
diff --git a/lib/ext2fs/expanddir.c b/lib/ext2fs/expanddir.c
index d0f7287..ecc13ae 100644
--- a/lib/ext2fs/expanddir.c
+++ b/lib/ext2fs/expanddir.c
@@ -67,22 +67,15 @@ static int expand_dir_proc(ext2_filsys	fs,
 		es->done = 1;
 		retval = ext2fs_write_dir_block4(fs, new_blk, block, 0,
 						 es->dir);
-	} else {
-		retval = ext2fs_get_mem(fs->blocksize, &block);
-		if (retval) {
-			es->err = retval;
-			return BLOCK_ABORT;
-		}
-		memset(block, 0, fs->blocksize);
-		retval = io_channel_write_blk64(fs->io, new_blk, 1, block);
-	}
+		ext2fs_free_mem(&block);
+	} else
+		retval = ext2fs_zero_blocks2(fs, new_blk, 1, NULL, NULL);
 	if (blockcnt >= 0)
 		es->goal = new_blk;
 	if (retval) {
 		es->err = retval;
 		return BLOCK_ABORT;
 	}
-	ext2fs_free_mem(&block);
 	*blocknr = new_blk;
 
 	if (es->done)
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 689f6a6..5ecaa85 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1504,6 +1504,9 @@ extern errcode_t ext2fs_zero_blocks(ext2_filsys fs, blk_t blk, int num,
 				    blk_t *ret_blk, int *ret_count);
 extern errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
 				     blk64_t *ret_blk, int *ret_count);
+extern errcode_t ext2fs_zero_blocks3(ext2_filsys fs, blk64_t blk, int num,
+				     blk64_t *ret_blk, int *ret_count,
+				     const void *block_buf);
 extern errcode_t ext2fs_create_journal_superblock(ext2_filsys fs,
 						  __u32 num_blocks, int flags,
 						  char  **ret_jsb);
diff --git a/lib/ext2fs/freefs.c b/lib/ext2fs/freefs.c
index 89a157b..ea9742e 100644
--- a/lib/ext2fs/freefs.c
+++ b/lib/ext2fs/freefs.c
@@ -61,6 +61,7 @@ void ext2fs_free(ext2_filsys fs)
 
 	fs->magic = 0;
 
+	ext2fs_zero_blocks2(NULL, 0, 0, NULL, NULL);
 	ext2fs_free_mem(&fs);
 }
 
diff --git a/lib/ext2fs/mkjournal.c b/lib/ext2fs/mkjournal.c
index 5be425c..2cd7ca5 100644
--- a/lib/ext2fs/mkjournal.c
+++ b/lib/ext2fs/mkjournal.c
@@ -148,12 +148,14 @@ errfree:
  * attempt to free the static zeroizing buffer.  (This is to keep
  * programs that check for memory leaks happy.)
  */
-#define STRIDE_LENGTH 8
-errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
-			      blk64_t *ret_blk, int *ret_count)
+#define MAX_STRIDE_LENGTH (4194304 / fs->blocksize)
+errcode_t ext2fs_zero_blocks3(ext2_filsys fs, blk64_t blk, int num,
+			      blk64_t *ret_blk, int *ret_count,
+			      const void *block_buf)
 {
 	int		j, count;
-	static char	*buf;
+	static void	*buf;
+	static unsigned	stride_length;
 	errcode_t	retval;
 
 	/* If fs is null, clean up the static buffer and return */
@@ -165,29 +167,56 @@ errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
 		return 0;
 	}
 
+	/* Deal with zeroing less than 1 block */
+	if (num <= 0)
+		return 0;
+
 	/* Try a zero out command, if supported */
 	retval = io_channel_zeroout(fs->io, blk, num);
 	if (retval == 0)
 		return 0;
 
+	/* If the user gave us a buffer, write that out */
+	if (block_buf) {
+		retval = io_channel_write_blk64(fs->io, blk, num, block_buf);
+		if (retval) {
+			if (ret_count)
+				*ret_count = num;
+			if (ret_blk)
+				*ret_blk = blk;
+		}
+
+		return retval;
+	}
+
 	/* Allocate the zeroizing buffer if necessary */
-	if (!buf) {
-		buf = malloc(fs->blocksize * STRIDE_LENGTH);
-		if (!buf)
-			return ENOMEM;
-		memset(buf, 0, fs->blocksize * STRIDE_LENGTH);
+	if (num > stride_length) {
+		void *p;
+		unsigned new_stride = num;
+
+		if (new_stride > MAX_STRIDE_LENGTH)
+			new_stride = MAX_STRIDE_LENGTH;
+		if (new_stride == stride_length)
+			goto skip_alloc;
+		p = realloc(buf, fs->blocksize * new_stride);
+		if (!p)
+			return EXT2_ET_NO_MEMORY;
+		buf = p;
+		stride_length = new_stride;
+		memset(buf, 0, fs->blocksize * stride_length);
 	}
+skip_alloc:
 	/* OK, do the write loop */
 	j=0;
 	while (j < num) {
-		if (blk % STRIDE_LENGTH) {
-			count = STRIDE_LENGTH - (blk % STRIDE_LENGTH);
+		if (blk % stride_length) {
+			count = stride_length - (blk % stride_length);
 			if (count > (num - j))
 				count = num - j;
 		} else {
 			count = num - j;
-			if (count > STRIDE_LENGTH)
-				count = STRIDE_LENGTH;
+			if (count > stride_length)
+				count = stride_length;
 		}
 		retval = io_channel_write_blk64(fs->io, blk, count, buf);
 		if (retval) {
@@ -202,6 +231,12 @@ errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
 	return 0;
 }
 
+errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
+			      blk64_t *ret_blk, int *ret_count)
+{
+	return ext2fs_zero_blocks3(fs, blk, num, ret_blk, ret_count, NULL);
+}
+
 errcode_t ext2fs_zero_blocks(ext2_filsys fs, blk_t blk, int num,
 			     blk_t *ret_blk, int *ret_count)
 {
@@ -372,20 +407,20 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 	retval = ext2fs_block_iterate3(fs, journal_ino, BLOCK_FLAG_APPEND,
 				       0, mkjournal_proc, &es);
 	if (retval)
-		goto errout;
+		goto out2;
 	if (es.err) {
 		retval = es.err;
-		goto errout;
+		goto out2;
 	}
 	if (es.zero_count) {
 		retval = ext2fs_zero_blocks2(fs, es.blk_to_zero,
 					    es.zero_count, 0, 0);
 		if (retval)
-			goto errout;
+			goto out2;
 	}
 
 	if ((retval = ext2fs_read_inode(fs, journal_ino, &inode)))
-		goto errout;
+		goto out2;
 
 	inode_size = (unsigned long long)fs->blocksize * num_blocks;
 	ext2fs_iblk_add_blocks(fs, &inode, es.newblocks);
@@ -394,10 +429,10 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 	inode.i_mode = LINUX_S_IFREG | 0600;
 	retval = ext2fs_inode_size_set(fs, &inode, inode_size);
 	if (retval)
-		goto errout;
+		goto out2;
 
 	if ((retval = ext2fs_write_new_inode(fs, journal_ino, &inode)))
-		goto errout;
+		goto out2;
 	retval = 0;
 
 	memcpy(fs->super->s_jnl_blocks, inode.i_block, EXT2_N_BLOCKS*4);
@@ -406,8 +441,6 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 	fs->super->s_jnl_backup_type = EXT3_JNL_BACKUP_BLOCKS;
 	ext2fs_mark_super_dirty(fs);
 
-errout:
-	ext2fs_zero_blocks2(0, 0, 0, 0, 0);
 out2:
 	ext2fs_free_mem(&buf);
 	return retval;
diff --git a/misc/mke2fs.c b/misc/mke2fs.c
index c6a2c2f..149330c 100644
--- a/misc/mke2fs.c
+++ b/misc/mke2fs.c
@@ -437,7 +437,6 @@ static void write_inode_tables(ext2_filsys fs, int lazy_flag, int itable_zeroed)
 				sync();
 		}
 	}
-	ext2fs_zero_blocks2(0, 0, 0, 0, 0);
 	ext2fs_numeric_progress_close(fs, &progress,
 				      _("done                            \n"));
 
@@ -626,7 +625,6 @@ static void create_journal_dev(ext2_filsys fs)
 		count -= c;
 		ext2fs_numeric_progress_update(fs, &progress, blk);
 	}
-	ext2fs_zero_blocks2(0, 0, 0, 0, 0);
 
 	ext2fs_numeric_progress_close(fs, &progress, NULL);
 write_superblock:
diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index e71acf3..93df070 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -762,11 +762,11 @@ static errcode_t adjust_superblock(ext2_resize_t rfs, blk64_t new_size)
 		/*
 		 * Write out the new inode table
 		 */
-		retval = io_channel_write_blk64(fs->io,
-						ext2fs_inode_table_loc(fs, i),
-						fs->inode_blocks_per_group,
-						rfs->itable_buf);
-		if (retval) goto errout;
+		retval = ext2fs_zero_blocks2(fs, ext2fs_inode_table_loc(fs, i),
+					     fs->inode_blocks_per_group, NULL,
+					     NULL);
+		if (retval)
+			goto errout;
 
 		io_channel_flush(fs->io);
 		if (rfs->progress) {
@@ -2244,15 +2244,11 @@ static errcode_t fix_resize_inode(ext2_filsys fs)
 {
 	struct ext2_inode	inode;
 	errcode_t		retval;
-	char			*block_buf = NULL;
 
 	if (!(fs->super->s_feature_compat &
 	      EXT2_FEATURE_COMPAT_RESIZE_INODE))
 		return 0;
 
-	retval = ext2fs_get_mem(fs->blocksize, &block_buf);
-	if (retval) goto errout;
-
 	retval = ext2fs_read_inode(fs, EXT2_RESIZE_INO, &inode);
 	if (retval) goto errout;
 
@@ -2272,19 +2268,16 @@ static errcode_t fix_resize_inode(ext2_filsys fs)
 		exit(1);
 	}
 
-	memset(block_buf, 0, fs->blocksize);

[PATCH 25/39] libext2fs: support allocating uninit blocks in bmap2()

[Darrick J. Wong]

As part of supporting fallocate-like functionality, extend
ext2fs_bmap() with two flags -- BMAP_UNINIT and BMAP_ZERO.  The first
will cause it to mark/set a block uninitialized, if it's part of an
extent based file.  For a block mapped file, the mapping is put in,
but there is no way to remember the uninitialized status.  The second
flag causes the block to be zeroed to support the use case of
emulating uninitialized blocks on a block-map file by zeroing them.

Eventually fallocate or fuse2fs or somebody will use these.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/bmap.c   |    9 +++++++--
 lib/ext2fs/ext2fs.h |    2 ++
 2 files changed, 9 insertions(+), 2 deletions(-)


diff --git a/lib/ext2fs/bmap.c b/lib/ext2fs/bmap.c
index c1d0e6f..e936ec1 100644
--- a/lib/ext2fs/bmap.c
+++ b/lib/ext2fs/bmap.c
@@ -214,10 +214,13 @@ static errcode_t extent_bmap(ext2_filsys fs, ext2_ino_t ino,
 	errcode_t		retval = 0;
 	blk64_t			blk64 = 0;
 	int			alloc = 0;
+	int			set_flags;
+
+	set_flags = bmap_flags & BMAP_UNINIT ? EXT2_EXTENT_SET_BMAP_UNINIT : 0;
 
 	if (bmap_flags & BMAP_SET) {
 		retval = ext2fs_extent_set_bmap(handle, block,
-						*phys_blk, 0);
+						*phys_blk, set_flags);
 		return retval;
 	}
 	retval = ext2fs_extent_goto(handle, block);
@@ -254,7 +257,7 @@ got_block:
 		alloc++;
 	set_extent:
 		retval = ext2fs_extent_set_bmap(handle, block,
-						blk64, 0);
+						blk64, set_flags);
 		if (retval) {
 			ext2fs_block_alloc_stats2(fs, blk64, -1);
 			return retval;
@@ -440,6 +443,8 @@ errcode_t ext2fs_bmap2(ext2_filsys fs, ext2_ino_t ino, struct ext2_inode *inode,
 	if (retval == 0)
 		*phys_blk = blk32;
 done:
+	if (*phys_blk && retval == 0 && (bmap_flags & BMAP_ZERO))
+		retval = ext2fs_zero_blocks2(fs, *phys_blk, 1, NULL, NULL);
 	if (buf)
 		ext2fs_free_mem(&buf);
 	if (handle)
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 5ecaa85..664c3c0 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -533,6 +533,8 @@ typedef struct ext2_icount *ext2_icount_t;
  */
 #define BMAP_ALLOC	0x0001
 #define BMAP_SET	0x0002
+#define BMAP_UNINIT	0x0004
+#define BMAP_ZERO	0x0008
 
 /*
  * Returned flags from ext2fs_bmap

[PATCH 26/39] libext2fs: file IO routines should handle uninit blocks

[Darrick J. Wong]

The file IO routines do not handle uninit blocks at all.  The read
method should check for the uninit flag and return a buffer of zeroes,
and the write routine should convert unwritten extents.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/fileio.c         |   24 ++++++++++++++++++++++--
 tests/f_uninit_cat/expect   |  Bin
 tests/f_uninit_cat/image.gz |  Bin
 tests/f_uninit_cat/name     |    1 +
 tests/f_uninit_cat/script   |   37 +++++++++++++++++++++++++++++++++++++
 5 files changed, 60 insertions(+), 2 deletions(-)
 create mode 100644 tests/f_uninit_cat/expect
 create mode 100644 tests/f_uninit_cat/image.gz
 create mode 100644 tests/f_uninit_cat/name
 create mode 100755 tests/f_uninit_cat/script


diff --git a/lib/ext2fs/fileio.c b/lib/ext2fs/fileio.c
index 1d5032a..d0a05d6 100644
--- a/lib/ext2fs/fileio.c
+++ b/lib/ext2fs/fileio.c
@@ -123,6 +123,8 @@ errcode_t ext2fs_file_flush(ext2_file_t file)
 {
 	errcode_t	retval;
 	ext2_filsys fs;
+	int		ret_flags;
+	blk64_t		dontcare;
 
 	EXT2_CHECK_MAGIC(file, EXT2_ET_MAGIC_EXT2_FILE);
 	fs = file->fs;
@@ -131,6 +133,22 @@ errcode_t ext2fs_file_flush(ext2_file_t file)
 	    !(file->flags & EXT2_FILE_BUF_DIRTY))
 		return 0;
 
+	/* Is this an uninit block? */
+	if (file->physblock && file->inode.i_flags & EXT4_EXTENTS_FL) {
+		retval = ext2fs_bmap2(fs, file->ino, &file->inode, BMAP_BUFFER,
+				      0, file->blockno, &ret_flags, &dontcare);
+		if (retval)
+			return retval;
+		if (ret_flags & BMAP_RET_UNINIT) {
+			retval = ext2fs_bmap2(fs, file->ino, &file->inode,
+					      BMAP_BUFFER, BMAP_SET,
+					      file->blockno, 0,
+					      &file->physblock);
+			if (retval)
+				return retval;
+		}
+	}
+
 	/*
 	 * OK, the physical block hasn't been allocated yet.
 	 * Allocate it.
@@ -185,15 +203,17 @@ static errcode_t load_buffer(ext2_file_t file, int dontfill)
 {
 	ext2_filsys	fs = file->fs;
 	errcode_t	retval;
+	int		ret_flags;
 
 	if (!(file->flags & EXT2_FILE_BUF_VALID)) {
 		retval = ext2fs_bmap2(fs, file->ino, &file->inode,
-				     BMAP_BUFFER, 0, file->blockno, 0,
+				     BMAP_BUFFER, 0, file->blockno, &ret_flags,
 				     &file->physblock);
 		if (retval)
 			return retval;
 		if (!dontfill) {
-			if (file->physblock) {
+			if (file->physblock &&
+			    !(ret_flags & BMAP_RET_UNINIT)) {
 				retval = io_channel_read_blk64(fs->io,
 							       file->physblock,
 							       1, file->buf);
diff --git a/tests/f_uninit_cat/expect b/tests/f_uninit_cat/expect
new file mode 100644
index 0000000000000000000000000000000000000000..0c0a5cf84cc58483ca33e257259607e4ca54ebbe
GIT binary patch
literal 3623
zcmeH{v2KGf5Qa1J6t`4rrKVs4QNvoLQ-&;^E5{tf3b8An9fdx94LB%6-ypu_bol%I
zbN}EyCjwT%#}UOzsurZuHPR~_IxSAVb5#S$U!-I|p!pqIOM}8{(*s%KgmnfdX!S27
zv{Igz7is&6EABXh4H{GeL1?FJuq*F~)@b(w<j!aAEv0I-IddzuN-UE7Ze)klQw1zf
z*9D9tJZEp&6DX~g-rdU9X-6-wP?Ral@**smY_HP#9k_J_k|0ZJJh-+Y5Zr=OQu*WI
zzT5W-@CqqUc6h-Kw#pib1XJyFD+TYSVSnstoOY;MdxX!9x0FDZLgoRM0<1<bgJtNx
zmG9H!`&bOV#$T9q`K)6>#|E(61l{JQn-!~Bkq1RSFzev!`&hG6*uNSB@QW^D#ROsk
a51YU>R!kr!@URIy?86rqj|s#C7RevINPdd|

literal 0
HcmV?d00001

diff --git a/tests/f_uninit_cat/image.gz b/tests/f_uninit_cat/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..d2ae66cb35f93dd4022e85f8667365487a7b24ea
GIT binary patch
literal 1553
zcmb2|=3wBo77AfvetUOgws4>X!-w+e5!ulJUCtMqn*6x7X0!`+M-&JY2D}M!S)+8+
zDVJ6IMapY?1&f!@9@sy4qUHCltf(-cFi@sM<Z<;*mRk{C3zur%doRAP`s7{J>E%1$
zf6YFjs>qO9YUaC2aZcCS-(DQb8_$NVE7;Mo<=)q*|3~ghM6S5L{?7@i*S^PYuMLg<
z{yO)bMDN+#HZAQNWmni(e$ISz=k4?0&B2S~=hf!a%c;LRqnoqOeE0R(Td}i${(5!v
zN9*zV=lkXUet4HAR(*B4`0223$*Xyed2c)RXUFctq3v?k4*osNhJB`=>Q$cfF5YY)
zq&Ig<I|BqHJYRiXzO>c;>t5Mgf0ATh%=q$g`@RWJoj2;Pd;ar6z(>h_e;2CHpSyp1
zK~2@ae|Oey(P??=HSMlJ&Y}8++jalfhj#s8_ga)C-tdS0iM_&q#-IEr>K*<uf0Cc@
zA1uVcut9k1xj*G<CR6-R{q_2++YoEG%ju)G)m_7H5^{`CS3sx*%4*qvh1Y5X@893w
ze(dD=mG_O$YD^1}+TT9sT+6vfTb`$HdzF3Z(w=*|H~uB`<f~6#9=s)v-CcQutl0EN
zDLbP6mj5|_XrHg$w}ZTW^>6Fj_h@afy|?67dd1pb`uxmif2pqt{Qt!?S^L#|zi0nX
zFI}X(^4;uQx9mT!qeB0OhF!1kR=hH+G@<KquWi^F`Rv=r-e*s4zxMk7(bMbaFY_=>
zsFGV7C4lT;jqS<~nYIjGr#U#DKDAdsmKtS`hQMeD44Duxd!&D1J4-kNg8%~n1^y`^

literal 0
HcmV?d00001

diff --git a/tests/f_uninit_cat/name b/tests/f_uninit_cat/name
new file mode 100644
index 0000000..f6b5674
--- /dev/null
+++ b/tests/f_uninit_cat/name
@@ -0,0 +1 @@
+cat a file with uninit blocks
diff --git a/tests/f_uninit_cat/script b/tests/f_uninit_cat/script
new file mode 100755
index 0000000..11dbb6e
--- /dev/null
+++ b/tests/f_uninit_cat/script
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+if test -x $DEBUGFS_EXE; then
+FSCK_OPT=-fy
+IMAGE=$test_dir/image.gz
+
+gzip -d < $IMAGE > $TMPFILE
+#e2label $TMPFILE test_filesys
+
+# Run fsck to fix things?
+EXP=$test_dir/expect
+OUT=$test_name.log
+rm -rf $test_name.failed $test_name.ok
+
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed > $OUT
+echo "Exit status is $?" >> $OUT
+
+echo "debugfs cat uninit file" >> $OUT
+echo "ex /a" > $TMPFILE.cmd
+echo "cat /a" >> $TMPFILE.cmd
+$DEBUGFS_EXE -w $TMPFILE -f $TMPFILE.cmd >> $OUT.new 2>&1
+echo >> $OUT.new
+sed -f $cmd_dir/filter.sed < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE
+
+# Figure out what happened
+if cmp -s $EXP $OUT; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff -u $EXP $OUT >> $test_name.failed
+fi
+unset EXP OUT FSCK_OPT IMAGE
+else #if test -a -x $DEBUGFS_EXE; then
+        echo "$test_name: $test_description: skipped"
+fi 

[PATCH 27/39] resize2fs: convert fs to and from 64bit mode

[Darrick J. Wong]

resize2fs does its magic by loading a filesystem, duplicating the
in-memory image of that fs, moving relevant blocks out of the way of
whatever new metadata get created, and finally writing everything back
out to disk.  Enabling 64bit mode enlarges the group descriptors,
which makes resize2fs a reasonable vehicle for taking care of the rest
of the bookkeeping requirements, so add to resize2fs the ability to
convert a filesystem to 64bit mode and back.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Cc: TR Reardon <thomas_reardon@hotmail.com>
---
 resize/main.c         |   39 +++++
 resize/resize2fs.8.in |   18 ++
 resize/resize2fs.c    |  363 +++++++++++++++++++++++++++++++++++++++++++++++--
 resize/resize2fs.h    |    3 
 4 files changed, 402 insertions(+), 21 deletions(-)


diff --git a/resize/main.c b/resize/main.c
index e017792..425ee63 100644
--- a/resize/main.c
+++ b/resize/main.c
@@ -42,7 +42,7 @@ static char *device_name, *io_options;
 static void usage (char *prog)
 {
 	fprintf (stderr, _("Usage: %s [-d debug_flags] [-f] [-F] [-M] [-P] "
-			   "[-p] device [new_size]\n\n"), prog);
+			   "[-p] device [-b|-s|new_size]\n\n"), prog);
 
 	exit (1);
 }
@@ -203,7 +203,7 @@ int main (int argc, char ** argv)
 	if (argc && *argv)
 		program_name = *argv;
 
-	while ((c = getopt (argc, argv, "d:fFhMPpS:")) != EOF) {
+	while ((c = getopt(argc, argv, "d:fFhMPpS:bs")) != EOF) {
 		switch (c) {
 		case 'h':
 			usage(program_name);
@@ -229,6 +229,12 @@ int main (int argc, char ** argv)
 		case 'S':
 			use_stride = atoi(optarg);
 			break;
+		case 'b':
+			flags |= RESIZE_ENABLE_64BIT;
+			break;
+		case 's':
+			flags |= RESIZE_DISABLE_64BIT;
+			break;
 		default:
 			usage(program_name);
 		}
@@ -393,6 +399,10 @@ int main (int argc, char ** argv)
 		if (sys_page_size > blocksize)
 			new_size &= ~((sys_page_size / blocksize)-1);
 	}
+	/* If changing 64bit, don't change the filesystem size. */
+	if (flags & (RESIZE_DISABLE_64BIT | RESIZE_ENABLE_64BIT)) {
+		new_size = ext2fs_blocks_count(fs->super);
+	}
 	if (!EXT2_HAS_INCOMPAT_FEATURE(fs->super,
 				       EXT4_FEATURE_INCOMPAT_64BIT)) {
 		/* Take 16T down to 2^32-1 blocks */
@@ -444,7 +454,30 @@ int main (int argc, char ** argv)
 			blocksize / 1024, new_size);
 		exit(1);
 	}
-	if (new_size == ext2fs_blocks_count(fs->super)) {
+	if ((flags & RESIZE_DISABLE_64BIT) && (flags & RESIZE_ENABLE_64BIT)) {
+		fprintf(stderr, _("Cannot set and unset 64bit feature.\n"));
+		exit(1);
+	} else if (flags & (RESIZE_DISABLE_64BIT | RESIZE_ENABLE_64BIT)) {
+		if (new_size >= (1ULL << 32)) {
+			fprintf(stderr, _("Cannot change the 64bit feature "
+				"on a filesystem that is larger than "
+				"2^32 blocks.\n"));
+			exit(1);
+		}
+		if (mount_flags & EXT2_MF_MOUNTED) {
+			fprintf(stderr, _("Cannot change the 64bit feature "
+				"while the filesystem is mounted.\n"));
+			exit(1);
+		}
+		if (flags & RESIZE_ENABLE_64BIT &&
+		    !EXT2_HAS_INCOMPAT_FEATURE(fs->super,
+				EXT3_FEATURE_INCOMPAT_EXTENTS)) {
+			fprintf(stderr, _("Please enable the extents feature "
+				"with tune2fs before enabling the 64bit "
+				"feature.\n"));
+			exit(1);
+		}
+	} else if (new_size == ext2fs_blocks_count(fs->super)) {
 		fprintf(stderr, _("The filesystem is already %llu (%dk) "
 			"blocks long.  Nothing to do!\n\n"), new_size,
 			blocksize / 1024);
diff --git a/resize/resize2fs.8.in b/resize/resize2fs.8.in
index 86495c6..0129bfc 100644
--- a/resize/resize2fs.8.in
+++ b/resize/resize2fs.8.in
@@ -8,7 +8,7 @@ resize2fs \- ext2/ext3/ext4 file system resizer
 .SH SYNOPSIS
 .B resize2fs
 [
-.B \-fFpPM
+.B \-fFpPMbs
 ]
 [
 .B \-d
@@ -86,8 +86,21 @@ to shrink the size of filesystem.  Then you may use
 to shrink the size of the partition.  When shrinking the size of
 the partition, make sure you do not make it smaller than the new size
 of the ext2 filesystem!
+.PP
+The
+.B \-b
+and
+.B \-s
+options enable and disable the 64bit feature, respectively.  The resize2fs
+program will, of course, take care of resizing the block group descriptors
+and moving other data blocks out of the way, as needed.  It is not possible
+to resize the filesystem concurrent with changing the 64bit status.
 .SH OPTIONS
 .TP
+.B \-b
+Turns on the 64bit feature, resizes the group descriptors as necessary, and
+moves other metadata out of the way.
+.TP
 .B \-d \fIdebug-flags
 Turns on various resize2fs debugging features, if they have been compiled
 into the binary.
@@ -127,6 +140,9 @@ of what the program is doing.
 .B \-P
 Print the minimum size of the filesystem and exit.
 .TP
+.B \-s
+Turns off the 64bit feature and frees blocks that are no longer in use.
+.TP
 .B \-S \fIRAID-stride
 The
 .B resize2fs
diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 93df070..a33aaf3 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -57,6 +57,9 @@ static errcode_t mark_table_blocks(ext2_filsys fs,
 static errcode_t clear_sparse_super2_last_group(ext2_resize_t rfs);
 static errcode_t reserve_sparse_super2_last_group(ext2_resize_t rfs,
 						 ext2fs_block_bitmap meta_bmap);
+static errcode_t resize_group_descriptors(ext2_resize_t rfs, blk64_t new_size);
+static errcode_t move_bg_metadata(ext2_resize_t rfs);
+static errcode_t zero_high_bits_in_inodes(ext2_resize_t rfs);
 
 /*
  * Some helper CPP macros
@@ -123,13 +126,30 @@ errcode_t resize_fs(ext2_filsys fs, blk64_t *new_size, int flags,
 	if (retval)
 		goto errout;
 
+	init_resource_track(&rtrack, "resize_group_descriptors", fs->io);
+	retval = resize_group_descriptors(rfs, *new_size);
+	if (retval)
+		goto errout;
+	print_resource_track(rfs, &rtrack, fs->io);
+
+	init_resource_track(&rtrack, "move_bg_metadata", fs->io);
+	retval = move_bg_metadata(rfs);
+	if (retval)
+		goto errout;
+	print_resource_track(rfs, &rtrack, fs->io);
+
+	init_resource_track(&rtrack, "zero_high_bits_in_metadata", fs->io);
+	retval = zero_high_bits_in_inodes(rfs);
+	if (retval)
+		goto errout;
+	print_resource_track(rfs, &rtrack, fs->io);
+
 	init_resource_track(&rtrack, "adjust_superblock", fs->io);
 	retval = adjust_superblock(rfs, *new_size);
 	if (retval)
 		goto errout;
 	print_resource_track(rfs, &rtrack, fs->io);
 
-
 	init_resource_track(&rtrack, "fix_uninit_block_bitmaps 2", fs->io);
 	fix_uninit_block_bitmaps(rfs->new_fs);
 	print_resource_track(rfs, &rtrack, fs->io);
@@ -235,6 +255,321 @@ errout:
 	return retval;
 }
 
+/* Keep the size of the group descriptor region constant */
+static void adjust_reserved_gdt_blocks(ext2_filsys old_fs, ext2_filsys fs)
+{
+	if ((fs->super->s_feature_compat &
+	     EXT2_FEATURE_COMPAT_RESIZE_INODE) &&
+	    (old_fs->desc_blocks != fs->desc_blocks)) {
+		int new;
+
+		new = ((int) fs->super->s_reserved_gdt_blocks) +
+			(old_fs->desc_blocks - fs->desc_blocks);
+		if (new < 0)
+			new = 0;
+		if (new > (int) fs->blocksize/4)
+			new = fs->blocksize/4;
+		fs->super->s_reserved_gdt_blocks = new;
+	}
+}
+
+/* Toggle 64bit mode */
+static errcode_t resize_group_descriptors(ext2_resize_t rfs, blk64_t new_size)
+{
+	void *o, *n, *new_group_desc;
+	dgrp_t i;
+	int copy_size;
+	errcode_t retval;
+
+	if (!(rfs->flags & (RESIZE_DISABLE_64BIT | RESIZE_ENABLE_64BIT)))
+		return 0;
+
+	if (new_size != ext2fs_blocks_count(rfs->new_fs->super) ||
+	    ext2fs_blocks_count(rfs->new_fs->super) >= (1ULL << 32) ||
+	    (rfs->flags & RESIZE_DISABLE_64BIT &&
+	     rfs->flags & RESIZE_ENABLE_64BIT))
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	if (rfs->flags & RESIZE_DISABLE_64BIT) {
+		rfs->new_fs->super->s_feature_incompat &=
+				~EXT4_FEATURE_INCOMPAT_64BIT;
+		rfs->new_fs->super->s_desc_size = EXT2_MIN_DESC_SIZE;
+	} else if (rfs->flags & RESIZE_ENABLE_64BIT) {
+		rfs->new_fs->super->s_feature_incompat |=
+				EXT4_FEATURE_INCOMPAT_64BIT;
+		rfs->new_fs->super->s_desc_size = EXT2_MIN_DESC_SIZE_64BIT;
+	}
+
+	if (EXT2_DESC_SIZE(rfs->old_fs->super) ==
+	    EXT2_DESC_SIZE(rfs->new_fs->super))
+		return 0;
+
+	o = rfs->new_fs->group_desc;
+	rfs->new_fs->desc_blocks = ext2fs_div_ceil(
+			rfs->old_fs->group_desc_count,
+			EXT2_DESC_PER_BLOCK(rfs->new_fs->super));
+	retval = ext2fs_get_arrayzero(rfs->new_fs->desc_blocks,
+				      rfs->old_fs->blocksize, &new_group_desc);
+	if (retval)
+		return retval;
+
+	n = new_group_desc;
+
+	if (EXT2_DESC_SIZE(rfs->old_fs->super) <=
+	    EXT2_DESC_SIZE(rfs->new_fs->super))
+		copy_size = EXT2_DESC_SIZE(rfs->old_fs->super);
+	else
+		copy_size = EXT2_DESC_SIZE(rfs->new_fs->super);
+	for (i = 0; i < rfs->old_fs->group_desc_count; i++) {
+		memcpy(n, o, copy_size);
+		n += EXT2_DESC_SIZE(rfs->new_fs->super);
+		o += EXT2_DESC_SIZE(rfs->old_fs->super);
+	}
+
+	ext2fs_free_mem(&rfs->new_fs->group_desc);
+	rfs->new_fs->group_desc = new_group_desc;
+
+	for (i = 0; i < rfs->old_fs->group_desc_count; i++)
+		ext2fs_group_desc_csum_set(rfs->new_fs, i);
+
+	adjust_reserved_gdt_blocks(rfs->old_fs, rfs->new_fs);
+
+	return 0;
+}
+
+/* Move bitmaps/inode tables out of the way. */
+static errcode_t move_bg_metadata(ext2_resize_t rfs)
+{
+	dgrp_t i;
+	blk64_t b, c, d, old_desc_blocks, new_desc_blocks, j;
+	ext2fs_block_bitmap old_map, new_map;
+	int old, new;
+	errcode_t retval;
+	int cluster_ratio;
+
+	if (!(rfs->flags & (RESIZE_DISABLE_64BIT | RESIZE_ENABLE_64BIT)))
+		return 0;
+
+	retval = ext2fs_allocate_block_bitmap(rfs->old_fs, "oldfs", &old_map);
+	if (retval)
+		return retval;
+
+	retval = ext2fs_allocate_block_bitmap(rfs->new_fs, "newfs", &new_map);
+	if (retval)
+		goto out;
+
+	if (EXT2_HAS_INCOMPAT_FEATURE(rfs->old_fs->super,
+				      EXT2_FEATURE_INCOMPAT_META_BG)) {
+		old_desc_blocks = rfs->old_fs->super->s_first_meta_bg;
+		new_desc_blocks = rfs->new_fs->super->s_first_meta_bg;
+	} else {
+		old_desc_blocks = rfs->old_fs->desc_blocks +
+				rfs->old_fs->super->s_reserved_gdt_blocks;
+		new_desc_blocks = rfs->new_fs->desc_blocks +
+				rfs->new_fs->super->s_reserved_gdt_blocks;
+	}
+
+	/* Construct bitmaps of super/descriptor blocks in old and new fs */
+	for (i = 0; i < rfs->old_fs->group_desc_count; i++) {
+		retval = ext2fs_super_and_bgd_loc2(rfs->old_fs, i, &b, &c, &d,
+						   NULL);
+		if (retval)
+			goto out;
+		if (b)
+			ext2fs_mark_block_bitmap2(old_map, b);
+		for (j = 0; c != 0 && j < old_desc_blocks; j++)
+			ext2fs_mark_block_bitmap2(old_map, c + j);
+		if (d)
+			ext2fs_mark_block_bitmap2(old_map, d);
+
+		retval = ext2fs_super_and_bgd_loc2(rfs->new_fs, i, &b, &c, &d,
+						   NULL);
+		if (retval)
+			goto out;
+		if (b)
+			ext2fs_mark_block_bitmap2(new_map, b);
+		for (j = 0; c != 0 && j < new_desc_blocks; j++)
+			ext2fs_mark_block_bitmap2(new_map, c + j);
+		if (d)
+			ext2fs_mark_block_bitmap2(new_map, d);
+	}
+
+	cluster_ratio = EXT2FS_CLUSTER_RATIO(rfs->new_fs);
+
+	/* Find changes in block allocations for bg metadata */
+	for (b = EXT2FS_B2C(rfs->old_fs,
+			    rfs->old_fs->super->s_first_data_block);
+	     b < ext2fs_blocks_count(rfs->new_fs->super);
+	     b += cluster_ratio) {
+		old = ext2fs_test_block_bitmap2(old_map, b);
+		new = ext2fs_test_block_bitmap2(new_map, b);
+
+		if (old && !new) {
+			/* mark old_map, unmark new_map */
+			if (cluster_ratio == 1)
+				ext2fs_unmark_block_bitmap2(
+						rfs->new_fs->block_map, b);
+		} else if (!old && new)
+			; /* unmark old_map, mark new_map */
+		else {
+			ext2fs_unmark_block_bitmap2(old_map, b);
+			ext2fs_unmark_block_bitmap2(new_map, b);
+		}
+	}
+
+	/*
+	 * new_map now shows blocks that have been newly allocated.
+	 * old_map now shows blocks that have been newly freed.
+	 */
+
+	/*
+	 * Move any conflicting bitmaps and inode tables.  Ensure that we
+	 * don't try to free clusters associated with bitmaps or tables.
+	 */
+	for (i = 0; i < rfs->old_fs->group_desc_count; i++) {
+		b = ext2fs_block_bitmap_loc(rfs->new_fs, i);
+		if (ext2fs_test_block_bitmap2(new_map, b))
+			ext2fs_block_bitmap_loc_set(rfs->new_fs, i, 0);
+		else if (ext2fs_test_block_bitmap2(old_map, b))
+			ext2fs_unmark_block_bitmap2(old_map, b);
+
+		b = ext2fs_inode_bitmap_loc(rfs->new_fs, i);
+		if (ext2fs_test_block_bitmap2(new_map, b))
+			ext2fs_inode_bitmap_loc_set(rfs->new_fs, i, 0);
+		else if (ext2fs_test_block_bitmap2(old_map, b))
+			ext2fs_unmark_block_bitmap2(old_map, b);
+
+		c = ext2fs_inode_table_loc(rfs->new_fs, i);
+		for (b = 0;
+		     b < rfs->new_fs->inode_blocks_per_group;
+		     b++) {
+			if (ext2fs_test_block_bitmap2(new_map, b + c))
+				ext2fs_inode_table_loc_set(rfs->new_fs, i, 0);
+			else if (ext2fs_test_block_bitmap2(old_map, b + c))
+				ext2fs_unmark_block_bitmap2(old_map, b + c);
+		}
+	}
+
+	/* Free unused clusters */
+	for (b = 0;
+	     cluster_ratio > 1 && b < ext2fs_blocks_count(rfs->new_fs->super);
+	     b += cluster_ratio)
+		if (ext2fs_test_block_bitmap2(old_map, b))
+			ext2fs_unmark_block_bitmap2(rfs->new_fs->block_map, b);
+out:
+	if (old_map)
+		ext2fs_free_block_bitmap(old_map);
+	if (new_map)
+		ext2fs_free_block_bitmap(new_map);
+	return retval;
+}
+
+/* Zero out the high bits of extent fields */
+static errcode_t zero_high_bits_in_extents(ext2_filsys fs, ext2_ino_t ino,
+				 struct ext2_inode *inode)
+{
+	ext2_extent_handle_t	handle;
+	struct ext2fs_extent	extent;
+	int			op = EXT2_EXTENT_ROOT;
+	errcode_t		errcode;
+
+	if (!(inode->i_flags & EXT4_EXTENTS_FL))
+		return 0;
+
+	errcode = ext2fs_extent_open(fs, ino, &handle);
+	if (errcode)
+		return errcode;
+
+	while (1) {
+		errcode = ext2fs_extent_get(handle, op, &extent);
+		if (errcode)
+			break;
+
+		op = EXT2_EXTENT_NEXT_SIB;
+
+		if (extent.e_pblk > (1ULL << 32)) {
+			extent.e_pblk &= (1ULL << 32) - 1;
+			errcode = ext2fs_extent_replace(handle, 0, &extent);
+			if (errcode)
+				break;
+		}
+	}
+
+	/* Ok if we run off the end */
+	if (errcode == EXT2_ET_EXTENT_NO_NEXT)
+		errcode = 0;
+	return errcode;
+}
+
+/* Zero out the high bits of inodes. */
+static errcode_t zero_high_bits_in_inodes(ext2_resize_t rfs)
+{
+	ext2_filsys	fs = rfs->new_fs;
+	int length = EXT2_INODE_SIZE(fs->super);
+	struct ext2_inode *inode = NULL;
+	ext2_inode_scan	scan = NULL;
+	errcode_t	retval;
+	ext2_ino_t	ino;
+
+	if (!(rfs->flags & (RESIZE_DISABLE_64BIT | RESIZE_ENABLE_64BIT)))
+		return 0;
+
+	if (fs->super->s_creator_os != EXT2_OS_LINUX)
+		return 0;
+
+	retval = ext2fs_open_inode_scan(fs, 0, &scan);
+	if (retval)
+		return retval;
+
+	retval = ext2fs_get_mem(length, &inode);
+	if (retval)
+		goto out;
+
+	do {
+		retval = ext2fs_get_next_inode_full(scan, &ino, inode, length);
+		if (retval)
+			goto out;
+		if (!ino)
+			break;
+		if (!ext2fs_test_inode_bitmap2(fs->inode_map, ino))
+			continue;
+
+		/*
+		 * Here's how we deal with high block number fields:
+		 *
+		 *  - i_size_high has been been written out with i_size_lo
+		 *    since the ext2 days, so no conversion is needed.
+		 *
+		 *  - i_blocks_hi is guarded by both the huge_file feature and
+		 *    inode flags and has always been written out with
+		 *    i_blocks_lo if the feature is set.  The field is only
+		 *    ever read if both feature and inode flag are set, so
+		 *    we don't need to zero it now.
+		 *
+		 *  - i_file_acl_high can be uninitialized, so zero it if
+		 *    it isn't already.
+		 */
+		if (inode->osd2.linux2.l_i_file_acl_high) {
+			inode->osd2.linux2.l_i_file_acl_high = 0;
+			retval = ext2fs_write_inode_full(fs, ino, inode,
+							 length);
+			if (retval)
+				goto out;
+		}
+
+		retval = zero_high_bits_in_extents(fs, ino, inode);
+		if (retval)
+			goto out;
+	} while (ino);
+
+out:
+	if (inode)
+		ext2fs_free_mem(&inode);
+	if (scan)
+		ext2fs_close_inode_scan(scan);
+	return retval;
+}
+
 /*
  * Clean up the bitmaps for unitialized bitmaps
  */
@@ -458,7 +793,8 @@ retry:
 	/*
 	 * Reallocate the group descriptors as necessary.
 	 */
-	if (old_fs->desc_blocks != fs->desc_blocks) {
+	if (EXT2_DESC_SIZE(old_fs->super) == EXT2_DESC_SIZE(fs->super) &&
+	    old_fs->desc_blocks != fs->desc_blocks) {
 		retval = ext2fs_resize_mem(old_fs->desc_blocks *
 					   fs->blocksize,
 					   fs->desc_blocks * fs->blocksize,
@@ -477,20 +813,11 @@ retry:
 	 * number of descriptor blocks, then adjust
 	 * s_reserved_gdt_blocks if possible to avoid needing to move
 	 * the inode table either now or in the future.
+	 *
+	 * Note: If we're converting to 64bit mode, we did this earlier.
 	 */
-	if ((fs->super->s_feature_compat &
-	     EXT2_FEATURE_COMPAT_RESIZE_INODE) &&
-	    (old_fs->desc_blocks != fs->desc_blocks)) {
-		int new;
-
-		new = ((int) fs->super->s_reserved_gdt_blocks) +
-			(old_fs->desc_blocks - fs->desc_blocks);
-		if (new < 0)
-			new = 0;
-		if (new > (int) fs->blocksize/4)
-			new = fs->blocksize/4;
-		fs->super->s_reserved_gdt_blocks = new;
-	}
+	if (EXT2_DESC_SIZE(old_fs->super) == EXT2_DESC_SIZE(fs->super))
+		adjust_reserved_gdt_blocks(old_fs, fs);
 
 	if ((fs->super->s_feature_incompat & EXT2_FEATURE_INCOMPAT_META_BG) &&
 	    (fs->super->s_first_meta_bg > fs->desc_blocks)) {
@@ -1018,7 +1345,9 @@ static errcode_t blocks_to_move(ext2_resize_t rfs)
 	if (retval)
 		goto errout;
 
-	if (old_blocks == new_blocks) {
+	if (EXT2_DESC_SIZE(rfs->old_fs->super) ==
+	    EXT2_DESC_SIZE(rfs->new_fs->super) &&
+	    old_blocks == new_blocks) {
 		retval = 0;
 		goto errout;
 	}
@@ -1538,7 +1867,7 @@ static errcode_t progress_callback(ext2_filsys fs,
 static errcode_t migrate_ea_block(ext2_resize_t rfs, ext2_ino_t ino,
 				  struct ext2_inode *inode, int *changed)
 {
-	char *buf;
+	char *buf = NULL;
 	blk64_t new_block;
 	errcode_t err = 0;
 
diff --git a/resize/resize2fs.h b/resize/resize2fs.h
index 7aeab91..829fcd8 100644
--- a/resize/resize2fs.h
+++ b/resize/resize2fs.h
@@ -82,6 +82,9 @@ typedef struct ext2_sim_progress *ext2_sim_progmeter;
 #define RESIZE_PERCENT_COMPLETE		0x0100
 #define RESIZE_VERBOSE			0x0200
 
+#define RESIZE_ENABLE_64BIT		0x0400
+#define RESIZE_DISABLE_64BIT		0x0800
+
 /*
  * This structure is used for keeping track of how much resources have
  * been used for a particular resize2fs pass.

[PATCH 28/39] tests: test resize2fs 32->64 and 64->32bit conversion code

[Darrick J. Wong]

Add some simple tests to check that flex_bg and meta_bg filesystems
can be converted between 32 and 64bit layouts.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/r_32to64bit/expect.gz      |  Bin
 tests/r_32to64bit/name           |    1 +
 tests/r_32to64bit/script         |   75 ++++++++++++++++++++++++++++++++++++++
 tests/r_32to64bit_meta/expect.gz |  Bin
 tests/r_32to64bit_meta/name      |    1 +
 tests/r_32to64bit_meta/script    |   75 ++++++++++++++++++++++++++++++++++++++
 tests/r_64to32bit/expect.gz      |  Bin
 tests/r_64to32bit/name           |    1 +
 tests/r_64to32bit/script         |   75 ++++++++++++++++++++++++++++++++++++++
 tests/r_64to32bit_meta/expect.gz |  Bin
 tests/r_64to32bit_meta/name      |    1 +
 tests/r_64to32bit_meta/script    |   75 ++++++++++++++++++++++++++++++++++++++
 12 files changed, 304 insertions(+)
 create mode 100644 tests/r_32to64bit/expect.gz
 create mode 100644 tests/r_32to64bit/name
 create mode 100644 tests/r_32to64bit/script
 create mode 100644 tests/r_32to64bit_meta/expect.gz
 create mode 100644 tests/r_32to64bit_meta/name
 create mode 100644 tests/r_32to64bit_meta/script
 create mode 100644 tests/r_64to32bit/expect.gz
 create mode 100644 tests/r_64to32bit/name
 create mode 100644 tests/r_64to32bit/script
 create mode 100644 tests/r_64to32bit_meta/expect.gz
 create mode 100644 tests/r_64to32bit_meta/name
 create mode 100644 tests/r_64to32bit_meta/script


diff --git a/tests/r_32to64bit/expect.gz b/tests/r_32to64bit/expect.gz
new file mode 100644
index 0000000000000000000000000000000000000000..4ba35c8ffb7ca9ef3159f1c40dd507c5fdcff180
GIT binary patch
literal 3144
zcmZ`*X*3j!8dm!7W$ELyg<=L-hK3p<Qwo!<Fxhv5NJhvyq#<RPu`{E{lI&a4Fv3_8
zAsR&X-C!(P$B=BdzI%V&bML?Byz6t`^PK0r;y4bL$5V=CCs|iB?s$vU@<zS=e1K(p
zl;lLdN&2y5yCca-pSG9cdBkxNp+L@y$vgWxw90e!t+7C80cA1K1*kVm%?(Hb3_`>O
zSMCXfc7|O+K3V%l?fp(cH2thwAKa(zRM(C`{ad<->*q7FRre`vvgP&Ob771bL{NYx
zWMwtLCGNx+vYN8;(CTn+uXu&##6W}v93Ma+El-AmhNurifAnU!F>S%Db`S$m`b81N
zr+A?_-yAB8_Mtf3MJqG{ZEj%T-JzE|@qoF{{<uRgXly+zJN@t{t5&1OY>)RMJ>>mr
z`rn$RO*<Js!aEMup-zu_H3NGN0}p#&>2o@W3kDw*yq5W5F<QH@wpWcif{J$Qx*NQX
z&%R&B9GI9f)vZBn%`LiB3$2uvPB3<e&ob<fqx-*CXID6pf3_SXiRz%%3hzypW4a_m
zcYV>dlk58h32-0vkok~)T2loj!FTqs%TzaNB5-OP>Mst~l$h?X6R+#)7=f1UkF6!&
zrWv%OuX$e?qh=wOJjO88ovibHZ^2g}{maG^y-ZDK<l|lHzMR@@!lgiFJL{Cp;t#*-
zPxF~?8*%&KNr<~++{0wWHvCmb73!J81>cD#B^lL~u*ez66E;a^$rKdf#IqeVaPr9o
zZSBZ9^9$1(mM$jVI|(a=KLDWgN=vm{jE1=DE|J}RIhPquT$O5qW}a1VHYE~D6B+V)
zqF%9uj0JD+sjy?}`!qCacV}X37cGO8EioQcDUqW<#QNq0OdTZRM+ZG6D{-mRvuji9
zIt;AEE5}k7)owQevdX(5e1vNVZCC5OkHM)bW}a4kzl}zizPht80(JL0s|xl9J^!=v
zT=MxDrzjl8oU`VJBL$3MD2n^3EzoSVQJy}-oRBRi5)aE73qI(^G|$Fac0B|PbTfS<
z@xa$VBFp{;?u_EOq1*>(NPDD0mXH{DZsW$Ng-H9!B&UR6w>BIqZ{VhgIYk@uEoA~A
zV(LpSG-4MUEGHN&^l-G;=P)i#^D%(*lQc6>gY#8MjF`RCOHQB7-n$nD%r#uqvp&;&
z@u-(AjMI|V!Gdw8LR<YVJKWy%QV@O+v}hvW8Wi1zv?~X#|HT~&A)O>$oW`V)Q9?eM
zD7;4h^LFTo51*oV)ayR<T+I1IAz;NE$=U}8q>Bmd0Sf>OyN!tm?Q1kqw_b_%X&F=i
zhyP;vRFR^s#`e^LlT@z_DeILDP762@RKi3FZQxOvei6+Im_R$^VxrtJR_X+3gbk2i
zG6HN*P|5(aLxs|NEdH2uItzSm$VLAdF2JF~=CvVFqSI|`^o+`o?q9g8mb)hyUMQBE
z`NK*?=2E2kz_JOx{hTJk?s`IK1#5N}t)sw}m<;2bF$!0}>q3SMip$^LeNAHz?~<|U
zY57LyQ?!z{*@*unziISLq5oD@ufcQUl1_4b_KAzxo~&ogvb)F}?#eGu;Dl#0Ijl)>
z_mh{|W|mz)N-u8}D{Yu?*P2E9yhw>?2mS6Bx&<lA1DGrvLTBzL+(TIaAqKVw2BVLY
zfN~2rwdw`K)1$>?wD_y~j6`h>{fDhQ<jM&h@wOT^wNL<?D6fsjOJA-5R(V6yBwlIf
z*nvjDfepGKTw0g;9BX`4i`o7LAR?;pAihZ99;{7i@`a9~e9#Gg_}+Jt$*Z5jE(Y>F
zu?>IqAKuCn5{t{2PGvF2Ef4zz@CwwdSHh%}*|h%1ZcY7Q1tpOb{s#DZILMkSim$Pn
zy3Y55Ub^RJNF0%_#vl_@i2O}zd{U#K+Yiaalc7XK(go%6tun$6|AyW?Tskf@H9vR@
zpclL5R9<9Gpd0zX26cuK-I=LBSimOw$WRr1!%c`z;_JBe*$4UawT)p(cwKUzwM7nv
z>!|OeO!KO{*SlwsMkS<Nhc@$Y{RcrA4J*8<QgPHOpcL=I1jlwX%3_UXCV60mQY#Qt
z;dETeD`&4<;6ckAKY{hlm7F**+!OX4pneWZw9}N&=1)pCi}f&;0&!MitwY2dnLs3q
zWtb+kQM`Lyf_v0=Po^T;cRc0gpvjiOBR2xyJO0mFj2Pj?5WorJ2ng@WUj4>6Xpsk#
zX6TM&qMD5ri?PH@FEu%Fss2tmF2Ni$F-Ao5Bi}qO8mNAmRZ?B$HAffSa?dq3)!*5~
zx(Cs0B*PZH9e19f(|JRQJTlDiGh7BL4E;6NziBlS$&-!|6!b_*j>xC!#TZoc=aGrm
zKkLGVU5_4Lzh!Wj1lJFDId@&|tkw1S)FM|I7J0|{+_2*90(i)oOmKro>Ndih3vLm@
ziViiDy+6N^zUmpn6A_c3m^zts+ZU83lULK~Xli%M#Udnfp{OHTWNT0kj$8iWhb#I3
z4?B;~<wM_rpN+F#;)c@sB;{onO?E4unbdOTM0dLP5GIp&UA}zePx<-+Kl$M@Kv|0u
zc-|gg6miW`2+qCR(2)y#BV|^<yBYAVPt*z+n^*wzC@}$vw}d%i%O3wkE}aAD8%?|N
zjg)P(5wd`J+JiY^<-~aK%xt7rl1hvR-aR1s*&oeYMYH)=PIY4w5eGbl7S>T3%>vec
z+iPd3n7CDmyfuyCqW9V2wkd*OTNavqc@@{tdoXFMsGBg!o1B+V7Si|W$$#%wE2ouP
zT_sp;VH>3ZNVT%R)bP3Tf$?Y`Px7>zz&p3%-)XK2qK)kZUN2q(hTKV)&y(_9`pb+&
zss#=zc)|?$(*$fRaUgXB)}u?BLGLHT2eEsm8ubs_6+Jjwm4N5gmA$XCvA3g<Uet(5
zG29VJ$#eH&r;|Q$ntQ30720YzK%iz_AaxgPao6Ua`Sk#R^b)Sbm8a{pIrWico=zA_
z(u~|#(|J!BrG9=i(b&NSk1S7wRt3K<L2UsI_^nu^J^vsC-xRD&GhFC#2j@veSmRHr
zfz4EU9CVgOhwaQHiC-Jdiuz=46r1G3`*g#FbIVKyBG55+!i;EX-*b;Vkv~l~Ly^V#
z(VnSW->!_P8BGPbBuyI_YRa<oL~6mr>~m`IU6iM%Sp2^VeR*NtdMnTIyUU@D<=31C
zL2|#Hgj<)1GCPgt2swRx|BJkH%Uy))GYeXb1M}l^3I2au$p}{M(p9a@r<x*_*kR&r
z;j?#2X}D+`v(H(jxT5%dhdH=_IhR#)sZ@a#3~2@YMhoRA|H@6iYCg_h(9v1gW6{c+
z>|dH`49qZZjhb)fh&kQH@iv5=&pP7Tu%LCy3k#G*{WT!yyfU;T7={e);<}kjB0jlH
z?z}txXGLrJ?d<YsNhvwbhKyF#(rE5=FMwVufqc`_5VtZWWQ{U&2hDQm_g!i$lB%Kq
z;wL2xfZKNK6%7goX~~i<^k+azmGcy7Fuw6JpsE7<Exm7o1mO}J+l&8Eg$(SMDy$I!
zzg3|My~cM>IHSJg6aJObr>gvvJ%&eD8$~;3|8a8l0(KZ%C29!QwecbSM-)~13q38)
zf>KtrVAoCT@YWYtYOl!5(;~c(l>2Sv0FUcUorM=gDIiunO^jgPN|>+E^RS7;yA%xw
z`QW>pxB8$}fb9CVN6?xj?v>x??@G4J^Y=n#1(AvJ>J2#J=YzfP&r_w8mj9e<dj~v1
zT;uu{dcd^*^;eQx&j?}aHh_Uyrf+wY(&gPk8kQ&^g-ed<h$&4+c*4&8Q`T=vCjxQD
z;|Db(O|gOLBg~0ALqTJu6*1+jm?f@CVag$f-ab4`{W)n_>aF>3hvVI(<{7Hn{nirW
zBh8i-D$T3^NlqYx+Ou7BXx~1z6f`!1Y|=QA!CXe6pZK(_4D4^o_<USlEBsQJj+xlm
z35Z&2>L+-sM75!b>c{;Q9j?d@59a6YutUw&am}y8577dd`)jWBM#rnWO;+^Av_g&D
zfxg<~e2rb|Qg;}|e+|9Vedt5-n~Ki&J2nhjYxxyz@LvOG?i<pN`f8j08#d*1hcRCM
z(|P<+WA|72p#-7(Fy9ZoH2!Nx+W&{gLw&VpvX{n({`cU2qQ}QWwa3P_X+(Xj6AR0~
E0Hg9z@c;k-

literal 0
HcmV?d00001

diff --git a/tests/r_32to64bit/name b/tests/r_32to64bit/name
new file mode 100644
index 0000000..fb45fab
--- /dev/null
+++ b/tests/r_32to64bit/name
@@ -0,0 +1 @@
+convert flex_bg 32bit fs to 64bit fs
diff --git a/tests/r_32to64bit/script b/tests/r_32to64bit/script
new file mode 100644
index 0000000..73493a4
--- /dev/null
+++ b/tests/r_32to64bit/script
@@ -0,0 +1,75 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fn
+OUT=$test_name.log
+EXP=$test_dir/expect
+CONF=$TMPFILE.conf
+
+gzip -d < $EXP.gz > $EXP
+
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,sparse_super,filetype,dir_index,ext_attr,resize_inode
+		blocksize = 1024
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /
+		hugefiles_slack = 0
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 1M
+		zero_hugefiles = false
+	}
+ENDL
+
+echo "resize2fs test" > $OUT
+
+MKE2FS_CONFIG=$CONF $MKE2FS -F -T ext4h $TMPFILE 524288 >> $OUT 2>&1
+rm -rf $CONF
+
+# dump and check
+$DUMPE2FS $TMPFILE 2>&1 | egrep "(GDT|bitmap|descriptor|^Group|Inode table|features)" >> $OUT
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+# resize it
+echo "resize2fs test.img -b" >> $OUT
+$RESIZE2FS $TMPFILE -b -f 2>&1 >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+# dump and check
+$DUMPE2FS $TMPFILE 2>&1 | egrep "(GDT|bitmap|descriptor|^Group|Inode table|features)" >> $OUT
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+rm $TMPFILE
+
+#
+# Do the verification
+#
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" -e 's/test_filesys:.*//g' -i $OUT
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+fi
+
+rm $EXP
+
+unset IMAGE FSCK_OPT OUT EXP CONF
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
+
diff --git a/tests/r_32to64bit_meta/expect.gz b/tests/r_32to64bit_meta/expect.gz
new file mode 100644
index 0000000000000000000000000000000000000000..d6985ecb82b5b7bd9f5046ba99a7e6cf2d60963a
GIT binary patch
literal 2956
zcmb7^S3Dbx8i%b?($=a~TkRRt-fgMbHb|`0s@Q6@_<|}v6{JcBRcb{kV#JC)(pXLG
zJq~I{%@VQ4>38nVxjT2y#e4Do-~aP_1(8>&BIf0+X=t}HEc``HSmP=N+fbtN*sfZ_
zPI5yfF(wX=i8173sS4Yynw{LnHqLDJUWAFD8qp$%9xzjqt^Uf$8yWQ8kJ7)$Nle|5
z>b{YDy|Z~*HxNtfndcbYXV}Bp*;Oy~2u(f&X&ceEn@8^Y{896K^5tLCv)TfUlmjea
zZf4LA+cdL)IT_g?V!k0~5tP2A4Pu@aU+C!qcE<0KTFVxs&0KQ^-LEdV1SVPEI4L0y
zX}%gW<h6+NA&6aRQ!(0D4ory)M2}VE{1m=DdmedsGR!|ANVVtpK(DPmXJ$5JEppL=
z^65NIK)?60pSJM&$?0}*+Y`NDF3rj1hk)jYnFaP$ko@ZQ0wpr{yjdOkHnTWYV}9uR
zMkZq9iAO!Y=EIU|b!cU4!&J+&M`raiTRAQnvj=33Qb=2te*^8uiSwFX1ZCl*S2sf2
zu_C$qWoXlslRvf&{~%-I=of4)8?kh9g4cP=gYDkl_Bx8c)M+j~L@Z>mG<~C7MAmmX
zc)nX|bF9nUT8)3YTRiiv?|@aX%E`$eI0|Xgv$){lI)nN~p9fL2!W!27LU$?k42_6f
z#ZTz+(`RjP#~r;&{`K+h8?LZshVQ@K+u+efcqiWag$+dkwQdpxaM2&a<-caKBT2Oi
z(8P0Ot(x8BeSzGufCP*+DraU-0l$PZMH9CxcIGY@)aD&8kI$P9cUmub`yXl^y$QJ)
zMR(2SOZ#RcV6xY#c-z85>_z@6wI)CYXm>dG8h*2s)O-_<<$+N9T!GPh$>o5Kj{_fm
z9ro1~dkp6~{o1R|#4=zVQi%G1suPbJ)$io|WhkwhGeiTn{H>Y#hzxYre%inThz?37
zee!NkBrnCp!+6qIF;r5A)83o_V>mronVY}3g*cr_55)FM##AyUrM8zIv>d$a7IMzU
z(a_S0&Mbj;ZnZ0gQ;TpT^_E0|g25FZTv(&iJuvEjzdPMuhFy2gD7%AngL*oOn28@4
ziE$?Oz{Sm!d6YT@{3^u`YFBZ>s7m!a2AWdVRV|gOfm?-A7vwiajuxaKvooJoh#YEH
zDnHiUZ#!wgP=DHl+MdIsYUm}zzRp-*dCO~v_Zq<HwjG@_4)?iv(O-8U1b0SL?3?oe
zEGI3wFE4PBHXtQ%@nd7oyEC87SVGmE%J_;a1h*nC24wUeqLoD&%fP7ua8u7{0nGQa
zex@_CKsIYbgjI_3SXzy3dZmY^oWptV04&_H*3$V|IKK?izh4$N0E&4JvB?6=WbA07
zR^|$OM*hzAFaSOhl1;kEbNwJXnOM69NE?torinV9^AgDJ(Hv5IDkK~5B-6r^?Mf0N
z)?9n(w>Qd7z9*zWv8xOl9n^U2XY`Ox5=by4-IE#edLXpzJmT$Uw+jM^$4<n@F7SQ7
z!#(#imLMC$ZNNFl@W~#e^XeG~`1z=4K0QCHiXompT&>N+ou?zc+s&>PwI_!GTkAHZ
z)3D#}N+0t0mWMOx`;3vca$!&qz6G<8@pc;s23r#k9TSS$<9}r1c1*GIE8qSe0zMEn
zw!O7nTxh2o!5opA`bfc@NS6>qv%>k#k0L6I*HdI+>ynt9b$5Xs{Co_PbDi!frJgTV
zks=oj>zWgk{uRK&hSEah+NY)-iFZi*<oRfz>5Z*9-6jluC6rBw0jCDW#>tqVVd=6C
zN>)j!cFD?pUSW{^c(FBIXhv;kucz$JKKF71*df9HZg~~wNcd-t*5nkGSdsb#E_Rt}
z@tD{*+%llDX;UnVA4eLZ_FW5;BnbM`sKnMTyR#GOQ~HV87h4I2t}t3v%atV_!)B_|
zU-$qrr2L3|>!y61k_*Z`MuHh*ohs!P91t!t5~H867u3Jk6Qkh{S0srLw9||-R={5r
zu}nmgT2(N$*gAmS=>~wO*IEGt!_l!_UGh8VJ6V*TC}$$xn<1>Ktm4LD-S+^N84dOL
zUZkXfGtRS8R%o2EmOX818CYXuW7+;=oZd{I0=K%a=T`aQM7CPho~B4aT4tQJBXv;c
zyXo59c2P(<kUJ^I+`og4t)eT|Vf{Dn;`Kt31k@37@8O`6xudLneEo|@*AH*r6)G~d
zG?`3GV*6ma7!ogNxteSaw?`~(L|uuxV-;_adiAcwW&<{HUEMnkbPp2G^zJ)-I7!0#
zSwzp(@4fPB9O73~{iD&Sy{XQx@40Dn?VsxkcmdUSV+k*rc@}1Dh7u;@!pbAxI6OAI
z&Id*owR^H~+)%n;`k4IZ{5moFI`PwYQsv@PcMOWEIEDS(AS95F>bgi(SvI&Enk;l1
z<33^CAq2q-@S#h!3p>BO7XsO;G6;ZEF8c|DFC0Q-Tev?sjg*;EQu?|)t2g7eG?8Zs
z$&NQ44k{__l9+Yau<}h!Bc1fCjQ8Z*_ewFc6|6vqU*8z{D%UNWEM(x&Up!b^=E`o(
zflb%z!N}*;Lu(9Q;BmS*C7au(ET0gEgR=7Sflq=lJxliT$0A}S#1&uUM2bYP#;VtS
z#&sq_gL^Q?1M{&uSH-qXhd=G_b)~khh{H857(~C)!C&8mRySJ)c3Hc?-H3)Rx%)*5
zI~>e%+9ndV11{{5TKG`Is=gEH8=p%;J6$8EnxfdQ#KGkRubHgS5&u*F51A2ZOl~53
zB@=IA-+Py*`q5Vk;GWbQvcPReV@4X{_!=adqbV1Cz=1r5ozj@zv;`VQ-FvGAcr(Lg
z*_C}u$(~O2Z3WnSR~lC2TEJO>(ygMTmsXpW5zi{JCV7=E_#w4yS=Mx3&IOicZf}zl
zmGW%%*B3P+Jb~!ZFz|&7<d;C3cILW3Y{v0(bWwJ`I1fh~qhH)PlkYzR?w}6Oa0UiH
zt0`LHYx6V?hAY)b6BR|H`zjuYTwJN;#%BZ8{2l%=etVEgcsUwn)Ou)Yq_Vc0a-Jzt
zXj?F@{3___6i#C@4sy)h%Fs$6eB-M2j=8VVke=A%o@zg^nIL3f%d(IMwk65ahXwqR
zMDdR#M<uk7$GXYVj>VMb6>4DvyA+#a2#lwdv9inFT)};eE`bm;E9aUC_SV^n(Kk+k
zT%A%eLvCn+?cml(I~4;|P4i*l)$N8q>#4Ddddh_PW!8483ES8qf3@_k>v$6-z;tT=
zq_d!7oTeeE^-U+f_C#Z}OOrOi`U#6n()F&e=LO@>C+-roRe#+<qSakWuq*$(2d0n1
z9%udXzCPun^x%Un?oCMoNDYnjf!ZVyk6j~qIH4>qFPGnS%G5?{aam5!q0_<WX5T-4
zlo5B2lYE~<hm9$@|KLjld()ETQTr{3P35|EVo>Tv30y;9I%vaCpxW)I|DkrxUj?c_
zvIt?)N-)4zmiz7DD8w}}zoqd#%|ooJElmlt1M$eYWHho)mk;AY&O|J%26~_CK(kK#
zD~j-(yjILhkMVJH&kpd{OINHy)wiAeF;9Z|+&T@^nzlGE8jibHkG%LullA8TYMY3y
zzCjRR$4hg-=*^Qp#DvFsAnd#aJys{E_8`hu(203=b+CA>b#3Otn;bDu5rn+zm|x6)
z%S`LfJLgAo>7H%)e0o>UF&n#p9y_FHIe*{UW>_fS_Nr^SG|fF--0H>|SV!Q*mtQ47
z&g6vY>t0AI@Dt1sd&7Cn|Am)l`~*uSr8%<Ge5)*KL<cpha~^$>G$MWu-Re_PoU7i~
zRcIM7+aV91Hvb=Z1`Eq6(DrR6y(&|*hceFIR_*?oa~g@FKNIx&l6H%>D|30As=07%
fIH;utsvtLEMx^{NUbfXhFWC+In1f{;sHpx8)}i25

literal 0
HcmV?d00001

diff --git a/tests/r_32to64bit_meta/name b/tests/r_32to64bit_meta/name
new file mode 100644
index 0000000..d83492e
--- /dev/null
+++ b/tests/r_32to64bit_meta/name
@@ -0,0 +1 @@
+convert meta_bg 32bit fs to 64bit fs
diff --git a/tests/r_32to64bit_meta/script b/tests/r_32to64bit_meta/script
new file mode 100644
index 0000000..5d02114
--- /dev/null
+++ b/tests/r_32to64bit_meta/script
@@ -0,0 +1,75 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fn
+OUT=$test_name.log
+EXP=$test_dir/expect
+CONF=$TMPFILE.conf
+
+gzip -d < $EXP.gz > $EXP
+
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,sparse_super,filetype,dir_index,ext_attr,meta_bg,^resize_inode
+		blocksize = 1024
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /
+		hugefiles_slack = 0
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 1M
+		zero_hugefiles = false
+	}
+ENDL
+
+echo "resize2fs test" > $OUT
+
+MKE2FS_CONFIG=$CONF $MKE2FS -F -T ext4h $TMPFILE 524288 >> $OUT 2>&1
+rm -rf $CONF
+
+# dump and check
+$DUMPE2FS $TMPFILE 2>&1 | egrep "(GDT|bitmap|descriptor|^Group|Inode table|features)" >> $OUT
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+# resize it
+echo "resize2fs test.img -b" >> $OUT
+$RESIZE2FS $TMPFILE -b -f 2>&1 >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+# dump and check
+$DUMPE2FS $TMPFILE 2>&1 | egrep "(GDT|bitmap|descriptor|^Group|Inode table|features)" >> $OUT
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+rm $TMPFILE
+
+#
+# Do the verification
+#
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" -e 's/test_filesys:.*//g' -i $OUT
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+fi
+
+rm $EXP
+
+unset IMAGE FSCK_OPT OUT EXP CONF
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
+
diff --git a/tests/r_64to32bit/expect.gz b/tests/r_64to32bit/expect.gz
new file mode 100644
index 0000000000000000000000000000000000000000..375a749e42b2a49724e874761048ac4cb91e010b
GIT binary patch
literal 3179
zcmaKsS5Om*5{6Y!ibxYsFc^wTGg8DLMG#16M^KOwq(qAJ5W$1A0E!?;6CojBFdUjn
zFG48NL4t&iNhA=El7O_(o0Q9$xi9zSzU|KL{`=4DHw%pCJb{`3S)O8B!kHogbpo%;
zzVFAK@qg#E8R@z@T(f8>(%1Uc2r$FX-Ujm;IZwLT*5g4PhiA!e-3uI)>xLw^fCn^w
zzV7nWE-u9)1MUaw0Km=Xm|FwlG3|bv^l%tZZInu1V*1`j-3G)2(#vquxIR&D=f#cD
zWiowvZz<ajiPT^Lc6`6k={zxHIB+>8?DC>xN!f(<7_h$GuLkH_Pv6?ecBjv8o6FQL
zCIm>@KexVGrl@(LrJi)-t`Zj!oyZ;Goj@5Jypp(XU8FNU2{ovv{-L*r1==_3b-6nn
z?us9bEfVy%yZYx@e@vV1x*Z~KD4m{ze~meK9!7QW;2Xq)cV=u2ZPMLG`0Lg9%3=Wh
z@$IR~>5nqjW$K;64wxGSH^-dfW-hutSENv>Ut^X$&(?fB(6o1}ISAq?B4;t*j!AX5
zhtrt2&B^&5?FJpklDEXLntt1-w+t9=Ux*QO9@csrk#C#L6hI*MjoRQ`0z=grnnR+~
zw-*kaYP+lG^pK;F+8n-_gQTO7pS}c~bZ@@5FAF?c^LN3)o|pbYNqA3<K<aEra~;wC
z_gqcJ@F*Gny1o49^zbZ^1vlGGjYwE8(#ayM3@)$CI>!qH`PUOdC=Mu11B>nS5a##7
zh469Yc*!;pmicwFqCMeH`t03iwZ`^-PF2;p(CJ&e5t_^c=8kVzmd=c6;HnVOqj6Tf
z*hdK>#^ZFkCnKGhvh=}J75}T@>>XC*%<wEabm(AuH$JQBXmTE@h;I%$T+4#I3)miP
zGH88c*Dm>&IoIa7DUZe60102g`m+sn5fA7OX%!9B4rY|B&xktNkiqn<JzDQO_cfx{
zlRDfwnO}0mfn)ecR6@Uvq_l`&Sk-GG_?SHR+v$^Q!IW2A+JtfJuz>6=APGwI{;_GN
z-+o30%ISOp!1n^f)&hP#$baeuPOb5bIP;Cr+h?4MfKxQo{BMYj={E71mdvzh(x<Ch
z0~ORcAP=8Ye~@n{$$EG+?}KNfzD%GXi$5`=Vbsu!Lu{*&Jm>1nd5Z5PR=?${NTSfG
z3qEDJP8zu{GWCDiaz0%jNFAYl)D%*8OkS<+mNE(wutZwAju@4@Hz-Xr<p1fv1zix=
z5ayi{&r!|_#LWkC#;k6%N<-C^+4oo0%y~~zZL5YdN~QMLVK=}$UFz(9yeARb8<rg3
z(lmv3b{VM&!~l^GQ5+&~1+YP2sYOx6-k`d>Byiy))<D6>^DO=(C*Ogr2hIAbK7JN)
z#yltaydL;-?)2qg^?nU7lr-#pE+&7EuySfK*f#b=y!RE*X^XVyh1{r;wLU#Ka=FUn
zTVv(|o3#YL=mwV+^@@MEzf^IL&2<Ia+a4t$Cw$c*nybo}WK<0CUJp~>a)h_Bj}XsO
zb<rrm2#RmK+(&YZE7>;K;A|s>LIex1u}O3p;DX*?`|csox53XwvYPt9nKA5E^&_{W
z<L6BE%dC^STeObzw~QU*p9=Kyz&iO^kggtZu4myTOOI48tVeQ*(yFCtim+7X5@-c=
zb^J|ou#-W%iI^bSAnO`n*2VV5Wo1>9B5ktc4GUF<md8qZ!Tr{IkeBBrx%(<xH6~2{
z7L3_)ExJ|tMXS?R_ZoW^3roeD_Ic@KtKhm5WCdm$5=%ZO!E53Q)xFT>jl5A}{qElQ
z#q!VUxhkaY1W%N+MagF^_6JZ#(SoI%*|Q{3gcJ^dd5=e0?!ItYy~du~Qk8S&xq}__
z^yScs9+^(aXa4w1i}ZMRYo>~*V2ubyxp!?+`Kgt4Q^Py&aQ*v{JJ=ei+bew3lbu)6
zx>NhHg9s68=L=g_6sb>%J@dcur)Yxx@=N=9*V}tzB$U6d*6CKuE38RG$$MveC)-Mk
z0#9g53#+K6R`JwGM&1hxcc99bbZv?k6>AkG3fsQKKUq;Ma(*}FM-jQ1xB9{PzA^rN
z)={6L6vOauG1cj1U@&{nH0(g(Q^y6Ux~vwhKn(4iWyZ^`?Pu|zLD4d*IX$vXOUaOJ
zZy#T6)gcf!oXt(zba8zMF|kml;pLyFxe4D41x%cg_K6(WM+JpPU!Tww+nC-021D{z
z^E}>Hmx^GfdWuCWtiKsNkv<dB(4t`wGZedoa)ewK<}qxEA^k$F2e;k_AqvauVelxZ
ziH1_~X?$C4C+e}1VjtQQ6-`Aw7W#C}(rI_k0PK>Ncm7p(2ygtYkqy23801Fz6&2nT
z?||v9eh{Ll++JT|bQL!F$~5hhh=^xu%F7YIkC3aOCPUUgVLRr+1Q>kFCGW~r&4*WR
zYXk0?oM93(1Tg@iyQ4++@8JrKp6K!Z?tHbx6zzuZa@{7#Uq~d^Y4_+knqve1pq%n*
zC1yx7Ra9G{D+-O>p`x+DDhiFQ{+6zgb5LM{7xdhIy_My!&*ldC#U0Hbi+fUSf1ewB
z_C`P$?c)_{qxWBK;`0ODIsofR3|D;5@d`7egn?JxhBne(&eUovOQK5|#M1RgR9t@;
z3SPW%#w|*8Xj9fS^$}w_h{qm-QB+JD7fqRA*S(O@Yl3{dg_dNiymrYhnxXo&ysT10
z9pOs(#~7BZ)i3WDqCZv4nbFC5`Ut}a;@87e58Uc1ev>(9hsu<9y^S5W#!OU0=xntY
zALTP@n_Ms$_i9*(yHQG7wYl*3JJ|8G;D@gv2-_l55dpxbS6K1dIauZk&4^ZcGH%9A
zxZ()~EopAY8~C#rWD`r^CJocN9HY@!Zs1bno`Y>V((0@~Li~cCAimVbo&V@%mkomx
zK)5XNhMR<{lDY@g`=h!!kbvDe#FzTo&H!@lDWQe<awFq4f3ww|sp!H00C?!3rnU#f
zBhALI)3QWiWi!#o!N_<oM?RyyQje^{>PCMqn?rk2F4)wSzPI^RaIP{q(W-LWWc|nS
zID-jA%a*en)NmKaMwZsw$WpNMvvS6d$A!51&RUV^@v{0^f8IV~2ltf;7pj`D`WSi`
z17jeUf84S5wZp66YQ02wWzkky#czEwtD>4L&oXa!nQ)!1&G#JtM5Em2Zdw1wLSU93
zc1%OI#K{wP<`lpgRvl&-_@c`J<ZaIPs`31eGCODh;8yo%^kMQM`Y>3fC|3MSTMuVr
ze%U32bm64r3we3CLvid)7l|KoyX&sj0-DUIjKNKCbF2rHyjV|&6AN0zBzS#xZs8sm
zQvW9JOiUO^Vo4T%Q>vQ(v2w9iUIFfSJbvXj<u%L|e5}6S%JYW1lLr~Xkz)JO;irvl
zd?r{$LHmxB!ewQxq9EvE5Q#@Or?Nc!SxL8w7H4+nqw9*+9il1d=s}P_&Z;A6o4|j?
z<m_9S(d(d67VHVb7W8j9kCAy3>fbM1;c_kg_~xyi9u$#dA}U2lni*vEp|toyBj10`
zzZ4a$j=uWZlVU9kIY!bG@$a7y#M+FDXyO!{iA(QfhkwqHua{0m70}W^6fA-?>mCE?
zUynxg3Rl=XQA^8CsuKG9zd#f#P_Ryer@1Jn@wN?8TVr*xf+3&uEK1fQrxFStVHpdT
zM(R<^rS~k8p24gILYWR3Lz}z`#2<x<zvQV_>y=v7y5O=3z<sBXb`7&*Ufr1(x#OcR
z=qnV<nX%-3Dw-A@jf_v_;_qs!+jQuZ@o~)GIb802dhmSyy^cM8zKt}NX$BgN@hfzC
zhy?$c6pQcSqfgdG)PB6Y<R=)LS{gVCI()WHV(vI)x{ov*Jk?x`S=t<6EN>6{C-q)D
z(r=lZbnsntD_Jcf^Rd(nqhLTFFP%A1KyKPm<eTsrZ(QDEtHvS2bN>>b;Axl+ulevv
zJaG1i!!LV6TRnObM<(sbJZ)WCwwZiLr~CfdyYzDTXt?gEd12(ppSA`Jwa*UkENcQy
z+@Ynh3P5m0#BNqDNGFdtx@P4|3jE9NaMqg?bDV*y84FcajEE+}{)&fPQ(_$mm`2#|
zr>2zwktDS`kj{0)?s4}}T5}cSSmb}h3xjEVSstXCIuIOX{;!GvVs~<tuz!(AK*4Bh
rSi*ijD8iEj|3BfeM|eaIXd(fz>-2v*n{x>J5o}?rH%;OmpE&U!+d_0F

literal 0
HcmV?d00001

diff --git a/tests/r_64to32bit/name b/tests/r_64to32bit/name
new file mode 100644
index 0000000..4c82371
--- /dev/null
+++ b/tests/r_64to32bit/name
@@ -0,0 +1 @@
+convert flex_bg 64bit fs to 32bit fs
diff --git a/tests/r_64to32bit/script b/tests/r_64to32bit/script
new file mode 100644
index 0000000..83a37bd
--- /dev/null
+++ b/tests/r_64to32bit/script
@@ -0,0 +1,75 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fn
+OUT=$test_name.log
+EXP=$test_dir/expect
+CONF=$TMPFILE.conf
+
+gzip -d < $EXP.gz > $EXP
+
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,sparse_super,filetype,dir_index,ext_attr,resize_inode,64bit
+		blocksize = 1024
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /
+		hugefiles_slack = 0
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 1M
+		zero_hugefiles = false
+	}
+ENDL
+
+echo "resize2fs test" > $OUT
+
+MKE2FS_CONFIG=$CONF $MKE2FS -F -T ext4h $TMPFILE 524288 >> $OUT 2>&1
+rm -rf $CONF
+
+# dump and check
+$DUMPE2FS $TMPFILE 2>&1 | egrep "(GDT|bitmap|descriptor|^Group|Inode table|features)" >> $OUT
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+# resize it
+echo "resize2fs test.img -s" >> $OUT
+$RESIZE2FS $TMPFILE -s -f 2>&1 >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+# dump and check
+$DUMPE2FS $TMPFILE 2>&1 | egrep "(GDT|bitmap|descriptor|^Group|Inode table|features)" >> $OUT
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+rm $TMPFILE
+
+#
+# Do the verification
+#
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" -e 's/test_filesys:.*//g' -i $OUT
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+fi
+
+rm $EXP
+
+unset IMAGE FSCK_OPT OUT EXP CONF
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
+
diff --git a/tests/r_64to32bit_meta/expect.gz b/tests/r_64to32bit_meta/expect.gz
new file mode 100644
index 0000000000000000000000000000000000000000..e39d01f1bc21038a73c3678f11dd66fba8c25aeb
GIT binary patch
literal 2924
zcmb7_XHXLg630UXl&VAt<q0B1ArvXns|ZL5U?2e!kP;vqO`0GbrAwER1h5gLmq0*>
zp+uw_h>8?Jnh-@Pj}A&d-psq3ySY#I?LV`#v;W!s?e3-2Gl0k$Svc_2p9SWA;wC(a
z#E(=_3Av_GWm;w*@pXNFY+`3!tl`CTr0`|Z)YwWBg|<9!6fR1lREZ`*bM?g(UTk#R
z*hXvR^ZK<9E~xEvuKaME;X*Gb1#C{$&t$Y5wvi{aw>!&RmY;W9wyfW<<I-R*67Eha
z>!dS8n)dBytCcO5cakQe#|{>@OLR+<<~n!g*SFozb%v_#G|$!P28T<lP&pw5(nTHN
zF%o)4Wwr6!tjl?8Jxj?S#F-yG_o{){4I8_((C0r}s`VM%kI`yvPY&2;x;&GzwaKyf
zT6;NS-7i`tMt$WI;L)zGfb;YV>Ltg_{XcWax8F0X$*yhGyx_swd@T+QZCf3RcI3nj
zVIUPBv<;|jVQSkNom`t1783*0-8=L1YZP7C*%klD9#8K554LsV2O$fANs`3gu>7+V
zvwH*6F&Bx=jL-|sEsX8EF%vP+p!DaC;p>x`6H8}*4VTVjzU@ewl3aPVm$@~&M$+a!
z3<mOAPB5Zk%!P)m%BYUNS`P2NY`VU4<k=kavM6I>Nr+`~{{8|NdUEpO{cHSJVs*@u
zxmBpT>rsgrx_l81)88waBc<iqpEj8U^V{wPiglgso!Z>_{m7RvmI3TsaKdzDDu?_I
zVIQCR-5OQ&A&va|O8S;uci)DdVI<-0hGc@}B3e^OLPyyjW6kl0VI+?5&FWx#YG3zY
zet$`q8uRODDl?{dJO{2*{PemNu;=P+XV!p+nupR-?L4LNwckJKb-!i52Y<@a-1v6;
z363zX!hrbKNIKf0`aR$w0^BL^6#7*TI5?8;r6cO{Ts}<(z97Glwj+1_cD>;jQQkt!
zDj*{3w?+w<nUb59Ye%f1UVX8cg^Kg8yP9p6BBvQ&-8rccnk1ORAdIzv!%ibuGnux?
zD&J(`#O!+B%NnRe3!#Mz$0dic261vehTggh;BtQ4_CS<BRhLUbehE<)i{ZI}_mHPP
zkceB%o>G1w)%3|uoT1+!L3$BY=I&f%{KSJ)m&KE_+NW-hc9IglXm}l*^d--NT>x5^
z&gRhKl5tFaA)#scBhc(Xy4?n0z_8u}6)4mXsbH8qF)Wh#Pv4<}VUQ6~rtZeFl<5~p
z`NeEQ5wP%+4%4G~2AXf+S~BgP0{=K67P{KqIkTponiNx4sbsoQZty8TT>t<p!k5I2
zy=1;}1F;3@6#bN|1$e#YqneR$y6C<Cz-bV6H5bTNFr39kHOA3V1f}=)dZfSwT^d6Q
zn#Oja`rZS@h0Gnf<tbQaJG+8x2@$(x=$e+czZVblW_)}}pYF?RCWPEHXq4$sA5Hk+
zb<SImd{(Oo?hBcBP<?6d9mEuhB?`-h8VO}Oh82ucFARuy;M`E|PbnuA4bB^1Zce(>
zSm|j?>T-dE#5Zxlbn<XnVgyJaQHi}drN*ub4K5@r&$a|Do8L@SBgUy)q5ze2)M;}s
zF?er}G6GJAxeD+y@VDbstLs@bixsb$rUlSC+@&Eiz;eRIc&fS&n%Q7v(Fata5Z(Rs
zyidNz*aM9%l;Bi~;BPP-_OQ58w9=@92ZGr>;5A<Iu))CDKnY0;#6%mL_S3brf`mr7
zh5|j*O}fWp-{EQvI`#EA@k+U`D-@$qk?}X}G@2_V8z3gm#*x=mY>HHsHVe8WW(>Ev
zXVC<aUNb*~7=?bmjp2Hnn_$xN&Vp0h%q+-O%y@wE^L;4-^>9hk1{ns&HMqr!dkP#F
z<t)g{d_fDV-jN9w?;5MD$?}b@ev8>`QIwGc;R57R@4A8m6l~3M@t#8xc2#5`j$EmZ
z&(^999)FjWJ)F>9O!|C|FId5{uy*mBir`Fj?Q-%UV$A1=<ab%(9fx7gK3Hah{UTUu
zeJO8!RJp(3-2=*tz11YA$!C?_4&Xd1`^0sSuc6jJpiCNviE>Zkgu68C_D?6|ip$d+
z!dwUZ<|Fv?ox0R6;FSkeIAq+eqItn2I;4|g42(OY)tD6w#=ETKNy3~hr2k?5N14v6
zyfs;?T1qx&?XoYK+3NWSa?2H{RrNvM&&*73JM11~G%aw^+Rh~Ws~4NRb_o&vof-cO
z?av@yn3Q2sIiO^y!D4jA=1L5CRJSW;pUnwTYY?EX@xCWi0H2T_uKn<Tp6CE_J{2iM
z2IOM=+ZPhWida=@D+s__>F#j$kOSq@5nclz=gy~@{4hKZdMM$R{&#x-0<IIGz+{oe
z4WnPAS16c0;QFPF?phrN@v@|s|FF;V<ftsqIN<`K=%wEsmqrC*0e@g3*gl0wdQIIE
z%C_4TCPTczpC|+D*hWk%O|($rP2y|E*sVUnqJ=>5dM}kaB54CJcjXORcDy@*0qZ{R
zd|q8zzh}XSia=uh^vg*YN}{6o)=Qa>UocmDvH@`ee!FptY4sh=ru~6Yy@FLgV0WMo
z&WoFHh?GN<%z&H&8z6Bn4m}FS*S%1r>f|ZZa&kQEVwrXuq=THDYq63r2k^|z1~Rz|
zG18%xxwL^$gk@7ijB?r%C74*}Js*KazKiQMP)K5>i}pH&)Ma~PlMRw!;0y}n{Ph?w
zYMP7V;4b*iOHwLj?-y%{98b%|Zn@WdD>|`WcLEH|sa$cWnz)=*5sv#Xe=XvKmp78c
z@9D@BYpoy&b{V@0dzN{(5K$YKJ55M;yH#Un3y3gEi-~o`(clp<)u?K;F<{Z!o^3`H
zv90pFBd?7Mr?(iSksAFBaO#%|Dt6VX$k^LReOv1A__=UA3JA628U{Isr*x!Kt^6C9
z`9+V9rFk2n#{$?IWhU#gKm<VBn$(2N;g{d8cI)*t$G(E_sedF~tnnRvd;MFskbVAf
z4ze<%JxBFU)48tK(97xGf6cT<5hB%LfX~YkY*-#sw1=jZp3<5utVMzTbuGpN(46ra
zN#R}tB-ehZg2ET>pWu#H1g~}$#PAy!LI4pV0<Q{#uAgLELbJ+E->8C7$ba+T#l^-G
z*(X})-it+|V+~9QK83YNkaXR{sC3>?C|?7E+2kz=6be5{-`L=aHE?&4^C-;gR=o^I
zD&4vHnD?>?k{_s}senC+r*Ov_c&hUiAyE!9^)y;B1ut~fI2-$F)J?<T#M^=V4MI~Z
zq7-RAex(oMqe7MLM48QzZeRxKCsCPeFalYA>LYz_4rGgnI3@g<;uEAIX@$#qk0H9e
z%aB#oBBWV&!~P~Bi|(`|`%*k?7Pmjyd{X>f>lqX4F!)v_!qEuxpEp7t$0+J<aP6&0
zl{EJ#$s{41u4uYdPP2?3xUcG)g7_ucS{3>;gp`N|o|>srem=bT!SP3S30&itJeRo3
z4|jJ7A%AtYcTyU*CpV(K-}B^T|No|<?<;M3-d%hf1{3sqLSkHM>eL%4o6+7<%?zs8
z^bf97Z8zoZhWltj&6T~wTQql6$yiG?e<o2C9-*nM*lnbmbSQj0dFZMdUGRm;`~LFk
zo-38QU<g6#n0p{oD^emtqbyijs-J~5>>e@h8YKVuTO)Z&<l8)cq=GuGJ(GHO?BKCZ
zNl{PIGn~n#9kYgYe>q3$?Mz&mZ)n4A{q(7;lb50)TOiA%!^1-06yLSxqrikwZ`o~>
zS=*=;@zwBu%a}Io-jW>dXb(d8O!)EYXP9ibW9qa2M^X(2H;>_WHf`-FtzZ!%OP@E>
Y2I|58$YVa1zGLR)AG6a^_5i>?0Su9%RsaA1

literal 0
HcmV?d00001

diff --git a/tests/r_64to32bit_meta/name b/tests/r_64to32bit_meta/name
new file mode 100644
index 0000000..e99ed8b
--- /dev/null
+++ b/tests/r_64to32bit_meta/name
@@ -0,0 +1 @@
+convert meta_bg 64bit fs to 32bit fs
diff --git a/tests/r_64to32bit_meta/script b/tests/r_64to32bit_meta/script
new file mode 100644
index 0000000..e2190ce
--- /dev/null
+++ b/tests/r_64to32bit_meta/script
@@ -0,0 +1,75 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fn
+OUT=$test_name.log
+EXP=$test_dir/expect
+CONF=$TMPFILE.conf
+
+gzip -d < $EXP.gz > $EXP
+
+cat > $CONF << ENDL
+[fs_types]
+	ext4h = {
+		features = has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,sparse_super,filetype,dir_index,ext_attr,meta_bg,^resize_inode,64bit
+		blocksize = 1024
+		inode_size = 256
+		make_hugefiles = true
+		hugefiles_dir = /
+		hugefiles_slack = 0
+		hugefiles_name = aaaaa
+		hugefiles_digits = 4
+		hugefiles_size = 1M
+		zero_hugefiles = false
+	}
+ENDL
+
+echo "resize2fs test" > $OUT
+
+MKE2FS_CONFIG=$CONF $MKE2FS -F -T ext4h $TMPFILE 524288 >> $OUT 2>&1
+rm -rf $CONF
+
+# dump and check
+$DUMPE2FS $TMPFILE 2>&1 | egrep "(GDT|bitmap|descriptor|^Group|Inode table|features)" >> $OUT
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+# resize it
+echo "resize2fs test.img -s" >> $OUT
+$RESIZE2FS $TMPFILE -s -f 2>&1 >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+# dump and check
+$DUMPE2FS $TMPFILE 2>&1 | egrep "(GDT|bitmap|descriptor|^Group|Inode table|features)" >> $OUT
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+status=$?
+echo Exit status is $status >> $OUT
+
+rm $TMPFILE
+
+#
+# Do the verification
+#
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" -e 's/test_filesys:.*//g' -i $OUT
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+fi
+
+rm $EXP
+
+unset IMAGE FSCK_OPT OUT EXP CONF
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
+

[PATCH 29/39] libext2fs: find inode goal when allocating blocks

[Darrick J. Wong]

Try to be a little smarter about where we go to allocate blocks for a
inode.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass2.c         |    3 ++-
 lib/ext2fs/alloc.c     |   10 ++++++++++
 lib/ext2fs/bmap.c      |    5 +++--
 lib/ext2fs/expanddir.c |    2 +-
 lib/ext2fs/ext2fs.h    |    1 +
 lib/ext2fs/ext_attr.c  |    4 +---
 lib/ext2fs/extent.c    |   10 ++--------
 lib/ext2fs/mkdir.c     |    3 ++-
 lib/ext2fs/symlink.c   |    3 ++-
 9 files changed, 24 insertions(+), 17 deletions(-)


diff --git a/e2fsck/pass2.c b/e2fsck/pass2.c
index 2060ed2..fa17f20 100644
--- a/e2fsck/pass2.c
+++ b/e2fsck/pass2.c
@@ -1807,7 +1807,8 @@ static int allocate_dir_block(e2fsck_t ctx,
 	pctx->errcode = ext2fs_map_cluster_block(fs, db->ino, &inode,
 						 db->blockcnt, &blk);
 	if (pctx->errcode || blk == 0) {
-		pctx->errcode = ext2fs_new_block2(fs, 0,
+		blk = ext2fs_find_inode_goal(fs, db->ino);
+		pctx->errcode = ext2fs_new_block2(fs, blk,
 						  ctx->block_found_map, &blk);
 		if (pctx->errcode) {
 			pctx->str = "ext2fs_new_block";
diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
index 82af18e..00cc9ba 100644
--- a/lib/ext2fs/alloc.c
+++ b/lib/ext2fs/alloc.c
@@ -303,3 +303,13 @@ void ext2fs_set_alloc_block_callback(ext2_filsys fs,
 
 	fs->get_alloc_block = func;
 }
+
+blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino)
+{
+	dgrp_t	group = ext2fs_group_of_ino(fs, ino);
+	__u8	log_flex = fs->super->s_log_groups_per_flex;
+
+	if (log_flex)
+		group = group & ~((1 << (log_flex)) - 1);
+	return ext2fs_group_first_block2(fs, group);
+}
diff --git a/lib/ext2fs/bmap.c b/lib/ext2fs/bmap.c
index e936ec1..b0d9087 100644
--- a/lib/ext2fs/bmap.c
+++ b/lib/ext2fs/bmap.c
@@ -247,7 +247,7 @@ got_block:
 		retval = extent_bmap(fs, ino, inode, handle, block_buf,
 				     0, block-1, 0, blocks_alloc, &blk64);
 		if (retval)
-			blk64 = 0;
+			blk64 = ext2fs_find_inode_goal(fs, ino);
 		retval = ext2fs_alloc_block2(fs, blk64, block_buf,
 					     &blk64);
 		if (retval)
@@ -357,7 +357,8 @@ errcode_t ext2fs_bmap2(ext2_filsys fs, ext2_ino_t ino, struct ext2_inode *inode,
 		}
 
 		*phys_blk = inode_bmap(inode, block);
-		b = block ? inode_bmap(inode, block-1) : 0;
+		b = block ? inode_bmap(inode, block-1) :
+			    ext2fs_find_inode_goal(fs, ino);
 
 		if ((*phys_blk == 0) && (bmap_flags & BMAP_ALLOC)) {
 			retval = ext2fs_alloc_block(fs, b, block_buf, &b);
diff --git a/lib/ext2fs/expanddir.c b/lib/ext2fs/expanddir.c
index ecc13ae..e8dff30 100644
--- a/lib/ext2fs/expanddir.c
+++ b/lib/ext2fs/expanddir.c
@@ -104,7 +104,7 @@ errcode_t ext2fs_expand_dir(ext2_filsys fs, ext2_ino_t dir)
 
 	es.done = 0;
 	es.err = 0;
-	es.goal = 0;
+	es.goal = ext2fs_find_inode_goal(fs, dir);
 	es.newblocks = 0;
 	es.dir = dir;
 
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 664c3c0..7941c5d 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -697,6 +697,7 @@ extern void ext2fs_set_alloc_block_callback(ext2_filsys fs,
 					    errcode_t (**old)(ext2_filsys fs,
 							      blk64_t goal,
 							      blk64_t *ret));
+blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino);
 
 /* alloc_sb.c */
 extern int ext2fs_reserve_super_and_bgd(ext2_filsys fs,
diff --git a/lib/ext2fs/ext_attr.c b/lib/ext2fs/ext_attr.c
index c397e00..43c19c5 100644
--- a/lib/ext2fs/ext_attr.c
+++ b/lib/ext2fs/ext_attr.c
@@ -374,7 +374,6 @@ static errcode_t prep_ea_block_for_write(ext2_filsys fs, ext2_ino_t ino,
 {
 	struct ext2_ext_attr_header *header;
 	void *block_buf = NULL;
-	dgrp_t grp;
 	blk64_t blk, goal;
 	errcode_t err;
 
@@ -420,8 +419,7 @@ static errcode_t prep_ea_block_for_write(ext2_filsys fs, ext2_ino_t ino,
 	}
 
 	/* Allocate a block */
-	grp = ext2fs_group_of_ino(fs, ino);
-	goal = ext2fs_inode_table_loc(fs, grp);
+	goal = ext2fs_find_inode_goal(fs, ino);
 	err = ext2fs_alloc_block2(fs, goal, NULL, &blk);
 	if (err)
 		goto out2;
diff --git a/lib/ext2fs/extent.c b/lib/ext2fs/extent.c
index c9ef701..4c6fbbf 100644
--- a/lib/ext2fs/extent.c
+++ b/lib/ext2fs/extent.c
@@ -1012,14 +1012,8 @@ static errcode_t extent_node_split(ext2_extent_handle_t handle,
 		goto done;
 	}
 
-	if (!goal_blk) {
-		dgrp_t	group = ext2fs_group_of_ino(handle->fs, handle->ino);
-		__u8	log_flex = handle->fs->super->s_log_groups_per_flex;
-
-		if (log_flex)
-			group = group & ~((1 << (log_flex)) - 1);
-		goal_blk = ext2fs_group_first_block2(handle->fs, group);
-	}
+	if (!goal_blk)
+		goal_blk = ext2fs_find_inode_goal(handle->fs, handle->ino);
 	retval = ext2fs_alloc_block2(handle->fs, goal_blk, block_buf,
 				    &new_node_pblk);
 	if (retval)
diff --git a/lib/ext2fs/mkdir.c b/lib/ext2fs/mkdir.c
index c4c7967..c88ff9e 100644
--- a/lib/ext2fs/mkdir.c
+++ b/lib/ext2fs/mkdir.c
@@ -69,7 +69,8 @@ errcode_t ext2fs_mkdir(ext2_filsys fs, ext2_ino_t parent, ext2_ino_t inum,
 	 * Allocate a data block for the directory
 	 */
 	if (!inline_data) {
-		retval = ext2fs_new_block2(fs, 0, 0, &blk);
+		retval = ext2fs_new_block2(fs, ext2fs_find_inode_goal(fs, ino),
+					   NULL, &blk);
 		if (retval)
 			goto cleanup;
 	}
diff --git a/lib/ext2fs/symlink.c b/lib/ext2fs/symlink.c
index f6eb6b6..e268ed4 100644
--- a/lib/ext2fs/symlink.c
+++ b/lib/ext2fs/symlink.c
@@ -53,7 +53,8 @@ errcode_t ext2fs_symlink(ext2_filsys fs, ext2_ino_t parent, ext2_ino_t ino,
 	 */
 	fastlink = (target_len < sizeof(inode.i_block));
 	if (!fastlink) {
-		retval = ext2fs_new_block2(fs, 0, 0, &blk);
+		retval = ext2fs_new_block2(fs, ext2fs_find_inode_goal(fs, ino),
+					   NULL, &blk);
 		if (retval)
 			goto cleanup;
 		retval = ext2fs_get_mem(fs->blocksize, &block_buf);

[PATCH 30/39] libext2fs: find/alloc a range of empty blocks

[Darrick J. Wong]

Provide a function that, given a goal pblk and a range, will try to
find a run of free blocks to satisfy the allocation.  By default the
function will look anywhere in the filesystem for the run, though this
can be constrained with optional flags.  One flag indicates that the
range must start at the goal block; the other flag indicates that we
should not return a range shorter than len.

v2: Add a second function to allocate a range of blocks.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/alloc.c  |  141 +++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/ext2fs/ext2fs.h |   11 ++++
 2 files changed, 152 insertions(+)


diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
index 00cc9ba..45791ff 100644
--- a/lib/ext2fs/alloc.c
+++ b/lib/ext2fs/alloc.c
@@ -26,6 +26,16 @@
 #include "ext2_fs.h"
 #include "ext2fs.h"
 
+#define min(a, b) ((a) < (b) ? (a) : (b))
+
+#undef DEBUG
+
+#ifdef DEBUG
+# define dbg_printf(f, a...)  do {printf(f, ## a); fflush(stdout); } while (0)
+#else
+# define dbg_printf(f, a...)
+#endif
+
 /*
  * Clear the uninit block bitmap flag if necessary
  */
@@ -313,3 +323,134 @@ blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino)
 		group = group & ~((1 << (log_flex)) - 1);
 	return ext2fs_group_first_block2(fs, group);
 }
+
+/*
+ * Starting at _goal_, scan around the filesystem to find a run of free blocks
+ * that's at least _len_ blocks long.  Possible flags:
+ * - EXT2_NEWRANGE_EXACT_GOAL: The range of blocks must start at _goal_.
+ * - EXT2_NEWRANGE_MIN_LENGTH: do not return a allocation shorter than _len_.
+ * - EXT2_NEWRANGE_ZERO_BLOCKS: Zero blocks pblk to pblk+plen before returning.
+ *
+ * The starting block is returned in _pblk_ and the length is returned via
+ * _plen_.  The blocks are not marked in the bitmap; the caller must mark
+ * however much of the returned run they actually use, hopefully via
+ * ext2fs_block_alloc_stats_range().
+ *
+ * This function can return a range that is longer than what was requested.
+ */
+errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
+			   blk64_t len, ext2fs_block_bitmap map, blk64_t *pblk,
+			   blk64_t *plen)
+{
+	errcode_t retval;
+	blk64_t start, end, b;
+	int looped = 0;
+	blk64_t max_blocks = ext2fs_blocks_count(fs->super);
+
+	dbg_printf("%s: flags=0x%x goal=%llu len=%llu\n", __func__, flags,
+		   goal, len);
+	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
+	if (len == 0 || (flags & ~EXT2_NEWRANGE_ALL_FLAGS))
+		return EXT2_ET_INVALID_ARGUMENT;
+	if (!map)
+		map = fs->block_map;
+	if (!map)
+		return EXT2_ET_NO_BLOCK_BITMAP;
+	if (!goal || goal >= ext2fs_blocks_count(fs->super))
+		goal = fs->super->s_first_data_block;
+
+	start = goal;
+	while (!looped || start <= goal) {
+		retval = ext2fs_find_first_zero_block_bitmap2(map, start,
+							      max_blocks - 1,
+							      &start);
+		if (retval == ENOENT) {
+			/*
+			 * If there are no free blocks beyond the starting
+			 * point, try scanning the whole filesystem, unless the
+			 * user told us only to allocate from _goal_, or if
+			 * we're already scanning the whole filesystem.
+			 */
+			if (flags & EXT2_NEWRANGE_FIXED_GOAL ||
+			    start == fs->super->s_first_data_block)
+				goto fail;
+			start = fs->super->s_first_data_block;
+			continue;
+		} else if (retval)
+			goto errout;
+
+		if (flags & EXT2_NEWRANGE_FIXED_GOAL && start != goal)
+			goto fail;
+
+		b = min(start + len - 1, max_blocks - 1);
+		retval =  ext2fs_find_first_set_block_bitmap2(map, start, b,
+							      &end);
+		if (retval == ENOENT)
+			end = b + 1;
+		else if (retval)
+			goto errout;
+
+		if (!(flags & EXT2_NEWRANGE_MIN_LENGTH) ||
+		    (end - start) >= len) {
+			/* Success! */
+			*pblk = start;
+			*plen = end - start;
+			dbg_printf("%s: new_range goal=%llu--%llu "
+				   "blk=%llu--%llu %llu\n",
+				   __func__, goal, goal + len - 1,
+				   *pblk, *pblk + *plen - 1, *plen);
+
+			for (b = start; b < end;
+			     b += fs->super->s_blocks_per_group)
+				clear_block_uninit(fs,
+						ext2fs_group_of_blk2(fs, b));
+			return 0;
+		}
+
+		if (flags & EXT2_NEWRANGE_FIXED_GOAL)
+			goto fail;
+		start = end;
+		if (start >= max_blocks) {
+			if (looped)
+				goto fail;
+			looped = 1;
+			start = fs->super->s_first_data_block;
+		}
+	}
+
+fail:
+	retval = EXT2_ET_BLOCK_ALLOC_FAIL;
+errout:
+	return retval;
+}
+
+errcode_t ext2fs_alloc_range(ext2_filsys fs, int flags, blk64_t goal,
+			     blk_t len, blk64_t *ret)
+{
+	int newr_flags = EXT2_NEWRANGE_MIN_LENGTH;
+	errcode_t retval;
+	blk64_t plen;
+
+	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
+	if (len == 0 || (flags & ~EXT2_ALLOCRANGE_ALL_FLAGS))
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	if (flags & EXT2_ALLOCRANGE_FIXED_GOAL)
+		newr_flags |= EXT2_NEWRANGE_FIXED_GOAL;
+
+	retval = ext2fs_new_range(fs, newr_flags, goal, len, NULL, ret, &plen);
+	if (retval)
+		return retval;
+
+	if (plen < len)
+		return EXT2_ET_BLOCK_ALLOC_FAIL;
+
+	if (flags & EXT2_ALLOCRANGE_ZERO_BLOCKS) {
+		retval = ext2fs_zero_blocks2(fs, *ret, len, NULL, NULL);
+		if (retval)
+			return retval;
+	}
+
+	ext2fs_block_alloc_stats_range(fs, *ret, len, +1);
+	return retval;
+}
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 7941c5d..91e1bef 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -698,6 +698,17 @@ extern void ext2fs_set_alloc_block_callback(ext2_filsys fs,
 							      blk64_t goal,
 							      blk64_t *ret));
 blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino);
+#define EXT2_NEWRANGE_FIXED_GOAL	(0x1)
+#define EXT2_NEWRANGE_MIN_LENGTH	(0x2)
+#define EXT2_NEWRANGE_ALL_FLAGS		(0x3)
+errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
+			   blk64_t len, ext2fs_block_bitmap map, blk64_t *pblk,
+			   blk64_t *plen);
+#define EXT2_ALLOCRANGE_FIXED_GOAL	(0x1)
+#define EXT2_ALLOCRANGE_ZERO_BLOCKS	(0x2)
+#define EXT2_ALLOCRANGE_ALL_FLAGS	(0x3)
+errcode_t ext2fs_alloc_range(ext2_filsys fs, int flags, blk64_t goal,
+			     blk_t len, blk64_t *ret);
 
 /* alloc_sb.c */
 extern int ext2fs_reserve_super_and_bgd(ext2_filsys fs,

[PATCH 31/39] libext2fs: add new hooks to support large allocations

[Darrick J. Wong]

Add a new get_alloc_blocks hook and a block_alloc_stats_range hook so
that e2fsck can capture allocation requests spanning more than a
block to its block_found_map.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 e2fsck/pass1.c           |   45 +++++++++++++++++++++++++++++++++++++++++++++
 lib/ext2fs/alloc.c       |   37 ++++++++++++++++++++++++++++++++++++-
 lib/ext2fs/alloc_stats.c |   16 ++++++++++++++++
 lib/ext2fs/ext2fs.h      |   16 ++++++++++++++++
 4 files changed, 113 insertions(+), 1 deletion(-)


diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c
index a963849..2d59f63 100644
--- a/e2fsck/pass1.c
+++ b/e2fsck/pass1.c
@@ -3802,6 +3802,26 @@ static errcode_t e2fsck_get_alloc_block(ext2_filsys fs, blk64_t goal,
 	return (0);
 }
 
+static errcode_t e2fsck_new_range(ext2_filsys fs, int flags, blk64_t goal,
+				  blk64_t len, blk64_t *pblk, blk64_t *plen)
+{
+	e2fsck_t ctx = (e2fsck_t) fs->priv_data;
+	errcode_t	retval;
+
+	if (ctx->block_found_map)
+		return ext2fs_new_range(fs, flags, goal, len,
+					ctx->block_found_map, pblk, plen);
+
+	if (!fs->block_map) {
+		retval = ext2fs_read_block_bitmap(fs);
+		if (retval)
+			return retval;
+	}
+
+	return ext2fs_new_range(fs, flags, goal, len, fs->block_map,
+				pblk, plen);
+}
+
 static void e2fsck_block_alloc_stats(ext2_filsys fs, blk64_t blk, int inuse)
 {
 	e2fsck_t ctx = (e2fsck_t) fs->priv_data;
@@ -3821,6 +3841,28 @@ static void e2fsck_block_alloc_stats(ext2_filsys fs, blk64_t blk, int inuse)
 	}
 }
 
+static void e2fsck_block_alloc_stats_range(ext2_filsys fs, blk64_t blk,
+					   blk_t num, int inuse)
+{
+	e2fsck_t ctx = (e2fsck_t) fs->priv_data;
+
+	/* Never free a critical metadata block */
+	if (ctx->block_found_map &&
+	    ctx->block_metadata_map &&
+	    inuse < 0 &&
+	    ext2fs_test_block_bitmap_range2(ctx->block_metadata_map, blk, num))
+		return;
+
+	if (ctx->block_found_map) {
+		if (inuse > 0)
+			ext2fs_mark_block_bitmap_range2(ctx->block_found_map,
+							blk, num);
+		else
+			ext2fs_unmark_block_bitmap_range2(ctx->block_found_map,
+							blk, num);
+	}
+}
+
 void e2fsck_use_inode_shortcuts(e2fsck_t ctx, int use_shortcuts)
 {
 	ext2_filsys fs = ctx->fs;
@@ -3844,4 +3886,7 @@ void e2fsck_intercept_block_allocations(e2fsck_t ctx)
 	ext2fs_set_alloc_block_callback(ctx->fs, e2fsck_get_alloc_block, 0);
 	ext2fs_set_block_alloc_stats_callback(ctx->fs,
 						e2fsck_block_alloc_stats, 0);
+	ext2fs_set_new_range_callback(ctx->fs, e2fsck_new_range, NULL);
+	ext2fs_set_block_alloc_stats_range_callback(ctx->fs,
+					e2fsck_block_alloc_stats_range, NULL);
 }
diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
index 45791ff..8070801 100644
--- a/lib/ext2fs/alloc.c
+++ b/lib/ext2fs/alloc.c
@@ -346,12 +346,32 @@ errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
 	blk64_t start, end, b;
 	int looped = 0;
 	blk64_t max_blocks = ext2fs_blocks_count(fs->super);
+	errcode_t (*nrf)(ext2_filsys fs, int flags, blk64_t goal,
+			 blk64_t len, blk64_t *pblk, blk64_t *plen);
 
 	dbg_printf("%s: flags=0x%x goal=%llu len=%llu\n", __func__, flags,
 		   goal, len);
 	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
 	if (len == 0 || (flags & ~EXT2_NEWRANGE_ALL_FLAGS))
 		return EXT2_ET_INVALID_ARGUMENT;
+
+	if (!map && fs->new_range) {
+		/*
+		 * In case there are clients out there whose new_range
+		 * handlers call ext2fs_new_range with a NULL block map,
+		 * temporarily swap out the function pointer so that we don't
+		 * end up in an infinite loop.
+		 */
+		nrf = fs->new_range;
+		fs->new_range = NULL;
+		retval = nrf(fs, flags, goal, len, pblk, plen);
+		fs->new_range = nrf;
+		if (retval)
+			return retval;
+		start = *pblk;
+		end = *pblk + *plen;
+		goto allocated;
+	}
 	if (!map)
 		map = fs->block_map;
 	if (!map)
@@ -399,7 +419,7 @@ errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
 				   "blk=%llu--%llu %llu\n",
 				   __func__, goal, goal + len - 1,
 				   *pblk, *pblk + *plen - 1, *plen);
-
+allocated:
 			for (b = start; b < end;
 			     b += fs->super->s_blocks_per_group)
 				clear_block_uninit(fs,
@@ -424,6 +444,21 @@ errout:
 	return retval;
 }
 
+void ext2fs_set_new_range_callback(ext2_filsys fs,
+	errcode_t (*func)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen),
+	errcode_t (**old)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen))
+{
+	if (!fs || fs->magic != EXT2_ET_MAGIC_EXT2FS_FILSYS)
+		return;
+
+	if (old)
+		*old = fs->new_range;
+
+	fs->new_range = func;
+}
+
 errcode_t ext2fs_alloc_range(ext2_filsys fs, int flags, blk64_t goal,
 			     blk_t len, blk64_t *ret)
 {
diff --git a/lib/ext2fs/alloc_stats.c b/lib/ext2fs/alloc_stats.c
index 3d3697c..5999082 100644
--- a/lib/ext2fs/alloc_stats.c
+++ b/lib/ext2fs/alloc_stats.c
@@ -145,4 +145,20 @@ void ext2fs_block_alloc_stats_range(ext2_filsys fs, blk64_t blk,
 	}
 	ext2fs_mark_super_dirty(fs);
 	ext2fs_mark_bb_dirty(fs);
+	if (fs->block_alloc_stats_range)
+		(fs->block_alloc_stats_range)(fs, blk, num, inuse);
+}
+
+void ext2fs_set_block_alloc_stats_range_callback(ext2_filsys fs,
+	void (*func)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse),
+	void (**old)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse))
+{
+	if (!fs || fs->magic != EXT2_ET_MAGIC_EXT2FS_FILSYS)
+		return;
+	if (old)
+		*old = fs->block_alloc_stats_range;
+
+	fs->block_alloc_stats_range = func;
 }
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 91e1bef..b1169cd 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -285,6 +285,12 @@ struct struct_ext2_filsys {
 
 	io_channel			journal_io;
 	char				*journal_name;
+
+	/* New block range allocation hooks */
+	errcode_t (*new_range)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen);
+	void (*block_alloc_stats_range)(ext2_filsys fs, blk64_t blk, blk_t num,
+					int inuse);
 };
 
 #if EXT2_FLAT_INCLUDES
@@ -697,6 +703,16 @@ extern void ext2fs_set_alloc_block_callback(ext2_filsys fs,
 					    errcode_t (**old)(ext2_filsys fs,
 							      blk64_t goal,
 							      blk64_t *ret));
+extern void ext2fs_set_new_range_callback(ext2_filsys fs,
+	errcode_t (*func)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen),
+	errcode_t (**old)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen));
+extern void ext2fs_set_block_alloc_stats_range_callback(ext2_filsys fs,
+	void (*func)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse),
+	void (**old)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse));
 blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino);
 #define EXT2_NEWRANGE_FIXED_GOAL	(0x1)
 #define EXT2_NEWRANGE_MIN_LENGTH	(0x2)

[PATCH 32/39] libext2fs: implement fallocate

[Darrick J. Wong]

Create a library function to perform fallocation on arbitrary files,
and wire up a few users for this function.  This is a bit more intense
than Ted's original mk_hugefiles implementation since we have to honor
any blocks that may already be allocated to the file.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/Makefile.in |    8 
 lib/ext2fs/bmap.c      |    2 
 lib/ext2fs/ext2fs.h    |   10 +
 lib/ext2fs/fallocate.c |  853 ++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 872 insertions(+), 1 deletion(-)
 create mode 100644 lib/ext2fs/fallocate.c


diff --git a/lib/ext2fs/Makefile.in b/lib/ext2fs/Makefile.in
index dc3e144..cd7dcc5 100644
--- a/lib/ext2fs/Makefile.in
+++ b/lib/ext2fs/Makefile.in
@@ -79,6 +79,7 @@ OBJS= $(DEBUGFS_LIB_OBJS) $(RESIZE_LIB_OBJS) $(E2IMAGE_LIB_OBJS) \
 	expanddir.o \
 	ext_attr.o \
 	extent.o \
+	fallocate.o \
 	fileio.o \
 	finddev.o \
 	flushb.o \
@@ -763,6 +764,13 @@ extent.o: $(srcdir)/extent.c $(top_builddir)/lib/config.h \
  $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
  $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/ext2_ext_attr.h \
  $(srcdir)/bitops.h $(srcdir)/e2image.h
+fallocate.o: $(srcdir)/fallocate.c $(top_builddir)/lib/config.h \
+ $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fsP.h \
+ $(srcdir)/ext2fs.h $(srcdir)/ext2_fs.h $(srcdir)/ext3_extents.h \
+ $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
+ $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/ext2_ext_attr.h \
+ $(srcdir)/bitops.h $(srcdir)/e2image.h
 fileio.o: $(srcdir)/fileio.c $(top_builddir)/lib/config.h \
  $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
  $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
diff --git a/lib/ext2fs/bmap.c b/lib/ext2fs/bmap.c
index b0d9087..0cb3cc8 100644
--- a/lib/ext2fs/bmap.c
+++ b/lib/ext2fs/bmap.c
@@ -67,7 +67,7 @@ static _BMAP_INLINE_ errcode_t block_ind_bmap(ext2_filsys fs, int flags,
 #endif
 
 	if (!b && (flags & BMAP_ALLOC)) {
-		b = nr ? ((blk_t *) block_buf)[nr-1] : 0;
+		b = nr ? ((blk_t *) block_buf)[nr-1] : ind;
 		retval = ext2fs_alloc_block(fs, b,
 					    block_buf + fs->blocksize, &b);
 		if (retval)
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index b1169cd..1b97c93 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1259,6 +1259,16 @@ extern errcode_t ext2fs_extent_goto2(ext2_extent_handle_t handle,
 				     int leaf_level, blk64_t blk);
 extern errcode_t ext2fs_extent_fix_parents(ext2_extent_handle_t handle);
 
+/* fallocate.c */
+#define EXT2_FALLOCATE_ZERO_BLOCKS	(0x1)
+#define EXT2_FALLOCATE_FORCE_INIT	(0x2)
+#define EXT2_FALLOCATE_FORCE_UNINIT	(0x4)
+#define EXT2_FALLOCATE_INIT_BEYOND_EOF	(0x8)
+#define EXT2_FALLOCATE_ALL_FLAGS	(0xF)
+errcode_t ext2fs_fallocate(ext2_filsys fs, int flags, ext2_ino_t ino,
+			   struct ext2_inode *inode, blk64_t goal,
+			   blk64_t start, blk64_t len);
+
 /* fileio.c */
 extern errcode_t ext2fs_file_open2(ext2_filsys fs, ext2_ino_t ino,
 				   struct ext2_inode *inode,
diff --git a/lib/ext2fs/fallocate.c b/lib/ext2fs/fallocate.c
new file mode 100644
index 0000000..bf453e1
--- /dev/null
+++ b/lib/ext2fs/fallocate.c
@@ -0,0 +1,853 @@
+/*
+ * fallocate.c -- Allocate large chunks of file.
+ *
+ * Copyright (C) 2014 Oracle.
+ *
+ * %Begin-Header%
+ * This file may be redistributed under the terms of the GNU Library
+ * General Public License, version 2.
+ * %End-Header%
+ */
+
+#include "config.h"
+
+#include "ext2_fs.h"
+#include "ext2fs.h"
+#define min(a, b) ((a) < (b) ? (a) : (b))
+
+#undef DEBUG
+
+#ifdef DEBUG
+# define dbg_printf(f, a...)  do {printf(f, ## a); fflush(stdout); } while (0)
+#else
+# define dbg_printf(f, a...)
+#endif
+
+/*
+ * Extent-based fallocate code.
+ *
+ * Find runs of unmapped logical blocks by starting at start and walking the
+ * extents until we reach the end of the range we want.
+ *
+ * For each run of unmapped blocks, try to find the extents on either side of
+ * the range.  If there's a left extent that can grow by at least a cluster and
+ * there are lblocks between start and the next lcluster after start, see if
+ * there's an implied cluster allocation; if so, zero the blocks (if the left
+ * extent is initialized) and adjust the extent.  Ditto for the blocks between
+ * the end of the last full lcluster and end, if there's a right extent.
+ *
+ * Try to attach as much as we can to the left extent, then try to attach as
+ * much as we can to the right extent.  For the remainder, try to allocate the
+ * whole range; map in whatever we get; and repeat until we're done.
+ *
+ * To attach to a left extent, figure out the maximum amount we can add to the
+ * extent and try to allocate that much, and append if successful.  To attach
+ * to a right extent, figure out the max we can add to the extent, try to
+ * allocate that much, and prepend if successful.
+ *
+ * We need an alloc_range function that tells us how much we can allocate given
+ * a maximum length and one of a suggested start, a fixed start, or a fixed end
+ * point.
+ *
+ * Every time we modify the extent tree we also need to update the block stats.
+ *
+ * At the end, update i_blocks and i_size appropriately.
+ */
+
+static void dbg_print_extent(char *desc, struct ext2fs_extent *extent)
+{
+#ifdef DEBUG
+	if (desc)
+		printf("%s: ", desc);
+	printf("extent: lblk %llu--%llu, len %u, pblk %llu, flags: ",
+	       extent->e_lblk, extent->e_lblk + extent->e_len - 1,
+	       extent->e_len, extent->e_pblk);
+	if (extent->e_flags & EXT2_EXTENT_FLAGS_LEAF)
+		fputs("LEAF ", stdout);
+	if (extent->e_flags & EXT2_EXTENT_FLAGS_UNINIT)
+		fputs("UNINIT ", stdout);
+	if (extent->e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT)
+		fputs("2ND_VISIT ", stdout);
+	if (!extent->e_flags)
+		fputs("(none)", stdout);
+	fputc('\n', stdout);
+	fflush(stdout);
+#endif
+}
+
+static errcode_t claim_range(ext2_filsys fs, struct ext2_inode *inode,
+			     blk64_t blk, blk64_t len)
+{
+	blk64_t	clusters;
+
+	clusters = (len + EXT2FS_CLUSTER_RATIO(fs) - 1) /
+		   EXT2FS_CLUSTER_RATIO(fs);
+	ext2fs_block_alloc_stats_range(fs, blk,
+			clusters * EXT2FS_CLUSTER_RATIO(fs), +1);
+	return ext2fs_iblk_add_blocks(fs, inode, clusters);
+}
+
+static errcode_t ext_falloc_helper(ext2_filsys fs,
+				   int flags,
+				   ext2_ino_t ino,
+				   struct ext2_inode *inode,
+				   ext2_extent_handle_t handle,
+				   struct ext2fs_extent *left_ext,
+				   struct ext2fs_extent *right_ext,
+				   blk64_t range_start, blk64_t range_len,
+				   blk64_t alloc_goal)
+{
+	struct ext2fs_extent	newex, ex;
+	int			op;
+	blk64_t			fillable, pblk, plen, x, y;
+	blk64_t			eof_blk = 0, cluster_fill = 0;
+	errcode_t		err;
+	blk_t			max_extent_len, max_uninit_len, max_init_len;
+
+#ifdef DEBUG
+	printf("%s: ", __func__);
+	if (left_ext)
+		printf("left_ext=%llu--%llu, ", left_ext->e_lblk,
+		       left_ext->e_lblk + left_ext->e_len - 1);
+	if (right_ext)
+		printf("right_ext=%llu--%llu, ", right_ext->e_lblk,
+		       right_ext->e_lblk + right_ext->e_len - 1);
+	printf("start=%llu len=%llu, goal=%llu\n", range_start, range_len,
+	       alloc_goal);
+	fflush(stdout);
+#endif
+	/* Can't create initialized extents past EOF? */
+	if (!(flags & EXT2_FALLOCATE_INIT_BEYOND_EOF))
+		eof_blk = EXT2_I_SIZE(inode) / fs->blocksize;
+
+	/* The allocation goal must be as far into a cluster as range_start. */
+	alloc_goal = (alloc_goal & ~EXT2FS_CLUSTER_MASK(fs)) |
+		     (range_start & EXT2FS_CLUSTER_MASK(fs));
+
+	max_uninit_len = EXT_UNINIT_MAX_LEN & ~EXT2FS_CLUSTER_MASK(fs);
+	max_init_len = EXT_INIT_MAX_LEN & ~EXT2FS_CLUSTER_MASK(fs);
+
+	/* We must lengthen the left extent to the end of the cluster */
+	if (left_ext && EXT2FS_CLUSTER_RATIO(fs) > 1) {
+		/* How many more blocks can be attached to left_ext? */
+		if (left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)
+			fillable = max_uninit_len - left_ext->e_len;
+		else
+			fillable = max_init_len - left_ext->e_len;
+
+		if (fillable > range_len)
+			fillable = range_len;
+		if (fillable == 0)
+			goto expand_right;
+
+		/*
+		 * If range_start isn't on a cluster boundary, try an
+		 * implied cluster allocation for left_ext.
+		 */
+		cluster_fill = EXT2FS_CLUSTER_RATIO(fs) -
+			       (range_start & EXT2FS_CLUSTER_MASK(fs));
+		cluster_fill &= EXT2FS_CLUSTER_MASK(fs);
+		if (cluster_fill == 0)
+			goto expand_right;
+
+		if (cluster_fill > fillable)
+			cluster_fill = fillable;
+
+		/* Don't expand an initialized left_ext beyond EOF */
+		if (!(flags & EXT2_FALLOCATE_INIT_BEYOND_EOF)) {
+			x = left_ext->e_lblk + left_ext->e_len - 1;
+			dbg_printf("%s: lend=%llu newlend=%llu eofblk=%llu\n",
+				   __func__, x, x + cluster_fill, eof_blk);
+			if (eof_blk >= x && eof_blk <= x + cluster_fill)
+				cluster_fill = eof_blk - x;
+			if (cluster_fill == 0)
+				goto expand_right;
+		}
+
+		err = ext2fs_extent_goto(handle, left_ext->e_lblk);
+		if (err)
+			goto expand_right;
+		left_ext->e_len += cluster_fill;
+		range_start += cluster_fill;
+		range_len -= cluster_fill;
+		alloc_goal += cluster_fill;
+
+		dbg_print_extent("ext_falloc clus left+", left_ext);
+		err = ext2fs_extent_replace(handle, 0, left_ext);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		/* Zero blocks */
+		if (!(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)) {
+			err = ext2fs_zero_blocks2(fs, left_ext->e_pblk +
+						  left_ext->e_len -
+						  cluster_fill, cluster_fill,
+						  NULL, NULL);
+			if (err)
+				goto out;
+		}
+	}
+
+expand_right:
+	/* We must lengthen the right extent to the beginning of the cluster */
+	if (right_ext && EXT2FS_CLUSTER_RATIO(fs) > 1) {
+		/* How much can we attach to right_ext? */
+		if (right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)
+			fillable = max_uninit_len - right_ext->e_len;
+		else
+			fillable = max_init_len - right_ext->e_len;
+
+		if (fillable > range_len)
+			fillable = range_len;
+		if (fillable == 0)
+			goto try_merge;
+
+		/*
+		 * If range_end isn't on a cluster boundary, try an implied
+		 * cluster allocation for right_ext.
+		 */
+		cluster_fill = right_ext->e_lblk & EXT2FS_CLUSTER_MASK(fs);
+		if (cluster_fill == 0)
+			goto try_merge;
+
+		err = ext2fs_extent_goto(handle, right_ext->e_lblk);
+		if (err)
+			goto out;
+
+		if (cluster_fill > fillable)
+			cluster_fill = fillable;
+		right_ext->e_lblk -= cluster_fill;
+		right_ext->e_pblk -= cluster_fill;
+		right_ext->e_len += cluster_fill;
+		range_len -= cluster_fill;
+
+		dbg_print_extent("ext_falloc clus right+", right_ext);
+		err = ext2fs_extent_replace(handle, 0, right_ext);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		/* Zero blocks if necessary */
+		if (!(right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)) {
+			err = ext2fs_zero_blocks2(fs, right_ext->e_pblk,
+						  cluster_fill, NULL, NULL);
+			if (err)
+				goto out;
+		}
+	}
+
+try_merge:
+	/* Merge both extents together, perhaps? */
+	if (left_ext && right_ext) {
+		/* Are the two extents mergeable? */
+		if ((left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT) !=
+		    (right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT))
+			goto try_left;
+
+		/* User requires init/uninit but extent is uninit/init. */
+		if (((flags & EXT2_FALLOCATE_FORCE_INIT) &&
+		     (left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)) ||
+		    ((flags & EXT2_FALLOCATE_FORCE_UNINIT) &&
+		     !(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)))
+			goto try_left;
+
+		/*
+		 * Skip initialized extent unless user wants to zero blocks
+		 * or requires init extent.
+		 */
+		if (!(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (!(flags & EXT2_FALLOCATE_ZERO_BLOCKS) ||
+		     !(flags & EXT2_FALLOCATE_FORCE_INIT)))
+			goto try_left;
+
+		/* Will it even fit? */
+		x = left_ext->e_len + range_len + right_ext->e_len;
+		if (x > (left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT ?
+				max_uninit_len : max_init_len))
+			goto try_left;
+
+		err = ext2fs_extent_goto(handle, left_ext->e_lblk);
+		if (err)
+			goto try_left;
+
+		/* Allocate blocks */
+		y = left_ext->e_pblk + left_ext->e_len;
+		err = ext2fs_new_range(fs, EXT2_NEWRANGE_FIXED_GOAL |
+				       EXT2_NEWRANGE_MIN_LENGTH, y,
+				       right_ext->e_pblk - y + 1, NULL,
+				       &pblk, &plen);
+		if (err)
+			goto try_left;
+		if (pblk + plen != right_ext->e_pblk)
+			goto try_left;
+		err = claim_range(fs, inode, pblk, plen);
+		if (err)
+			goto out;
+
+		/* Modify extents */
+		left_ext->e_len = x;
+		dbg_print_extent("ext_falloc merge", left_ext);
+		err = ext2fs_extent_replace(handle, 0, left_ext);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+		err = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT_LEAF, &newex);
+		if (err)
+			goto out;
+		err = ext2fs_extent_delete(handle, 0);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+		*right_ext = *left_ext;
+
+		/* Zero blocks */
+		if (!(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, range_start, range_len,
+						  NULL, NULL);
+			if (err)
+				goto out;
+		}
+
+		return 0;
+	}
+
+try_left:
+	/* Extend the left extent */
+	if (left_ext) {
+		/* How many more blocks can be attached to left_ext? */
+		if (left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)
+			fillable = max_uninit_len - left_ext->e_len;
+		else if (flags & EXT2_FALLOCATE_ZERO_BLOCKS)
+			fillable = max_init_len - left_ext->e_len;
+		else
+			fillable = 0;
+
+		/* User requires init/uninit but extent is uninit/init. */
+		if (((flags & EXT2_FALLOCATE_FORCE_INIT) &&
+		     (left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)) ||
+		    ((flags & EXT2_FALLOCATE_FORCE_UNINIT) &&
+		     !(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)))
+			goto try_right;
+
+		if (fillable > range_len)
+			fillable = range_len;
+
+		/* Don't expand an initialized left_ext beyond EOF */
+		x = left_ext->e_lblk + left_ext->e_len - 1;
+		if (!(flags & EXT2_FALLOCATE_INIT_BEYOND_EOF)) {
+			dbg_printf("%s: lend=%llu newlend=%llu eofblk=%llu\n",
+				   __func__, x, x + fillable, eof_blk);
+			if (eof_blk >= x && eof_blk <= x + fillable)
+				fillable = eof_blk - x;
+		}
+
+		if (fillable == 0)
+			goto try_right;
+
+		/* Test if the right edge of the range is already mapped? */
+		if (EXT2FS_CLUSTER_RATIO(fs) > 1) {
+			err = ext2fs_map_cluster_block(fs, ino, inode,
+					x + fillable, &pblk);
+			if (err)
+				goto out;
+			if (pblk)
+				fillable -= 1 + ((x + fillable)
+						 & EXT2FS_CLUSTER_MASK(fs));
+			if (fillable == 0)
+				goto try_right;
+		}
+
+		/* Allocate range of blocks */
+		x = left_ext->e_pblk + left_ext->e_len;
+		err = ext2fs_new_range(fs, EXT2_NEWRANGE_FIXED_GOAL |
+				EXT2_NEWRANGE_MIN_LENGTH,
+				x, fillable, NULL, &pblk, &plen);
+		if (err)
+			goto try_right;
+		err = claim_range(fs, inode, pblk, plen);
+		if (err)
+			goto out;
+
+		/* Modify left_ext */
+		err = ext2fs_extent_goto(handle, left_ext->e_lblk);
+		if (err)
+			goto out;
+		range_start += plen;
+		range_len -= plen;
+		left_ext->e_len += plen;
+		dbg_print_extent("ext_falloc left+", left_ext);
+		err = ext2fs_extent_replace(handle, 0, left_ext);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		/* Zero blocks if necessary */
+		if (!(left_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, pblk, plen, NULL, NULL);
+			if (err)
+				goto out;
+		}
+	}
+
+try_right:
+	/* Extend the right extent */
+	if (right_ext) {
+		/* How much can we attach to right_ext? */
+		if (right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)
+			fillable = max_uninit_len - right_ext->e_len;
+		else if (flags & EXT2_FALLOCATE_ZERO_BLOCKS)
+			fillable = max_init_len - right_ext->e_len;
+		else
+			fillable = 0;
+
+		/* User requires init/uninit but extent is uninit/init. */
+		if (((flags & EXT2_FALLOCATE_FORCE_INIT) &&
+		     (right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)) ||
+		    ((flags & EXT2_FALLOCATE_FORCE_UNINIT) &&
+		     !(right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT)))
+			goto try_anywhere;
+
+		if (fillable > range_len)
+			fillable = range_len;
+		if (fillable == 0)
+			goto try_anywhere;
+
+		/* Test if the left edge of the range is already mapped? */
+		if (EXT2FS_CLUSTER_RATIO(fs) > 1) {
+			err = ext2fs_map_cluster_block(fs, ino, inode,
+					right_ext->e_lblk - fillable, &pblk);
+			if (err)
+				goto out;
+			if (pblk)
+				fillable -= EXT2FS_CLUSTER_RATIO(fs) -
+						((right_ext->e_lblk - fillable)
+						 & EXT2FS_CLUSTER_MASK(fs));
+			if (fillable == 0)
+				goto try_anywhere;
+		}
+
+		/*
+		 * FIXME: It would be nice if we could handle allocating a
+		 * variable range from a fixed end point instead of just
+		 * skipping to the general allocator if the whole range is
+		 * unavailable.
+		 */
+		err = ext2fs_new_range(fs, EXT2_NEWRANGE_FIXED_GOAL |
+				EXT2_NEWRANGE_MIN_LENGTH,
+				right_ext->e_pblk - fillable,
+				fillable, NULL, &pblk, &plen);
+		if (err)
+			goto try_anywhere;
+		err = claim_range(fs, inode,
+			      pblk & ~EXT2FS_CLUSTER_MASK(fs),
+			      plen + (pblk & EXT2FS_CLUSTER_MASK(fs)));
+		if (err)
+			goto out;
+
+		/* Modify right_ext */
+		err = ext2fs_extent_goto(handle, right_ext->e_lblk);
+		if (err)
+			goto out;
+		range_len -= plen;
+		right_ext->e_lblk -= plen;
+		right_ext->e_pblk -= plen;
+		right_ext->e_len += plen;
+		dbg_print_extent("ext_falloc right+", right_ext);
+		err = ext2fs_extent_replace(handle, 0, right_ext);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		/* Zero blocks if necessary */
+		if (!(right_ext->e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, pblk,
+					plen + cluster_fill, NULL, NULL);
+			if (err)
+				goto out;
+		}
+	}
+
+try_anywhere:
+	/* Try implied cluster alloc on the left and right ends */
+	if (range_len > 0 && (range_start & EXT2FS_CLUSTER_MASK(fs))) {
+		cluster_fill = EXT2FS_CLUSTER_RATIO(fs) -
+			       (range_start & EXT2FS_CLUSTER_MASK(fs));
+		cluster_fill &= EXT2FS_CLUSTER_MASK(fs);
+		if (cluster_fill > range_len)
+			cluster_fill = range_len;
+		newex.e_lblk = range_start;
+		err = ext2fs_map_cluster_block(fs, ino, inode, newex.e_lblk,
+					       &pblk);
+		if (err)
+			goto out;
+		if (pblk == 0)
+			goto try_right_implied;
+		newex.e_pblk = pblk;
+		newex.e_len = cluster_fill;
+		newex.e_flags = (flags & EXT2_FALLOCATE_FORCE_INIT ? 0 :
+				 EXT2_EXTENT_FLAGS_UNINIT);
+		dbg_print_extent("ext_falloc iclus left+", &newex);
+		ext2fs_extent_goto(handle, newex.e_lblk);
+		err = ext2fs_extent_get(handle, EXT2_EXTENT_CURRENT,
+					&ex);
+		if (err == EXT2_ET_NO_CURRENT_NODE)
+			ex.e_lblk = 0;
+		else if (err)
+			goto out;
+
+		if (ex.e_lblk > newex.e_lblk)
+			op = 0; /* insert before */
+		else
+			op = EXT2_EXTENT_INSERT_AFTER;
+		dbg_printf("%s: inserting %s lblk %llu newex=%llu\n",
+			   __func__, op ? "after" : "before", ex.e_lblk,
+			   newex.e_lblk);
+		err = ext2fs_extent_insert(handle, op, &newex);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		if (!(newex.e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, newex.e_pblk,
+						  newex.e_len, NULL, NULL);
+			if (err)
+				goto out;
+		}
+
+		range_start += cluster_fill;
+		range_len -= cluster_fill;
+	}
+
+try_right_implied:
+	y = range_start + range_len;
+	if (range_len > 0 && (y & EXT2FS_CLUSTER_MASK(fs))) {
+		cluster_fill = y & EXT2FS_CLUSTER_MASK(fs);
+		if (cluster_fill > range_len)
+			cluster_fill = range_len;
+		newex.e_lblk = y & ~EXT2FS_CLUSTER_MASK(fs);
+		err = ext2fs_map_cluster_block(fs, ino, inode, newex.e_lblk,
+					       &pblk);
+		if (err)
+			goto out;
+		if (pblk == 0)
+			goto no_implied;
+		newex.e_pblk = pblk;
+		newex.e_len = cluster_fill;
+		newex.e_flags = (flags & EXT2_FALLOCATE_FORCE_INIT ? 0 :
+				 EXT2_EXTENT_FLAGS_UNINIT);
+		dbg_print_extent("ext_falloc iclus right+", &newex);
+		ext2fs_extent_goto(handle, newex.e_lblk);
+		err = ext2fs_extent_get(handle, EXT2_EXTENT_CURRENT,
+					&ex);
+		if (err == EXT2_ET_NO_CURRENT_NODE)
+			ex.e_lblk = 0;
+		else if (err)
+			goto out;
+
+		if (ex.e_lblk > newex.e_lblk)
+			op = 0; /* insert before */
+		else
+			op = EXT2_EXTENT_INSERT_AFTER;
+		dbg_printf("%s: inserting %s lblk %llu newex=%llu\n",
+			   __func__, op ? "after" : "before", ex.e_lblk,
+			   newex.e_lblk);
+		err = ext2fs_extent_insert(handle, op, &newex);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		if (!(newex.e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, newex.e_pblk,
+						  newex.e_len, NULL, NULL);
+			if (err)
+				goto out;
+		}
+
+		range_len -= cluster_fill;
+	}
+
+no_implied:
+	if (range_len == 0)
+		return 0;
+
+	newex.e_lblk = range_start;
+	if (flags & EXT2_FALLOCATE_FORCE_INIT) {
+		max_extent_len = max_init_len;
+		newex.e_flags = 0;
+	} else {
+		max_extent_len = max_uninit_len;
+		newex.e_flags = EXT2_EXTENT_FLAGS_UNINIT;
+	}
+	pblk = alloc_goal;
+	y = range_len;
+	for (x = 0; x < y;) {
+		cluster_fill = newex.e_lblk & EXT2FS_CLUSTER_MASK(fs);
+		fillable = min(range_len + cluster_fill, max_extent_len);
+		err = ext2fs_new_range(fs, 0, pblk & ~EXT2FS_CLUSTER_MASK(fs),
+				       fillable,
+				       NULL, &pblk, &plen);
+		if (err)
+			goto out;
+		err = claim_range(fs, inode, pblk, plen);
+		if (err)
+			goto out;
+
+		/* Create extent */
+		newex.e_pblk = pblk + cluster_fill;
+		newex.e_len = plen - cluster_fill;
+		dbg_print_extent("ext_falloc create", &newex);
+		ext2fs_extent_goto(handle, newex.e_lblk);
+		err = ext2fs_extent_get(handle, EXT2_EXTENT_CURRENT,
+					&ex);
+		if (err == EXT2_ET_NO_CURRENT_NODE)
+			ex.e_lblk = 0;
+		else if (err)
+			goto out;
+
+		if (ex.e_lblk > newex.e_lblk)
+			op = 0; /* insert before */
+		else
+			op = EXT2_EXTENT_INSERT_AFTER;
+		dbg_printf("%s: inserting %s lblk %llu newex=%llu\n",
+			   __func__, op ? "after" : "before", ex.e_lblk,
+			   newex.e_lblk);
+		err = ext2fs_extent_insert(handle, op, &newex);
+		if (err)
+			goto out;
+		err = ext2fs_extent_fix_parents(handle);
+		if (err)
+			goto out;
+
+		if (!(newex.e_flags & EXT2_EXTENT_FLAGS_UNINIT) &&
+		    (flags & EXT2_FALLOCATE_ZERO_BLOCKS)) {
+			err = ext2fs_zero_blocks2(fs, pblk, plen, NULL, NULL);
+			if (err)
+				goto out;
+		}
+
+		/* Update variables at end of loop */
+		x += plen - cluster_fill;
+		range_len -= plen - cluster_fill;
+		newex.e_lblk += plen - cluster_fill;
+		pblk += plen - cluster_fill;
+		if (pblk >= ext2fs_blocks_count(fs->super))
+			pblk = fs->super->s_first_data_block;
+	}
+
+out:
+	return err;
+}
+
+static errcode_t extent_fallocate(ext2_filsys fs, int flags, ext2_ino_t ino,
+				      struct ext2_inode *inode, blk64_t goal,
+				      blk64_t start, blk64_t len)
+{
+	ext2_extent_handle_t	handle;
+	struct ext2fs_extent	left_extent, right_extent;
+	struct ext2fs_extent	*left_adjacent, *right_adjacent;
+	errcode_t		err;
+	blk64_t			range_start, range_end = 0, end, next;
+	blk64_t			count, goal_distance;
+
+	end = start + len - 1;
+	err = ext2fs_extent_open2(fs, ino, inode, &handle);
+	if (err)
+		return err;
+
+	/*
+	 * Find the extent closest to the start of the alloc range.  We don't
+	 * check the return value because _goto() sets the current node to the
+	 * next-lowest extent if 'start' is in a hole; or the next-highest
+	 * extent if there aren't any lower ones; or doesn't set a current node
+	 * if there was a real error reading the extent tree.  In that case,
+	 * _get() will error out.
+	 */
+start_again:
+	ext2fs_extent_goto(handle, start);
+	err = ext2fs_extent_get(handle, EXT2_EXTENT_CURRENT, &left_extent);
+	if (err == EXT2_ET_NO_CURRENT_NODE) {
+		blk64_t max_blocks = ext2fs_blocks_count(fs->super);
+
+		if (goal == ~0ULL)
+			goal = ext2fs_find_inode_goal(fs, ino);
+		err = ext2fs_find_first_zero_block_bitmap2(fs->block_map,
+						goal, max_blocks - 1, &goal);
+		goal += start;
+		err = ext_falloc_helper(fs, flags, ino, inode, handle, NULL,
+					NULL, start, len, goal);
+		goto errout;
+	} else if (err)
+		goto errout;
+
+	dbg_print_extent("ext_falloc initial", &left_extent);
+	next = left_extent.e_lblk + left_extent.e_len;
+	if (left_extent.e_lblk > start) {
+		/* The nearest extent we found was beyond start??? */
+		goal = left_extent.e_pblk - (left_extent.e_lblk - start);
+		err = ext_falloc_helper(fs, flags, ino, inode, handle, NULL,
+					&left_extent, start,
+					left_extent.e_lblk - start, goal);
+		if (err)
+			goto errout;
+
+		goto start_again;
+	} else if (next >= start) {
+		range_start = next;
+		left_adjacent = &left_extent;
+	} else {
+		range_start = start;
+		left_adjacent = NULL;
+	}
+	goal = left_extent.e_pblk + (range_start - left_extent.e_lblk);
+	goal_distance = range_start - next;
+
+	do {
+		err = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT_LEAF,
+					   &right_extent);
+		dbg_printf("%s: ino=%d get next =%d\n", __func__, ino,
+			   (int)err);
+		dbg_print_extent("ext_falloc next", &right_extent);
+		/* Stop if we've seen this extent before */
+		if (!err && right_extent.e_lblk <= left_extent.e_lblk)
+			err = EXT2_ET_EXTENT_NO_NEXT;
+
+		if (err && err != EXT2_ET_EXTENT_NO_NEXT)
+			goto errout;
+		if (err == EXT2_ET_EXTENT_NO_NEXT ||
+		    right_extent.e_lblk > end + 1) {
+			range_end = end;
+			right_adjacent = NULL;
+		} else {
+			/* Handle right_extent.e_lblk <= end */
+			range_end = right_extent.e_lblk - 1;
+			right_adjacent = &right_extent;
+		}
+		if (err != EXT2_ET_EXTENT_NO_NEXT &&
+		    goal_distance > (range_end - right_extent.e_lblk)) {
+			goal = right_extent.e_pblk -
+					(right_extent.e_lblk - range_start);
+			goal_distance = range_end - right_extent.e_lblk;
+		}
+
+		dbg_printf("%s: ino=%d rstart=%llu rend=%llu\n", __func__, ino,
+			   range_start, range_end);
+		err = 0;
+		if (range_start <= range_end) {
+			count = range_end - range_start + 1;
+			err = ext_falloc_helper(fs, flags, ino, inode, handle,
+						left_adjacent, right_adjacent,
+						range_start, count, goal);
+			if (err)
+				goto errout;
+		}
+
+		if (range_end == end)
+			break;
+
+		err = ext2fs_extent_goto(handle, right_extent.e_lblk);
+		if (err)
+			goto errout;
+		next = right_extent.e_lblk + right_extent.e_len;
+		left_extent = right_extent;
+		left_adjacent = &left_extent;
+		range_start = next;
+		goal = left_extent.e_pblk + (range_start - left_extent.e_lblk);
+		goal_distance = range_start - next;
+	} while (range_end < end);
+
+errout:
+	ext2fs_extent_free(handle);
+	return err;
+}
+
+/*
+ * Map physical blocks to a range of logical blocks within a file.  The range
+ * of logical blocks are (start, start + len).  If there are already extents,
+ * the mappings will try to extend the mappings; otherwise, it will try to map
+ * start as if logical block 0 points to goal.  If goal is ~0ULL, then the goal
+ * is calculated based on the inode group.
+ *
+ * Flags:
+ * - EXT2_FALLOCATE_ZERO_BLOCKS: Zero the blocks that are allocated.
+ * - EXT2_FALLOCATE_FORCE_INIT: Create only initialized extents.
+ * - EXT2_FALLOCATE_FORCE_UNINIT: Create only uninitialized extents.
+ * - EXT2_FALLOCATE_INIT_BEYOND_EOF: Create extents beyond EOF.
+ *
+ * If neither FORCE_INIT nor FORCE_UNINIT are specified, this function will
+ * try to expand any extents it finds, zeroing blocks as necessary.
+ */
+errcode_t ext2fs_fallocate(ext2_filsys fs, int flags, ext2_ino_t ino,
+			   struct ext2_inode *inode, blk64_t goal,
+			   blk64_t start, blk64_t len)
+{
+	struct ext2_inode	inode_buf;
+	blk64_t			blk, x;
+	errcode_t		err;
+
+	if (((flags & EXT2_FALLOCATE_FORCE_INIT) &&
+	    (flags & EXT2_FALLOCATE_FORCE_UNINIT)) ||
+	   (flags & ~EXT2_FALLOCATE_ALL_FLAGS))
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	if (len > ext2fs_blocks_count(fs->super))
+		return EXT2_ET_BLOCK_ALLOC_FAIL;
+	else if (len == 0)
+		return 0;
+
+	/* Read inode structure if necessary */
+	if (!inode) {
+		err = ext2fs_read_inode(fs, ino, &inode_buf);
+		if (err)
+			return err;
+		inode = &inode_buf;
+	}
+	dbg_printf("%s: ino=%d start=%llu len=%llu goal=%llu\n", __func__, ino,
+		   start, len, goal);
+
+	if (inode->i_flags & EXT4_EXTENTS_FL) {
+		err = extent_fallocate(fs, flags, ino, inode, goal, start, len);
+		goto out;
+	}
+
+	/* XXX: Allocate a bunch of blocks the slow way */
+	for (blk = start; blk < start + len; blk++) {
+		err = ext2fs_bmap2(fs, ino, inode, NULL, 0, blk, 0, &x);
+		if (err)
+			return err;
+		if (x)
+			continue;
+
+		err = ext2fs_bmap2(fs, ino, inode, NULL,
+				   BMAP_ALLOC | BMAP_UNINIT | BMAP_ZERO, blk,
+				   0, &x);
+		if (err)
+			return err;
+	}
+
+out:
+	if (inode == &inode_buf)
+		ext2fs_write_inode(fs, ino, inode);
+	return err;
+}

[PATCH 33/39] libext2fs: use fallocate for creating journals and hugefiles

[Darrick J. Wong]

Use the new fallocate API for creating the journal and the mk_hugefile
feature.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/mkjournal.c |  134 +++++++-----------------------------------------
 misc/mk_hugefiles.c    |   96 ++++------------------------------
 2 files changed, 30 insertions(+), 200 deletions(-)


diff --git a/lib/ext2fs/mkjournal.c b/lib/ext2fs/mkjournal.c
index 2cd7ca5..9378314 100644
--- a/lib/ext2fs/mkjournal.c
+++ b/lib/ext2fs/mkjournal.c
@@ -250,89 +250,6 @@ errcode_t ext2fs_zero_blocks(ext2_filsys fs, blk_t blk, int num,
 }
 
 /*
- * Helper function for creating the journal using direct I/O routines
- */
-struct mkjournal_struct {
-	int		num_blocks;
-	int		newblocks;
-	blk64_t		goal;
-	blk64_t		blk_to_zero;
-	int		zero_count;
-	int		flags;
-	char		*buf;
-	errcode_t	err;
-};
-
-static int mkjournal_proc(ext2_filsys	fs,
-			  blk64_t	*blocknr,
-			  e2_blkcnt_t	blockcnt,
-			  blk64_t	ref_block EXT2FS_ATTR((unused)),
-			  int		ref_offset EXT2FS_ATTR((unused)),
-			  void		*priv_data)
-{
-	struct mkjournal_struct *es = (struct mkjournal_struct *) priv_data;
-	blk64_t	new_blk;
-	errcode_t	retval;
-
-	if (*blocknr) {
-		es->goal = *blocknr;
-		return 0;
-	}
-	if (blockcnt &&
-	    (EXT2FS_B2C(fs, es->goal) == EXT2FS_B2C(fs, es->goal+1)))
-		new_blk = es->goal+1;
-	else {
-		es->goal &= ~EXT2FS_CLUSTER_MASK(fs);
-		retval = ext2fs_new_block2(fs, es->goal, 0, &new_blk);
-		if (retval) {
-			es->err = retval;
-			return BLOCK_ABORT;
-		}
-		ext2fs_block_alloc_stats2(fs, new_blk, +1);
-		es->newblocks++;
-	}
-	if (blockcnt >= 0)
-		es->num_blocks--;
-
-	retval = 0;
-	if (blockcnt <= 0)
-		retval = io_channel_write_blk64(fs->io, new_blk, 1, es->buf);
-	else if (!(es->flags & EXT2_MKJOURNAL_LAZYINIT)) {
-		if (es->zero_count) {
-			if ((es->blk_to_zero + es->zero_count == new_blk) &&
-			    (es->zero_count < 1024))
-				es->zero_count++;
-			else {
-				retval = ext2fs_zero_blocks2(fs,
-							     es->blk_to_zero,
-							     es->zero_count,
-							     0, 0);
-				es->zero_count = 0;
-			}
-		}
-		if (es->zero_count == 0) {
-			es->blk_to_zero = new_blk;
-			es->zero_count = 1;
-		}
-	}
-
-	if (blockcnt == 0)
-		memset(es->buf, 0, fs->blocksize);
-
-	if (retval) {
-		es->err = retval;
-		return BLOCK_ABORT;
-	}
-	*blocknr = es->goal = new_blk;
-
-	if (es->num_blocks == 0)
-		return (BLOCK_CHANGED | BLOCK_ABORT);
-	else
-		return BLOCK_CHANGED;
-
-}
-
-/*
  * Calculate the initial goal block to be roughly at the middle of the
  * filesystem.  Pick a group that has the largest number of free
  * blocks.
@@ -373,7 +290,8 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 	errcode_t		retval;
 	struct ext2_inode	inode;
 	unsigned long long	inode_size;
-	struct mkjournal_struct	es;
+	int			falloc_flags = EXT2_FALLOCATE_FORCE_INIT;
+	blk64_t			zblk;
 
 	if ((retval = ext2fs_create_journal_superblock(fs, num_blocks, flags,
 						       &buf)))
@@ -390,40 +308,16 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 		goto out2;
 	}
 
-	es.num_blocks = num_blocks;
-	es.newblocks = 0;
-	es.buf = buf;
-	es.err = 0;
-	es.flags = flags;
-	es.zero_count = 0;
-	es.goal = (goal != ~0ULL) ? goal : get_midpoint_journal_block(fs);
+	if (goal == ~0ULL)
+		goal = get_midpoint_journal_block(fs);
 
-	if (fs->super->s_feature_incompat & EXT3_FEATURE_INCOMPAT_EXTENTS) {
+	if (fs->super->s_feature_incompat & EXT3_FEATURE_INCOMPAT_EXTENTS)
 		inode.i_flags |= EXT4_EXTENTS_FL;
-		if ((retval = ext2fs_write_inode(fs, journal_ino, &inode)))
-			goto out2;
-	}
 
-	retval = ext2fs_block_iterate3(fs, journal_ino, BLOCK_FLAG_APPEND,
-				       0, mkjournal_proc, &es);
-	if (retval)
-		goto out2;
-	if (es.err) {
-		retval = es.err;
-		goto out2;
-	}
-	if (es.zero_count) {
-		retval = ext2fs_zero_blocks2(fs, es.blk_to_zero,
-					    es.zero_count, 0, 0);
-		if (retval)
-			goto out2;
-	}
-
-	if ((retval = ext2fs_read_inode(fs, journal_ino, &inode)))
-		goto out2;
+	if (!(flags & EXT2_MKJOURNAL_LAZYINIT))
+		falloc_flags |= EXT2_FALLOCATE_ZERO_BLOCKS;
 
 	inode_size = (unsigned long long)fs->blocksize * num_blocks;
-	ext2fs_iblk_add_blocks(fs, &inode, es.newblocks);
 	inode.i_mtime = inode.i_ctime = fs->now ? fs->now : time(0);
 	inode.i_links_count = 1;
 	inode.i_mode = LINUX_S_IFREG | 0600;
@@ -431,9 +325,21 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 	if (retval)
 		goto out2;
 
+	retval = ext2fs_fallocate(fs, falloc_flags, journal_ino,
+				  &inode, goal, 0, num_blocks);
+	if (retval)
+		goto out2;
+
 	if ((retval = ext2fs_write_new_inode(fs, journal_ino, &inode)))
 		goto out2;
-	retval = 0;
+
+	retval = ext2fs_bmap2(fs, journal_ino, &inode, NULL, 0, 0, NULL, &zblk);
+	if (retval)
+		goto out2;
+
+	retval = io_channel_write_blk64(fs->io, zblk, 1, buf);
+	if (retval)
+		goto out2;
 
 	memcpy(fs->super->s_jnl_blocks, inode.i_block, EXT2_N_BLOCKS*4);
 	fs->super->s_jnl_blocks[15] = inode.i_size_high;
diff --git a/misc/mk_hugefiles.c b/misc/mk_hugefiles.c
index 3e4274c..5ac1114 100644
--- a/misc/mk_hugefiles.c
+++ b/misc/mk_hugefiles.c
@@ -258,12 +258,7 @@ static errcode_t mk_hugefile(ext2_filsys fs, blk64_t num,
 
 {
 	errcode_t		retval;
-	blk64_t			lblk, bend = 0;
-	__u64			size;
-	blk64_t			left;
-	blk64_t			count = 0;
 	struct ext2_inode	inode;
-	ext2_extent_handle_t	handle;
 
 	retval = ext2fs_new_inode(fs, 0, LINUX_S_IFREG, NULL, ino);
 	if (retval)
@@ -283,85 +278,20 @@ static errcode_t mk_hugefile(ext2_filsys fs, blk64_t num,
 
 	ext2fs_inode_alloc_stats2(fs, *ino, +1, 0);
 
-	retval = ext2fs_extent_open2(fs, *ino, &inode, &handle);
+	if (EXT2_HAS_INCOMPAT_FEATURE(fs->super,
+				      EXT3_FEATURE_INCOMPAT_EXTENTS))
+		inode.i_flags |= EXT4_EXTENTS_FL;
+	retval = ext2fs_fallocate(fs,
+				  EXT2_FALLOCATE_FORCE_INIT |
+				  EXT2_FALLOCATE_ZERO_BLOCKS,
+				  *ino, &inode, ~0ULL, 0, num);
 	if (retval)
 		return retval;
-
-	lblk = 0;
-	left = num ? num : 1;
-	while (left) {
-		blk64_t pblk, end;
-		blk64_t n = left;
-
-		retval =  ext2fs_find_first_zero_block_bitmap2(fs->block_map,
-			goal, ext2fs_blocks_count(fs->super) - 1, &end);
-		if (retval)
-			goto errout;
-		goal = end;
-
-		retval =  ext2fs_find_first_set_block_bitmap2(fs->block_map, goal,
-			       ext2fs_blocks_count(fs->super) - 1, &bend);
-		if (retval == ENOENT) {
-			bend = ext2fs_blocks_count(fs->super);
-			if (num == 0)
-				left = 0;
-		}
-		if (!num || bend - goal < left)
-			n = bend - goal;
-		pblk = goal;
-		if (num)
-			left -= n;
-		goal += n;
-		count += n;
-		ext2fs_block_alloc_stats_range(fs, pblk, n, +1);
-
-		if (zero_hugefile) {
-			blk64_t ret_blk;
-			retval = ext2fs_zero_blocks2(fs, pblk, n,
-						     &ret_blk, NULL);
-
-			if (retval)
-				com_err(program_name, retval,
-					_("while zeroing block %llu "
-					  "for hugefile"), ret_blk);
-		}
-
-		while (n) {
-			blk64_t l = n;
-			struct ext2fs_extent newextent;
-
-			if (l > EXT_INIT_MAX_LEN)
-				l = EXT_INIT_MAX_LEN;
-
-			newextent.e_len = l;
-			newextent.e_pblk = pblk;
-			newextent.e_lblk = lblk;
-			newextent.e_flags = 0;
-
-			retval = ext2fs_extent_insert(handle,
-					EXT2_EXTENT_INSERT_AFTER, &newextent);
-			if (retval)
-				return retval;
-			pblk += l;
-			lblk += l;
-			n -= l;
-		}
-	}
-
-	retval = ext2fs_read_inode(fs, *ino, &inode);
-	if (retval)
-		goto errout;
-
-	retval = ext2fs_iblk_add_blocks(fs, &inode,
-					count / EXT2FS_CLUSTER_RATIO(fs));
-	if (retval)
-		goto errout;
-	size = (__u64) count * fs->blocksize;
-	retval = ext2fs_inode_size_set(fs, &inode, size);
+	retval = ext2fs_inode_size_set(fs, &inode, num * fs->blocksize);
 	if (retval)
-		goto errout;
+		return retval;
 
-	retval = ext2fs_write_new_inode(fs, *ino, &inode);
+	retval = ext2fs_write_inode(fs, *ino, &inode);
 	if (retval)
 		goto errout;
 
@@ -379,13 +309,7 @@ retry:
 		goto retry;
 	}
 
-	if (retval)
-		goto errout;
-
 errout:
-	if (handle)
-		ext2fs_extent_free(handle);

[PATCH 34/39] debugfs: implement fallocate

[Darrick J. Wong]

Implement a fallocate function for debugfs, and add some tests to
demonstrate that it works (more or less).

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 debugfs/debug_cmds.ct                |    3 +
 debugfs/debugfs.c                    |   36 +++++++
 debugfs/debugfs.h                    |    1 
 tests/d_fallocate/expect.gz          |  Bin
 tests/d_fallocate/name               |    1 
 tests/d_fallocate/script             |  175 ++++++++++++++++++++++++++++++++++
 tests/d_fallocate_bigalloc/expect.gz |  Bin
 tests/d_fallocate_bigalloc/name      |    1 
 tests/d_fallocate_bigalloc/script    |  176 ++++++++++++++++++++++++++++++++++
 tests/d_fallocate_blkmap/expect      |   58 +++++++++++
 tests/d_fallocate_blkmap/name        |    1 
 tests/d_fallocate_blkmap/script      |   85 ++++++++++++++++
 12 files changed, 537 insertions(+)
 create mode 100644 tests/d_fallocate/expect.gz
 create mode 100644 tests/d_fallocate/name
 create mode 100644 tests/d_fallocate/script
 create mode 100644 tests/d_fallocate_bigalloc/expect.gz
 create mode 100644 tests/d_fallocate_bigalloc/name
 create mode 100644 tests/d_fallocate_bigalloc/script
 create mode 100644 tests/d_fallocate_blkmap/expect
 create mode 100644 tests/d_fallocate_blkmap/name
 create mode 100644 tests/d_fallocate_blkmap/script


diff --git a/debugfs/debug_cmds.ct b/debugfs/debug_cmds.ct
index c6f6d6c..34dad9e 100644
--- a/debugfs/debug_cmds.ct
+++ b/debugfs/debug_cmds.ct
@@ -157,6 +157,9 @@ request do_dirsearch, "Search a directory for a particular filename",
 request do_bmap, "Calculate the logical->physical block mapping for an inode",
 	bmap;
 
+request do_fallocate, "Allocate uninitialized blocks to an inode",
+	fallocate;
+
 request do_punch, "Punch (or truncate) blocks from an inode by deallocating them",
 	punch, truncate;
 
diff --git a/debugfs/debugfs.c b/debugfs/debugfs.c
index db85028..b30a5ab 100644
--- a/debugfs/debugfs.c
+++ b/debugfs/debugfs.c
@@ -2073,6 +2073,42 @@ void do_punch(int argc, char *argv[])
 		return;
 	}
 }
+
+void do_fallocate(int argc, char *argv[])
+{
+	ext2_ino_t	ino;
+	blk64_t		start, end;
+	int		err;
+	errcode_t	errcode;
+
+	if (common_args_process(argc, argv, 3, 4, argv[0],
+				"<file> start_blk [end_blk]",
+				CHECK_FS_RW | CHECK_FS_BITMAPS))
+		return;
+
+	ino = string_to_inode(argv[1]);
+	if (!ino)
+		return;
+	err = strtoblk(argv[0], argv[2], "logical block", &start);
+	if (err)
+		return;
+	if (argc == 4) {
+		err = strtoblk(argv[0], argv[3], "logical block", &end);
+		if (err)
+			return;
+	} else
+		end = ~0;
+
+	errcode = ext2fs_fallocate(current_fs, EXT2_FALLOCATE_INIT_BEYOND_EOF,
+				   ino, NULL, ~0ULL, start, end - start + 1);
+
+	if (errcode) {
+		com_err(argv[0], errcode,
+			"while fallocating inode %u from %llu to %llu\n", ino,
+			(unsigned long long) start, (unsigned long long) end);
+		return;
+	}
+}
 #endif /* READ_ONLY */
 
 void do_symlink(int argc, char *argv[])
diff --git a/debugfs/debugfs.h b/debugfs/debugfs.h
index e163d0a..76bb22c 100644
--- a/debugfs/debugfs.h
+++ b/debugfs/debugfs.h
@@ -166,6 +166,7 @@ extern void do_imap(int argc, char **argv);
 extern void do_set_current_time(int argc, char **argv);
 extern void do_supported_features(int argc, char **argv);
 extern void do_punch(int argc, char **argv);
+extern void do_fallocate(int argc, char **argv);
 extern void do_symlink(int argc, char **argv);
 
 extern void do_dump_mmp(int argc, char **argv);
diff --git a/tests/d_fallocate/expect.gz b/tests/d_fallocate/expect.gz
new file mode 100644
index 0000000000000000000000000000000000000000..59d6263ab66ce86f46d3a617269427f13b9ddfde
GIT binary patch
literal 3386
zcmd^9`#%%<|F0F1NTyt~b#&a;ajzk_gm5ryOmaDn+{Jd#ka2d}BC1v6<kmD|x#tqQ
zkY#R#h;YJAm=wa+n9QZm`aHhp@%?^3|H9{&*X#LszFzO=>+*g--k@x$J@nLb2*5s$
ze*kf^`zY3J9U`SDQPgqM9JBK9@IHqlC(p>U&h4W)3{ih=gVcG~{dj)jHg{Ka-z3A-
ze7486F=YWDS;H|`U7TX~vGey~)29MCtHPGiNe)N2wwchr<AI&vv{&8AJd>Le)J8w+
z-)`fL+$zk9j;rMCq8hEP1lpudX|Uf{LXqREAZv2v$j%I{twQeFFJxq)joF9wku2Y8
ztSWCY@<*S>m^1G=r%Pw&T6);*p2giow|Ke1s`-yqb<Y{*pIb&d2In7e^?zqe@6=7h
zzZuP5P#;1iVzEhog=Dp4a5Lup4{yyJDjXO@QsHG}<oou?!1h(x{F>q+OWD=ri5=>R
zQ-{i43>9+F*Ek+R`iYgVDC{@XRQI`*--gzM+v9ebTZx!&=6AR+Xk6j4kj4p$t3nOu
zxl3yP&_m5Q45}@_sHK(+><AOS8Km+)=6Vg<J!BhHYOlAf7gz2Ce{(bWwzDFf;h{tf
z-3a)LaL*t3q}agiW@emq@8}M@V&Y+CN_-<$fZ7rwuu*w`cVm{)Hd-fkeXN)hV#jE!
z3jDlEWQJ9K6)d8d%;$`3gS51bO~t^riIq4t&*!a7hSily(M4})y7yo~da!>0c0{8o
zE{&2K8>IV_;%;~1I&mkqoe3IDV~BYC*ezu$YuS(hkGPZjmhWwL12p>3E~8X~iW>=T
zm2@ARPRP&%q6g|#KtTkq>78pu2VQ6JHOz{q9T9D4q9gnR4_y&lSv!dw@a%H&Zg2rQ
zE7xh(CKGIGhvzzy_7!Cd7Q72o9iDNqoeNH+0?JDp4YxEWU`UD-L*@Pj((hVuC(>TG
z^1E8s1x<aX6(W8egGH+2Zj-}aU%R0VO7==rfmNPi<?<gd_L96HmKA-x#|d-J3&!Le
zNtk~x@Vd7CoHPU3&;bl6$%YF2c$deSX0pi*ifB1Yv=&Uxl5R&<Px=C}T<GK3>wU4`
z#4uSv%58YCnsyUfFNTXh;|{sv&nF})@J<peDMWZMNP7*f_nC{gc8|vT>vsZ0N!R)@
zu;nD)9Fjx2e*zW=a4#hm2i1*OMy{dTrh4iY_-hE+En0am$q|H9!4|Q<_LAnmk_QSy
zY5iAQ_ttiXHgjJ94H*5=_htB(S)_rF&K+X4FLmNh#NwxBrLnD6`d=>Vvw`FHw0%M?
zjYMW}71%B7*NARlDbb{cciEs*JgDxRG`2`aM3d#dAl369>yotB_(EDQi7780bJLDI
z)KfM@l-Z`uZSh&8N5qyM2l94klENDLO{k@`h^8n)p93k0w}tih5z%VJl44HyokRA1
zhhO;?w+_G_rc+eNZ$p!eKXJ>S64lnw@J~ImBAPCbt9OfNsZygbl}Rn^3AveZ%BbfZ
z)B9-gO9ci^H-Re|Fr`T28-f_Ldi3*G?hR{9ALCJ!k@xLma0lYpKw&iV_`AJx{jF+w
znE2Sgaitni(^W~g_eF)6=>+0Zfme~iaLCfyuS=+@8OM3{Y}gwO>PM0WH0F=ymHA6C
zf5d@m)H!7wm&dvXh#N+4pQ`&&_XK4G$`czu>~SZA3K-47ty$5L%T-_d1iv+z9aTb4
zemDRe)2o*p=Um#hH~+!?U8c;3Co@N*l7hTpIhwOlw5ym(9~wrlfSoT##c_XUx7p{L
z7cGnH$CyPDPM<i<rY@nU9sJZO+Hfmq@hWO~rlXPO<$H+|nOwzS6JUz9nO7|rZ7ofT
zy^g;IgnRo&vr>(rpmnn>GbPj$(7-5cJq3MS-?}lx+6Ed_r_{qjD-W{x3C9t2O4=+!
zxJ7rPwdII;?fB2Mt(FUF-Iu1P`jdqz=qJU8&8jye^0{T>*{dl2MR@_8zG>0+B#BUU
z)tjzD6jtr@r@mcq(4EdXIOuh35Md&?w}`G!%L2$nzpVZ^zqD5JgiQ$F$K}P$mM_^d
zdb*ZduRde6Nf{MCEUX*(*}o?Cj(V%+n2y81j76J{q&q-vokSOTQk8yWsT|c0lsNo}
zx&=;5@sw=+3#IlJz5Lfx_wlgb#j-K28i*lkJxYnQ-lJPPF0=NoR<&ioZl?Kc{LVt4
zfkp~ny_;>7{2-RIcYLW!2E`Xvptf>U{HaQR)K>Tew@So17W-|`SX2S}xi3b0#yadu
zJOkdcbZH}LBO2NOGj4{NHo(%PA_9*z+fSAj*PFbZk!-F~0R?tx)>7}GzxY1-yd8Je
zYwtOk?w8t*x1VmR5*J*DHEEdsWv4rtEA&6KR^A+eK8?UHPb9g9H&}HDHJHaEae1E3
z*D#VR*%g{GyzUSS(yF{D+aT64iiaKB=w^H&QXw|OxhhL8Zl7KqPVD-M4<B5CFcebV
zUykM4BE6nKaJ45dAprsTb^$7%UR_>#Ws5BLKuT5@#spmCU3rkWz9k%bK;HN~EFhm%
z%IuNeipO~nm8Cn+gX3F@z^5?!@`rI><8i3(+y&g4+Lr11t&gwwD?o)@qlYioGAXu*
zE4Vyh<}&&`u2aX>vfP9&XQYs6SBi{s055t0(4hd&PdUp*Ft7^*{A$0#@gU22Y5Pbi
zyo^MLyi~Xh{)D`~qLiq4HU{@71ak3+RDcWkxd%8R`FdU#-Znz#F7J~HQ4s9_1k}rC
z4jjuQ*!oj!)%VXfNX&xekvj4RaI}A)Z2v(Xdjgl2;tUqLg7JS|i1GsqtB3XzU4R<i
z8=1irX^q@4D?JR&^E=YAtX%ttt6V^{?SPhgZb^vA6OII}gg`(3lXipE21JCnZ$y*+
z{m*u6)BBf|RLk$V22+AnU;cl1JZH?c)W32FA<hMVc}KXhso9Q*P&sB%ebz(pNXywG
zoIa#F4;!Zw#thRL`jDm@n<I&_l+zZJz%%KqnIP=eUV_G|B;TI0wVZ+KzyrwvWU4bx
zT)1Y5@Y`Xx(i@R)Qx@t4{=TZ^p3zo8E6=Z^lvU0;fr`BZA2g(8$~_zN4K{qihUwR*
z5q=SL1cm-|NK(RtKyX|dKEoV3MrbC8n;JPo9a3;C8H$a>2A2#OrcZ<_scOaE-FA_=
zqVFA{3rRS1^I1=Wor#}P)s&sdF`bLKM+<^?eLox%+q_o?2Mp6)>4dO6+PQ?Eerjq%
zPzJmxoMeiM1h>qt*ygp7tNLKIGTCO;#OF{lx{J4sz7xEg6!s7JKMZQyES&8mgMav5
zo19b>lGlDN%;0DyGsiYYe5(hS4@v+8J=T}NzDAdWbma3Kh3^0_vM6su0Y~y2E8_Rh
zb~wp&4&VPmx7}ZkmiKX$m^JuO9;qdd+XU(XGg;d=UmyK$_5T-3D;?ZF4>S9&s3?AC
zuBH72jsZa~;529Ou0PoDc0+{fAhc|Mxo6Buc@E1o>!-;ag*?Z4dZ8C!M5i_e?(&Oj
z*N0daX`e$~xmR4I`^#xXZWS!3MRQT8NFlgFM!Ih-w8Pl2qFf*NC(ilwJLV@ztx?65
z616-8sVzJ97caLNea8O;AUWmY--k6-P#uskddHr+3S^BqFL1aovo18ajA;+xtCMsf
zj?#>=DC1CxpJN)JZ)8u^>rUB2pLS|p19Dgjg&l4Q`~D*k4YW$tngWS>oAPr;+YOpj
zqbg1syJfnL7#g=J|EYy9i&XwI@Rl?1*I|S*Nvrd5q%tY+kP{FvY<kFv7@7X97kcyV
zT2v&c)~y-$(bKem6wzd7>L*+Jrqsh!n-W3rFs;Ftm`ghV?cOvgWX+#cbdz!zanUIt
zd6=G1bc0YL-0j}r|2vb~@g-%lh6;9XmN5;_poz<_DWH*|?9M$Qtz8OdkBjh)FCbBM
z*RZ=msj?G&+^dt=$pBkq@tq7|Az$voqg`Jl>=t}Ca(hcSZ!iGa6ip8FapKZztf9sF
zAKefFE&HIq&;Rw{xr8^Yv-d&EFR+Voe0;vE6$?GWckG?(o8Fz;MHr-~u9u|j434L#
z^Wp1#n7XRsT_Kw&rupZ$;{o^h8$FR*HDnQhUM`W}6?Ye2_ziIGb^Nm;k3D<-1q+Lr
AoB#j-

literal 0
HcmV?d00001

diff --git a/tests/d_fallocate/name b/tests/d_fallocate/name
new file mode 100644
index 0000000..72d0ed3
--- /dev/null
+++ b/tests/d_fallocate/name
@@ -0,0 +1 @@
+fallocate sparse files and big files
diff --git a/tests/d_fallocate/script b/tests/d_fallocate/script
new file mode 100644
index 0000000..829a1bd
--- /dev/null
+++ b/tests/d_fallocate/script
@@ -0,0 +1,175 @@
+if test -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fy
+OUT=$test_name.log
+if [ -f $test_dir/expect.gz ]; then
+	EXP=$test_name.tmp
+	gunzip < $test_dir/expect.gz > $EXP
+else
+	EXP=$test_dir/expect
+fi
+
+cp /dev/null $OUT
+
+cat > $TMPFILE.conf << ENDL
+[fs_types]
+ext4 = {
+        base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr,^has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,64bit
+        blocksize = 1024
+        inode_size = 256
+        inode_ratio = 16384
+}
+ENDL
+MKE2FS_CONFIG=$TMPFILE.conf $MKE2FS -F -o Linux -b 1024 -O ^bigalloc -T ext4 $TMPFILE 65536 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT 2>&1
+rm -rf $TMPFILE.conf
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+echo "debugfs write files" >> $OUT
+make_file() {
+	name="$1"
+	start="$2"
+	flag="$3"
+
+	cat << ENDL
+write /dev/null $name
+sif /$name size 40960
+eo /$name
+set_bmap $flag 10 $((start + 10))
+set_bmap $flag 13 $((start + 13))
+set_bmap $flag 26 $((start + 26))
+set_bmap $flag 29 $((start + 29))
+ec
+sif /$name blocks 8
+setb $((start + 10))
+setb $((start + 13))
+setb $((start + 26))
+setb $((start + 29))
+ENDL
+}
+
+#Files we create:
+# a: fallocate a 40k file
+# b*: falloc sparse file starting at b*
+# c*: falloc spare file ending at c*
+# d: midcluster to midcluster, surrounding sparse
+# e: partial middle cluster alloc
+# f: one big file
+# g*: falloc sparse init file starting at g*
+# h*: falloc sparse init file ending at h*
+# i: midcluster to midcluster, surrounding sparse init
+# j: partial middle cluster alloc
+# k: one big init file
+base=5000
+cat > $TMPFILE.cmd << ENDL
+write /dev/null a
+sif /a size 40960
+fallocate /a 0 39
+ENDL
+echo "ex /a" >> $TMPFILE.cmd2
+
+make_file sample $base --uninit >> $TMPFILE.cmd
+echo "ex /sample" >> $TMPFILE.cmd2
+base=10000
+
+for i in 8 9 10 11 12 13 14 15; do
+	make_file b$i $(($base + (40 * ($i - 8)))) --uninit >> $TMPFILE.cmd
+	echo "fallocate /b$i $i 39" >> $TMPFILE.cmd
+	echo "ex /b$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file c$i $(($base + 320 + (40 * ($i - 24)))) --uninit >> $TMPFILE.cmd
+	echo "fallocate /c$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /c$i" >> $TMPFILE.cmd2
+done
+
+make_file d $(($base + 640)) --uninit >> $TMPFILE.cmd
+echo "fallocate /d 4 35" >> $TMPFILE.cmd
+echo "ex /d" >> $TMPFILE.cmd2
+
+make_file e $(($base + 680)) --uninit >> $TMPFILE.cmd
+echo "fallocate /e 19 20" >> $TMPFILE.cmd
+echo "ex /e" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null f
+sif /f size 1024
+eo /f
+set_bmap --uninit 0 9000
+ec
+sif /f blocks 2
+setb 9000
+fallocate /f 0 8999
+ENDL
+echo "ex /f" >> $TMPFILE.cmd2
+
+# Now do it again, but with initialized blocks
+base=20000
+for i in 8 9 10 11 12 13 14 15; do
+	make_file g$i $(($base + (40 * ($i - 8)))) >> $TMPFILE.cmd
+	echo "fallocate /g$i $i 39" >> $TMPFILE.cmd
+	echo "ex /g$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file h$i $(($base + 320 + (40 * ($i - 24)))) >> $TMPFILE.cmd
+	echo "fallocate /h$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /h$i" >> $TMPFILE.cmd2
+done
+
+make_file i $(($base + 640)) >> $TMPFILE.cmd
+echo "fallocate /i 4 35" >> $TMPFILE.cmd
+echo "ex /i" >> $TMPFILE.cmd2
+
+make_file j $(($base + 680)) >> $TMPFILE.cmd
+echo "fallocate /j 19 20" >> $TMPFILE.cmd
+echo "ex /j" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null k
+sif /k size 1024
+eo /k
+set_bmap 0 19000
+ec
+sif /k blocks 2
+setb 19000
+fallocate /k 0 8999
+sif /k size 9216000
+ENDL
+echo "ex /k" >> $TMPFILE.cmd2
+
+$DEBUGFS_EXE -w $TMPFILE -f $TMPFILE.cmd > /dev/null 2>&1
+$DEBUGFS_EXE $TMPFILE -f $TMPFILE.cmd2 >> $OUT.new 2>&1
+sed -f $cmd_dir/filter.sed < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE.cmd $TMPFILE.cmd2
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+rm -f $TMPFILE
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f $test_name.tmp
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
diff --git a/tests/d_fallocate_bigalloc/expect.gz b/tests/d_fallocate_bigalloc/expect.gz
new file mode 100644
index 0000000000000000000000000000000000000000..1f88dcdc60b63e9984e21d50ecdee982bd327aed
GIT binary patch
literal 2799
zcmbV}`9BkmAIB-Oxl(gga~0-_v3+e=d=R-+gq%Z%$yv(CB4=|gxn>i|kgxj)IbyCk
zCdYD{xt1K|>XP;G_4)h>-(TME=lgg)-jB!oC6ff;dKY16f9yEZ)r~OQB!&~>r~*WJ
zYfT=RwyqRWxD&lv$XwjDpN~zOjFfclD8bh0Q^#lPa=uME$`eo-e&)9(!%Lq=E&r0n
z<zDcR-gU?BVBWb+Zje=L+AF$Of8Pm<Y`=-u7{7m5Uvaq&)PloBYUE8!jyt$IwmbeL
z(-M5!JR-_ff?HSyUn|u$%2wOn|8i!HON5R;a$|b<H6XGd9uIquZN;^64i>A)71*h%
z<w~-FOx8hoox9qTIqkdmDh(q_x5}7}%=h6U9vgoup!5Mi7(^ej(DKr$vMTO_1)~Eo
zIUZ2+iz+?t9oRrFxl-ftBtSS=fizj>SXGJGut;td&u@5pyk+@M)dG{@{Z^<F+LXlh
zeH-SyESjJvJY4O>sh#^0+j`ny-l43et7mh5p_tX={dN5i{&T}f8PHbQIPTj%{p6ng
zjDAJ=Eq$|>g3kUE8=r=M{|rAIqOHjWxZp|~#=~y_Lr0NPe5dl3nT)QkS{>&%i|=Q?
z9LAk%KYiU@H)k&L*(5uTQ|A;o!9uLECK0ye5@u*1Nwz|xt@CdiiNO2~+=B~4>5X+K
z-5Z)J^$nO2)A%YOMy7%DL8RmBaPjQH9=?OiiH#e78Uk$C9x~d9@4jJMEYAG{|J8Zi
zn+e1a?EBXzVRbIgHm~0rD%q{wUtQALX_K;7p39sLy)Hh0D~9d3>?xG~WY%qEoh1Am
z*HJf|h2kxJOsP4MtEN9+d{L-J>Mn=|4fLokeB`oui`0^yOtx-~T<v8baUGc*nd<L(
zDmNE(?Q!ZKOw*NuT0TNEZ~t(EX>ZLXK``gpOPRE^67$p%%2QR>))SSINY*LqA<*q*
zOM>0)tq3>Lz}~2RJRFmoXPBN!!5gNVQ~IM6)6LSU4?iu#`!_i(U7B6a{$BQixyi15
zc;2n7-gG)V2!raU@5!@yZ0}3wQFc-0nZ4=u3+9kr3N_tk1vuQc0?hMCRh!R6@Y(K%
zh?hJs$wdi+uI_n)ZF#XOr^8e_>~yH<qy)%|JV;{x7qD$C+y;-yecr3AnF1O^T2q2c
z?=In~Ajobk$_5{n`<$k%NdWC4t=YkUUqJPbbo8e_RSlt)mHByIaN&{xC3OeVq-#1x
zxqo^!`k|g(9Otr;3K9gjU{9p-1^teK704C^BjbR^|J6@*alImPL@BnCs$+LSl#MsD
zKrs!Rk;thJ(8N|*ei(fxMH;x27ZP{IRB9NCGZeOMezbEIc}c-+f$Wds8NI_5Di6>x
zgu<qSyAf5-hF1Z4oU_|xnH@cnxxMg{q+eD`*iJfyAGRQAQ=#F9!m21@wf43tu;KJ9
z3<^WHB8_3kh&k}09f~FlLt0SI$3ukjkZBOQm9FQV0XGy@YriA$PSjET2+S@e;!BR9
zGVv(S*Nfd^C%ogRJF0>-kauVGvXU(uh0@$LD1+xo?&pIXC6hVBx@nM=;Jm(N6*B$^
zZ0O6$s1GLRZTpsUt2_H#QCMFgaiMMb)x~~U0=XL%@SJA7#NTzqv10KdJ9&`8TN;O<
z*k*khOWj*Q)-lFcP-dE%ToSqadz_&-0=OaYXa}vFfcs!uYd=)i{7TKz#Z9tY-D+vQ
zQ+AE~Y}|yL!P!S$3Y$2rO(9RCU>!v{)2MsH2og_wF3#UmCe1YUwM$M~0oZA~KC1U(
zM$RDbqu5X-n?CN`S-&jxT9lu*UapjFpJ(vcmF5&k<3XaE$)yl|<`TziH~`ud!{sQr
z*wnKhnCZrvjzNFTfXBiv+MEgskN`Zu!;PC-d*12Q7Jm1NowJCE5a$D9JsSZhG~j)f
z(@T=Al8v8_xI2mNoXf58j%`xIx?wjZgzD`V&yLBSt1z@!7?1<MZ93K}`k_#T!0-ix
zV%_<-uh(7>Yl`@XEryerp#f$uq-pm7llmG@Sntjhg?Mox`YaDx&Dt%Mzr3$|Ir#M4
zN)gVT1H#be@Mgs2_@Ex{)vz@E4dYPQo5x3cY^?IYSP^8{k{Zy^Cs1fxTtZ%o*e?Z?
z2w<8Sybp^77^=Rf5}ifG-|B+m)~rM~&x?u7HOGGt_0&CQK45Z|5LKm<la5j+)&O~-
z`~v8g{nT$@p@MJY$r`DObYtm)&wmk9<Yj*kUvaG3jR?;yaQ`@q%7A=RF>Zr@-7ad@
zE|M_TNS#P3$Tt5t_?HH#BAd!w84YKTsLD)k5N>=i<q%(M8r%tIs~Y7HI<;6CXayeC
zLSEH2w7)sj0h&)zRs-4qJ|#UJQu5~A!{iFHy9}u-R?%x})=-Z?O7%Qaj#ves(R6?s
z+#M;<K68;6kJfdDt_bvxMr-mc1F1gh)I=;*>v4e&&tv6?JdtE0>hsIDS}2V4XtqeX
znP_<w+N2)M`|r_U1UnY(t4N)q06(c#i#nho?>%{j4M+207Ic(GABdp%OdbnIufFmb
zHrxio7j-Ck=FmbAuQ#A~G@2T6wSG;OC~N_j;$aw$7RR7<m1aUjc6qIHQgbZm83a-)
z#|cVIebqg8N^Q&kP$Y3sR6JWRn%!2bRZwmPy3OV|`4Gka<KxiO^Dx!{s=6L~Q<Lwq
zQhoHzYY@7{{CQm?)uVBML%^zv+blCaZS3Lvto!+l$}h;;HXhfeO}mwBoPWQbyq(k~
z*?(N5u|6}7B;cauxCwP>qmx-QkCi9W*W@#3%4!6eL8(oJ)BMY#;=Kn(d#*+j-bQ5a
z0-;!KZzGwit<0w*`<McPU1L@A>`wg%t<;4IF7>kVVU!tw*1w5PtPw!Lj!<IZD<hT!
zoi6W$8ROaOiO?$s7T3rX-Y!R<{bV-iM1|ASis47`+6WXXM~v6ucjo;a&E5mtDM>nN
z^Dywyn13G!#-xf1eljCtsdJa8VBVJkcF{-ak^qjHn8TBvq2$C(O}tFy2RN7toOnPP
z^r52fBa<Y`gLcj~F%(ywpz-}c66AMuCJbnLlY|zm>&#G;>?*HL6ifDWaO&ajg{qc=
zcw=sQN?nw+;yXza&Ky#3dcB4w3fsV?1R2yLt2}j)Ga*nbJ_-DhjX|g6hy$+jJdZJu
zH37xDr>zu7!p9k&r>zs7u0cnOyE;OMJ`7K^u3!=H&Qa_ED7sN3)IQ%^<9EonRhJql
zdVHYrse9cz;l5m$B<`aVvN;@94UYGA3_Ow56|}iN)Nx-jy}=4m4esq$X9%e!^m}Km
zeGRCV@pk_aXnT=ycxgBw*~<dcFIT^Eb?OCdCfV4hx1jpW-Fl&fk$7XgKQdMu%s(4s
zdM*ecgcQ|Y2*I@LJ<tlt{2ng>nh~0Hg3W|q7!AMBH9;ONK6g_(analw5nFuD!mn8l
z&b;cy5Gq)UKTQq*SR*o~LX3Ty3%ZNX88L(uw3#BL0HvC>_>mA}{QoP_9Szxw+==1a
zYPd1}w_2QWQq^qAA7Iju_#1=UOaEETiC8wMsyrNs|J9Pz)AD;iaEbh9Kd~UDbs}X@
z!##>buAemg@g!(b<A=h0+u^oD1ZS1z!TMWG|IhX{4u_CG-~RaU2jnpLWG9p5GqS!y
YfH#>;eXfG{^+a;}+(<JuzRbn-KS;4<*#H0l

literal 0
HcmV?d00001

diff --git a/tests/d_fallocate_bigalloc/name b/tests/d_fallocate_bigalloc/name
new file mode 100644
index 0000000..915645c
--- /dev/null
+++ b/tests/d_fallocate_bigalloc/name
@@ -0,0 +1 @@
+fallocate sparse files and big files with bigalloc
diff --git a/tests/d_fallocate_bigalloc/script b/tests/d_fallocate_bigalloc/script
new file mode 100644
index 0000000..dae8699
--- /dev/null
+++ b/tests/d_fallocate_bigalloc/script
@@ -0,0 +1,176 @@
+if test -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fy
+OUT=$test_name.log
+if [ -f $test_dir/expect.gz ]; then
+	EXP=$test_name.tmp
+	gunzip < $test_dir/expect.gz > $EXP
+else
+	EXP=$test_dir/expect
+fi
+
+cp /dev/null $OUT
+
+cat > $TMPFILE.conf << ENDL
+[fs_types]
+ext4 = {
+	cluster_size = 8192
+        base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr,^has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,64bit
+        blocksize = 1024
+        inode_size = 256
+        inode_ratio = 16384
+}
+ENDL
+MKE2FS_CONFIG=$TMPFILE.conf $MKE2FS -F -o Linux -b 1024 -O bigalloc -T ext4 $TMPFILE 65536 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT 2>&1
+rm -rf $TMPFILE.conf
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+echo "debugfs write files" >> $OUT
+make_file() {
+	name="$1"
+	start="$2"
+	flag="$3"
+
+	cat << ENDL
+write /dev/null $name
+sif /$name size 40960
+eo /$name
+set_bmap $flag 10 $((start + 10))
+set_bmap $flag 13 $((start + 13))
+set_bmap $flag 26 $((start + 26))
+set_bmap $flag 29 $((start + 29))
+ec
+sif /$name blocks 32
+setb $((start + 10))
+setb $((start + 13))
+setb $((start + 26))
+setb $((start + 29))
+ENDL
+}
+
+#Files we create:
+# a: fallocate a 40k file
+# b*: falloc sparse file starting at b*
+# c*: falloc spare file ending at c*
+# d: midcluster to midcluster, surrounding sparse
+# e: partial middle cluster alloc
+# f: one big file
+# g*: falloc sparse init file starting at g*
+# h*: falloc sparse init file ending at h*
+# i: midcluster to midcluster, surrounding sparse init
+# j: partial middle cluster alloc
+# k: one big init file
+base=5000
+cat > $TMPFILE.cmd << ENDL
+write /dev/null a
+sif /a size 40960
+fallocate /a 0 39
+ENDL
+echo "ex /a" >> $TMPFILE.cmd2
+
+make_file sample $base --uninit >> $TMPFILE.cmd
+echo "ex /sample" >> $TMPFILE.cmd2
+base=10000
+
+for i in 8 9 10 11 12 13 14 15; do
+	make_file b$i $(($base + (40 * ($i - 8)))) --uninit >> $TMPFILE.cmd
+	echo "fallocate /b$i $i 39" >> $TMPFILE.cmd
+	echo "ex /b$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file c$i $(($base + 320 + (40 * ($i - 24)))) --uninit >> $TMPFILE.cmd
+	echo "fallocate /c$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /c$i" >> $TMPFILE.cmd2
+done
+
+make_file d $(($base + 640)) --uninit >> $TMPFILE.cmd
+echo "fallocate /d 4 35" >> $TMPFILE.cmd
+echo "ex /d" >> $TMPFILE.cmd2
+
+make_file e $(($base + 680)) --uninit >> $TMPFILE.cmd
+echo "fallocate /e 19 20" >> $TMPFILE.cmd
+echo "ex /e" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null f
+sif /f size 1024
+eo /f
+set_bmap --uninit 0 9000
+ec
+sif /f blocks 16
+setb 9000
+fallocate /f 0 8999
+ENDL
+echo "ex /f" >> $TMPFILE.cmd2
+
+# Now do it again, but with initialized blocks
+base=20000
+for i in 8 9 10 11 12 13 14 15; do
+	make_file g$i $(($base + (40 * ($i - 8)))) >> $TMPFILE.cmd
+	echo "fallocate /g$i $i 39" >> $TMPFILE.cmd
+	echo "ex /g$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file h$i $(($base + 320 + (40 * ($i - 24)))) >> $TMPFILE.cmd
+	echo "fallocate /h$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /h$i" >> $TMPFILE.cmd2
+done
+
+make_file i $(($base + 640)) >> $TMPFILE.cmd
+echo "fallocate /i 4 35" >> $TMPFILE.cmd
+echo "ex /i" >> $TMPFILE.cmd2
+
+make_file j $(($base + 680)) >> $TMPFILE.cmd
+echo "fallocate /j 19 20" >> $TMPFILE.cmd
+echo "ex /j" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null k
+sif /k size 1024
+eo /k
+set_bmap 0 19000
+ec
+sif /k blocks 16
+setb 19000
+fallocate /k 0 8999
+sif /k size 9216000
+ENDL
+echo "ex /k" >> $TMPFILE.cmd2
+
+$DEBUGFS_EXE -w $TMPFILE -f $TMPFILE.cmd > /dev/null 2>&1
+$DEBUGFS_EXE $TMPFILE -f $TMPFILE.cmd2 >> $OUT.new 2>&1
+sed -f $cmd_dir/filter.sed < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE.cmd $TMPFILE.cmd2
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+rm -f $TMPFILE
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f $test_name.tmp
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
diff --git a/tests/d_fallocate_blkmap/expect b/tests/d_fallocate_blkmap/expect
new file mode 100644
index 0000000..f7ae606
--- /dev/null
+++ b/tests/d_fallocate_blkmap/expect
@@ -0,0 +1,58 @@
+Creating filesystem with 65536 1k blocks and 4096 inodes
+Superblock backups stored on blocks: 
+	8193, 24577, 40961, 57345
+
+Allocating group tables:    \b\b\bdone                            
+Writing inode tables:    \b\b\bdone                            
+Writing superblocks and filesystem accounting information:    \b\b\bdone
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 11/4096 files (0.0% non-contiguous), 2340/65536 blocks
+Exit status is 0
+debugfs write files
+debugfs: stat /a
+Inode: 12   Type: regular    Mode:  0666   Flags: 0x0
+Generation: 0    Version: 0x00000000:00000000
+User:     0   Group:     0   Size: 40960
+File ACL: 0    Directory ACL: 0
+Links: 1   Blockcount: 82
+Fragment:  Address: 0    Number: 0    Size: 0
+Size of extra inode fields: 28
+BLOCKS:
+(0-1):1312-1313, (2-11):8000-8009, (IND):8010, (12-39):8011-8038
+TOTAL: 41
+
+debugfs: stat /b
+Inode: 13   Type: regular    Mode:  0666   Flags: 0x0
+Generation: 0    Version: 0x00000000:00000000
+User:     0   Group:     0   Size: 10240000
+File ACL: 0    Directory ACL: 0
+Links: 1   Blockcount: 20082
+Fragment:  Address: 0    Number: 0    Size: 0
+Size of extra inode fields: 28
+BLOCKS:
+(0-11):10000-10011, (IND):10012, (12-267):10013-10268, (DIND):10269, (IND):10270, (268-523):10271-10526, (IND):10527, (524-779):10528-10783, (IND):10784, (780-1035):10785-11040, (IND):11041, (1036-1291):11042-11297, (IND):11298, (1292-1547):11299-11554, (IND):11555, (1548-1803):11556-11811, (IND):11812, (1804-2059):11813-12068, (IND):12069, (2060-2315):12070-12325, (IND):12326, (2316-2571):12327-12582, (IND):12583, (2572-2827):12584-12839, (IND):12840, (2828-3083):12841-13096, (IND):13097, (3084-3339):13098-13353, (IND):13354, (3340-3595):13355-13610, (IND):13611, (3596-3851):13612-13867, (IND):13868, (3852-4107):13869-14124, (IND):14125, (4108-4363):14126-14381, (IND):14382, (4364-4619):14383-14638, (IND):14639, (4620-4875):14640-14895, (IND):14896, (4876-5131):14897-15152, (IND):15153, 
 (5132-5387):15154-15409, (IND):15410, (5388-5643):15411-15666, (IND):15667, (5644-5899):15668-15923, (IND):15924, (5900-6155):15925-16180, (IND):16181, (6156-6411):16182-16437, (IND):16438,!
  (6412-6667):16439-16694, (IND):16695, (6668-6923):16696-16951, (IND):16952, (6924-7179):16953-17208, (IND):17209, (7180-7435):17210-17465, (IND):17466, (7436-7691):17467-17722, (IND):17723, (7692-7947):17724-17979, (IND):17980, (7948-8203):17981-18236, (IND):18237, (8204-8459):18238-18493, (IND):18494, (8460-8715):18495-18750, (IND):18751, (8716-8971):18752-19007, (IND):19008, (8972-9227):19009-19264, (IND):19265, (9228-9483):19266-19521, (IND):19522, (9484-9739):19523-19778, (IND):19779, (9740-9995):19780-20035, (IND):20036, (9996-9999):20037-20040
+TOTAL: 10041
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Free blocks count wrong for group #0 (6841, counted=6840).
+Fix? yes
+
+Free blocks count wrong for group #1 (1551, counted=1550).
+Fix? yes
+
+Free blocks count wrong (53116, counted=53114).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 13/4096 files (7.7% non-contiguous), 12422/65536 blocks
+Exit status is 1
diff --git a/tests/d_fallocate_blkmap/name b/tests/d_fallocate_blkmap/name
new file mode 100644
index 0000000..ba2b61d
--- /dev/null
+++ b/tests/d_fallocate_blkmap/name
@@ -0,0 +1 @@
+fallocate sparse files and big files on a blockmap fs
diff --git a/tests/d_fallocate_blkmap/script b/tests/d_fallocate_blkmap/script
new file mode 100644
index 0000000..da83cd1
--- /dev/null
+++ b/tests/d_fallocate_blkmap/script
@@ -0,0 +1,85 @@
+if test -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fy
+OUT=$test_name.log
+if [ -f $test_dir/expect.gz ]; then
+	EXP=$test_name.tmp
+	gunzip < $test_dir/expect.gz > $EXP1
+else
+	EXP=$test_dir/expect
+fi
+
+cp /dev/null $OUT
+
+cat > $TMPFILE.conf << ENDL
+[fs_types]
+ext4 = {
+        base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr,^has_journal,^extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,^64bit
+        blocksize = 1024
+        inode_size = 256
+        inode_ratio = 16384
+}
+ENDL
+MKE2FS_CONFIG=$TMPFILE.conf $MKE2FS -F -o Linux -b 1024 -O ^bigalloc -T ext4 $TMPFILE 65536 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT 2>&1
+rm -rf $TMPFILE.conf
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+echo "debugfs write files" >> $OUT
+
+#Files we create:
+# a: fallocate a 40k file
+# k: one big file
+base=5000
+cat > $TMPFILE.cmd << ENDL
+write /dev/null a
+sif /a bmap[2] 8000
+sif /a size 40960
+sif /a i_blocks 2
+setb 8000
+fallocate /a 0 39
+
+write /dev/null b
+sif /b size 10240000
+sif /b bmap[0] 10000
+sif /b i_blocks 2
+setb 10000
+fallocate /b 0 9999
+ENDL
+echo "stat /a" >> $TMPFILE.cmd2
+echo "stat /b" >> $TMPFILE.cmd2
+
+$DEBUGFS_EXE -w $TMPFILE -f $TMPFILE.cmd > /dev/null 2>&1
+$DEBUGFS_EXE $TMPFILE -f $TMPFILE.cmd2 >> $OUT.new 2>&1
+sed -f $cmd_dir/filter.sed -e '/^.*time:.*$/d' < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE.cmd $TMPFILE.cmd2
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+rm -f $TMPFILE
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f $test_name.tmp
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi

[PATCH 35/39] tests: test debugfs punch command

[Darrick J. Wong]

Test punching out various parts of sparse files.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/d_punch/expect          |  143 +++++++++++++++++++++++++++++++++++++++++
 tests/d_punch/name            |    1 
 tests/d_punch/script          |  129 +++++++++++++++++++++++++++++++++++++
 tests/d_punch_bigalloc/expect |  142 +++++++++++++++++++++++++++++++++++++++++
 tests/d_punch_bigalloc/name   |    1 
 tests/d_punch_bigalloc/script |  130 +++++++++++++++++++++++++++++++++++++
 6 files changed, 546 insertions(+)
 create mode 100644 tests/d_punch/expect
 create mode 100644 tests/d_punch/name
 create mode 100644 tests/d_punch/script
 create mode 100644 tests/d_punch_bigalloc/expect
 create mode 100644 tests/d_punch_bigalloc/name
 create mode 100644 tests/d_punch_bigalloc/script


diff --git a/tests/d_punch/expect b/tests/d_punch/expect
new file mode 100644
index 0000000..a5b7fd8
--- /dev/null
+++ b/tests/d_punch/expect
@@ -0,0 +1,143 @@
+Creating filesystem with 65536 1k blocks and 4096 inodes
+Superblock backups stored on blocks: 
+	8193, 24577, 40961, 57345
+
+Allocating group tables:    \b\b\bdone                            
+Writing inode tables:    \b\b\bdone                            
+Writing superblocks and filesystem accounting information:    \b\b\bdone
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 11/4096 files (0.0% non-contiguous), 2345/65536 blocks
+Exit status is 0
+debugfs write files
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+debugfs: ex /sample
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1323              0
+ 1/ 1   1/  5     0 -     9  1313 -  1322     10 Uninit
+ 1/ 1   2/  5    11 -    12  1324 -  1325      2 Uninit
+ 1/ 1   3/  5    14 -    25  1327 -  1338     12 Uninit
+ 1/ 1   4/  5    27 -    28  1340 -  1341      2 Uninit
+ 1/ 1   5/  5    30 -    39  1343 -  1352     10 Uninit
+debugfs: ex /b8
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1390              0
+ 1/ 1   1/  4     0 -     0  1326 -  1326      1 Uninit
+ 1/ 1   2/  4     1 -     1  1339 -  1339      1 Uninit
+ 1/ 1   3/  4     2 -     2  1342 -  1342      1 Uninit
+ 1/ 1   4/  4     3 -     7  1353 -  1357      5 Uninit
+debugfs: ex /b9
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1368              0
+ 1/ 1   1/  1     0 -     8  1358 -  1366      9 Uninit
+debugfs: ex /b10
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1378              0
+ 1/ 1   1/  2     0 -     0  1367 -  1367      1 Uninit
+ 1/ 1   2/  2     1 -     9  1369 -  1377      9 Uninit
+debugfs: ex /b11
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1389              0
+ 1/ 1   1/  1     0 -     9  1379 -  1388     10 Uninit
+debugfs: ex /b12
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1401              0
+ 1/ 1   1/  2     0 -     9  1391 -  1400     10 Uninit
+ 1/ 1   2/  2    11 -    11  1402 -  1402      1 Uninit
+debugfs: ex /b13
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1413              0
+ 1/ 1   1/  2     0 -     9  1403 -  1412     10 Uninit
+ 1/ 1   2/  2    11 -    12  1414 -  1415      2 Uninit
+debugfs: ex /b14
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1426              0
+ 1/ 1   1/  2     0 -     9  1416 -  1425     10 Uninit
+ 1/ 1   2/  2    11 -    12  1427 -  1428      2 Uninit
+debugfs: ex /b15
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1439              0
+ 1/ 1   1/  3     0 -     9  1429 -  1438     10 Uninit
+ 1/ 1   2/  3    11 -    12  1440 -  1441      2 Uninit
+ 1/ 1   3/  3    14 -    14  1443 -  1443      1 Uninit
+debugfs: ex /c24
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    25 - 4294967295  1453         4294967271
+ 1/ 1   1/  3    25 -    25  1468 -  1468      1 Uninit
+ 1/ 1   2/  3    27 -    28  1470 -  1471      2 Uninit
+ 1/ 1   3/  3    30 -    39  1473 -  1482     10 Uninit
+debugfs: ex /c25
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    27 - 4294967295  1483         4294967269
+ 1/ 1   1/  2    27 -    28  1485 -  1486      2 Uninit
+ 1/ 1   2/  2    30 -    39  1488 -  1497     10 Uninit
+debugfs: ex /c26
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    27 - 4294967295  1484         4294967269
+ 1/ 1   1/  2    27 -    28  1498 -  1499      2 Uninit
+ 1/ 1   2/  2    30 -    39  1501 -  1510     10 Uninit
+debugfs: ex /c27
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    28 - 4294967295  1487         4294967268
+ 1/ 1   1/  2    28 -    28  1512 -  1512      1 Uninit
+ 1/ 1   2/  2    30 -    39  1514 -  1523     10 Uninit
+debugfs: ex /c28
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    30 - 4294967295  1500         4294967266
+ 1/ 1   1/  1    30 -    39  1526 -  1535     10 Uninit
+debugfs: ex /c29
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    30 - 4294967295  1511         4294967266
+ 1/ 1   1/  1    30 -    39  1537 -  1546     10 Uninit
+debugfs: ex /c30
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    31 - 4294967295  1513         4294967265
+ 1/ 1   1/  1    31 -    39  1549 -  1557      9 Uninit
+debugfs: ex /c31
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    32 - 4294967295  1524         4294967264
+ 1/ 1   1/  1    32 -    39  1560 -  1567      8 Uninit
+debugfs: ex /d
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1525              0
+ 1/ 1   1/  3     0 -     0  1442 -  1442      1 Uninit
+ 1/ 1   2/  3     1 -     3  1444 -  1446      3 Uninit
+ 1/ 1   3/  3    36 -    39  1573 -  1576      4 Uninit
+debugfs: ex /e
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1547              0
+ 1/ 1   1/ 11     0 -     5  1447 -  1452      6 Uninit
+ 1/ 1   2/ 11     6 -     9  1454 -  1457      4 Uninit
+ 1/ 1   3/ 11    11 -    12  1459 -  1460      2 Uninit
+ 1/ 1   4/ 11    14 -    18  1462 -  1466      5 Uninit
+ 1/ 1   5/ 11    21 -    21  1472 -  1472      1 Uninit
+ 1/ 1   6/ 11    22 -    22  1536 -  1536      1 Uninit
+ 1/ 1   7/ 11    23 -    23  1548 -  1548      1 Uninit
+ 1/ 1   8/ 11    24 -    25  1558 -  1559      2 Uninit
+ 1/ 1   9/ 11    27 -    28  1569 -  1570      2 Uninit
+ 1/ 1  10/ 11    30 -    30  1572 -  1572      1 Uninit
+ 1/ 1  11/ 11    31 -    39  1577 -  1585      9 Uninit
+debugfs: ex /f
+Level Entries       Logical      Physical Length Flags
+ 0/ 0   1/  2     0 -     0  9000 -  9000      1 Uninit
+ 0/ 0   2/  2  8999 -  8999 17999 - 17999      1 Uninit
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Free blocks count wrong for group #1 (7934, counted=7933).
+Fix? yes
+
+Free blocks count wrong (62923, counted=62922).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 32/4096 files (43.8% non-contiguous), 2614/65536 blocks
+Exit status is 1
diff --git a/tests/d_punch/name b/tests/d_punch/name
new file mode 100644
index 0000000..724639f
--- /dev/null
+++ b/tests/d_punch/name
@@ -0,0 +1 @@
+punch sparse files and big files
diff --git a/tests/d_punch/script b/tests/d_punch/script
new file mode 100644
index 0000000..db6c4dd
--- /dev/null
+++ b/tests/d_punch/script
@@ -0,0 +1,129 @@
+if test -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fy
+OUT=$test_name.log
+if [ -f $test_dir/expect.gz ]; then
+	EXP=$test_name.tmp
+	gunzip < $test_dir/expect.gz > $EXP1
+else
+	EXP=$test_dir/expect
+fi
+
+cp /dev/null $OUT
+
+cat > $TMPFILE.conf << ENDL
+[fs_types]
+ext4 = {
+        base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr,^has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,64bit
+        blocksize = 1024
+        inode_size = 256
+        inode_ratio = 16384
+}
+ENDL
+MKE2FS_CONFIG=$TMPFILE.conf $MKE2FS -F -o Linux -b 1024 -O ^bigalloc -T ext4 $TMPFILE 65536 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT 2>&1
+rm -rf $TMPFILE.conf
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+echo "debugfs write files" >> $OUT
+make_file() {
+	name="$1"
+	start="$2"
+	flag="$3"
+
+	cat << ENDL
+write /dev/null $name
+fallocate /$name 0 39
+punch /$name 10 10
+punch /$name 13 13
+punch /$name 26 26
+punch /$name 29 29
+ENDL
+}
+
+#Files we create:
+# a: punch a 40k file
+# b*: punch sparse file starting at b*
+# c*: punch spare file ending at c*
+# d: midcluster to midcluster, surrounding sparse
+# e: partial middle cluster alloc
+# f: one big file
+base=5000
+cat > $TMPFILE.cmd << ENDL
+write /dev/null a
+fallocate /a 0 39
+punch /a 0 39
+ENDL
+echo "ex /a" >> $TMPFILE.cmd2
+
+make_file sample $base --uninit >> $TMPFILE.cmd
+echo "ex /sample" >> $TMPFILE.cmd2
+base=10000
+
+for i in 8 9 10 11 12 13 14 15; do
+	make_file b$i $(($base + (40 * ($i - 8)))) --uninit >> $TMPFILE.cmd
+	echo "punch /b$i $i 39" >> $TMPFILE.cmd
+	echo "ex /b$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file c$i $(($base + 320 + (40 * ($i - 24)))) --uninit >> $TMPFILE.cmd
+	echo "punch /c$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /c$i" >> $TMPFILE.cmd2
+done
+
+make_file d $(($base + 640)) --uninit >> $TMPFILE.cmd
+echo "punch /d 4 35" >> $TMPFILE.cmd
+echo "ex /d" >> $TMPFILE.cmd2
+
+make_file e $(($base + 680)) --uninit >> $TMPFILE.cmd
+echo "punch /e 19 20" >> $TMPFILE.cmd
+echo "ex /e" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null f
+sif /f size 1024
+eo /f
+set_bmap --uninit 0 9000
+ec
+sif /f blocks 2
+setb 9000
+fallocate /f 0 8999
+punch /f 1 8998
+ENDL
+echo "ex /f" >> $TMPFILE.cmd2
+
+$DEBUGFS_EXE -w $TMPFILE -f $TMPFILE.cmd > /dev/null 2>&1
+$DEBUGFS_EXE $TMPFILE -f $TMPFILE.cmd2 >> $OUT.new 2>&1
+sed -f $cmd_dir/filter.sed < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE.cmd $TMPFILE.cmd2
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+rm -f $TMPFILE
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f $test_name.tmp
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
diff --git a/tests/d_punch_bigalloc/expect b/tests/d_punch_bigalloc/expect
new file mode 100644
index 0000000..adbef89
--- /dev/null
+++ b/tests/d_punch_bigalloc/expect
@@ -0,0 +1,142 @@
+
+Warning: the bigalloc feature is still under development
+See https://ext4.wiki.kernel.org/index.php/Bigalloc for more information
+
+Creating filesystem with 65536 1k blocks and 4096 inodes
+
+Allocating group tables:    \b\b\bdone                            
+Writing inode tables:    \b\b\bdone                            
+Writing superblocks and filesystem accounting information:    \b\b\bdone
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 11/4096 files (9.1% non-contiguous), 1144/65536 blocks
+Exit status is 0
+debugfs write files
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+debugfs: ex /sample
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1184              0
+ 1/ 1   1/  5     0 -     9  1144 -  1153     10 Uninit
+ 1/ 1   2/  5    11 -    12  1155 -  1156      2 Uninit
+ 1/ 1   3/  5    14 -    25  1158 -  1169     12 Uninit
+ 1/ 1   4/  5    27 -    28  1171 -  1172      2 Uninit
+ 1/ 1   5/  5    30 -    39  1174 -  1183     10 Uninit
+debugfs: ex /b8
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1232              0
+ 1/ 1   1/  1     0 -     7  1192 -  1199      8 Uninit
+debugfs: ex /b9
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1248              0
+ 1/ 1   1/  1     0 -     8  1200 -  1208      9 Uninit
+debugfs: ex /b10
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1272              0
+ 1/ 1   1/  1     0 -     9  1216 -  1225     10 Uninit
+debugfs: ex /b11
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1296              0
+ 1/ 1   1/  2     0 -     7  1240 -  1247      8 Uninit
+ 1/ 1   2/  2     8 -     9  1256 -  1257      2 Uninit
+debugfs: ex /b12
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1320              0
+ 1/ 1   1/  3     0 -     7  1264 -  1271      8 Uninit
+ 1/ 1   2/  3     8 -     9  1280 -  1281      2 Uninit
+ 1/ 1   3/  3    11 -    11  1283 -  1283      1 Uninit
+debugfs: ex /b13
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1344              0
+ 1/ 1   1/  3     0 -     7  1288 -  1295      8 Uninit
+ 1/ 1   2/  3     8 -     9  1304 -  1305      2 Uninit
+ 1/ 1   3/  3    11 -    12  1307 -  1308      2 Uninit
+debugfs: ex /b14
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1368              0
+ 1/ 1   1/  3     0 -     7  1312 -  1319      8 Uninit
+ 1/ 1   2/  3     8 -     9  1328 -  1329      2 Uninit
+ 1/ 1   3/  3    11 -    12  1331 -  1332      2 Uninit
+debugfs: ex /b15
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1392              0
+ 1/ 1   1/  4     0 -     7  1336 -  1343      8 Uninit
+ 1/ 1   2/  4     8 -     9  1352 -  1353      2 Uninit
+ 1/ 1   3/  4    11 -    12  1355 -  1356      2 Uninit
+ 1/ 1   4/  4    14 -    14  1358 -  1358      1 Uninit
+debugfs: ex /c24
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    25 - 4294967295  1416         4294967271
+ 1/ 1   1/  3    25 -    25  1401 -  1401      1 Uninit
+ 1/ 1   2/  3    27 -    28  1403 -  1404      2 Uninit
+ 1/ 1   3/  3    30 -    39  1406 -  1415     10 Uninit
+debugfs: ex /c25
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    27 - 4294967295  1440         4294967269
+ 1/ 1   1/  2    27 -    28  1427 -  1428      2 Uninit
+ 1/ 1   2/  2    30 -    39  1430 -  1439     10 Uninit
+debugfs: ex /c26
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    27 - 4294967295  1464         4294967269
+ 1/ 1   1/  2    27 -    28  1451 -  1452      2 Uninit
+ 1/ 1   2/  2    30 -    39  1454 -  1463     10 Uninit
+debugfs: ex /c27
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    28 - 4294967295  1488         4294967268
+ 1/ 1   1/  2    28 -    28  1476 -  1476      1 Uninit
+ 1/ 1   2/  2    30 -    39  1478 -  1487     10 Uninit
+debugfs: ex /c28
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    30 - 4294967295  1512         4294967266
+ 1/ 1   1/  1    30 -    39  1502 -  1511     10 Uninit
+debugfs: ex /c29
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    30 - 4294967295  1536         4294967266
+ 1/ 1   1/  1    30 -    39  1526 -  1535     10 Uninit
+debugfs: ex /c30
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    31 - 4294967295  1560         4294967265
+ 1/ 1   1/  1    31 -    39  1551 -  1559      9 Uninit
+debugfs: ex /c31
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    32 - 4294967295  1584         4294967264
+ 1/ 1   1/  1    32 -    39  1576 -  1583      8 Uninit
+debugfs: ex /d
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1600              0
+ 1/ 1   1/  2     0 -     3  1360 -  1363      4 Uninit
+ 1/ 1   2/  2    36 -    39  1596 -  1599      4 Uninit
+debugfs: ex /e
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1624              0
+ 1/ 1   1/  8     0 -     9  1376 -  1385     10 Uninit
+ 1/ 1   2/  8    11 -    12  1387 -  1388      2 Uninit
+ 1/ 1   3/  8    14 -    15  1390 -  1391      2 Uninit
+ 1/ 1   4/  8    16 -    18  1568 -  1570      3 Uninit
+ 1/ 1   5/  8    21 -    23  1573 -  1575      3 Uninit
+ 1/ 1   6/  8    24 -    25  1608 -  1609      2 Uninit
+ 1/ 1   7/  8    27 -    28  1611 -  1612      2 Uninit
+ 1/ 1   8/  8    30 -    39  1614 -  1623     10 Uninit
+debugfs: ex /f
+Level Entries       Logical      Physical Length Flags
+ 0/ 0   1/  2     0 -     0  9000 -  9000      1 Uninit
+ 0/ 0   2/  2  8999 -  8999 17999 - 17999      1 Uninit
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Free blocks count wrong for group #0 (7987, counted=7986).
+Fix? yes
+
+Free blocks count wrong (63896, counted=63888).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 32/4096 files (43.8% non-contiguous), 1648/65536 blocks
+Exit status is 1
diff --git a/tests/d_punch_bigalloc/name b/tests/d_punch_bigalloc/name
new file mode 100644
index 0000000..6d61ebe
--- /dev/null
+++ b/tests/d_punch_bigalloc/name
@@ -0,0 +1 @@
+punch sparse files and big files with bigalloc
diff --git a/tests/d_punch_bigalloc/script b/tests/d_punch_bigalloc/script
new file mode 100644
index 0000000..5784154
--- /dev/null
+++ b/tests/d_punch_bigalloc/script
@@ -0,0 +1,130 @@
+if test -x $DEBUGFS_EXE; then
+
+FSCK_OPT=-fy
+OUT=$test_name.log
+if [ -f $test_dir/expect.gz ]; then
+	EXP=$test_name.tmp
+	gunzip < $test_dir/expect.gz > $EXP1
+else
+	EXP=$test_dir/expect
+fi
+
+cp /dev/null $OUT
+
+cat > $TMPFILE.conf << ENDL
+[fs_types]
+ext4 = {
+	cluster_size = 8192
+        base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr,^has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,64bit
+        blocksize = 1024
+        inode_size = 256
+        inode_ratio = 16384
+}
+ENDL
+MKE2FS_CONFIG=$TMPFILE.conf $MKE2FS -F -o Linux -b 1024 -O bigalloc -T ext4 $TMPFILE 65536 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT 2>&1
+rm -rf $TMPFILE.conf
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+echo "debugfs write files" >> $OUT
+make_file() {
+	name="$1"
+	start="$2"
+	flag="$3"
+
+	cat << ENDL
+write /dev/null $name
+fallocate /$name 0 39
+punch /$name 10 10
+punch /$name 13 13
+punch /$name 26 26
+punch /$name 29 29
+ENDL
+}
+
+#Files we create:
+# a: punch a 40k file
+# b*: punch sparse file starting at b*
+# c*: punch spare file ending at c*
+# d: midcluster to midcluster, surrounding sparse
+# e: partial middle cluster alloc
+# f: one big file
+base=5000
+cat > $TMPFILE.cmd << ENDL
+write /dev/null a
+fallocate /a 0 39
+punch /a 0 39
+ENDL
+echo "ex /a" >> $TMPFILE.cmd2
+
+make_file sample $base --uninit >> $TMPFILE.cmd
+echo "ex /sample" >> $TMPFILE.cmd2
+base=10000
+
+for i in 8 9 10 11 12 13 14 15; do
+	make_file b$i $(($base + (40 * ($i - 8)))) --uninit >> $TMPFILE.cmd
+	echo "punch /b$i $i 39" >> $TMPFILE.cmd
+	echo "ex /b$i" >> $TMPFILE.cmd2
+done
+
+for i in 24 25 26 27 28 29 30 31; do
+	make_file c$i $(($base + 320 + (40 * ($i - 24)))) --uninit >> $TMPFILE.cmd
+	echo "punch /c$i 0 $i" >> $TMPFILE.cmd
+	echo "ex /c$i" >> $TMPFILE.cmd2
+done
+
+make_file d $(($base + 640)) --uninit >> $TMPFILE.cmd
+echo "punch /d 4 35" >> $TMPFILE.cmd
+echo "ex /d" >> $TMPFILE.cmd2
+
+make_file e $(($base + 680)) --uninit >> $TMPFILE.cmd
+echo "punch /e 19 20" >> $TMPFILE.cmd
+echo "ex /e" >> $TMPFILE.cmd2
+
+cat >> $TMPFILE.cmd << ENDL
+write /dev/null f
+sif /f size 1024
+eo /f
+set_bmap --uninit 0 9000
+ec
+sif /f blocks 16
+setb 9000
+fallocate /f 0 8999
+punch /f 1 8998
+ENDL
+echo "ex /f" >> $TMPFILE.cmd2
+
+$DEBUGFS_EXE -w $TMPFILE -f $TMPFILE.cmd > /dev/null 2>&1
+$DEBUGFS_EXE $TMPFILE -f $TMPFILE.cmd2 >> $OUT.new 2>&1
+sed -f $cmd_dir/filter.sed < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE.cmd $TMPFILE.cmd2
+
+$FSCK -fy -N test_filesys $TMPFILE > $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+rm -f $OUT.new
+
+rm -f $TMPFILE
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+	rm -f $test_name.tmp
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
+
+else #if test -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi

[PATCH 37/39] fuse2fs: translate ACL structures

[Darrick J. Wong]

Translate "native" ACL structures into ext4 ACL structures when
reading or writing the ACL EAs.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 configure       |    5 +
 configure.in    |    8 +-
 lib/config.h.in |    3 +
 misc/fuse2fs.c  |  263 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 271 insertions(+), 8 deletions(-)


diff --git a/configure b/configure
index 31efdc9..364cf5e 100755
--- a/configure
+++ b/configure
@@ -12418,7 +12418,7 @@ fi
 done
 
 fi
-for ac_header in  	dirent.h 	errno.h 	execinfo.h 	getopt.h 	malloc.h 	mntent.h 	paths.h 	semaphore.h 	setjmp.h 	signal.h 	stdarg.h 	stdint.h 	stdlib.h 	termios.h 	termio.h 	unistd.h 	utime.h 	attr/xattr.h 	linux/falloc.h 	linux/fd.h 	linux/major.h 	linux/loop.h 	net/if_dl.h 	netinet/in.h 	sys/disklabel.h 	sys/disk.h 	sys/file.h 	sys/ioctl.h 	sys/mkdev.h 	sys/mman.h 	sys/mount.h 	sys/prctl.h 	sys/resource.h 	sys/select.h 	sys/socket.h 	sys/sockio.h 	sys/stat.h 	sys/syscall.h 	sys/sysctl.h 	sys/sysmacros.h 	sys/time.h 	sys/types.h 	sys/un.h 	sys/wait.h
+for ac_header in  	dirent.h 	errno.h 	execinfo.h 	getopt.h 	malloc.h 	mntent.h 	paths.h 	semaphore.h 	setjmp.h 	signal.h 	stdarg.h 	stdint.h 	stdlib.h 	termios.h 	termio.h 	unistd.h 	utime.h 	attr/xattr.h 	linux/falloc.h 	linux/fd.h 	linux/major.h 	linux/loop.h 	net/if_dl.h 	netinet/in.h 	sys/acl.h 	sys/disklabel.h 	sys/disk.h 	sys/file.h 	sys/ioctl.h 	sys/mkdev.h 	sys/mman.h 	sys/mount.h 	sys/prctl.h 	sys/resource.h 	sys/select.h 	sys/socket.h 	sys/sockio.h 	sys/stat.h 	sys/syscall.h 	sys/sysctl.h 	sys/sysmacros.h 	sys/time.h 	sys/types.h 	sys/un.h 	sys/wait.h
 do :
   as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
 ac_fn_c_check_header_mongrel "$LINENO" "$ac_header" "$as_ac_Header" "$ac_includes_default"
@@ -13209,6 +13209,7 @@ else
 do :
   as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
 ac_fn_c_check_header_compile "$LINENO" "$ac_header" "$as_ac_Header" "#define _FILE_OFFSET_BITS	64
+#define FUSE_USE_VERSION 29
 "
 if eval test \"x\$"$as_ac_Header"\" = x"yes"; then :
   cat >>confdefs.h <<_ACEOF
@@ -13227,6 +13228,7 @@ done
 
 	cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
+#define FUSE_USE_VERSION 29
 #ifdef __linux__
 #include <linux/fs.h>
 #include <linux/falloc.h>
@@ -13346,6 +13348,7 @@ else
 do :
   as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
 ac_fn_c_check_header_compile "$LINENO" "$ac_header" "$as_ac_Header" "#define _FILE_OFFSET_BITS	64
+#define FUSE_USE_VERSION 29
 #ifdef __linux__
 # include <linux/fs.h>
 # include <linux/falloc.h>
diff --git a/configure.in b/configure.in
index eb4b6ad..9cb6aaf 100644
--- a/configure.in
+++ b/configure.in
@@ -935,6 +935,7 @@ AC_CHECK_HEADERS(m4_flatten([
 	linux/loop.h
 	net/if_dl.h
 	netinet/in.h
+	sys/acl.h
 	sys/disklabel.h
 	sys/disk.h
 	sys/file.h
@@ -1177,10 +1178,12 @@ then
 else
 	AC_CHECK_HEADERS([pthread.h fuse.h], [],
 [AC_MSG_FAILURE([Cannot find fuse2fs headers.])],
-[#define _FILE_OFFSET_BITS	64])
+[#define _FILE_OFFSET_BITS	64
+#define FUSE_USE_VERSION 29])
 
 	AC_PREPROC_IFELSE(
-[AC_LANG_PROGRAM([[#ifdef __linux__
+[AC_LANG_PROGRAM([[#define FUSE_USE_VERSION 29
+#ifdef __linux__
 #include <linux/fs.h>
 #include <linux/falloc.h>
 #include <linux/xattr.h>
@@ -1195,6 +1198,7 @@ fi
 ,
 AC_CHECK_HEADERS([pthread.h fuse.h], [], [FUSE_CMT="#"],
 [#define _FILE_OFFSET_BITS	64
+#define FUSE_USE_VERSION 29
 #ifdef __linux__
 # include <linux/fs.h>
 # include <linux/falloc.h>
diff --git a/lib/config.h.in b/lib/config.h.in
index b10e91d..7005940 100644
--- a/lib/config.h.in
+++ b/lib/config.h.in
@@ -467,6 +467,9 @@
 /* Define to 1 if you have the `sysconf' function. */
 #undef HAVE_SYSCONF
 
+/* Define to 1 if you have the <sys/acl.h> header file. */
+#undef HAVE_SYS_ACL_H
+
 /* Define to 1 if you have the <sys/disklabel.h> header file. */
 #undef HAVE_SYS_DISKLABEL_H
 
diff --git a/misc/fuse2fs.c b/misc/fuse2fs.c
index bc667d7..ec0b743 100644
--- a/misc/fuse2fs.c
+++ b/misc/fuse2fs.c
@@ -18,9 +18,15 @@
 # include <linux/falloc.h>
 # include <linux/xattr.h>
 # define FUSE_PLATFORM_OPTS	",nonempty,big_writes"
+# ifdef HAVE_SYS_ACL_H
+#  define TRANSLATE_LINUX_ACLS
+# endif
 #else
 # define FUSE_PLATFORM_OPTS	""
 #endif
+#ifdef TRANSLATE_LINUX_ACLS
+# include <sys/acl.h>
+#endif
 #include <sys/ioctl.h>
 #include <unistd.h>
 #include <fuse.h>
@@ -85,6 +91,200 @@ static ext2_filsys global_fs; /* Try not to use this directly */
 
 errcode_t ext2fs_run_ext3_journal(ext2_filsys *fs);
 
+/* ACL translation stuff */
+#ifdef TRANSLATE_LINUX_ACLS
+/*
+ * Copied from acl_ea.h in libacl source; ACLs have to be sent to and from fuse
+ * in this format... at least on Linux.
+ */
+#define ACL_EA_ACCESS		"system.posix_acl_access"
+#define ACL_EA_DEFAULT		"system.posix_acl_default"
+
+#define ACL_EA_VERSION		0x0002
+
+typedef struct {
+	u_int16_t	e_tag;
+	u_int16_t	e_perm;
+	u_int32_t	e_id;
+} acl_ea_entry;
+
+typedef struct {
+	u_int32_t	a_version;
+	acl_ea_entry	a_entries[0];
+} acl_ea_header;
+
+static inline size_t acl_ea_size(int count)
+{
+	return sizeof(acl_ea_header) + count * sizeof(acl_ea_entry);
+}
+
+static inline int acl_ea_count(size_t size)
+{
+	if (size < sizeof(acl_ea_header))
+		return -1;
+	size -= sizeof(acl_ea_header);
+	if (size % sizeof(acl_ea_entry))
+		return -1;
+	return size / sizeof(acl_ea_entry);
+}
+
+/*
+ * ext4 ACL structures, copied from fs/ext4/acl.h.
+ */
+#define EXT4_ACL_VERSION	0x0001
+
+typedef struct {
+	__u16		e_tag;
+	__u16		e_perm;
+	__u32		e_id;
+} ext4_acl_entry;
+
+typedef struct {
+	__u16		e_tag;
+	__u16		e_perm;
+} ext4_acl_entry_short;
+
+typedef struct {
+	__u32		a_version;
+} ext4_acl_header;
+
+static inline size_t ext4_acl_size(int count)
+{
+	if (count <= 4) {
+		return sizeof(ext4_acl_header) +
+		       count * sizeof(ext4_acl_entry_short);
+	} else {
+		return sizeof(ext4_acl_header) +
+		       4 * sizeof(ext4_acl_entry_short) +
+		       (count - 4) * sizeof(ext4_acl_entry);
+	}
+}
+
+static inline int ext4_acl_count(size_t size)
+{
+	ssize_t s;
+
+	size -= sizeof(ext4_acl_header);
+	s = size - 4 * sizeof(ext4_acl_entry_short);
+	if (s < 0) {
+		if (size % sizeof(ext4_acl_entry_short))
+			return -1;
+		return size / sizeof(ext4_acl_entry_short);
+	} else {
+		if (s % sizeof(ext4_acl_entry))
+			return -1;
+		return s / sizeof(ext4_acl_entry) + 4;
+	}
+}
+
+static errcode_t fuse_to_ext4_acl(acl_ea_header *facl, size_t facl_sz,
+				  ext4_acl_header **eacl, size_t *eacl_sz)
+{
+	int i, facl_count;
+	ext4_acl_header *h;
+	size_t h_sz;
+	ext4_acl_entry *e;
+	acl_ea_entry *a;
+	void *hptr;
+	errcode_t err;
+
+	facl_count = acl_ea_count(facl_sz);
+	h_sz = ext4_acl_size(facl_count);
+	if (facl_count < 0 || facl->a_version != ACL_EA_VERSION)
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	err = ext2fs_get_mem(h_sz, &h);
+	if (err)
+		return err;
+
+	h->a_version = ext2fs_cpu_to_le32(EXT4_ACL_VERSION);
+	hptr = h + 1;
+	for (i = 0, a = facl->a_entries; i < facl_count; i++, a++) {
+		e = hptr;
+		e->e_tag = ext2fs_cpu_to_le16(a->e_tag);
+		e->e_perm = ext2fs_cpu_to_le16(a->e_perm);
+
+		switch (a->e_tag) {
+		case ACL_USER:
+		case ACL_GROUP:
+			e->e_id = ext2fs_cpu_to_le32(a->e_id);
+			hptr += sizeof(ext4_acl_entry);
+			break;
+		case ACL_USER_OBJ:
+		case ACL_GROUP_OBJ:
+		case ACL_MASK:
+		case ACL_OTHER:
+			hptr += sizeof(ext4_acl_entry_short);
+			break;
+		default:
+			err = EXT2_ET_INVALID_ARGUMENT;
+			goto out;
+		}
+	}
+
+	*eacl = h;
+	*eacl_sz = h_sz;
+	return err;
+out:
+	ext2fs_free_mem(&h);
+	return err;
+}
+
+static errcode_t ext4_to_fuse_acl(acl_ea_header **facl, size_t *facl_sz,
+				  ext4_acl_header *eacl, size_t eacl_sz)
+{
+	int i, eacl_count;
+	acl_ea_header *f;
+	ext4_acl_entry *e;
+	acl_ea_entry *a;
+	size_t f_sz;
+	void *hptr;
+	errcode_t err;
+
+	eacl_count = ext4_acl_count(eacl_sz);
+	f_sz = acl_ea_size(eacl_count);
+	if (eacl_count < 0 ||
+	    eacl->a_version != ext2fs_cpu_to_le32(EXT4_ACL_VERSION))
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	err = ext2fs_get_mem(f_sz, &f);
+	if (err)
+		return err;
+
+	f->a_version = ACL_EA_VERSION;
+	hptr = eacl + 1;
+	for (i = 0, a = f->a_entries; i < eacl_count; i++, a++) {
+		e = hptr;
+		a->e_tag = ext2fs_le16_to_cpu(e->e_tag);
+		a->e_perm = ext2fs_le16_to_cpu(e->e_perm);
+
+		switch (a->e_tag) {
+		case ACL_USER:
+		case ACL_GROUP:
+			a->e_id = ext2fs_le32_to_cpu(e->e_id);
+			hptr += sizeof(ext4_acl_entry);
+			break;
+		case ACL_USER_OBJ:
+		case ACL_GROUP_OBJ:
+		case ACL_MASK:
+		case ACL_OTHER:
+			hptr += sizeof(ext4_acl_entry_short);
+			break;
+		default:
+			err = EXT2_ET_INVALID_ARGUMENT;
+			goto out;
+		}
+	}
+
+	*facl = f;
+	*facl_sz = f_sz;
+	return err;
+out:
+	ext2fs_free_mem(&f);
+	return err;
+}
+#endif /* TRANSLATE_LINUX_ACLS */
+
 /*
  * ext2_file_t contains a struct inode, so we can't leave files open.
  * Use this as a proxy instead.
@@ -2143,6 +2343,30 @@ static int op_statfs(const char *path, struct statvfs *buf)
 	return 0;
 }
 
+typedef errcode_t (*xattr_xlate_get)(void **cooked_buf, size_t *cooked_sz,
+				     const void *raw_buf, size_t raw_sz);
+typedef errcode_t (*xattr_xlate_set)(const void *cooked_buf, size_t cooked_sz,
+				     void **raw_buf, size_t *raw_sz);
+struct xattr_translate {
+	const char *prefix;
+	xattr_xlate_get get;
+	xattr_xlate_set set;
+};
+
+#define XATTR_TRANSLATOR(p, g, s) \
+	{.prefix = (p), \
+	 .get = (xattr_xlate_get)(g), \
+	 .set = (xattr_xlate_set)(s)}
+
+static struct xattr_translate xattr_translators[] = {
+#ifdef TRANSLATE_LINUX_ACLS
+	XATTR_TRANSLATOR(ACL_EA_ACCESS, ext4_to_fuse_acl, fuse_to_ext4_acl),
+	XATTR_TRANSLATOR(ACL_EA_DEFAULT, ext4_to_fuse_acl, fuse_to_ext4_acl),
+#endif
+	XATTR_TRANSLATOR(NULL, NULL, NULL),
+};
+#undef XATTR_TRANSLATOR
+
 static int op_getxattr(const char *path, const char *key, char *value,
 		       size_t len)
 {
@@ -2150,8 +2374,9 @@ static int op_getxattr(const char *path, const char *key, char *value,
 	struct fuse2fs *ff = (struct fuse2fs *)ctxt->private_data;
 	ext2_filsys fs;
 	struct ext2_xattr_handle *h;
-	void *ptr;
-	size_t plen;
+	struct xattr_translate *xt;
+	void *ptr, *cptr;
+	size_t plen, clen;
 	ext2_ino_t ino;
 	errcode_t err;
 	int ret = 0;
@@ -2194,6 +2419,17 @@ static int op_getxattr(const char *path, const char *key, char *value,
 		goto out2;
 	}
 
+	for (xt = xattr_translators; xt->prefix != NULL; xt++) {
+		if (strncmp(key, xt->prefix, strlen(xt->prefix)) == 0) {
+			err = xt->get(&cptr, &clen, ptr, plen);
+			if (err)
+				goto out3;
+			ext2fs_free_mem(&ptr);
+			ptr = cptr;
+			plen = clen;
+		}
+	}
+
 	if (!len) {
 		ret = plen;
 	} else if (len < plen) {
@@ -2203,6 +2439,7 @@ static int op_getxattr(const char *path, const char *key, char *value,
 		ret = plen;
 	}
 
+out3:
 	ext2fs_free_mem(&ptr);
 out2:
 	err = ext2fs_xattrs_close(&h);
@@ -2317,6 +2554,9 @@ static int op_setxattr(const char *path, const char *key, const char *value,
 	struct fuse2fs *ff = (struct fuse2fs *)ctxt->private_data;
 	ext2_filsys fs;
 	struct ext2_xattr_handle *h;
+	struct xattr_translate *xt;
+	void *cvalue;
+	size_t clen;
 	ext2_ino_t ino;
 	errcode_t err;
 	int ret = 0;
@@ -2356,19 +2596,32 @@ static int op_setxattr(const char *path, const char *key, const char *value,
 		goto out2;
 	}
 
-	err = ext2fs_xattr_set(h, key, value, len);
+	cvalue = (void *)value;
+	clen = len;
+	for (xt = xattr_translators; xt->prefix != NULL; xt++) {
+		if (strncmp(key, xt->prefix, strlen(xt->prefix)) == 0) {
+			err = xt->set(value, len, &cvalue, &clen);
+			if (err)
+				goto out3;
+		}
+	}
+
+	err = ext2fs_xattr_set(h, key, cvalue, clen);
 	if (err) {
 		ret = translate_error(fs, ino, err);
-		goto out2;
+		goto out3;
 	}
 
 	err = ext2fs_xattrs_write(h);
 	if (err) {
 		ret = translate_error(fs, ino, err);
-		goto out2;
+		goto out3;
 	}
 
 	ret = update_ctime(fs, ino, NULL);
+out3:
+	if (cvalue != value)
+		ext2fs_free_mem(&cvalue);
 out2:
 	err = ext2fs_xattrs_close(&h);
 	if (!ret && err)

[PATCH 38/39] fuse2fs: handle 64-bit dates correctly

[Darrick J. Wong]

Fix fuse2fs' interpretation of 64-bit date quantities to match the
kernel.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/fuse2fs.c |   31 ++++++++++++++++++++++---------
 1 file changed, 22 insertions(+), 9 deletions(-)


diff --git a/misc/fuse2fs.c b/misc/fuse2fs.c
index ec0b743..2746a27 100644
--- a/misc/fuse2fs.c
+++ b/misc/fuse2fs.c
@@ -353,15 +353,24 @@ static int __translate_error(ext2_filsys fs, errcode_t err, ext2_ino_t ino,
 
 static inline __u32 ext4_encode_extra_time(const struct timespec *time)
 {
-	return (sizeof(time->tv_sec) > 4 ?
-		(time->tv_sec >> 32) & EXT4_EPOCH_MASK : 0) |
-	       ((time->tv_nsec << EXT4_EPOCH_BITS) & EXT4_NSEC_MASK);
+	__u32 extra = sizeof(time->tv_sec) > 4 ?
+			((time->tv_sec - (__s32)time->tv_sec) >> 32) &
+			EXT4_EPOCH_MASK : 0;
+	return extra | (time->tv_nsec << EXT4_EPOCH_BITS);
 }
 
 static inline void ext4_decode_extra_time(struct timespec *time, __u32 extra)
 {
-	if (sizeof(time->tv_sec) > 4)
-		time->tv_sec |= (__u64)((extra) & EXT4_EPOCH_MASK) << 32;
+	if (sizeof(time->tv_sec) > 4 && (extra & EXT4_EPOCH_MASK)) {
+		__u64 extra_bits = extra & EXT4_EPOCH_MASK;
+		/*
+		 * Prior to kernel 3.14?, we had a broken decode function,
+		 * wherein we effectively did this:
+		 * if (extra_bits == 3)
+		 *     extra_bits = 0;
+		 */
+		time->tv_sec += extra_bits << 32;
+	}
 	time->tv_nsec = ((extra) & EXT4_NSEC_MASK) >> EXT4_EPOCH_BITS;
 }
 
@@ -387,7 +396,7 @@ do {									       \
 	(timespec)->tv_sec = (signed)((raw_inode)->xtime);		       \
 	if (EXT4_FITS_IN_INODE(raw_inode, xtime ## _extra))		       \
 		ext4_decode_extra_time((timespec),			       \
-				       raw_inode->xtime ## _extra);	       \
+				       (raw_inode)->xtime ## _extra);	       \
 	else								       \
 		(timespec)->tv_nsec = 0;				       \
 } while (0)
@@ -749,6 +758,7 @@ static int stat_inode(ext2_filsys fs, ext2_ino_t ino, struct stat *statbuf)
 	dev_t fakedev = 0;
 	errcode_t err;
 	int ret = 0;
+	struct timespec tv;
 
 	memset(&inode, 0, sizeof(inode));
 	err = ext2fs_read_inode_full(fs, ino, (struct ext2_inode *)&inode,
@@ -766,9 +776,12 @@ static int stat_inode(ext2_filsys fs, ext2_ino_t ino, struct stat *statbuf)
 	statbuf->st_size = EXT2_I_SIZE(&inode);
 	statbuf->st_blksize = fs->blocksize;
 	statbuf->st_blocks = blocks_from_inode(fs, &inode);
-	statbuf->st_atime = inode.i_atime;
-	statbuf->st_mtime = inode.i_mtime;
-	statbuf->st_ctime = inode.i_ctime;
+	EXT4_INODE_GET_XTIME(i_atime, &tv, &inode);
+	statbuf->st_atime = tv.tv_sec;
+	EXT4_INODE_GET_XTIME(i_mtime, &tv, &inode);
+	statbuf->st_mtime = tv.tv_sec;
+	EXT4_INODE_GET_XTIME(i_ctime, &tv, &inode);
+	statbuf->st_ctime = tv.tv_sec;
 	if (LINUX_S_ISCHR(inode.i_mode) ||
 	    LINUX_S_ISBLK(inode.i_mode)) {
 		if (inode.i_block[0])

[PATCH 39/39] fuse2fs: implement fallocate

[Darrick J. Wong]

Use the (new) ext2fs_fallocate() to fallocate file space.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/fuse2fs.c |   58 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)


diff --git a/misc/fuse2fs.c b/misc/fuse2fs.c
index 2746a27..466461e 100644
--- a/misc/fuse2fs.c
+++ b/misc/fuse2fs.c
@@ -3300,7 +3300,63 @@ out:
 static int fallocate_helper(struct fuse_file_info *fp, int mode, off_t offset,
 			    off_t len)
 {
-	return -EOPNOTSUPP;
+	struct fuse_context *ctxt = fuse_get_context();
+	struct fuse2fs *ff = (struct fuse2fs *)ctxt->private_data;
+	struct fuse2fs_file_handle *fh = (struct fuse2fs_file_handle *)fp->fh;
+	ext2_filsys fs;
+	struct ext2_inode_large inode;
+	blk64_t start, end;
+	__u64 fsize;
+	errcode_t err;
+	int flags;
+
+	FUSE2FS_CHECK_CONTEXT(ff);
+	fs = ff->fs;
+	FUSE2FS_CHECK_MAGIC(fs, fh, FUSE2FS_FILE_MAGIC);
+	start = offset / fs->blocksize;
+	end = (offset + len - 1) / fs->blocksize;
+	dbg_printf("%s: ino=%d mode=0x%x start=%jd end=%llu\n", __func__,
+		   fh->ino, mode, offset / fs->blocksize, end);
+	if (!fs_can_allocate(ff, len / fs->blocksize))
+		return -ENOSPC;
+
+	memset(&inode, 0, sizeof(inode));
+	err = ext2fs_read_inode_full(fs, fh->ino, (struct ext2_inode *)&inode,
+				     sizeof(inode));
+	if (err)
+		return err;
+	fsize = EXT2_I_SIZE(&inode);
+
+	/* Allocate a bunch of blocks */
+	flags = (mode & FL_KEEP_SIZE_FLAG ? 0 :
+			EXT2_FALLOCATE_INIT_BEYOND_EOF);
+	err = ext2fs_fallocate(fs, flags, fh->ino,
+			       (struct ext2_inode *)&inode,
+			       ~0ULL, start, end - start + 1);
+	if (err && err != EXT2_ET_BLOCK_ALLOC_FAIL)
+		return translate_error(fs, fh->ino, err);
+
+	/* Update i_size */
+	if (!(mode & FL_KEEP_SIZE_FLAG)) {
+		if (offset + len > fsize) {
+			err = ext2fs_inode_size_set(fs,
+						(struct ext2_inode *)&inode,
+						offset + len);
+			if (err)
+				return translate_error(fs, fh->ino, err);
+		}
+	}
+
+	err = update_mtime(fs, fh->ino, &inode);
+	if (err)
+		return err;
+
+	err = ext2fs_write_inode_full(fs, fh->ino, (struct ext2_inode *)&inode,
+				      sizeof(inode));
+	if (err)
+		return translate_error(fs, fh->ino, err);
+
+	return err;
 }
 
 static errcode_t clean_block_middle(ext2_filsys fs, ext2_ino_t ino,

Re: [PATCH 18/39] e2fsck: opportunistically recalculate unused inode count and inode_uninit after pass 5

[Darrick J. Wong]

On Sat, Oct 25, 2014 at 01:58:19PM -0700, Darrick J. Wong wrote:
> At the end of pass5, update the unused inode count to cover any newly
> unused inodes and set inode_uninit if none of the inodes in the group
> are used.  This will save us time on future fsck runs since we don't
> need to scan those inodes any more.
> 
> Since it's not an error for the unused inode count to be smaller than
> necessary, don't bother if the fs is mounted and we're not fixing
> anything else.
> 
> On an aged filesystem that has experienced at least one massive inode
> die-off, this has been shown to reduce future e2fsck times by around
> 10%.  The effect is diluted on less heavily used filesystems.

I didn't notice that ext2fs_set_gdt_csum() quietly takes care of this if the
filesystem's open for writes.  Therefore, I withdraw this patch and apologize
for the noise.

However, the BLOCK_UNINIT piece is still needed, so I'll issue a new patch #19.

--D

> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  e2fsck/pass5.c                          |   38 +++++++++++++++++++++++++++++++
>  e2fsck/problem.c                        |    5 ++++
>  e2fsck/problem.h                        |    3 ++
>  tests/f_bad_bbitmap/expect.1            |    3 ++
>  tests/f_bad_bmap_csum/expect.1          |    3 ++
>  tests/f_bad_gdt_csum/expect.1           |    7 +++++-
>  tests/f_bad_ibitmap/expect.1            |    3 ++
>  tests/f_bad_inode_csum/expect.1         |    3 ++
>  tests/f_idata_and_extents/expect.1      |    3 ++
>  tests/f_illitable_flexbg/expect.1       |   12 ++++++++++
>  tests/f_inlinedata_repair/expect.1      |    3 ++
>  tests/f_invalid_extent_symlink/expect.1 |    3 ++
>  tests/f_jnl_64bit/expect.1              |    6 +++++
>  tests/f_super_bad_csum/expect.1         |    8 +++++--
>  tests/f_unused_itable/expect.1          |    6 +++++
>  15 files changed, 103 insertions(+), 3 deletions(-)
> 
> 
> diff --git a/e2fsck/pass5.c b/e2fsck/pass5.c
> index 64fb7fe..1a225fb 100644
> --- a/e2fsck/pass5.c
> +++ b/e2fsck/pass5.c
> @@ -821,6 +821,44 @@ do_counts:
>  			} else
>  				ext2fs_unmark_valid(fs);
>  		}
> +
> +		/*
> +		 * Opportunistically update the unused inodes count
> +		 * and set inode_uninit so we can skip scanning unused
> +		 * inodes during future fsck runs.  However, it's not
> +		 * an error if the unused inode count is smaller
> +		 * than necessary, so don't bother the user if FS is
> +		 * mounted and we haven't fixed anything else, to
> +		 * minimize unnecessary reboots.
> +		 */
> +		if (((ctx->options & E2F_OPT_NO) ||
> +		     !(ctx->mount_flags & EXT2_MF_MOUNTED) ||
> +		     ext2fs_test_changed(fs)) &&
> +		    ext2fs_has_group_desc_csum(fs) &&
> +		    !ext2fs_bg_flags_test(fs, i, EXT2_BG_INODE_UNINIT)) {
> +			ext2_ino_t start, end, j;
> +
> +			start = (fs->super->s_inodes_per_group * i) + 1;
> +			end = (fs->super->s_inodes_per_group * (i + 1));
> +			pctx.group = i;
> +			pctx.ino = ext2fs_bg_itable_unused(fs, i);
> +			end -= pctx.ino;
> +			for (j = end; j >= start; j--)
> +				if (ext2fs_test_inode_bitmap2(ctx->inode_used_map, j))
> +					break;
> +
> +			pctx.ino2 = fs->super->s_inodes_per_group -
> +						(j - start + 1);
> +			if (pctx.ino != pctx.ino2 &&
> +			    fix_problem(ctx, PR_5_UNUSED_INODES_COUNT_GROUP,
> +					&pctx)) {
> +				if (pctx.ino2 == fs->super->s_inodes_per_group)
> +					ext2fs_bg_flags_set(fs, i,
> +							EXT2_BG_INODE_UNINIT);
> +				ext2fs_bg_itable_unused_set(fs, i, pctx.ino2);
> +				ext2fs_mark_super_dirty(fs);
> +			}
> +		}
>  	}
>  	if (free_inodes != fs->super->s_free_inodes_count) {
>  		pctx.group = -1;
> diff --git a/e2fsck/problem.c b/e2fsck/problem.c
> index a4da64b..9c2b2e6 100644
> --- a/e2fsck/problem.c
> +++ b/e2fsck/problem.c
> @@ -1899,6 +1899,11 @@ static struct e2fsck_problem problem_table[] = {
>  	  N_("@g %g @b @B does not match checksum.\n"),
>  	  PROMPT_FIX, PR_LATCH_BBITMAP | PR_PREEN_OK },
>  
> +	/* Unused inode count for group wrong */
> +	{ PR_5_UNUSED_INODES_COUNT_GROUP,
> +	  N_("Unused @i count wrong for @g #%g (%i, counted=%j).\n"),
> +	  PROMPT_FIX, PR_PREEN_OK | PR_NO_OK },
> +
>  	/* Post-Pass 5 errors */
>  
>  	/* Recreate journal if E2F_FLAG_JOURNAL_INODE flag is set */
> diff --git a/e2fsck/problem.h b/e2fsck/problem.h
> index 3c28166..6a71fc8 100644
> --- a/e2fsck/problem.h
> +++ b/e2fsck/problem.h
> @@ -1139,6 +1139,9 @@ struct problem_context {
>  /* Block bitmap checksum does not match */
>  #define PR_5_BLOCK_BITMAP_CSUM_INVALID	0x05001B
>  
> +/* Unused inode count for group wrong */
> +#define PR_5_UNUSED_INODES_COUNT_GROUP	0x05001C
> +
>  /*
>   * Post-Pass 5 errors
>   */
> diff --git a/tests/f_bad_bbitmap/expect.1 b/tests/f_bad_bbitmap/expect.1
> index 71ad1bb..af3f632 100644
> --- a/tests/f_bad_bbitmap/expect.1
> +++ b/tests/f_bad_bbitmap/expect.1
> @@ -9,6 +9,9 @@ Pass 5: Checking group summary information
>  Block bitmap differences:  -(8--10) -(12--17) -(19--31)
>  Fix? yes
>  
> +Unused inode count wrong for group #0 (0, counted=117).
> +Fix? yes
> +
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks
> diff --git a/tests/f_bad_bmap_csum/expect.1 b/tests/f_bad_bmap_csum/expect.1
> index ca8f77f..7616751 100644
> --- a/tests/f_bad_bmap_csum/expect.1
> +++ b/tests/f_bad_bmap_csum/expect.1
> @@ -6,6 +6,9 @@ Pass 2: Checking directory structure
>  Pass 3: Checking directory connectivity
>  Pass 4: Checking reference counts
>  Pass 5: Checking group summary information
> +Unused inode count wrong for group #0 (0, counted=117).
> +Fix? yes
> +
>  Inode bitmap differences: Group 0 inode bitmap does not match checksum.
>  FIXED.
>  Block bitmap differences: Group 0 block bitmap does not match checksum.
> diff --git a/tests/f_bad_gdt_csum/expect.1 b/tests/f_bad_gdt_csum/expect.1
> index e14c897..0f496e0 100644
> --- a/tests/f_bad_gdt_csum/expect.1
> +++ b/tests/f_bad_gdt_csum/expect.1
> @@ -6,5 +6,10 @@ Pass 2: Checking directory structure
>  Pass 3: Checking directory connectivity
>  Pass 4: Checking reference counts
>  Pass 5: Checking group summary information
> +Unused inode count wrong for group #0 (0, counted=117).
> +Fix? yes
> +
> +
> +test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks
> -Exit status is 0
> +Exit status is 1
> diff --git a/tests/f_bad_ibitmap/expect.1 b/tests/f_bad_ibitmap/expect.1
> index ea17523..81689e0 100644
> --- a/tests/f_bad_ibitmap/expect.1
> +++ b/tests/f_bad_ibitmap/expect.1
> @@ -9,6 +9,9 @@ Pass 5: Checking group summary information
>  Inode bitmap differences:  -(12--32)
>  Fix? yes
>  
> +Unused inode count wrong for group #0 (0, counted=117).
> +Fix? yes
> +
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks
> diff --git a/tests/f_bad_inode_csum/expect.1 b/tests/f_bad_inode_csum/expect.1
> index b3c628d..b28a349 100644
> --- a/tests/f_bad_inode_csum/expect.1
> +++ b/tests/f_bad_inode_csum/expect.1
> @@ -117,6 +117,9 @@ Fix? yes
>  Free inodes count wrong for group #0 (0, counted=32).
>  Fix? yes
>  
> +Unused inode count wrong for group #0 (0, counted=32).
> +Fix? yes
> +
>  Free inodes count wrong (0, counted=32).
>  Fix? yes
>  
> diff --git a/tests/f_idata_and_extents/expect.1 b/tests/f_idata_and_extents/expect.1
> index 7f7fbf3..aba063a 100644
> --- a/tests/f_idata_and_extents/expect.1
> +++ b/tests/f_idata_and_extents/expect.1
> @@ -26,6 +26,9 @@ Fix? yes
>  Free inodes count wrong for group #0 (105, counted=106).
>  Fix? yes
>  
> +Unused inode count wrong for group #0 (103, counted=105).
> +Fix? yes
> +
>  Free inodes count wrong (105, counted=106).
>  Fix? yes
>  
> diff --git a/tests/f_illitable_flexbg/expect.1 b/tests/f_illitable_flexbg/expect.1
> index fa42a0f..04ce65b 100644
> --- a/tests/f_illitable_flexbg/expect.1
> +++ b/tests/f_illitable_flexbg/expect.1
> @@ -18,6 +18,18 @@ Pass 5: Checking group summary information
>  Inode bitmap differences:  -(65--128)
>  Fix? yes
>  
> +Unused inode count wrong for group #0 (0, counted=52).
> +Fix? yes
> +
> +Unused inode count wrong for group #1 (0, counted=64).
> +Fix? yes
> +
> +Unused inode count wrong for group #2 (0, counted=64).
> +Fix? yes
> +
> +Unused inode count wrong for group #3 (0, counted=64).
> +Fix? yes
> +
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 12/256 files (0.0% non-contiguous), 31163/32768 blocks
> diff --git a/tests/f_inlinedata_repair/expect.1 b/tests/f_inlinedata_repair/expect.1
> index faba192..f7eebd3 100644
> --- a/tests/f_inlinedata_repair/expect.1
> +++ b/tests/f_inlinedata_repair/expect.1
> @@ -69,6 +69,9 @@ Pass 5: Checking group summary information
>  Directories count wrong for group #0 (7, counted=8).
>  Fix? yes
>  
> +Unused inode count wrong for group #0 (90, counted=91).
> +Fix? yes
> +
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 28/128 files (0.0% non-contiguous), 18/512 blocks
> diff --git a/tests/f_invalid_extent_symlink/expect.1 b/tests/f_invalid_extent_symlink/expect.1
> index 7bda0b7..cbb2ac5 100644
> --- a/tests/f_invalid_extent_symlink/expect.1
> +++ b/tests/f_invalid_extent_symlink/expect.1
> @@ -6,6 +6,9 @@ Clear? yes
>  Pass 3: Checking directory connectivity
>  Pass 4: Checking reference counts
>  Pass 5: Checking group summary information
> +Unused inode count wrong for group #0 (4, counted=5).
> +Fix? yes
> +
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 11/16 files (9.1% non-contiguous), 21/100 blocks
> diff --git a/tests/f_jnl_64bit/expect.1 b/tests/f_jnl_64bit/expect.1
> index e360e2f..915076b 100644
> --- a/tests/f_jnl_64bit/expect.1
> +++ b/tests/f_jnl_64bit/expect.1
> @@ -7,6 +7,12 @@ Pass 5: Checking group summary information
>  Free blocks count wrong (14059, counted=12712).
>  Fix? yes
>  
> +Unused inode count wrong for group #0 (0, counted=2876).
> +Fix? yes
> +
> +Unused inode count wrong for group #1 (1395, counted=3958).
> +Fix? yes
> +
>  Free inodes count wrong (8181, counted=6834).
>  Fix? yes
>  
> diff --git a/tests/f_super_bad_csum/expect.1 b/tests/f_super_bad_csum/expect.1
> index 25ced5c..a606fab 100644
> --- a/tests/f_super_bad_csum/expect.1
> +++ b/tests/f_super_bad_csum/expect.1
> @@ -5,8 +5,12 @@ Pass 2: Checking directory structure
>  Pass 3: Checking directory connectivity
>  Pass 4: Checking reference counts
>  Pass 5: Checking group summary information
> -Inode bitmap differences: Group 1 inode bitmap does not match checksum.
> -FIXED.
> +Unused inode count wrong for group #0 (0, counted=501).
> +Fix? yes
> +
> +Unused inode count wrong for group #1 (0, counted=512).
> +Fix? yes
> +
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 11/1024 files (0.0% non-contiguous), 1557/16384 blocks
> diff --git a/tests/f_unused_itable/expect.1 b/tests/f_unused_itable/expect.1
> index a4da987..fbc6538 100644
> --- a/tests/f_unused_itable/expect.1
> +++ b/tests/f_unused_itable/expect.1
> @@ -19,9 +19,15 @@ Pass 5: Checking group summary information
>  Free inodes count wrong for group #0 (53, counted=51).
>  Fix? yes
>  
> +Unused inode count wrong for group #0 (0, counted=51).
> +Fix? yes
> +
>  Free inodes count wrong for group #1 (64, counted=58).
>  Fix? yes
>  
> +Unused inode count wrong for group #1 (0, counted=58).
> +Fix? yes
> +
>  
>  test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
>  test_filesys: 19/128 files (0.0% non-contiguous), 165/1000 blocks
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 19/39] libext2fs: set BLOCK_UNINIT for non-last blockgroups if all blocks are free

[Darrick J. Wong]

Set BLOCK_UNINIT in any group whose blocks are all unused, so long as
it isn't the last group.  This helps us speed up future e2fsck runs
and mounts because we don't need to read or checksum block bitmaps for
these groups.

v2: Take care of this in the library, since e2fsck always updates these
summary counts.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 lib/ext2fs/csum.c |    5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/ext2fs/csum.c b/lib/ext2fs/csum.c
index 12f6fe9..6dcefb9 100644
--- a/lib/ext2fs/csum.c
+++ b/lib/ext2fs/csum.c
@@ -854,6 +854,11 @@ errcode_t ext2fs_set_gdt_csum(ext2_filsys fs)
 		__u32 old_unused = ext2fs_bg_itable_unused(fs, i);
 		__u32 old_flags = ext2fs_bg_flags(fs, i);
 		__u32 old_free_inodes_count = ext2fs_bg_free_inodes_count(fs, i);
+		__u32 old_free_blocks_count = ext2fs_bg_free_blocks_count(fs, i);
+
+		if (old_free_blocks_count == sb->s_blocks_per_group &&
+		    i != fs->group_desc_count - 1)
+			ext2fs_bg_flags_set(fs, i, EXT2_BG_BLOCK_UNINIT);
 
 		if (old_free_inodes_count == sb->s_inodes_per_group) {
 			ext2fs_bg_flags_set(fs, i, EXT2_BG_INODE_UNINIT);

[PATCH v2 13/39] resize2fs: set bg flags and unused inode count when resizing

[Darrick J. Wong]

Recalculate the unused inode count and the block/inode uninit flags
when resizing a filesystem.  This can speed up future e2fsck runs
considerably and will reduce mount times.

v2: Use a library function to take care of this.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 resize/resize2fs.c              |    4 ++++
 tests/r_fixup_lastbg/expect     |   36 ++++++++++++++++++++++++++++++++++++
 tests/r_fixup_lastbg/script     |   37 +++++++++++++++++++++++++++++++++++++
 tests/r_fixup_lastbg_big/expect |   39 +++++++++++++++++++++++++++++++++++++++
 tests/r_fixup_lastbg_big/script |   37 +++++++++++++++++++++++++++++++++++++
 5 files changed, 153 insertions(+)
 create mode 100644 tests/r_fixup_lastbg/expect
 create mode 100755 tests/r_fixup_lastbg/script
 create mode 100644 tests/r_fixup_lastbg_big/expect
 create mode 100755 tests/r_fixup_lastbg_big/script

diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index a8a6850..6d96a35 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -198,6 +198,10 @@ errcode_t resize_fs(ext2_filsys fs, blk64_t *new_size, int flags,
 	if (retval)
 		goto errout;
 
+	retval = ext2fs_set_gdt_csum(rfs->new_fs);
+	if (retval)
+		goto errout;
+
 	rfs->new_fs->super->s_state &= ~EXT2_ERROR_FS;
 	rfs->new_fs->flags &= ~EXT2_FLAG_MASTER_SB_ONLY;
 
diff --git a/tests/r_fixup_lastbg/expect b/tests/r_fixup_lastbg/expect
new file mode 100644
index 0000000..f9945b3
--- /dev/null
+++ b/tests/r_fixup_lastbg/expect
@@ -0,0 +1,36 @@
+Creating filesystem with 20000 1k blocks and 1248 inodes
+Superblock backups stored on blocks: 
+	8193
+
+Allocating group tables:    \b\b\bdone                            
+Writing inode tables:    \b\b\bdone                            
+Creating journal (1024 blocks): done
+Writing superblocks and filesystem accounting information:    \b\b\bdone
+
+Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Group 2: (Blocks 16385-19999)
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Resizing the filesystem on test.img to 20004 (1k) blocks.
+The filesystem on test.img is now 20004 (1k) blocks long.
+
+Group 2: (Blocks 16385-20003) [INODE_UNINIT]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3619 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-20003
+  Free inodes: 833-1248
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 11/1248 files (0.0% non-contiguous), 1517/20004 blocks
diff --git a/tests/r_fixup_lastbg/script b/tests/r_fixup_lastbg/script
new file mode 100755
index 0000000..ad223d4
--- /dev/null
+++ b/tests/r_fixup_lastbg/script
@@ -0,0 +1,37 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+test_description="fix up last bg when expanding within the last bg"
+
+EXP=$test_dir/expect
+OUT=$test_name.out
+LOG=$test_name.log
+E2FSCK=../e2fsck/e2fsck
+
+$MKE2FS -T ext4 -b 1024 -F -U 56d3ee50-8532-4f29-8181-d7c6ea4a94a6 $TMPFILE 20000 > $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$DEBUGFS -R "set_bg 2 itable_unused 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 flags 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 checksum 0xd318" -w $TMPFILE > /dev/null 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+dd if=/dev/zero of=$TMPFILE bs=1 count=1 seek=$((1024 * 20004)) conv=notrunc >> $OUT 2> /dev/null
+$RESIZE2FS_EXE -f -p $TMPFILE >> $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$E2FSCK -fy $TMPFILE >> $OUT 2>&1
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" < $OUT > $LOG
+rm -rf $OUT
+
+cmp -s $LOG $EXP
+RC=$?
+if [ $RC -eq 0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff -u $EXP $LOG > $test_name.failed
+fi
+
+unset EXP LOG OUT E2FSCK
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi
diff --git a/tests/r_fixup_lastbg_big/expect b/tests/r_fixup_lastbg_big/expect
new file mode 100644
index 0000000..1aa36fc
--- /dev/null
+++ b/tests/r_fixup_lastbg_big/expect
@@ -0,0 +1,39 @@
+Creating filesystem with 20000 1k blocks and 1248 inodes
+Superblock backups stored on blocks: 
+	8193
+
+Allocating group tables:    \b\b\bdone                            
+Writing inode tables:    \b\b\bdone                            
+Creating journal (1024 blocks): done
+Writing superblocks and filesystem accounting information:    \b\b\bdone
+
+Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Group 2: (Blocks 16385-19999)
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Resizing the filesystem on test.img to 40000 (1k) blocks.
+The filesystem on test.img is now 40000 (1k) blocks long.
+
+Group 2: (Blocks 16385-24576) [INODE_UNINIT, BLOCK_UNINIT]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  8192 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-24576
+  Free inodes: 833-1248
+Group 3: (Blocks 24577-32768) [INODE_UNINIT]
+  Backup superblock at 24577, Group descriptors at 24578-24578
+  Reserved GDT blocks at 24579-24656
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 11/2080 files (0.0% non-contiguous), 1809/40000 blocks
diff --git a/tests/r_fixup_lastbg_big/script b/tests/r_fixup_lastbg_big/script
new file mode 100755
index 0000000..4c4a351
--- /dev/null
+++ b/tests/r_fixup_lastbg_big/script
@@ -0,0 +1,37 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+test_description="fix up last bg when expanding beyond the last bg"
+
+EXP=$test_dir/expect
+OUT=$test_name.out
+LOG=$test_name.log
+E2FSCK=../e2fsck/e2fsck
+
+$MKE2FS -T ext4 -b 1024 -F -U 56d3ee50-8532-4f29-8181-d7c6ea4a94a6 $TMPFILE 20000 > $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$DEBUGFS -R "set_bg 2 itable_unused 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 flags 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 checksum 0xd318" -w $TMPFILE > /dev/null 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+dd if=/dev/zero of=$TMPFILE bs=1 count=1 seek=$((1024 * 40000)) conv=notrunc >> $OUT 2> /dev/null
+$RESIZE2FS_EXE -f -p $TMPFILE >> $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$E2FSCK -fy $TMPFILE >> $OUT 2>&1
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" < $OUT > $LOG
+rm -rf $OUT
+
+cmp -s $LOG $EXP
+RC=$?
+if [ $RC -eq 0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff -u $EXP $LOG > $test_name.failed
+fi
+
+unset EXP LOG OUT E2FSCK
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi

[PATCH 40/39] e2fsck: fix reporting of unknown htree block inode number

[Darrick J. Wong]

Sami Liedes reports that e2fsck fails to report the correct directory
inode number during a pass2 check for unexpected HTREE blocks.
Provide the inode number in the problem report.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reported-by: Sami Liedes <sami.liedes@iki.fi>
---
 e2fsck/pass2.c |    1 +
 1 file changed, 1 insertion(+)

diff --git a/e2fsck/pass2.c b/e2fsck/pass2.c
index fa17f20..f645229 100644
--- a/e2fsck/pass2.c
+++ b/e2fsck/pass2.c
@@ -1006,6 +1006,7 @@ inline_read_fail:
 	dx_dir = e2fsck_get_dx_dir_info(ctx, ino);
 	if (dx_dir && dx_dir->numblocks) {
 		if (db->blockcnt >= dx_dir->numblocks) {
+			pctx.dir = ino;
 			if (fix_problem(ctx, PR_2_UNEXPECTED_HTREE_BLOCK,
 					&pctx)) {
 				clear_htree(ctx, ino);

[PATCH 41/39] mke2fs: warn if enabling metadata_csum on a pre-3.18 kernel

[Darrick J. Wong]

The metadata_csum feature (really, the journal checksum disk format)
didn't stabilize until the 3.18 kernel, at which point the companion
journal_csum feature was turned on by default if metadata_csum was
enabled.  Therefore, warn the user if they try to create such a
filesystem on a pre-3.18 kernel.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 misc/mke2fs.c    |    8 ++++++++
 tests/filter.sed |    1 +
 2 files changed, 9 insertions(+)

diff --git a/misc/mke2fs.c b/misc/mke2fs.c
index 149330c..aeb852f 100644
--- a/misc/mke2fs.c
+++ b/misc/mke2fs.c
@@ -2213,6 +2213,14 @@ profile_error:
 			blocksize, sys_page_size);
 	}
 
+	/* Metadata checksumming wasn't totally stable before 3.18. */
+	if (is_before_linux_ver(3, 18, 0) &&
+	    (fs_param.s_feature_ro_compat &
+					EXT4_FEATURE_RO_COMPAT_METADATA_CSUM))
+		fprintf(stderr, _("Suggestion: Use Linux kernel >= 3.18 for "
+			"improved stability of the metadata and journal "
+			"checksum features.\n"));
+
 	/*
 	 * On newer kernels we do have lazy_itable_init support. So pick the
 	 * right default in case ext4 module is not loaded.
diff --git a/tests/filter.sed b/tests/filter.sed
index d9a336c..d07e9b8 100644
--- a/tests/filter.sed
+++ b/tests/filter.sed
@@ -23,3 +23,4 @@ s/\\015//g
 /whichever comes first/d
 s/, csum 0x\([0-9a-f]*\)//g
 s/ csum 0x\([0-9a-f]*\)//g
+/^Suggestion:/d

Re: [PATCH 01/39] misc: fix compiler warnings

[Theodore Ts'o]

On Sat, Oct 25, 2014 at 01:56:30PM -0700, Darrick J. Wong wrote:
> Fix some gcc-4.8 warnings.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

					- Ted

Re: [PATCH 02/39] e2fuzz: exercise fuzzed blocks more aggressively

[Theodore Ts'o]

On Sat, Oct 25, 2014 at 01:56:36PM -0700, Darrick J. Wong wrote:
> Exercise fuzzed metadata blocks more aggressively by expanding up to
> 50000 files (instead of just test.1, which might not hit anything).
> Fix a typo while we're at it.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

					- Ted

Re: [PATCH 03/39] libext2fs: directory iteration mustn't walk off the buffer end

[Theodore Ts'o]

On Sat, Oct 25, 2014 at 01:56:42PM -0700, Darrick J. Wong wrote:
> When we're iterating a directory, the loop control code reads the
> length of the next directory record, failing to account for the fact
> that there must be at least 8 bytes (the minimum size of a directory
> entry) left in the buffer to read the next directory record.  Fix the
> loop conditional so that we don't read off the end of the buffer.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> Reported-by: Sami Liedes <sami.liedes@iki.fi>

Applied, thanks.

					- Ted

Re: [PATCH 04/39] libext2fs: zero the EA block buffer before filling it

[Theodore Ts'o]

On Sat, Oct 25, 2014 at 01:56:49PM -0700, Darrick J. Wong wrote:
> When writing an extended attribute (EA) block, it's quite possible
> that the EA formatting code will not write the entire buffer.
> Therefore, we must zero the buffer beforehand to avoid writing random
> heap contents to disk.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> Reported-by: Sami Liedes <sami.liedes@iki.fi>

Applied, thanks.

					- Ted

Re: [PATCH 05/39] libext2fs: don't memcpy identical pointers when writing a cache block

[Theodore Ts'o]

On Sat, Oct 25, 2014 at 01:56:55PM -0700, Darrick J. Wong wrote:
> Sami Liedes found a scenario where we could memcpy incorrectly:
> 
> If a block read fails during an e2fsck run, the UNIX IO manager will
> call the io->read_error routine with a pointer to the internal block
> cache.  The e2fsck read error handler immediately tries to write the
> buffer back out to disk(!), at which point the block write code will
> try to copy the buffer contents back into the block cache.  Normally
> this is fine, but not when the write buffer is the cache itself!
> 
> So, plumb in a trivial check for this condition.  A more thorough
> solution would pass a duplicated buffer to the IO error handlers, but
> I don't know if that happens frequently enough to be worth the extra
> point of failure.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> Reported-by: Sami Liedes <sami.liedes@iki.fi>

Applied, thanks.

	 				- Ted

Re: [PATCH 06/39] misc: fix broken libmagic interaction with plausibility check

[Theodore Ts'o]

On Sat, Oct 25, 2014 at 01:57:01PM -0700, Darrick J. Wong wrote:
> If we get as far as calling libmagic, return the correct error code so
> that mkfs asks for confirmation if libmagic finds something and
> doesn't ask if nothing is found.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted

Re: [PATCH 07/39] tune2fs: speed up rewriting extent tree when enabling metadata_csum

[Theodore Ts'o]

On Sat, Oct 25, 2014 at 01:57:08PM -0700, Darrick J. Wong wrote:
> When enabling checksums, tune2fs naively rewrites every extent in the
> entire tree!  This is unnecessary since we only need to rewrite each
> extent tree block; therefore, only rewrite the extent if it's the
> first one in an internal extent tree block.
> 
> Also, don't bother iterating the extent tree when clearing checksums.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

					- Ted

Re: [PATCH 08/39] tune2fs: don't change metadata_csum on a mounted fs

[Theodore Ts'o]

On Sat, Oct 25, 2014 at 01:57:15PM -0700, Darrick J. Wong wrote:
> Don't let users change metadata_csum on a mounted filesystem because
> there's no way to tell the kernel to turn on the feature; there's no
> way to prevent the kernel from rewriting on-disk structures while
> tune2fs is also rewriting them; and there's no way to tell the kernel
> to reload them after tune2fs is finished.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted

Re: [PATCH 40/39] e2fsck: fix reporting of unknown htree block inode number

[Theodore Ts'o]

On Mon, Oct 27, 2014 at 04:31:09PM -0700, Darrick J. Wong wrote:
> Sami Liedes reports that e2fsck fails to report the correct directory
> inode number during a pass2 check for unexpected HTREE blocks.
> Provide the inode number in the problem report.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> Reported-by: Sami Liedes <sami.liedes@iki.fi>

Applied, thanks.

					- Ted

Re: [PATCH 17/39] e2fsck: fix dangling pointer when dir_info array is resized

[Theodore Ts'o]

On Sat, Oct 25, 2014 at 01:58:12PM -0700, Darrick J. Wong wrote:
> e2fsck uses an array to store directory usage information during pass
> 3; the usage context also contains a pointer to the last directory
> looked up.  When expanding the dir_info array, this cache pointer
> needs to be cleared if the array resize changed the pointer location,
> or else we'll later walk off the end of this dead pointer.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> Reported-by: Sami Liedes <sami.liedes@iki.fi>

Applied, thanks.

					- Ted

Re: [PATCH 41/39] mke2fs: warn if enabling metadata_csum on a pre-3.18 kernel

[Theodore Ts'o]

On Mon, Oct 27, 2014 at 04:32:31PM -0700, Darrick J. Wong wrote:
> The metadata_csum feature (really, the journal checksum disk format)
> didn't stabilize until the 3.18 kernel, at which point the companion
> journal_csum feature was turned on by default if metadata_csum was
> enabled.  Therefore, warn the user if they try to create such a
> filesystem on a pre-3.18 kernel.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Applied, thanks.

						- Ted