[PATCH] ext4/030: Ext4 online resize tests.

[PATCH] ext4/030: Ext4 online resize tests.

[harshads]

Add tests for Ext4 online resize feature. Once online resizing with
bigalloc support is merged into Ext4, these tests can be used to
sanitize the working of that feature. In order to test online resizing
with bigalloc, set MKFS_OPTIONS="-O bigalloc -C <cluster_size>" and
run the test.

Signed-off-by: Harshad Shirwadkar <harshads@google.com>
---
 tests/ext4/030     | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 21 ++++++++++++
 tests/ext4/group   |  1 +
 3 files changed, 115 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

diff --git a/tests/ext4/030 b/tests/ext4/030
new file mode 100755
index 00000000..c958e10c
--- /dev/null
+++ b/tests/ext4/030
@@ -0,0 +1,93 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize test.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <harshads@google.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_scratch
+rm -f $seqres.full
+TESTDIR="${SCRATCH_MNT}/scratchdir"
+TESTFILE="${TESTDIR}/testfile"
+
+mb2bytes()
+{
+	expr $1 \* 1024 \* 1024
+}
+
+# Takes two arguments: $1 = initial size (mb), $2 = new size (mb).
+work()
+{
+	local original_size=$1
+	local final_size=$2
+
+	echo "+ create scratch fs of size $original_size MB"
+	_scratch_mkfs_sized $(mb2bytes $original_size) 4096 > /dev/null 2>&1
+	dumpe2fs -g "${SCRATCH_DEV}" > /dev/null 2>&1 || \
+		_notrun "dumpe2fs -g not supported"
+
+	echo "+ mount scratch fs"
+	_scratch_mount > /dev/null 2>&1
+
+	echo "+ resize fs to $final_size MB"
+	resize2fs -f "${SCRATCH_DEV}" ${final_size}M >> $seqres.full 2>&1 || \
+		_fail "resize2fs failed"
+
+	echo "+ umount fs"
+	umount "${SCRATCH_MNT}"
+
+	echo "+ check fs"
+	e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || \
+		_fail "fsck should not fail"
+}
+
+## Try resizing to different sizes.
+work 128 1024
+work 1024 2048
+work 1024 4096
+work 1024 5120
+
+status=0
+exit
diff --git a/tests/ext4/030.out b/tests/ext4/030.out
new file mode 100644
index 00000000..14c24cc5
--- /dev/null
+++ b/tests/ext4/030.out
@@ -0,0 +1,21 @@
+QA output created by 030
++ create scratch fs of size 128 MB
++ mount scratch fs
++ resize fs to 1024 MB
++ umount fs
++ check fs
++ create scratch fs of size 1024 MB
++ mount scratch fs
++ resize fs to 2048 MB
++ umount fs
++ check fs
++ create scratch fs of size 1024 MB
++ mount scratch fs
++ resize fs to 4096 MB
++ umount fs
++ check fs
++ create scratch fs of size 1024 MB
++ mount scratch fs
++ resize fs to 5120 MB
++ umount fs
++ check fs
diff --git a/tests/ext4/group b/tests/ext4/group
index 257bb646..f2beeaf8 100644
--- a/tests/ext4/group
+++ b/tests/ext4/group
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag
-- 
2.15.0.403.gc27cc4dac6-goog

Re: [PATCH] ext4/030: Ext4 online resize tests.

[Dave Chinner]

On Tue, Oct 31, 2017 at 05:11:59PM -0700, harshads wrote:
> Add tests for Ext4 online resize feature. Once online resizing with
> bigalloc support is merged into Ext4, these tests can be used to
> sanitize the working of that feature. In order to test online resizing
> with bigalloc, set MKFS_OPTIONS="-O bigalloc -C <cluster_size>" and
> run the test.
> 
> Signed-off-by: Harshad Shirwadkar <harshads@google.com>

You should do this on a loop device, that way you can test
much bigger sizes and all the critical numbers around 32 bit
overflows in the storage stack (2T, 4T, 8T, 16T, etc). e.g.
see xfs/078.

Also, you should add a test that does grow at ENOSPC will exercising
the fs with fstress. See xfs/104.

Perhaps some of the XFS specific tests in the growfs group could be
turned into generic tests with the right infrastructure, so you get
more coverage on ext4 without a whole lot of extra work...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[PATCH v2] ext4/030: Ext4 online resize tests.

[harshads]

Add tests for Ext4 online resize feature.

Signed-off-by: Harshad Shirwadkar <harshads@google.com>
---
 common/ext4        |  42 +++++++++++++++
 tests/ext4/030     | 101 ++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 4 files changed, 292 insertions(+)
 create mode 100644 common/ext4
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

diff --git a/common/ext4 b/common/ext4
new file mode 100644
index 00000000..71966bef
--- /dev/null
+++ b/common/ext4
@@ -0,0 +1,42 @@
+#
+# Common ext4 functions
+#
+
+_ext4_online_resize()
+{
+	local image_file=$1
+	local image_mount=$2
+	local original_size=$3
+	local final_size=$4
+	local cluster_size=$5
+
+	echo "+ truncate image file to ${final_size}"
+	# If 'final_size' is not suffixed (like 24576), then the
+	# caller assumes that it is the number of blocks.
+	if [[ "${final_size: -1}" == [a-zA-Z] ]]; then
+		truncate ${image_file} -s ${final_size}
+	else
+		truncate ${image_file} -s ${final_size} -o 4096
+	fi
+	echo "+ create fs on image file ${original_size}"
+	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode,metadata_csum \
+		-C ${cluster_size} -b 4096 \
+		${image_file} ${original_size} > /dev/null 2>&1
+
+	dumpe2fs -g ${image_file} > /dev/null 2>&1 || \
+		_notrun "dumpe2fs -g not supported"
+
+	echo "+ mount image file"
+	mount -t ${FSTYP} ${image_file} ${image_mount} > /dev/null 2>&1
+
+	echo "+ resize fs to ${final_size}"
+	resize2fs -f ${image_file} ${final_size} >> $seqres.full 2>&1 || \
+		_fail "resize2fs failed"
+
+	echo "+ umount fs"
+	umount ${image_mount}
+
+	echo "+ check fs"
+	e2fsck -fn ${image_file} >> $seqres.full 2>&1 || \
+	    _fail "fsck should not fail"
+}
diff --git a/tests/ext4/030 b/tests/ext4/030
new file mode 100755
index 00000000..2415fb65
--- /dev/null
+++ b/tests/ext4/030
@@ -0,0 +1,101 @@
+#! /bin/bash
+# FS QA Test ext4/031
+#
+# Ext4 online resize tests with small and crucial resizes.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <harshads@google.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+	umount ${IMG_MNT} > /dev/null 2>&1
+	rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+. ./common/ext4
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+
+IMG_FILE=$TEST_DIR/$seq.fs
+IMG_MNT=$TEST_DIR/$seq.mnt
+
+rm -f $seqres.full
+
+rmdir $IMG_MNT 2>/dev/null
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+for cluster_size in 4096 16384 65536
+do
+	blocks_per_cluster=`expr $cluster_size / 4096`
+	echo "+ set cluster size to ${cluster_size}"
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((16384 * $blocks_per_cluster)) \
+		$((24576 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((32767 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((32768 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((65536 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((491521 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((507904 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((527488 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((5274880 * $blocks_per_cluster)) ${cluster_size}
+done
+
+status=0
+exit
diff --git a/tests/ext4/030.out b/tests/ext4/030.out
new file mode 100644
index 00000000..5f6441e6
--- /dev/null
+++ b/tests/ext4/030.out
@@ -0,0 +1,148 @@
+QA output created by 030
++ set cluster size to 4096
++ truncate image file to 24576
++ create fs on image file 16384
++ mount image file
++ resize fs to 24576
++ umount fs
++ check fs
++ truncate image file to 32767
++ create fs on image file 24576
++ mount image file
++ resize fs to 32767
++ umount fs
++ check fs
++ truncate image file to 32768
++ create fs on image file 24576
++ mount image file
++ resize fs to 32768
++ umount fs
++ check fs
++ truncate image file to 65536
++ create fs on image file 24576
++ mount image file
++ resize fs to 65536
++ umount fs
++ check fs
++ truncate image file to 491521
++ create fs on image file 24576
++ mount image file
++ resize fs to 491521
++ umount fs
++ check fs
++ truncate image file to 507904
++ create fs on image file 24576
++ mount image file
++ resize fs to 507904
++ umount fs
++ check fs
++ truncate image file to 527488
++ create fs on image file 24576
++ mount image file
++ resize fs to 527488
++ umount fs
++ check fs
++ truncate image file to 5274880
++ create fs on image file 24576
++ mount image file
++ resize fs to 5274880
++ umount fs
++ check fs
++ set cluster size to 16384
++ truncate image file to 98304
++ create fs on image file 65536
++ mount image file
++ resize fs to 98304
++ umount fs
++ check fs
++ truncate image file to 131068
++ create fs on image file 98304
++ mount image file
++ resize fs to 131068
++ umount fs
++ check fs
++ truncate image file to 131072
++ create fs on image file 98304
++ mount image file
++ resize fs to 131072
++ umount fs
++ check fs
++ truncate image file to 262144
++ create fs on image file 98304
++ mount image file
++ resize fs to 262144
++ umount fs
++ check fs
++ truncate image file to 1966084
++ create fs on image file 98304
++ mount image file
++ resize fs to 1966084
++ umount fs
++ check fs
++ truncate image file to 2031616
++ create fs on image file 98304
++ mount image file
++ resize fs to 2031616
++ umount fs
++ check fs
++ truncate image file to 2109952
++ create fs on image file 98304
++ mount image file
++ resize fs to 2109952
++ umount fs
++ check fs
++ truncate image file to 21099520
++ create fs on image file 98304
++ mount image file
++ resize fs to 21099520
++ umount fs
++ check fs
++ set cluster size to 65536
++ truncate image file to 393216
++ create fs on image file 262144
++ mount image file
++ resize fs to 393216
++ umount fs
++ check fs
++ truncate image file to 524272
++ create fs on image file 393216
++ mount image file
++ resize fs to 524272
++ umount fs
++ check fs
++ truncate image file to 524288
++ create fs on image file 393216
++ mount image file
++ resize fs to 524288
++ umount fs
++ check fs
++ truncate image file to 1048576
++ create fs on image file 393216
++ mount image file
++ resize fs to 1048576
++ umount fs
++ check fs
++ truncate image file to 7864336
++ create fs on image file 393216
++ mount image file
++ resize fs to 7864336
++ umount fs
++ check fs
++ truncate image file to 8126464
++ create fs on image file 393216
++ mount image file
++ resize fs to 8126464
++ umount fs
++ check fs
++ truncate image file to 8439808
++ create fs on image file 393216
++ mount image file
++ resize fs to 8439808
++ umount fs
++ check fs
++ truncate image file to 84398080
++ create fs on image file 393216
++ mount image file
++ resize fs to 84398080
++ umount fs
++ check fs
diff --git a/tests/ext4/group b/tests/ext4/group
index 257bb646..f29d3de6 100644
--- a/tests/ext4/group
+++ b/tests/ext4/group
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag
-- 
2.15.0.531.g2ccb3012c9-goog

Re: [PATCH v2] ext4/030: Ext4 online resize tests.

[Eryu Guan]

On Mon, Dec 04, 2017 at 07:43:35PM -0800, harshads wrote:
> Add tests for Ext4 online resize feature.

The commit log describes the test as a generic ext4 online resize test,
but the actual code is really testing online resize with bigalloc. So
please either update the commit log or make the test generic enough.

> 
> Signed-off-by: Harshad Shirwadkar <harshads@google.com>
> ---
>  common/ext4        |  42 +++++++++++++++
>  tests/ext4/030     | 101 ++++++++++++++++++++++++++++++++++++
>  tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/ext4/group   |   1 +
>  4 files changed, 292 insertions(+)
>  create mode 100644 common/ext4
>  create mode 100755 tests/ext4/030
>  create mode 100644 tests/ext4/030.out
> 
> diff --git a/common/ext4 b/common/ext4
> new file mode 100644
> index 00000000..71966bef
> --- /dev/null
> +++ b/common/ext4
> @@ -0,0 +1,42 @@
> +#
> +# Common ext4 functions
> +#
> +
> +_ext4_online_resize()

With the hardcoded mkfs option "-O bigalloc,resize_inode,metadata_csum",
I don't think this is a function that can be shared with other tests
easily. Maybe it should be moved to the test itself.

> +{
> +	local image_file=$1
> +	local image_mount=$2
> +	local original_size=$3
> +	local final_size=$4
> +	local cluster_size=$5
> +
> +	echo "+ truncate image file to ${final_size}"
> +	# If 'final_size' is not suffixed (like 24576), then the
> +	# caller assumes that it is the number of blocks.
> +	if [[ "${final_size: -1}" == [a-zA-Z] ]]; then
> +		truncate ${image_file} -s ${final_size}

Use xfs_io to do truncate, truncate(1) is not always available (very old
distros don't have it), though that's unlikely to happen these days, why
not using xfs_io when we have better choice :)

	$XFS_IO_PROG -c "truncate <size>" ${image_file}

> +	else
> +		truncate ${image_file} -s ${final_size} -o 4096
> +	fi
> +	echo "+ create fs on image file ${original_size}"
> +	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode,metadata_csum \
> +		-C ${cluster_size} -b 4096 \
> +		${image_file} ${original_size} > /dev/null 2>&1

We should make sure the userspace tool and kernel support thease
features first. Currently we have a helper function
_require_ext4_mkfs_feature() that checks if a given feature is supported
by e2fsprogs, but it doesn't check if kernel knows the feature too. Ted
is working a rework of that helper, but that's not finalized & merged
yet.

https://www.spinics.net/lists/fstests/msg07880.html

The new helper is _require_scratch_ext4_feature, which is creating new
filesystem with the given feature on $SCRATCH_DEV and try to mount it,
and _notrun if any step fails.

Perhaps we could make it work on sparse fs image not on $SCRATCH_DEV, so
that would be quicker and address Ted's concern too (mkfs twice on
$SCRATCH_DEV, once in the _require rule, once in the test).

Assuming the new helper is "_require_ext4_feature", then you can add the
following rules in your test

_require_ext4_feature "bigalloc,resize_inode,metadata_csum"

> +
> +	dumpe2fs -g ${image_file} > /dev/null 2>&1 || \
> +		_notrun "dumpe2fs -g not supported"

Use $DUMPE2FS_PROG, and this should in the test itself, not a worker
function. Maybe this could be made into a new _require rule too.

> +
> +	echo "+ mount image file"
> +	mount -t ${FSTYP} ${image_file} ${image_mount} > /dev/null 2>&1

$MOUNT_PROG

> +
> +	echo "+ resize fs to ${final_size}"
> +	resize2fs -f ${image_file} ${final_size} >> $seqres.full 2>&1 || \
> +		_fail "resize2fs failed"
> +
> +	echo "+ umount fs"
> +	umount ${image_mount}

$UMOUNT_PROG

> +
> +	echo "+ check fs"
> +	e2fsck -fn ${image_file} >> $seqres.full 2>&1 || \
> +	    _fail "fsck should not fail"

Assign a loop device to the fs image and call _check_generic_filesystem
with the device.

> +}
> diff --git a/tests/ext4/030 b/tests/ext4/030
> new file mode 100755
> index 00000000..2415fb65
> --- /dev/null
> +++ b/tests/ext4/030
> @@ -0,0 +1,101 @@
> +#! /bin/bash
> +# FS QA Test ext4/031
                     ^^^ 030? 031?
> +#
> +# Ext4 online resize tests with small and crucial resizes.

Same here, test description needs to match the actual test.

> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
> +#
> +# Author: Harshad Shirwadkar <harshads@google.com>
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -f $tmp.*
> +	umount ${IMG_MNT} > /dev/null 2>&1
> +	rm -f ${IMG_FILE} > /dev/null 2>&1
> +}
> +
> +# get standard environment and checks
> +. ./common/rc
> +. ./common/ext4

This should be done in common/rc (e.g. how common/xfs is sourced), not
in individual test. But if we move the resize test worker into the test
itself, common/ext4 is not needed.

> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +_supported_fs ext4
> +_supported_os Linux
> +
> +_require_loop
> +
> +IMG_FILE=$TEST_DIR/$seq.fs
> +IMG_MNT=$TEST_DIR/$seq.mnt
> +
> +rm -f $seqres.full
> +
> +rmdir $IMG_MNT 2>/dev/null
> +mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
> +
> +for cluster_size in 4096 16384 65536
> +do

We prefer this for-do-done format in fstests :)

for ...; do
	<work>
done

> +	blocks_per_cluster=`expr $cluster_size / 4096`
> +	echo "+ set cluster size to ${cluster_size}"
> +	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +		$((16384 * $blocks_per_cluster)) \
> +		$((24576 * $blocks_per_cluster)) ${cluster_size}

These size numbers are a bit hard to read, add human readable comments
on these numbers?

Thanks,
Eryu

> +
> +	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +		$((24576 * $blocks_per_cluster)) \
> +		$((32767 * $blocks_per_cluster)) ${cluster_size}
> +
> +	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +		$((24576 * $blocks_per_cluster)) \
> +		$((32768 * $blocks_per_cluster)) ${cluster_size}
> +
> +	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +		$((24576 * $blocks_per_cluster)) \
> +		$((65536 * $blocks_per_cluster)) ${cluster_size}
> +
> +	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +		$((24576 * $blocks_per_cluster)) \
> +		$((491521 * $blocks_per_cluster)) ${cluster_size}
> +
> +	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +		$((24576 * $blocks_per_cluster)) \
> +		$((507904 * $blocks_per_cluster)) ${cluster_size}
> +
> +	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +		$((24576 * $blocks_per_cluster)) \
> +		$((527488 * $blocks_per_cluster)) ${cluster_size}
> +
> +	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +		$((24576 * $blocks_per_cluster)) \
> +		$((5274880 * $blocks_per_cluster)) ${cluster_size}
> +done
> +
> +status=0
> +exit
> diff --git a/tests/ext4/030.out b/tests/ext4/030.out
> new file mode 100644
> index 00000000..5f6441e6
> --- /dev/null
> +++ b/tests/ext4/030.out
> @@ -0,0 +1,148 @@
> +QA output created by 030
> ++ set cluster size to 4096
> ++ truncate image file to 24576
> ++ create fs on image file 16384
> ++ mount image file
> ++ resize fs to 24576
> ++ umount fs
> ++ check fs
> ++ truncate image file to 32767
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 32767
> ++ umount fs
> ++ check fs
> ++ truncate image file to 32768
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 32768
> ++ umount fs
> ++ check fs
> ++ truncate image file to 65536
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 65536
> ++ umount fs
> ++ check fs
> ++ truncate image file to 491521
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 491521
> ++ umount fs
> ++ check fs
> ++ truncate image file to 507904
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 507904
> ++ umount fs
> ++ check fs
> ++ truncate image file to 527488
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 527488
> ++ umount fs
> ++ check fs
> ++ truncate image file to 5274880
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 5274880
> ++ umount fs
> ++ check fs
> ++ set cluster size to 16384
> ++ truncate image file to 98304
> ++ create fs on image file 65536
> ++ mount image file
> ++ resize fs to 98304
> ++ umount fs
> ++ check fs
> ++ truncate image file to 131068
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 131068
> ++ umount fs
> ++ check fs
> ++ truncate image file to 131072
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 131072
> ++ umount fs
> ++ check fs
> ++ truncate image file to 262144
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 262144
> ++ umount fs
> ++ check fs
> ++ truncate image file to 1966084
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 1966084
> ++ umount fs
> ++ check fs
> ++ truncate image file to 2031616
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 2031616
> ++ umount fs
> ++ check fs
> ++ truncate image file to 2109952
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 2109952
> ++ umount fs
> ++ check fs
> ++ truncate image file to 21099520
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 21099520
> ++ umount fs
> ++ check fs
> ++ set cluster size to 65536
> ++ truncate image file to 393216
> ++ create fs on image file 262144
> ++ mount image file
> ++ resize fs to 393216
> ++ umount fs
> ++ check fs
> ++ truncate image file to 524272
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 524272
> ++ umount fs
> ++ check fs
> ++ truncate image file to 524288
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 524288
> ++ umount fs
> ++ check fs
> ++ truncate image file to 1048576
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 1048576
> ++ umount fs
> ++ check fs
> ++ truncate image file to 7864336
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 7864336
> ++ umount fs
> ++ check fs
> ++ truncate image file to 8126464
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 8126464
> ++ umount fs
> ++ check fs
> ++ truncate image file to 8439808
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 8439808
> ++ umount fs
> ++ check fs
> ++ truncate image file to 84398080
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 84398080
> ++ umount fs
> ++ check fs
> diff --git a/tests/ext4/group b/tests/ext4/group
> index 257bb646..f29d3de6 100644
> --- a/tests/ext4/group
> +++ b/tests/ext4/group
> @@ -32,6 +32,7 @@
>  027 auto quick fsmap
>  028 auto quick fsmap
>  029 auto quick fsmap
> +030 auto quick ioctl resize
>  271 auto rw quick
>  301 aio auto ioctl rw stress defrag
>  302 aio auto ioctl rw stress defrag
> -- 
> 2.15.0.531.g2ccb3012c9-goog
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3] ext4/030: Ext4 online resize with bigalloc tests.

[harshads]

Add tests to verify Ext4 online resizing feature with bigalloc feature
enabled. We test various resizing scenarios with different cluster
sizes.

Signed-off-by: Harshad Shirwadkar <harshads@google.com>
---
 common/rc          |  23 ++++++++
 tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 4 files changed, 330 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

diff --git a/common/rc b/common/rc
index 9216efdb..052dadae 100644
--- a/common/rc
+++ b/common/rc
@@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
     _scratch_unmount
 }
 
+# Check whether the specified feature whether it is supported by
+# mkfs.ext4 and the kernel by using a sparse file image.
+_require_ext4_feature()
+{
+    if [ -z "$1" ]; then
+        echo "Usage: _require_loop_ext4_feature feature"
+        exit 1
+    fi
+    local test_image="${TEST_DIR}/test_image"
+    local test_mnt="${TEST_DIR}/test_mnt"
+    rm -rf $test_image $test_mnt
+    mkdir -p $test_mnt || _fail "cannot create loopback mount point"
+    $XFS_IO_PROG -f -c "truncate 512m" $test_image >/dev/null 2>&1
+    $MKFS_EXT4_PROG -F $MKFS_OPTIONS -O "$1" $test_image 512m >/dev/null 2>&1 \
+	|| _notrun "mkfs.ext4 doesn't support $1 feature"
+    _mount -t $FSTYP `_common_dev_mount_options $test_image $test_mnt` \
+        >/dev/null 2>&1 \
+	|| _notrun "Kernel doesn't support the ext4 feature(s): $1"
+    $UMOUNT_PROG $test_mnt >/dev/null 2>&1
+    rm -f $test_image >/dev/null 2>&1
+    rmdir $test_mnt >/dev/null 2>&1
+}
+
 # this test requires that external log/realtime devices are not in use
 #
 _require_nonexternal()
diff --git a/tests/ext4/030 b/tests/ext4/030
new file mode 100755
index 00000000..add063a6
--- /dev/null
+++ b/tests/ext4/030
@@ -0,0 +1,158 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize tests of small and crucial resizes with bigalloc
+# feature.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <harshads@google.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_bytes2blk()
+{
+    local bytes=$1
+    BLKSIZ=4096
+    echo $((bytes / $BLKSIZ))
+}
+
+_ext4_online_resize()
+{
+    local image_file=$1
+    local image_mount=$2
+    local original_size=$3
+    local final_size=$4
+    local cluster_size=$5
+
+    echo "+ truncate image file to $(_bytes2blk $final_size)"
+    # If 'final_size' is not suffixed (like 24576), then the
+    # caller assumes that it is the number of blocks.
+    $XFS_IO_PROG -f -c "truncate ${final_size}" ${image_file}
+    LOOP_DEVICE=`_create_loop_device $image_file || _fail "losetup failed"`
+
+    echo "+ create fs on image file $(_bytes2blk $original_size)"
+    ${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode,metadata_csum \
+		-C ${cluster_size} -b 4096 \
+		${LOOP_DEVICE} $(_bytes2blk $original_size) > /dev/null 2>&1 || \
+	_fail "mkfs failed"
+
+    echo "+ mount image file"
+    $MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${image_mount} > \
+		/dev/null 2>&1 || _fail "mount failed"
+
+    echo "+ resize fs to $(_bytes2blk $final_size)"
+    resize2fs -f ${LOOP_DEVICE} $(_bytes2blk $final_size) >> $seqres.full 2>&1 || \
+	_fail "resize2fs failed"
+
+    echo "+ umount fs"
+    $UMOUNT_PROG ${image_mount}
+
+    echo "+ check fs"
+    _check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+    _destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
+}
+
+_cleanup()
+{
+    cd /
+    [ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
+    rm -f $tmp.*
+    umount ${IMG_MNT} > /dev/null 2>&1
+    rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+_require_ext4_feature "bigalloc,resize_inode,metadata_csum"
+
+IMG_FILE=$TEST_DIR/$seq.fs
+IMG_MNT=$TEST_DIR/$seq.mnt
+
+rm -f $seqres.full
+
+rmdir $IMG_MNT 2>/dev/null
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+## We perform resizing to various multiples of block group sizes to
+## ensure that we cover maximum edge cases in the kernel code.
+for cluster_size in 4096 16384 65536; do
+    echo "+ set cluster size to ${cluster_size}"
+    ## Extending from 1/2 block group to 2/3 block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((16384 * $cluster_size)) \
+			$((24576 * $cluster_size)) ${cluster_size}
+
+    ## Extending from 1/2 block group to one cluster less than
+    ## full block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((32767 * $cluster_size)) ${cluster_size}
+
+    ## Extending from 1/2 block group to a full block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((32768 * $cluster_size)) ${cluster_size}
+
+    ## Extending from 1/2 block group to 2 full block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((65536 * $cluster_size)) ${cluster_size}
+
+    ## Extending from 1/2 block group to 15 block groups and one
+    ## cluster.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((491521 * $cluster_size)) ${cluster_size}
+
+    ## Extending from 1/2 block group to 15 full and one (last)
+    ## half block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((507904 * $cluster_size)) ${cluster_size}
+
+    ## Extending from 1/2 block group to 16 full block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((524288 * $cluster_size)) ${cluster_size}
+
+    ## Extending from 1/2 block group to 160 full block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((5242880 * $cluster_size)) ${cluster_size}
+done
+
+status=0
+exit
diff --git a/tests/ext4/030.out b/tests/ext4/030.out
new file mode 100644
index 00000000..17cc4d99
--- /dev/null
+++ b/tests/ext4/030.out
@@ -0,0 +1,148 @@
+QA output created by 030
++ set cluster size to 4096
++ truncate image file to 24576
++ create fs on image file 16384
++ mount image file
++ resize fs to 24576
++ umount fs
++ check fs
++ truncate image file to 32767
++ create fs on image file 24576
++ mount image file
++ resize fs to 32767
++ umount fs
++ check fs
++ truncate image file to 32768
++ create fs on image file 24576
++ mount image file
++ resize fs to 32768
++ umount fs
++ check fs
++ truncate image file to 65536
++ create fs on image file 24576
++ mount image file
++ resize fs to 65536
++ umount fs
++ check fs
++ truncate image file to 491521
++ create fs on image file 24576
++ mount image file
++ resize fs to 491521
++ umount fs
++ check fs
++ truncate image file to 507904
++ create fs on image file 24576
++ mount image file
++ resize fs to 507904
++ umount fs
++ check fs
++ truncate image file to 524288
++ create fs on image file 24576
++ mount image file
++ resize fs to 524288
++ umount fs
++ check fs
++ truncate image file to 5242880
++ create fs on image file 24576
++ mount image file
++ resize fs to 5242880
++ umount fs
++ check fs
++ set cluster size to 16384
++ truncate image file to 98304
++ create fs on image file 65536
++ mount image file
++ resize fs to 98304
++ umount fs
++ check fs
++ truncate image file to 131068
++ create fs on image file 98304
++ mount image file
++ resize fs to 131068
++ umount fs
++ check fs
++ truncate image file to 131072
++ create fs on image file 98304
++ mount image file
++ resize fs to 131072
++ umount fs
++ check fs
++ truncate image file to 262144
++ create fs on image file 98304
++ mount image file
++ resize fs to 262144
++ umount fs
++ check fs
++ truncate image file to 1966084
++ create fs on image file 98304
++ mount image file
++ resize fs to 1966084
++ umount fs
++ check fs
++ truncate image file to 2031616
++ create fs on image file 98304
++ mount image file
++ resize fs to 2031616
++ umount fs
++ check fs
++ truncate image file to 2097152
++ create fs on image file 98304
++ mount image file
++ resize fs to 2097152
++ umount fs
++ check fs
++ truncate image file to 20971520
++ create fs on image file 98304
++ mount image file
++ resize fs to 20971520
++ umount fs
++ check fs
++ set cluster size to 65536
++ truncate image file to 393216
++ create fs on image file 262144
++ mount image file
++ resize fs to 393216
++ umount fs
++ check fs
++ truncate image file to 524272
++ create fs on image file 393216
++ mount image file
++ resize fs to 524272
++ umount fs
++ check fs
++ truncate image file to 524288
++ create fs on image file 393216
++ mount image file
++ resize fs to 524288
++ umount fs
++ check fs
++ truncate image file to 1048576
++ create fs on image file 393216
++ mount image file
++ resize fs to 1048576
++ umount fs
++ check fs
++ truncate image file to 7864336
++ create fs on image file 393216
++ mount image file
++ resize fs to 7864336
++ umount fs
++ check fs
++ truncate image file to 8126464
++ create fs on image file 393216
++ mount image file
++ resize fs to 8126464
++ umount fs
++ check fs
++ truncate image file to 8388608
++ create fs on image file 393216
++ mount image file
++ resize fs to 8388608
++ umount fs
++ check fs
++ truncate image file to 83886080
++ create fs on image file 393216
++ mount image file
++ resize fs to 83886080
++ umount fs
++ check fs
diff --git a/tests/ext4/group b/tests/ext4/group
index 257bb646..f29d3de6 100644
--- a/tests/ext4/group
+++ b/tests/ext4/group
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag
-- 
2.15.1.620.gb9897f4670-goog

Re: [PATCH v3] ext4/030: Ext4 online resize with bigalloc tests.

[Amir Goldstein]

On Thu, Jan 4, 2018 at 3:18 AM, harshads <harshads@google.com> wrote:
> Add tests to verify Ext4 online resizing feature with bigalloc feature
> enabled. We test various resizing scenarios with different cluster
> sizes.
>
> Signed-off-by: Harshad Shirwadkar <harshads@google.com>
> ---
>  common/rc          |  23 ++++++++
>  tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/ext4/group   |   1 +
>  4 files changed, 330 insertions(+)
>  create mode 100755 tests/ext4/030
>  create mode 100644 tests/ext4/030.out
>
> diff --git a/common/rc b/common/rc
> index 9216efdb..052dadae 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
>      _scratch_unmount
>  }
>
> +# Check whether the specified feature whether it is supported by
> +# mkfs.ext4 and the kernel by using a sparse file image.
> +_require_ext4_feature()

1. please explain why this loop variant is needed
2. it would be great if you could also change callers to
_require_scratch_ext4_feature
    to use _require_scratch_feature and plug
_require_scratch_ext4_feature in there
3. probably best to post this as a separate patch from the test itself

> +{
> +    if [ -z "$1" ]; then
> +        echo "Usage: _require_loop_ext4_feature feature"

So which is it, _require_loop_ext4_feature or _require_ext4_feature?
First one sounds better to me, given that you explain why the loop
variant is needed.
If it is needed, will it be useful to have for other fs?
Then better implement _require_loop_feature and make ext4
a specific case.

Cheers,
Amir.

Re: [PATCH v3] ext4/030: Ext4 online resize with bigalloc tests.

[Eryu Guan]

On Thu, Jan 04, 2018 at 10:09:08AM +0200, Amir Goldstein wrote:
> On Thu, Jan 4, 2018 at 3:18 AM, harshads <harshads@google.com> wrote:
> > Add tests to verify Ext4 online resizing feature with bigalloc feature
> > enabled. We test various resizing scenarios with different cluster
> > sizes.
> >
> > Signed-off-by: Harshad Shirwadkar <harshads@google.com>
> > ---
> >  common/rc          |  23 ++++++++
> >  tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/ext4/group   |   1 +
> >  4 files changed, 330 insertions(+)
> >  create mode 100755 tests/ext4/030
> >  create mode 100644 tests/ext4/030.out
> >
> > diff --git a/common/rc b/common/rc
> > index 9216efdb..052dadae 100644
> > --- a/common/rc
> > +++ b/common/rc
> > @@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
> >      _scratch_unmount
> >  }
> >
> > +# Check whether the specified feature whether it is supported by
> > +# mkfs.ext4 and the kernel by using a sparse file image.
> > +_require_ext4_feature()
> 
> 1. please explain why this loop variant is needed
> 2. it would be great if you could also change callers to
> _require_scratch_ext4_feature
>     to use _require_scratch_feature and plug
> _require_scratch_ext4_feature in there
> 3. probably best to post this as a separate patch from the test itself

I haven't went through the whole patch yet, just want to point out that
the _require_scratch_ext4_feature helper has been added in commit
be341e36fd02 ("common: rework _require_ext4_mkfs_feature"), but not
enabled in the more generic _require_scratch_feature helper yet.

It'd be good to plug the ext4 helper to the generic helper in a separate
patch, as Amir suggested, and perhaps converting all existing callers of
the ext4 helper to the generic helper.

Thanks,
Eryu

> 
> > +{
> > +    if [ -z "$1" ]; then
> > +        echo "Usage: _require_loop_ext4_feature feature"
> 
> So which is it, _require_loop_ext4_feature or _require_ext4_feature?
> First one sounds better to me, given that you explain why the loop
> variant is needed.
> If it is needed, will it be useful to have for other fs?
> Then better implement _require_loop_feature and make ext4
> a specific case.
> 
> Cheers,
> Amir.

Re: [PATCH v3] ext4/030: Ext4 online resize with bigalloc tests.

[Harshad Shirwadkar]

On Thu, Jan 4, 2018 at 12:20 AM, Eryu Guan <eguan@redhat.com> wrote:
> On Thu, Jan 04, 2018 at 10:09:08AM +0200, Amir Goldstein wrote:
>> On Thu, Jan 4, 2018 at 3:18 AM, harshads <harshads@google.com> wrote:
>> > Add tests to verify Ext4 online resizing feature with bigalloc feature
>> > enabled. We test various resizing scenarios with different cluster
>> > sizes.
>> >
>> > Signed-off-by: Harshad Shirwadkar <harshads@google.com>
>> > ---
>> >  common/rc          |  23 ++++++++
>> >  tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >  tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
>> >  tests/ext4/group   |   1 +
>> >  4 files changed, 330 insertions(+)
>> >  create mode 100755 tests/ext4/030
>> >  create mode 100644 tests/ext4/030.out
>> >
>> > diff --git a/common/rc b/common/rc
>> > index 9216efdb..052dadae 100644
>> > --- a/common/rc
>> > +++ b/common/rc
>> > @@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
>> >      _scratch_unmount
>> >  }
>> >
>> > +# Check whether the specified feature whether it is supported by
>> > +# mkfs.ext4 and the kernel by using a sparse file image.
>> > +_require_ext4_feature()
>>
>> 1. please explain why this loop variant is needed

As discussed on a previous thread, the loop variant will avoid the
need to mkfs twice on scratch device (once in rule and once in test).
Also, this particular test doesn't need scratch device at all. So, if
I decide to use _require_scratch_ext4_feature, I will have to do
_require_scratch just for the rule even though the actual test doesn't
need scratch. So, having loop variant helps.

>> 2. it would be great if you could also change callers to
>> _require_scratch_ext4_feature
>>     to use _require_scratch_feature and plug
>> _require_scratch_ext4_feature in there

Okay, that sounds good.

>> 3. probably best to post this as a separate patch from the test itself

Okay.

>
> I haven't went through the whole patch yet, just want to point out that
> the _require_scratch_ext4_feature helper has been added in commit
> be341e36fd02 ("common: rework _require_ext4_mkfs_feature"), but not
> enabled in the more generic _require_scratch_feature helper yet.
>
> It'd be good to plug the ext4 helper to the generic helper in a separate
> patch, as Amir suggested, and perhaps converting all existing callers of
> the ext4 helper to the generic helper.
>
> Thanks,
> Eryu
>
>>
>> > +{
>> > +    if [ -z "$1" ]; then
>> > +        echo "Usage: _require_loop_ext4_feature feature"
>>
>> So which is it, _require_loop_ext4_feature or _require_ext4_feature?
>> First one sounds better to me, given that you explain why the loop
>> variant is needed.
>> If it is needed, will it be useful to have for other fs?
>> Then better implement _require_loop_feature and make ext4
>> a specific case.

Thanks for pointing this out. I agree that _require_loop_ext4_feature
is better. Sure, I'll add a _require_loop_feature. Alright, I'll do
_require_loop stuff in a different patch.

- Harshad.

>>
>> Cheers,
>> Amir.

Re: [PATCH v3] ext4/030: Ext4 online resize with bigalloc tests.

[Eryu Guan]

On Fri, Jan 05, 2018 at 01:21:30PM -0800, Harshad Shirwadkar wrote:
> On Thu, Jan 4, 2018 at 12:20 AM, Eryu Guan <eguan@redhat.com> wrote:
> > On Thu, Jan 04, 2018 at 10:09:08AM +0200, Amir Goldstein wrote:
> >> On Thu, Jan 4, 2018 at 3:18 AM, harshads <harshads@google.com> wrote:
> >> > Add tests to verify Ext4 online resizing feature with bigalloc feature
> >> > enabled. We test various resizing scenarios with different cluster
> >> > sizes.
> >> >
> >> > Signed-off-by: Harshad Shirwadkar <harshads@google.com>
> >> > ---
> >> >  common/rc          |  23 ++++++++
> >> >  tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >  tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
> >> >  tests/ext4/group   |   1 +
> >> >  4 files changed, 330 insertions(+)
> >> >  create mode 100755 tests/ext4/030
> >> >  create mode 100644 tests/ext4/030.out
> >> >
> >> > diff --git a/common/rc b/common/rc
> >> > index 9216efdb..052dadae 100644
> >> > --- a/common/rc
> >> > +++ b/common/rc
> >> > @@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
> >> >      _scratch_unmount
> >> >  }
> >> >
> >> > +# Check whether the specified feature whether it is supported by
> >> > +# mkfs.ext4 and the kernel by using a sparse file image.
> >> > +_require_ext4_feature()
> >>
> >> 1. please explain why this loop variant is needed
> 
> As discussed on a previous thread, the loop variant will avoid the
> need to mkfs twice on scratch device (once in rule and once in test).
> Also, this particular test doesn't need scratch device at all. So, if
> I decide to use _require_scratch_ext4_feature, I will have to do
> _require_scratch just for the rule even though the actual test doesn't
> need scratch. So, having loop variant helps.

Hmm, as the _require_scratch_ext4_feature helper is already there and
already does mkfs twice (with a very small fs size, so the additional
mkfs won't add much time), I think it'd be much simpler to just use
_require_scratch and _require_scratch_feature and create all the test
images on $SCRATCH_DEV, a loop device variant doesn't seem that
necessary to me.

Thanks,
Eryu

> 
> >> 2. it would be great if you could also change callers to
> >> _require_scratch_ext4_feature
> >>     to use _require_scratch_feature and plug
> >> _require_scratch_ext4_feature in there
> 
> Okay, that sounds good.
> 
> >> 3. probably best to post this as a separate patch from the test itself
> 
> Okay.
> 
> >
> > I haven't went through the whole patch yet, just want to point out that
> > the _require_scratch_ext4_feature helper has been added in commit
> > be341e36fd02 ("common: rework _require_ext4_mkfs_feature"), but not
> > enabled in the more generic _require_scratch_feature helper yet.
> >
> > It'd be good to plug the ext4 helper to the generic helper in a separate
> > patch, as Amir suggested, and perhaps converting all existing callers of
> > the ext4 helper to the generic helper.
> >
> > Thanks,
> > Eryu
> >
> >>
> >> > +{
> >> > +    if [ -z "$1" ]; then
> >> > +        echo "Usage: _require_loop_ext4_feature feature"
> >>
> >> So which is it, _require_loop_ext4_feature or _require_ext4_feature?
> >> First one sounds better to me, given that you explain why the loop
> >> variant is needed.
> >> If it is needed, will it be useful to have for other fs?
> >> Then better implement _require_loop_feature and make ext4
> >> a specific case.
> 
> Thanks for pointing this out. I agree that _require_loop_ext4_feature
> is better. Sure, I'll add a _require_loop_feature. Alright, I'll do
> _require_loop stuff in a different patch.
> 
> - Harshad.
> 
> >>
> >> Cheers,
> >> Amir.

Re: [PATCH v3] ext4/030: Ext4 online resize with bigalloc tests.

[Harshad Shirwadkar]

Okay, I will keep this patch clean and will just use
_require_scratch_* functions. Let me submit another patch that does
that cleaning up that Amir suggested.

On Sun, Jan 7, 2018 at 7:31 AM, Eryu Guan <eguan@redhat.com> wrote:
> On Fri, Jan 05, 2018 at 01:21:30PM -0800, Harshad Shirwadkar wrote:
>> On Thu, Jan 4, 2018 at 12:20 AM, Eryu Guan <eguan@redhat.com> wrote:
>> > On Thu, Jan 04, 2018 at 10:09:08AM +0200, Amir Goldstein wrote:
>> >> On Thu, Jan 4, 2018 at 3:18 AM, harshads <harshads@google.com> wrote:
>> >> > Add tests to verify Ext4 online resizing feature with bigalloc feature
>> >> > enabled. We test various resizing scenarios with different cluster
>> >> > sizes.
>> >> >
>> >> > Signed-off-by: Harshad Shirwadkar <harshads@google.com>
>> >> > ---
>> >> >  common/rc          |  23 ++++++++
>> >> >  tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >  tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >  tests/ext4/group   |   1 +
>> >> >  4 files changed, 330 insertions(+)
>> >> >  create mode 100755 tests/ext4/030
>> >> >  create mode 100644 tests/ext4/030.out
>> >> >
>> >> > diff --git a/common/rc b/common/rc
>> >> > index 9216efdb..052dadae 100644
>> >> > --- a/common/rc
>> >> > +++ b/common/rc
>> >> > @@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
>> >> >      _scratch_unmount
>> >> >  }
>> >> >
>> >> > +# Check whether the specified feature whether it is supported by
>> >> > +# mkfs.ext4 and the kernel by using a sparse file image.
>> >> > +_require_ext4_feature()
>> >>
>> >> 1. please explain why this loop variant is needed
>>
>> As discussed on a previous thread, the loop variant will avoid the
>> need to mkfs twice on scratch device (once in rule and once in test).
>> Also, this particular test doesn't need scratch device at all. So, if
>> I decide to use _require_scratch_ext4_feature, I will have to do
>> _require_scratch just for the rule even though the actual test doesn't
>> need scratch. So, having loop variant helps.
>
> Hmm, as the _require_scratch_ext4_feature helper is already there and
> already does mkfs twice (with a very small fs size, so the additional
> mkfs won't add much time), I think it'd be much simpler to just use
> _require_scratch and _require_scratch_feature and create all the test
> images on $SCRATCH_DEV, a loop device variant doesn't seem that
> necessary to me.
>
> Thanks,
> Eryu
>
>>
>> >> 2. it would be great if you could also change callers to
>> >> _require_scratch_ext4_feature
>> >>     to use _require_scratch_feature and plug
>> >> _require_scratch_ext4_feature in there
>>
>> Okay, that sounds good.
>>
>> >> 3. probably best to post this as a separate patch from the test itself
>>
>> Okay.
>>
>> >
>> > I haven't went through the whole patch yet, just want to point out that
>> > the _require_scratch_ext4_feature helper has been added in commit
>> > be341e36fd02 ("common: rework _require_ext4_mkfs_feature"), but not
>> > enabled in the more generic _require_scratch_feature helper yet.
>> >
>> > It'd be good to plug the ext4 helper to the generic helper in a separate
>> > patch, as Amir suggested, and perhaps converting all existing callers of
>> > the ext4 helper to the generic helper.
>> >
>> > Thanks,
>> > Eryu
>> >
>> >>
>> >> > +{
>> >> > +    if [ -z "$1" ]; then
>> >> > +        echo "Usage: _require_loop_ext4_feature feature"
>> >>
>> >> So which is it, _require_loop_ext4_feature or _require_ext4_feature?
>> >> First one sounds better to me, given that you explain why the loop
>> >> variant is needed.
>> >> If it is needed, will it be useful to have for other fs?
>> >> Then better implement _require_loop_feature and make ext4
>> >> a specific case.
>>
>> Thanks for pointing this out. I agree that _require_loop_ext4_feature
>> is better. Sure, I'll add a _require_loop_feature. Alright, I'll do
>> _require_loop stuff in a different patch.
>>
>> - Harshad.
>>
>> >>
>> >> Cheers,
>> >> Amir.

[PATCH v4] ext4/030: Ext4 online resize with bigalloc tests.

[harshads]

Add tests to verify Ext4 online resizing feature with bigalloc feature
enabled. We test various resizing scenarios with different cluster
sizes.

Signed-off-by: Harshad Shirwadkar <harshads@google.com>
---
 tests/ext4/030     | 156 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 3 files changed, 305 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

diff --git a/tests/ext4/030 b/tests/ext4/030
new file mode 100755
index 00000000..99ca9362
--- /dev/null
+++ b/tests/ext4/030
@@ -0,0 +1,156 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize tests of small and crucial resizes with bigalloc
+# feature.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <harshads@google.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_bytes2blk()
+{
+    local bytes=$1
+    BLKSIZ=4096
+    echo $((bytes / $BLKSIZ))
+}
+
+_ext4_online_resize()
+{
+    local image_file=$1
+    local image_mount=$2
+    local original_size=$3
+    local final_size=$4
+    local cluster_size=$5
+
+    echo "+ truncate image file to $(_bytes2blk $final_size)"
+    $XFS_IO_PROG -f -c "truncate ${final_size}" ${image_file}
+    LOOP_DEVICE=`_create_loop_device $image_file || _fail "losetup failed"`
+
+    echo "+ create fs on image file $(_bytes2blk $original_size)"
+    ${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode,metadata_csum \
+		-C ${cluster_size} -b 4096 \
+		${LOOP_DEVICE} $(_bytes2blk $original_size) > /dev/null 2>&1 || \
+	_fail "mkfs failed"
+
+    echo "+ mount image file"
+    $MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${image_mount} > \
+		/dev/null 2>&1 || _fail "mount failed"
+
+    echo "+ resize fs to $(_bytes2blk $final_size)"
+    resize2fs -f ${LOOP_DEVICE} $(_bytes2blk $final_size) >> $seqres.full 2>&1 || \
+	_fail "resize2fs failed"
+
+    echo "+ umount fs"
+    $UMOUNT_PROG ${image_mount}
+
+    echo "+ check fs"
+    _check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+    _destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
+}
+
+_cleanup()
+{
+    cd /
+    [ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
+    rm -f $tmp.*
+    umount ${IMG_MNT} > /dev/null 2>&1
+    rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+_require_scratch
+_require_scratch_ext4_feature "bigalloc,resize_inode,metadata_csum"
+
+IMG_FILE=$TEST_DIR/$seq.fs
+IMG_MNT=$TEST_DIR/$seq.mnt
+
+rm -f $seqres.full
+
+rmdir $IMG_MNT 2>/dev/null
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+## We perform resizing to various multiples of block group sizes to
+## ensure that we cover maximum edge cases in the kernel code.
+for cluster_size in 4096 16384 65536; do
+    echo "+ set cluster size to ${cluster_size}"
+    ## Extending a 1/2 block group to a 2/3 block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((16384 * $cluster_size)) \
+			$((24576 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to one cluster less than a
+    ## full block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((32767 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to a full block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((32768 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 2 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((65536 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 15 block groups and one
+    ## cluster.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((491521 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 15 and a half block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((507904 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 16 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((524288 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 160 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((5242880 * $cluster_size)) ${cluster_size}
+done
+
+status=0
+exit
diff --git a/tests/ext4/030.out b/tests/ext4/030.out
new file mode 100644
index 00000000..17cc4d99
--- /dev/null
+++ b/tests/ext4/030.out
@@ -0,0 +1,148 @@
+QA output created by 030
++ set cluster size to 4096
++ truncate image file to 24576
++ create fs on image file 16384
++ mount image file
++ resize fs to 24576
++ umount fs
++ check fs
++ truncate image file to 32767
++ create fs on image file 24576
++ mount image file
++ resize fs to 32767
++ umount fs
++ check fs
++ truncate image file to 32768
++ create fs on image file 24576
++ mount image file
++ resize fs to 32768
++ umount fs
++ check fs
++ truncate image file to 65536
++ create fs on image file 24576
++ mount image file
++ resize fs to 65536
++ umount fs
++ check fs
++ truncate image file to 491521
++ create fs on image file 24576
++ mount image file
++ resize fs to 491521
++ umount fs
++ check fs
++ truncate image file to 507904
++ create fs on image file 24576
++ mount image file
++ resize fs to 507904
++ umount fs
++ check fs
++ truncate image file to 524288
++ create fs on image file 24576
++ mount image file
++ resize fs to 524288
++ umount fs
++ check fs
++ truncate image file to 5242880
++ create fs on image file 24576
++ mount image file
++ resize fs to 5242880
++ umount fs
++ check fs
++ set cluster size to 16384
++ truncate image file to 98304
++ create fs on image file 65536
++ mount image file
++ resize fs to 98304
++ umount fs
++ check fs
++ truncate image file to 131068
++ create fs on image file 98304
++ mount image file
++ resize fs to 131068
++ umount fs
++ check fs
++ truncate image file to 131072
++ create fs on image file 98304
++ mount image file
++ resize fs to 131072
++ umount fs
++ check fs
++ truncate image file to 262144
++ create fs on image file 98304
++ mount image file
++ resize fs to 262144
++ umount fs
++ check fs
++ truncate image file to 1966084
++ create fs on image file 98304
++ mount image file
++ resize fs to 1966084
++ umount fs
++ check fs
++ truncate image file to 2031616
++ create fs on image file 98304
++ mount image file
++ resize fs to 2031616
++ umount fs
++ check fs
++ truncate image file to 2097152
++ create fs on image file 98304
++ mount image file
++ resize fs to 2097152
++ umount fs
++ check fs
++ truncate image file to 20971520
++ create fs on image file 98304
++ mount image file
++ resize fs to 20971520
++ umount fs
++ check fs
++ set cluster size to 65536
++ truncate image file to 393216
++ create fs on image file 262144
++ mount image file
++ resize fs to 393216
++ umount fs
++ check fs
++ truncate image file to 524272
++ create fs on image file 393216
++ mount image file
++ resize fs to 524272
++ umount fs
++ check fs
++ truncate image file to 524288
++ create fs on image file 393216
++ mount image file
++ resize fs to 524288
++ umount fs
++ check fs
++ truncate image file to 1048576
++ create fs on image file 393216
++ mount image file
++ resize fs to 1048576
++ umount fs
++ check fs
++ truncate image file to 7864336
++ create fs on image file 393216
++ mount image file
++ resize fs to 7864336
++ umount fs
++ check fs
++ truncate image file to 8126464
++ create fs on image file 393216
++ mount image file
++ resize fs to 8126464
++ umount fs
++ check fs
++ truncate image file to 8388608
++ create fs on image file 393216
++ mount image file
++ resize fs to 8388608
++ umount fs
++ check fs
++ truncate image file to 83886080
++ create fs on image file 393216
++ mount image file
++ resize fs to 83886080
++ umount fs
++ check fs
diff --git a/tests/ext4/group b/tests/ext4/group
index 257bb646..f29d3de6 100644
--- a/tests/ext4/group
+++ b/tests/ext4/group
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag
-- 
2.16.0.rc0.223.g4a4ac83678-goog

Re: [PATCH v4] ext4/030: Ext4 online resize with bigalloc tests.

[Eryu Guan]

On Sun, Jan 07, 2018 at 08:18:15PM -0800, harshads wrote:
> Add tests to verify Ext4 online resizing feature with bigalloc feature
> enabled. We test various resizing scenarios with different cluster
> sizes.
> 
> Signed-off-by: Harshad Shirwadkar <harshads@google.com>

I have some more comments inline, sorry for not bringing up all comments
in previous reviews.. And please cc ext4 list for new ext4 test.

One minor issue regarding to the patch summary:

ext4/030: Ext4 online resize with bigalloc tests

there's no need to include a fixed test seq number in summary for new
test, new tests usually will be re-numbered on commit.

> ---
>  tests/ext4/030     | 156 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/ext4/030.out | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/ext4/group   |   1 +
>  3 files changed, 305 insertions(+)
>  create mode 100755 tests/ext4/030
>  create mode 100644 tests/ext4/030.out
> 
> diff --git a/tests/ext4/030 b/tests/ext4/030
> new file mode 100755
> index 00000000..99ca9362
> --- /dev/null
> +++ b/tests/ext4/030
> @@ -0,0 +1,156 @@
> +#! /bin/bash
> +# FS QA Test ext4/030
> +#
> +# Ext4 online resize tests of small and crucial resizes with bigalloc
> +# feature.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
> +#
> +# Author: Harshad Shirwadkar <harshads@google.com>
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_bytes2blk()

Usually the leading underscore is only used for common helper names, not
a local function, "bytes2blk" is fine.

> +{
> +    local bytes=$1

And please indent with tab not 4 spaces.

> +    BLKSIZ=4096

BLKSIZ is always 4096 and is only used in this function, move it out of
bytes2blk? e.g.

BLKSIZ=4096
bytes2blk()
{
...
}

This leads me to wonder that do we need to test other block sizes like
1k and 2k?

> +    echo $((bytes / $BLKSIZ))
> +}
> +
> +_ext4_online_resize()
> +{
> +    local image_file=$1
> +    local image_mount=$2
> +    local original_size=$3
> +    local final_size=$4
> +    local cluster_size=$5
> +
> +    echo "+ truncate image file to $(_bytes2blk $final_size)"

Append such logs to $seqres.full too? I might be easier for debugging.

> +    $XFS_IO_PROG -f -c "truncate ${final_size}" ${image_file}
> +    LOOP_DEVICE=`_create_loop_device $image_file || _fail "losetup failed"`
> +
> +    echo "+ create fs on image file $(_bytes2blk $original_size)"
> +    ${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode,metadata_csum \

Why do we need "metadata_csum" here? Some comments would be fine if it's
really needed. Better to have comments on the "resize_inode" feature
too.

> +		-C ${cluster_size} -b 4096 \
> +		${LOOP_DEVICE} $(_bytes2blk $original_size) > /dev/null 2>&1 || \
> +	_fail "mkfs failed"
> +
> +    echo "+ mount image file"
> +    $MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${image_mount} > \
> +		/dev/null 2>&1 || _fail "mount failed"
> +
> +    echo "+ resize fs to $(_bytes2blk $final_size)"
> +    resize2fs -f ${LOOP_DEVICE} $(_bytes2blk $final_size) >> $seqres.full 2>&1 || \
> +	_fail "resize2fs failed"

Hmm, online resize support for bigalloc ext4 is added in v4.15-rc1,
tests would fail on old kernels. I think we need a new _require rule to
make sure current kernel does support ext4 online resize with bigalloc
feature.

> +
> +    echo "+ umount fs"
> +    $UMOUNT_PROG ${image_mount}
> +
> +    echo "+ check fs"
> +    _check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
> +	_fail "fsck should not fail"
> +    _destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
> +}
> +
> +_cleanup()
> +{
> +    cd /
> +    [ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
> +    rm -f $tmp.*
> +    umount ${IMG_MNT} > /dev/null 2>&1

$UMOUNT_PROG

> +    rm -f ${IMG_FILE} > /dev/null 2>&1
> +}
> +
> +# get standard environment and checks
> +. ./common/rc
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +_supported_fs ext4
> +_supported_os Linux
> +
> +_require_loop
> +_require_scratch
> +_require_scratch_ext4_feature "bigalloc,resize_inode,metadata_csum"
> +
> +IMG_FILE=$TEST_DIR/$seq.fs
> +IMG_MNT=$TEST_DIR/$seq.mnt

We required scratch device, then just create fs images on $SCRATCH_MNT?
i.e.

rm -f $seqres.full
_scratch_mkfs >>$seqres.full 2>&1
_scratch_mount

IMG_FILE=$SCRATCH_MNT/$seq.fs
IMG_MNT=$SCRATCH_MNT/$seq.mnt

> +
> +rm -f $seqres.full
> +
> +rmdir $IMG_MNT 2>/dev/null

Then there's no need to do rmdir on newly created filesystem.

> +mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
> +
> +## We perform resizing to various multiples of block group sizes to
> +## ensure that we cover maximum edge cases in the kernel code.

I found that your reply to Ted in the ext4 bigalloc online resize path
thread provided more details of the test:

"... ensure that all different code paths that resize IOCTL triggers get
executed. At a high level, these different code paths are - 1) extending
last block group, 2) adding new block groups 3) conversion of file
system to meta-bg."

I think it's better to add similar comments in the test too.

Thanks,
Eryu

> +for cluster_size in 4096 16384 65536; do
> +    echo "+ set cluster size to ${cluster_size}"
> +    ## Extending a 1/2 block group to a 2/3 block group.
> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +			$((16384 * $cluster_size)) \
> +			$((24576 * $cluster_size)) ${cluster_size}
> +
> +    ## Extending a 2/3rd block group to one cluster less than a
> +    ## full block group.
> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +			$((24576 * $cluster_size)) \
> +			$((32767 * $cluster_size)) ${cluster_size}
> +
> +    ## Extending a 2/3rd block group to a full block group.
> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +			$((24576 * $cluster_size)) \
> +			$((32768 * $cluster_size)) ${cluster_size}
> +
> +    ## Extending a 2/3rd block group to 2 block groups.
> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +			$((24576 * $cluster_size)) \
> +			$((65536 * $cluster_size)) ${cluster_size}
> +
> +    ## Extending a 2/3rd block group to 15 block groups and one
> +    ## cluster.
> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +			$((24576 * $cluster_size)) \
> +			$((491521 * $cluster_size)) ${cluster_size}
> +
> +    ## Extending a 2/3rd block group to 15 and a half block groups.
> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +			$((24576 * $cluster_size)) \
> +			$((507904 * $cluster_size)) ${cluster_size}
> +
> +    ## Extending a 2/3rd block group to 16 block groups.
> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +			$((24576 * $cluster_size)) \
> +			$((524288 * $cluster_size)) ${cluster_size}
> +
> +    ## Extending a 2/3rd block group to 160 block groups.
> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
> +			$((24576 * $cluster_size)) \
> +			$((5242880 * $cluster_size)) ${cluster_size}
> +done
> +
> +status=0
> +exit
> diff --git a/tests/ext4/030.out b/tests/ext4/030.out
> new file mode 100644
> index 00000000..17cc4d99
> --- /dev/null
> +++ b/tests/ext4/030.out
> @@ -0,0 +1,148 @@
> +QA output created by 030
> ++ set cluster size to 4096
> ++ truncate image file to 24576
> ++ create fs on image file 16384
> ++ mount image file
> ++ resize fs to 24576
> ++ umount fs
> ++ check fs
> ++ truncate image file to 32767
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 32767
> ++ umount fs
> ++ check fs
> ++ truncate image file to 32768
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 32768
> ++ umount fs
> ++ check fs
> ++ truncate image file to 65536
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 65536
> ++ umount fs
> ++ check fs
> ++ truncate image file to 491521
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 491521
> ++ umount fs
> ++ check fs
> ++ truncate image file to 507904
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 507904
> ++ umount fs
> ++ check fs
> ++ truncate image file to 524288
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 524288
> ++ umount fs
> ++ check fs
> ++ truncate image file to 5242880
> ++ create fs on image file 24576
> ++ mount image file
> ++ resize fs to 5242880
> ++ umount fs
> ++ check fs
> ++ set cluster size to 16384
> ++ truncate image file to 98304
> ++ create fs on image file 65536
> ++ mount image file
> ++ resize fs to 98304
> ++ umount fs
> ++ check fs
> ++ truncate image file to 131068
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 131068
> ++ umount fs
> ++ check fs
> ++ truncate image file to 131072
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 131072
> ++ umount fs
> ++ check fs
> ++ truncate image file to 262144
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 262144
> ++ umount fs
> ++ check fs
> ++ truncate image file to 1966084
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 1966084
> ++ umount fs
> ++ check fs
> ++ truncate image file to 2031616
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 2031616
> ++ umount fs
> ++ check fs
> ++ truncate image file to 2097152
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 2097152
> ++ umount fs
> ++ check fs
> ++ truncate image file to 20971520
> ++ create fs on image file 98304
> ++ mount image file
> ++ resize fs to 20971520
> ++ umount fs
> ++ check fs
> ++ set cluster size to 65536
> ++ truncate image file to 393216
> ++ create fs on image file 262144
> ++ mount image file
> ++ resize fs to 393216
> ++ umount fs
> ++ check fs
> ++ truncate image file to 524272
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 524272
> ++ umount fs
> ++ check fs
> ++ truncate image file to 524288
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 524288
> ++ umount fs
> ++ check fs
> ++ truncate image file to 1048576
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 1048576
> ++ umount fs
> ++ check fs
> ++ truncate image file to 7864336
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 7864336
> ++ umount fs
> ++ check fs
> ++ truncate image file to 8126464
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 8126464
> ++ umount fs
> ++ check fs
> ++ truncate image file to 8388608
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 8388608
> ++ umount fs
> ++ check fs
> ++ truncate image file to 83886080
> ++ create fs on image file 393216
> ++ mount image file
> ++ resize fs to 83886080
> ++ umount fs
> ++ check fs
> diff --git a/tests/ext4/group b/tests/ext4/group
> index 257bb646..f29d3de6 100644
> --- a/tests/ext4/group
> +++ b/tests/ext4/group
> @@ -32,6 +32,7 @@
>  027 auto quick fsmap
>  028 auto quick fsmap
>  029 auto quick fsmap
> +030 auto quick ioctl resize
>  271 auto rw quick
>  301 aio auto ioctl rw stress defrag
>  302 aio auto ioctl rw stress defrag
> -- 
> 2.16.0.rc0.223.g4a4ac83678-goog
> 

Re: [PATCH v4] ext4/030: Ext4 online resize with bigalloc tests.

[Harshad Shirwadkar]

On Wed, Jan 10, 2018 at 11:07 PM, Eryu Guan <eguan@redhat.com> wrote:
> On Sun, Jan 07, 2018 at 08:18:15PM -0800, harshads wrote:
>> Add tests to verify Ext4 online resizing feature with bigalloc feature
>> enabled. We test various resizing scenarios with different cluster
>> sizes.
>>
>> Signed-off-by: Harshad Shirwadkar <harshads@google.com>
>
> I have some more comments inline, sorry for not bringing up all comments
> in previous reviews.. And please cc ext4 list for new ext4 test.
>
> One minor issue regarding to the patch summary:
>
> ext4/030: Ext4 online resize with bigalloc tests
>
> there's no need to include a fixed test seq number in summary for new
> test, new tests usually will be re-numbered on commit.
Done.
>
>> ---
>>  tests/ext4/030     | 156 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  tests/ext4/030.out | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++
>>  tests/ext4/group   |   1 +
>>  3 files changed, 305 insertions(+)
>>  create mode 100755 tests/ext4/030
>>  create mode 100644 tests/ext4/030.out
>>
>> diff --git a/tests/ext4/030 b/tests/ext4/030
>> new file mode 100755
>> index 00000000..99ca9362
>> --- /dev/null
>> +++ b/tests/ext4/030
>> @@ -0,0 +1,156 @@
>> +#! /bin/bash
>> +# FS QA Test ext4/030
>> +#
>> +# Ext4 online resize tests of small and crucial resizes with bigalloc
>> +# feature.
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
>> +#
>> +# Author: Harshad Shirwadkar <harshads@google.com>
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
>> +#-----------------------------------------------------------------------
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +tmp=/tmp/$$
>> +status=1     # failure is the default!
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +_bytes2blk()
>
> Usually the leading underscore is only used for common helper names, not
> a local function, "bytes2blk" is fine.
Done.
>
>> +{
>> +    local bytes=$1
>
> And please indent with tab not 4 spaces.
Done.
>
>> +    BLKSIZ=4096
>
> BLKSIZ is always 4096 and is only used in this function, move it out of
> bytes2blk? e.g.
>
> BLKSIZ=4096
> bytes2blk()
> {
> ...
> }
>
Done.
> This leads me to wonder that do we need to test other block sizes like
> 1k and 2k?
As of now, I have performed testing with 4k and 2k block sizes.
Everything looks good. With 1k block size, there are a few small
fixes. So I was we could get this version of the test in. When 1k
block fixes are in the kernel, we could enhance the test suite to test
that.
>
>> +    echo $((bytes / $BLKSIZ))
>> +}
>> +
>> +_ext4_online_resize()
>> +{
>> +    local image_file=$1
>> +    local image_mount=$2
>> +    local original_size=$3
>> +    local final_size=$4
>> +    local cluster_size=$5
>> +
>> +    echo "+ truncate image file to $(_bytes2blk $final_size)"
>
> Append such logs to $seqres.full too? I might be easier for debugging.
Done.
>
>> +    $XFS_IO_PROG -f -c "truncate ${final_size}" ${image_file}
>> +    LOOP_DEVICE=`_create_loop_device $image_file || _fail "losetup failed"`
>> +
>> +    echo "+ create fs on image file $(_bytes2blk $original_size)"
>> +    ${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode,metadata_csum \
>
> Why do we need "metadata_csum" here? Some comments would be fine if it's
> really needed. Better to have comments on the "resize_inode" feature
> too.
>
>> +             -C ${cluster_size} -b 4096 \
>> +             ${LOOP_DEVICE} $(_bytes2blk $original_size) > /dev/null 2>&1 || \
>> +     _fail "mkfs failed"
>> +
>> +    echo "+ mount image file"
>> +    $MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${image_mount} > \
>> +             /dev/null 2>&1 || _fail "mount failed"
>> +
>> +    echo "+ resize fs to $(_bytes2blk $final_size)"
>> +    resize2fs -f ${LOOP_DEVICE} $(_bytes2blk $final_size) >> $seqres.full 2>&1 || \
>> +     _fail "resize2fs failed"
>
> Hmm, online resize support for bigalloc ext4 is added in v4.15-rc1,
> tests would fail on old kernels. I think we need a new _require rule to
> make sure current kernel does support ext4 online resize with bigalloc
> feature.
>
>> +
>> +    echo "+ umount fs"
>> +    $UMOUNT_PROG ${image_mount}
>> +
>> +    echo "+ check fs"
>> +    _check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
>> +     _fail "fsck should not fail"
>> +    _destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
>> +}
>> +
>> +_cleanup()
>> +{
>> +    cd /
>> +    [ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
>> +    rm -f $tmp.*
>> +    umount ${IMG_MNT} > /dev/null 2>&1
>
> $UMOUNT_PROG
Done.
>
>> +    rm -f ${IMG_FILE} > /dev/null 2>&1
>> +}
>> +
>> +# get standard environment and checks
>> +. ./common/rc
>> +
>> +# remove previous $seqres.full before test
>> +rm -f $seqres.full
>> +
>> +# real QA test starts here
>> +_supported_fs ext4
>> +_supported_os Linux
>> +
>> +_require_loop
>> +_require_scratch
>> +_require_scratch_ext4_feature "bigalloc,resize_inode,metadata_csum"
>> +
>> +IMG_FILE=$TEST_DIR/$seq.fs
>> +IMG_MNT=$TEST_DIR/$seq.mnt
>
> We required scratch device, then just create fs images on $SCRATCH_MNT?
> i.e.
>
> rm -f $seqres.full
> _scratch_mkfs >>$seqres.full 2>&1
> _scratch_mount
>
> IMG_FILE=$SCRATCH_MNT/$seq.fs
> IMG_MNT=$SCRATCH_MNT/$seq.mnt
Done.
>
>> +
>> +rm -f $seqres.full
>> +
>> +rmdir $IMG_MNT 2>/dev/null
>
> Then there's no need to do rmdir on newly created filesystem.
Done.
>
>> +mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
>> +
>> +## We perform resizing to various multiples of block group sizes to
>> +## ensure that we cover maximum edge cases in the kernel code.
>
> I found that your reply to Ted in the ext4 bigalloc online resize path
> thread provided more details of the test:
>
> "... ensure that all different code paths that resize IOCTL triggers get
> executed. At a high level, these different code paths are - 1) extending
> last block group, 2) adding new block groups 3) conversion of file
> system to meta-bg."
>
> I think it's better to add similar comments in the test too.
I added comments in the test.
>
> Thanks,
> Eryu
>
>> +for cluster_size in 4096 16384 65536; do
>> +    echo "+ set cluster size to ${cluster_size}"
>> +    ## Extending a 1/2 block group to a 2/3 block group.
>> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
>> +                     $((16384 * $cluster_size)) \
>> +                     $((24576 * $cluster_size)) ${cluster_size}
>> +
>> +    ## Extending a 2/3rd block group to one cluster less than a
>> +    ## full block group.
>> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
>> +                     $((24576 * $cluster_size)) \
>> +                     $((32767 * $cluster_size)) ${cluster_size}
>> +
>> +    ## Extending a 2/3rd block group to a full block group.
>> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
>> +                     $((24576 * $cluster_size)) \
>> +                     $((32768 * $cluster_size)) ${cluster_size}
>> +
>> +    ## Extending a 2/3rd block group to 2 block groups.
>> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
>> +                     $((24576 * $cluster_size)) \
>> +                     $((65536 * $cluster_size)) ${cluster_size}
>> +
>> +    ## Extending a 2/3rd block group to 15 block groups and one
>> +    ## cluster.
>> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
>> +                     $((24576 * $cluster_size)) \
>> +                     $((491521 * $cluster_size)) ${cluster_size}
>> +
>> +    ## Extending a 2/3rd block group to 15 and a half block groups.
>> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
>> +                     $((24576 * $cluster_size)) \
>> +                     $((507904 * $cluster_size)) ${cluster_size}
>> +
>> +    ## Extending a 2/3rd block group to 16 block groups.
>> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
>> +                     $((24576 * $cluster_size)) \
>> +                     $((524288 * $cluster_size)) ${cluster_size}
>> +
>> +    ## Extending a 2/3rd block group to 160 block groups.
>> +    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
>> +                     $((24576 * $cluster_size)) \
>> +                     $((5242880 * $cluster_size)) ${cluster_size}
>> +done
>> +
>> +status=0
>> +exit
>> diff --git a/tests/ext4/030.out b/tests/ext4/030.out
>> new file mode 100644
>> index 00000000..17cc4d99
>> --- /dev/null
>> +++ b/tests/ext4/030.out
>> @@ -0,0 +1,148 @@
>> +QA output created by 030
>> ++ set cluster size to 4096
>> ++ truncate image file to 24576
>> ++ create fs on image file 16384
>> ++ mount image file
>> ++ resize fs to 24576
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 32767
>> ++ create fs on image file 24576
>> ++ mount image file
>> ++ resize fs to 32767
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 32768
>> ++ create fs on image file 24576
>> ++ mount image file
>> ++ resize fs to 32768
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 65536
>> ++ create fs on image file 24576
>> ++ mount image file
>> ++ resize fs to 65536
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 491521
>> ++ create fs on image file 24576
>> ++ mount image file
>> ++ resize fs to 491521
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 507904
>> ++ create fs on image file 24576
>> ++ mount image file
>> ++ resize fs to 507904
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 524288
>> ++ create fs on image file 24576
>> ++ mount image file
>> ++ resize fs to 524288
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 5242880
>> ++ create fs on image file 24576
>> ++ mount image file
>> ++ resize fs to 5242880
>> ++ umount fs
>> ++ check fs
>> ++ set cluster size to 16384
>> ++ truncate image file to 98304
>> ++ create fs on image file 65536
>> ++ mount image file
>> ++ resize fs to 98304
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 131068
>> ++ create fs on image file 98304
>> ++ mount image file
>> ++ resize fs to 131068
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 131072
>> ++ create fs on image file 98304
>> ++ mount image file
>> ++ resize fs to 131072
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 262144
>> ++ create fs on image file 98304
>> ++ mount image file
>> ++ resize fs to 262144
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 1966084
>> ++ create fs on image file 98304
>> ++ mount image file
>> ++ resize fs to 1966084
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 2031616
>> ++ create fs on image file 98304
>> ++ mount image file
>> ++ resize fs to 2031616
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 2097152
>> ++ create fs on image file 98304
>> ++ mount image file
>> ++ resize fs to 2097152
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 20971520
>> ++ create fs on image file 98304
>> ++ mount image file
>> ++ resize fs to 20971520
>> ++ umount fs
>> ++ check fs
>> ++ set cluster size to 65536
>> ++ truncate image file to 393216
>> ++ create fs on image file 262144
>> ++ mount image file
>> ++ resize fs to 393216
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 524272
>> ++ create fs on image file 393216
>> ++ mount image file
>> ++ resize fs to 524272
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 524288
>> ++ create fs on image file 393216
>> ++ mount image file
>> ++ resize fs to 524288
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 1048576
>> ++ create fs on image file 393216
>> ++ mount image file
>> ++ resize fs to 1048576
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 7864336
>> ++ create fs on image file 393216
>> ++ mount image file
>> ++ resize fs to 7864336
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 8126464
>> ++ create fs on image file 393216
>> ++ mount image file
>> ++ resize fs to 8126464
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 8388608
>> ++ create fs on image file 393216
>> ++ mount image file
>> ++ resize fs to 8388608
>> ++ umount fs
>> ++ check fs
>> ++ truncate image file to 83886080
>> ++ create fs on image file 393216
>> ++ mount image file
>> ++ resize fs to 83886080
>> ++ umount fs
>> ++ check fs
>> diff --git a/tests/ext4/group b/tests/ext4/group
>> index 257bb646..f29d3de6 100644
>> --- a/tests/ext4/group
>> +++ b/tests/ext4/group
>> @@ -32,6 +32,7 @@
>>  027 auto quick fsmap
>>  028 auto quick fsmap
>>  029 auto quick fsmap
>> +030 auto quick ioctl resize
>>  271 auto rw quick
>>  301 aio auto ioctl rw stress defrag
>>  302 aio auto ioctl rw stress defrag
>> --
>> 2.16.0.rc0.223.g4a4ac83678-goog
>>

[PATCH v4] ext4: Ext4 online resize with bigalloc tests.

[harshads]

Add tests to verify Ext4 online resizing feature with bigalloc feature
enabled. We test various resizing scenarios with different cluster
sizes.

Signed-off-by: Harshad Shirwadkar <harshads@google.com>
---
 tests/ext4/030     | 174 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 172 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 3 files changed, 347 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

diff --git a/tests/ext4/030 b/tests/ext4/030
new file mode 100755
index 00000000..9e91b96a
--- /dev/null
+++ b/tests/ext4/030
@@ -0,0 +1,174 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize tests of small and crucial resizes with bigalloc
+# feature.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <harshads@google.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,	 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+BLK_SIZ=4096
+CLUSTER_SIZ=4096
+
+IMG_FILE=$SCRATCH_DIR/$seq.fs
+IMG_MNT=$SCRATCH_DIR/$seq.mnt
+
+## Num clusters to blocks.
+c2b()
+{
+	local clusters=$1
+	echo $(($clusters * $CLUSTER_SIZ / $BLK_SIZ))
+}
+
+_ext4_online_resize()
+{
+	## sizes are in number of blocks.
+	local original_size=$1
+	local final_size=$2
+
+	## Start with a clean image file.
+	echo "" > ${IMG_FILE}
+	echo "+++ truncate image file to $final_size" | \
+		tee -a $seqres.full
+	$XFS_IO_PROG -f -c "truncate $(($final_size * $BLK_SIZ))" ${IMG_FILE}
+	LOOP_DEVICE=`_create_loop_device $IMG_FILE || _fail "losetup failed"`
+
+	echo "+++ create fs on image file $original_size" | \
+		tee -a $seqres.full
+
+	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode -C $CLUSTER_SIZ \
+		-b $BLK_SIZ ${LOOP_DEVICE} $original_size >> \
+		$seqres.full 2>&1 || _fail "mkfs failed"
+
+	echo "+++ mount image file" | tee -a $seqres.full
+	$MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \
+		/dev/null 2>&1 || _fail "mount failed"
+
+	echo "+++ resize fs to $final_size" | tee -a $seqres.full
+
+	resize2fs -f ${LOOP_DEVICE} $final_size >> \
+		  $seqres.full 2>&1 || return 1
+
+	echo "+++ umount fs" | tee -a $seqres.full
+	$UMOUNT_PROG ${IMG_MNT}
+
+	echo "+++ check fs" | tee -a $seqres.full
+	_check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
+		_fail "fsck should not fail"
+	_destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
+}
+
+_cleanup()
+{
+	cd /
+	[ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
+	rm -f $tmp.*
+	$UMOUNT_PROG ${IMG_MNT} > /dev/null 2>&1
+	rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+_require_scratch
+# We use resize_inode to make sure that block group descriptor table
+# can be extended.
+_require_scratch_ext4_feature "bigalloc,resize_inode,metadata_csum"
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+rm -f $seqres.full
+
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+# Check if online resizing with bigalloc is supported by the kernel
+_ext4_online_resize 4096 8192 || \
+	_notrun "Kernel doesn't support online resizing with bigalloc"
+
+## We perform resizing to various multiples of block group sizes to
+## ensure that we cover maximum edge cases in the kernel code.
+for CLUSTER_SIZ in 4096 16384 65536; do
+	echo "++ set cluster size to $CLUSTER_SIZ" | tee -a $seqres.full
+
+	##
+	## Extend last group tests
+	##
+
+	## Extending a 1/2 block group to a 2/3 block group.
+	_ext4_online_resize $(c2b 16384) $(c2b 24576) \
+		|| _fail "resize failed"
+	## Extending a 2/3rd block group to one cluster less than a
+	## full block group.
+	_ext4_online_resize $(c2b 24576) $(c2b 32767) \
+		|| _fail "resize failed"
+	## Extending a 2/3rd block group to a full block group.
+	_ext4_online_resize $(c2b 24576) $(c2b 32768) \
+		|| _fail "resize failed"
+
+	##
+	## Extend last group and add more block groups.
+	##
+
+	## Extending a 2/3rd block group to 2 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 65536) \
+		|| _fail "resize failed"
+	## Extending a 2/3rd block group to 15 block groups and one
+	## cluster.
+	_ext4_online_resize $(c2b 24576) $(c2b 491521) \
+		|| _fail "resize failed"
+	## Extending a 2/3rd block group to 15 and a half block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 507904) \
+		|| _fail "resize failed"
+
+	## Extending a 2/3rd block group to 16 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 524288) \
+		|| _fail "resize failed"
+	## Extending a 2/3rd block group to 160 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 5242880) \
+		|| _fail "resize failed"
+
+	##
+	## Convert to meta_bg.
+	##
+
+	## Extending a 2/3rd block group to 1280 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 41943040) \
+		|| _fail "resize failed"
+done
+
+status=0
+exit
diff --git a/tests/ext4/030.out b/tests/ext4/030.out
new file mode 100644
index 00000000..49304654
--- /dev/null
+++ b/tests/ext4/030.out
@@ -0,0 +1,172 @@
+QA output created by 030
++++ truncate image file to 8192
++++ create fs on image file 4096
++++ mount image file
++++ resize fs to 8192
++++ umount fs
++++ check fs
+++ set cluster size to 4096
++++ truncate image file to 24576
++++ create fs on image file 16384
++++ mount image file
++++ resize fs to 24576
++++ umount fs
++++ check fs
++++ truncate image file to 32767
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 32767
++++ umount fs
++++ check fs
++++ truncate image file to 32768
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 32768
++++ umount fs
++++ check fs
++++ truncate image file to 65536
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 65536
++++ umount fs
++++ check fs
++++ truncate image file to 491521
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 491521
++++ umount fs
++++ check fs
++++ truncate image file to 507904
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 507904
++++ umount fs
++++ check fs
++++ truncate image file to 524288
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 524288
++++ umount fs
++++ check fs
++++ truncate image file to 5242880
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 5242880
++++ umount fs
++++ check fs
++++ truncate image file to 41943040
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 41943040
++++ umount fs
++++ check fs
+++ set cluster size to 16384
++++ truncate image file to 98304
++++ create fs on image file 65536
++++ mount image file
++++ resize fs to 98304
++++ umount fs
++++ check fs
++++ truncate image file to 131068
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 131068
++++ umount fs
++++ check fs
++++ truncate image file to 131072
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 131072
++++ umount fs
++++ check fs
++++ truncate image file to 262144
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 262144
++++ umount fs
++++ check fs
++++ truncate image file to 1966084
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 1966084
++++ umount fs
++++ check fs
++++ truncate image file to 2031616
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 2031616
++++ umount fs
++++ check fs
++++ truncate image file to 2097152
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 2097152
++++ umount fs
++++ check fs
++++ truncate image file to 20971520
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 20971520
++++ umount fs
++++ check fs
++++ truncate image file to 167772160
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 167772160
++++ umount fs
++++ check fs
+++ set cluster size to 65536
++++ truncate image file to 393216
++++ create fs on image file 262144
++++ mount image file
++++ resize fs to 393216
++++ umount fs
++++ check fs
++++ truncate image file to 524272
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 524272
++++ umount fs
++++ check fs
++++ truncate image file to 524288
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 524288
++++ umount fs
++++ check fs
++++ truncate image file to 1048576
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 1048576
++++ umount fs
++++ check fs
++++ truncate image file to 7864336
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 7864336
++++ umount fs
++++ check fs
++++ truncate image file to 8126464
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 8126464
++++ umount fs
++++ check fs
++++ truncate image file to 8388608
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 8388608
++++ umount fs
++++ check fs
++++ truncate image file to 83886080
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 83886080
++++ umount fs
++++ check fs
++++ truncate image file to 671088640
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 671088640
++++ umount fs
++++ check fs
diff --git a/tests/ext4/group b/tests/ext4/group
index 257bb646..f29d3de6 100644
--- a/tests/ext4/group
+++ b/tests/ext4/group
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag
-- 
2.16.0.rc1.238.g530d649a79-goog

Re: [PATCH v4] ext4: Ext4 online resize with bigalloc tests.

[Eryu Guan]

On Tue, Jan 23, 2018 at 01:53:57PM -0800, harshads wrote:
> Add tests to verify Ext4 online resizing feature with bigalloc feature
> enabled. We test various resizing scenarios with different cluster
> sizes.
> 
> Signed-off-by: Harshad Shirwadkar <harshads@google.com>

Looks good to me, thanks! Just two minor issues inline.

> ---
>  tests/ext4/030     | 174 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/ext4/030.out | 172 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/ext4/group   |   1 +
>  3 files changed, 347 insertions(+)
>  create mode 100755 tests/ext4/030
>  create mode 100644 tests/ext4/030.out
> 
> diff --git a/tests/ext4/030 b/tests/ext4/030
> new file mode 100755
> index 00000000..9e91b96a
> --- /dev/null
> +++ b/tests/ext4/030
> @@ -0,0 +1,174 @@
> +#! /bin/bash
> +# FS QA Test ext4/030
> +#
> +# Ext4 online resize tests of small and crucial resizes with bigalloc
> +# feature.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
> +#
> +# Author: Harshad Shirwadkar <harshads@google.com>
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,	 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +BLK_SIZ=4096
> +CLUSTER_SIZ=4096
> +
> +IMG_FILE=$SCRATCH_DIR/$seq.fs
> +IMG_MNT=$SCRATCH_DIR/$seq.mnt
> +
> +## Num clusters to blocks.
> +c2b()
> +{
> +	local clusters=$1
> +	echo $(($clusters * $CLUSTER_SIZ / $BLK_SIZ))
> +}
> +
> +_ext4_online_resize()
> +{
> +	## sizes are in number of blocks.
> +	local original_size=$1
> +	local final_size=$2
> +
> +	## Start with a clean image file.
> +	echo "" > ${IMG_FILE}
> +	echo "+++ truncate image file to $final_size" | \
> +		tee -a $seqres.full
> +	$XFS_IO_PROG -f -c "truncate $(($final_size * $BLK_SIZ))" ${IMG_FILE}
> +	LOOP_DEVICE=`_create_loop_device $IMG_FILE || _fail "losetup failed"`
> +
> +	echo "+++ create fs on image file $original_size" | \
> +		tee -a $seqres.full
> +
> +	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode -C $CLUSTER_SIZ \
> +		-b $BLK_SIZ ${LOOP_DEVICE} $original_size >> \
> +		$seqres.full 2>&1 || _fail "mkfs failed"
> +
> +	echo "+++ mount image file" | tee -a $seqres.full
> +	$MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \
> +		/dev/null 2>&1 || _fail "mount failed"
> +
> +	echo "+++ resize fs to $final_size" | tee -a $seqres.full
> +
> +	resize2fs -f ${LOOP_DEVICE} $final_size >> \
> +		  $seqres.full 2>&1 || return 1
> +
> +	echo "+++ umount fs" | tee -a $seqres.full
> +	$UMOUNT_PROG ${IMG_MNT}
> +
> +	echo "+++ check fs" | tee -a $seqres.full
> +	_check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
> +		_fail "fsck should not fail"
> +	_destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
> +}
> +
> +_cleanup()
> +{
> +	cd /
> +	[ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
> +	rm -f $tmp.*
> +	$UMOUNT_PROG ${IMG_MNT} > /dev/null 2>&1
> +	rm -f ${IMG_FILE} > /dev/null 2>&1
> +}
> +
> +# get standard environment and checks
> +. ./common/rc
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +_supported_fs ext4
> +_supported_os Linux
> +
> +_require_loop
> +_require_scratch
> +# We use resize_inode to make sure that block group descriptor table
> +# can be extended.
> +_require_scratch_ext4_feature "bigalloc,resize_inode,metadata_csum"

What about metadata_csum? Seems it's not used in the actual resize test.
Meant to delete it?

> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_scratch_mount
> +
> +rm -f $seqres.full
> +
> +mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
> +
> +# Check if online resizing with bigalloc is supported by the kernel
> +_ext4_online_resize 4096 8192 || \
> +	_notrun "Kernel doesn't support online resizing with bigalloc"

Hmm, how do we tell if it's a test failure or lack of kernel support?
How about adding an extra argument to ext4_online_resize to indicate
that this is a test run, so it checks resize2fs output to see if the
failure is really due to lacking support in kernel. Otherwise we treat
all resize2fs failures as bugs.

e.g. resize2fs prints:
"resize2fs: Operation not supported While checking for on-line resizing support"

Thanks,
Eryu

[PATCH v5] ext4: Ext4 online resize with bigalloc tests.

[harshads]

Add tests to verify Ext4 online resizing feature with bigalloc feature
enabled. We test various resizing scenarios with different cluster
sizes.

Signed-off-by: Harshad Shirwadkar <harshads@google.com>
---
 tests/ext4/030     | 171 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 172 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 3 files changed, 344 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

diff --git a/tests/ext4/030 b/tests/ext4/030
new file mode 100755
index 00000000..1d42b514
--- /dev/null
+++ b/tests/ext4/030
@@ -0,0 +1,171 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize tests of small and crucial resizes with bigalloc
+# feature.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <harshads@google.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,	 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+BLK_SIZ=4096
+CLUSTER_SIZ=4096
+
+IMG_FILE=$SCRATCH_DIR/$seq.fs
+IMG_MNT=$SCRATCH_DIR/$seq.mnt
+
+## Num clusters to blocks.
+c2b()
+{
+	local clusters=$1
+	echo $(($clusters * $CLUSTER_SIZ / $BLK_SIZ))
+}
+
+_ext4_online_resize()
+{
+	## sizes are in number of blocks.
+	local original_size=$1
+	local final_size=$2
+	local check_if_supported=$([ "$3" = "1" ] && echo "1" || echo "0")
+
+	## Start with a clean image file.
+	echo "" > ${IMG_FILE}
+	echo "+++ truncate image file to $final_size" | \
+		tee -a $seqres.full
+	$XFS_IO_PROG -f -c "truncate $(($final_size * $BLK_SIZ))" ${IMG_FILE}
+	LOOP_DEVICE=`_create_loop_device $IMG_FILE || _fail "losetup failed"`
+
+	echo "+++ create fs on image file $original_size" | \
+		tee -a $seqres.full
+
+	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode -C $CLUSTER_SIZ \
+		-b $BLK_SIZ ${LOOP_DEVICE} $original_size >> \
+		$seqres.full 2>&1 || _fail "mkfs failed"
+
+	echo "+++ mount image file" | tee -a $seqres.full
+	$MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \
+		/dev/null 2>&1 || _fail "mount failed"
+
+	echo "+++ resize fs to $final_size" | tee -a $seqres.full
+
+	resize_output=$(resize2fs -f ${LOOP_DEVICE} $final_size 2>&1)
+	if [ $? -ne 0 ]; then
+		if [ $check_if_supported -eq 1 ]; then
+			echo $resize_output | \
+			grep "resize2fs: Operation not supported While checking for on-line resizing support" \
+				&& _notrun "online resizing not supported with bigalloc"
+		fi
+		_fail "resize failed"
+	fi
+	echo $resize_output >> $seqres.full
+	echo "+++ umount fs" | tee -a $seqres.full
+	$UMOUNT_PROG ${IMG_MNT}
+
+	echo "+++ check fs" | tee -a $seqres.full
+	_check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
+		_fail "fsck should not fail"
+	_destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
+}
+
+_cleanup()
+{
+	cd /
+	[ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
+	rm -f $tmp.*
+	$UMOUNT_PROG ${IMG_MNT} > /dev/null 2>&1
+	rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+_require_scratch
+# We use resize_inode to make sure that block group descriptor table
+# can be extended.
+_require_scratch_ext4_feature "bigalloc,resize_inode"
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+rm -f $seqres.full
+
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+# Check if online resizing with bigalloc is supported by the kernel
+_ext4_online_resize 4096 8192 1
+
+## We perform resizing to various multiples of block group sizes to
+## ensure that we cover maximum edge cases in the kernel code.
+for CLUSTER_SIZ in 4096 16384 65536; do
+	echo "++ set cluster size to $CLUSTER_SIZ" | tee -a $seqres.full
+
+	##
+	## Extend last group tests
+	##
+
+	## Extending a 1/2 block group to a 2/3 block group.
+	_ext4_online_resize $(c2b 16384) $(c2b 24576)
+	## Extending a 2/3rd block group to one cluster less than a
+	## full block group.
+	_ext4_online_resize $(c2b 24576) $(c2b 32767)
+	## Extending a 2/3rd block group to a full block group.
+	_ext4_online_resize $(c2b 24576) $(c2b 32768)
+
+	##
+	## Extend last group and add more block groups.
+	##
+
+	## Extending a 2/3rd block group to 2 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 65536)
+	## Extending a 2/3rd block group to 15 block groups and one
+	## cluster.
+	_ext4_online_resize $(c2b 24576) $(c2b 491521)
+	## Extending a 2/3rd block group to 15 and a half block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 507904)
+	## Extending a 2/3rd block group to 16 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 524288)
+	## Extending a 2/3rd block group to 160 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 5242880)
+
+	##
+	## Convert to meta_bg.
+	##
+
+	## Extending a 2/3rd block group to 1280 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 41943040)
+done
+
+status=0
+exit
diff --git a/tests/ext4/030.out b/tests/ext4/030.out
new file mode 100644
index 00000000..49304654
--- /dev/null
+++ b/tests/ext4/030.out
@@ -0,0 +1,172 @@
+QA output created by 030
++++ truncate image file to 8192
++++ create fs on image file 4096
++++ mount image file
++++ resize fs to 8192
++++ umount fs
++++ check fs
+++ set cluster size to 4096
++++ truncate image file to 24576
++++ create fs on image file 16384
++++ mount image file
++++ resize fs to 24576
++++ umount fs
++++ check fs
++++ truncate image file to 32767
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 32767
++++ umount fs
++++ check fs
++++ truncate image file to 32768
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 32768
++++ umount fs
++++ check fs
++++ truncate image file to 65536
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 65536
++++ umount fs
++++ check fs
++++ truncate image file to 491521
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 491521
++++ umount fs
++++ check fs
++++ truncate image file to 507904
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 507904
++++ umount fs
++++ check fs
++++ truncate image file to 524288
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 524288
++++ umount fs
++++ check fs
++++ truncate image file to 5242880
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 5242880
++++ umount fs
++++ check fs
++++ truncate image file to 41943040
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 41943040
++++ umount fs
++++ check fs
+++ set cluster size to 16384
++++ truncate image file to 98304
++++ create fs on image file 65536
++++ mount image file
++++ resize fs to 98304
++++ umount fs
++++ check fs
++++ truncate image file to 131068
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 131068
++++ umount fs
++++ check fs
++++ truncate image file to 131072
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 131072
++++ umount fs
++++ check fs
++++ truncate image file to 262144
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 262144
++++ umount fs
++++ check fs
++++ truncate image file to 1966084
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 1966084
++++ umount fs
++++ check fs
++++ truncate image file to 2031616
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 2031616
++++ umount fs
++++ check fs
++++ truncate image file to 2097152
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 2097152
++++ umount fs
++++ check fs
++++ truncate image file to 20971520
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 20971520
++++ umount fs
++++ check fs
++++ truncate image file to 167772160
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 167772160
++++ umount fs
++++ check fs
+++ set cluster size to 65536
++++ truncate image file to 393216
++++ create fs on image file 262144
++++ mount image file
++++ resize fs to 393216
++++ umount fs
++++ check fs
++++ truncate image file to 524272
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 524272
++++ umount fs
++++ check fs
++++ truncate image file to 524288
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 524288
++++ umount fs
++++ check fs
++++ truncate image file to 1048576
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 1048576
++++ umount fs
++++ check fs
++++ truncate image file to 7864336
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 7864336
++++ umount fs
++++ check fs
++++ truncate image file to 8126464
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 8126464
++++ umount fs
++++ check fs
++++ truncate image file to 8388608
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 8388608
++++ umount fs
++++ check fs
++++ truncate image file to 83886080
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 83886080
++++ umount fs
++++ check fs
++++ truncate image file to 671088640
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 671088640
++++ umount fs
++++ check fs
diff --git a/tests/ext4/group b/tests/ext4/group
index 257bb646..f29d3de6 100644
--- a/tests/ext4/group
+++ b/tests/ext4/group
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag
-- 
2.16.0.rc1.238.g530d649a79-goog

Re: [PATCH v5] ext4: Ext4 online resize with bigalloc tests.

[Eryu Guan]

On Wed, Jan 24, 2018 at 02:58:12PM -0800, harshads wrote:
> Add tests to verify Ext4 online resizing feature with bigalloc feature
> enabled. We test various resizing scenarios with different cluster
> sizes.
> 
> Signed-off-by: Harshad Shirwadkar <harshads@google.com>

Thanks for all the revisions! I made some minor edits and queued it for
next release. Please see below.

> ---
>  tests/ext4/030     | 171 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/ext4/030.out | 172 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/ext4/group   |   1 +
>  3 files changed, 344 insertions(+)
>  create mode 100755 tests/ext4/030
>  create mode 100644 tests/ext4/030.out
> 
> diff --git a/tests/ext4/030 b/tests/ext4/030
> new file mode 100755
> index 00000000..1d42b514
> --- /dev/null
> +++ b/tests/ext4/030
> @@ -0,0 +1,171 @@
> +#! /bin/bash
> +# FS QA Test ext4/030
> +#
> +# Ext4 online resize tests of small and crucial resizes with bigalloc
> +# feature.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
> +#
> +# Author: Harshad Shirwadkar <harshads@google.com>
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,	 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +BLK_SIZ=4096
> +CLUSTER_SIZ=4096
> +
> +IMG_FILE=$SCRATCH_DIR/$seq.fs
> +IMG_MNT=$SCRATCH_DIR/$seq.mnt
> +
> +## Num clusters to blocks.
> +c2b()
> +{
> +	local clusters=$1
> +	echo $(($clusters * $CLUSTER_SIZ / $BLK_SIZ))
> +}
> +
> +_ext4_online_resize()

I removed the leading underscore for local function.

> +{
> +	## sizes are in number of blocks.
> +	local original_size=$1
> +	local final_size=$2
> +	local check_if_supported=$([ "$3" = "1" ] && echo "1" || echo "0")

	local check_if_supported=${3:-0}

> +
> +	## Start with a clean image file.
> +	echo "" > ${IMG_FILE}
> +	echo "+++ truncate image file to $final_size" | \
> +		tee -a $seqres.full
> +	$XFS_IO_PROG -f -c "truncate $(($final_size * $BLK_SIZ))" ${IMG_FILE}
> +	LOOP_DEVICE=`_create_loop_device $IMG_FILE || _fail "losetup failed"`

No need to call _fail here, _create_loop_device already did that.

> +
> +	echo "+++ create fs on image file $original_size" | \
> +		tee -a $seqres.full
> +
> +	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode -C $CLUSTER_SIZ \
> +		-b $BLK_SIZ ${LOOP_DEVICE} $original_size >> \
> +		$seqres.full 2>&1 || _fail "mkfs failed"
> +
> +	echo "+++ mount image file" | tee -a $seqres.full
> +	$MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \
> +		/dev/null 2>&1 || _fail "mount failed"
> +
> +	echo "+++ resize fs to $final_size" | tee -a $seqres.full
> +
> +	resize_output=$(resize2fs -f ${LOOP_DEVICE} $final_size 2>&1)

I changed this to let resize2fs dump output to a tmp file, saving
multi-line outputs into a variable would cause subtle problems that are
hard to find/debug. e.g. echoing a such variable to a pipe without
proper quotes would change the output in subtle ways, and we did hit
such problems before.

> +	if [ $? -ne 0 ]; then
> +		if [ $check_if_supported -eq 1 ]; then
> +			echo $resize_output | \
> +			grep "resize2fs: Operation not supported While checking for on-line resizing support" \
> +				&& _notrun "online resizing not supported with bigalloc"
> +		fi
> +		_fail "resize failed"
> +	fi
> +	echo $resize_output >> $seqres.full

Then grep the tmp file and append it to $seqres.full.

Thanks,
Eryu

> +	echo "+++ umount fs" | tee -a $seqres.full
> +	$UMOUNT_PROG ${IMG_MNT}
> +
> +	echo "+++ check fs" | tee -a $seqres.full
> +	_check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
> +		_fail "fsck should not fail"
> +	_destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
> +}
> +
> +_cleanup()
> +{
> +	cd /
> +	[ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
> +	rm -f $tmp.*
> +	$UMOUNT_PROG ${IMG_MNT} > /dev/null 2>&1
> +	rm -f ${IMG_FILE} > /dev/null 2>&1
> +}
> +
> +# get standard environment and checks
> +. ./common/rc
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +_supported_fs ext4
> +_supported_os Linux
> +
> +_require_loop
> +_require_scratch
> +# We use resize_inode to make sure that block group descriptor table
> +# can be extended.
> +_require_scratch_ext4_feature "bigalloc,resize_inode"
> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_scratch_mount
> +
> +rm -f $seqres.full
> +
> +mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
> +
> +# Check if online resizing with bigalloc is supported by the kernel
> +_ext4_online_resize 4096 8192 1
> +
> +## We perform resizing to various multiples of block group sizes to
> +## ensure that we cover maximum edge cases in the kernel code.
> +for CLUSTER_SIZ in 4096 16384 65536; do
> +	echo "++ set cluster size to $CLUSTER_SIZ" | tee -a $seqres.full
> +
> +	##
> +	## Extend last group tests
> +	##
> +
> +	## Extending a 1/2 block group to a 2/3 block group.
> +	_ext4_online_resize $(c2b 16384) $(c2b 24576)
> +	## Extending a 2/3rd block group to one cluster less than a
> +	## full block group.
> +	_ext4_online_resize $(c2b 24576) $(c2b 32767)
> +	## Extending a 2/3rd block group to a full block group.
> +	_ext4_online_resize $(c2b 24576) $(c2b 32768)
> +
> +	##
> +	## Extend last group and add more block groups.
> +	##
> +
> +	## Extending a 2/3rd block group to 2 block groups.
> +	_ext4_online_resize $(c2b 24576) $(c2b 65536)
> +	## Extending a 2/3rd block group to 15 block groups and one
> +	## cluster.
> +	_ext4_online_resize $(c2b 24576) $(c2b 491521)
> +	## Extending a 2/3rd block group to 15 and a half block groups.
> +	_ext4_online_resize $(c2b 24576) $(c2b 507904)
> +	## Extending a 2/3rd block group to 16 block groups.
> +	_ext4_online_resize $(c2b 24576) $(c2b 524288)
> +	## Extending a 2/3rd block group to 160 block groups.
> +	_ext4_online_resize $(c2b 24576) $(c2b 5242880)
> +
> +	##
> +	## Convert to meta_bg.
> +	##
> +
> +	## Extending a 2/3rd block group to 1280 block groups.
> +	_ext4_online_resize $(c2b 24576) $(c2b 41943040)
> +done
> +
> +status=0
> +exit
> diff --git a/tests/ext4/030.out b/tests/ext4/030.out
> new file mode 100644
> index 00000000..49304654
> --- /dev/null
> +++ b/tests/ext4/030.out
> @@ -0,0 +1,172 @@
> +QA output created by 030
> ++++ truncate image file to 8192
> ++++ create fs on image file 4096
> ++++ mount image file
> ++++ resize fs to 8192
> ++++ umount fs
> ++++ check fs
> +++ set cluster size to 4096
> ++++ truncate image file to 24576
> ++++ create fs on image file 16384
> ++++ mount image file
> ++++ resize fs to 24576
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 32767
> ++++ create fs on image file 24576
> ++++ mount image file
> ++++ resize fs to 32767
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 32768
> ++++ create fs on image file 24576
> ++++ mount image file
> ++++ resize fs to 32768
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 65536
> ++++ create fs on image file 24576
> ++++ mount image file
> ++++ resize fs to 65536
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 491521
> ++++ create fs on image file 24576
> ++++ mount image file
> ++++ resize fs to 491521
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 507904
> ++++ create fs on image file 24576
> ++++ mount image file
> ++++ resize fs to 507904
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 524288
> ++++ create fs on image file 24576
> ++++ mount image file
> ++++ resize fs to 524288
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 5242880
> ++++ create fs on image file 24576
> ++++ mount image file
> ++++ resize fs to 5242880
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 41943040
> ++++ create fs on image file 24576
> ++++ mount image file
> ++++ resize fs to 41943040
> ++++ umount fs
> ++++ check fs
> +++ set cluster size to 16384
> ++++ truncate image file to 98304
> ++++ create fs on image file 65536
> ++++ mount image file
> ++++ resize fs to 98304
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 131068
> ++++ create fs on image file 98304
> ++++ mount image file
> ++++ resize fs to 131068
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 131072
> ++++ create fs on image file 98304
> ++++ mount image file
> ++++ resize fs to 131072
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 262144
> ++++ create fs on image file 98304
> ++++ mount image file
> ++++ resize fs to 262144
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 1966084
> ++++ create fs on image file 98304
> ++++ mount image file
> ++++ resize fs to 1966084
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 2031616
> ++++ create fs on image file 98304
> ++++ mount image file
> ++++ resize fs to 2031616
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 2097152
> ++++ create fs on image file 98304
> ++++ mount image file
> ++++ resize fs to 2097152
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 20971520
> ++++ create fs on image file 98304
> ++++ mount image file
> ++++ resize fs to 20971520
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 167772160
> ++++ create fs on image file 98304
> ++++ mount image file
> ++++ resize fs to 167772160
> ++++ umount fs
> ++++ check fs
> +++ set cluster size to 65536
> ++++ truncate image file to 393216
> ++++ create fs on image file 262144
> ++++ mount image file
> ++++ resize fs to 393216
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 524272
> ++++ create fs on image file 393216
> ++++ mount image file
> ++++ resize fs to 524272
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 524288
> ++++ create fs on image file 393216
> ++++ mount image file
> ++++ resize fs to 524288
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 1048576
> ++++ create fs on image file 393216
> ++++ mount image file
> ++++ resize fs to 1048576
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 7864336
> ++++ create fs on image file 393216
> ++++ mount image file
> ++++ resize fs to 7864336
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 8126464
> ++++ create fs on image file 393216
> ++++ mount image file
> ++++ resize fs to 8126464
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 8388608
> ++++ create fs on image file 393216
> ++++ mount image file
> ++++ resize fs to 8388608
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 83886080
> ++++ create fs on image file 393216
> ++++ mount image file
> ++++ resize fs to 83886080
> ++++ umount fs
> ++++ check fs
> ++++ truncate image file to 671088640
> ++++ create fs on image file 393216
> ++++ mount image file
> ++++ resize fs to 671088640
> ++++ umount fs
> ++++ check fs
> diff --git a/tests/ext4/group b/tests/ext4/group
> index 257bb646..f29d3de6 100644
> --- a/tests/ext4/group
> +++ b/tests/ext4/group
> @@ -32,6 +32,7 @@
>  027 auto quick fsmap
>  028 auto quick fsmap
>  029 auto quick fsmap
> +030 auto quick ioctl resize
>  271 auto rw quick
>  301 aio auto ioctl rw stress defrag
>  302 aio auto ioctl rw stress defrag
> -- 
> 2.16.0.rc1.238.g530d649a79-goog
>