[PATCH 0/3] xfstests: Add a group of reservation space test

[PATCH 0/3] xfstests: Add a group of reservation space test

[WuBo]

This patch set add a group of reservation space test. Especailly for btrfs.
It includes three parts: copy workload, prealloc and write posix.

For test 264, I hope it's usefull for Josef's reserve improve work.
For test 265 and 266, the current btrfs is not pass yet.

TO avoid fill the huge disk, I just resize the filesystem size which NOT 
all filesystems supportted. Here I just make these tests support btrfs only,
but it's can ream to ext3/4 if needed.

Wu Bo (3):
  xfstests 264: add a copy and reserve test
  xfstests 265: add a prealloc and reserve test
  xfstests 266: add a write and reserve test

 264     |  158 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 264.out |    6 ++
 265     |  107 ++++++++++++++++++++++++++++++++++++++++++
 265.out |    5 ++
 266     |  104 +++++++++++++++++++++++++++++++++++++++++
 266.out |    5 ++
 group   |    3 +
 7 files changed, 388 insertions(+), 0 deletions(-)
 create mode 100755 264
 create mode 100644 264.out
 create mode 100755 265
 create mode 100644 265.out
 create mode 100755 266
 create mode 100644 266.out

-- 
1.7.3.1

[PATCH 1/3] xfstests 264: add a copy and reserve test

[WuBo]

This test is a stress test. It creates a set of threads for coping small files 
into disk. I use a 2G disk for test, the ENOSPC arises usually but the disk is 
not full under kenerl 3.0 with intel64.

Signed-off-by: Wu Bo <Wu.Bo@cn.fujitsu.com>
---
 264     |  158 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 264.out |    6 ++
 group   |    1 +
 3 files changed, 165 insertions(+), 0 deletions(-)
 create mode 100755 264
 create mode 100644 264.out

diff --git a/264 b/264
new file mode 100755
index 0000000..8df56c2
--- /dev/null
+++ b/264
@@ -0,0 +1,158 @@
+#! /bin/bash
+# FS QA Test No. 264
+#
+# reservation test with heavy cp workload
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2011-2012 Fujitsu, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+#creator
+owner=wu.bo@cn.fujitsu.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=0	# success is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $TEST_DIR/* $tmp.*
+	
+	if [ $_resize -eq 1 ]
+	then
+		btrfs filesystem resize ${_old}B $TEST_DIR > /dev/null 2>&1
+	fi
+	
+	_cleanup_testdir
+}
+
+. ./common.rc
+. ./common.filter
+
+threads=0  # will be set later
+count=1
+
+_threads_set()
+{
+	_cpu_num=`cat /proc/cpuinfo | grep "processor" | wc -l`
+	threads=$(( $_cpu_num * 50 ))
+	if [ $threads -gt 200 ]
+	then
+		threads=200
+	fi
+
+	echo "create $threads threads to run..."
+}
+
+_file_create()
+{
+	_i=0
+
+	if ! mkdir $TEST_DIR/origin
+	then
+		echo "mkdir origin err"
+		status=1
+		exit
+	fi
+
+	cd $TEST_DIR/origin
+
+	_disksize=`df --block-size=1 | grep $TEST_DEV | awk '{print $2}'`
+	_disksize=$(( $_disksize / 3 ))
+	_num=$(( $_disksize / $count / $threads / 4096 ))
+	_count=$count
+	while [ $_i -lt $_num ]
+	do
+		dd if=/dev/zero of=file_$_i bs=4096 count=$_count > /dev/null 2>&1
+		_i=$(( $_i + 1 ))
+	done
+
+	cd $here
+}
+
+_porter()
+{
+	_suffix=$1
+
+	if ! mkdir $TEST_DIR/sub_$_suffix
+	then
+		echo "mkdir sub_xxx err"
+		status=1
+		exit
+	fi
+
+	cp -r $TEST_DIR/origin $TEST_DIR/sub_$_suffix
+
+	sync
+}
+
+_do_workload()
+{
+	_pid=1
+	
+	rm -rf $TEST_DIR/*
+
+	_threads_set
+	_file_create
+
+	_threads=$threads
+
+	while [ $_pid -lt $_threads ]
+	do
+		_porter $_pid &
+
+		_pid=$(( $_pid + 1 ))
+	done
+
+	wait
+}
+
+# real QA test starts here
+_supported_fs btrfs
+_supported_os IRIX Linux
+
+_setup_testdir
+_check_test_fs
+
+echo "------------------------------"
+echo "start the workload"
+echo "------------------------------"
+	
+_resize=0
+_new=$(( 2 * 1024 * 1024 * 1024 )) # 2G
+_old=`df --block-size=1 | grep $TEST_DEV | awk '{print $2}'`
+if [ $_new -lt $_old ]
+then
+	btrfs filesystem resize ${_new}B $TEST_DIR > /dev/null 2>&1
+	if [ $? -ne 0 ]
+	then
+		echo "btrfs filesystem resize err"
+		status=1
+		exit
+	fi
+	_resize=1
+fi
+
+_do_workload
+
+echo "done"
+exit
diff --git a/264.out b/264.out
new file mode 100644
index 0000000..b218578
--- /dev/null
+++ b/264.out
@@ -0,0 +1,6 @@
+QA output created by 264
+------------------------------
+start the workload
+------------------------------
+create 100 threads to run...
+done
diff --git a/group b/group
index 2a8970c..301aaec 100644
--- a/group
+++ b/group
@@ -377,3 +377,4 @@ deprecated
 261 auto quick quota
 262 auto quick quota
 263 rw auto quick
+264 auto rw
-- 
1.7.3.1

Re: [PATCH 1/3] xfstests 264: add a copy and reserve test

[Christoph Hellwig]

On Thu, Nov 03, 2011 at 11:08:55AM +0800, WuBo wrote:
> This test is a stress test. It creates a set of threads for coping small files 
> into disk. I use a 2G disk for test, the ENOSPC arises usually but the disk is 
> not full under kenerl 3.0 with intel64.

It seems like you really want to use _scratch_mkfs_sized instead of the
btrfs-specific resize option and make the test generic.  Also we already
have used up these test numbers in the xfstests-dev repository, and I
will commit another batch of tests today.

Re: [PATCH 1/3] xfstests 264: add a copy and reserve test

[WuBo]

On 11/03/2011 02:55 PM, Christoph Hellwig wrote:
> On Thu, Nov 03, 2011 at 11:08:55AM +0800, WuBo wrote:
>> This test is a stress test. It creates a set of threads for coping small files 
>> into disk. I use a 2G disk for test, the ENOSPC arises usually but the disk is 
>> not full under kenerl 3.0 with intel64.
> 
> It seems like you really want to use _scratch_mkfs_sized instead of the
> btrfs-specific resize option and make the test generic.  Also we already
OK, thanks for _scratch_mkfs_sized, It's will make these tests generic.

> have used up these test numbers in the xfstests-dev repository, and I
> will commit another batch of tests today.
I will change my test numbers in next version.

thanks,
wubo

> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

[PATCH 2/3] xfstests 265: add a prealloc and reserve test

[WuBo]

This test is for preallocation test. If the disk is full, just with a prealloc
file has some free space that prealloc early. We need to check whether the write
to the free space is success or not.

Signed-off-by: Wu Bo <Wu.Bo@cn.fujitsu.com>
---
 265     |  107 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 265.out |    5 +++
 group   |    1 +
 3 files changed, 113 insertions(+), 0 deletions(-)
 create mode 100755 265
 create mode 100644 265.out

diff --git a/265 b/265
new file mode 100755
index 0000000..13c3520
--- /dev/null
+++ b/265
@@ -0,0 +1,107 @@
+#! /bin/bash
+# FS QA Test No. 265
+#
+# preallocation test
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2011-2012 Fujitsu, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+#creator
+owner=wu.bo@cn.fujitsu.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=0    # success is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	rm -f $TEST_DIR/* $tmp.*
+	
+	if [ $_resize -eq 1 ]
+	then
+		btrfs filesystem resize ${_old}B $TEST_DIR > /dev/null 2>&1
+	fi
+	
+	_cleanup_testdir
+}
+
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs btrfs
+_supported_os IRIX Linux
+
+_setup_testdir
+_check_test_fs
+
+echo "------------------------------"
+echo "preallocation test"
+echo "------------------------------"
+
+_resize=0
+_new=$(( 1 * 1024 * 1024 * 1024 )) # 1G
+_old=`df --block-size=1 | grep $TEST_DEV | awk '{print $2}'`
+if [ $_new -lt $_old ]
+then
+	btrfs filesystem resize ${_new}B $TEST_DIR > /dev/null 2>&1
+	if [ $? -ne 0 ]
+	then
+		echo "btrfs filesystem resize err"
+		status=1
+		exit
+	fi
+	_resize=1
+fi
+
+rm -rf $TEST_DIR/*
+cd $TEST_DIR
+dd if=/dev/zero of=test bs=4K count=1 > /dev/null 2>&1
+if [ $? -ne 0 ]
+then
+	echo "create file err"
+	status=1
+	exit
+fi
+
+fallocate -n -o 4K -l 1M test > /dev/null 2>&1
+if [ $? -ne 0 ]
+then
+	echo "fallocate file err"
+	status=1
+	exit
+fi
+
+dd if=/dev/zero of=tmp1 bs=1M > /dev/null 2>&1
+dd if=/dev/zero of=tmp2 bs=4K > /dev/null 2>&1
+sync
+
+dd if=/dev/zero of=test seek=1 bs=4K count=2 conv=notrunc > /dev/null 2>&1
+if [ $? -ne 0 ]
+then
+	echo "fill prealloc range err"
+	status=1
+	exit
+fi
+
+echo "done"
+exit
diff --git a/265.out b/265.out
new file mode 100644
index 0000000..202ca6a
--- /dev/null
+++ b/265.out
@@ -0,0 +1,5 @@
+QA output created by 265
+------------------------------
+preallocation test
+------------------------------
+done
diff --git a/group b/group
index 301aaec..f76ab8c 100644
--- a/group
+++ b/group
@@ -378,3 +378,4 @@ deprecated
 262 auto quick quota
 263 rw auto quick
 264 auto rw
+265 auto rw
-- 
1.7.3.1

Re: [PATCH 2/3] xfstests 265: add a prealloc and reserve test

[Ben Myers]

Hi Wu Bo,

On Thu, Nov 03, 2011 at 11:09:00AM +0800, WuBo wrote:
> This test is for preallocation test. If the disk is full, just with a prealloc
> file has some free space that prealloc early. We need to check whether the write
> to the free space is success or not.
> 
> Signed-off-by: Wu Bo <Wu.Bo@cn.fujitsu.com>

This test is failing for me because I don't have fallocate installed.  I
suggest the test could to be changed to check for binaries it uses,
possibly the version of those binaries, and then not run unless the
right ones are installed.  But the best I can do right now is make a
note of it.

Just a heads up.  ;)

Regards,
Ben

Re: [PATCH 2/3] xfstests 265: add a prealloc and reserve test

[Christoph Hellwig]

On Tue, Nov 15, 2011 at 12:21:13PM -0600, Ben Myers wrote:
> Hi Wu Bo,
> 
> On Thu, Nov 03, 2011 at 11:09:00AM +0800, WuBo wrote:
> > This test is for preallocation test. If the disk is full, just with a prealloc
> > file has some free space that prealloc early. We need to check whether the write
> > to the free space is success or not.
> > 
> > Signed-off-by: Wu Bo <Wu.Bo@cn.fujitsu.com>
> 
> This test is failing for me because I don't have fallocate installed.  I
> suggest the test could to be changed to check for binaries it uses,
> possibly the version of those binaries, and then not run unless the
> right ones are installed.  But the best I can do right now is make a
> note of it.

It might be even better to just use the xfs_io falloc command as we
generally expect an uptodate xfs_io for use with xfstests.

Re: [PATCH 2/3] xfstests 265: add a prealloc and reserve test

[WuBo]

On 11/16/2011 03:08 AM, Christoph Hellwig wrote:
> On Tue, Nov 15, 2011 at 12:21:13PM -0600, Ben Myers wrote:
>> Hi Wu Bo,
>>
>> On Thu, Nov 03, 2011 at 11:09:00AM +0800, WuBo wrote:
>>> This test is for preallocation test. If the disk is full, just with a prealloc
>>> file has some free space that prealloc early. We need to check whether the write
>>> to the free space is success or not.
>>>
>>> Signed-off-by: Wu Bo <Wu.Bo@cn.fujitsu.com>
>>
>> This test is failing for me because I don't have fallocate installed.  I
>> suggest the test could to be changed to check for binaries it uses,
>> possibly the version of those binaries, and then not run unless the
>> right ones are installed.  But the best I can do right now is make a
>> note of it.
> 
> It might be even better to just use the xfs_io falloc command as we
> generally expect an uptodate xfs_io for use with xfstests.

Got it.

thanks,
wubo

> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

[PATCH 3/3] xfstests 266: add a write and reserve test

[WuBo]

This test is for write-posix test. If writing a file when the disk is almost
full, the posix wants the call to write as much as possible but not none.

quote the POSIX:
If a write() requests that more bytes be written than there is room for
(for example, [XSI] [Option Start] the process' file size limit or
[Option End] the physical end of a medium), only as many bytes as there
is room for shall be written. For example, suppose there is space for 20
bytes more in a file before reaching a limit. A write of 512 bytes will
return 20. The next write of a non-zero number of bytes would give a
failure return (except as noted below).

Signed-off-by: Wu Bo <Wu.Bo@cn.fujitsu.com>
---
 266     |  104 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 266.out |    5 +++
 group   |    1 +
 3 files changed, 110 insertions(+), 0 deletions(-)
 create mode 100755 266
 create mode 100644 266.out

diff --git a/266 b/266
new file mode 100755
index 0000000..6331ff8
--- /dev/null
+++ b/266
@@ -0,0 +1,104 @@
+#! /bin/bash
+# FS QA Test No. 266
+#
+# The posix write test. when write size is larger than disk free size,
+# should write as more as possible
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2011-2012 Fujitsu, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+#creator
+owner=wu.bo@cn.fujitsu.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=0    # success is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	rm -f $TEST_DIR/* $tmp.*
+	
+	if [ $_resize -eq 1 ]
+	then
+		btrfs filesystem resize ${_old}B $TEST_DIR > /dev/null 2>&1
+	fi
+	
+	_cleanup_testdir
+}
+
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs btrfs
+_supported_os IRIX Linux
+
+_setup_testdir
+_check_test_fs
+
+echo "------------------------------"
+echo "write lack test"
+echo "------------------------------"
+
+_resize=0
+_new=$(( 1 * 1024 * 1024 * 1024 )) # 1G
+_old=`df --block-size=1 | grep $TEST_DEV | awk '{print $2}'`
+if [ $_new -lt $_old ]
+then
+	btrfs filesystem resize ${_new}B $TEST_DIR > /dev/null 2>&1
+	if [ $? -ne 0 ]
+	then
+		echo "btrfs filesystem resize err"
+		status=1
+		exit
+	fi
+	_resize=1
+fi
+
+rm -rf $TEST_DIR/*
+cd $TEST_DIR
+dd if=/dev/zero of=tmp1 bs=4K count=1 > /dev/null 2>&1
+if [ $? -ne 0 ]
+then
+	echo "create file err"
+	status=1
+	exit
+fi
+
+dd if=/dev/zero of=tmp2 bs=1M > /dev/null 2>&1
+dd if=/dev/zero of=tmp3 bs=4K > /dev/null 2>&1
+sync
+
+rm -f tmp1
+sync
+
+dd if=/dev/zero of=tmp1 bs=8K count=1 > /dev/null 2>&1
+_filesize=`du tmp1 | awk '{print $1}'`
+if [ $_filesize -ne 4 ]
+then
+	echo "write file err"
+	status=1
+	exit
+fi
+
+echo "done"
+exit
diff --git a/266.out b/266.out
new file mode 100644
index 0000000..ce3340c
--- /dev/null
+++ b/266.out
@@ -0,0 +1,5 @@
+QA output created by 266
+------------------------------
+write lack test
+------------------------------
+done
diff --git a/group b/group
index f76ab8c..cf93ef9 100644
--- a/group
+++ b/group
@@ -379,3 +379,4 @@ deprecated
 263 rw auto quick
 264 auto rw
 265 auto rw
+266 auto rw
-- 
1.7.3.1

[PATCH] change fallocate to xfs_io falloc

[WuBo]

Thanks to Ben for pointing it that the fallocate maybe uninstall 
without checking it, and change fallocate to xfs_io as Christoph
suggested.

Signed-off-by: Wu Bo <Wu.Bo@cn.fujitsu.com>
---
 274 |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/274 b/274
index b658004..bcf5a11 100755
--- a/274
+++ b/274
@@ -46,6 +46,7 @@ _cleanup()
 _supported_fs generic
 _supported_os IRIX Linux
 _require_scratch
+_require_xfs_io_falloc
 
 echo "------------------------------"
 echo "preallocation test"
@@ -67,7 +68,7 @@ then
 	exit
 fi
 
-fallocate -n -o 4K -l 1M test >/dev/null 2>&1
+$XFS_IO_PROG -F -f -c 'falloc 4096 1m' test >/dev/null
 if [ $? -ne 0 ]
 then
 	echo "fallocate file err"
-- 
1.7.3.1

Re: [PATCH] change fallocate to xfs_io falloc

[Ben Myers]

Hey Wu Bo,

On Thu, Nov 17, 2011 at 01:16:13PM +0800, WuBo wrote:
> Thanks to Ben for pointing it that the fallocate maybe uninstall 
> without checking it, and change fallocate to xfs_io as Christoph
> suggested.
> 
> Signed-off-by: Wu Bo <Wu.Bo@cn.fujitsu.com>

Nice!  It works great for me.  ;)

Reviewed-by: Ben Myers <bpm@sgi.com>

Thanks,
	Ben

> ---
>  274 |    3 ++-
>  1 files changed, 2 insertions(+), 1 deletions(-)
> 
> diff --git a/274 b/274
> index b658004..bcf5a11 100755
> --- a/274
> +++ b/274
> @@ -46,6 +46,7 @@ _cleanup()
>  _supported_fs generic
>  _supported_os IRIX Linux
>  _require_scratch
> +_require_xfs_io_falloc
>  
>  echo "------------------------------"
>  echo "preallocation test"
> @@ -67,7 +68,7 @@ then
>  	exit
>  fi
>  
> -fallocate -n -o 4K -l 1M test >/dev/null 2>&1
> +$XFS_IO_PROG -F -f -c 'falloc 4096 1m' test >/dev/null
>  if [ $? -ne 0 ]
>  then
>  	echo "fallocate file err"
> -- 
> 1.7.3.1
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs