Review: fix test 004 to account for reserved space

Review: fix test 004 to account for reserved space

[David Chinner]

With the changes to use some space by default in only in memory
as a reserved pool, df and statfs will now output a fre block
count that is slightly different to  what is held in the superblock.

Update the qa test to account for this change.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

---
 xfstests/004 |   35 +++++++++++++++++++++++++----------
 1 file changed, 25 insertions(+), 10 deletions(-)

Index: xfs-cmds/xfstests/004
===================================================================
--- xfs-cmds.orig/xfstests/004	2006-11-14 19:57:39.000000000 +1100
+++ xfs-cmds/xfstests/004	2007-05-04 16:38:03.957537306 +1000
@@ -67,21 +67,36 @@ xfs_db -r -c "freesp -s" $SCRATCH_DEV >$
 echo "xfs_db for $SCRATCH_DEV" >>$seq.full
 cat $tmp.xfs_db >>$seq.full
 
+eval `$XFS_IO_PROG -x -c resblks $SCRATCH_MNT 2>&1 \
+	| $AWK_PROG '/available/ { printf "resblks=%u\n", $5 }'`
+echo "resblks gave: resblks=$resblks" >>$seq.full
+
 # check the 'blocks' field from freesp command is OK
 # since 2.6.18, df does not report the 4 blocks per AG that cannot
 # be allocated, hence we check for that exact mismatch.
+# since ~2.6.22, reserved blocks are used by default and df does
+# not report them, hence check for an exact mismatch.
 perl -ne '
-	    BEGIN	{ $avail ='$avail' * 512;
-			  $answer="(no xfs_db free blocks line?)" }
-	    /free blocks (\d+)$/	|| next;
-	    $freesp = $1 * '$dbsize';
-	    if ($freesp == $avail)	{ $answer = "yes"; }
-	    else {
+	BEGIN	{ $avail ='$avail' * 512;
+		  $answer="(no xfs_db free blocks line?)" }
+	/free blocks (\d+)$/	|| next;
+	$freesp = $1 * '$dbsize';
+	if ($freesp == $avail) {
+		$answer = "yes";
+	} else {
 		$avail = $avail + (('$agcount' + 1) * '$dbsize' * 4);
-		if ($freesp == $avail)	{ $answer = "yes"; }
-		else			{ $answer = "no ($freesp != $avail)"; }
-	    }
-	    END	{ print "$answer\n" }
+		if ($freesp == $avail) {
+			$answer = "yes";
+		} else {
+			$avail = $avail + ('$resblks' * '$dbsize');
+			if ($freesp == $avail) {
+				$answer = "yes";
+			} else {
+				$answer = "no ($freesp != $avail)";
+			}
+		}
+	}
+	END	{ print "$answer\n" }
 	' <$tmp.xfs_db >$tmp.ans
 ans="`cat $tmp.ans`"
 echo "Checking blocks column same as df: $ans"

Re: Review: fix test 004 to account for reserved space

[David Chinner]

Ping?

On Mon, Jun 04, 2007 at 04:33:28PM +1000, David Chinner wrote:
> With the changes to use some space by default in only in memory
> as a reserved pool, df and statfs will now output a fre block
> count that is slightly different to  what is held in the superblock.
> 
> Update the qa test to account for this change.
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> Principal Engineer
> SGI Australian Software Group
> 
> ---
>  xfstests/004 |   35 +++++++++++++++++++++++++----------
>  1 file changed, 25 insertions(+), 10 deletions(-)
> 
> Index: xfs-cmds/xfstests/004
> ===================================================================
> --- xfs-cmds.orig/xfstests/004	2006-11-14 19:57:39.000000000 +1100
> +++ xfs-cmds/xfstests/004	2007-05-04 16:38:03.957537306 +1000
> @@ -67,21 +67,36 @@ xfs_db -r -c "freesp -s" $SCRATCH_DEV >$
>  echo "xfs_db for $SCRATCH_DEV" >>$seq.full
>  cat $tmp.xfs_db >>$seq.full
>  
> +eval `$XFS_IO_PROG -x -c resblks $SCRATCH_MNT 2>&1 \
> +	| $AWK_PROG '/available/ { printf "resblks=%u\n", $5 }'`
> +echo "resblks gave: resblks=$resblks" >>$seq.full
> +
>  # check the 'blocks' field from freesp command is OK
>  # since 2.6.18, df does not report the 4 blocks per AG that cannot
>  # be allocated, hence we check for that exact mismatch.
> +# since ~2.6.22, reserved blocks are used by default and df does
> +# not report them, hence check for an exact mismatch.
>  perl -ne '
> -	    BEGIN	{ $avail ='$avail' * 512;
> -			  $answer="(no xfs_db free blocks line?)" }
> -	    /free blocks (\d+)$/	|| next;
> -	    $freesp = $1 * '$dbsize';
> -	    if ($freesp == $avail)	{ $answer = "yes"; }
> -	    else {
> +	BEGIN	{ $avail ='$avail' * 512;
> +		  $answer="(no xfs_db free blocks line?)" }
> +	/free blocks (\d+)$/	|| next;
> +	$freesp = $1 * '$dbsize';
> +	if ($freesp == $avail) {
> +		$answer = "yes";
> +	} else {
>  		$avail = $avail + (('$agcount' + 1) * '$dbsize' * 4);
> -		if ($freesp == $avail)	{ $answer = "yes"; }
> -		else			{ $answer = "no ($freesp != $avail)"; }
> -	    }
> -	    END	{ print "$answer\n" }
> +		if ($freesp == $avail) {
> +			$answer = "yes";
> +		} else {
> +			$avail = $avail + ('$resblks' * '$dbsize');
> +			if ($freesp == $avail) {
> +				$answer = "yes";
> +			} else {
> +				$answer = "no ($freesp != $avail)";
> +			}
> +		}
> +	}
> +	END	{ print "$answer\n" }
>  	' <$tmp.xfs_db >$tmp.ans
>  ans="`cat $tmp.ans`"
>  echo "Checking blocks column same as df: $ans"

-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: Review: fix test 004 to account for reserved space

[Timothy Shimmin]

I have to go home now, but I'll look at this one soon.

--Tim


David Chinner wrote:
> Ping?
> 
> On Mon, Jun 04, 2007 at 04:33:28PM +1000, David Chinner wrote:
>> With the changes to use some space by default in only in memory
>> as a reserved pool, df and statfs will now output a fre block
>> count that is slightly different to  what is held in the superblock.
>>
>> Update the qa test to account for this change.
>>
>> Cheers,
>>
>> Dave.
>> -- 
>> Dave Chinner
>> Principal Engineer
>> SGI Australian Software Group
>>
>> ---
>>  xfstests/004 |   35 +++++++++++++++++++++++++----------
>>  1 file changed, 25 insertions(+), 10 deletions(-)
>>
>> Index: xfs-cmds/xfstests/004
>> ===================================================================
>> --- xfs-cmds.orig/xfstests/004	2006-11-14 19:57:39.000000000 +1100
>> +++ xfs-cmds/xfstests/004	2007-05-04 16:38:03.957537306 +1000
>> @@ -67,21 +67,36 @@ xfs_db -r -c "freesp -s" $SCRATCH_DEV >$
>>  echo "xfs_db for $SCRATCH_DEV" >>$seq.full
>>  cat $tmp.xfs_db >>$seq.full
>>  
>> +eval `$XFS_IO_PROG -x -c resblks $SCRATCH_MNT 2>&1 \
>> +	| $AWK_PROG '/available/ { printf "resblks=%u\n", $5 }'`
>> +echo "resblks gave: resblks=$resblks" >>$seq.full
>> +
>>  # check the 'blocks' field from freesp command is OK
>>  # since 2.6.18, df does not report the 4 blocks per AG that cannot
>>  # be allocated, hence we check for that exact mismatch.
>> +# since ~2.6.22, reserved blocks are used by default and df does
>> +# not report them, hence check for an exact mismatch.
>>  perl -ne '
>> -	    BEGIN	{ $avail ='$avail' * 512;
>> -			  $answer="(no xfs_db free blocks line?)" }
>> -	    /free blocks (\d+)$/	|| next;
>> -	    $freesp = $1 * '$dbsize';
>> -	    if ($freesp == $avail)	{ $answer = "yes"; }
>> -	    else {
>> +	BEGIN	{ $avail ='$avail' * 512;
>> +		  $answer="(no xfs_db free blocks line?)" }
>> +	/free blocks (\d+)$/	|| next;
>> +	$freesp = $1 * '$dbsize';
>> +	if ($freesp == $avail) {
>> +		$answer = "yes";
>> +	} else {
>>  		$avail = $avail + (('$agcount' + 1) * '$dbsize' * 4);
>> -		if ($freesp == $avail)	{ $answer = "yes"; }
>> -		else			{ $answer = "no ($freesp != $avail)"; }
>> -	    }
>> -	    END	{ print "$answer\n" }
>> +		if ($freesp == $avail) {
>> +			$answer = "yes";
>> +		} else {
>> +			$avail = $avail + ('$resblks' * '$dbsize');
>> +			if ($freesp == $avail) {
>> +				$answer = "yes";
>> +			} else {
>> +				$answer = "no ($freesp != $avail)";
>> +			}
>> +		}
>> +	}
>> +	END	{ print "$answer\n" }
>>  	' <$tmp.xfs_db >$tmp.ans
>>  ans="`cat $tmp.ans`"
>>  echo "Checking blocks column same as df: $ans"
> 

Re: Review: fix test 004 to account for reserved space

[Tim Shimmin]

Hi Dave,

Yes that looks not bad.

However, your last proposal, prompted from Nathan, was to adjust 
f_bavail differently
than bfree to account for how much was available in the reserved block 
space.

struct statfs {
  long    f_type;     /* type of filesystem (see below) */
  long    f_bsize;    /* optimal transfer block size */
  long    f_blocks;   /* total data blocks in file system */
  long    f_bfree;    /* free blocks in fs */                    <------ 
includes reserved
  long    f_bavail;   /* free blocks avail to non-superuser */   <------ 
doesn't include reserved
  long    f_files;    /* total file nodes in file system */
  long    f_ffree;    /* free file nodes in fs */
  fsid_t  f_fsid;     /* file system id */
  long    f_namelen;  /* maximum length of filenames */
  long    f_spare[6]; /* spare for later */
};

And looking at df code it uses bfree in calculation of space used...
the 3rd field of df output.

df code:
---------------------------------------------------------------
     total = fsu.fsu_blocks;
     available = fsu.fsu_bavail;
     negate_available = (fsu.fsu_bavail_top_bit_set
              & (available != UINTMAX_MAX));
     available_to_root = fsu.fsu_bfree;
   }

 used = UINTMAX_MAX;
 negate_used = false;
 if (total != UINTMAX_MAX && available_to_root != UINTMAX_MAX)
   {
     used = total - available_to_root;
     negate_used = (total < available_to_root);
   }

printf (" %*s %*s %*s ",
      width + col1_adjustment,
      df_readable (false, total,
               buf[0], input_units, output_units),
      width, df_readable (negate_used, used,
                  buf[1], input_units, output_units),
      width, df_readable (negate_available, available,
                  buf[2], input_units, output_units));
-----------------------------------------------------------------
example df output:
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3             18864128   7476720  11387408  40% /
-------------------------------------------------------------------

And indeed we set "used" as a shell variable in test 004 from df but 
don't seem
to do anything with it.
I think we should check it to verify that we have set bfree correctly.

Cheers,
Tim.

David Chinner wrote:
> Ping?
>
> On Mon, Jun 04, 2007 at 04:33:28PM +1000, David Chinner wrote:
>   
>> With the changes to use some space by default in only in memory
>> as a reserved pool, df and statfs will now output a fre block
>> count that is slightly different to  what is held in the superblock.
>>
>> Update the qa test to account for this change.
>>
>> Cheers,
>>
>> Dave.
>> -- 
>> Dave Chinner
>> Principal Engineer
>> SGI Australian Software Group
>>
>> ---
>>  xfstests/004 |   35 +++++++++++++++++++++++++----------
>>  1 file changed, 25 insertions(+), 10 deletions(-)
>>
>> Index: xfs-cmds/xfstests/004
>> ===================================================================
>> --- xfs-cmds.orig/xfstests/004	2006-11-14 19:57:39.000000000 +1100
>> +++ xfs-cmds/xfstests/004	2007-05-04 16:38:03.957537306 +1000
>> @@ -67,21 +67,36 @@ xfs_db -r -c "freesp -s" $SCRATCH_DEV >$
>>  echo "xfs_db for $SCRATCH_DEV" >>$seq.full
>>  cat $tmp.xfs_db >>$seq.full
>>  
>> +eval `$XFS_IO_PROG -x -c resblks $SCRATCH_MNT 2>&1 \
>> +	| $AWK_PROG '/available/ { printf "resblks=%u\n", $5 }'`
>> +echo "resblks gave: resblks=$resblks" >>$seq.full
>> +
>>  # check the 'blocks' field from freesp command is OK
>>  # since 2.6.18, df does not report the 4 blocks per AG that cannot
>>  # be allocated, hence we check for that exact mismatch.
>> +# since ~2.6.22, reserved blocks are used by default and df does
>> +# not report them, hence check for an exact mismatch.
>>  perl -ne '
>> -	    BEGIN	{ $avail ='$avail' * 512;
>> -			  $answer="(no xfs_db free blocks line?)" }
>> -	    /free blocks (\d+)$/	|| next;
>> -	    $freesp = $1 * '$dbsize';
>> -	    if ($freesp == $avail)	{ $answer = "yes"; }
>> -	    else {
>> +	BEGIN	{ $avail ='$avail' * 512;
>> +		  $answer="(no xfs_db free blocks line?)" }
>> +	/free blocks (\d+)$/	|| next;
>> +	$freesp = $1 * '$dbsize';
>> +	if ($freesp == $avail) {
>> +		$answer = "yes";
>> +	} else {
>>  		$avail = $avail + (('$agcount' + 1) * '$dbsize' * 4);
>> -		if ($freesp == $avail)	{ $answer = "yes"; }
>> -		else			{ $answer = "no ($freesp != $avail)"; }
>> -	    }
>> -	    END	{ print "$answer\n" }
>> +		if ($freesp == $avail) {
>> +			$answer = "yes";
>> +		} else {
>> +			$avail = $avail + ('$resblks' * '$dbsize');
>> +			if ($freesp == $avail) {
>> +				$answer = "yes";
>> +			} else {
>> +				$answer = "no ($freesp != $avail)";
>> +			}
>> +		}
>> +	}
>> +	END	{ print "$answer\n" }
>>  	' <$tmp.xfs_db >$tmp.ans
>>  ans="`cat $tmp.ans`"
>>  echo "Checking blocks column same as df: $ans"
>>     
>
>   

Re: Review: fix test 004 to account for reserved space

[Christoph Hellwig]

On Mon, Jun 04, 2007 at 04:33:28PM +1000, David Chinner wrote:
> With the changes to use some space by default in only in memory
> as a reserved pool, df and statfs will now output a fre block
> count that is slightly different to  what is held in the superblock.
> 
> Update the qa test to account for this change.

I think we should rather subtract the amount of internally reserved blocks
from the return value in xfs_statvfs.

Re: Review: fix test 004 to account for reserved space

[David Chinner]

On Sat, Jun 16, 2007 at 08:55:08PM +0100, Christoph Hellwig wrote:
> On Mon, Jun 04, 2007 at 04:33:28PM +1000, David Chinner wrote:
> > With the changes to use some space by default in only in memory
> > as a reserved pool, df and statfs will now output a fre block
> > count that is slightly different to  what is held in the superblock.
> > 
> > Update the qa test to account for this change.
> 
> I think we should rather subtract the amount of internally reserved blocks
> from the return value in xfs_statvfs.

Which return value?

With this patch:


---
 fs/xfs/xfs_vfsops.c |    1 +
 1 file changed, 1 insertion(+)

Index: 2.6.x-xfs-new/fs/xfs/xfs_vfsops.c
===================================================================
--- 2.6.x-xfs-new.orig/fs/xfs/xfs_vfsops.c	2007-06-08 21:46:29.000000000 +1000
+++ 2.6.x-xfs-new/fs/xfs/xfs_vfsops.c	2007-06-12 13:08:49.933837815 +1000
@@ -876,6 +876,7 @@ xfs_statvfs(
 	statp->f_blocks = sbp->sb_dblocks - lsize;
 	statp->f_bfree = statp->f_bavail =
 				sbp->sb_fdblocks - XFS_ALLOC_SET_ASIDE(mp);
+	statp->f_bfree += mp->m_resblks_avail;
 	fakeinos = statp->f_bfree << sbp->sb_inopblog;
 #if XFS_BIG_INUMS
 	fakeinos += mp->m_inoadd;


An strace of df --block-size=4k gives:

statfs("/mnt/test", {f_type=0x58465342, f_bsize=4096, f_blocks=1048616, f_bfree=874158, f_bavail=873134, f_files=4204672, f_ffree=4191008, f_fsid={2072, 0}, f_namelen=255, f_frsize=4096}) = 0
write(1, "/dev/sdb8              1048616  "..., 66/dev/sdb8              1048616    174458    873134  17% /mnt/test
) = 66
statfs("/mnt/scratch", {f_type=0x58465342, f_bsize=4096, f_blocks=1248496, f_bfree=1248392, f_bavail=1247368, f_files=5004224, f_ffree=5004220, f_fsid={2073, 0}, f_namelen=255, f_frsize=4096}) = 0
write(1, "/dev/sdb9              1248496  "..., 69/dev/sdb9              1248496       104   1247368   1% /mnt/scratch

Is this what you were thinking of?

Note that this still requires the fix to the qa test because
the value in the on disk superblock matches f_bfree, not f_bavail
and df appears to output f_bavail....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group