* [PATCH] xfstests: Improve test 219 to work with all filesystems @ 2011-05-17 14:55 Jan Kara 2011-05-17 23:13 ` Dave Chinner 0 siblings, 1 reply; 7+ messages in thread From: Jan Kara @ 2011-05-17 14:55 UTC (permalink / raw) To: xfs; +Cc: Jan Kara Different filesystems account different amount of metadata in quota. Thus it is impractical to check for a particular amount of space occupied by a file because there is no right value. Change the test to verify whether the amount of space before quotacheck and after quotacheck is the same as other quota tests do. Signed-off-by: Jan Kara <jack@suse.cz> --- 219 | 4 +--- 219.out | 18 ++---------------- common.quota | 52 ++++++++++++++++++++++++++++++++++++++-------------- 3 files changed, 41 insertions(+), 33 deletions(-) diff --git a/219 b/219 index 836d703..7c22dac 100755 --- a/219 +++ b/219 @@ -76,9 +76,7 @@ test_accounting() for file in $SCRATCH_MNT/{buffer,direct,mmap}; do $here/src/lstat64 $file | head -3 | _filter_scratch done - - repquota -$type -n $SCRATCH_MNT | grep -v "^#0" | _filter_scratch | - awk '/^#/ { if (seen[$1]) next; seen[$1]++; } { print; }' + _check_quota_usage } # real QA test starts here diff --git a/219.out b/219.out index fadfafc..7a86b94 100644 --- a/219.out +++ b/219.out @@ -22,14 +22,7 @@ QA output created by 219 File: "SCRATCH_MNT/mmap" Size: 49152 Filetype: Regular File Mode: (0644/-rw-r--r--) Uid: (1) Gid: (2) -*** Report for user quotas on device SCRATCH_DEV -Block grace time: 7days; Inode grace time: 7days - Block limits File limits -User used soft hard grace used soft hard grace ----------------------------------------------------------------------- -#1 -- 144 0 0 3 0 0 - - +Comparing user usage ### test group accounting @@ -53,11 +46,4 @@ User used soft hard grace used soft hard grace File: "SCRATCH_MNT/mmap" Size: 49152 Filetype: Regular File Mode: (0644/-rw-r--r--) Uid: (1) Gid: (2) -*** Report for group quotas on device SCRATCH_DEV -Block grace time: 7days; Inode grace time: 7days - Block limits File limits -Group used soft hard grace used soft hard grace ----------------------------------------------------------------------- -#2 -- 144 0 0 3 0 0 - - +Comparing group usage diff --git a/common.quota b/common.quota index 3c87ce1..9cb9304 100644 --- a/common.quota +++ b/common.quota @@ -236,36 +236,60 @@ _check_quota_usage() { # Sync to get delalloc to disk sync + USRQUOTA=0 + GRPQUOTA=0 + QMNTOPT="" + if echo $MOUNT_OPTIONS | grep -E "uquota|usrquota|uqnoenforce" &>/dev/null; then + USRQUOTA=1 + QMNTOPT=",usrquota" + fi + if echo $MOUNT_OPTIONS | grep -E "gquota|grpquota|gqnoenforce" &>/dev/null; then + GRPQUOTA=1 + QMNTOPT=$QMNTOPT",grpquota" + fi VFS_QUOTA=0 if [ $FSTYP = "ext2" -o $FSTYP = "ext3" -o $FSTYP = "ext4" -o $FSTYP = "reiserfs" ]; then VFS_QUOTA=1 quotaon -f -u -g $SCRATCH_MNT 2>/dev/null fi - repquota -u -n $SCRATCH_MNT | grep -v "^#0" | _filter_scratch | - sort >$tmp.user.orig - repquota -g -n $SCRATCH_MNT | grep -v "^#0" | _filter_scratch | - sort >$tmp.group.orig + if [ $USRQUOTA == 1 ]; then + repquota -u -n $SCRATCH_MNT | grep -v "^#0" | + _filter_scratch | sort >$tmp.user.orig + fi + if [ $GRPQUOTA == 1 ]; then + repquota -g -n $SCRATCH_MNT | grep -v "^#0" | + _filter_scratch | sort >$tmp.group.orig + fi if [ $VFS_QUOTA -eq 1 ]; then quotacheck -u -g $SCRATCH_MNT 2>/dev/null else # use XFS method to force quotacheck mount -o remount,noquota $SCRATCH_DEV - mount -o remount,usrquota,grpquota $SCRATCH_DEV + mount -o remount$QMNTOPT $SCRATCH_DEV + fi + if [ $USRQUOTA == 1 ]; then + repquota -u -n $SCRATCH_MNT | grep -v "^#0" | + _filter_scratch | sort >$tmp.user.checked + fi + if [ $GRPQUOTA == 1 ]; then + repquota -g -n $SCRATCH_MNT | grep -v "^#0" | + _filter_scratch | sort >$tmp.group.checked fi - repquota -u -n $SCRATCH_MNT | grep -v "^#0" | _filter_scratch | - sort >$tmp.user.checked - repquota -g -n $SCRATCH_MNT | grep -v "^#0" | _filter_scratch | - sort >$tmp.group.checked if [ $VFS_QUOTA -eq 1 ]; then quotaon -u -g $SCRATCH_MNT 2>/dev/null fi - { + if [ $USRQUOTA == 1 ]; then echo "Comparing user usage" - diff $tmp.user.orig $tmp.user.checked - } && { + if ! diff $tmp.user.orig $tmp.user.checked; then + return 1 + fi + fi + if [ $GRPQUOTA == 1 ]; then echo "Comparing group usage" - diff $tmp.group.orig $tmp.group.checked - } + if ! diff $tmp.group.orig $tmp.group.checked; then + return 1 + fi + fi } # make sure this script returns success -- 1.6.0.2 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] xfstests: Improve test 219 to work with all filesystems 2011-05-17 14:55 [PATCH] xfstests: Improve test 219 to work with all filesystems Jan Kara @ 2011-05-17 23:13 ` Dave Chinner 2011-05-18 8:24 ` Jan Kara 0 siblings, 1 reply; 7+ messages in thread From: Dave Chinner @ 2011-05-17 23:13 UTC (permalink / raw) To: Jan Kara; +Cc: xfs On Tue, May 17, 2011 at 04:55:04PM +0200, Jan Kara wrote: > Different filesystems account different amount of metadata in quota. Thus it is > impractical to check for a particular amount of space occupied by a file > because there is no right value. Change the test to verify whether the amount > of space before quotacheck and after quotacheck is the same as other quota > tests do. Except that the purpose of the test the accounting correctly matches the blocks allocated via direct IO, buffered IO and mmap, not that quota is consistent over a remount. IOWs, The numbers do actually matter - for example the recent changes to speculative delayed allocation beyond EOF for buffered IO in XFS could be causing large numbers of blocks to be left after EOF incorrectly, but the exact block number check used in the test would catch that. The method you propose would not catch it at all, and we'd be oblivous to an undesirable change in behaviour. IMO, a better filter function would be the way to go - one that takes into account that there might be some metadata blocks allocated but not less than 3x48k should have be allocated to the quotas... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] xfstests: Improve test 219 to work with all filesystems 2011-05-17 23:13 ` Dave Chinner @ 2011-05-18 8:24 ` Jan Kara 2011-05-18 23:43 ` Dave Chinner 0 siblings, 1 reply; 7+ messages in thread From: Jan Kara @ 2011-05-18 8:24 UTC (permalink / raw) To: Dave Chinner; +Cc: Jan Kara, xfs On Wed 18-05-11 09:13:01, Dave Chinner wrote: > On Tue, May 17, 2011 at 04:55:04PM +0200, Jan Kara wrote: > > Different filesystems account different amount of metadata in quota. Thus it is > > impractical to check for a particular amount of space occupied by a file > > because there is no right value. Change the test to verify whether the amount > > of space before quotacheck and after quotacheck is the same as other quota > > tests do. > > Except that the purpose of the test the accounting correctly matches > the blocks allocated via direct IO, buffered IO and mmap, not that > quota is consistent over a remount. > > IOWs, The numbers do actually matter - for example the recent > changes to speculative delayed allocation beyond EOF for buffered IO > in XFS could be causing large numbers of blocks to be left after EOF > incorrectly, but the exact block number check used in the test would > catch that. The method you propose would not catch it at all, and > we'd be oblivous to an undesirable change in behaviour. Hmm, I guess we think of different errors to catch with the test. I was more thinking that the test tries to catch errors where we forget to account allocated blocks in quota or so. But you are right, there are other tests to catch this although not testing e.g. direct IO I think. > IMO, a better filter function would be the way to go - one > that takes into account that there might be some metadata blocks > allocated but not less than 3x48k should have be allocated to the > quotas... OK, but if I just check that the amount of space is >= 3x48k, your sample problem with xfs would pass anyway. What would be nice is to know the right value but that depends on fs type and also fs parameters (fs block size in ext3/ext4) so it would be a bit large chunk of code to compute the right value - that's why I chose quotacheck to do the work for me... But I guess I can do that if you think it's worth it. Honza -- Jan Kara <jack@suse.cz> SUSE Labs, CR _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] xfstests: Improve test 219 to work with all filesystems 2011-05-18 8:24 ` Jan Kara @ 2011-05-18 23:43 ` Dave Chinner 2011-05-19 10:49 ` Jan Kara 0 siblings, 1 reply; 7+ messages in thread From: Dave Chinner @ 2011-05-18 23:43 UTC (permalink / raw) To: Jan Kara; +Cc: xfs On Wed, May 18, 2011 at 10:24:22AM +0200, Jan Kara wrote: > On Wed 18-05-11 09:13:01, Dave Chinner wrote: > > On Tue, May 17, 2011 at 04:55:04PM +0200, Jan Kara wrote: > > > Different filesystems account different amount of metadata in quota. Thus it is > > > impractical to check for a particular amount of space occupied by a file > > > because there is no right value. Change the test to verify whether the amount > > > of space before quotacheck and after quotacheck is the same as other quota > > > tests do. > > > > Except that the purpose of the test the accounting correctly matches > > the blocks allocated via direct IO, buffered IO and mmap, not that > > quota is consistent over a remount. > > > > IOWs, The numbers do actually matter - for example the recent > > changes to speculative delayed allocation beyond EOF for buffered IO > > in XFS could be causing large numbers of blocks to be left after EOF > > incorrectly, but the exact block number check used in the test would > > catch that. The method you propose would not catch it at all, and > > we'd be oblivous to an undesirable change in behaviour. > Hmm, I guess we think of different errors to catch with the test. I was > more thinking that the test tries to catch errors where we forget to > account allocated blocks in quota or so. But you are right, there are other > tests to catch this although not testing e.g. direct IO I think. > > > IMO, a better filter function would be the way to go - one > > that takes into account that there might be some metadata blocks > > allocated but not less than 3x48k should have be allocated to the > > quotas... > OK, but if I just check that the amount of space is >= 3x48k, your > sample problem with xfs would pass anyway. Not if it was done like we do with the randholes tests (008), where we use the _within_tolerance function to determine if the number of holes is accceptible. > > What would be nice is to know > the right value but that depends on fs type and also fs parameters (fs > block size in ext3/ext4) so it would be a bit large chunk of code to > compute the right value - that's why I chose quotacheck to do the work for > me... But I guess I can do that if you think it's worth it. Yes, block size will alter the number of blocks, but remember that requota is reporting it in kilobytes, so the number should always be around 144. Hence checking the result is 144 +/- 5% would probably cover all filesystems and all configurations without an issue, and it would still catch gross increasing in disk usage... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] xfstests: Improve test 219 to work with all filesystems 2011-05-18 23:43 ` Dave Chinner @ 2011-05-19 10:49 ` Jan Kara 2011-05-19 11:15 ` Dave Chinner 0 siblings, 1 reply; 7+ messages in thread From: Jan Kara @ 2011-05-19 10:49 UTC (permalink / raw) To: Dave Chinner; +Cc: Jan Kara, xfs On Thu 19-05-11 09:43:18, Dave Chinner wrote: > On Wed, May 18, 2011 at 10:24:22AM +0200, Jan Kara wrote: > > On Wed 18-05-11 09:13:01, Dave Chinner wrote: > > > On Tue, May 17, 2011 at 04:55:04PM +0200, Jan Kara wrote: > > > > Different filesystems account different amount of metadata in quota. Thus it is > > > > impractical to check for a particular amount of space occupied by a file > > > > because there is no right value. Change the test to verify whether the amount > > > > of space before quotacheck and after quotacheck is the same as other quota > > > > tests do. > > > > > > Except that the purpose of the test the accounting correctly matches > > > the blocks allocated via direct IO, buffered IO and mmap, not that > > > quota is consistent over a remount. > > > > > > IOWs, The numbers do actually matter - for example the recent > > > changes to speculative delayed allocation beyond EOF for buffered IO > > > in XFS could be causing large numbers of blocks to be left after EOF > > > incorrectly, but the exact block number check used in the test would > > > catch that. The method you propose would not catch it at all, and > > > we'd be oblivous to an undesirable change in behaviour. > > Hmm, I guess we think of different errors to catch with the test. I was > > more thinking that the test tries to catch errors where we forget to > > account allocated blocks in quota or so. But you are right, there are other > > tests to catch this although not testing e.g. direct IO I think. > > > > > IMO, a better filter function would be the way to go - one > > > that takes into account that there might be some metadata blocks > > > allocated but not less than 3x48k should have be allocated to the > > > quotas... > > OK, but if I just check that the amount of space is >= 3x48k, your > > sample problem with xfs would pass anyway. > > Not if it was done like we do with the randholes tests (008), where we use > the _within_tolerance function to determine if the number of holes is > accceptible. > > > > > What would be nice is to know > > the right value but that depends on fs type and also fs parameters (fs > > block size in ext3/ext4) so it would be a bit large chunk of code to > > compute the right value - that's why I chose quotacheck to do the work for > > me... But I guess I can do that if you think it's worth it. > > Yes, block size will alter the number of blocks, but remember that > requota is reporting it in kilobytes, so the number should always be > around 144. Hence checking the result is 144 +/- 5% would probably > cover all filesystems and all configurations without an issue, and > it would still catch gross increasing in disk usage... Ah, OK, that makes sense and works for the common cases I'm aware of (it won't work for example for ext4 with 64k block size, or ocfs2 with large cluster size where allocation unit can be upto 1M). Anyway, it has probably the best effort/result ratio so I'll do this. Thanks for ideas. Honza _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] xfstests: Improve test 219 to work with all filesystems 2011-05-19 10:49 ` Jan Kara @ 2011-05-19 11:15 ` Dave Chinner 2011-05-19 11:50 ` Jan Kara 0 siblings, 1 reply; 7+ messages in thread From: Dave Chinner @ 2011-05-19 11:15 UTC (permalink / raw) To: Jan Kara; +Cc: xfs On Thu, May 19, 2011 at 12:49:08PM +0200, Jan Kara wrote: > On Thu 19-05-11 09:43:18, Dave Chinner wrote: > > On Wed, May 18, 2011 at 10:24:22AM +0200, Jan Kara wrote: > > > On Wed 18-05-11 09:13:01, Dave Chinner wrote: > > > > On Tue, May 17, 2011 at 04:55:04PM +0200, Jan Kara wrote: > > > > > Different filesystems account different amount of metadata in quota. Thus it is > > > > > impractical to check for a particular amount of space occupied by a file > > > > > because there is no right value. Change the test to verify whether the amount > > > > > of space before quotacheck and after quotacheck is the same as other quota > > > > > tests do. > > > > > > > > Except that the purpose of the test the accounting correctly matches > > > > the blocks allocated via direct IO, buffered IO and mmap, not that > > > > quota is consistent over a remount. > > > > > > > > IOWs, The numbers do actually matter - for example the recent > > > > changes to speculative delayed allocation beyond EOF for buffered IO > > > > in XFS could be causing large numbers of blocks to be left after EOF > > > > incorrectly, but the exact block number check used in the test would > > > > catch that. The method you propose would not catch it at all, and > > > > we'd be oblivous to an undesirable change in behaviour. > > > Hmm, I guess we think of different errors to catch with the test. I was > > > more thinking that the test tries to catch errors where we forget to > > > account allocated blocks in quota or so. But you are right, there are other > > > tests to catch this although not testing e.g. direct IO I think. > > > > > > > IMO, a better filter function would be the way to go - one > > > > that takes into account that there might be some metadata blocks > > > > allocated but not less than 3x48k should have be allocated to the > > > > quotas... > > > OK, but if I just check that the amount of space is >= 3x48k, your > > > sample problem with xfs would pass anyway. > > > > Not if it was done like we do with the randholes tests (008), where we use > > the _within_tolerance function to determine if the number of holes is > > accceptible. > > > > > > > > What would be nice is to know > > > the right value but that depends on fs type and also fs parameters (fs > > > block size in ext3/ext4) so it would be a bit large chunk of code to > > > compute the right value - that's why I chose quotacheck to do the work for > > > me... But I guess I can do that if you think it's worth it. > > > > Yes, block size will alter the number of blocks, but remember that > > requota is reporting it in kilobytes, so the number should always be > > around 144. Hence checking the result is 144 +/- 5% would probably > > cover all filesystems and all configurations without an issue, and > > it would still catch gross increasing in disk usage... > Ah, OK, that makes sense and works for the common cases I'm aware of (it > won't work for example for ext4 with 64k block size, or ocfs2 with large > cluster size where allocation unit can be upto 1M). Anyway, it has probably > the best effort/result ratio so I'll do this. Thanks for ideas. The write size of 48k is completely arbitrary, so if we need to change it to work better with larger block sizes, then that is easy to do... Anyway, there are lots of other tests that will break with 64k block sizes, so this is the least of our worries at this point... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] xfstests: Improve test 219 to work with all filesystems 2011-05-19 11:15 ` Dave Chinner @ 2011-05-19 11:50 ` Jan Kara 0 siblings, 0 replies; 7+ messages in thread From: Jan Kara @ 2011-05-19 11:50 UTC (permalink / raw) To: Dave Chinner; +Cc: Jan Kara, xfs On Thu 19-05-11 21:15:21, Dave Chinner wrote: > On Thu, May 19, 2011 at 12:49:08PM +0200, Jan Kara wrote: > > On Thu 19-05-11 09:43:18, Dave Chinner wrote: > > > On Wed, May 18, 2011 at 10:24:22AM +0200, Jan Kara wrote: > > > > On Wed 18-05-11 09:13:01, Dave Chinner wrote: > > > > > On Tue, May 17, 2011 at 04:55:04PM +0200, Jan Kara wrote: > > > > > > Different filesystems account different amount of metadata in quota. Thus it is > > > > > > impractical to check for a particular amount of space occupied by a file > > > > > > because there is no right value. Change the test to verify whether the amount > > > > > > of space before quotacheck and after quotacheck is the same as other quota > > > > > > tests do. > > > > > > > > > > Except that the purpose of the test the accounting correctly matches > > > > > the blocks allocated via direct IO, buffered IO and mmap, not that > > > > > quota is consistent over a remount. > > > > > > > > > > IOWs, The numbers do actually matter - for example the recent > > > > > changes to speculative delayed allocation beyond EOF for buffered IO > > > > > in XFS could be causing large numbers of blocks to be left after EOF > > > > > incorrectly, but the exact block number check used in the test would > > > > > catch that. The method you propose would not catch it at all, and > > > > > we'd be oblivous to an undesirable change in behaviour. > > > > Hmm, I guess we think of different errors to catch with the test. I was > > > > more thinking that the test tries to catch errors where we forget to > > > > account allocated blocks in quota or so. But you are right, there are other > > > > tests to catch this although not testing e.g. direct IO I think. > > > > > > > > > IMO, a better filter function would be the way to go - one > > > > > that takes into account that there might be some metadata blocks > > > > > allocated but not less than 3x48k should have be allocated to the > > > > > quotas... > > > > OK, but if I just check that the amount of space is >= 3x48k, your > > > > sample problem with xfs would pass anyway. > > > > > > Not if it was done like we do with the randholes tests (008), where we use > > > the _within_tolerance function to determine if the number of holes is > > > accceptible. > > > > > > > > > > > What would be nice is to know > > > > the right value but that depends on fs type and also fs parameters (fs > > > > block size in ext3/ext4) so it would be a bit large chunk of code to > > > > compute the right value - that's why I chose quotacheck to do the work for > > > > me... But I guess I can do that if you think it's worth it. > > > > > > Yes, block size will alter the number of blocks, but remember that > > > requota is reporting it in kilobytes, so the number should always be > > > around 144. Hence checking the result is 144 +/- 5% would probably > > > cover all filesystems and all configurations without an issue, and > > > it would still catch gross increasing in disk usage... > > Ah, OK, that makes sense and works for the common cases I'm aware of (it > > won't work for example for ext4 with 64k block size, or ocfs2 with large > > cluster size where allocation unit can be upto 1M). Anyway, it has probably > > the best effort/result ratio so I'll do this. Thanks for ideas. > > The write size of 48k is completely arbitrary, so if we need to > change it to work better with larger block sizes, then that is > easy to do... > > Anyway, there are lots of other tests that will break with 64k block > sizes, so this is the least of our worries at this point... Yes, I think that as well. So when we come across a case we want to test which will fail, we can change the amount written. So far I just changed the test so that it works for my ext3 & ext4 tests... Honza _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2011-05-19 11:50 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-05-17 14:55 [PATCH] xfstests: Improve test 219 to work with all filesystems Jan Kara 2011-05-17 23:13 ` Dave Chinner 2011-05-18 8:24 ` Jan Kara 2011-05-18 23:43 ` Dave Chinner 2011-05-19 10:49 ` Jan Kara 2011-05-19 11:15 ` Dave Chinner 2011-05-19 11:50 ` Jan Kara
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox