Re: [PATCH] xfs/larp: Make test failures debuggable

[Alli <allison.henderson@oracle.com>]

On Sat, 2022-05-07 at 04:02 +0800, Zorro Lang wrote:
> On Fri, May 06, 2022 at 06:08:08PM +0000, Catherine Hoang wrote:
> > > On May 6, 2022, at 9:40 AM, Zorro Lang <zlang@redhat.com> wrote:
> > > 
> > > On Fri, May 06, 2022 at 09:14:42AM -0700, Darrick J. Wong wrote:
> > > > On Fri, May 06, 2022 at 05:51:41PM +1000, Dave Chinner wrote:
> > > > > From: Dave Chinner <dchinner@redhat.com>
> > > > > 
> > > > > Md5sum output for attributes created combined program output
> > > > > and
> > > > > attribute values. This adds variable path names to the
> > > > > md5sum, so
> > > > > there's no way to tell if the md5sum is actually correct for
> > > > > the
> > > > > given attribute value that is returned as it's not constant
> > > > > from
> > > > > test to test. Hence we can't actually say that the output is
> > > > > correct
> > > > > because we can't reproduce exactly what we are hashing
> > > > > easily.
> > > > > 
> > > > > Indeed, the last attr test in series (node w/ replace) had an
> > > > > invalid md5sum. The attr value being produced after recovery
> > > > > was
> > > > > correct, but the md5sum output match was failing. Golden
> > > > > output
> > > > > appears to be wrong.
> > > > > 
> > > > > Fix this issue by seperately dumping all the attributes on
> > > > > the inode
> > > > > via a list operation to indicate their size, then dump the
> > > > > value of
> > > > > the test attribute directly to md5sum. This means the md5sum
> > > > > for
> > > > > the attributes using the same fixed values are all identical,
> > > > > so
> > > > > it's easy to tell if the md5sum for a given test is correct.
> > > > > We also
> > > > > check that all attributes that should be present after
> > > > > recovery are
> > > > > still there (e.g. checks recovery didn't trash innocent
> > > > > bystanders).
> > > > > 
> > > > > Further, the attribute replace tests replace an attribute
> > > > > with an
> > > > > identical value, hence there is no way to tell if recovery
> > > > > has
> > > > > resulted in the original being left behind or the new
> > > > > attribute
> > > > > being fully recovered because both have the same name and
> > > > > value.
> > > > > When replacing the attribute value, use a different sized
> > > > > value so
> > > > > it is trivial to determine that we've recovered the new
> > > > > attribute
> > > > > value correctly.
> > > > > 
> > > > > Also, the test runs on the scratch device - there is no need
> > > > > to
> > > > > remove the testdir in _cleanup. Doing so prevents post-mortem
> > > > > failure analysis because it burns the dead, broken corpse to
> > > > > ash and
> > > > > leaves no way of to determine cause of death.
> > > > > 
> > > > > Signed-off-by: Dave Chinner <dchinner@redhat.com>
> > > > > ---
> > > > > 
> > > > > Hi Catherine,
> > > > > 
> > > > > These are all the mods I needed to make to be able to
> > > > > understand the
> > > > > test failures I was getting as I debugged the new LARP
> > > > > recovery
> > > > > algorithm I've written.  You'll need to massage the test
> > > > > number in
> > > > > this patch to apply it on top of your patch.
> > > > > 
> > > > > I haven't added any new test cases yet, nor have I done
> > > > > anything to
> > > > > manage the larp sysfs knob, but we'll need to do those in the
> > > > > near
> > > > > future.
> > > > > 
> > > > > Zorro, can you consider merging this test in the near
> > > > > future?  We're
> > > > > right at the point of merging the upstream kernel code and so
> > > > > really
> > > > > need to start growing the test coverage of this feature, and
> > > > > this
> > > > > test should simply not-run on kernels that don't have the
> > > > > feature
> > > > > enabled....
> > > > > 
> > > > > Cheers,
> > > > > 
> > > > > Dave.
> > > > > ---
> > > > > 
> > > > > tests/xfs/600     |  20 +++++-----
> > > > > tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++-
> > > > > -----------------
> > > > > 2 files changed, 85 insertions(+), 44 deletions(-)
> > > > > 
> > > > > diff --git a/tests/xfs/600 b/tests/xfs/600
> > > > > index 252cdf27..84704646 100755
> > > > > --- a/tests/xfs/600
> > > > > +++ b/tests/xfs/600
> > > > > @@ -16,7 +16,7 @@ _begin_fstest auto quick attr
> > > > > 
> > > > > _cleanup()
> > > > > {
> > > > > -	rm -rf $tmp.* $testdir
> > > > > +	rm -rf $tmp.*
> > > > > 	test -w /sys/fs/xfs/debug/larp && \
> > > > > 		echo 0 > /sys/fs/xfs/debug/larp
> > > > 
> > > > Blergh, this ^^^^^^^^^ is going to need fixing too.
> > > > 
> > > > Please save the old value, then write it back in the _cleanup
> > > > function.
> > > 
> > > Ok, I'm going to do that when I merge it, if Catherine wouldn't
> > > like to do
> > > more changes in a V8 patch. If this case still need more changes,
> > > please tell
> > > me in time, and then it might have to wait the fstests release
> > > after next, if
> > > too late.
> > > 
> > > Thanks,
> > > Zorro
> > 
> > Based on Dave’s feedback, it looks like the patch will need a few
> > more
> > changes before it’s ready.
> 
> Great, that would be really helpful if you'd like to rebase this
> patch to fstests
> for-next branch. And how about combine Dave's patch with your patch?
> I don't think
> it's worth merging two seperated patches for one signle new case.
> What does Dave think?
> 
> I just merged below two patchset[1] into xfs-5.18-fixes-1, then tried
> to test this case
> (with you and Dave's patch). But it always failed as [2]. Please make
> sure it works
> as expected with those kernel patches still reviewing, I can't be
> sure I have the
> testing environment to guarantee that. Or I'd like to push this case
> after I see
> its target kernel patchset be merged by xfs-linux at first.
> 
> Thanks,
> Zorro
> 
> [1]
> [PATCH v29 00/15] xfs: Log Attribute Replay
> [PATCH 00/10 v2] xfs: LARP - clean up xfs_attr_set_iter state machine
> 
> 
> [2] out.bad output:
> *** mkfs
> *** mount FS
> attr_set: Input/output error
> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output
> error
> xfs_logprint: unknown log operation type (1246)
> Bad data in log
> mount: /mnt/scratch: mount(2) system call failed: Structure needs
> cleaning.
I think that happens when you dont have the corresponding xfsprogs
updates.  The userspace log print does not recognize the new log item.

Assuming you are running with an unpatched xfsprogs, I would have
thought the lack of error injects might have stopped the test from
running though?

Allison

> > Catherine
> > > > <slightly ot rant>
> > > > 
> > > > These sysfs knobs are a pain because they all reset to defaults
> > > > if
> > > > xfs.ko gets cycled.  I know, I know, at least Dave and Ted
> > > > don't do
> > > > modular kernels and so never see this happen, but I do.  I bet
> > > > Dave also
> > > > hasn't ever run xfs/434 or xfs/436, which might be why I'm the
> > > > only one
> > > > seeing dquot leaks with 5.19-next.
> > > > 
> > > > (I might be able to lift the xfs-as-module requirement if "echo
> > > > 1 >
> > > > /sys/kernel/slab/*/validate" does what I think it might, since
> > > > all we
> > > > want to do is look for slab leaks, and those tests
> > > > rmmod/modprobe as a
> > > > brute force way of making the slab debug code check for leaks.)
> > > > 
> > > > In case anyone's wondering, a solution to the knobs getting
> > > > unset after
> > > > an rmmod/modprobe cycle is to add a file
> > > > /etc/udev/rules.d/99-fstester.rules containing:
> > > > 
> > > > ACTION=="add|change", SUBSYSTEM=="module",
> > > > DEVPATH=="/module/${module}", RUN+="/bin/sh -c \"echo ${value}
> > > > > ${knob}\""
> > > > 
> > > > which should be enough to keep LARP turned on.
> > > > 
> > > > <end rant>
> > > > 
> > > > Anyway, since this is a proposed test, I say that with this
> > > > applied and
> > > > the debug knob bits fixed, the whole thing is
> > > > Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> > > > 
> > > > --D
> > > > 
> > > > > }
> > > > > @@ -46,7 +46,9 @@ test_attr_replay()
> > > > > 	touch $testfile
> > > > > 
> > > > > 	# Verify attr recovery
> > > > > -	{ $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 |
> > > > > _filter_scratch
> > > > > +	$ATTR_PROG -l $testfile | _filter_scratch
> > > > > +	echo -n "$attr_name: "
> > > > > +	$ATTR_PROG -q -g $attr_name $testfile | md5sum;
> > > > > 
> > > > > 	echo ""
> > > > > }
> > > > > @@ -157,19 +159,19 @@ create_test_file remote_file2 1
> > > > > $attr64k
> > > > > test_attr_replay remote_file2 "attr_name2" $attr64k "s"
> > > > > "larp"
> > > > > test_attr_replay remote_file2 "attr_name2" $attr64k "r"
> > > > > "larp"
> > > > > 
> > > > > -# replace shortform
> > > > > +# replace shortform with different value
> > > > > create_test_file sf_file 2 $attr64
> > > > > -test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
> > > > > +test_attr_replay sf_file "attr_name2" $attr16 "s" "larp"
> > > > > 
> > > > > -# replace leaf
> > > > > -create_test_file leaf_file 2 $attr1k
> > > > > -test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
> > > > > +# replace leaf with different value
> > > > > +create_test_file leaf_file 3 $attr1k
> > > > > +test_attr_replay leaf_file "attr_name2" $attr256 "s" "larp"
> > > > > 
> > > > > -# replace node
> > > > > +# replace node with a different value
> > > > > create_test_file node_file 1 $attr64k
> > > > > $ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
> > > > > 		>> $seqres.full
> > > > > -test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
> > > > > +test_attr_replay node_file "attr_name2" $attr256 "s" "larp"
> > > > > 
> > > > > echo "*** done"
> > > > > status=0
> > > > > diff --git a/tests/xfs/600.out b/tests/xfs/600.out
> > > > > index 96b1d7d9..fe25ea3e 100644
> > > > > --- a/tests/xfs/600.out
> > > > > +++ b/tests/xfs/600.out
> > > > > @@ -4,146 +4,185 @@ QA output created by 600
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1':
> > > > > Input/output error
> > > > > -21d850f99c43cc13abbe34838a8a3c8a  -
> > > > > +Attribute "attr_name" has a 65 byte value for
> > > > > SCRATCH_MNT/testdir/empty_file1
> > > > > +attr_name: cfbe2a33be4601d2b655d099a18378fc  -
> > > > > 
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file1
> > > > > +attr_name: attr_get: No data available
> > > > > +Could not get "attr_name" for
> > > > > /mnt/scratch/testdir/empty_file1
> > > > > d41d8cd98f00b204e9800998ecf8427e  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2':
> > > > > Input/output error
> > > > > -2ff89c2935debc431745ec791be5421a  -
> > > > > +Attribute "attr_name" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/empty_file2
> > > > > +attr_name: 9fd415c49d67afc4b78fad4055a3a376  -
> > > > > 
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file2
> > > > > +attr_name: attr_get: No data available
> > > > > +Could not get "attr_name" for
> > > > > /mnt/scratch/testdir/empty_file2
> > > > > d41d8cd98f00b204e9800998ecf8427e  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file3
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3':
> > > > > Input/output error
> > > > > -5d24b314242c52176c98ac4bd685da8b  -
> > > > > +Attribute "attr_name" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/empty_file3
> > > > > +attr_name: 7f6fd1b6d872108bd44bd143cbcdfa19  -
> > > > > 
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file3
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file3
> > > > > +attr_name: attr_get: No data available
> > > > > +Could not get "attr_name" for
> > > > > /mnt/scratch/testdir/empty_file3
> > > > > d41d8cd98f00b204e9800998ecf8427e  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1':
> > > > > Input/output error
> > > > > -5a7b559a70d8e92b4f3c6f7158eead08  -
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > +Attribute "attr_name2" has a 65 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > +attr_name2: cfbe2a33be4601d2b655d099a18378fc  -
> > > > > 
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/inline_file1
> > > > > d41d8cd98f00b204e9800998ecf8427e  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2':
> > > > > Input/output error
> > > > > -5717d5e66c70be6bdb00ecbaca0b7749  -
> > > > > +Attribute "attr_name2" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > +attr_name2: 9fd415c49d67afc4b78fad4055a3a376  -
> > > > > 
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/inline_file2
> > > > > d41d8cd98f00b204e9800998ecf8427e  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3':
> > > > > Input/output error
> > > > > -5c929964efd1b243aa8cceb6524f4810  -
> > > > > +Attribute "attr_name2" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19  -
> > > > > 
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/inline_file3
> > > > > d41d8cd98f00b204e9800998ecf8427e  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1':
> > > > > Input/output error
> > > > > -51ccb5cdfc9082060f0f94a8a108fea0  -
> > > > > +Attribute "attr_name2" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > +attr_name2: 9fd415c49d67afc4b78fad4055a3a376  -
> > > > > 
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/extent_file1
> > > > > d41d8cd98f00b204e9800998ecf8427e  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name4" for
> > > > > SCRATCH_MNT/testdir/extent_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file2':
> > > > > Input/output error
> > > > > -8d530bbe852d8bca83b131d5b3e497f5  -
> > > > > +Attribute "attr_name4" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file2
> > > > > +Attribute "attr_name2" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file2
> > > > > +Attribute "attr_name3" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file2
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file2
> > > > > +attr_name4: 9fd415c49d67afc4b78fad4055a3a376  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name4" for
> > > > > SCRATCH_MNT/testdir/extent_file3
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file3':
> > > > > Input/output error
> > > > > -5d77c4d3831a35bcbbd6e7677119ce9a  -
> > > > > +Attribute "attr_name4" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file3
> > > > > +Attribute "attr_name2" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file3
> > > > > +Attribute "attr_name3" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file3
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file3
> > > > > +attr_name4: 9fd415c49d67afc4b78fad4055a3a376  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4':
> > > > > Input/output error
> > > > > -6707ec2431e4dbea20e17da0816520bb  -
> > > > > +Attribute "attr_name2" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19  -
> > > > > 
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/extent_file4
> > > > > d41d8cd98f00b204e9800998ecf8427e  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1':
> > > > > Input/output error
> > > > > -767ebca3e4a6d24170857364f2bf2a3c  -
> > > > > +Attribute "attr_name2" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > +Attribute "attr_name1" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > +attr_name2: 9fd415c49d67afc4b78fad4055a3a376  -
> > > > > 
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > +Attribute "attr_name1" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/remote_file1
> > > > > d41d8cd98f00b204e9800998ecf8427e  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2':
> > > > > Input/output error
> > > > > -fd84ddec89237e6d34a1703639efaebf  -
> > > > > +Attribute "attr_name2" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > +Attribute "attr_name1" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19  -
> > > > > 
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > +Attribute "attr_name1" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/remote_file2
> > > > > d41d8cd98f00b204e9800998ecf8427e  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for SCRATCH_MNT/testdir/sf_file
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/sf_file':
> > > > > Input/output error
> > > > > -34aaa49662bafb46c76e377454685071  -
> > > > > +Attribute "attr_name1" has a 64 byte value for
> > > > > SCRATCH_MNT/testdir/sf_file
> > > > > +Attribute "attr_name2" has a 17 byte value for
> > > > > SCRATCH_MNT/testdir/sf_file
> > > > > +attr_name2: 9a6eb1bc9da3c66a9b495dfe2fe8a756  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for SCRATCH_MNT/testdir/leaf_file
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/leaf_file':
> > > > > Input/output error
> > > > > -664e95ec28830ffb367c0950026e0d21  -
> > > > > +Attribute "attr_name2" has a 257 byte value for
> > > > > SCRATCH_MNT/testdir/leaf_file
> > > > > +Attribute "attr_name3" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/leaf_file
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/leaf_file
> > > > > +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a  -
> > > > > 
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for SCRATCH_MNT/testdir/node_file
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/node_file':
> > > > > Input/output error
> > > > > -bb37a78ce26472eeb711e3559933db42  -
> > > > > +Attribute "attr_name2" has a 257 byte value for
> > > > > SCRATCH_MNT/testdir/node_file
> > > > > +Attribute "attr_name1" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/node_file
> > > > > +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a  -
> > > > > 
> > > > > *** done
> > > > > -- 
> > > > > Dave Chinner
> > > > > david@fromorbit.com