xfstests test #74 failure

xfstests test #74 failure

[Ted Ts'o]

Has anyone else noticed failures with xfstests #74?  What I'm finding
is that if compile the fstest program so that it is statically linked,
I can't get it to fail:

candygram:~/xfstests# ldd /vdb/fstest.static 
		      not a dynamic executable

However ,if it is dynamically linked withthe system C library, it
works just fine:

candygram:~/xfstests# ldd /vdb/fstest.dyn
		      linux-gate.so.1 =>  (0xffffe000)
		      libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb765d000)
		      /lib/ld-linux.so.2 (0xb77bc000)

The failure case looks like this.  If it doesn't fail the first time,
it almost certainly will fail the second time the command is repeated.
(And when I run test #74, it usually fails during the 2nd fstest
invocation, which adds a -m option to command line below, but the -m
doesn't seem necessary to cause the failure; what seems to be
important is running fstest the second time.)

candygram:~/xfstests# /vdb/fstest.dyn -n 3 -F -l 10 -f 5 -s 31457280 -b 512 -p /vdb/fstest.2875.2
num_children=3 file_size=31457280 num_files=5 loop_count=10 block_size=512
mmap=0 sync=0 prealloc=0
Total data size 471.9 Mbyte
Child 2 cleaning /vdb/fstest.2875.2/child2
Child 1 cleaning /vdb/fstest.2875.2/child1
Child 0 cleaning /vdb/fstest.2875.2/child0
Child 1 loop 0
Child 0 loop 0
Child 2 loop 0
Child 0 loop 1
Child 1 loop 1
Child 2 loop 1
Corruption in child 0 fnum 3 at offset 22327432
Correct:   5c 5c 5c 5c 5c 5c 5c 5c 5c 5c 5c 5c 5c 5c 5c 5c 5c [ 1344.124844] fstest.dyn used greatest stack depth: 5436 bytes left
5c 5c 5c 
Incorrect: 07 f2 ba 78 49 d0 a3 18 bd e5 c7 02 28 99 c5 4c 07 f2 ba 78 Corruption length: 0

Child exited with status 1
Child 2 loop 2
Child 1 loop 2
Child 1 loop 3
Child 2 loop 3
Child 1 loop 4

The exact same command-line works fine if I use statically linked
binary (fstest.dyn).  It also seems to work fine if I double the
amount of memory in my VM (from 1024 to 2048 megs).  At this point I'm
not sure whether it's a bug in fstest, or in ext4.  It is reproducible with 4k block size, 1k
block size, 4k/nodelalloc/noextents, nojournal mode. 

I'm using the libc6 in Debian unstable, version 2.13-18, both when
compiling and the shared library runtime.

If it's an ext4 bug, it's not a recent regression; I can replicate
this with v3.1-rc3, v3.0, 2.6.39, and 2.6.38.  However, I can *not*
replicate this with ext3, so this smells like an ext4 bug.  I have
absolutely no idea why linking dynamically vs. statically would make a
difference, though....

					- Ted

KVM-based xfstests appliance

[Ted Ts'o]

On Sun, Aug 28, 2011 at 07:54:37PM -0400, Ted Ts'o wrote:
> Has anyone else noticed failures with xfstests #74?  What I'm finding
> is that if compile the fstest program so that it is statically linked,
> I can't get it to fail:

I've put up an my KVM-based xfstests appliance here:

ftp://ftp.kernel.org/pub/linux/kernel/people/tytso/V-xfstests.tar.gz

Hopefully this will be useful for other people who are trying to do QA
work for ext4.  I find it's much more convenient to test out a kernel
by running "kvm-xfstests smoke", or "kvm-xfstests full".

BTW, I've found that the xfstests 74 test failure is very timing
dependent.  It doesn't fail when I use another system which has faster
disks than dm-crypt'ed laptop drives, and where writeback mode is
enabled with lots of memory available for caching purposes.  It seems
to require memory pressure and slower disks.  I should check and see
whether it involves ext4_writepage() getting invoked out of the direct
reclaim path...

	     	     	    	      	  	  - Ted

Re: KVM-based xfstests appliance

[Tao Ma]

Hi Ted,
On 08/29/2011 09:32 PM, Ted Ts'o wrote:
> On Sun, Aug 28, 2011 at 07:54:37PM -0400, Ted Ts'o wrote:
>> Has anyone else noticed failures with xfstests #74?  What I'm finding
>> is that if compile the fstest program so that it is statically linked,
>> I can't get it to fail:
> 
> I've put up an my KVM-based xfstests appliance here:
> 
> ftp://ftp.kernel.org/pub/linux/kernel/people/tytso/V-xfstests.tar.gz
It seems that this file is missing in the new ftp.kernel.org. Would you
mind upload it again or is there any other site that we can try to get it?

Thanks
Tao
> 
> Hopefully this will be useful for other people who are trying to do QA
> work for ext4.  I find it's much more convenient to test out a kernel
> by running "kvm-xfstests smoke", or "kvm-xfstests full".
> 
> BTW, I've found that the xfstests 74 test failure is very timing
> dependent.  It doesn't fail when I use another system which has faster
> disks than dm-crypt'ed laptop drives, and where writeback mode is
> enabled with lots of memory available for caching purposes.  It seems
> to require memory pressure and slower disks.  I should check and see
> whether it involves ext4_writepage() getting invoked out of the direct
> reclaim path...
> 
> 	     	     	    	      	  	  - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html