Re: [Qemu-devel] [RFC] TCG unit testing

[Aurelien Jarno <aurelien@aurel32.net>]

On Fri, Aug 23, 2013 at 12:47:43PM -0700, Richard Henderson wrote:
> I've been thinking for a while about how to reliably test TCG backends, and
> maybe how to do regression testing on them.  Having to begin the test from a
> guest binary, especially considering the vast cross-compilation problem, is
> pretty much a non-starter.

Given the regular breakages we have with TCG, it's really a gread idea.
Usually most of the targets are working correctly so the problem is not
found immediately.

> I've been thinking of a truly stripped down target for the purpose, with a
> special-purpose machine loop and main to go with it.  I.e. avoid vl.c.

I think we should use as much as possible of the current infrastructure,
having two different main loops means they are going to be
desynchronised at some point.

Ideally it would be nice to have a machine called TCG (maybe with a few
variants for big/little endian, 32 and 64-bits) which loads TCG "code"
from a text file and execute it. Then we can imagine zillions of small
testcases like the GCC ones, and more can be added when a problem is
discovered and fixed.

> My current idea is that the test file consists of 3 sections: guest memory
> layout, raw TBs, and expected results.  Perhaps best explained with some examples:
> 
> (1a) I've split up this test into two TBs to prevent some constant folding.
> (1b) The guest machine should have enough registers to make it easy to perform
> lots of tests at once.  Even better if we can avoid the complication of memory.
> 
> --------------------------------------------
> # Test basic arithmetic, esp with constant arguments
> CODE
> 0:	movi_i32	a0, 0x12345678
> 	movi_i32	a1, 0x87654321
> 	movi_tl		pc, 1
> 	exit_tb		0
> 
> 1:	add_i32		a2, a1, a0
> 	sub_i32		a1, a1, a0
> 	movi_i32	t0, 1
> 	add_i32		a3, a0, t0
> 	sub_i32		a4, a0, t0
> 	movi_i32	t0, -1
> 	add_i32		a5, a0, t0
> 	sub_i32		a6, a0, t0
> 	movi_i32	t0, -0x5678
> 	add_i32		a7, a0, t0
> 	sub_i32		a8, a0, t0
> 	movi_i32	t0, 0x123456
> 	add_i32		a9, a0, t0
> 	sub_i32		a10, a0, t0
> 	movi_i32	t0, 0x12341234
> 	add_i32		a11, a0, t0
> 	sub_i32		a12, a0, t0
> 	call		done
> 
> RESULT
> 	a0	0x12345678
> 	a1	0x7530eca9
> 	a2	0x99999999
> 	a3	0x12345679
> 	a4	0x12345677
> 	a5	0x12345677
> 	a6	0x12345679
> 	a7	0x12340000
> 	a8	0x1234acf0
> 	a9	0x12468ace
> 	a10	0x12222222
> 	a11	0x246868ac
> 	a12	0x4444
> -------------------------------------------
> 
> (2) The ram layout should be flexible enough to test specific scenarios
> 
> -------------------------------------------
> # For 64-bit guest, 32-bit host, make sure we compare high-part of tlb
> MEM
> 	ram	0x0_0000_0000, 4096
> 	ram	0x1_0000_0000, 4096

Maybe we can provide a few helper in the simple TCG machine to create
and remove TLB entries, or to memset the memory. That way each code that
is run can use a different memory mapping.

> CODE
> 0:	movi_i64	b0, 0x12345678abcdef
> 	movi_i64	t1, 0
> 	movi_i64	t2, 0x1_0000_0000
> 	qemu_st32	b0, t1
> 	qemu_ld32	b1, t1
> 	qemu_ld32	b2, t2
> 	call		done
> 
> RESULT
> 	b0	0x12345678abcdef
> 	b1	0x12345678abcdef
> 	b2	0
> -------------------------------------------
> 
> (3a) If we let "ioram" be defined via memory_region_init_io, with the "/l" and
> "/b" qualifiers controlling the .endianness field of the region, then we can
> test the memory access path all the way through.
> (3b) The final test in the result section would *not* use the device interface
> from memory.h, but rather examine the backing ram directly.
> 
> -------------------------------------------
> MEM
>         ioram/l 0, 4096                         # little-endian device
> 	ioram/b 4096, 4096			# big-endian device
> CODE
> 0:      movi_tl         t0, 0x12345678
>         movi_tl         t1, 0
>         qemu_st32       t0, t1
> 	qemu_ld32	a0, t1
> 	movi_tl		t1, 4096
> 	qemu_st32	t0, t1
> 	qemu_ld32	a1, t1
>         call            done
> RESULT
> 	a0	0x12345678
> 	a1	0x12345678
>         mem/b   0, 0x78, 0x56, 0x34, 0x12       # little-endian bytes
> 	mem/b	4096, 0x12, 0x34, 0x56, 0x78	# big-endian bytes
> -------------------------------------------
> 
> Thoughts?  Anything that we ought to be testing that I haven't thought of here,
> and that this sort of structure couldn't support?

As said above this kind of structure might be difficult to maintain
while we increase it with a lot more tests.

I think the call to helpers, with variable number of arguments and
types, should be tested. Probably also access to the host memory through
the non-qemu ld/st functions. And maybe the condition ops, especially
with some corner cases with signed/unsigned comparisons.

The problem I also see is the testing of optional ops.

Also what about user mode? At least the memory access is different.

> It may be some time before I can progress this enough to usefulness, but I
> wanted to get this written down while it is fresh in my head.

That would be really great if it can be implemented at some point.
Thanks for working on that.

-- 
Aurelien Jarno                          GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net