[Qemu-devel] [RFC] ARM test generator

[Qemu-devel] [RFC] ARM test generator

[Laurent Desnogues]

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]

Hello,

this is a very simple test generator for ARM.

The aim is to help test instruction correctness since several people
already noticed some problems, that are not always easy to
reproduce in the absence of test infrastructure (a big word for such
a simple thing).

The process is simple:
  - an input file gives a list of instructions with inputs
  - C and asm files are generated and compiled for ARM
  - the test can then be run on a reference platform and with
    arm-linux-user qemu, and outputs can be compared.

The sample input file I provide demonstrates bugs that are
corrected by my proposed patch:

http://lists.gnu.org/archive/html/qemu-devel/2008-06/msg00699.html

This should not be taken too seriously, this is just an aid to
establish communication between patchers and maintainers :-)

BTW I put this under BSD license,  let me know if that is a problem
for inclusion into qemu (in case that ever happens, which I hope
will...).

Comments are obviously very welcome.


Laurent

[-- Attachment #2: test-arm-v00.tar.bz2 --]
[-- Type: application/x-bzip2, Size: 6333 bytes --]

Re: [Qemu-devel] [RFC] ARM test generator

[Paul Brook]

[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]

On Tuesday 08 July 2008, Laurent Desnogues wrote:
> Hello,
>
> this is a very simple test generator for ARM.
>
> The aim is to help test instruction correctness since several people
> already noticed some problems, that are not always easy to
> reproduce in the absence of test infrastructure (a big word for such
> a simple thing).

I suggest creating a separate project for this.

I'm unconvinced how much value this actually adds unless you can get fairly 
extensive coverage.  i.e. the interesting bit is how you generate 
test-arm.txt and test-ref.txt. You should also cover different combinations 
of overlapping source/destination operands.

I wrote reasonably extensive self-checking tests for a subset of NEON. However 
it ended up a bit of a mess, needs rewriting (probably in something other 
than C), doesn't cover some of the hard instructions, and the input generator 
needs a lot of improvement.  I've attached it for reference.

GCC also has a set of tests for the NEON compiler intrinsics.  We've 
considered augmenting those to be runtime tests.

Paul

[-- Attachment #2: test-neon.tar.bz2 --]
[-- Type: application/x-tbz, Size: 8399 bytes --]

Re: [Qemu-devel] [RFC] ARM test generator

[Laurent Desnogues]

On Tue, Jul 8, 2008 at 3:26 PM, Paul Brook <paul@codesourcery.com> wrote:
> I'm unconvinced how much value this actually adds unless you can get fairly
> extensive coverage.  i.e. the interesting bit is how you generate
> test-arm.txt and test-ref.txt.

Of course the problem is the data coverage (as opposed to code
coverage) for which no tool will tell us how close we are to 100%.

For test-ref.txt generation, the solution is trivial:  use real hardware.
As long as you are not testing unpredictable or implementation
defined behaviours, that can be considered as a reference that
qemu should match.

> You should also cover different combinations
> of overlapping source/destination operands.

That's a good point.

Anyway, the aim of my generator is not to fully validate qemu ARM
implementation.  I just wanted some simple stuff to provide
demonstrable evidence of a problem when words are not enough :)


Laurent

Re: [Qemu-devel] [RFC] ARM test generator

[Paul Brook]

> For test-ref.txt generation, the solution is trivial:  use real hardware.
> As long as you are not testing unpredictable or implementation
> defined behaviours, that can be considered as a reference that
> qemu should match.

For any reasonable coverage test I'd expect test-ref.txt to be huge, which 
makes testing without reference hardware infeasible (and certainly not 
suitable for inclusion in qemu SVN). AFAIK For ARMv7 then there are currently 
zero generally available hardware platforms, and ARMv6 hardware is still 
fairly rare.

Paul

Re: [Qemu-devel] [RFC] ARM test generator

[Laurent Desnogues]

On Tue, Jul 8, 2008 at 5:28 PM, Paul Brook <paul@codesourcery.com> wrote:
> For any reasonable coverage test I'd expect test-ref.txt to be huge, which
> makes testing without reference hardware infeasible (and certainly not
> suitable for inclusion in qemu SVN). AFAIK For ARMv7 then there are currently
> zero generally available hardware platforms, and ARMv6 hardware is still
> fairly rare.

I agree the lack of easily available ARMv7 hardware is a problem though
some people are already getting board (mine should be in within 2 weeks).
As far as ARMv6 is concerned the Nokia n8x0 use an ARMv6 processor;
again that's not widespread enough.  (BTW Nokia n8x0 dev environment
uses qemu and fails running ffmpeg ARMv6.)

I also admit that putting the ref in svn is not the way to go.

So you would have to trust someone to provide you with a ref output to
check patch correctness.  Basically we're back to the starting point:  you
need to be convinced :-)


Laurent