Question about TCG backend correctness

Question about TCG backend correctness

[LIU Zhiwei]

Hi folks,

     For TCG front end, we can test it with tools, such as RISU. But I 
don't know if  there are some tools that can help
to verify the correctness of a TCG backend.

     Can someone share the tools or the experience to debug RISC-V 
backend?  Thanks very much.

Best Regards,
Zhiwei

Re: Question about TCG backend correctness

[Alex Bennée]

LIU Zhiwei <zhiwei_liu@linux.alibaba.com> writes:

> Hi folks,
>
>     For TCG front end, we can test it with tools, such as RISU. But I
> don't know if  there are some tools that can help
> to verify the correctness of a TCG backend.
>
>     Can someone share the tools or the experience to debug RISC-V
> backend?  Thanks very much.

It's mostly down to inspection or debugging failures. Sometimes you can
attempt to shake out bugs by running a stacked QEMU. e.g.

   qemu-system-aarch64 on x86 host
   qemu-system-aarch64 on qemu-system-riscv64 on x86 host

and see if the two aarch64 guests run the same. This of course gets very
tricky running full OS kernels because as soon as time and async
interrupts get involved you will get divergence anyway. This would work
best with simple straight line test cases (e.g. check-tcg test cases).

I've long wanted to have the ability to have TCG unit tests where a
virtual processor could be defined for the purpose of directly
exercising TCG. This would be mainly to check tcg_optimize() works
correctly but no doubt could be extended to verify the eventual backend
output. The problem with implementing such an approach has been the
amount of boilerplate needed to define a simple frontend. Maybe this
will get simpler as we slowly move to a single binary build but there is
still quite of lot of things TCG needs to know about the guest it is
emulating.

If you would like to attempt improve the testing situation for TCG and its
backend I'm fully supportive.

-- 
Alex Bennée

Re: Question about TCG backend correctness

[LIU Zhiwei]

On 2022/10/17 18:30, Alex Bennée wrote:
> LIU Zhiwei <zhiwei_liu@linux.alibaba.com> writes:
>
>> Hi folks,
>>
>>      For TCG front end, we can test it with tools, such as RISU. But I
>> don't know if  there are some tools that can help
>> to verify the correctness of a TCG backend.
>>
>>      Can someone share the tools or the experience to debug RISC-V
>> backend?  Thanks very much.
> It's mostly down to inspection or debugging failures.
Agree.
>   Sometimes you can
> attempt to shake out bugs by running a stacked QEMU. e.g.
>
>     qemu-system-aarch64 on x86 host
>     qemu-system-aarch64 on qemu-system-riscv64 on x86 host
>
> and see if the two aarch64 guests run the same.
We currently depend on the guest behaviors to test the correctness of 
TCG backend. And if the guest doesn't
behave correctly, it is hard to find the exact bug in backend.

Maybe I can run RISU on qemu-aarch64(x86) and qemu-aarch64(risc-v) to 
check the RISC-V backend.

>   This of course gets very
> tricky running full OS kernels because as soon as time and async
> interrupts get involved you will get divergence anyway. This would work
> best with simple straight line test cases (e.g. check-tcg test cases).
>
> I've long wanted to have the ability to have TCG unit tests where a
> virtual processor could be defined for the purpose of directly
> exercising TCG.

We already have many ISAs as the front end of TCG. Will the virtual 
processor here be some
different?

> This would be mainly to check tcg_optimize() works
> correctly but no doubt could be extended to verify the eventual backend
> output. The problem with implementing such an approach has been the
> amount of boilerplate needed to define a simple frontend.
Sorry, I don't get it.
>   Maybe this
> will get simpler as we slowly move to a single binary build but there is
> still quite of lot of things TCG needs to know about the guest it is
> emulating.
>
> If you would like to attempt improve the testing situation for TCG and its
> backend I'm fully supportive.

If I can find a  good way, I really like to do some further work.

Thanks，
Zhiwei

>

Re: Question about TCG backend correctness

[Richard Henderson]

On 10/18/22 01:27, LIU Zhiwei wrote:
> Maybe I can run RISU on qemu-aarch64(x86) and qemu-aarch64(risc-v) to check the RISC-V 
> backend.

This is a good start for debugging a tcg backend. It's not comprehensive, because RISU 
executes one instruction at a time then raises an exception to check the results.  This 
means that the tcg optimizer doesn't have much to work with, which means that the tcg 
backend is not as stressed as it could be.

>> I've long wanted to have the ability to have TCG unit tests where a
>> virtual processor could be defined for the purpose of directly
>> exercising TCG.
> 
> We already have many ISAs as the front end of TCG. Will the virtual processor here be some
> different?

It wouldn't.  This is my argument against creating a new virtual processor.

I do think we should be better about creating regression tests for bugs fixed, in the form 
of small focused assembly test cases which get run via check-tcg.


r~

Re: Question about TCG backend correctness

[Alex Bennée]

Richard Henderson <richard.henderson@linaro.org> writes:

> On 10/18/22 01:27, LIU Zhiwei wrote:
>> Maybe I can run RISU on qemu-aarch64(x86) and qemu-aarch64(risc-v)
>> to check the RISC-V backend.
>
> This is a good start for debugging a tcg backend. It's not
> comprehensive, because RISU executes one instruction at a time then
> raises an exception to check the results.  This means that the tcg
> optimizer doesn't have much to work with, which means that the tcg
> backend is not as stressed as it could be.
>
>>> I've long wanted to have the ability to have TCG unit tests where a
>>> virtual processor could be defined for the purpose of directly
>>> exercising TCG.
>> We already have many ISAs as the front end of TCG. Will the virtual
>> processor here be some
>> different?
>
> It wouldn't.  This is my argument against creating a new virtual
> processor.

I'm not wedded to the idea - but it could be a super simple processor
with a lot less extra baggage than a full ISA - purely for throwing TCG
ops at rather than decoding any machine code.

> I do think we should be better about creating regression tests for
> bugs fixed, in the form of small focused assembly test cases which get
> run via check-tcg.

Can we detect optimisation failures with guest tests? Have we written
any so far?

Is it worth doing anything to lower the barrier of entry for these small
micro tests or is the current check-tcg framework enough?

>
>
> r~


-- 
Alex Bennée

Re: Question about TCG backend correctness

[Richard Henderson]

On 10/18/22 19:22, Alex Bennée wrote:
> I'm not wedded to the idea - but it could be a super simple processor
> with a lot less extra baggage than a full ISA - purely for throwing TCG
> ops at rather than decoding any machine code.

I'd be surprised if you don't wind up with a similar amount of code as for a full ISA, and 
a lot more baggage on the side besides.  But feel free to prove me wrong...

OTOH, if we really did split out a libtcg, disconnected from any target and parameterized, 
that would make the sort of non-ISA unit testing you're thinking about vastly easier, 
because you'd be able to write a driver program that isn't "qemu", without all of the 
target baggage that would currently make this infeasible.

And, very most likely, such a parameterized libtcg will be required for full heterogeneous 
emulation...

> Can we detect optimisation failures with guest tests?

No, and that's where some sort of specific harness might be helpful.

> Is it worth doing anything to lower the barrier of entry for these small
> micro tests or is the current check-tcg framework enough?

I think the current check-tcg framework probably sufficient.
Generally I think

int main()
{
    asm("test");
    assert(result);
    return 0;
}

is what these micro tests should be aiming for.


r~

Re: Question about TCG backend correctness

[LIU Zhiwei]

[-- Attachment #1: Type: text/plain, Size: 2435 bytes --]


On 2022/10/18 13:22, Richard Henderson wrote:
> On 10/18/22 01:27, LIU Zhiwei wrote:
>> Maybe I can run RISU on qemu-aarch64(x86) and qemu-aarch64(risc-v) to 
>> check the RISC-V backend.
>
> This is a good start for debugging a tcg backend. 

After fixing some bugs, RISU can run  some instructions now. Thanks.


IMHO,  the next 2 requests should be satisfied for TCG test.

1. Add a unit test case for any lower level and small function is easy.

    For example, the function in risc-v backend,

    static bool tcg_out_mov(TCGContext *s, TCGType type, TCGReg ret,
    TCGReg arg)
      {
          if (ret == arg) {
              return true;
          }
          switch (type) {
          case TCG_TYPE_I32:
          case TCG_TYPE_I64:
              tcg_out_opc_imm(s, OPC_ADDI, ret, arg, 0);
              break;
          default:
              g_assert_not_reached();
          }
          return true;
      }

    Write a unit test case for it is not easy currently.  I don't know
    how to fill the TCGContext struct and other parameters.
    And there is no test framework where I can reuse.

    The others may say that we can run a lot of benchmark to ensure we
    have a good coverage and don't
    need such a low level test case. Some reasonable. But it will lead
    to a very high test burden and it is very hard to get
    a good coverage only through high level test.


2. Add a unit test case for any high level function is easy, such as 
tcg_gen_code.

    The check-tcg test belongs to the high level testing.

Best Regards，
Zhiwei

> It's not comprehensive, because RISU executes one instruction at a 
> time then raises an exception to check the results.  This means that 
> the tcg optimizer doesn't have much to work with, which means that the 
> tcg backend is not as stressed as it could be.
>
>>> I've long wanted to have the ability to have TCG unit tests where a
>>> virtual processor could be defined for the purpose of directly
>>> exercising TCG.
>>
>> We already have many ISAs as the front end of TCG. Will the virtual 
>> processor here be some
>> different?
>
> It wouldn't.  This is my argument against creating a new virtual 
> processor.
>
> I do think we should be better about creating regression tests for 
> bugs fixed, in the form of small focused assembly test cases which get 
> run via check-tcg.
>
>
> r~

[-- Attachment #2: Type: text/html, Size: 3782 bytes --]

Re: Question about TCG backend correctness

[Alex Bennée]

LIU Zhiwei <zhiwei_liu@linux.alibaba.com> writes:

> On 2022/10/18 13:22, Richard Henderson wrote:
>
>  On 10/18/22 01:27, LIU Zhiwei wrote: 
>
>  Maybe I can run RISU on qemu-aarch64(x86) and qemu-aarch64(risc-v) to check the RISC-V backend. 
>
>  This is a good start for debugging a tcg backend. 
>
> After fixing some bugs, RISU can run  some instructions now. Thanks.
>
> IMHO,  the next 2 requests should be satisfied for TCG test.
>
> 1. Add a unit test case for any lower level and small function is easy. 
>
>  For example, the function in risc-v backend, 
>
>  static bool tcg_out_mov(TCGContext *s, TCGType type, TCGReg ret, TCGReg arg)
>   {
>       if (ret == arg) {
>           return true;
>       }
>       switch (type) {
>       case TCG_TYPE_I32:
>       case TCG_TYPE_I64:
>           tcg_out_opc_imm(s, OPC_ADDI, ret, arg, 0);
>           break;
>       default:
>           g_assert_not_reached();
>       }
>       return true;
>   }
>
>  Write a unit test case for it is not easy currently.  I don't know how to fill the TCGContext struct and other
>  parameters.
>  And there is no test framework where I can reuse.
>
>  The others may say that we can run a lot of benchmark to ensure we have a good coverage and don't
>  need such a low level test case. Some reasonable. But it will lead to a very high test burden and it is very hard to
>  get
>  a good coverage only through high level test.

You can at the very least use the gcov build to identify which bits of
the backend are not exercised by "check-tcg" and maybe write a few more
tests to fill it in. It's not a direct exercising of the code but it at
least ensures it has run and worked at least once.

>
> 2. Add a unit test case for any high level function is easy, such as tcg_gen_code. 
>
>  The check-tcg test belongs to the high level testing. 
>
> Best Regards，
> Zhiwei
>
>  It's not comprehensive, because RISU executes one instruction at a time then raises an exception to check the
>  results.  This means that the tcg optimizer doesn't have much to work with, which means that the tcg backend is not
>  as stressed as it could be. 
>
>  I've long wanted to have the ability to have TCG unit tests where a 
>  virtual processor could be defined for the purpose of directly 
>  exercising TCG. 
>
>  We already have many ISAs as the front end of TCG. Will the virtual processor here be some 
>  different? 
>
>  It wouldn't.  This is my argument against creating a new virtual processor. 
>
>  I do think we should be better about creating regression tests for bugs fixed, in the form of small focused assembly
>  test cases which get run via check-tcg. 
>
>  r~ 


-- 
Alex Bennée