[Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

[Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

[Alex Bennée]

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - changes to the language for the lesser known architectures
---
 _posts/2018-06-21-tcg-testing.md | 139 +++++++++++++++++++++++++++++++
 1 file changed, 139 insertions(+)
 create mode 100644 _posts/2018-06-21-tcg-testing.md

diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md
new file mode 100644
index 0000000..dfc504c
--- /dev/null
+++ b/_posts/2018-06-21-tcg-testing.md
@@ -0,0 +1,139 @@
+---
+layout: post
+title:  "QEMU TCG Tests"
+date:   2018-06-21 10:30:00:00 +0000
+last_modified_at: 2018-06-21 10:30:00:00 +0000
+author: Alex Bennée
+categories: [testing, docker, compilation, tcg]
+---
+
+Ever since I started working on QEMU there was a small directory
+called tests/tcg that was in a perpetually broken state. It contains
+tests that exercise QEMU's ability to work across architectures using
+the power of the Tiny Code Generator. However as these tests needed to
+be compiled for the *guest* architectures and not the *host*
+architecture - known as cross-compiling - most developers never ran
+them. As the tests were hardly ever built inevitably a certain amount
+of bit-rot set in.
+
+# Cross Compilers
+
+In the old days cross-compilation setups were almost all hand-crafted
+affairs which involved building versions of binutils, gcc and a basic
+libc. If you couldn't get someone to give you a pre-built tarball it
+was something you laboured through once and hopefully never had to
+touch again. There were even dedicated scripts like crosstool-ng which
+attempted to make the process of patching and configuring your
+toolchain easier.
+
+While the distributions have improved their support for cross
+compilers over the years there are still plenty of variations in how
+they are deployed. It is hard for a project like QEMU which has to
+build on a wide range of operating systems and architectures to
+seamlessly use any given distributions compiler setup. However for
+those with cross compilers to hand `configure` now accepts two
+additional flags:
+
+    --cross-cc-$(ARCH)
+    --cross-cc-flags-$(ARCH)
+
+With a compiler specified for each guest architecture you want to test
+the build system can now build and run the tests. However for
+developers that don't have cross compilers around they can now take
+advantage of QEMU's docker images.
+
+# Enter Docker Containers
+
+If you work in IT you would be hard pressed not to have noticed the
+hype around Docker and the concept of containerisation over the last
+few years. Put simply containers allow you to define a known working
+set of software that gets run in an isolated environment for a given
+task. While this has many uses for QEMU it allows us to define build
+environments that any developer can run without having to mess around
+with their preferred host setup.
+
+Over the last few years QEMU's build system has been expanding the
+number of docker images it supports. Most of this has been in service
+of our CI testing such as [Patchew](https://patchew.org/QEMU/) and
+[Shippable](https://app.shippable.com/github/qemu/qemu/dashboard) but
+any developer with a docker setup can run the exact same images. For
+example if you want to check your patches won't break when compiled on
+a 32 bit ARM system you can run:
+
+    make docker-test-build@debian-armhf-cross J=n
+
+instead of tracking down a piece of ARM hardware to actually build on.
+Run `make docker` in your source tree to see the range of builds and
+tests it can support.
+
+# make check-tcg
+
+With the latest work [merged into
+master](https://git.qemu.org/?p=qemu.git;a=commit;h=de44c044420d1139480fa50c2d5be19223391218) we can now
+take advantage of both hand configured and docker based cross
+compilers to build test cases for TCG again. To run the TCG tests
+after you have built QEMU:
+
+    make check-tcg
+
+and the build system will build and run all the tests it can for your
+configured targets.
+
+# Rules for tests/tcg
+
+So now we have the infrastructure in place to add new tests what rules
+need to be followed to add new tests? 
+
+Well the first thing to note is currently all the tests are for
+linux-user versions of QEMU. This means the tests are all currently
+user-space tests that have access to the Linux syscall ABI.
+
+Another thing to note is the tests are standalone from the rest of the
+QEMU test infrastructure. To keep things simple they are compiled as
+standalone "static" binaries. As the cross-compilation setup can be
+quite rudimentary for some of the rarer architectures we only compile
+against a standard libc. There is no support for linking to other
+libraries like for example glib. Thread and maths support is part of
+glibc so shouldn't be a problem.
+
+Finally when writing new tests consider if it really is architecture
+specific or can be added to `tests/tcg/multiarch`. The multiarch tests
+are re-built for every supported architecture and should be the
+default place for anything explicitly testing syscalls and other
+common parts of the code base.
+
+# What's Next
+
+My hope with this work is we can start adding more tests to
+systematically defend functionality in linux-user. In fact I hope the
+first port of call to reproducing a crash would be writing a test case
+that can be added to our growing library of tests.
+
+Another thing that needs sorting out is getting toolchains for all of
+the less common architectures. The current work relies heavily on the
+excellent work of the Debian toolchain team in making multiarch
+aware cross compilers available in their distribution. However QEMU
+supports a lot more architectures than QEMU, some only as system
+emulations. In principle supporting them is as easy as adding another
+docker recipe but it might be these recipes end up having to compile
+the compilers from source.
+
+The `tests/tcg` directory still contains a number of source files we
+don't build. 
+
+The cris and openrisc directories contain user-space tests which just
+need the support of a toolchain and the relevant Makefile plumbing to
+be added.
+
+The lm32, mips and xtensa targets have a set of tests that need a
+system emulator. Aside from adding the compilers as docker images some
+additional work is needed to handle the differences between plain
+linux-user tests which can simply return an exit code to getting the
+results from a qemu-system emulation. Some architectures have
+semi-hosting support already for this while others report their test
+status over a simple serial link which will need to be parsed and
+handled in custom versions of the
+[`run-%:`](https://git.qemu.org/?p=qemu.git;a=blob;f=tests/tcg/Makefile;h=bf064153900a438e4ad8e2d79eaaac8a27d66062;hb=HEAD#l95)
+rule.
+
+
-- 
2.17.1

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

[Paolo Bonzini]

On 26/06/2018 16:35, Alex Bennée wrote:
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Pushed with a little bit of editing, thanks!

diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-28-tcg-testing.md
similarity index 83%
rename from _posts/2018-06-21-tcg-testing.md
rename to _posts/2018-06-28-tcg-testing.md
index e61312e..026e726 100644
--- a/_posts/2018-06-21-tcg-testing.md
+++ b/_posts/2018-06-28-tcg-testing.md
@@ -7,39 +7,39 @@ author: Alex Bennée
 categories: [testing, docker, compilation, tcg]
 ---
 
-Ever since I started working on QEMU there was a small directory
-called tests/tcg that was in a perpetually broken state. It contains
+Ever since I started working on QEMU, a small directory
+called `tests/tcg` has been in a perpetually broken state. It contains
 tests that exercise QEMU's ability to work across architectures using
 the power of the Tiny Code Generator. However as these tests needed to
 be compiled for the *guest* architectures and not the *host*
-architecture - known as cross-compiling - most developers never ran
-them. As the tests were hardly ever built inevitably a certain amount
-of bit-rot set in.
+architecture&mdash;this is known as cross-compiling&mdash;most developers
+never ran them. As the tests were hardly ever built inevitably a certain
+amount of bit-rot set in.
 
 # Cross Compilers
 
-In the old days cross-compilation setups were almost all hand-crafted
+In the old days, cross-compilation setups were almost all hand-crafted
 affairs which involved building versions of binutils, gcc and a basic
-libc. If you couldn't get someone to give you a pre-built tarball it
+libc. If you couldn't get someone to give you a pre-built tarball, it
 was something you laboured through once and hopefully never had to
 touch again. There were even dedicated scripts like crosstool-ng which
 attempted to make the process of patching and configuring your
 toolchain easier.
 
 While the distributions have improved their support for cross
-compilers over the years there are still plenty of variations in how
+compilers over the years, there are still plenty of variations in how
 they are deployed. It is hard for a project like QEMU which has to
 build on a wide range of operating systems and architectures to
 seamlessly use any given distributions compiler setup. However for
-those with cross compilers to hand `configure` now accepts two
+those with cross compilers at hand `configure` now accepts two
 additional flags:
 
     --cross-cc-$(ARCH)
     --cross-cc-flags-$(ARCH)
 
 With a compiler specified for each guest architecture you want to test
-the build system can now build and run the tests. However for
-developers that don't have cross compilers around they can now take
+the build system can now build and run the tests. For
+developers that don't have cross compilers around, they can take
 advantage of QEMU's docker images.
 
 # Enter Docker Containers
@@ -66,11 +66,11 @@ instead of tracking down a piece of ARM hardware to actually build on.
 Run `make docker` in your source tree to see the range of builds and
 tests it can support.
 
-# make check-tcg
+# `make check-tcg`
 
 With the latest work [merged into
 master](https://git.qemu.org/?p=qemu.git;a=commit;h=de44c044420d1139480fa50c2d5be19223391218) we can now
-take advantage of both hand configured and docker based cross
+take advantage of either hand-configured and Docker-based cross
 compilers to build test cases for TCG again. To run the TCG tests
 after you have built QEMU:
 
@@ -79,16 +79,16 @@ after you have built QEMU:
 and the build system will build and run all the tests it can for your
 configured targets.
 
-# Rules for tests/tcg
+# Rules for `tests/tcg`
 
 So now we have the infrastructure in place to add new tests what rules
 need to be followed to add new tests? 
 
-Well the first thing to note is currently all the tests are for
-linux-user versions of QEMU. This means the tests are all currently
+Well the first thing to note is currently all the tests are for the
+`linux-user` variant of QEMU. This means the tests are all currently
 user-space tests that have access to the Linux syscall ABI.
 
-Another thing to note is the tests are standalone from the rest of the
+Another thing to note is the tests are separate from the rest of the
 QEMU test infrastructure. To keep things simple they are compiled as
 standalone "static" binaries. As the cross-compilation setup can be
 quite rudimentary for some of the rarer architectures we only compile
@@ -99,10 +99,10 @@ glibc so shouldn't be a problem.
 Finally when writing new tests consider if it really is architecture
 specific or can be added to `tests/tcg/multiarch`. The multiarch tests
 are re-built for every supported architecture and should be the
-default place for anything explicitly testing syscalls and other
+default place for anything that tests syscalls or other
 common parts of the code base.
 
-# What's Next
+# What's next
 
 My hope with this work is we can start adding more tests to
 systematically defend functionality in linux-user. In fact I hope the
@@ -135,5 +135,3 @@ status over a simple serial link which will need to be parsed and
 handled in custom versions of the
 [`run-%:`](https://git.qemu.org/?p=qemu.git;a=blob;f=tests/tcg/Makefile;h=bf064153900a438e4ad8e2d79eaaac8a27d66062;hb=HEAD#l95)
 rule.
-
-

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

[no-reply]

Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20180626143538.5390-1-alex.bennee@linaro.org
Subject: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
1efad9bff0 blogposts: add post about the new check-tcg infrastructure

=== OUTPUT BEGIN ===
Checking PATCH 1/1: blogposts: add post about the new check-tcg infrastructure...
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#14: 
new file mode 100644

ERROR: trailing whitespace
#103: FILE: _posts/2018-06-21-tcg-testing.md:85:
+need to be followed to add new tests? $

ERROR: trailing whitespace
#140: FILE: _posts/2018-06-21-tcg-testing.md:122:
+don't build. $

total: 2 errors, 1 warnings, 139 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

[Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

[Alex Bennée]

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++
 1 file changed, 129 insertions(+)
 create mode 100644 _posts/2018-06-21-tcg-testing.md

diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md
new file mode 100644
index 0000000..85abb7a
--- /dev/null
+++ b/_posts/2018-06-21-tcg-testing.md
@@ -0,0 +1,129 @@
+---
+layout: post
+title:  "QEMU TCG Tests"
+date:   2018-06-21 10:30:00:00 +0000
+last_modified_at: 2018-06-21 10:30:00:00 +0000
+author: Alex Bennée
+categories: [testing, docker, compilation, tcg]
+---
+
+Ever since I started working on QEMU there was a small directory
+called tests/tcg that was in a perpetually broken state. It contains
+tests that exercise QEMU's ability to work across architectures using
+the power of the Tiny Code Generator. However as these tests needed to
+be compiled for the *guest* architectures and not the *host*
+architecture - known as cross-compiling - most developers never ran
+them. As the tests were hardly ever built inevitably a certain amount
+of bit-rot set in.
+
+# Cross Compilers
+
+In the old days cross-compilation setups were almost all hand-crafted
+affairs which involved building versions of binutils, gcc and a basic
+libc. If you couldn't get someone to give you a pre-built tarball it
+was something you laboured through once and hopefully never had to
+touch again. There were even dedicated scripts like crosstool-ng which
+attempted to make the process of patching and configuring your
+toolchain easier.
+
+While the distributions have improved their support for cross
+compilers over the years there are still plenty of variations in how
+they are deployed. It is hard for a project like QEMU which has to
+build on a wide range of operating systems and architectures to
+seamlessly use any given distributions compiler setup. However for
+those with cross compilers to hand `configure` now accepts two
+additional flags:
+
+    --cross-cc-$(ARCH)
+    --cross-cc-flags-$(ARCH)
+
+With a compiler specified for each guest architecture you want to test
+the build system can now build and run the tests. However for
+developers that don't have cross compilers around they can now take
+advantage of QEMU's docker images.
+
+# Enter Docker Containers
+
+If you work in IT you would be hard pressed not to have noticed the
+hype around Docker and the concept of containerisation over the last
+few years. Put simply containers allow you to define a known working
+set of software that gets run in an isolated environment for a given
+task. While this has many uses for QEMU it allows us to define build
+environments that any developer can run without having to mess around
+with their preferred host setup.
+
+Over the last few years QEMU's build system has been expanding the
+number of docker images it supports. Most of this has been in service
+of our CI testing such as [Patchew](https://patchew.org/QEMU/) and
+[Shippable](https://app.shippable.com/github/qemu/qemu/dashboard) but
+any developer with a docker setup can run the exact same images. For
+example if you want to check your patches won't break when compiled on
+a 32 bit ARM system you can run:
+
+    make docker-test-build@debian-armhf-cross J=n
+
+instead of tracking down a piece of ARM hardware to actually build on.
+Run `make docker` in your source tree to see the range of builds and
+tests it can support.
+
+# make check-tcg
+
+With the latest work [merged into
+master](https://git.qemu.org/?p=qemu.git;a=commit;h=TODO) we can now
+take advantage of both hand configured and docker based cross
+compilers to build test cases for TCG again. To run the TCG tests
+after you have built QEMU:
+
+    make check-tcg
+
+and the build system will build and run all the tests it can for your
+configured targets.
+
+# Rules for tests/tcg
+
+So now we have the infrastructure in place to add new tests what rules
+need to be followed to add new tests? 
+
+Well the first thing to note is currently all the tests are for
+linux-user versions of QEMU. This means the tests are all currently
+user-space tests that have access to the Linux syscall ABI.
+
+Another thing to note is the tests are standalone from the rest of the
+QEMU test infrastructure. To keep things simple they are compiled as
+standalone "static" binaries. As the cross-compilation setup can be
+quite rudimentary for some of the rarer architectures we only compile
+against a standard libc. There is no support for linking to other
+libraries like for example glib. Thread and maths support is part of
+glibc so shouldn't be a problem.
+
+Finally when writing new tests consider if it really is architecture
+specific or can be added to `tests/tcg/multiarch`. The multiarch tests
+are re-built for every supported architecture and should be the
+default place for anything explicitly testing syscalls and other
+common parts of the code base.
+
+# What's Next
+
+My hope with this work is we can start adding more tests to
+systematically defend functionality in linux-user. In fact I hope the
+first port of call to reproducing a crash would be writing a test case
+that can be added to our growing library of tests.
+
+Another thing that needs sorting out is getting toolchains for all of
+the less common architectures. The current work relies heavily on the
+excellent work of the Debian toolchain team in making multiarch
+aware cross compilers available in their distribution. However QEMU
+supports a lot more architectures than QEMU, some only as system
+emulations. In principle supporting them is as easy as adding another
+docker recipe but it might be these recipes end up having to compile
+the compilers from source.
+
+The `tests/tcg` directory still contains a number of source files we
+don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have
+a set of tests that need a system emulator. Now we have the
+infrastructure for compiling I hope we can get support for system
+tests added fairly quickly. There will need to be some work to figure
+out a nice common way to pass results back to the build-system. For
+linux-user this is simple as all programs can simply return their exit
+code however for system emulation this is a little more involved.
+
-- 
2.17.1

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

[no-reply]

Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20180621184116.910-1-alex.bennee@linaro.org
Subject: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]               patchew/20180621184116.910-1-alex.bennee@linaro.org -> patchew/20180621184116.910-1-alex.bennee@linaro.org
Switched to a new branch 'test'
0a765327ad blogposts: add post about the new check-tcg infrastructure

=== OUTPUT BEGIN ===
Checking PATCH 1/1: blogposts: add post about the new check-tcg infrastructure...
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#13: 
new file mode 100644

ERROR: trailing whitespace
#102: FILE: _posts/2018-06-21-tcg-testing.md:85:
+need to be followed to add new tests? $

total: 1 errors, 1 warnings, 129 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

[Philippe Mathieu-Daudé]

On 06/21/2018 06:12 PM, no-reply@patchew.org wrote:
> This series seems to have some coding style problems. See output below for
> more information:
> 
> Type: series
> Message-id: 20180621184116.910-1-alex.bennee@linaro.org
> Subject: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure
[...]> === OUTPUT BEGIN ===
> Checking PATCH 1/1: blogposts: add post about the new check-tcg infrastructure...
> WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
> #13: 
> new file mode 100644
> 
> ERROR: trailing whitespace
> #102: FILE: _posts/2018-06-21-tcg-testing.md:85:
> +need to be followed to add new tests? $

=)

> 
> total: 1 errors, 1 warnings, 129 lines checked
> 
> Your patch has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

[Max Filippov]

On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++
>  1 file changed, 129 insertions(+)
>  create mode 100644 _posts/2018-06-21-tcg-testing.md
>
> diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md

[...]

> +The `tests/tcg` directory still contains a number of source files we
> +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have
> +a set of tests that need a system emulator. Now we have the
> +infrastructure for compiling I hope we can get support for system
> +tests added fairly quickly. There will need to be some work to figure
> +out a nice common way to pass results back to the build-system. For
> +linux-user this is simple as all programs can simply return their exit
> +code however for system emulation this is a little more involved.

xtensa tests pass exit codes to the build system through semihosting calls.
If any of them fails make check fails as well.

-- 
Thanks.
-- Max

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

[Alex Bennée]

Max Filippov <jcmvbkbc@gmail.com> writes:

> On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>  _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++
>>  1 file changed, 129 insertions(+)
>>  create mode 100644 _posts/2018-06-21-tcg-testing.md
>>
>> diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md
>
> [...]
>
>> +The `tests/tcg` directory still contains a number of source files we
>> +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have
>> +a set of tests that need a system emulator. Now we have the
>> +infrastructure for compiling I hope we can get support for system
>> +tests added fairly quickly. There will need to be some work to figure
>> +out a nice common way to pass results back to the build-system. For
>> +linux-user this is simple as all programs can simply return their exit
>> +code however for system emulation this is a little more involved.
>
> xtensa tests pass exit codes to the build system through semihosting calls.
> If any of them fails make check fails as well.

I've re-written that section as:

  The `tests/tcg` directory still contains a number of source files we
  don't build.

  The cris and openrisc directories contain user-space tests which just
  need the support of a toolchain and the relevant Makefile plumbing to
  be added.

  The lm32, mips and xtensa targets have a set of tests that need a
  system emulator. Aside from adding the compilers as docker images some
  additional work is needed to handle the differences between plain
  linux-user tests which can simply return an exit code to getting the
  results from a qemu-system emulation. Some architectures have
  semi-hosting support already for this while others report their test
  status over a simple serial link which will need to be parsed and
  handled in the `run-%:` test rule.

How is that?

Any chance you could look into what it would take to package up the
xtensa toolchain in a docker container? Are they simply tarballs of
binaries?

--
Alex Bennée

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

[Max Filippov]

On Fri, Jun 22, 2018 at 2:03 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Max Filippov <jcmvbkbc@gmail.com> writes:
>
>> On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> ---
>>>  _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++
>>>  1 file changed, 129 insertions(+)
>>>  create mode 100644 _posts/2018-06-21-tcg-testing.md
>>>
>>> diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md
>>
>> [...]
>>
>>> +The `tests/tcg` directory still contains a number of source files we
>>> +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have
>>> +a set of tests that need a system emulator. Now we have the
>>> +infrastructure for compiling I hope we can get support for system
>>> +tests added fairly quickly. There will need to be some work to figure
>>> +out a nice common way to pass results back to the build-system. For
>>> +linux-user this is simple as all programs can simply return their exit
>>> +code however for system emulation this is a little more involved.
>>
>> xtensa tests pass exit codes to the build system through semihosting calls.
>> If any of them fails make check fails as well.
>
> I've re-written that section as:
>
>   The `tests/tcg` directory still contains a number of source files we
>   don't build.
>
>   The cris and openrisc directories contain user-space tests which just
>   need the support of a toolchain and the relevant Makefile plumbing to
>   be added.
>
>   The lm32, mips and xtensa targets have a set of tests that need a
>   system emulator. Aside from adding the compilers as docker images some
>   additional work is needed to handle the differences between plain
>   linux-user tests which can simply return an exit code to getting the
>   results from a qemu-system emulation. Some architectures have
>   semi-hosting support already for this while others report their test
>   status over a simple serial link which will need to be parsed and
>   handled in the `run-%:` test rule.
>
> How is that?

'run-%' goal is only present in xtensa Makefile, other test suites have
explicit loop in the 'check' goal. Otherwise LGTM.

> Any chance you could look into what it would take to package up the
> xtensa toolchain in a docker container?

Can you point me to an example?

> Are they simply tarballs of binaries?

Yes, we have that option:

  https://github.com/foss-xtensa/toolchain/releases

Or they may be built from source.

-- 
Thanks.
-- Max

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

[Alex Bennée]

Max Filippov <jcmvbkbc@gmail.com> writes:

> On Fri, Jun 22, 2018 at 2:03 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> Max Filippov <jcmvbkbc@gmail.com> writes:
>>
>>> On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>> ---
>>>>  _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++
>>>>  1 file changed, 129 insertions(+)
>>>>  create mode 100644 _posts/2018-06-21-tcg-testing.md
>>>>
>>>> diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md
>>>
>>> [...]
>>>
>>>> +The `tests/tcg` directory still contains a number of source files we
>>>> +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have
>>>> +a set of tests that need a system emulator. Now we have the
>>>> +infrastructure for compiling I hope we can get support for system
>>>> +tests added fairly quickly. There will need to be some work to figure
>>>> +out a nice common way to pass results back to the build-system. For
>>>> +linux-user this is simple as all programs can simply return their exit
>>>> +code however for system emulation this is a little more involved.
>>>
>>> xtensa tests pass exit codes to the build system through semihosting calls.
>>> If any of them fails make check fails as well.
>>
>> I've re-written that section as:
>>
>>   The `tests/tcg` directory still contains a number of source files we
>>   don't build.
>>
>>   The cris and openrisc directories contain user-space tests which just
>>   need the support of a toolchain and the relevant Makefile plumbing to
>>   be added.
>>
>>   The lm32, mips and xtensa targets have a set of tests that need a
>>   system emulator. Aside from adding the compilers as docker images some
>>   additional work is needed to handle the differences between plain
>>   linux-user tests which can simply return an exit code to getting the
>>   results from a qemu-system emulation. Some architectures have
>>   semi-hosting support already for this while others report their test
>>   status over a simple serial link which will need to be parsed and
>>   handled in the `run-%:` test rule.
>>
>> How is that?
>
> 'run-%' goal is only present in xtensa Makefile, other test suites have
> explicit loop in the 'check' goal. Otherwise LGTM.

The TCG tests added a generic run-% rule:

  run-%: %
          $(call run-test, $<, $(QEMU) $<, "$< on $(TARGET_NAME)")

Although individual tests can override it.

>
>> Any chance you could look into what it would take to package up the
>> xtensa toolchain in a docker container?
>
> Can you point me to an example?

Well this was the base attempt:

https://github.com/stsquad/qemu/commit/daee9dc013b13b13e81f720176c5e379c05ed0e0

So I guess we need to add the overlay to the image?

>
>> Are they simply tarballs of binaries?
>
> Yes, we have that option:
>
>   https://github.com/foss-xtensa/toolchain/releases
>
> Or they may be built from source.


--
Alex Bennée