[Qemu-devel] [PATCH 0/3] tests/acceptance: Add tests for the Leon3 board

[Qemu-devel] [PATCH 0/3] tests/acceptance: Add tests for the Leon3 board

[Philippe Mathieu-Daudé]

Quick tests worth to avoid regressions, idea from
https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04177.html
"Maintainers, please tell us how to boot your machines"

Regards,

Phil.

Philippe Mathieu-Daudé (3):
  tests/acceptance: Add test that boots the HelenOS microkernel on Leon3
  tests/acceptance: Add test that boots Linux up to BusyBox on Leon3
  .travis.yml: Let the avocado job run the Leon3 test

 .travis.yml                             |  2 +-
 MAINTAINERS                             |  1 +
 tests/acceptance/machine_sparc_leon3.py | 89 +++++++++++++++++++++++++
 3 files changed, 91 insertions(+), 1 deletion(-)
 create mode 100644 tests/acceptance/machine_sparc_leon3.py

-- 
2.19.1

[Qemu-devel] [PATCH 1/3] tests/acceptance: Add test that boots the HelenOS microkernel on Leon3

[Philippe Mathieu-Daudé]

Release notes:
http://www.helenos.org/wiki/Download#HelenOS0.6.0

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 MAINTAINERS                             |  1 +
 tests/acceptance/machine_sparc_leon3.py | 58 +++++++++++++++++++++++++
 2 files changed, 59 insertions(+)
 create mode 100644 tests/acceptance/machine_sparc_leon3.py

diff --git a/MAINTAINERS b/MAINTAINERS
index cad58b9487..8660e13ef2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1158,6 +1158,7 @@ S: Maintained
 F: hw/sparc/leon3.c
 F: hw/*/grlib*
 F: include/hw/*/grlib*
+F: tests/acceptance/machine_sparc_leon3.py
 
 S390 Machines
 -------------
diff --git a/tests/acceptance/machine_sparc_leon3.py b/tests/acceptance/machine_sparc_leon3.py
new file mode 100644
index 0000000000..0bbae44f85
--- /dev/null
+++ b/tests/acceptance/machine_sparc_leon3.py
@@ -0,0 +1,58 @@
+# Functional test that boots a Leon3 machine and checks its serial console.
+#
+# Copyright (c) Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import os
+import logging
+
+from avocado import skipIf
+from avocado_qemu import Test
+
+
+class Leon3Machine(Test):
+
+    timeout = 60
+
+    # TODO refactor to MachineTest
+    def wait_for_console_pattern(self, success_message, failure_message=None):
+        """
+        Waits for messages to appear on the console, while logging the content
+
+        :param success_message: if this message appears, test succeeds
+        :param failure_message: if this message appears, test fails
+        """
+        console = self.vm.console_socket.makefile()
+        console_logger = logging.getLogger('console')
+        while True:
+            msg = console.readline().strip()
+            if not msg:
+                continue
+            console_logger.debug(msg)
+            if success_message in msg:
+                break
+            if failure_message and failure_message in msg:
+                fail = 'Failure message found in console: %s' % failure_message
+                self.fail(fail)
+
+    def test_leon3_helenos_uimage(self):
+        """
+        :avocado: tags=arch:sparc
+        :avocado: tags=machine:leon3
+        :avocado: tags=binfmt:uimage
+        """
+        kernel_url = ('http://www.helenos.org/releases/'
+                      'HelenOS-0.6.0-sparc32-leon3.bin')
+        kernel_hash = 'a88c9cfdb8430c66650e5290a08765f9bf049a30'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_machine('leon3_generic')
+        self.vm.set_console()
+        self.vm.add_args('-kernel', kernel_path)
+
+        self.vm.launch()
+
+        self.wait_for_console_pattern('Copyright (c) 2001-2014 HelenOS project')
+        self.wait_for_console_pattern('Booting the kernel ...')
-- 
2.19.1

[Qemu-devel] [PATCH 2/3] tests/acceptance: Add test that boots Linux up to BusyBox on Leon3

[Philippe Mathieu-Daudé]

Gaisler provides convenient images:
https://www.gaisler.com/index.php/downloads/linux

HOWTO build:
https://www.gaisler.com/index.php/products/operating-systems/linux

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/machine_sparc_leon3.py | 31 +++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/tests/acceptance/machine_sparc_leon3.py b/tests/acceptance/machine_sparc_leon3.py
index 0bbae44f85..b3039b5d30 100644
--- a/tests/acceptance/machine_sparc_leon3.py
+++ b/tests/acceptance/machine_sparc_leon3.py
@@ -37,6 +37,11 @@ class Leon3Machine(Test):
                 fail = 'Failure message found in console: %s' % failure_message
                 self.fail(fail)
 
+    def exec_command_and_wait_for_pattern(self, command, success_message):
+        command += '\n'
+        self.vm.console_socket.sendall(command.encode())
+        self.wait_for_console_pattern(success_message)
+
     def test_leon3_helenos_uimage(self):
         """
         :avocado: tags=arch:sparc
@@ -56,3 +61,29 @@ class Leon3Machine(Test):
 
         self.wait_for_console_pattern('Copyright (c) 2001-2014 HelenOS project')
         self.wait_for_console_pattern('Booting the kernel ...')
+
+    def test_leon3_linux_kernel_4_9_busybox(self):
+        """
+        :avocado: tags=arch:sparc
+        :avocado: tags=machine:leon3
+        """
+        kernel_url = ('https://www.gaisler.com/anonftp/linux/linux-4/images/'
+                     'leon-linux-4.9/leon-linux-4.9-1.0/up/image.ram')
+        kernel_hash = '289bd1bcca10cda76d0ef2264a8657adc251f5f5'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_machine('leon3_generic')
+        self.vm.set_console()
+        self.vm.add_args('-kernel', kernel_path)
+
+        self.vm.launch()
+
+        self.wait_for_console_pattern('TYPE: Leon3 System-on-a-Chip')
+
+        self.wait_for_console_pattern('Welcome to Buildroot')
+
+        self.wait_for_console_pattern('buildroot login:')
+        self.exec_command_and_wait_for_pattern('root', '#')
+        uname = 'Linux buildroot 4.9.54-00018-g62dab2c #2 ' \
+                'Wed Oct 18 09:45:51 CEST 2017 sparc GNU/Linux'
+        self.exec_command_and_wait_for_pattern('uname -a', uname)
-- 
2.19.1

[Qemu-devel] [PATCH 3/3] .travis.yml: Let the avocado job run the Leon3 test

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
If this list continues to grow we can
- split it (as other jobs)
- move them to GitLab where we can have multi-stage jobs,
  avocado tests run on top of build jobs.
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index aeb9b211cd..37708b3438 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -231,7 +231,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,sparc-softmmu"
         - TEST_CMD="make check-acceptance"
       after_failure:
         - cat tests/results/latest/job.log
-- 
2.19.1

Re: [Qemu-devel] [PATCH 0/3] tests/acceptance: Add tests for the Leon3 board

[KONRAD Frederic]

Hi Philippe,

Thanks for that!

I'm not aware at all of the tests/acceptance/* stuff.. How can we launch those
tests?

Appart of that it looks good to me :).

Regards,
Fred

Le 6/27/19 à 1:53 PM, Philippe Mathieu-Daudé a écrit :
> Quick tests worth to avoid regressions, idea from
> https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04177.html
> "Maintainers, please tell us how to boot your machines"
> 
> Regards,
> 
> Phil.
> 
> Philippe Mathieu-Daudé (3):
>    tests/acceptance: Add test that boots the HelenOS microkernel on Leon3
>    tests/acceptance: Add test that boots Linux up to BusyBox on Leon3
>    .travis.yml: Let the avocado job run the Leon3 test
> 
>   .travis.yml                             |  2 +-
>   MAINTAINERS                             |  1 +
>   tests/acceptance/machine_sparc_leon3.py | 89 +++++++++++++++++++++++++
>   3 files changed, 91 insertions(+), 1 deletion(-)
>   create mode 100644 tests/acceptance/machine_sparc_leon3.py
> 

Re: [Qemu-devel] [PATCH 0/3] tests/acceptance: Add tests for the Leon3 board

[Philippe Mathieu-Daudé]

On 6/27/19 2:28 PM, KONRAD Frederic wrote:
> Hi Philippe,
> 
> Thanks for that!
> 
> I'm not aware at all of the tests/acceptance/* stuff.. How can we launch
> those
> tests?

$ make subdir-sparc-softmmu
$ make check-venv
$ tests/venv/bin/python -m avocado --show=app run
tests/acceptance/machine_sparc_leon3.py
JOB ID     : 12900968820fcd9ba2a03b9cfe2d060508c1d91c
JOB LOG    :
/home/phil/avocado/job-results/job-2019-06-27T14.38-1290096/job.log
 (1/2)
tests/acceptance/machine_sparc_leon3.py:Leon3Machine.test_leon3_helenos_uimage:
PASS (1.10 s)
 (2/2)
tests/acceptance/machine_sparc_leon3.py:Leon3Machine.test_leon3_linux_kernel_4_9_busybox:
PASS (3.72 s)
RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
CANCEL 0
JOB TIME   : 5.32 s

Due to a pending issue with the chardev console [*] , the Avocado
framework sometime hangs, so meanwhile I run tests in a loop and they
eventually succeed :S

[*] https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg01879.html

To get the console, and filter uimage tests:

$ tests/venv/bin/python -m avocado --show=app,console run -t
binfmt:uimage tests/acceptance/machine_sparc_leon3.py
JOB ID     : b1fb8e8a101c6a45d2f15e57a9faafee94cdf2b5
JOB LOG    :
/home/phil/avocado/job-results/job-2019-06-27T14.41-b1fb8e8/job.log
 (1/1)
tests/acceptance/machine_sparc_leon3.py:Leon3Machine.test_leon3_helenos_uimage:
 console: HelenOS bootloader, release 0.6.0 (Elastic Horse)
console: Built on 2014-12-21 20:17:42 for sparc32
console: Copyright (c) 2001-2014 HelenOS project
console: 0x4000bf20|0x4000bf20: kernel image (496640/128466 bytes)
console: 0x4002b4f2|0x4002b4f2: ns image (154195/66444 bytes)
console: 0x4003b87e|0x4003b87e: loader image (153182/66437 bytes)
console: 0x4004bc03|0x4004bc03: init image (155339/66834 bytes)
console: 0x4005c115|0x4005c115: locsrv image (162063/70267 bytes)
console: 0x4006d390|0x4006d390: rd image (152678/65889 bytes)
console: 0x4007d4f1|0x4007d4f1: vfs image (168480/73394 bytes)
console: 0x4008f3a3|0x4008f3a3: logger image (158034/68368 bytes)
console: 0x4009feb3|0x4009feb3: ext4fs image (234510/100301 bytes)
console: 0x400b8680|0x400b8680: initrd image (8388608/1668901 bytes)
console: ABMA devices:
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: <0:3000> at 0x00000000 irq 0
console: Memory size: 0 MB
console: Inflating components ... initrd ext4fs logger vfs rd locsrv
init loader ns kernel Booting the kernel ...
PASS (1.11 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
CANCEL 0
JOB TIME   : 1.48 s

The other test often timeouts due to [*]:

$ tests/venv/bin/python -m avocado --show=app,console run
tests/acceptance/machine_sparc_leon3.py
 (1/2)
tests/acceptance/machine_sparc_leon3.py:Leon3Machine.test_leon3_helenos_uimage:
PASS (1.19 s)
 (2/2)
tests/acceptance/machine_sparc_leon3.py:Leon3Machine.test_leon3_linux_kernel_4_9_busybox:
 console: PROMLIB: Sun Boot Prom Version 0 Revision 0
console: Linux version 4.9.54-00018-g62dab2c (andreas@andreas) (gcc
version 4.9.4 (Cobham Gaisler Linux 4.9 Toolchain 1.0) ) #2 Wed Oct 18
09:45:51 CEST 2017
console: bootconsole [earlyprom0] enabled
console: ARCH: LEON
console: TYPE: Leon3 System-on-a-Chip
console: Ethernet address: 00:00:7c:cc:01:45
console: CACHE: direct mapped cache, set size 1k
console: CACHE: not flushing on every context switch
console: OF stdout device is: /a::a
console: PROM: Built device tree with 8629 bytes of memory.
console: Booting Linux...
console: Built 1 zonelists in Zone order, mobility grouping on.  Total
pages: 30732
console: Kernel command line: console=ttyS0,38400 init=/sbin/init
console: PID hash table entries: 512 (order: -1, 2048 bytes)
console: Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
console: Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
console: Sorting __ex_table...
console: Memory: 120576K/123952K available (3894K kernel code, 155K
rwdata, 692K rodata, 2188K init, 129K bss, 3376K reserved, 0K
cma-reserved, 0K highmem)
console: NR_IRQS:64
console: Console: colour dummy device 80x25
console: console [ttyS0] enabled
console: console [ttyS0] enabled
console: bootconsole [earlyprom0] disabled
console: bootconsole [earlyprom0] disabled
console: clocksource: timer_cs: mask: 0xffffffffffffffff max_cycles:
0x1d854df40, max_idle_ns: 3526361616960 ns
console: Calibrating delay loop... 133.01 BogoMIPS (lpj=665088)
console: pid_max: default: 32768 minimum: 301
console: Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
console: Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
console: devtmpfs: initialized
console: clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
max_idle_ns: 19112604462750000 ns
console: futex hash table entries: 256 (order: -1, 3072 bytes)
console: NET: Registered protocol family 16
console: SCSI subsystem initialized
console: clocksource: Switched to clocksource timer_cs
console: FS-Cache: Loaded
console: CacheFiles: Loaded
console: NET: Registered protocol family 2
console: TCP established hash table entries: 1024 (order: 0, 4096 bytes)
console: TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
console: TCP: Hash tables configured (established 1024 bind 1024)
console: UDP hash table entries: 256 (order: 0, 4096 bytes)
console: UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
console: NET: Registered protocol family 1
console: RPC: Registered named UNIX socket transport module.
console: RPC: Registered udp transport module.
console: RPC: Registered tcp transport module.
console: RPC: Registered tcp NFSv4.1 backchannel transport module.
|console: workingset: timestamp_bits=30 max_order=15 bucket_order=0
console: FS-Cache: Netfs 'nfs' registered for caching
console: NFS: Registering the id_resolver key type
console: Key type id_resolver registered
console: Key type id_legacy registered
console: jitterentropy: Initialization failed with host not compliant
with requirements: 2
console: io scheduler noop registered
console: io scheduler deadline registered
console: io scheduler cfq registered (default)
console: Serial: GRLIB APBUART driver
console: ffd0fce4: ttyS0 at MMIO 0x80000100 (irq = 3, base_baud =
2500000) is a GRLIB/APBUART
console: grlib-apbuart at 0x80000100, irq 3
console: brd: module loaded
console: loop: module loaded
console: NET: Registered protocol family 10
console: sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
console: NET: Registered protocol family 17
console: Key type dns_resolver registered
console: leon: power management initialized
console: Freeing unused kernel memory: 2188K
console: This architecture does not have kernel memory protection.
/console: Starting lograndom: fast init done
console: ging: OK
-console: Initializing random number generator... done.
console: Starting network: OK
console: Welcome to Buildroot
INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout
reached\nOriginal status: ERROR\n{'name':
'2-tests/acceptance/machine_sparc_leon3.py:Leon3Machine.test_leon3_linux_kernel_4_9_busybox',
'logdir': '/home/phil/avocado/job-results/job-2019-06-27T14.43... (60.44 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 1 |
CANCEL 0
JOB TIME   : 62.10 s

I guess Cleber is working on a fix, but it is unrelated to this series.

> Appart of that it looks good to me :).

Thanks!

> Regards,
> Fred
> 
> Le 6/27/19 à 1:53 PM, Philippe Mathieu-Daudé a écrit :
>> Quick tests worth to avoid regressions, idea from
>> https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04177.html
>> "Maintainers, please tell us how to boot your machines"
>>
>> Regards,
>>
>> Phil.
>>
>> Philippe Mathieu-Daudé (3):
>>    tests/acceptance: Add test that boots the HelenOS microkernel on Leon3
>>    tests/acceptance: Add test that boots Linux up to BusyBox on Leon3
>>    .travis.yml: Let the avocado job run the Leon3 test
>>
>>   .travis.yml                             |  2 +-
>>   MAINTAINERS                             |  1 +
>>   tests/acceptance/machine_sparc_leon3.py | 89 +++++++++++++++++++++++++
>>   3 files changed, 91 insertions(+), 1 deletion(-)
>>   create mode 100644 tests/acceptance/machine_sparc_leon3.py
>>

Re: [PATCH 0/3] tests/acceptance: Add tests for the Leon3 board

[Philippe Mathieu-Daudé]

ping?

On 6/27/19 1:53 PM, Philippe Mathieu-Daudé wrote:
> Quick tests worth to avoid regressions, idea from
> https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04177.html
> "Maintainers, please tell us how to boot your machines"
> 
> Regards,
> 
> Phil.
> 
> Philippe Mathieu-Daudé (3):
>    tests/acceptance: Add test that boots the HelenOS microkernel on Leon3
>    tests/acceptance: Add test that boots Linux up to BusyBox on Leon3
>    .travis.yml: Let the avocado job run the Leon3 test
> 
>   .travis.yml                             |  2 +-
>   MAINTAINERS                             |  1 +
>   tests/acceptance/machine_sparc_leon3.py | 89 +++++++++++++++++++++++++
>   3 files changed, 91 insertions(+), 1 deletion(-)
>   create mode 100644 tests/acceptance/machine_sparc_leon3.py
>