[Qemu-devel] [PATCH 0/2] tests/acceptance: Add test of NeXTcube framebuffer using OCR

[Qemu-devel] [PATCH 0/2] tests/acceptance: Add test of NeXTcube framebuffer using OCR

[Philippe Mathieu-Daudé]

Hi,

I was looking at Thomas' last series [*] where he adds the
NeXTcube machine, thinking about enforcing a new rule "new
machines must have tests". Then I realized the UART is not
yet implemented, so our current sample tests are not helpful.

Since the framebuffer is working, I gave a try at dumping the
screen content via the HMP 'screendump' command, then parsing
the screenshot with an OCR tool.

The default ROM dump the bootlog to a console. Using the old
good tesseract tool we can recover some useful words to be
sure the guest is sane, its framebuffer is definitively working.

This test takes less than 6s on Travis-CI:
https://travis-ci.org/philmd/qemu/builds/552174983#L1836

   AVOCADO tests/acceptance
 (3/9) /home/travis/build/philmd/qemu/tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer:  PASS (5.69 s)

Regards,

Phil.

Based-on: 20190628181536.13729-1-huth@tuxfamily.org
[*] "m68k: Add basic support for the NeXTcube machine"
https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg06393.html

Philippe Mathieu-Daudé (2):
  tests/acceptance: Add test of NeXTcube framebuffer using OCR
  .travis.yml: Let the avocado job run the NeXTcube tests

 .travis.yml                               |  5 ++-
 tests/acceptance/machine_m68k_nextcube.py | 50 +++++++++++++++++++++++
 2 files changed, 54 insertions(+), 1 deletion(-)
 create mode 100644 tests/acceptance/machine_m68k_nextcube.py

-- 
2.19.1

[Qemu-devel] [PATCH 1/2] tests/acceptance: Add test of NeXTcube framebuffer using OCR

[Philippe Mathieu-Daudé]

Add a test of the NeXTcube framebuffer using the Tesseract OCR
engine on a screenshot of the framebuffer device.

The test is very quick:

  $ avocado --show=app,ocr run tests/acceptance/machine_m68k_nextcube.py
  JOB ID     : f7d3c27976047036dc568183baf64c04863d9985
  JOB LOG    : ~/avocado/job-results/job-2019-06-29T16.18-f7d3c27/job.log
  (1/1) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer: |ocr:
  ue r pun Honl'flx ; 5‘ 55‘
  avg ncaaaaa 25 MHZ, memary jag m
  Backplane slat «a
  Ethernet address a a r a r3 2
  Memgry sackets aea canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled
  Memgry sackets a and 1 canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled
  [...]
  Yestlnq the rpu, 5::
  system test raneg Errar egge 51
  Egg: cammand
  Default pggc devlce nut fauna
  NEXY>I
  PASS (3.59 s)
  RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 3.97 s

Documentation on how to install tesseract:
  https://github.com/tesseract-ocr/tesseract/wiki#installation

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/machine_m68k_nextcube.py | 50 +++++++++++++++++++++++
 1 file changed, 50 insertions(+)
 create mode 100644 tests/acceptance/machine_m68k_nextcube.py

diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
new file mode 100644
index 0000000000..cf061292a7
--- /dev/null
+++ b/tests/acceptance/machine_m68k_nextcube.py
@@ -0,0 +1,50 @@
+# Functional test that boots a VM and run OCR on the framebuffer
+#
+# Copyright (c) Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import logging
+import time
+import distutils.spawn
+
+from avocado import skipUnless
+from avocado_qemu import Test
+from avocado.utils import process
+
+
+class NextCubeMachine(Test):
+
+    timeout = 15
+
+    @skipUnless(distutils.spawn.find_executable('tesseract'),
+                'tesseract OCR tool not installed')
+    def test_bootrom_framebuffer(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:next-cube
+        :avocado: tags=device:framebuffer
+        """
+        rom_url = ('http://www.nextcomputers.org/NeXTfiles/Software/ROM_Files/'
+                   '68040_Non-Turbo_Chipset/Rev_2.5_v66.BIN')
+        rom_hash = 'b3534796abae238a0111299fc406a9349f7fee24'
+        rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
+
+        self.vm.set_machine('next-cube')
+        self.vm.add_args('-bios', rom_path)
+        self.vm.launch()
+
+        self.log.info('VM launched, waiting for display')
+        # FIXME how to catch the 'displaysurface_create 1120x832' trace-event?
+        time.sleep(2)
+
+        screenshot_path = self.workdir + "dump"
+        self.vm.command('human-monitor-command',
+                        command_line='screendump %s' % screenshot_path)
+
+        console_logger = logging.getLogger('ocr')
+        proc = process.run("tesseract %s stdout" % screenshot_path)
+        console_logger.debug(proc.stdout_text)
+        self.assertIn('Backplane', proc.stdout_text)
+        self.assertIn('Ethernet address', proc.stdout_text)
-- 
2.19.1

[Qemu-devel] [PATCH 2/2] .travis.yml: Let the avocado job run the NeXTcube tests

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .travis.yml | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index aeb9b211cd..16907b5a78 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -231,7 +231,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu"
         - TEST_CMD="make check-acceptance"
       after_failure:
         - cat tests/results/latest/job.log
@@ -240,6 +240,9 @@ matrix:
           packages:
             - python3-pip
             - python3.5-venv
+            - tesseract-ocr
+
+
     # Using newer GCC with sanitizers
     - addons:
         apt:
-- 
2.19.1

Re: [Qemu-devel] [PATCH 1/2] tests/acceptance: Add test of NeXTcube framebuffer using OCR

[Thomas Huth]

On 29/06/2019 17.00, Philippe Mathieu-Daudé wrote:
> Add a test of the NeXTcube framebuffer using the Tesseract OCR
> engine on a screenshot of the framebuffer device.

Wow, that's a funny idea, I like it!

> The test is very quick:
> 
>   $ avocado --show=app,ocr run tests/acceptance/machine_m68k_nextcube.py
>   JOB ID     : f7d3c27976047036dc568183baf64c04863d9985
>   JOB LOG    : ~/avocado/job-results/job-2019-06-29T16.18-f7d3c27/job.log
>   (1/1) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer: |ocr:
>   ue r pun Honl'flx ; 5‘ 55‘
>   avg ncaaaaa 25 MHZ, memary jag m
>   Backplane slat «a
>   Ethernet address a a r a r3 2
>   Memgry sackets aea canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled
>   Memgry sackets a and 1 canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled>   [...]
>   Yestlnq the rpu, 5::
>   system test raneg Errar egge 51
>   Egg: cammand
>   Default pggc devlce nut fauna
>   NEXY>I

Hmm, the quality of the text is rather bad...

> Documentation on how to install tesseract:
>   https://github.com/tesseract-ocr/tesseract/wiki#installation

... according to that documentation, you're also supposed to install a
language pack ... have you tried whether it gets better with
tesseract-ocr-eng ?

 Thomas

Re: [Qemu-devel] [PATCH 1/2] tests/acceptance: Add test of NeXTcube framebuffer using OCR

[Philippe Mathieu-Daudé]

On 6/30/19 9:08 PM, Thomas Huth wrote:
> On 29/06/2019 17.00, Philippe Mathieu-Daudé wrote:
>> Add a test of the NeXTcube framebuffer using the Tesseract OCR
>> engine on a screenshot of the framebuffer device.
> 
> Wow, that's a funny idea, I like it!
> 
>> The test is very quick:
>>
>>   $ avocado --show=app,ocr run tests/acceptance/machine_m68k_nextcube.py
>>   JOB ID     : f7d3c27976047036dc568183baf64c04863d9985
>>   JOB LOG    : ~/avocado/job-results/job-2019-06-29T16.18-f7d3c27/job.log
>>   (1/1) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer: |ocr:
>>   ue r pun Honl'flx ; 5‘ 55‘
>>   avg ncaaaaa 25 MHZ, memary jag m
>>   Backplane slat «a
>>   Ethernet address a a r a r3 2
>>   Memgry sackets aea canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled
>>   Memgry sackets a and 1 canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled>   [...]
>>   Yestlnq the rpu, 5::
>>   system test raneg Errar egge 51
>>   Egg: cammand
>>   Default pggc devlce nut fauna
>>   NEXY>I
> 
> Hmm, the quality of the text is rather bad...
> 
>> Documentation on how to install tesseract:
>>   https://github.com/tesseract-ocr/tesseract/wiki#installation
> 
> ... according to that documentation, you're also supposed to install a
> language pack ... have you tried whether it gets better with
> tesseract-ocr-eng ?

Oops, I should had RTFM...

$ docker run -it -v /tmp:/tmp ubuntu:18.10
root@ed15d4d1ff4b:/# apt update && apt install -y tesseract-ocr-eng
[...]
root@ed15d4d1ff4b:/# tesseract /tmp/screenshot.pbm stdout
Warning. Invalid resolution 0 dpi. Using 70 instead.
Estimating resolution as 109
CPU HCesasa 25 Wz, memory 168 nS
Backplane slot #0

Ethernet address: 0:0:f:0:13:2

Nemory sockets 8-3 configured for 16MB parity page mode SIMMs but have
16MB page mode SIMMs installed
Nemory sockets @ and 1 configured for 16MB parity page mode SINMs but
have 16MB page mode SIMMs installed
Nemory sockets 4-7 configured for 16MB parity page mode SIMMs but have
16MB page mode SIMMs installed
Nemory sockets 2 and 3 configured for 16MB parity page mode SIMMs but
have 16MB page mode SIMMs installed
Nemory sockets 8-11 configured for 16MB parity page mode SIMMs but have
16"B page mode SIMMs installed
Nemory sockets 4 and 5 configured for 16MB parity page mode SIMMs but
have 16MB page mode SIMs installed
Nemory sockets 42-18 configured for 16MB parity page mode SIMMs but have
16MB page mode SIMNs installed
Nemory sockets 6 and 7 configured for 16MB parity page mode SIMMs but
have 16MB page mode SIMMs installed
Nemory size 64nB

Testing the FPU, SCC

System test failed. Error code 51

Boot command
Default boat device not found
Next>

Way better!