[PATCH 0/2] tests/acceptance/integratorcp: Verify Tux is displayed on framebuffer

[PATCH 0/2] tests/acceptance/integratorcp: Verify Tux is displayed on framebuffer

[Philippe Mathieu-Daudé]

Hi,

This is a simple Avocado test that use OpenCV to find the
Tux logo on the framebuffer screen dump.

It is very fast, around 4 seconds on my laptop.

Tux found by OpenCV: https://pasteboard.co/ISz7kr8.png

Regards,

Phil.

Based-on: <20200131170233.14584-1-thuth@redhat.com>
"tests/acceptance: Add a test for the integratorcp arm machine"
https://www.mail-archive.com/qemu-devel@nongnu.org/msg675828.html

Philippe Mathieu-Daudé (2):
  tests/acceptance: Extract boot_integratorcp() from test_integratorcp()
  tests/acceptance/integratorcp: Verify Tux is displayed on framebuffer

 tests/acceptance/machine_arm_integratorcp.py | 67 ++++++++++++++++++--
 1 file changed, 61 insertions(+), 6 deletions(-)

-- 
2.21.1

[PATCH 1/2] tests/acceptance: Extract boot_integratorcp() from test_integratorcp()

[Philippe Mathieu-Daudé]

As we want to re-use this code, extract it as a new function.
Since we are using the PL011 serial console, add a Avocado tag
to ease filtering of tests.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/machine_arm_integratorcp.py | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/tests/acceptance/machine_arm_integratorcp.py b/tests/acceptance/machine_arm_integratorcp.py
index 4f9ab40f2f..748819916d 100644
--- a/tests/acceptance/machine_arm_integratorcp.py
+++ b/tests/acceptance/machine_arm_integratorcp.py
@@ -19,12 +19,7 @@ class IntegratorMachine(Test):
 
     timeout = 90
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
-    def test_integratorcp(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:integratorcp
-        """
+    def boot_integratorcp(self):
         kernel_url = ('https://github.com/zayac/qemu-arm/raw/master/'
                       'arm-test/kernel/zImage.integrator')
         kernel_hash = '0d7adba893c503267c946a3cbdc63b4b54f25468'
@@ -40,4 +35,13 @@ class IntegratorMachine(Test):
                          '-initrd', initrd_path,
                          '-append', 'printk.time=0 console=ttyAMA0')
         self.vm.launch()
+
+    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    def test_integratorcp(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:integratorcp
+        :avocado: tags=device:pl011
+        """
+        self.boot_integratorcp()
         wait_for_console_pattern(self, 'Log in as root')
-- 
2.21.1

[PATCH 2/2] tests/acceptance/integratorcp: Verify Tux is displayed on framebuffer

[Philippe Mathieu-Daudé]

Add a test that verifies the Tux logo is displayed on the framebuffer.

We simply follow the OpenCV "Template Matching with Multiple Objects"
tutorial, replacing Lionel Messi by Tux:
https://docs.opencv.org/4.2.0/d4/dc6/tutorial_py_template_matching.html

When OpenCV and NumPy are installed, this test can be run using:

  $ AVOCADO_ALLOW_UNTRUSTED_CODE=hmmm \
    avocado --show=app,framebuffer run -t device:framebuffer \
      tests/acceptance/machine_arm_integratorcp.py
  JOB ID     : 8c46b0f8269242e87d738247883ea2a470df949e
  JOB LOG    : avocado/job-results/job-2020-01-31T21.38-8c46b0f/job.log
   (1/1) tests/acceptance/machine_arm_integratorcp.py:IntegratorMachine.test_framebuffer_tux_logo:
  framebuffer: found Tux at position [x, y] = (0, 0)
  PASS (3.96 s)
  RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 4.23 s

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
The resulting match is https://pasteboard.co/ISz7kr8.png

For matching SMP machines, these suggestion might work:
https://stackoverflow.com/questions/50579050/template-matching-with-multiple-objects-in-opencv-python
---
 tests/acceptance/machine_arm_integratorcp.py | 51 ++++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/tests/acceptance/machine_arm_integratorcp.py b/tests/acceptance/machine_arm_integratorcp.py
index 748819916d..a706fe1b44 100644
--- a/tests/acceptance/machine_arm_integratorcp.py
+++ b/tests/acceptance/machine_arm_integratorcp.py
@@ -9,11 +9,26 @@
 # later.  See the COPYING file in the top-level directory.
 
 import os
+import logging
 
 from avocado import skipUnless
 from avocado_qemu import Test
 from avocado_qemu import wait_for_console_pattern
 
+
+NUMPY_AVAILABLE = True
+try:
+    import numpy as np
+except ImportError:
+    NUMPY_AVAILABLE = False
+
+CV2_AVAILABLE = True
+try:
+    import cv2
+except ImportError:
+    CV2_AVAILABLE = False
+
+
 class IntegratorMachine(Test):
     """Boots the Linux kernel and checks that the console is operational"""
 
@@ -45,3 +60,39 @@ class IntegratorMachine(Test):
         """
         self.boot_integratorcp()
         wait_for_console_pattern(self, 'Log in as root')
+
+    @skipUnless(NUMPY_AVAILABLE, 'Python NumPy not installed')
+    @skipUnless(CV2_AVAILABLE, 'Python OpenCV not installed')
+    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    def test_framebuffer_tux_logo(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:integratorcp
+        :avocado: tags=device:pl110
+        :avocado: tags=device:framebuffer
+        """
+        screendump_path = os.path.join(self.workdir, "screendump.pbm")
+        tuxlogo_url = ('https://github.com/torvalds/linux/raw/v2.6.12/'
+                       'drivers/video/logo/logo_linux_vga16.ppm')
+        tuxlogo_hash = '3991c2ddbd1ddaecda7601f8aafbcf5b02dc86af'
+        tuxlogo_path = self.fetch_asset(tuxlogo_url, asset_hash=tuxlogo_hash)
+
+        self.boot_integratorcp()
+        framebuffer_ready = 'Console: switching to colour frame buffer device'
+        wait_for_console_pattern(self, framebuffer_ready)
+        self.vm.command('human-monitor-command', command_line='stop')
+        self.vm.command('human-monitor-command',
+                        command_line='screendump %s' % screendump_path)
+        logger = logging.getLogger('framebuffer')
+
+        cpu_count = 1
+        match_threshold = 0.92
+        screendump_bgr = cv2.imread(screendump_path)
+        screendump_gray = cv2.cvtColor(screendump_bgr, cv2.COLOR_BGR2GRAY)
+        result = cv2.matchTemplate(screendump_gray, cv2.imread(tuxlogo_path, 0),
+                                   cv2.TM_CCOEFF_NORMED)
+        loc = np.where(result >= match_threshold)
+        tux_count = 0
+        for tux_count, pt in enumerate(zip(*loc[::-1]), start=1):
+            logger.debug('found Tux at position [x, y] = %s', pt)
+        self.assertGreaterEqual(tux_count, cpu_count)
-- 
2.21.1

Re: [PATCH 1/2] tests/acceptance: Extract boot_integratorcp() from test_integratorcp()

[Thomas Huth]

On 31/01/2020 22.11, Philippe Mathieu-Daudé wrote:
> As we want to re-use this code, extract it as a new function.
> Since we are using the PL011 serial console, add a Avocado tag
> to ease filtering of tests.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/machine_arm_integratorcp.py | 16 ++++++++++------
>  1 file changed, 10 insertions(+), 6 deletions(-)
> 
> diff --git a/tests/acceptance/machine_arm_integratorcp.py b/tests/acceptance/machine_arm_integratorcp.py
> index 4f9ab40f2f..748819916d 100644
> --- a/tests/acceptance/machine_arm_integratorcp.py
> +++ b/tests/acceptance/machine_arm_integratorcp.py
> @@ -19,12 +19,7 @@ class IntegratorMachine(Test):
>  
>      timeout = 90
>  
> -    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
> -    def test_integratorcp(self):
> -        """
> -        :avocado: tags=arch:arm
> -        :avocado: tags=machine:integratorcp
> -        """
> +    def boot_integratorcp(self):
>          kernel_url = ('https://github.com/zayac/qemu-arm/raw/master/'
>                        'arm-test/kernel/zImage.integrator')
>          kernel_hash = '0d7adba893c503267c946a3cbdc63b4b54f25468'
> @@ -40,4 +35,13 @@ class IntegratorMachine(Test):
>                           '-initrd', initrd_path,
>                           '-append', 'printk.time=0 console=ttyAMA0')

I wonder whether you might want to move the "console=ttyAMA0" to the
test_integratorcp(), too, to get the text in the framebuffer in the
second test instead?

>          self.vm.launch()
> +
> +    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
> +    def test_integratorcp(self):
> +        """
> +        :avocado: tags=arch:arm
> +        :avocado: tags=machine:integratorcp
> +        :avocado: tags=device:pl011
> +        """
> +        self.boot_integratorcp()
>          wait_for_console_pattern(self, 'Log in as root')
> 

Anyway, patch looks fine,

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 2/2] tests/acceptance/integratorcp: Verify Tux is displayed on framebuffer

[Thomas Huth]

On 31/01/2020 22.11, Philippe Mathieu-Daudé wrote:
> Add a test that verifies the Tux logo is displayed on the framebuffer.
> 
> We simply follow the OpenCV "Template Matching with Multiple Objects"
> tutorial, replacing Lionel Messi by Tux:
> https://docs.opencv.org/4.2.0/d4/dc6/tutorial_py_template_matching.html

What a cool idea :-)

If you like, there are some more images with Tux in the QEMU advent
calendar 2018 which you could use for testing the framebuffer:

 https://www.qemu-advent-calendar.org/2018/download/day13.tar.xz
 https://www.qemu-advent-calendar.org/2018/download/day15.tar.xz
 https://www.qemu-advent-calendar.org/2018/download/day16.tar.xz
 https://www.qemu-advent-calendar.org/2018/download/day19.tar.xz

These two use a slightly different tux:

 https://www.qemu-advent-calendar.org/2018/download/day09.tar.xz
 https://www.qemu-advent-calendar.org/2018/download/day11.tar.xz

And day 22 (MIPS64) and 23 (SPARC64) use a framebuffer, too, but they
seem to be broken with the current version from the git master branch.

 Thomas

Re: [PATCH 1/2] tests/acceptance: Extract boot_integratorcp() from test_integratorcp()

[Wainer dos Santos Moschetta]

On 1/31/20 7:11 PM, Philippe Mathieu-Daudé wrote:
> As we want to re-use this code, extract it as a new function.
> Since we are using the PL011 serial console, add a Avocado tag
> to ease filtering of tests.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   tests/acceptance/machine_arm_integratorcp.py | 16 ++++++++++------
>   1 file changed, 10 insertions(+), 6 deletions(-)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/tests/acceptance/machine_arm_integratorcp.py b/tests/acceptance/machine_arm_integratorcp.py
> index 4f9ab40f2f..748819916d 100644
> --- a/tests/acceptance/machine_arm_integratorcp.py
> +++ b/tests/acceptance/machine_arm_integratorcp.py
> @@ -19,12 +19,7 @@ class IntegratorMachine(Test):
>   
>       timeout = 90
>   
> -    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
> -    def test_integratorcp(self):
> -        """
> -        :avocado: tags=arch:arm
> -        :avocado: tags=machine:integratorcp
> -        """
> +    def boot_integratorcp(self):
>           kernel_url = ('https://github.com/zayac/qemu-arm/raw/master/'
>                         'arm-test/kernel/zImage.integrator')
>           kernel_hash = '0d7adba893c503267c946a3cbdc63b4b54f25468'
> @@ -40,4 +35,13 @@ class IntegratorMachine(Test):
>                            '-initrd', initrd_path,
>                            '-append', 'printk.time=0 console=ttyAMA0')
>           self.vm.launch()
> +
> +    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
> +    def test_integratorcp(self):
> +        """
> +        :avocado: tags=arch:arm
> +        :avocado: tags=machine:integratorcp
> +        :avocado: tags=device:pl011
> +        """
> +        self.boot_integratorcp()
>           wait_for_console_pattern(self, 'Log in as root')

Re: [PATCH 2/2] tests/acceptance/integratorcp: Verify Tux is displayed on framebuffer

[Wainer dos Santos Moschetta]

On 1/31/20 7:11 PM, Philippe Mathieu-Daudé wrote:
> Add a test that verifies the Tux logo is displayed on the framebuffer.
>
> We simply follow the OpenCV "Template Matching with Multiple Objects"
> tutorial, replacing Lionel Messi by Tux:

So Tux > Messi??!! I knew that!

> https://docs.opencv.org/4.2.0/d4/dc6/tutorial_py_template_matching.html
>
> When OpenCV and NumPy are installed, this test can be run using:
>
>    $ AVOCADO_ALLOW_UNTRUSTED_CODE=hmmm \
>      avocado --show=app,framebuffer run -t device:framebuffer \
>        tests/acceptance/machine_arm_integratorcp.py
>    JOB ID     : 8c46b0f8269242e87d738247883ea2a470df949e
>    JOB LOG    : avocado/job-results/job-2020-01-31T21.38-8c46b0f/job.log
>     (1/1) tests/acceptance/machine_arm_integratorcp.py:IntegratorMachine.test_framebuffer_tux_logo:
>    framebuffer: found Tux at position [x, y] = (0, 0)
>    PASS (3.96 s)
>    RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>    JOB TIME   : 4.23 s
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> The resulting match is https://pasteboard.co/ISz7kr8.png
>
> For matching SMP machines, these suggestion might work:
> https://stackoverflow.com/questions/50579050/template-matching-with-multiple-objects-in-opencv-python
> ---
>   tests/acceptance/machine_arm_integratorcp.py | 51 ++++++++++++++++++++
>   1 file changed, 51 insertions(+)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/tests/acceptance/machine_arm_integratorcp.py b/tests/acceptance/machine_arm_integratorcp.py
> index 748819916d..a706fe1b44 100644
> --- a/tests/acceptance/machine_arm_integratorcp.py
> +++ b/tests/acceptance/machine_arm_integratorcp.py
> @@ -9,11 +9,26 @@
>   # later.  See the COPYING file in the top-level directory.
>   
>   import os
> +import logging
>   
>   from avocado import skipUnless
>   from avocado_qemu import Test
>   from avocado_qemu import wait_for_console_pattern
>   
> +
> +NUMPY_AVAILABLE = True
> +try:
> +    import numpy as np
> +except ImportError:
> +    NUMPY_AVAILABLE = False
> +
> +CV2_AVAILABLE = True
> +try:
> +    import cv2
> +except ImportError:
> +    CV2_AVAILABLE = False
> +
> +
>   class IntegratorMachine(Test):
>       """Boots the Linux kernel and checks that the console is operational"""
>   
> @@ -45,3 +60,39 @@ class IntegratorMachine(Test):
>           """
>           self.boot_integratorcp()
>           wait_for_console_pattern(self, 'Log in as root')
> +
> +    @skipUnless(NUMPY_AVAILABLE, 'Python NumPy not installed')
> +    @skipUnless(CV2_AVAILABLE, 'Python OpenCV not installed')
> +    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
> +    def test_framebuffer_tux_logo(self):
> +        """
> +        :avocado: tags=arch:arm
> +        :avocado: tags=machine:integratorcp
> +        :avocado: tags=device:pl110
> +        :avocado: tags=device:framebuffer
> +        """
> +        screendump_path = os.path.join(self.workdir, "screendump.pbm")
> +        tuxlogo_url = ('https://github.com/torvalds/linux/raw/v2.6.12/'
> +                       'drivers/video/logo/logo_linux_vga16.ppm')
> +        tuxlogo_hash = '3991c2ddbd1ddaecda7601f8aafbcf5b02dc86af'
> +        tuxlogo_path = self.fetch_asset(tuxlogo_url, asset_hash=tuxlogo_hash)
> +
> +        self.boot_integratorcp()
> +        framebuffer_ready = 'Console: switching to colour frame buffer device'
> +        wait_for_console_pattern(self, framebuffer_ready)
> +        self.vm.command('human-monitor-command', command_line='stop')
> +        self.vm.command('human-monitor-command',
> +                        command_line='screendump %s' % screendump_path)
> +        logger = logging.getLogger('framebuffer')
> +
> +        cpu_count = 1
> +        match_threshold = 0.92
> +        screendump_bgr = cv2.imread(screendump_path)
> +        screendump_gray = cv2.cvtColor(screendump_bgr, cv2.COLOR_BGR2GRAY)
> +        result = cv2.matchTemplate(screendump_gray, cv2.imread(tuxlogo_path, 0),
> +                                   cv2.TM_CCOEFF_NORMED)
> +        loc = np.where(result >= match_threshold)
> +        tux_count = 0
> +        for tux_count, pt in enumerate(zip(*loc[::-1]), start=1):
> +            logger.debug('found Tux at position [x, y] = %s', pt)
> +        self.assertGreaterEqual(tux_count, cpu_count)

Re: [PATCH 1/2] tests/acceptance: Extract boot_integratorcp() from test_integratorcp()

[Philippe Mathieu-Daudé]

On 2/1/20 8:06 AM, Thomas Huth wrote:
> On 31/01/2020 22.11, Philippe Mathieu-Daudé wrote:
>> As we want to re-use this code, extract it as a new function.
>> Since we are using the PL011 serial console, add a Avocado tag
>> to ease filtering of tests.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>   tests/acceptance/machine_arm_integratorcp.py | 16 ++++++++++------
>>   1 file changed, 10 insertions(+), 6 deletions(-)
>>
>> diff --git a/tests/acceptance/machine_arm_integratorcp.py b/tests/acceptance/machine_arm_integratorcp.py
>> index 4f9ab40f2f..748819916d 100644
>> --- a/tests/acceptance/machine_arm_integratorcp.py
>> +++ b/tests/acceptance/machine_arm_integratorcp.py
>> @@ -19,12 +19,7 @@ class IntegratorMachine(Test):
>>   
>>       timeout = 90
>>   
>> -    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
>> -    def test_integratorcp(self):
>> -        """
>> -        :avocado: tags=arch:arm
>> -        :avocado: tags=machine:integratorcp
>> -        """
>> +    def boot_integratorcp(self):
>>           kernel_url = ('https://github.com/zayac/qemu-arm/raw/master/'
>>                         'arm-test/kernel/zImage.integrator')
>>           kernel_hash = '0d7adba893c503267c946a3cbdc63b4b54f25468'
>> @@ -40,4 +35,13 @@ class IntegratorMachine(Test):
>>                            '-initrd', initrd_path,
>>                            '-append', 'printk.time=0 console=ttyAMA0')
> 
> I wonder whether you might want to move the "console=ttyAMA0" to the
> test_integratorcp(), too, to get the text in the framebuffer in the
> second test instead?

This is not a test, but a common method used by both tests, so both use it.

> 
>>           self.vm.launch()
>> +
>> +    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
>> +    def test_integratorcp(self):
>> +        """
>> +        :avocado: tags=arch:arm
>> +        :avocado: tags=machine:integratorcp
>> +        :avocado: tags=device:pl011
>> +        """
>> +        self.boot_integratorcp()
>>           wait_for_console_pattern(self, 'Log in as root')
>>
> 
> Anyway, patch looks fine,
> 
> Reviewed-by: Thomas Huth <thuth@redhat.com>

Thanks!

Peter, Thomas previous patch and these series are reviewed. Can you 
queue them in your target-arm.next tree?