[Buildroot] [PATCH 1/2] package/glslsandbox-player: bump to version v2023.04.05

[Buildroot] [PATCH 1/2] package/glslsandbox-player: bump to version v2023.04.05

[Julien Olivain]

License hash changed due to copyright year update.

Signed-off-by: Julien Olivain <ju.o@free.fr>
---
 package/glslsandbox-player/glslsandbox-player.hash | 4 ++--
 package/glslsandbox-player/glslsandbox-player.mk   | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/package/glslsandbox-player/glslsandbox-player.hash b/package/glslsandbox-player/glslsandbox-player.hash
index bc8946c62b..91af3c9c75 100644
--- a/package/glslsandbox-player/glslsandbox-player.hash
+++ b/package/glslsandbox-player/glslsandbox-player.hash
@@ -1,3 +1,3 @@
 # Locally calculated
-sha256  b4af34889faa6d3a904e980f23aeb720dfb614d50aa01b9b1874fc2ae77dbcf4  glslsandbox-player-2021.08.24.tar.gz
-sha256  970d45d8a3dfd042d303008294f49db8c0c464f7687aa6e28a01b0926e74df85  LICENSE
+sha256  8adf89f65a060958e6ba2aa52a28f3189fde23b5478958ad4b8b6a7e82007587  glslsandbox-player-2023.04.05.tar.gz
+sha256  cda47d436d3aeb02d595ddd316d841494926d286327383b5d63734115204a8b2  LICENSE
diff --git a/package/glslsandbox-player/glslsandbox-player.mk b/package/glslsandbox-player/glslsandbox-player.mk
index 241f3475d2..0066f43567 100644
--- a/package/glslsandbox-player/glslsandbox-player.mk
+++ b/package/glslsandbox-player/glslsandbox-player.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-GLSLSANDBOX_PLAYER_VERSION = 2021.08.24
+GLSLSANDBOX_PLAYER_VERSION = 2023.04.05
 GLSLSANDBOX_PLAYER_SITE = $(call github,jolivain,glslsandbox-player,v$(GLSLSANDBOX_PLAYER_VERSION))
 GLSLSANDBOX_PLAYER_AUTORECONF = YES
 GLSLSANDBOX_PLAYER_DEPENDENCIES = libegl libgles host-pkgconf
-- 
2.40.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/2] support/testing/tests/package/test_glslsandbox_player.py: new runtime test

[Julien Olivain]

Signed-off-by: Julien Olivain <ju.o@free.fr>
---
Patch series tested on branch master at commit a4fec34 with commands:

    utils/docker-run make check-package
    ...
    0 warnings generated

    support/testing/run-tests \
        -d dl -o output_folder \
        tests.package.test_glslsandbox_player
    ...
    OK
---
 DEVELOPERS                                    |  2 +
 .../tests/package/test_glslsandbox_player.py  | 50 +++++++++++++++++++
 .../linux-vkms.fragment                       |  1 +
 3 files changed, 53 insertions(+)
 create mode 100644 support/testing/tests/package/test_glslsandbox_player.py
 create mode 100644 support/testing/tests/package/test_glslsandbox_player/linux-vkms.fragment

diff --git a/DEVELOPERS b/DEVELOPERS
index 7aa5980df7..d2730f0cef 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -1737,6 +1737,8 @@ F:	support/testing/tests/package/sample_python_pyalsa.py
 F:	support/testing/tests/package/sample_python_spake2.py
 F:	support/testing/tests/package/test_ddrescue.py
 F:	support/testing/tests/package/test_ddrescue/
+F:	support/testing/tests/package/test_glslsandbox_player.py
+F:	support/testing/tests/package/test_glslsandbox_player/
 F:	support/testing/tests/package/test_gnupg2.py
 F:	support/testing/tests/package/test_highway.py
 F:	support/testing/tests/package/test_hwloc.py
diff --git a/support/testing/tests/package/test_glslsandbox_player.py b/support/testing/tests/package/test_glslsandbox_player.py
new file mode 100644
index 0000000000..3b0dd60395
--- /dev/null
+++ b/support/testing/tests/package/test_glslsandbox_player.py
@@ -0,0 +1,50 @@
+import os
+
+import infra.basetest
+
+
+class TestGlslsandboxPlayer(infra.basetest.BRTest):
+    config = \
+        """
+        BR2_aarch64=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
+        BR2_LINUX_KERNEL=y
+        BR2_LINUX_KERNEL_CUSTOM_VERSION=y
+        BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="6.1.32"
+        BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
+        BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/aarch64-virt/linux.config"
+        BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES="{}"
+        BR2_PACKAGE_LIBDRM=y
+        BR2_PACKAGE_MESA3D=y
+        BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_SWRAST=y
+        BR2_PACKAGE_MESA3D_LLVM=y
+        BR2_PACKAGE_MESA3D_OPENGL_EGL=y
+        BR2_PACKAGE_MESA3D_OPENGL_ES=y
+        BR2_PACKAGE_GLSLSANDBOX_PLAYER=y
+        BR2_PACKAGE_GLSLSANDBOX_PLAYER_KMS=y
+        BR2_TARGET_ROOTFS_CPIO=y
+        BR2_TARGET_ROOTFS_CPIO_GZIP=y
+        # BR2_TARGET_ROOTFS_TAR is not set
+        """.format(
+            infra.filepath("tests/package/test_glslsandbox_player/linux-vkms.fragment"))
+
+    def test_run(self):
+        img = os.path.join(self.builddir, "images", "rootfs.cpio.gz")
+        kern = os.path.join(self.builddir, "images", "Image")
+        self.emulator.boot(arch="aarch64",
+                           kernel=kern,
+                           kernel_cmdline=["console=ttyAMA0"],
+                           options=["-M", "virt", "-cpu", "cortex-a57", "-m", "256M", "-initrd", img])
+        self.emulator.login()
+
+        # We force a small resolution in order to keep a relatively
+        # fast software rendering
+        cmd = "GSP_DRM_MODE=640x480 "
+        # We run 3 frames of a reduced resolution of SimpleMandel
+        cmd += "glslsandbox-player -S SimpleMandel -w0 -f3 -R8 -N -vv -r1 -D"
+        self.assertRunOk(cmd, timeout=30)
+
+        # Since we render 3 frames and request a dump of the last one,
+        # a ppm image file is expected to be present
+        self.assertRunOk("test -s SimpleMandel-00002.ppm")
diff --git a/support/testing/tests/package/test_glslsandbox_player/linux-vkms.fragment b/support/testing/tests/package/test_glslsandbox_player/linux-vkms.fragment
new file mode 100644
index 0000000000..ec2ed4460c
--- /dev/null
+++ b/support/testing/tests/package/test_glslsandbox_player/linux-vkms.fragment
@@ -0,0 +1 @@
+CONFIG_DRM_VKMS=y
-- 
2.40.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/glslsandbox-player: bump to version v2023.04.05

[Thomas Petazzoni via buildroot]

Hello Julien,

On Wed,  7 Jun 2023 22:24:38 +0200
Julien Olivain <ju.o@free.fr> wrote:

> License hash changed due to copyright year update.
> 
> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
>  package/glslsandbox-player/glslsandbox-player.hash | 4 ++--
>  package/glslsandbox-player/glslsandbox-player.mk   | 2 +-
>  2 files changed, 3 insertions(+), 3 deletions(-)

Both applied to master. Nice to see an OpenGL test case in automated
tests! I'm curious to understand more how it works:

- How does glslsandboxplayer knows that we want the frames to be
  generated as .ppm files?

- What role does the DRM vkms driver play in the story?

Thanks for this work!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/glslsandbox-player: bump to version v2023.04.05

[Julien Olivain]

Hi Thomas,

I'm glad to see there is some interest in graphical runtime testing!

On 29/07/2023 23:39, Thomas Petazzoni wrote:
> Hello Julien,
> 
> On Wed,  7 Jun 2023 22:24:38 +0200
> Julien Olivain <ju.o@free.fr> wrote:
> 
>> License hash changed due to copyright year update.
>> 
>> Signed-off-by: Julien Olivain <ju.o@free.fr>
>> ---
>>  package/glslsandbox-player/glslsandbox-player.hash | 4 ++--
>>  package/glslsandbox-player/glslsandbox-player.mk   | 2 +-
>>  2 files changed, 3 insertions(+), 3 deletions(-)
> 
> Both applied to master. Nice to see an OpenGL test case in automated
> tests! I'm curious to understand more how it works:
> 
> - How does glslsandboxplayer knows that we want the frames to be
>   generated as .ppm files?

The last frame screen dump as a .ppm is requested with the '-D'
argument in the program invocation, see [1] and [2].

I'll provide here a bit more insights, in case other people would like
to write other graphical tests ;)

glslsandbox-player was designed as a tool for stressing, benchmarking,
testing and debugging OpenGL ES 2 GPU drivers. So it includes its own
screenshot feature. It was also designed to run in reduced
environments (non-linux, static-only, C90, POSIX.1, ...). This is why
there is no fancy getopt_long() or other dependencies on modern
graphics formats libraries.

glslsandbox-player is implementing the screenshot with a good old
glReadPixel() call, then a fwrite() in [3].

Even if all of this is a bit outdated, it still continuing to catch
hardware and GPU driver software bugs in modern implementations (up to
OpenGL ES 3.2, and also in Vulkan drivers, thanks to Mesa Zink [4] or
other adaptation layers).

You could achieve the same behavior with any application having its
own builtin screenshot feature, for example with weston:

     # start compositor (with debug protocol to allow screenshots)
     weston --debug ...other-options...
     # do the screenshot by calling the client program
     weston-screenshooter

This will generate a files with the date, matching
"wayland-screenshot-*.png".


Regarding the validity of the output:

This test is "just" checking a non-empty PPM file is generated. This
is because, quoting the OpenGL ES 2.0 spec at [5] in Appendix A, "The
OpenGL ES specification is not pixel exact". I did not wanted to
introduce a test that would fail every now an then. I also wanted to
include a sufficiently "complex" workload, while remaining fast (a
Mandel fractal was a good candidate for that).

Testing for a non-empty file could be seen as a bit weak. In fact, it
implicitly relies on the fact that glslsandbox-player validates return
codes of every library, system and OpenGL calls. It is programmed to
fail in case an unexpected code is seen. See for example [6]. Since
the PPM screenshot is made at the end of the rendering, and cleanup
routines are also checking return codes, the mere presence of this PPM
file AND a clean return code of the glslsandbox-player invocation
means a lot. It means that all the plumbing between all the involved
components returned "OK": compilation of LLVM, Mesa, libdrm, Kernel,
client application... and a simple execution of all of that also
returned "OK".

This is a peculiarity of glslsandbox-player. A "normal" OpenGL
application would not validate every single call, mainly for
performance reasons (this is why glslsandbox-player has also those
--disable-strict-{egl,gles} build time configure options).

Since OpenGL ES has some invariance rules (continue reading [6]), some
image checksums could be added in other graphic tests.

The glslsandbox-player in its testsuite some of this kind of tests,
for example [7].  I thought it was not really relevant to include
those kind of checks in this test.

Also, the DRM VKMS display driver includes CRC checksumming functions,
see [8] [9] through debugfs, which could be leveraged to validate the
Kernel display controller "view" is the same as the user space client
screenshots. For example, we can compare the CRC of the screenshot
with the CRC of the display controller driver. See below for VKMS
details.


> - What role does the DRM vkms driver play in the story?

To run correctly, OpenGL ES requires a "valid context" created by EGL
(with eglCreateContext()). The EGL API always requires a "valid
display" to be initialized, see [10].

There is many non-standard hacks to run offscreen without a valid
display, but they are outside the Khronos specs, and sometimes non
portable from one GPU driver version to another.

Without a display, we usually end up with eglInitialize() failing with
error EGL_BAD_DISPLAY or EGL_NOT_INITIALIZED and stop there.

With KMS/DRM, the user space client application is also controlling
its swap chain, see [11], so the application needs a DRM driver to
talk to.

For such a test to work, the tested application must discover the DRM
driver on its own (or must be configurable to use a specific DRM
driver). glslsandbox-player can do both, see [12] and [13]. Here,
since we have only one DRM vkms driver, we let the application
discover it.

To me, VKMS with Mesa llvmpipe/swrast is the best software-only
solution for the moment. It allows to run Qemu with "-display none"
while emulating some kind of display in the virtual
machine. Obviously, this limits to simple GPU workloads that could
still fit in a software llvmpipe/swrast implementation. With a Qemu
Aarch64 virt (possibly with SMP), we can still reach decent
performance for simple runtime tests.

Virtio-GPU would have been another option, but requires QEmu emulate a
display, and access an OpenGL implementation. This would put too many
requirements on the testing environment.

Finally, other GPU vendor implementation usually require to emulate
some hardware to probe their drivers. This solution can be ruled out
for now.


> Thanks for this work!
> 
> Thomas

I hope this demystifies a bit all the hidden magic behind this test.
I also hope to see other graphic runtime tests! ;)

Best regards,

Julien.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/glslsandbox-player: bump to version v2023.04.05

[Thomas Petazzoni via buildroot]

Hello,

On Sun, 30 Jul 2023 14:12:44 +0200
Julien Olivain <ju.o@free.fr> wrote:

> I hope this demystifies a bit all the hidden magic behind this test.
> I also hope to see other graphic runtime tests! ;)

Thanks a lot for all those details, very useful. Minor nit: you've put
a lot of references to link [XYZ], but you forgot the list of those
links.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/glslsandbox-player: bump to version v2023.04.05

[Julien Olivain]

Hi Thomas,

As you mentioned, links were missing. I added them. Sorry about that.

On 30/07/2023 14:12, Julien Olivain wrote:
> Hi Thomas,
> 
> I'm glad to see there is some interest in graphical runtime testing!
> 
> On 29/07/2023 23:39, Thomas Petazzoni wrote:
>> Hello Julien,
>> 
>> On Wed,  7 Jun 2023 22:24:38 +0200
>> Julien Olivain <ju.o@free.fr> wrote:
>> 
>>> License hash changed due to copyright year update.
>>> 
>>> Signed-off-by: Julien Olivain <ju.o@free.fr>
>>> ---
>>>  package/glslsandbox-player/glslsandbox-player.hash | 4 ++--
>>>  package/glslsandbox-player/glslsandbox-player.mk   | 2 +-
>>>  2 files changed, 3 insertions(+), 3 deletions(-)
>> 
>> Both applied to master. Nice to see an OpenGL test case in automated
>> tests! I'm curious to understand more how it works:
>> 
>> - How does glslsandboxplayer knows that we want the frames to be
>>   generated as .ppm files?
> 
> The last frame screen dump as a .ppm is requested with the '-D'
> argument in the program invocation, see [1] and [2].
> 
> I'll provide here a bit more insights, in case other people would like
> to write other graphical tests ;)
> 
> glslsandbox-player was designed as a tool for stressing, benchmarking,
> testing and debugging OpenGL ES 2 GPU drivers. So it includes its own
> screenshot feature. It was also designed to run in reduced
> environments (non-linux, static-only, C90, POSIX.1, ...). This is why
> there is no fancy getopt_long() or other dependencies on modern
> graphics formats libraries.
> 
> glslsandbox-player is implementing the screenshot with a good old
> glReadPixel() call, then a fwrite() in [3].
> 
> Even if all of this is a bit outdated, it still continuing to catch
> hardware and GPU driver software bugs in modern implementations (up to
> OpenGL ES 3.2, and also in Vulkan drivers, thanks to Mesa Zink [4] or
> other adaptation layers).
> 
> You could achieve the same behavior with any application having its
> own builtin screenshot feature, for example with weston:
> 
>     # start compositor (with debug protocol to allow screenshots)
>     weston --debug ...other-options...
>     # do the screenshot by calling the client program
>     weston-screenshooter
> 
> This will generate a files with the date, matching
> "wayland-screenshot-*.png".
> 
> 
> Regarding the validity of the output:
> 
> This test is "just" checking a non-empty PPM file is generated. This
> is because, quoting the OpenGL ES 2.0 spec at [5] in Appendix A, "The
> OpenGL ES specification is not pixel exact". I did not wanted to
> introduce a test that would fail every now an then. I also wanted to
> include a sufficiently "complex" workload, while remaining fast (a
> Mandel fractal was a good candidate for that).
> 
> Testing for a non-empty file could be seen as a bit weak. In fact, it
> implicitly relies on the fact that glslsandbox-player validates return
> codes of every library, system and OpenGL calls. It is programmed to
> fail in case an unexpected code is seen. See for example [6]. Since
> the PPM screenshot is made at the end of the rendering, and cleanup
> routines are also checking return codes, the mere presence of this PPM
> file AND a clean return code of the glslsandbox-player invocation
> means a lot. It means that all the plumbing between all the involved
> components returned "OK": compilation of LLVM, Mesa, libdrm, Kernel,
> client application... and a simple execution of all of that also
> returned "OK".
> 
> This is a peculiarity of glslsandbox-player. A "normal" OpenGL
> application would not validate every single call, mainly for
> performance reasons (this is why glslsandbox-player has also those
> --disable-strict-{egl,gles} build time configure options).
> 
> Since OpenGL ES has some invariance rules (continue reading [6]), some
> image checksums could be added in other graphic tests.
> 
> The glslsandbox-player in its testsuite some of this kind of tests,
> for example [7].  I thought it was not really relevant to include
> those kind of checks in this test.
> 
> Also, the DRM VKMS display driver includes CRC checksumming functions,
> see [8] [9] through debugfs, which could be leveraged to validate the
> Kernel display controller "view" is the same as the user space client
> screenshots. For example, we can compare the CRC of the screenshot
> with the CRC of the display controller driver. See below for VKMS
> details.
> 
> 
>> - What role does the DRM vkms driver play in the story?
> 
> To run correctly, OpenGL ES requires a "valid context" created by EGL
> (with eglCreateContext()). The EGL API always requires a "valid
> display" to be initialized, see [10].
> 
> There is many non-standard hacks to run offscreen without a valid
> display, but they are outside the Khronos specs, and sometimes non
> portable from one GPU driver version to another.
> 
> Without a display, we usually end up with eglInitialize() failing with
> error EGL_BAD_DISPLAY or EGL_NOT_INITIALIZED and stop there.
> 
> With KMS/DRM, the user space client application is also controlling
> its swap chain, see [11], so the application needs a DRM driver to
> talk to.
> 
> For such a test to work, the tested application must discover the DRM
> driver on its own (or must be configurable to use a specific DRM
> driver). glslsandbox-player can do both, see [12] and [13]. Here,
> since we have only one DRM vkms driver, we let the application
> discover it.
> 
> To me, VKMS with Mesa llvmpipe/swrast is the best software-only
> solution for the moment. It allows to run Qemu with "-display none"
> while emulating some kind of display in the virtual
> machine. Obviously, this limits to simple GPU workloads that could
> still fit in a software llvmpipe/swrast implementation. With a Qemu
> Aarch64 virt (possibly with SMP), we can still reach decent
> performance for simple runtime tests.
> 
> Virtio-GPU would have been another option, but requires QEmu emulate a
> display, and access an OpenGL implementation. This would put too many
> requirements on the testing environment.
> 
> Finally, other GPU vendor implementation usually require to emulate
> some hardware to probe their drivers. This solution can be ruled out
> for now.
> 
> 
>> Thanks for this work!
>> 
>> Thomas
> 
> I hope this demystifies a bit all the hidden magic behind this test.
> I also hope to see other graphic runtime tests! ;)
> 
> Best regards,
> 
> Julien.

[1]  
https://git.buildroot.org/buildroot/tree/support/testing/tests/package/test_glslsandbox_player.py?id=1367f3e465500129998631419cf645c0180c71a1#n45
[2]  
https://github.com/jolivain/glslsandbox-player/blob/v2023.04.05/README.md#running-glslsandbox-player
[3]  
https://github.com/jolivain/glslsandbox-player/blob/v2023.04.05/src/glslsandbox-player.c#L274
[4]  https://docs.mesa3d.org/drivers/zink.html
[5]  
https://registry.khronos.org/OpenGL/specs/es/2.0/es_full_spec_2.0.pdf#page=169
[6]  
https://github.com/jolivain/glslsandbox-player/blob/v2023.04.05/src/gles_helper.c#L292
[7]  
https://github.com/jolivain/glslsandbox-player/blob/v2023.04.05/tests/tst-textures.at#L17
[8]  
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/gpu/drm/vkms/vkms_composer.c?h=v6.4.7#n107
[9]  https://docs.kernel.org/gpu/drm-uapi.html#display-crc-support
[10] 
https://registry.khronos.org/EGL/sdk/docs/man/html/eglInitialize.xhtml
[11] 
https://github.com/jolivain/glslsandbox-player/blob/v2023.04.05/src/native_gfx_kms.c#L761
[12] 
https://github.com/jolivain/glslsandbox-player/blob/v2023.04.05/src/native_gfx_kms.c#L416
[13] 
https://github.com/jolivain/glslsandbox-player/blob/v2023.04.05/src/native_gfx_kms.c#L526

Best regards,

Julien.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot