[PATCH 0/2] Re-enable the sh4eb target

[PATCH 0/2] Re-enable the sh4eb target

[Thomas Huth]

There are binaries available for sh4 running in big endian mode,
so the assumption that there is no usable machine for sh4eb was
wrong. Thus revert the patch that disabled sh4eb and add a proper
regression test for our CI.

Thomas Huth (2):
  Revert "Remove the unused sh4eb target"
  tests/functional: Add a test for sh4eb

 configs/devices/sh4eb-softmmu/default.mak |  3 +++
 configs/targets/sh4eb-softmmu.mak         |  2 ++
 qapi/machine.json                         |  2 +-
 tests/qtest/endianness-test.c             |  1 +
 tests/qtest/machine-none-test.c           |  1 +
 .gitlab-ci.d/buildtest.yml                |  2 +-
 .gitlab-ci.d/cirrus.yml                   |  2 +-
 .gitlab-ci.d/crossbuilds.yml              |  2 +-
 .travis.yml                               |  2 +-
 tests/functional/meson.build              |  3 +++
 tests/functional/test_sh4eb_r2d.py        | 33 +++++++++++++++++++++++
 tests/qemu-iotests/testenv.py             |  1 +
 tests/qtest/meson.build                   |  1 +
 13 files changed, 50 insertions(+), 5 deletions(-)
 create mode 100644 configs/devices/sh4eb-softmmu/default.mak
 create mode 100644 configs/targets/sh4eb-softmmu.mak
 create mode 100755 tests/functional/test_sh4eb_r2d.py

-- 
2.47.0

[PATCH 1/2] Revert "Remove the unused sh4eb target"

[Thomas Huth]

This reverts commit 73ceb12960e686b763415f0880cc5171ccce01cf.

The "r2d" machine can work in big endian mode, see:

 https://lore.kernel.org/qemu-devel/d6755445-1060-48a8-82b6-2f392c21f9b9@landley.net/

So the reasoning for removing sh4eb was wrong.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 configs/devices/sh4eb-softmmu/default.mak | 3 +++
 configs/targets/sh4eb-softmmu.mak         | 2 ++
 qapi/machine.json                         | 2 +-
 tests/qtest/endianness-test.c             | 1 +
 tests/qtest/machine-none-test.c           | 1 +
 .gitlab-ci.d/buildtest.yml                | 2 +-
 .gitlab-ci.d/cirrus.yml                   | 2 +-
 .gitlab-ci.d/crossbuilds.yml              | 2 +-
 .travis.yml                               | 2 +-
 tests/qemu-iotests/testenv.py             | 1 +
 tests/qtest/meson.build                   | 1 +
 11 files changed, 14 insertions(+), 5 deletions(-)
 create mode 100644 configs/devices/sh4eb-softmmu/default.mak
 create mode 100644 configs/targets/sh4eb-softmmu.mak

diff --git a/configs/devices/sh4eb-softmmu/default.mak b/configs/devices/sh4eb-softmmu/default.mak
new file mode 100644
index 0000000000..f18d1f6519
--- /dev/null
+++ b/configs/devices/sh4eb-softmmu/default.mak
@@ -0,0 +1,3 @@
+# Default configuration for sh4eb-softmmu
+
+include ../sh4-softmmu/default.mak
diff --git a/configs/targets/sh4eb-softmmu.mak b/configs/targets/sh4eb-softmmu.mak
new file mode 100644
index 0000000000..226b1fc698
--- /dev/null
+++ b/configs/targets/sh4eb-softmmu.mak
@@ -0,0 +1,2 @@
+TARGET_ARCH=sh4
+TARGET_BIG_ENDIAN=y
diff --git a/qapi/machine.json b/qapi/machine.json
index 3cc055b6ff..a6b8795b09 100644
--- a/qapi/machine.json
+++ b/qapi/machine.json
@@ -37,7 +37,7 @@
              'loongarch64', 'm68k', 'microblaze', 'microblazeel', 'mips', 'mips64',
              'mips64el', 'mipsel', 'or1k', 'ppc',
              'ppc64', 'riscv32', 'riscv64', 'rx', 's390x', 'sh4',
-             'sparc', 'sparc64', 'tricore',
+             'sh4eb', 'sparc', 'sparc64', 'tricore',
              'x86_64', 'xtensa', 'xtensaeb' ] }
 
 ##
diff --git a/tests/qtest/endianness-test.c b/tests/qtest/endianness-test.c
index f4872b0283..222d116fae 100644
--- a/tests/qtest/endianness-test.c
+++ b/tests/qtest/endianness-test.c
@@ -41,6 +41,7 @@ static const TestCase test_cases[] = {
     { "ppc64", "pseries-2.7", 0x10080000000ULL,
       .bswap = true, .superio = "i82378" },
     { "sh4", "r2d", 0xfe240000, .superio = "i82378" },
+    { "sh4eb", "r2d", 0xfe240000, .bswap = true, .superio = "i82378" },
     { "sparc64", "sun4u", 0x1fe02000000LL, .bswap = true },
     { "x86_64", "pc", -1 },
     {}
diff --git a/tests/qtest/machine-none-test.c b/tests/qtest/machine-none-test.c
index 9cf95bea1e..159b2a705a 100644
--- a/tests/qtest/machine-none-test.c
+++ b/tests/qtest/machine-none-test.c
@@ -42,6 +42,7 @@ static struct arch2cpu cpus_map[] = {
     { "ppc64", "power8e_v2.1" },
     { "s390x", "qemu" },
     { "sh4", "sh7750r" },
+    { "sh4eb", "sh7751r" },
     { "sparc", "LEON2" },
     { "sparc64", "Fujitsu Sparc64" },
     { "tricore", "tc1796" },
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 01e8470a69..b847196c3e 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -72,7 +72,7 @@ build-system-debian:
   variables:
     IMAGE: debian
     CONFIGURE_ARGS: --with-coroutine=sigaltstack
-    TARGETS: arm-softmmu i386-softmmu riscv64-softmmu sh4-softmmu
+    TARGETS: arm-softmmu i386-softmmu riscv64-softmmu sh4eb-softmmu
       sparc-softmmu xtensa-softmmu
     MAKE_CHECK_ARGS: check-build
 
diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index 6a7bafac7b..9427302383 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -57,7 +57,7 @@ x64-freebsd-14-build:
     CIRRUS_VM_RAM: 8G
     UPDATE_COMMAND: pkg update; pkg upgrade -y
     INSTALL_COMMAND: pkg install -y
-    CONFIGURE_ARGS: --target-list-exclude=arm-softmmu,i386-softmmu,microblaze-softmmu,mips64el-softmmu,mipsel-softmmu,mips-softmmu,ppc-softmmu,xtensa-softmmu
+    CONFIGURE_ARGS: --target-list-exclude=arm-softmmu,i386-softmmu,microblaze-softmmu,mips64el-softmmu,mipsel-softmmu,mips-softmmu,ppc-softmmu,sh4eb-softmmu,xtensa-softmmu
     TEST_TARGETS: check
 
 aarch64-macos-build:
diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 178f62869d..95dfc39224 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -176,7 +176,7 @@ cross-win64-system:
     EXTRA_CONFIGURE_OPTS: --enable-fdt=internal --disable-plugins
     CROSS_SKIP_TARGETS: alpha-softmmu avr-softmmu hppa-softmmu
                         m68k-softmmu microblazeel-softmmu
-                        or1k-softmmu rx-softmmu sh4-softmmu sparc64-softmmu
+                        or1k-softmmu rx-softmmu sh4eb-softmmu sparc64-softmmu
                         tricore-softmmu xtensaeb-softmmu
   artifacts:
     when: on_success
diff --git a/.travis.yml b/.travis.yml
index ad81bc5e1b..8fc1ae0cf2 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -223,7 +223,7 @@ jobs:
           - genisoimage
       env:
         - CONFIG="--disable-containers --audio-drv-list=sdl --disable-user
-            --target-list=arm-softmmu,avr-softmmu,microblaze-softmmu,sh4-softmmu,sparc64-softmmu,xtensaeb-softmmu"
+            --target-list=arm-softmmu,avr-softmmu,microblaze-softmmu,sh4eb-softmmu,sparc64-softmmu,xtensaeb-softmmu"
 
     - name: "[s390x] GCC (user)"
       arch: s390x
diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
index 8cd620c202..6326e46b7b 100644
--- a/tests/qemu-iotests/testenv.py
+++ b/tests/qemu-iotests/testenv.py
@@ -245,6 +245,7 @@ def __init__(self, source_dir: str, build_dir: str,
             ('riscv64', 'virt'),
             ('rx', 'gdbsim-r5f562n8'),
             ('sh4', 'r2d'),
+            ('sh4eb', 'r2d'),
             ('tricore', 'tricore_testboard')
         )
         for suffix, machine in machine_map:
diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index e8be8b3942..5c131754eb 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -186,6 +186,7 @@ qtests_ppc64 = \
   qtests_pci + ['migration-test', 'cpu-plug-test', 'drive_del-test']
 
 qtests_sh4 = (config_all_devices.has_key('CONFIG_ISA_TESTDEV') ? ['endianness-test'] : [])
+qtests_sh4eb = (config_all_devices.has_key('CONFIG_ISA_TESTDEV') ? ['endianness-test'] : [])
 
 qtests_sparc = ['prom-env-test', 'm48t59-test', 'boot-serial-test'] + \
   qtests_filter
-- 
2.47.0

Re: [PATCH 1/2] Revert "Remove the unused sh4eb target"

[Daniel P. Berrangé]

On Thu, Oct 24, 2024 at 10:27:34AM +0200, Thomas Huth wrote:
> This reverts commit 73ceb12960e686b763415f0880cc5171ccce01cf.
> 
> The "r2d" machine can work in big endian mode, see:
> 
>  https://lore.kernel.org/qemu-devel/d6755445-1060-48a8-82b6-2f392c21f9b9@landley.net/
> 
> So the reasoning for removing sh4eb was wrong.

This is a good example of why bypassing the deprecation process is generally
a bad idea. Even if we believe something is unusable, we may not be aware
of the full situation. The deprecation period gives us an opportunity to
solicit more diverse information with which we can make the final deletion
decision.

> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  configs/devices/sh4eb-softmmu/default.mak | 3 +++
>  configs/targets/sh4eb-softmmu.mak         | 2 ++
>  qapi/machine.json                         | 2 +-
>  tests/qtest/endianness-test.c             | 1 +
>  tests/qtest/machine-none-test.c           | 1 +
>  .gitlab-ci.d/buildtest.yml                | 2 +-
>  .gitlab-ci.d/cirrus.yml                   | 2 +-
>  .gitlab-ci.d/crossbuilds.yml              | 2 +-
>  .travis.yml                               | 2 +-
>  tests/qemu-iotests/testenv.py             | 1 +
>  tests/qtest/meson.build                   | 1 +
>  11 files changed, 14 insertions(+), 5 deletions(-)
>  create mode 100644 configs/devices/sh4eb-softmmu/default.mak
>  create mode 100644 configs/targets/sh4eb-softmmu.mak

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 1/2] Revert "Remove the unused sh4eb target"

[Rob Landley]

On 10/24/24 03:27, Thomas Huth wrote:
> This reverts commit 73ceb12960e686b763415f0880cc5171ccce01cf.
> 
> The "r2d" machine can work in big endian mode, see:
> 
>   https://lore.kernel.org/qemu-devel/d6755445-1060-48a8-82b6-2f392c21f9b9@landley.net/
> 
> So the reasoning for removing sh4eb was wrong.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>

Signed-off-by: Rob Landley <rob@landley.net>

Rob

Re: [PATCH 1/2] Revert "Remove the unused sh4eb target"

[Thomas Huth]

On 25/10/2024 18.09, Rob Landley wrote:
> On 10/24/24 03:27, Thomas Huth wrote:
>> This reverts commit 73ceb12960e686b763415f0880cc5171ccce01cf.
>>
>> The "r2d" machine can work in big endian mode, see:
>>
>>   https://lore.kernel.org/qemu-devel/ 
>> d6755445-1060-48a8-82b6-2f392c21f9b9@landley.net/
>>
>> So the reasoning for removing sh4eb was wrong.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
> 
> Signed-off-by: Rob Landley <rob@landley.net>

I'll take that as a "Reviewed-by" ;-)

  Thanks,
   Thomas

[PATCH 2/2] tests/functional: Add a test for sh4eb

[Thomas Huth]

Now that we are aware of binaries that are available for sh4eb,
we should make sure that there are no regressions with this
target and test it regularly in our CI.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/functional/meson.build       |  3 +++
 tests/functional/test_sh4eb_r2d.py | 33 ++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
 create mode 100755 tests/functional/test_sh4eb_r2d.py

diff --git a/tests/functional/meson.build b/tests/functional/meson.build
index d4b675bf63..d642c11516 100644
--- a/tests/functional/meson.build
+++ b/tests/functional/meson.build
@@ -160,6 +160,9 @@ tests_sh4_system_thorough = [
   'sh4_tuxrun',
 ]
 
+tests_sh4eb_system_thorough = [
+  'sh4eb_r2d',
+]
 
 tests_sparc_system_thorough = [
   'sparc_sun4m',
diff --git a/tests/functional/test_sh4eb_r2d.py b/tests/functional/test_sh4eb_r2d.py
new file mode 100755
index 0000000000..d9c022c8b8
--- /dev/null
+++ b/tests/functional/test_sh4eb_r2d.py
@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+#
+# Boot a Linux kernel on a r2d sh4eb machine and check the console
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import os
+import shutil
+
+from qemu_test import LinuxKernelTest, Asset
+from qemu_test import exec_command_and_wait_for_pattern
+from qemu_test.utils import archive_extract
+from unittest import skipUnless
+
+class R2dEBTest(LinuxKernelTest):
+
+    ASSET_TGZ = Asset(
+        'https://landley.net/bin/mkroot/0.8.11/sh4eb.tgz',
+        'be8c6cb5aef8406899dc5aa5e22b6aa45840eb886cdd3ced51555c10577ada2c')
+
+    def test_sh4eb_r2d(self):
+        self.set_machine('r2d')
+        file_path = self.ASSET_TGZ.fetch()
+        archive_extract(file_path, self.workdir)
+        self.vm.add_args('-append', 'console=ttySC1 noiotrap')
+        self.launch_kernel(os.path.join(self.workdir, 'sh4eb/linux-kernel'),
+                           initrd=os.path.join(self.workdir, 'sh4eb/initramfs.cpio.gz'),
+                           console_index=1, wait_for='Type exit when done')
+        exec_command_and_wait_for_pattern(self, 'exit', 'Restarting system')
+        shutil.rmtree(os.path.join(self.workdir, 'sh4eb'))
+
+if __name__ == '__main__':
+    LinuxKernelTest.main()
-- 
2.47.0

Re: [PATCH 2/2] tests/functional: Add a test for sh4eb

[Philippe Mathieu-Daudé]

On 24/10/24 05:27, Thomas Huth wrote:
> Now that we are aware of binaries that are available for sh4eb,
> we should make sure that there are no regressions with this
> target and test it regularly in our CI.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   tests/functional/meson.build       |  3 +++
>   tests/functional/test_sh4eb_r2d.py | 33 ++++++++++++++++++++++++++++++
>   2 files changed, 36 insertions(+)
>   create mode 100755 tests/functional/test_sh4eb_r2d.py

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH 2/2] tests/functional: Add a test for sh4eb

[Rob Landley]

On 10/24/24 03:27, Thomas Huth wrote:
> Now that we are aware of binaries that are available for sh4eb,
> we should make sure that there are no regressions with this
> target and test it regularly in our CI.

Any progress on restoring this? Didn't see it in "git pull" just now...

> +class R2dEBTest(LinuxKernelTest):
> +
> +    ASSET_TGZ = Asset(
> +        'https://landley.net/bin/mkroot/0.8.11/sh4eb.tgz',
> +        'be8c6cb5aef8406899dc5aa5e22b6aa45840eb886cdd3ced51555c10577ada2c')

Feel free to pull binaries from my site, but from a reliability 
perspective "some random dude got hit by a bus so a site went down that 
broke our test infrastructure" seems a bit dodgy. (Even the Internet 
Archive has been having reliability issues of late, and "as long as 
Brewster Kahle's dot-com money holds out" seems a similar bus number.)

Building it yourself from source seems more reliable. Is there any sort 
of policy here?

I build the toolchains, kernel, and userspace entirely from source and 
can provide the reproduction sequences for those if you like.

It's part of my mkroot project, which is attempting to build small qemu 
systems for every target supported by all of:

1) qemu
2) linux
3) musl-libc
4) gcc

And even do automated smoketests on them showing it can boot, run a 
shell script, and access a virtual block device and network connection:

   https://github.com/landley/toybox/blob/master/mkroot/testroot.sh

Alas a lot of targets I would LIKE to support are missing something in 
one or more of those 4 packages. (No sparc in musl, etc.)

At the moment this lack is most obvious to me for nommu support. If I 
want fdpic then linux/fs/Kconfig.binfmt says my options are:

config BINFMT_ELF_FDPIC
         bool "Kernel support for FDPIC ELF binaries"
         default y if !BINFMT_ELF
         depends on ARM || ((M68K || RISCV || SUPERH || XTENSA) && !MMU)

But gcc's source says my options are:

$ grep -lir fdpic gcc/config | xargs dirname | uniq

   gcc/config/frv
   gcc/config/sh
   gcc/config/bfin
   gcc/config/arm

Which is exactly TWO targets that overlap. (There were two more, but 
Linux commit f5a8eb632b56 removed bfin and frv in 2018, and while I've 
followed arm-fdpic development since 2016 I have yet to get it to work 
for me.)

(NOMMU support is a bit like eukaryote cells in biology: beneath most 
people's notice but EVERYWHERE, to the point that ecosystem physically 
outweighs the "more advanced" one if you add up each instance of both.)

While you can do PIE binaries on nommu (which is what buildroot and
Buildroot and https://github.com/gregungerer/simple-linux do for the 
other nommu architectures), that just doesn't scale. If you run two PIE 
instances of "bash" on nommu it loads two copies of the bash binary into 
memory, AND each one requires one big contiguous block of memory for the 
text, rodata, bss, and data segments all together.

The point of fdpic is those four segments relocate independently, so 
multiple instances of the same program can share text and rodata, and 
also use smaller chunks of memory for bss and data which is much easier 
to come by in system without an mmu to let virtual mappings collate 
fragmented physical memory. On nommu memory fragmentation is basically 
inevitable.

Rob

P.S. In theory I could try linux's nolibc.h instead of musl but building 
toybox under that is quite the lift, and I briefly had an llvm hexagon 
build working instead of gcc (copied from your test stuff, but 
https://compiler-rt.llvm.org/ is just apocalyptically badly designed 
that the easy way to get a portable build out of it honestly seems to be 
to write a new replacement library from scratch).

Re: [PATCH 2/2] tests/functional: Add a test for sh4eb

[Thomas Huth]

On 29/10/2024 18.58, Rob Landley wrote:
> On 10/24/24 03:27, Thomas Huth wrote:
>> Now that we are aware of binaries that are available for sh4eb,
>> we should make sure that there are no regressions with this
>> target and test it regularly in our CI.
> 
> Any progress on restoring this? Didn't see it in "git pull" just now...

I'll make sure to put the patches in my next pull request before the QEMU 
soft freeze starts next week.

>> +class R2dEBTest(LinuxKernelTest):
>> +
>> +    ASSET_TGZ = Asset(
>> +        'https://landley.net/bin/mkroot/0.8.11/sh4eb.tgz',
>> +        'be8c6cb5aef8406899dc5aa5e22b6aa45840eb886cdd3ced51555c10577ada2c')
> 
> Feel free to pull binaries from my site, but from a reliability perspective 
> "some random dude got hit by a bus so a site went down that broke our test 
> infrastructure" seems a bit dodgy. (Even the Internet Archive has been 
> having reliability issues of late, and "as long as Brewster Kahle's dot-com 
> money holds out" seems a similar bus number.)
> 
> Building it yourself from source seems more reliable. Is there any sort of 
> policy here?

We don't really have any infrastructure in place for building such assets 
within the QEMU CI, but the binaries get cached locally after the initial 
download, so we should at least be able to retrieve them from these caches 
in case your original site becomes unavailable ...

> And even do automated smoketests on them showing it can boot, run a shell 
> script, and access a virtual block device and network connection:
> 
>    https://github.com/landley/toybox/blob/master/mkroot/testroot.sh

... but if you're using github anyway, maybe you could also build the 
binaries via github actions and publish the assets there? That would make it 
easier for cloning and reproducing the stuff, I think.

Anyway, thank you very much for providing the binaries already on your site, 
that's really helpful!

  Thomas