[PATCH v4 0/5] Deprecate system emulation support for 32-bit x86 and arm hosts

[PATCH v4 0/5] Deprecate system emulation support for 32-bit x86 and arm hosts

[Thomas Huth]

We're struggling quite badly with our CI minutes on the shared
gitlab runners, so we urgently need to think of ways to cut down
our supported build and target environments. Hardly anybody uses
QEMU system emulation on 32-bit x86 and arm hosts anymore, so it's
time to deprecate these environments now.

This is a follow-up patch series from the previous discussion here:

 https://lore.kernel.org/qemu-devel/20230130114428.1297295-1-thuth@redhat.com/

where people still mentioned that there is still interest in certain
support for 32-bit host hardware. But as far as I could see, there is
no real need for system emulation on 32-bit x86 host and 32-bit arm
hosts anymore, so it should be fine if we drop these host environments
soon (these are also the two architectures that contribute the most to
the long test times in our CI, so we would benefit a lot by dropping
those).

v4:
- Drop the patch that deprecated qemu-system-arm since it still
  might be required to enforce 32-bit guests with TCG
- Only deprecate system emulation on 32-bit x86 hosts since user-mode
  emulation might still be useful in certain scenarios
- Add a sentence how to enforce 32-bit mode with qemu-system-x86_64

v3:
- Update some commit descriptions according to the suggestions in v2
- Added the Reviewed-bys from v2

v2:
- Split binary and host deprecation into separate patches
- Added patches to immediately drop the jobs from the CI

Thomas Huth (5):
  docs/about/deprecated: Deprecate 32-bit x86 hosts for system emulation
  docs/about/deprecated: Deprecate the qemu-system-i386 binary
  gitlab-ci.d/crossbuilds: Drop the i386 system emulation job
  docs/about/deprecated: Deprecate 32-bit arm hosts for system emulation
  gitlab-ci.d/crossbuilds: Drop the 32-bit arm system emulation jobs

 docs/about/deprecated.rst    | 34 ++++++++++++++++++++++++++++++++++
 .gitlab-ci.d/crossbuilds.yml | 24 ------------------------
 2 files changed, 34 insertions(+), 24 deletions(-)

-- 
2.31.1

[PATCH v4 1/5] docs/about/deprecated: Deprecate 32-bit x86 hosts for system emulation

[Thomas Huth]

Hardly anybody still uses 32-bit x86 hosts today, so we should start
deprecating them to stop wasting our time and CI minutes here.
For example, there are also still some unresolved problems with these:
When emulating 64-bit binaries in user mode, TCG does not honor atomicity
for 64-bit accesses, which is "perhaps worse than not working at all"
(quoting Richard). Let's simply make it clear that people should use
64-bit x86 hosts nowadays and we do not intend to fix/maintain the old
32-bit stuff.

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 docs/about/deprecated.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
index 15084f7bea..1ca9dc33d6 100644
--- a/docs/about/deprecated.rst
+++ b/docs/about/deprecated.rst
@@ -196,6 +196,17 @@ CI coverage support may bitrot away before the deprecation process
 completes. The little endian variants of MIPS (both 32 and 64 bit) are
 still a supported host architecture.
 
+System emulation on 32-bit x86 hosts (since 8.0)
+''''''''''''''''''''''''''''''''''''''''''''''''
+
+Support for 32-bit x86 host deployments is increasingly uncommon in mainstream
+OS distributions given the widespread availability of 64-bit x86 hardware.
+The QEMU project no longer considers 32-bit x86 support for system emulation to
+be an effective use of its limited resources, and thus intends to discontinue
+it. Since all recent x86 hardware from the past >10 years is capable of the
+64-bit x86 extensions, a corresponding 64-bit OS should be used instead.
+
+
 QEMU API (QAPI) events
 ----------------------
 
-- 
2.31.1

[PATCH v4 2/5] docs/about/deprecated: Deprecate the qemu-system-i386 binary

[Thomas Huth]

Aside from not supporting KVM on 32-bit hosts, the qemu-system-x86_64
binary is a proper superset of the qemu-system-i386 binary. With the
32-bit host support being deprecated, it is now also possible to
deprecate the qemu-system-i386 binary.

With regards to 32-bit KVM support in the x86 Linux kernel,
the developers confirmed that they do not need a recent
qemu-system-i386 binary here:

 https://lore.kernel.org/kvm/Y%2ffkTs5ajFy0hP1U@google.com/

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 docs/about/deprecated.rst | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
index 1ca9dc33d6..c4fcc6b33c 100644
--- a/docs/about/deprecated.rst
+++ b/docs/about/deprecated.rst
@@ -34,6 +34,20 @@ deprecating the build option and no longer defend it in CI. The
 ``--enable-gcov`` build option remains for analysis test case
 coverage.
 
+``qemu-system-i386`` binary (since 8.0)
+'''''''''''''''''''''''''''''''''''''''
+
+The ``qemu-system-i386`` binary was mainly useful for running with KVM
+on 32-bit x86 hosts, but most Linux distributions already removed their
+support for 32-bit x86 kernels, so hardly anybody still needs this. The
+``qemu-system-x86_64`` binary is a proper superset and can be used to
+run 32-bit guests by selecting a 32-bit CPU model, including KVM support
+on x86_64 hosts. Thus users are recommended to reconfigure their systems
+to use the ``qemu-system-x86_64`` binary instead. If a 32-bit CPU guest
+environment should be enforced, you can switch off the "long mode" CPU
+flag, e.g. with ``-cpu max,lm=off``.
+
+
 System emulator command line arguments
 --------------------------------------
 
-- 
2.31.1

Re: [PATCH v4 2/5] docs/about/deprecated: Deprecate the qemu-system-i386 binary

[Daniel P. Berrangé]

On Mon, Mar 06, 2023 at 09:46:55AM +0100, Thomas Huth wrote:
> Aside from not supporting KVM on 32-bit hosts, the qemu-system-x86_64
> binary is a proper superset of the qemu-system-i386 binary. With the
> 32-bit host support being deprecated, it is now also possible to
> deprecate the qemu-system-i386 binary.
> 
> With regards to 32-bit KVM support in the x86 Linux kernel,
> the developers confirmed that they do not need a recent
> qemu-system-i386 binary here:
> 
>  https://lore.kernel.org/kvm/Y%2ffkTs5ajFy0hP1U@google.com/
> 
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  docs/about/deprecated.rst | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
> index 1ca9dc33d6..c4fcc6b33c 100644
> --- a/docs/about/deprecated.rst
> +++ b/docs/about/deprecated.rst
> @@ -34,6 +34,20 @@ deprecating the build option and no longer defend it in CI. The
>  ``--enable-gcov`` build option remains for analysis test case
>  coverage.
>  
> +``qemu-system-i386`` binary (since 8.0)
> +'''''''''''''''''''''''''''''''''''''''
> +
> +The ``qemu-system-i386`` binary was mainly useful for running with KVM
> +on 32-bit x86 hosts, but most Linux distributions already removed their
> +support for 32-bit x86 kernels, so hardly anybody still needs this. The
> +``qemu-system-x86_64`` binary is a proper superset and can be used to
> +run 32-bit guests by selecting a 32-bit CPU model, including KVM support
> +on x86_64 hosts. Thus users are recommended to reconfigure their systems
> +to use the ``qemu-system-x86_64`` binary instead. If a 32-bit CPU guest
> +environment should be enforced, you can switch off the "long mode" CPU
> +flag, e.g. with ``-cpu max,lm=off``.

I had the idea to check this today and this is not quite sufficient,
because we have code that changes the family/model/stepping for
'max' which is target dependent:

#ifdef TARGET_X86_64
    object_property_set_int(OBJECT(cpu), "family", 15, &error_abort);
    object_property_set_int(OBJECT(cpu), "model", 107, &error_abort);
    object_property_set_int(OBJECT(cpu), "stepping", 1, &error_abort);
#else
    object_property_set_int(OBJECT(cpu), "family", 6, &error_abort);
    object_property_set_int(OBJECT(cpu), "model", 6, &error_abort);
    object_property_set_int(OBJECT(cpu), "stepping", 3, &error_abort);
#endif

The former is a 64-bit AMD model and the latter is a 32-bit model.

Seems LLVM was sensitive to this distinction to some extent:

   https://gitlab.com/qemu-project/qemu/-/issues/191


A further difference is that qemy-system-i686 does not appear to enable
the 'syscall' flag, but I've not figured out where that difference is
coming from in the code.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v4 2/5] docs/about/deprecated: Deprecate the qemu-system-i386 binary

[Thomas Huth]

On 06/03/2023 10.27, Daniel P. Berrangé wrote:
> On Mon, Mar 06, 2023 at 09:46:55AM +0100, Thomas Huth wrote:
>> Aside from not supporting KVM on 32-bit hosts, the qemu-system-x86_64
>> binary is a proper superset of the qemu-system-i386 binary. With the
>> 32-bit host support being deprecated, it is now also possible to
>> deprecate the qemu-system-i386 binary.
>>
>> With regards to 32-bit KVM support in the x86 Linux kernel,
>> the developers confirmed that they do not need a recent
>> qemu-system-i386 binary here:
>>
>>   https://lore.kernel.org/kvm/Y%2ffkTs5ajFy0hP1U@google.com/
>>
>> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
>> Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   docs/about/deprecated.rst | 14 ++++++++++++++
>>   1 file changed, 14 insertions(+)
>>
>> diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
>> index 1ca9dc33d6..c4fcc6b33c 100644
>> --- a/docs/about/deprecated.rst
>> +++ b/docs/about/deprecated.rst
>> @@ -34,6 +34,20 @@ deprecating the build option and no longer defend it in CI. The
>>   ``--enable-gcov`` build option remains for analysis test case
>>   coverage.
>>   
>> +``qemu-system-i386`` binary (since 8.0)
>> +'''''''''''''''''''''''''''''''''''''''
>> +
>> +The ``qemu-system-i386`` binary was mainly useful for running with KVM
>> +on 32-bit x86 hosts, but most Linux distributions already removed their
>> +support for 32-bit x86 kernels, so hardly anybody still needs this. The
>> +``qemu-system-x86_64`` binary is a proper superset and can be used to
>> +run 32-bit guests by selecting a 32-bit CPU model, including KVM support
>> +on x86_64 hosts. Thus users are recommended to reconfigure their systems
>> +to use the ``qemu-system-x86_64`` binary instead. If a 32-bit CPU guest
>> +environment should be enforced, you can switch off the "long mode" CPU
>> +flag, e.g. with ``-cpu max,lm=off``.
> 
> I had the idea to check this today and this is not quite sufficient,
> because we have code that changes the family/model/stepping for
> 'max' which is target dependent:
> 
> #ifdef TARGET_X86_64
>      object_property_set_int(OBJECT(cpu), "family", 15, &error_abort);
>      object_property_set_int(OBJECT(cpu), "model", 107, &error_abort);
>      object_property_set_int(OBJECT(cpu), "stepping", 1, &error_abort);
> #else
>      object_property_set_int(OBJECT(cpu), "family", 6, &error_abort);
>      object_property_set_int(OBJECT(cpu), "model", 6, &error_abort);
>      object_property_set_int(OBJECT(cpu), "stepping", 3, &error_abort);
> #endif
> 
> The former is a 64-bit AMD model and the latter is a 32-bit model.
> 
> Seems LLVM was sensitive to this distinction to some extent:
> 
>     https://gitlab.com/qemu-project/qemu/-/issues/191
> 
> A further difference is that qemy-system-i686 does not appear to enable
> the 'syscall' flag, but I've not figured out where that difference is
> coming from in the code.

Ugh, ok. I gave it a quick try with a patch like this:

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -4344,15 +4344,15 @@ static void max_x86_cpu_initfn(Object *obj)
       */
      object_property_set_str(OBJECT(cpu), "vendor", CPUID_VENDOR_AMD,
                              &error_abort);
-#ifdef TARGET_X86_64
-    object_property_set_int(OBJECT(cpu), "family", 15, &error_abort);
-    object_property_set_int(OBJECT(cpu), "model", 107, &error_abort);
-    object_property_set_int(OBJECT(cpu), "stepping", 1, &error_abort);
-#else
-    object_property_set_int(OBJECT(cpu), "family", 6, &error_abort);
-    object_property_set_int(OBJECT(cpu), "model", 6, &error_abort);
-    object_property_set_int(OBJECT(cpu), "stepping", 3, &error_abort);
-#endif
+    if (object_property_get_bool(obj, "lm", &error_abort)) {
+        object_property_set_int(obj, "family", 15, &error_abort);
+        object_property_set_int(obj, "model", 107, &error_abort);
+        object_property_set_int(obj, "stepping", 1, &error_abort);
+    } else {
+        object_property_set_int(obj, "family", 6, &error_abort);
+        object_property_set_int(obj, "model", 6, &error_abort);
+        object_property_set_int(obj, "stepping", 3, &error_abort);
+    }
      object_property_set_str(OBJECT(cpu), "model-id",
                              "QEMU TCG CPU version " QEMU_HW_VERSION,
                              &error_abort);

... but it seems like the "lm" property is not initialized
there yet, so this does not work... :-/

Giving that we have soft-freeze tomorrow, let's ignore this patch
for now and revisit this topic during the 8.1 cycle. But I'll
queue the other 4 patches to get some pressure out of our CI
during the freeze time.

  Thomas

Re: [PATCH v4 2/5] docs/about/deprecated: Deprecate the qemu-system-i386 binary

[Daniel P. Berrangé]

On Mon, Mar 06, 2023 at 10:54:15AM +0100, Thomas Huth wrote:
> On 06/03/2023 10.27, Daniel P. Berrangé wrote:
> > On Mon, Mar 06, 2023 at 09:46:55AM +0100, Thomas Huth wrote:
> > > Aside from not supporting KVM on 32-bit hosts, the qemu-system-x86_64
> > > binary is a proper superset of the qemu-system-i386 binary. With the
> > > 32-bit host support being deprecated, it is now also possible to
> > > deprecate the qemu-system-i386 binary.
> > > 
> > > With regards to 32-bit KVM support in the x86 Linux kernel,
> > > the developers confirmed that they do not need a recent
> > > qemu-system-i386 binary here:
> > > 
> > >   https://lore.kernel.org/kvm/Y%2ffkTs5ajFy0hP1U@google.com/
> > > 
> > > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> > > Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
> > > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > > ---
> > >   docs/about/deprecated.rst | 14 ++++++++++++++
> > >   1 file changed, 14 insertions(+)
> > > 
> > > diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
> > > index 1ca9dc33d6..c4fcc6b33c 100644
> > > --- a/docs/about/deprecated.rst
> > > +++ b/docs/about/deprecated.rst
> > > @@ -34,6 +34,20 @@ deprecating the build option and no longer defend it in CI. The
> > >   ``--enable-gcov`` build option remains for analysis test case
> > >   coverage.
> > > +``qemu-system-i386`` binary (since 8.0)
> > > +'''''''''''''''''''''''''''''''''''''''
> > > +
> > > +The ``qemu-system-i386`` binary was mainly useful for running with KVM
> > > +on 32-bit x86 hosts, but most Linux distributions already removed their
> > > +support for 32-bit x86 kernels, so hardly anybody still needs this. The
> > > +``qemu-system-x86_64`` binary is a proper superset and can be used to
> > > +run 32-bit guests by selecting a 32-bit CPU model, including KVM support
> > > +on x86_64 hosts. Thus users are recommended to reconfigure their systems
> > > +to use the ``qemu-system-x86_64`` binary instead. If a 32-bit CPU guest
> > > +environment should be enforced, you can switch off the "long mode" CPU
> > > +flag, e.g. with ``-cpu max,lm=off``.
> > 
> > I had the idea to check this today and this is not quite sufficient,
> > because we have code that changes the family/model/stepping for
> > 'max' which is target dependent:
> > 
> > #ifdef TARGET_X86_64
> >      object_property_set_int(OBJECT(cpu), "family", 15, &error_abort);
> >      object_property_set_int(OBJECT(cpu), "model", 107, &error_abort);
> >      object_property_set_int(OBJECT(cpu), "stepping", 1, &error_abort);
> > #else
> >      object_property_set_int(OBJECT(cpu), "family", 6, &error_abort);
> >      object_property_set_int(OBJECT(cpu), "model", 6, &error_abort);
> >      object_property_set_int(OBJECT(cpu), "stepping", 3, &error_abort);
> > #endif
> > 
> > The former is a 64-bit AMD model and the latter is a 32-bit model.
> > 
> > Seems LLVM was sensitive to this distinction to some extent:
> > 
> >     https://gitlab.com/qemu-project/qemu/-/issues/191
> > 
> > A further difference is that qemy-system-i686 does not appear to enable
> > the 'syscall' flag, but I've not figured out where that difference is
> > coming from in the code.
> 
> Ugh, ok. I gave it a quick try with a patch like this:
> 
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -4344,15 +4344,15 @@ static void max_x86_cpu_initfn(Object *obj)
>       */
>      object_property_set_str(OBJECT(cpu), "vendor", CPUID_VENDOR_AMD,
>                              &error_abort);
> -#ifdef TARGET_X86_64
> -    object_property_set_int(OBJECT(cpu), "family", 15, &error_abort);
> -    object_property_set_int(OBJECT(cpu), "model", 107, &error_abort);
> -    object_property_set_int(OBJECT(cpu), "stepping", 1, &error_abort);
> -#else
> -    object_property_set_int(OBJECT(cpu), "family", 6, &error_abort);
> -    object_property_set_int(OBJECT(cpu), "model", 6, &error_abort);
> -    object_property_set_int(OBJECT(cpu), "stepping", 3, &error_abort);
> -#endif
> +    if (object_property_get_bool(obj, "lm", &error_abort)) {
> +        object_property_set_int(obj, "family", 15, &error_abort);
> +        object_property_set_int(obj, "model", 107, &error_abort);
> +        object_property_set_int(obj, "stepping", 1, &error_abort);
> +    } else {
> +        object_property_set_int(obj, "family", 6, &error_abort);
> +        object_property_set_int(obj, "model", 6, &error_abort);
> +        object_property_set_int(obj, "stepping", 3, &error_abort);
> +    }
>      object_property_set_str(OBJECT(cpu), "model-id",
>                              "QEMU TCG CPU version " QEMU_HW_VERSION,
>                              &error_abort);
> 
> ... but it seems like the "lm" property is not initialized
> there yet, so this does not work... :-/
> 
> Giving that we have soft-freeze tomorrow, let's ignore this patch
> for now and revisit this topic during the 8.1 cycle. But I'll
> queue the other 4 patches to get some pressure out of our CI
> during the freeze time.

Yep, makes sense.

More generally the whole impl of the 'max' CPU feels somewhat
questionable even for qemu-system-i386. It exposes all features
that TCG supports. A large set of these features never existed
on *any* 32-bit silicon. Hands up who has seen 32-bit silicon
with AVX2 support ? From a correctness POV we should have
capped CPU features in some manner. Given the lack of interest
in 32-bit though, we've ignored the problem and it likely does
not affect apps anyway as they're not likely to be looking for
newish features.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v4 2/5] docs/about/deprecated: Deprecate the qemu-system-i386 binary

[Thomas Huth]

On 06/03/2023 10.27, Daniel P. Berrangé wrote:
> On Mon, Mar 06, 2023 at 09:46:55AM +0100, Thomas Huth wrote:
>> [...] If a 32-bit CPU guest
>> +environment should be enforced, you can switch off the "long mode" CPU
>> +flag, e.g. with ``-cpu max,lm=off``.
> 
> I had the idea to check this today and this is not quite sufficient,
[...]
> A further difference is that qemy-system-i686 does not appear to enable
> the 'syscall' flag, but I've not figured out where that difference is
> coming from in the code.

I think I just spotted this by accident in target/i386/cpu.c
around line 637:

#ifdef TARGET_X86_64
#define TCG_EXT2_X86_64_FEATURES (CPUID_EXT2_SYSCALL | CPUID_EXT2_LM)
#else
#define TCG_EXT2_X86_64_FEATURES 0
#endif

  Thomas

Re: [PATCH v4 2/5] docs/about/deprecated: Deprecate the qemu-system-i386 binary

[Daniel P. Berrangé]

On Mon, Mar 06, 2023 at 02:48:16PM +0100, Thomas Huth wrote:
> On 06/03/2023 10.27, Daniel P. Berrangé wrote:
> > On Mon, Mar 06, 2023 at 09:46:55AM +0100, Thomas Huth wrote:
> > > [...] If a 32-bit CPU guest
> > > +environment should be enforced, you can switch off the "long mode" CPU
> > > +flag, e.g. with ``-cpu max,lm=off``.
> > 
> > I had the idea to check this today and this is not quite sufficient,
> [...]
> > A further difference is that qemy-system-i686 does not appear to enable
> > the 'syscall' flag, but I've not figured out where that difference is
> > coming from in the code.
> 
> I think I just spotted this by accident in target/i386/cpu.c
> around line 637:
> 
> #ifdef TARGET_X86_64
> #define TCG_EXT2_X86_64_FEATURES (CPUID_EXT2_SYSCALL | CPUID_EXT2_LM)
> #else
> #define TCG_EXT2_X86_64_FEATURES 0
> #endif

Hmm, so right now the difference between qemu-system-i386 and
qemu-system-x86_64 is based on compile time conditionals. So we
have the burden of building everything twice and also a burden
of testing everything twice.

If we eliminate qemu-system-i386 we get rid of our own burden,
but users/mgmt apps need to adapt to force qemu-system-x86_64
to present a 32-bit system.

What about if we had qemu-system-i386 be a hardlink to
qemu-system-x86_64, and then changed behaviour based off the
executed binary name ?

ie if running qemu-system-i386, we could present a 32-bit CPU by
default. We eliminate all of our double compilation burden still.
We still have extra testing burden, but it is in a fairly narrow
area, so does not imply x2 the testing burden just $small-percentage
extra testing ?  That would means apps/users would not need to change
at all, but we still get most of the win we're after on the
QEMU side 

Essentially #ifdef TARGET_X86_64  would be change  'if (is_64bit) {...}'
in a handful of places, with 'bool is_64bit' initialized in main() from
argv[0] ?

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v4 2/5] docs/about/deprecated: Deprecate the qemu-system-i386 binary

[Thomas Huth]

On 06/03/2023 15.06, Daniel P. Berrangé wrote:
> On Mon, Mar 06, 2023 at 02:48:16PM +0100, Thomas Huth wrote:
>> On 06/03/2023 10.27, Daniel P. Berrangé wrote:
>>> On Mon, Mar 06, 2023 at 09:46:55AM +0100, Thomas Huth wrote:
>>>> [...] If a 32-bit CPU guest
>>>> +environment should be enforced, you can switch off the "long mode" CPU
>>>> +flag, e.g. with ``-cpu max,lm=off``.
>>>
>>> I had the idea to check this today and this is not quite sufficient,
>> [...]
>>> A further difference is that qemy-system-i686 does not appear to enable
>>> the 'syscall' flag, but I've not figured out where that difference is
>>> coming from in the code.
>>
>> I think I just spotted this by accident in target/i386/cpu.c
>> around line 637:
>>
>> #ifdef TARGET_X86_64
>> #define TCG_EXT2_X86_64_FEATURES (CPUID_EXT2_SYSCALL | CPUID_EXT2_LM)
>> #else
>> #define TCG_EXT2_X86_64_FEATURES 0
>> #endif
> 
> Hmm, so right now the difference between qemu-system-i386 and
> qemu-system-x86_64 is based on compile time conditionals. So we
> have the burden of building everything twice and also a burden
> of testing everything twice.
> 
> If we eliminate qemu-system-i386 we get rid of our own burden,
> but users/mgmt apps need to adapt to force qemu-system-x86_64
> to present a 32-bit system.
> 
> What about if we had qemu-system-i386 be a hardlink to
> qemu-system-x86_64, and then changed behaviour based off the
> executed binary name ?

We could also simply provide a shell script that runs:

  qemu-system-x86_64 -cpu qemu32 $*

... that'd sounds like the simplest solution to me.

  Thomas

Re: [PATCH v4 2/5] docs/about/deprecated: Deprecate the qemu-system-i386 binary

[Daniel P. Berrangé]

On Mon, Mar 06, 2023 at 03:18:23PM +0100, Thomas Huth wrote:
> On 06/03/2023 15.06, Daniel P. Berrangé wrote:
> > On Mon, Mar 06, 2023 at 02:48:16PM +0100, Thomas Huth wrote:
> > > On 06/03/2023 10.27, Daniel P. Berrangé wrote:
> > > > On Mon, Mar 06, 2023 at 09:46:55AM +0100, Thomas Huth wrote:
> > > > > [...] If a 32-bit CPU guest
> > > > > +environment should be enforced, you can switch off the "long mode" CPU
> > > > > +flag, e.g. with ``-cpu max,lm=off``.
> > > > 
> > > > I had the idea to check this today and this is not quite sufficient,
> > > [...]
> > > > A further difference is that qemy-system-i686 does not appear to enable
> > > > the 'syscall' flag, but I've not figured out where that difference is
> > > > coming from in the code.
> > > 
> > > I think I just spotted this by accident in target/i386/cpu.c
> > > around line 637:
> > > 
> > > #ifdef TARGET_X86_64
> > > #define TCG_EXT2_X86_64_FEATURES (CPUID_EXT2_SYSCALL | CPUID_EXT2_LM)
> > > #else
> > > #define TCG_EXT2_X86_64_FEATURES 0
> > > #endif
> > 
> > Hmm, so right now the difference between qemu-system-i386 and
> > qemu-system-x86_64 is based on compile time conditionals. So we
> > have the burden of building everything twice and also a burden
> > of testing everything twice.
> > 
> > If we eliminate qemu-system-i386 we get rid of our own burden,
> > but users/mgmt apps need to adapt to force qemu-system-x86_64
> > to present a 32-bit system.
> > 
> > What about if we had qemu-system-i386 be a hardlink to
> > qemu-system-x86_64, and then changed behaviour based off the
> > executed binary name ?
> 
> We could also simply provide a shell script that runs:
> 
>  qemu-system-x86_64 -cpu qemu32 $*
> 
> ... that'd sounds like the simplest solution to me.

That woudn't do the right thing if the user ran 'qemu-system-i386 -cpu max'
because their '-cpu max' would override the -cpu arg in the shell script
that forced 32-bit mode.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v4 2/5] docs/about/deprecated: Deprecate the qemu-system-i386 binary

[Daniel P. Berrangé]

On Mon, Mar 06, 2023 at 02:25:46PM +0000, Daniel P. Berrangé wrote:
> On Mon, Mar 06, 2023 at 03:18:23PM +0100, Thomas Huth wrote:
> > On 06/03/2023 15.06, Daniel P. Berrangé wrote:
> > > On Mon, Mar 06, 2023 at 02:48:16PM +0100, Thomas Huth wrote:
> > > > On 06/03/2023 10.27, Daniel P. Berrangé wrote:
> > > > > On Mon, Mar 06, 2023 at 09:46:55AM +0100, Thomas Huth wrote:
> > > > > > [...] If a 32-bit CPU guest
> > > > > > +environment should be enforced, you can switch off the "long mode" CPU
> > > > > > +flag, e.g. with ``-cpu max,lm=off``.
> > > > > 
> > > > > I had the idea to check this today and this is not quite sufficient,
> > > > [...]
> > > > > A further difference is that qemy-system-i686 does not appear to enable
> > > > > the 'syscall' flag, but I've not figured out where that difference is
> > > > > coming from in the code.
> > > > 
> > > > I think I just spotted this by accident in target/i386/cpu.c
> > > > around line 637:
> > > > 
> > > > #ifdef TARGET_X86_64
> > > > #define TCG_EXT2_X86_64_FEATURES (CPUID_EXT2_SYSCALL | CPUID_EXT2_LM)
> > > > #else
> > > > #define TCG_EXT2_X86_64_FEATURES 0
> > > > #endif
> > > 
> > > Hmm, so right now the difference between qemu-system-i386 and
> > > qemu-system-x86_64 is based on compile time conditionals. So we
> > > have the burden of building everything twice and also a burden
> > > of testing everything twice.
> > > 
> > > If we eliminate qemu-system-i386 we get rid of our own burden,
> > > but users/mgmt apps need to adapt to force qemu-system-x86_64
> > > to present a 32-bit system.
> > > 
> > > What about if we had qemu-system-i386 be a hardlink to
> > > qemu-system-x86_64, and then changed behaviour based off the
> > > executed binary name ?
> > 
> > We could also simply provide a shell script that runs:
> > 
> >  qemu-system-x86_64 -cpu qemu32 $*
> > 
> > ... that'd sounds like the simplest solution to me.
> 
> That woudn't do the right thing if the user ran 'qemu-system-i386 -cpu max'
> because their '-cpu max' would override the -cpu arg in the shell script
> that forced 32-bit mode.

It would also fail to work with SELinux, because policy restrictions
doesn't allow for an intermediate wrapper script to exec binaries.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v4 2/5] docs/about/deprecated: Deprecate the qemu-system-i386 binary

[Philippe Mathieu-Daudé]

On 6/3/23 15:06, Daniel P. Berrangé wrote:
> On Mon, Mar 06, 2023 at 02:48:16PM +0100, Thomas Huth wrote:
>> On 06/03/2023 10.27, Daniel P. Berrangé wrote:
>>> On Mon, Mar 06, 2023 at 09:46:55AM +0100, Thomas Huth wrote:
>>>> [...] If a 32-bit CPU guest
>>>> +environment should be enforced, you can switch off the "long mode" CPU
>>>> +flag, e.g. with ``-cpu max,lm=off``.
>>>
>>> I had the idea to check this today and this is not quite sufficient,
>> [...]
>>> A further difference is that qemy-system-i686 does not appear to enable
>>> the 'syscall' flag, but I've not figured out where that difference is
>>> coming from in the code.
>>
>> I think I just spotted this by accident in target/i386/cpu.c
>> around line 637:
>>
>> #ifdef TARGET_X86_64
>> #define TCG_EXT2_X86_64_FEATURES (CPUID_EXT2_SYSCALL | CPUID_EXT2_LM)
>> #else
>> #define TCG_EXT2_X86_64_FEATURES 0
>> #endif
> 
> Hmm, so right now the difference between qemu-system-i386 and
> qemu-system-x86_64 is based on compile time conditionals. So we
> have the burden of building everything twice and also a burden
> of testing everything twice.
> 
> If we eliminate qemu-system-i386 we get rid of our own burden,
> but users/mgmt apps need to adapt to force qemu-system-x86_64
> to present a 32-bit system.
> 
> What about if we had qemu-system-i386 be a hardlink to
> qemu-system-x86_64, and then changed behaviour based off the
> executed binary name ?
> 
> ie if running qemu-system-i386, we could present a 32-bit CPU by
> default. We eliminate all of our double compilation burden still.
> We still have extra testing burden, but it is in a fairly narrow
> area, so does not imply x2 the testing burden just $small-percentage
> extra testing ?  That would means apps/users would not need to change
> at all, but we still get most of the win we're after on the
> QEMU side
> 
> Essentially #ifdef TARGET_X86_64  would be change  'if (is_64bit) {...}'
> in a handful of places, with 'bool is_64bit' initialized in main() from
> argv[0] ?

That is what Alex suggested me to do with ARM binaries as a prototype
of unifying 32/64-bit binaries, avoiding to break users scripts.

[PATCH v4 3/5] gitlab-ci.d/crossbuilds: Drop the i386 system emulation job

[Thomas Huth]

Hardly anybody still uses 32-bit x86 environments for running QEMU with
full system emulation, so let's stop wasting our scarce CI minutes with
this job.

(There are still the 32-bit MinGW and TCI jobs around for having
some compile test coverage on 32-bit)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.d/crossbuilds.yml | 10 ----------
 1 file changed, 10 deletions(-)

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index d3a31a2112..a25cb87ae4 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -43,16 +43,6 @@ cross-arm64-user:
   variables:
     IMAGE: debian-arm64-cross
 
-cross-i386-system:
-  extends:
-    - .cross_system_build_job
-    - .cross_test_artifacts
-  needs:
-    job: i386-fedora-cross-container
-  variables:
-    IMAGE: fedora-i386-cross
-    MAKE_CHECK_ARGS: check-qtest
-
 cross-i386-user:
   extends:
     - .cross_user_build_job
-- 
2.31.1

Re: [PATCH v4 3/5] gitlab-ci.d/crossbuilds: Drop the i386 system emulation job

[Philippe Mathieu-Daudé]

On 6/3/23 09:46, Thomas Huth wrote:
> Hardly anybody still uses 32-bit x86 environments for running QEMU with
> full system emulation, so let's stop wasting our scarce CI minutes with
> this job.
> 
> (There are still the 32-bit MinGW and TCI jobs around for having
> some compile test coverage on 32-bit)
> 
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .gitlab-ci.d/crossbuilds.yml | 10 ----------
>   1 file changed, 10 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

[PATCH v4 4/5] docs/about/deprecated: Deprecate 32-bit arm hosts for system emulation

[Thomas Huth]

For running QEMU in system emulation mode, the user needs a rather
strong host system, i.e. not only an embedded low-frequency controller.
All recent beefy arm host machines should support 64-bit now, it's
unlikely that anybody is still seriously using QEMU on a 32-bit arm
CPU, so we deprecate the 32-bit arm hosts here to finally save use
some time and precious CI minutes.

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 docs/about/deprecated.rst | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
index c4fcc6b33c..f0de517dc2 100644
--- a/docs/about/deprecated.rst
+++ b/docs/about/deprecated.rst
@@ -220,6 +220,15 @@ be an effective use of its limited resources, and thus intends to discontinue
 it. Since all recent x86 hardware from the past >10 years is capable of the
 64-bit x86 extensions, a corresponding 64-bit OS should be used instead.
 
+System emulation on 32-bit arm hosts (since 8.0)
+''''''''''''''''''''''''''''''''''''''''''''''''
+
+Since QEMU needs a strong host machine for running full system emulation, and
+all recent powerful arm hosts support 64-bit, the QEMU project deprecates the
+support for running any system emulation on 32-bit arm hosts in general. Use
+64-bit arm hosts for system emulation instead. (Note: "user" mode emulation
+continuous to be supported on 32-bit arm hosts, too)
+
 
 QEMU API (QAPI) events
 ----------------------
-- 
2.31.1

Re: [PATCH v4 4/5] docs/about/deprecated: Deprecate 32-bit arm hosts for system emulation

[Philippe Mathieu-Daudé]

On 6/3/23 09:46, Thomas Huth wrote:
> For running QEMU in system emulation mode, the user needs a rather
> strong host system, i.e. not only an embedded low-frequency controller.
> All recent beefy arm host machines should support 64-bit now, it's
> unlikely that anybody is still seriously using QEMU on a 32-bit arm
> CPU, so we deprecate the 32-bit arm hosts here to finally save use
> some time and precious CI minutes.
> 
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   docs/about/deprecated.rst | 9 +++++++++
>   1 file changed, 9 insertions(+)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH v4 4/5] docs/about/deprecated: Deprecate 32-bit arm hosts for system emulation

[Richard Henderson]

On 3/6/23 00:46, Thomas Huth wrote:
> +continuous to be supported on 32-bit arm hosts, too)

"continues"

Acked-by: Richard Henderson <richard.henderson@linaro.org>


r~

[PATCH v4 5/5] gitlab-ci.d/crossbuilds: Drop the 32-bit arm system emulation jobs

[Thomas Huth]

Hardly anybody still uses 32-bit arm environments for running QEMU,
so let's stop wasting our scarce CI minutes with these jobs.

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.d/crossbuilds.yml | 14 --------------
 1 file changed, 14 deletions(-)

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index a25cb87ae4..61b8ac86ee 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -1,13 +1,6 @@
 include:
   - local: '/.gitlab-ci.d/crossbuild-template.yml'
 
-cross-armel-system:
-  extends: .cross_system_build_job
-  needs:
-    job: armel-debian-cross-container
-  variables:
-    IMAGE: debian-armel-cross
-
 cross-armel-user:
   extends: .cross_user_build_job
   needs:
@@ -15,13 +8,6 @@ cross-armel-user:
   variables:
     IMAGE: debian-armel-cross
 
-cross-armhf-system:
-  extends: .cross_system_build_job
-  needs:
-    job: armhf-debian-cross-container
-  variables:
-    IMAGE: debian-armhf-cross
-
 cross-armhf-user:
   extends: .cross_user_build_job
   needs:
-- 
2.31.1