[PATCH] qemuboot.conf: make cpus match built artifacts

[PATCH] qemuboot.conf: make cpus match built artifacts

[Martin Kelly]

Currently, the qemu CPUs for are specified as generic, but the built
artifacts are not. For example, we build x86-64 artifacts targeting
core2duo but run them in qemu with generic qemu/kvm CPUs. This causes
some packages that take advantage of the host architecture to crash
because they try to use CPU features not advertised by qemu. As an
example, Qt uses ssse3. When artifacts linked against Qt and built
targeting core2duo attempt to run on a generic qemu/kvm CPU, we get
the following crash:

Incompatible processor. This Qt build requires the following features:
     ssse3

We could fix this by making packages like Qt not take advantage of CPU
features. However, we will probably keep facing similar issues over
time, so it's better to resolve them in a more enduring way.

Fix this by making the qemu -cpu arguments match the built artifacts.

Signed-off-by: Martin Kelly <mkelly@xevo.com>
---
 meta/conf/machine/include/qemuboot-x86.inc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/conf/machine/include/qemuboot-x86.inc b/meta/conf/machine/include/qemuboot-x86.inc
index 06ac983..acd03a1 100644
--- a/meta/conf/machine/include/qemuboot-x86.inc
+++ b/meta/conf/machine/include/qemuboot-x86.inc
@@ -1,12 +1,12 @@
 # For runqemu
 IMAGE_CLASSES += "qemuboot"
 QB_SYSTEM_NAME_x86 = "qemu-system-i386"
-QB_CPU_x86 = "-cpu qemu32"
-QB_CPU_KVM_x86 = "-cpu kvm32"
+QB_CPU_x86 = "-cpu pentium2"
+QB_CPU_KVM_x86 = "-cpu pentium2"
 
 QB_SYSTEM_NAME_x86-64 = "qemu-system-x86_64"
 QB_CPU_x86-64 = "-cpu core2duo"
-QB_CPU_KVM_x86-64 = "-cpu kvm64"
+QB_CPU_KVM_x86-64 = "-cpu core2duo"
 
 QB_AUDIO_DRV = "alsa"
 QB_AUDIO_OPT = "-soundhw ac97,es1370"
-- 
2.7.4

Re: [PATCH] qemuboot.conf: make cpus match built artifacts

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 339 bytes --]

On 22 April 2017 at 01:21, Martin Kelly <mkelly@xevo.com> wrote:

> Currently, the qemu CPUs for are specified as generic, but the built
> artifacts are not. For example, we build x86-64 artifacts targeting
> core2duo but run them in qemu with generic qemu/kvm CPUs.
>

What MACHINE specifically are you referring to here?

Ross

[-- Attachment #2: Type: text/html, Size: 725 bytes --]

Re: [PATCH] qemuboot.conf: make cpus match built artifacts

[Martin Kelly]

On 04/23/2017 12:50 PM, Burton, Ross wrote:
>
> On 22 April 2017 at 01:21, Martin Kelly <mkelly@xevo.com
> <mailto:mkelly@xevo.com>> wrote:
>
>     Currently, the qemu CPUs for are specified as generic, but the built
>     artifacts are not. For example, we build x86-64 artifacts targeting
>     core2duo but run them in qemu with generic qemu/kvm CPUs.
>
>
> What MACHINE specifically are you referring to here?
>
> Ross

This is for qemux86 and qemux86-64.

Re: [PATCH] qemuboot.conf: make cpus match built artifacts

[Richard Purdie]

On Fri, 2017-04-21 at 17:21 -0700, Martin Kelly wrote:
> Currently, the qemu CPUs for are specified as generic, but the built
> artifacts are not. For example, we build x86-64 artifacts targeting
> core2duo but run them in qemu with generic qemu/kvm CPUs. This causes
> some packages that take advantage of the host architecture to crash
> because they try to use CPU features not advertised by qemu. As an
> example, Qt uses ssse3. When artifacts linked against Qt and built
> targeting core2duo attempt to run on a generic qemu/kvm CPU, we get
> the following crash:
> 
> Incompatible processor. This Qt build requires the following
> features:
>      ssse3
> 
> We could fix this by making packages like Qt not take advantage of
> CPU
> features. However, we will probably keep facing similar issues over
> time, so it's better to resolve them in a more enduring way.
> 
> Fix this by making the qemu -cpu arguments match the built artifacts.
> 
> Signed-off-by: Martin Kelly <mkelly@xevo.com>
> ---
>  meta/conf/machine/include/qemuboot-x86.inc | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/conf/machine/include/qemuboot-x86.inc
> b/meta/conf/machine/include/qemuboot-x86.inc
> index 06ac983..acd03a1 100644
> --- a/meta/conf/machine/include/qemuboot-x86.inc
> +++ b/meta/conf/machine/include/qemuboot-x86.inc
> @@ -1,12 +1,12 @@
>  # For runqemu
>  IMAGE_CLASSES += "qemuboot"
>  QB_SYSTEM_NAME_x86 = "qemu-system-i386"
> -QB_CPU_x86 = "-cpu qemu32"
> -QB_CPU_KVM_x86 = "-cpu kvm32"
> +QB_CPU_x86 = "-cpu pentium2"
> +QB_CPU_KVM_x86 = "-cpu pentium2"
> 
>  QB_SYSTEM_NAME_x86-64 = "qemu-system-x86_64"
>  QB_CPU_x86-64 = "-cpu core2duo"
> -QB_CPU_KVM_x86-64 = "-cpu kvm64"
> +QB_CPU_KVM_x86-64 = "-cpu core2duo"
>  
>  QB_AUDIO_DRV = "alsa"
>  QB_AUDIO_OPT = "-soundhw ac97,es1370"

We've had problems here in the past:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=b8819b02dc4ff29d678cc55887dfe6c2d109a67d
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a2a20adbfd03ed9f07a3aa38ee2c9a8d962d4bf9

so we once did use core2duo and had to switch away from it after kernel
issues. Admittedly that was some time ago and I don't know if those
issues were resolved. I do remember other debates about qemu32 verses
pentium2 and other tunes as well so I'm a little nervous about changes
in this area as the issues were subtle to track down and fix,
particularly given the range of host distro kernels we need to run kvm
against :(.

Cheers,

Richard
> 

Re: [PATCH] qemuboot.conf: make cpus match built artifacts

[Aníbal Limón]

[-- Attachment #1: Type: text/plain, Size: 2325 bytes --]

Hi Martin,

It looks that we need to improve the alignment between compiler flags
and runqemu configurations. I don't know if is the right way to change
the cpu's to pentium and core2due may be could cause side-effects.

I think will be better to review the qemuboot profiles and to provide a
mechanism to modify it.

I want to know, what was your build settings?

Finally this patch needs to be send to oe-core mailing list.

Best regards,
Anibal

On 04/21/2017 07:21 PM, Martin Kelly wrote:
> Currently, the qemu CPUs for are specified as generic, but the built
> artifacts are not. For example, we build x86-64 artifacts targeting
> core2duo but run them in qemu with generic qemu/kvm CPUs. This causes
> some packages that take advantage of the host architecture to crash
> because they try to use CPU features not advertised by qemu. As an
> example, Qt uses ssse3. When artifacts linked against Qt and built
> targeting core2duo attempt to run on a generic qemu/kvm CPU, we get
> the following crash:
> 
> Incompatible processor. This Qt build requires the following features:
>      ssse3
> 
> We could fix this by making packages like Qt not take advantage of CPU
> features. However, we will probably keep facing similar issues over
> time, so it's better to resolve them in a more enduring way.
> 
> Fix this by making the qemu -cpu arguments match the built artifacts.
> 
> Signed-off-by: Martin Kelly <mkelly@xevo.com>
> ---
>  meta/conf/machine/include/qemuboot-x86.inc | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/conf/machine/include/qemuboot-x86.inc b/meta/conf/machine/include/qemuboot-x86.inc
> index 06ac983..acd03a1 100644
> --- a/meta/conf/machine/include/qemuboot-x86.inc
> +++ b/meta/conf/machine/include/qemuboot-x86.inc
> @@ -1,12 +1,12 @@
>  # For runqemu
>  IMAGE_CLASSES += "qemuboot"
>  QB_SYSTEM_NAME_x86 = "qemu-system-i386"
> -QB_CPU_x86 = "-cpu qemu32"
> -QB_CPU_KVM_x86 = "-cpu kvm32"
> +QB_CPU_x86 = "-cpu pentium2"
> +QB_CPU_KVM_x86 = "-cpu pentium2"
>  
>  QB_SYSTEM_NAME_x86-64 = "qemu-system-x86_64"
>  QB_CPU_x86-64 = "-cpu core2duo"
> -QB_CPU_KVM_x86-64 = "-cpu kvm64"
> +QB_CPU_KVM_x86-64 = "-cpu core2duo"
>  
>  QB_AUDIO_DRV = "alsa"
>  QB_AUDIO_OPT = "-soundhw ac97,es1370"
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH] qemuboot.conf: make cpus match built artifacts

[Martin Kelly]

On 04/24/2017 09:50 AM, Aníbal Limón wrote:
> Hi Martin,
>
> It looks that we need to improve the alignment between compiler flags
> and runqemu configurations. I don't know if is the right way to change
> the cpu's to pentium and core2due may be could cause side-effects.
>
> I think will be better to review the qemuboot profiles and to provide a
> mechanism to modify it.

If we go this route, it seems like we should make it modifiable by 
distro layers. Currently, the qemuboot.conf settings live in OE-core, so 
I did not find a way to override the settings in a distro file. When I 
tried manually changing them, the setting did not take effect because 
the setting in OE-core got parsed last.

That said, I think it would be more ideal to make sure the settings work 
by default everywhere, without each layer having to tweak anything. As 
long as the compiled and runtime features match, I think we can achieve 
this goal.

>
> I want to know, what was your build settings?

To test the patch, I did:

git clone git://git.yoctoproject.org/poky
source oe-init-build-env
# manually edit MACHINE to be qemux86 or qemux86-64
bitbake core-image-minimal

I hit the initial bug when building AGL (Automotive Grade Linux), which 
pulls in meta-qt5, which surfaced the issue.

>
> Finally this patch needs to be send to oe-core mailing list.
>

Sorry, adding oe-core now.

Re: [PATCH] qemuboot.conf: make cpus match built artifacts

[Martin Kelly]

On 04/24/2017 09:50 AM, Aníbal Limón wrote:
> Hi Martin,
>
> It looks that we need to improve the alignment between compiler flags
> and runqemu configurations. I don't know if is the right way to change
> the cpu's to pentium and core2due may be could cause side-effects.
>
> I think will be better to review the qemuboot profiles and to provide a
> mechanism to modify it.

If we go this route, it seems like we should make it modifiable by 
distro layers. Currently, the qemuboot.conf settings live in OE-core, so 
I did not find a way to override the settings in a distro file. When I 
tried manually changing them, the setting did not take effect because 
the setting in OE-core got parsed last.

That said, I think it would be more ideal to make sure the settings work 
by default everywhere, without each layer having to tweak anything. As 
long as the compiled and runtime features match, I think we can achieve 
this goal.

>
> I want to know, what was your build settings?

To test the patch, I did:

git clone git://git.yoctoproject.org/poky
source oe-init-build-env
# manually edit MACHINE to be qemux86 or qemux86-64
bitbake core-image-minimal

I hit the initial bug when building AGL (Automotive Grade Linux), which 
pulls in meta-qt5, which surfaced the issue.

>
> Finally this patch needs to be send to oe-core mailing list.
>

Sorry, adding oe-core now.

Re: [PATCH] qemuboot.conf: make cpus match built artifacts

[Martin Kelly]

On 04/24/2017 09:31 AM, Richard Purdie wrote:
> On Fri, 2017-04-21 at 17:21 -0700, Martin Kelly wrote:
>> Currently, the qemu CPUs for are specified as generic, but the built
>> artifacts are not. For example, we build x86-64 artifacts targeting
>> core2duo but run them in qemu with generic qemu/kvm CPUs. This causes
>> some packages that take advantage of the host architecture to crash
>> because they try to use CPU features not advertised by qemu. As an
>> example, Qt uses ssse3. When artifacts linked against Qt and built
>> targeting core2duo attempt to run on a generic qemu/kvm CPU, we get
>> the following crash:
>>
>> Incompatible processor. This Qt build requires the following
>> features:
>>      ssse3
>>
>> We could fix this by making packages like Qt not take advantage of
>> CPU
>> features. However, we will probably keep facing similar issues over
>> time, so it's better to resolve them in a more enduring way.
>>
>> Fix this by making the qemu -cpu arguments match the built artifacts.
>>
>> Signed-off-by: Martin Kelly <mkelly@xevo.com>
>> ---
>>  meta/conf/machine/include/qemuboot-x86.inc | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/conf/machine/include/qemuboot-x86.inc
>> b/meta/conf/machine/include/qemuboot-x86.inc
>> index 06ac983..acd03a1 100644
>> --- a/meta/conf/machine/include/qemuboot-x86.inc
>> +++ b/meta/conf/machine/include/qemuboot-x86.inc
>> @@ -1,12 +1,12 @@
>>  # For runqemu
>>  IMAGE_CLASSES += "qemuboot"
>>  QB_SYSTEM_NAME_x86 = "qemu-system-i386"
>> -QB_CPU_x86 = "-cpu qemu32"
>> -QB_CPU_KVM_x86 = "-cpu kvm32"
>> +QB_CPU_x86 = "-cpu pentium2"
>> +QB_CPU_KVM_x86 = "-cpu pentium2"
>>
>>  QB_SYSTEM_NAME_x86-64 = "qemu-system-x86_64"
>>  QB_CPU_x86-64 = "-cpu core2duo"
>> -QB_CPU_KVM_x86-64 = "-cpu kvm64"
>> +QB_CPU_KVM_x86-64 = "-cpu core2duo"
>>
>>  QB_AUDIO_DRV = "alsa"
>>  QB_AUDIO_OPT = "-soundhw ac97,es1370"
>
> We've had problems here in the past:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=b8819b02dc4ff29d678cc55887dfe6c2d109a67d
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a2a20adbfd03ed9f07a3aa38ee2c9a8d962d4bf9
>
> so we once did use core2duo and had to switch away from it after kernel
> issues. Admittedly that was some time ago and I don't know if those
> issues were resolved. I do remember other debates about qemu32 verses
> pentium2 and other tunes as well so I'm a little nervous about changes
> in this area as the issues were subtle to track down and fix,
> particularly given the range of host distro kernels we need to run kvm
> against :(.
>

Got it; thanks for filling me in. Please let me know if I'm missing 
something, but I looked over the bugs and I think they are different 
issues (not caused by pentium2 or core2duo settings).

Bug #5817 [1][2] appears to be an issue with passing two -cpu arguments 
instead of just one. This didn't actually cause a crash or panic AFAICT, 
but it was confusing. It was fixed by always passing in kvm/kvm64 when 
--enable-kvm is set.

Bug #3908 [3][4] is caused by an interaction between -cpu host and the 
kernel. It makes sense that -cpu host could cause issues like this 
because there are no guarantees that the host CPU and the target CPU are 
100% compatible. In this bug, some kernels worked and others did not, 
which makes sense.

These bug fixes did fix the relevant bugs, but they caused the bug I'm 
seeing because they switched to generic CPUs (which advertise very few 
CPU features) without switching the actual compile target. Thus we are 
building for i586 (pentium2) and core2duo but running with generic CPUs 
that don't know about the features we have compiled in. Any program 
attempting to use CPU features that were compiled in but not enabled at 
runtime are going to crash, and that's exactly what I saw in Qt.

Obviously, a change like this warrants good testing, but by my analysis 
it should be safe. If there's some extra testing I can do to help out, 
or if there's  let me know and I'd be happy to do it.

[1] 
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a2a20adbfd03ed9f07a3aa38ee2c9a8d962d4bf9
[2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=5817
[3] 
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=b8819b02dc4ff29d678cc55887dfe6c2d109a67d
[4] https://bugzilla.yoctoproject.org/show_bug.cgi?id=3908

Re: [poky] [PATCH] qemuboot.conf: make cpus match built artifacts

[Martin Kelly]

On 04/24/2017 09:31 AM, Richard Purdie wrote:
> On Fri, 2017-04-21 at 17:21 -0700, Martin Kelly wrote:
>> Currently, the qemu CPUs for are specified as generic, but the built
>> artifacts are not. For example, we build x86-64 artifacts targeting
>> core2duo but run them in qemu with generic qemu/kvm CPUs. This causes
>> some packages that take advantage of the host architecture to crash
>> because they try to use CPU features not advertised by qemu. As an
>> example, Qt uses ssse3. When artifacts linked against Qt and built
>> targeting core2duo attempt to run on a generic qemu/kvm CPU, we get
>> the following crash:
>>
>> Incompatible processor. This Qt build requires the following
>> features:
>>      ssse3
>>
>> We could fix this by making packages like Qt not take advantage of
>> CPU
>> features. However, we will probably keep facing similar issues over
>> time, so it's better to resolve them in a more enduring way.
>>
>> Fix this by making the qemu -cpu arguments match the built artifacts.
>>
>> Signed-off-by: Martin Kelly <mkelly@xevo.com>
>> ---
>>  meta/conf/machine/include/qemuboot-x86.inc | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/conf/machine/include/qemuboot-x86.inc
>> b/meta/conf/machine/include/qemuboot-x86.inc
>> index 06ac983..acd03a1 100644
>> --- a/meta/conf/machine/include/qemuboot-x86.inc
>> +++ b/meta/conf/machine/include/qemuboot-x86.inc
>> @@ -1,12 +1,12 @@
>>  # For runqemu
>>  IMAGE_CLASSES += "qemuboot"
>>  QB_SYSTEM_NAME_x86 = "qemu-system-i386"
>> -QB_CPU_x86 = "-cpu qemu32"
>> -QB_CPU_KVM_x86 = "-cpu kvm32"
>> +QB_CPU_x86 = "-cpu pentium2"
>> +QB_CPU_KVM_x86 = "-cpu pentium2"
>>
>>  QB_SYSTEM_NAME_x86-64 = "qemu-system-x86_64"
>>  QB_CPU_x86-64 = "-cpu core2duo"
>> -QB_CPU_KVM_x86-64 = "-cpu kvm64"
>> +QB_CPU_KVM_x86-64 = "-cpu core2duo"
>>
>>  QB_AUDIO_DRV = "alsa"
>>  QB_AUDIO_OPT = "-soundhw ac97,es1370"
>
> We've had problems here in the past:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=b8819b02dc4ff29d678cc55887dfe6c2d109a67d
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a2a20adbfd03ed9f07a3aa38ee2c9a8d962d4bf9
>
> so we once did use core2duo and had to switch away from it after kernel
> issues. Admittedly that was some time ago and I don't know if those
> issues were resolved. I do remember other debates about qemu32 verses
> pentium2 and other tunes as well so I'm a little nervous about changes
> in this area as the issues were subtle to track down and fix,
> particularly given the range of host distro kernels we need to run kvm
> against :(.
>

Got it; thanks for filling me in. Please let me know if I'm missing 
something, but I looked over the bugs and I think they are different 
issues (not caused by pentium2 or core2duo settings).

Bug #5817 [1][2] appears to be an issue with passing two -cpu arguments 
instead of just one. This didn't actually cause a crash or panic AFAICT, 
but it was confusing. It was fixed by always passing in kvm/kvm64 when 
--enable-kvm is set.

Bug #3908 [3][4] is caused by an interaction between -cpu host and the 
kernel. It makes sense that -cpu host could cause issues like this 
because there are no guarantees that the host CPU and the target CPU are 
100% compatible. In this bug, some kernels worked and others did not, 
which makes sense.

These bug fixes did fix the relevant bugs, but they caused the bug I'm 
seeing because they switched to generic CPUs (which advertise very few 
CPU features) without switching the actual compile target. Thus we are 
building for i586 (pentium2) and core2duo but running with generic CPUs 
that don't know about the features we have compiled in. Any program 
attempting to use CPU features that were compiled in but not enabled at 
runtime are going to crash, and that's exactly what I saw in Qt.

Obviously, a change like this warrants good testing, but by my analysis 
it should be safe. If there's some extra testing I can do to help out, 
or if there's  let me know and I'd be happy to do it.

[1] 
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a2a20adbfd03ed9f07a3aa38ee2c9a8d962d4bf9
[2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=5817
[3] 
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=b8819b02dc4ff29d678cc55887dfe6c2d109a67d
[4] https://bugzilla.yoctoproject.org/show_bug.cgi?id=3908