qemu-system-i386: Could not install MSR_CORE_THREAD_COUNT handler: Success

qemu-system-i386: Could not install MSR_CORE_THREAD_COUNT handler: Success

[Vitaly Chikunov]

Hi,

QEMU 7.2.0 when run on 32-bit x86 architecture fails with:

  i586$ qemu-system-i386 -enable-kvm
  qemu-system-i386: Could not install MSR_CORE_THREAD_COUNT handler: Success
  i586$ qemu-system-x86_64 -enable-kvm
  qemu-system-x86_64: Could not install MSR_CORE_THREAD_COUNT handler: Success

Minimal reproducer is `qemu-system-i386 -enable-kvm'. And this only
happens on x86 (linux32 personality and binaries on x86_64 host):

  i586$ file /usr/bin/qemu-system-i386
  /usr/bin/qemu-system-i386: ELF 32-bit LSB pie executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, BuildID[sha1]=0ba1d953bcb7a691014255954f060ff404c8df90, for GNU/Linux 3.2.0, stripped
  i586$ /usr/bin/qemu-system-i386 --version
  QEMU emulator version 7.2.0 (qemu-7.2.0-alt1)
  Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

Thanks,

Re: qemu-system-i386: Could not install MSR_CORE_THREAD_COUNT handler: Success

[Alexander Graf]

Hi Vitaly,

This is a kvm kernel bug and should be fixed with the latest stable releases. Which kernel version are you running?

Thanks,

Alex


> Am 30.12.2022 um 15:30 schrieb Vitaly Chikunov <vt@altlinux.org>:
> 
> Hi,
> 
> QEMU 7.2.0 when run on 32-bit x86 architecture fails with:
> 
>  i586$ qemu-system-i386 -enable-kvm
>  qemu-system-i386: Could not install MSR_CORE_THREAD_COUNT handler: Success
>  i586$ qemu-system-x86_64 -enable-kvm
>  qemu-system-x86_64: Could not install MSR_CORE_THREAD_COUNT handler: Success
> 
> Minimal reproducer is `qemu-system-i386 -enable-kvm'. And this only
> happens on x86 (linux32 personality and binaries on x86_64 host):
> 
>  i586$ file /usr/bin/qemu-system-i386
>  /usr/bin/qemu-system-i386: ELF 32-bit LSB pie executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, BuildID[sha1]=0ba1d953bcb7a691014255954f060ff404c8df90, for GNU/Linux 3.2.0, stripped
>  i586$ /usr/bin/qemu-system-i386 --version
>  QEMU emulator version 7.2.0 (qemu-7.2.0-alt1)
>  Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
> 
> Thanks,
> 

Re: qemu-system-i386: Could not install MSR_CORE_THREAD_COUNT handler: Success

[Vitaly Chikunov]

Alexander,

On Fri, Dec 30, 2022 at 06:44:14PM +0100, Alexander Graf wrote:
> Hi Vitaly,
> 
> This is a kvm kernel bug and should be fixed with the latest stable releases. Which kernel version are you running?

This is on latest v6.0 stable - 6.0.15.

Maybe there could be workaround for such situations? (Or maybe it's
possible to make this error non-fatal?) We use qemu+kvm for testing and
now we cannot test on x86.

Thanks,


> 
> Thanks,
> 
> Alex
> 
> 
> > Am 30.12.2022 um 15:30 schrieb Vitaly Chikunov <vt@altlinux.org>:
> > 
> > Hi,
> > 
> > QEMU 7.2.0 when run on 32-bit x86 architecture fails with:
> > 
> >  i586$ qemu-system-i386 -enable-kvm
> >  qemu-system-i386: Could not install MSR_CORE_THREAD_COUNT handler: Success
> >  i586$ qemu-system-x86_64 -enable-kvm
> >  qemu-system-x86_64: Could not install MSR_CORE_THREAD_COUNT handler: Success
> > 
> > Minimal reproducer is `qemu-system-i386 -enable-kvm'. And this only
> > happens on x86 (linux32 personality and binaries on x86_64 host):
> > 
> >  i586$ file /usr/bin/qemu-system-i386
> >  /usr/bin/qemu-system-i386: ELF 32-bit LSB pie executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, BuildID[sha1]=0ba1d953bcb7a691014255954f060ff404c8df90, for GNU/Linux 3.2.0, stripped
> >  i586$ /usr/bin/qemu-system-i386 --version
> >  QEMU emulator version 7.2.0 (qemu-7.2.0-alt1)
> >  Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
> > 
> > Thanks,
> > 
> 

Re: qemu-system-i386: Could not install MSR_CORE_THREAD_COUNT handler: Success

[Alexander Graf]

Hi Vitaly,

On 30.12.22 19:16, Vitaly Chikunov wrote:
> Alexander,
>
> On Fri, Dec 30, 2022 at 06:44:14PM +0100, Alexander Graf wrote:
>> Hi Vitaly,
>>
>> This is a kvm kernel bug and should be fixed with the latest stable releases. Which kernel version are you running?
> This is on latest v6.0 stable - 6.0.15.
>
> Maybe there could be workaround for such situations? (Or maybe it's
> possible to make this error non-fatal?) We use qemu+kvm for testing and
> now we cannot test on x86.

I'm confused what's going wrong for you. I tried to reproduce the issue 
locally, but am unable to:

$ uname -a
Linux server 6.0.15-default #1 SMP PREEMPT_DYNAMIC Sat Dec 31 07:52:52 
CET 2022 x86_64 x86_64 x86_64 GNU/Linux
$ linux32 chroot .
$ uname -a
Linux server 6.0.15-default #1 SMP PREEMPT_DYNAMIC Sat Dec 31 07:52:52 
CET 2022 i686 GNU/Linux
$ cd qemu
$ file ./build/qemu-system-i386
./build/qemu-system-i386: ELF 32-bit LSB shared object, Intel 80386, 
version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, 
for GNU/Linux 3.2.0, 
BuildID[sha1]=f75e20572be5c604c121de4497397665c168aa4c, with debug_info, 
not stripped
$ ./build/qemu-system-i386 --version
QEMU emulator version 7.2.0 (v7.2.0-dirty)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
$ ./build/qemu-system-i386 -nographic -enable-kvm
SeaBIOS (version rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org)
[...]


Can you please double check whether your host kernel version is 6.0.15? 
Please paste the output of "uname -a".


Alex

Re: qemu-system-i386: Could not install MSR_CORE_THREAD_COUNT handler: Success

[Vitaly Chikunov]

Alexander,

On Sat, Dec 31, 2022 at 10:28:21AM +0100, Alexander Graf wrote:
> On 30.12.22 19:16, Vitaly Chikunov wrote:
> > On Fri, Dec 30, 2022 at 06:44:14PM +0100, Alexander Graf wrote:
> > > 
> > > This is a kvm kernel bug and should be fixed with the latest stable releases. Which kernel version are you running?
> > This is on latest v6.0 stable - 6.0.15.
> > 
> > Maybe there could be workaround for such situations? (Or maybe it's
> > possible to make this error non-fatal?) We use qemu+kvm for testing and
> > now we cannot test on x86.
> 
> I'm confused what's going wrong for you. I tried to reproduce the issue
> locally, but am unable to:
> 
> $ uname -a
> Linux server 6.0.15-default #1 SMP PREEMPT_DYNAMIC Sat Dec 31 07:52:52 CET
> 2022 x86_64 x86_64 x86_64 GNU/Linux
> $ linux32 chroot .
> $ uname -a
> Linux server 6.0.15-default #1 SMP PREEMPT_DYNAMIC Sat Dec 31 07:52:52 CET
> 2022 i686 GNU/Linux
> $ cd qemu
> $ file ./build/qemu-system-i386
> ./build/qemu-system-i386: ELF 32-bit LSB shared object, Intel 80386, version
> 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux
> 3.2.0, BuildID[sha1]=f75e20572be5c604c121de4497397665c168aa4c, with
> debug_info, not stripped
> $ ./build/qemu-system-i386 --version
> QEMU emulator version 7.2.0 (v7.2.0-dirty)
> Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
> $ ./build/qemu-system-i386 -nographic -enable-kvm
> SeaBIOS (version rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org)
> [...]
> 
> 
> Can you please double check whether your host kernel version is 6.0.15?
> Please paste the output of "uname -a".

Excuse me, I'm incorrectly reported kernel version I tried to boot instead
of host one. Host kernels are quite old, 5.15.59 and even 5.17.15 --
where failure is occurring.

I just tested on 5.15.85 and there is no failure.

  builder@i586:/.in$ uname -a
  Linux localhost.localdomain 5.15.85-std-def-alt1 #1 SMP Wed Dec 21 21:14:40 UTC 2022 i686 GNU/Linux
  builder@i586:/.in$ qemu-system-i386 -nographic -enable-kvm
  SeaBIOS (version 1.16.1-alt1)

Perhaps, one of solutions it to reboot our build fleet to newer kernels.
[This maybe hard, though, since special builder node image should be
created and reboot shall be coordinated through all systems, in compare,
updating QEMU would be easier since chroot is created on every build].

Thanks for your helpful replies!

Thanks,

> 
> 
> Alex

Re: qemu-system-i386: Could not install MSR_CORE_THREAD_COUNT handler: Success

[Alexander Graf]

Hi Vitaly,

On 31.12.22 11:17, Vitaly Chikunov wrote:
> Alexander,
>
> On Sat, Dec 31, 2022 at 10:28:21AM +0100, Alexander Graf wrote:
>> On 30.12.22 19:16, Vitaly Chikunov wrote:
>>> On Fri, Dec 30, 2022 at 06:44:14PM +0100, Alexander Graf wrote:
>>>> This is a kvm kernel bug and should be fixed with the latest stable releases. Which kernel version are you running?
>>> This is on latest v6.0 stable - 6.0.15.
>>>
>>> Maybe there could be workaround for such situations? (Or maybe it's
>>> possible to make this error non-fatal?) We use qemu+kvm for testing and
>>> now we cannot test on x86.
>> I'm confused what's going wrong for you. I tried to reproduce the issue
>> locally, but am unable to:
>>
>> $ uname -a
>> Linux server 6.0.15-default #1 SMP PREEMPT_DYNAMIC Sat Dec 31 07:52:52 CET
>> 2022 x86_64 x86_64 x86_64 GNU/Linux
>> $ linux32 chroot .
>> $ uname -a
>> Linux server 6.0.15-default #1 SMP PREEMPT_DYNAMIC Sat Dec 31 07:52:52 CET
>> 2022 i686 GNU/Linux
>> $ cd qemu
>> $ file ./build/qemu-system-i386
>> ./build/qemu-system-i386: ELF 32-bit LSB shared object, Intel 80386, version
>> 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux
>> 3.2.0, BuildID[sha1]=f75e20572be5c604c121de4497397665c168aa4c, with
>> debug_info, not stripped
>> $ ./build/qemu-system-i386 --version
>> QEMU emulator version 7.2.0 (v7.2.0-dirty)
>> Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
>> $ ./build/qemu-system-i386 -nographic -enable-kvm
>> SeaBIOS (version rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org)
>> [...]
>>
>>
>> Can you please double check whether your host kernel version is 6.0.15?
>> Please paste the output of "uname -a".
> Excuse me, I'm incorrectly reported kernel version I tried to boot instead
> of host one. Host kernels are quite old, 5.15.59 and even 5.17.15 --
> where failure is occurring.
>
> I just tested on 5.15.85 and there is no failure.


Awesome, great to hear :). That means everything works as expected at least.


>    builder@i586:/.in$ uname -a
>    Linux localhost.localdomain 5.15.85-std-def-alt1 #1 SMP Wed Dec 21 21:14:40 UTC 2022 i686 GNU/Linux
>    builder@i586:/.in$ qemu-system-i386 -nographic -enable-kvm
>    SeaBIOS (version 1.16.1-alt1)
>
> Perhaps, one of solutions it to reboot our build fleet to newer kernels.
> [This maybe hard, though, since special builder node image should be
> created and reboot shall be coordinated through all systems, in compare,
> updating QEMU would be easier since chroot is created on every build].


I understand that it may be slightly painful to update your build fleet, 
but given this is a genuine kernel bug that has a fix available upstream 
and it only happens on niche corner cases (i386 QEMU on x86-64 Linux 
kernels with the bug) that I doubt anyone will use in production, I'd 
prefer we keep the QEMU logic as is :).

In the meanwhile, while you're patching the build fleet, you can apply 
the patch below as part of your build process to ensure you don't fail 
due to the kernel bug. Just make sure to remove it again as soon as 
you're done with the fleet update :).


diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index a213209379..b9396bc7a6 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -2632,7 +2632,11 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
                  return ret;
              }
      }
+#ifdef __x86_64__
      if (kvm_vm_check_extension(s, KVM_CAP_X86_USER_SPACE_MSR)) {
+#else
+    if (0) {
+#endif
          bool r;

          ret = kvm_vm_enable_cap(s, KVM_CAP_X86_USER_SPACE_MSR, 0,

Alex

Re: qemu-system-i386: Could not install MSR_CORE_THREAD_COUNT handler: Success

[Vitaly Chikunov]

Alexander,

On Sat, Dec 31, 2022 at 12:34:45PM +0100, Alexander Graf wrote:
> On 31.12.22 11:17, Vitaly Chikunov wrote:
> > On Sat, Dec 31, 2022 at 10:28:21AM +0100, Alexander Graf wrote:
> > > On 30.12.22 19:16, Vitaly Chikunov wrote:
> > > > On Fri, Dec 30, 2022 at 06:44:14PM +0100, Alexander Graf wrote:
> > > > > This is a kvm kernel bug and should be fixed with the latest stable releases. Which kernel version are you running?
> > > > This is on latest v6.0 stable - 6.0.15.
> > > > 
> > > > Maybe there could be workaround for such situations? (Or maybe it's
> > > > possible to make this error non-fatal?) We use qemu+kvm for testing and
> > > > now we cannot test on x86.
> > > I'm confused what's going wrong for you. I tried to reproduce the issue
> > > locally, but am unable to:
> > > 
> > > $ uname -a
> > > Linux server 6.0.15-default #1 SMP PREEMPT_DYNAMIC Sat Dec 31 07:52:52 CET
> > > 2022 x86_64 x86_64 x86_64 GNU/Linux
> > > $ linux32 chroot .
> > > $ uname -a
> > > Linux server 6.0.15-default #1 SMP PREEMPT_DYNAMIC Sat Dec 31 07:52:52 CET
> > > 2022 i686 GNU/Linux
> > > $ cd qemu
> > > $ file ./build/qemu-system-i386
> > > ./build/qemu-system-i386: ELF 32-bit LSB shared object, Intel 80386, version
> > > 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux
> > > 3.2.0, BuildID[sha1]=f75e20572be5c604c121de4497397665c168aa4c, with
> > > debug_info, not stripped
> > > $ ./build/qemu-system-i386 --version
> > > QEMU emulator version 7.2.0 (v7.2.0-dirty)
> > > Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
> > > $ ./build/qemu-system-i386 -nographic -enable-kvm
> > > SeaBIOS (version rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org)
> > > [...]
> > > 
> > > 
> > > Can you please double check whether your host kernel version is 6.0.15?
> > > Please paste the output of "uname -a".
> > Excuse me, I'm incorrectly reported kernel version I tried to boot instead
> > of host one. Host kernels are quite old, 5.15.59 and even 5.17.15 --
> > where failure is occurring.
> > 
> > I just tested on 5.15.85 and there is no failure.
> 
> 
> Awesome, great to hear :). That means everything works as expected at least.
> 
> 
> >    builder@i586:/.in$ uname -a
> >    Linux localhost.localdomain 5.15.85-std-def-alt1 #1 SMP Wed Dec 21 21:14:40 UTC 2022 i686 GNU/Linux
> >    builder@i586:/.in$ qemu-system-i386 -nographic -enable-kvm
> >    SeaBIOS (version 1.16.1-alt1)
> > 
> > Perhaps, one of solutions it to reboot our build fleet to newer kernels.
> > [This maybe hard, though, since special builder node image should be
> > created and reboot shall be coordinated through all systems, in compare,
> > updating QEMU would be easier since chroot is created on every build].
> 
> 
> I understand that it may be slightly painful to update your build fleet, but
> given this is a genuine kernel bug that has a fix available upstream and it
> only happens on niche corner cases (i386 QEMU on x86-64 Linux kernels with
> the bug) that I doubt anyone will use in production, I'd prefer we keep the
> QEMU logic as is :).
> 
> In the meanwhile, while you're patching the build fleet, you can apply the
> patch below as part of your build process to ensure you don't fail due to
> the kernel bug. Just make sure to remove it again as soon as you're done
> with the fleet update :).

Thanks for the suggestions.
Best wishes for the New Year!

Vitaly,

> 
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index a213209379..b9396bc7a6 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -2632,7 +2632,11 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
>                  return ret;
>              }
>      }
> +#ifdef __x86_64__
>      if (kvm_vm_check_extension(s, KVM_CAP_X86_USER_SPACE_MSR)) {
> +#else
> +    if (0) {
> +#endif
>          bool r;
> 
>          ret = kvm_vm_enable_cap(s, KVM_CAP_X86_USER_SPACE_MSR, 0,
> 
> Alex
>