qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* qemu-user self emulation broken with default CPU on x86/x64
@ 2023-07-03 16:03 Pierrick Bouvier
  2023-07-03 18:04 ` Daniel P. Berrangé
  2023-07-03 21:05 ` GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) Daniel P. Berrangé
  0 siblings, 2 replies; 6+ messages in thread
From: Pierrick Bouvier @ 2023-07-03 16:03 UTC (permalink / raw)
  To: qemu-devel; +Cc: berrange, Richard Henderson, laurent

Hi everyone,

Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default 
cpu was updated to "max" instead of qemu32/qemu64.

This change "broke" qemu self emulation if this new default cpu is used.

$ ./qemu-x86_64 ./qemu-x86_64 --version
qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion 
`(isize & (isize - 1)) == 0' failed.
qemu: uncaught target signal 6 (Aborted) - core dumped
Aborted

By setting cpu back to qemu64, it works again.
$ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64  --version
qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6)
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers

Commenting assert does not work, as qemu aligned malloc fail shortly after.

I'm willing to fix it, but I'm not sure what is the issue with "max" cpu 
exactly. Is it missing CPU cache line, or something else?
Any guidance would be welcome.

I know it's not the most important problem on earth, but it's still 
surprising to meet this when you try to use qemu to emulate itself.

Regards,
Pierrick

[1] 
https://gitlab.com/qemu-project/qemu/-/commit/d135f781405f7c78153aa65e0327b05a4aa72e50


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: qemu-user self emulation broken with default CPU on x86/x64
  2023-07-03 16:03 qemu-user self emulation broken with default CPU on x86/x64 Pierrick Bouvier
@ 2023-07-03 18:04 ` Daniel P. Berrangé
  2023-07-03 21:05 ` GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) Daniel P. Berrangé
  1 sibling, 0 replies; 6+ messages in thread
From: Daniel P. Berrangé @ 2023-07-03 18:04 UTC (permalink / raw)
  To: Pierrick Bouvier; +Cc: qemu-devel, Richard Henderson, laurent

On Mon, Jul 03, 2023 at 06:03:08PM +0200, Pierrick Bouvier wrote:
> Hi everyone,
> 
> Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default cpu
> was updated to "max" instead of qemu32/qemu64.
> 
> This change "broke" qemu self emulation if this new default cpu is used.
> 
> $ ./qemu-x86_64 ./qemu-x86_64 --version
> qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion `(isize &
> (isize - 1)) == 0' failed.
> qemu: uncaught target signal 6 (Aborted) - core dumped
> Aborted
> 
> By setting cpu back to qemu64, it works again.
> $ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64  --version
> qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6)
> Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
> 
> Commenting assert does not work, as qemu aligned malloc fail shortly after.
> 
> I'm willing to fix it, but I'm not sure what is the issue with "max" cpu
> exactly. Is it missing CPU cache line, or something else?
> Any guidance would be welcome.

Yes, it appears that dcache info being reported by the 'max' CPU is
bogus. We can simply the test case with 'getconf'

With 'max' CPU model:

# getconf -a | grep DCACHE
LEVEL1_DCACHE_SIZE                 7273809
LEVEL1_DCACHE_ASSOC                1
LEVEL1_DCACHE_LINESIZE             2697

with 'qemu64' CPU model

# getconf -a | grep DCACHE
LEVEL1_DCACHE_SIZE                 0
LEVEL1_DCACHE_ASSOC                0
LEVEL1_DCACHE_LINESIZE             0

OR with Nehalem:

# getconf -a | grep DCACHE
LEVEL1_DCACHE_SIZE                 32768
LEVEL1_DCACHE_ASSOC                8
LEVEL1_DCACHE_LINESIZE             64

> 
> I know it's not the most important problem on earth, but it's still
> surprising to meet this when you try to use qemu to emulate itself.
> 
> Regards,
> Pierrick
> 
> [1] https://gitlab.com/qemu-project/qemu/-/commit/d135f781405f7c78153aa65e0327b05a4aa72e50
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 6+ messages in thread

* GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64)
  2023-07-03 16:03 qemu-user self emulation broken with default CPU on x86/x64 Pierrick Bouvier
  2023-07-03 18:04 ` Daniel P. Berrangé
@ 2023-07-03 21:05 ` Daniel P. Berrangé
  2023-07-04 17:30   ` Pierrick Bouvier
  2023-07-04 17:37   ` Florian Weimer
  1 sibling, 2 replies; 6+ messages in thread
From: Daniel P. Berrangé @ 2023-07-03 21:05 UTC (permalink / raw)
  To: Pierrick Bouvier
  Cc: qemu-devel, Richard Henderson, laurent, Florian Weimer,
	Sajan Karumanchi

On Mon, Jul 03, 2023 at 06:03:08PM +0200, Pierrick Bouvier wrote:
> Hi everyone,
> 
> Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default cpu
> was updated to "max" instead of qemu32/qemu64.
> 
> This change "broke" qemu self emulation if this new default cpu is used.
> 
> $ ./qemu-x86_64 ./qemu-x86_64 --version
> qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion `(isize &
> (isize - 1)) == 0' failed.
> qemu: uncaught target signal 6 (Aborted) - core dumped
> Aborted
> 
> By setting cpu back to qemu64, it works again.
> $ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64  --version
> qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6)
> Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
> 
> Commenting assert does not work, as qemu aligned malloc fail shortly after.
> 
> I'm willing to fix it, but I'm not sure what is the issue with "max" cpu
> exactly. Is it missing CPU cache line, or something else?

I've observed GLibC is issuing CPUID leaf 0x8000_001d

QEMU 'max' CPU model doesn't defnie xlevel, so QEMU makes it default
to the same as min_xlevel, which is calculated to be 0x8000_000a.

cpu_x86_cpuid() in QEMU sees CPUID leaf 0x8000_001d is above 0x8000_000a,
and so  considers it an invaild CPUID and thus forces it to report
0x0000_000d which is supposedly what an invalid CPUID leaf should do.


Net result: glibc is asking for 0x8000_001d, but getting back data
for 0x0000_000d.

This doesn't end happily for obvious reasons, getting garbage for
the dcache sizes.


The 'qemu64' CPU model also gets CPUID leaf 0x8000_001d capped back
to 0x0000_000d, but crucially qemu64 lacks the 'xsave' feature bit,
so QEMU returns all-zeroes for CPUID leaf 0x0000_000d. Still not
good, but this makes glibc report 0 for DCACHE_*, which in turn
avoids tripping up the nested qemu which queries DCACHE sysconf.

So the problem is thus more widespread than just 'max' CPU model.

Any QEMU CPU model with vendor=AuthenticAMD and the xsave feature,
and the xlevel unset, will cause glibc to report garbage for the
L1D cache info

Any QEMU CPU model with vendor=AuthenticAMD and without the xsave
feature, and the xlevel unset, will cause glibc to report zeroes
for L1D cache info

Neither is good, but the latter at least doesn't trip up the
nested QEMU when it queries L1D cache info.

I'm unsure if QEMU's behaviour is correct with calculating the
default 'xlevel' values for 'max', but I'm assuming the xlevel
was correct for Opteron_G4/5 since those are explicitly set
in the code for along time.

Over to the GLibC side, I see there was a recent change:

commit 103a469dc7755fd9e8ccf362f3dd4c55dc761908
Author: Sajan Karumanchi <sajan.karumanchi@amd.com>
Date:   Wed Jan 18 18:29:04 2023 +0100

    x86: Cache computation for AMD architecture.
    
    All AMD architectures cache details will be computed based on
    __cpuid__ `0x8000_001D` and the reference to __cpuid__ `0x8000_0006` will be
    zeroed out for future architectures.
    
    Reviewed-by: Premachandra Mallappa <premachandra.mallappa@amd.com>


This introduced the use of CPUID leaf 0x8000_001D. Before this point
glibc would use 0x8000_0000 and 0x8000_0005 to calculate the cache
size.  QEMU worked correctly with this implementation.

  https://sourceware.org/pipermail/libc-alpha/2023-January/144815.html

The reporter said

   "Though we have done the testing on Zen and pre-Zen architectures,
    we recommend to carryout the tests from your end too."

it is unclear if their testing would have covered Opteron_G4/Opteron_G5
architectures, and I not expecting to have had QEMU testing of course ?

I don't have any non-virtual pre-Zen silicon I could verify CPUID
behaviour on. I've not found historic versions of the AMD architecture
reference to see when they first documented 0x8000_001d as a valid
CPUID leaf for getting cache info.

IOW it is still unclear to me whether the root cause bug here is in
QEMU's emulation of CPUID 0x8000_001d, or whether this was actually
a real regression introduced in glibc >= 2.37

I'm tending towards glibc regression though.

Copying Florian and the original AMD patch author

Brief summary

With old glibc 2.36, using QEMU's  qemu64/max CPU models:

# qemu-x86_64-static -cpu qemu64 /bin/getconf -a | grep DCACHE
LEVEL1_DCACHE_SIZE                 65536
LEVEL1_DCACHE_ASSOC                2
LEVEL1_DCACHE_LINESIZE             64

# qemu-x86_64-static -cpu Opteron_G4 /bin/getconf -a | grep DCACHE
LEVEL1_DCACHE_SIZE                 65536
LEVEL1_DCACHE_ASSOC                2
LEVEL1_DCACHE_LINESIZE             64

# qemu-x86_64-static -cpu max /bin/getconf -a | grep DCACHE
LEVEL1_DCACHE_SIZE                 65536
LEVEL1_DCACHE_ASSOC                2
LEVEL1_DCACHE_LINESIZE             64


With new glibc 2.37:

# qemu-x86_64-static -cpu qemu64 /bin/getconf -a | grep DCACHE
LEVEL1_DCACHE_SIZE                 0
LEVEL1_DCACHE_ASSOC                0
LEVEL1_DCACHE_LINESIZE             0

# qemu-x86_64-static -cpu Opteron_G4 /bin/getconf -a | grep DCACHE
LEVEL1_DCACHE_SIZE                 693889
LEVEL1_DCACHE_ASSOC                1
LEVEL1_DCACHE_LINESIZE             833

# qemu-x86_64-static -cpu max /bin/getconf -a | grep DCACHE
LEVEL1_DCACHE_SIZE                 7273809
LEVEL1_DCACHE_ASSOC                1
LEVEL1_DCACHE_LINESIZE             2697

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64)
  2023-07-03 21:05 ` GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) Daniel P. Berrangé
@ 2023-07-04 17:30   ` Pierrick Bouvier
  2023-07-04 17:37   ` Florian Weimer
  1 sibling, 0 replies; 6+ messages in thread
From: Pierrick Bouvier @ 2023-07-04 17:30 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: qemu-devel, Richard Henderson, laurent, Florian Weimer,
	Sajan Karumanchi

Thanks for this deep analysis.

Even if the bug is potentially on glibc side, would that be worth to 
change something in qemu CPU description to avoid it (like changing cpuid)?

On 7/3/23 23:05, Daniel P. Berrangé wrote:
> On Mon, Jul 03, 2023 at 06:03:08PM +0200, Pierrick Bouvier wrote:
>> Hi everyone,
>>
>> Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default cpu
>> was updated to "max" instead of qemu32/qemu64.
>>
>> This change "broke" qemu self emulation if this new default cpu is used.
>>
>> $ ./qemu-x86_64 ./qemu-x86_64 --version
>> qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion `(isize &
>> (isize - 1)) == 0' failed.
>> qemu: uncaught target signal 6 (Aborted) - core dumped
>> Aborted
>>
>> By setting cpu back to qemu64, it works again.
>> $ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64  --version
>> qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6)
>> Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
>>
>> Commenting assert does not work, as qemu aligned malloc fail shortly after.
>>
>> I'm willing to fix it, but I'm not sure what is the issue with "max" cpu
>> exactly. Is it missing CPU cache line, or something else?
> 
> I've observed GLibC is issuing CPUID leaf 0x8000_001d
> 
> QEMU 'max' CPU model doesn't defnie xlevel, so QEMU makes it default
> to the same as min_xlevel, which is calculated to be 0x8000_000a.
> 
> cpu_x86_cpuid() in QEMU sees CPUID leaf 0x8000_001d is above 0x8000_000a,
> and so  considers it an invaild CPUID and thus forces it to report
> 0x0000_000d which is supposedly what an invalid CPUID leaf should do.
> 
> 
> Net result: glibc is asking for 0x8000_001d, but getting back data
> for 0x0000_000d.
> 
> This doesn't end happily for obvious reasons, getting garbage for
> the dcache sizes.
> 
> 
> The 'qemu64' CPU model also gets CPUID leaf 0x8000_001d capped back
> to 0x0000_000d, but crucially qemu64 lacks the 'xsave' feature bit,
> so QEMU returns all-zeroes for CPUID leaf 0x0000_000d. Still not
> good, but this makes glibc report 0 for DCACHE_*, which in turn
> avoids tripping up the nested qemu which queries DCACHE sysconf.
> 
> So the problem is thus more widespread than just 'max' CPU model.
> 
> Any QEMU CPU model with vendor=AuthenticAMD and the xsave feature,
> and the xlevel unset, will cause glibc to report garbage for the
> L1D cache info
> 
> Any QEMU CPU model with vendor=AuthenticAMD and without the xsave
> feature, and the xlevel unset, will cause glibc to report zeroes
> for L1D cache info
> 
> Neither is good, but the latter at least doesn't trip up the
> nested QEMU when it queries L1D cache info.
> 
> I'm unsure if QEMU's behaviour is correct with calculating the
> default 'xlevel' values for 'max', but I'm assuming the xlevel
> was correct for Opteron_G4/5 since those are explicitly set
> in the code for along time.
> 
> Over to the GLibC side, I see there was a recent change:
> 
> commit 103a469dc7755fd9e8ccf362f3dd4c55dc761908
> Author: Sajan Karumanchi <sajan.karumanchi@amd.com>
> Date:   Wed Jan 18 18:29:04 2023 +0100
> 
>      x86: Cache computation for AMD architecture.
>      
>      All AMD architectures cache details will be computed based on
>      __cpuid__ `0x8000_001D` and the reference to __cpuid__ `0x8000_0006` will be
>      zeroed out for future architectures.
>      
>      Reviewed-by: Premachandra Mallappa <premachandra.mallappa@amd.com>
> 
> 
> This introduced the use of CPUID leaf 0x8000_001D. Before this point
> glibc would use 0x8000_0000 and 0x8000_0005 to calculate the cache
> size.  QEMU worked correctly with this implementation.
> 
>    https://sourceware.org/pipermail/libc-alpha/2023-January/144815.html
> 
> The reporter said
> 
>     "Though we have done the testing on Zen and pre-Zen architectures,
>      we recommend to carryout the tests from your end too."
> 
> it is unclear if their testing would have covered Opteron_G4/Opteron_G5
> architectures, and I not expecting to have had QEMU testing of course ?
> 
> I don't have any non-virtual pre-Zen silicon I could verify CPUID
> behaviour on. I've not found historic versions of the AMD architecture
> reference to see when they first documented 0x8000_001d as a valid
> CPUID leaf for getting cache info.
> 
> IOW it is still unclear to me whether the root cause bug here is in
> QEMU's emulation of CPUID 0x8000_001d, or whether this was actually
> a real regression introduced in glibc >= 2.37
> 
> I'm tending towards glibc regression though.
> 
> Copying Florian and the original AMD patch author
> 
> Brief summary
> 
> With old glibc 2.36, using QEMU's  qemu64/max CPU models:
> 
> # qemu-x86_64-static -cpu qemu64 /bin/getconf -a | grep DCACHE
> LEVEL1_DCACHE_SIZE                 65536
> LEVEL1_DCACHE_ASSOC                2
> LEVEL1_DCACHE_LINESIZE             64
> 
> # qemu-x86_64-static -cpu Opteron_G4 /bin/getconf -a | grep DCACHE
> LEVEL1_DCACHE_SIZE                 65536
> LEVEL1_DCACHE_ASSOC                2
> LEVEL1_DCACHE_LINESIZE             64
> 
> # qemu-x86_64-static -cpu max /bin/getconf -a | grep DCACHE
> LEVEL1_DCACHE_SIZE                 65536
> LEVEL1_DCACHE_ASSOC                2
> LEVEL1_DCACHE_LINESIZE             64
> 
> 
> With new glibc 2.37:
> 
> # qemu-x86_64-static -cpu qemu64 /bin/getconf -a | grep DCACHE
> LEVEL1_DCACHE_SIZE                 0
> LEVEL1_DCACHE_ASSOC                0
> LEVEL1_DCACHE_LINESIZE             0
> 
> # qemu-x86_64-static -cpu Opteron_G4 /bin/getconf -a | grep DCACHE
> LEVEL1_DCACHE_SIZE                 693889
> LEVEL1_DCACHE_ASSOC                1
> LEVEL1_DCACHE_LINESIZE             833
> 
> # qemu-x86_64-static -cpu max /bin/getconf -a | grep DCACHE
> LEVEL1_DCACHE_SIZE                 7273809
> LEVEL1_DCACHE_ASSOC                1
> LEVEL1_DCACHE_LINESIZE             2697
> 
> With regards,
> Daniel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64)
  2023-07-03 21:05 ` GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) Daniel P. Berrangé
  2023-07-04 17:30   ` Pierrick Bouvier
@ 2023-07-04 17:37   ` Florian Weimer
  2023-07-05 13:08     ` Karumanchi, Sajan
  1 sibling, 1 reply; 6+ messages in thread
From: Florian Weimer @ 2023-07-04 17:37 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Pierrick Bouvier, qemu-devel, Richard Henderson, laurent,
	Sajan Karumanchi

* Daniel P. Berrangé:

> On Mon, Jul 03, 2023 at 06:03:08PM +0200, Pierrick Bouvier wrote:
>> Hi everyone,
>> 
>> Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default cpu
>> was updated to "max" instead of qemu32/qemu64.
>> 
>> This change "broke" qemu self emulation if this new default cpu is used.
>> 
>> $ ./qemu-x86_64 ./qemu-x86_64 --version
>> qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion `(isize &
>> (isize - 1)) == 0' failed.
>> qemu: uncaught target signal 6 (Aborted) - core dumped
>> Aborted
>> 
>> By setting cpu back to qemu64, it works again.
>> $ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64  --version
>> qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6)
>> Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
>> 
>> Commenting assert does not work, as qemu aligned malloc fail shortly after.
>> 
>> I'm willing to fix it, but I'm not sure what is the issue with "max" cpu
>> exactly. Is it missing CPU cache line, or something else?
>
> I've observed GLibC is issuing CPUID leaf 0x8000_001d
>
> QEMU 'max' CPU model doesn't defnie xlevel, so QEMU makes it default
> to the same as min_xlevel, which is calculated to be 0x8000_000a.
>
> cpu_x86_cpuid() in QEMU sees CPUID leaf 0x8000_001d is above 0x8000_000a,
> and so  considers it an invaild CPUID and thus forces it to report
> 0x0000_000d which is supposedly what an invalid CPUID leaf should do.
>
>
> Net result: glibc is asking for 0x8000_001d, but getting back data
> for 0x0000_000d.
>
> This doesn't end happily for obvious reasons, getting garbage for
> the dcache sizes.
>
>
> The 'qemu64' CPU model also gets CPUID leaf 0x8000_001d capped back
> to 0x0000_000d, but crucially qemu64 lacks the 'xsave' feature bit,
> so QEMU returns all-zeroes for CPUID leaf 0x0000_000d. Still not
> good, but this makes glibc report 0 for DCACHE_*, which in turn
> avoids tripping up the nested qemu which queries DCACHE sysconf.
>
> So the problem is thus more widespread than just 'max' CPU model.
>
> Any QEMU CPU model with vendor=AuthenticAMD and the xsave feature,
> and the xlevel unset, will cause glibc to report garbage for the
> L1D cache info
>
> Any QEMU CPU model with vendor=AuthenticAMD and without the xsave
> feature, and the xlevel unset, will cause glibc to report zeroes
> for L1D cache info
>
> Neither is good, but the latter at least doesn't trip up the
> nested QEMU when it queries L1D cache info.
>
> I'm unsure if QEMU's behaviour is correct with calculating the
> default 'xlevel' values for 'max', but I'm assuming the xlevel
> was correct for Opteron_G4/5 since those are explicitly set
> in the code for along time.

We are tracking this as:

  New AMD cache size computation logic does not work for some CPUs,
  hypervisors
  <https://sourceware.org/bugzilla/show_bug.cgi?id=30428>

I filed it after we resolved the earlier crashes because the data is
clearly not accurate.  I was also able to confirm that impacts more than
just hypervisors.

Sajan posted a first patch:

  [PATCH] x86: Fix for cache computation on AMD legacy cpus.
  <https://sourceware.org/pipermail/libc-alpha/2023-June/148763.html>

However, it changes the reported cache sizes on some older CPUs compared
to what we had before (although the values are no longer zero at least).

Thanks,
Florian



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64)
  2023-07-04 17:37   ` Florian Weimer
@ 2023-07-05 13:08     ` Karumanchi, Sajan
  0 siblings, 0 replies; 6+ messages in thread
From: Karumanchi, Sajan @ 2023-07-05 13:08 UTC (permalink / raw)
  To: Florian Weimer, Daniel P.Berrangé
  Cc: Pierrick Bouvier, qemu-devel@nongnu.org, Richard Henderson,
	laurent@vivier.eu, Mallappa, Premachandra

[-- Attachment #1: Type: text/plain, Size: 4158 bytes --]

[AMD Official Use Only - General]

++ Prem


Thanks & Regards,

Sajan K.



________________________________
From: Florian Weimer <fweimer@redhat.com>
Sent: Tuesday, July 4, 2023 11:07 PM
To: Daniel P.Berrangé <berrange@redhat.com>
Cc: Pierrick Bouvier <pierrick.bouvier@linaro.org>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; Richard Henderson <richard.henderson@linaro.org>; laurent@vivier.eu <laurent@vivier.eu>; Karumanchi, Sajan <Sajan.Karumanchi@amd.com>
Subject: Re: GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64)

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.


* Daniel P. Berrangé:

> On Mon, Jul 03, 2023 at 06:03:08PM +0200, Pierrick Bouvier wrote:
>> Hi everyone,
>>
>> Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default cpu
>> was updated to "max" instead of qemu32/qemu64.
>>
>> This change "broke" qemu self emulation if this new default cpu is used.
>>
>> $ ./qemu-x86_64 ./qemu-x86_64 --version
>> qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion `(isize &
>> (isize - 1)) == 0' failed.
>> qemu: uncaught target signal 6 (Aborted) - core dumped
>> Aborted
>>
>> By setting cpu back to qemu64, it works again.
>> $ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64  --version
>> qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6)
>> Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
>>
>> Commenting assert does not work, as qemu aligned malloc fail shortly after.
>>
>> I'm willing to fix it, but I'm not sure what is the issue with "max" cpu
>> exactly. Is it missing CPU cache line, or something else?
>
> I've observed GLibC is issuing CPUID leaf 0x8000_001d
>
> QEMU 'max' CPU model doesn't defnie xlevel, so QEMU makes it default
> to the same as min_xlevel, which is calculated to be 0x8000_000a.
>
> cpu_x86_cpuid() in QEMU sees CPUID leaf 0x8000_001d is above 0x8000_000a,
> and so  considers it an invaild CPUID and thus forces it to report
> 0x0000_000d which is supposedly what an invalid CPUID leaf should do.
>
>
> Net result: glibc is asking for 0x8000_001d, but getting back data
> for 0x0000_000d.
>
> This doesn't end happily for obvious reasons, getting garbage for
> the dcache sizes.
>
>
> The 'qemu64' CPU model also gets CPUID leaf 0x8000_001d capped back
> to 0x0000_000d, but crucially qemu64 lacks the 'xsave' feature bit,
> so QEMU returns all-zeroes for CPUID leaf 0x0000_000d. Still not
> good, but this makes glibc report 0 for DCACHE_*, which in turn
> avoids tripping up the nested qemu which queries DCACHE sysconf.
>
> So the problem is thus more widespread than just 'max' CPU model.
>
> Any QEMU CPU model with vendor=AuthenticAMD and the xsave feature,
> and the xlevel unset, will cause glibc to report garbage for the
> L1D cache info
>
> Any QEMU CPU model with vendor=AuthenticAMD and without the xsave
> feature, and the xlevel unset, will cause glibc to report zeroes
> for L1D cache info
>
> Neither is good, but the latter at least doesn't trip up the
> nested QEMU when it queries L1D cache info.
>
> I'm unsure if QEMU's behaviour is correct with calculating the
> default 'xlevel' values for 'max', but I'm assuming the xlevel
> was correct for Opteron_G4/5 since those are explicitly set
> in the code for along time.

We are tracking this as:

  New AMD cache size computation logic does not work for some CPUs,
  hypervisors
  <https://sourceware.org/bugzilla/show_bug.cgi?id=30428>

I filed it after we resolved the earlier crashes because the data is
clearly not accurate.  I was also able to confirm that impacts more than
just hypervisors.

Sajan posted a first patch:

  [PATCH] x86: Fix for cache computation on AMD legacy cpus.
  <https://sourceware.org/pipermail/libc-alpha/2023-June/148763.html>

However, it changes the reported cache sizes on some older CPUs compared
to what we had before (although the values are no longer zero at least).

Thanks,
Florian


[-- Attachment #2: Type: text/html, Size: 6631 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2023-07-05 13:42 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-07-03 16:03 qemu-user self emulation broken with default CPU on x86/x64 Pierrick Bouvier
2023-07-03 18:04 ` Daniel P. Berrangé
2023-07-03 21:05 ` GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) Daniel P. Berrangé
2023-07-04 17:30   ` Pierrick Bouvier
2023-07-04 17:37   ` Florian Weimer
2023-07-05 13:08     ` Karumanchi, Sajan

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).