* qemu-user self emulation broken with default CPU on x86/x64 @ 2023-07-03 16:03 Pierrick Bouvier 2023-07-03 18:04 ` Daniel P. Berrangé 2023-07-03 21:05 ` GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) Daniel P. Berrangé 0 siblings, 2 replies; 6+ messages in thread From: Pierrick Bouvier @ 2023-07-03 16:03 UTC (permalink / raw) To: qemu-devel; +Cc: berrange, Richard Henderson, laurent Hi everyone, Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default cpu was updated to "max" instead of qemu32/qemu64. This change "broke" qemu self emulation if this new default cpu is used. $ ./qemu-x86_64 ./qemu-x86_64 --version qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion `(isize & (isize - 1)) == 0' failed. qemu: uncaught target signal 6 (Aborted) - core dumped Aborted By setting cpu back to qemu64, it works again. $ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64 --version qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6) Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers Commenting assert does not work, as qemu aligned malloc fail shortly after. I'm willing to fix it, but I'm not sure what is the issue with "max" cpu exactly. Is it missing CPU cache line, or something else? Any guidance would be welcome. I know it's not the most important problem on earth, but it's still surprising to meet this when you try to use qemu to emulate itself. Regards, Pierrick [1] https://gitlab.com/qemu-project/qemu/-/commit/d135f781405f7c78153aa65e0327b05a4aa72e50 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: qemu-user self emulation broken with default CPU on x86/x64 2023-07-03 16:03 qemu-user self emulation broken with default CPU on x86/x64 Pierrick Bouvier @ 2023-07-03 18:04 ` Daniel P. Berrangé 2023-07-03 21:05 ` GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) Daniel P. Berrangé 1 sibling, 0 replies; 6+ messages in thread From: Daniel P. Berrangé @ 2023-07-03 18:04 UTC (permalink / raw) To: Pierrick Bouvier; +Cc: qemu-devel, Richard Henderson, laurent On Mon, Jul 03, 2023 at 06:03:08PM +0200, Pierrick Bouvier wrote: > Hi everyone, > > Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default cpu > was updated to "max" instead of qemu32/qemu64. > > This change "broke" qemu self emulation if this new default cpu is used. > > $ ./qemu-x86_64 ./qemu-x86_64 --version > qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion `(isize & > (isize - 1)) == 0' failed. > qemu: uncaught target signal 6 (Aborted) - core dumped > Aborted > > By setting cpu back to qemu64, it works again. > $ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64 --version > qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6) > Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers > > Commenting assert does not work, as qemu aligned malloc fail shortly after. > > I'm willing to fix it, but I'm not sure what is the issue with "max" cpu > exactly. Is it missing CPU cache line, or something else? > Any guidance would be welcome. Yes, it appears that dcache info being reported by the 'max' CPU is bogus. We can simply the test case with 'getconf' With 'max' CPU model: # getconf -a | grep DCACHE LEVEL1_DCACHE_SIZE 7273809 LEVEL1_DCACHE_ASSOC 1 LEVEL1_DCACHE_LINESIZE 2697 with 'qemu64' CPU model # getconf -a | grep DCACHE LEVEL1_DCACHE_SIZE 0 LEVEL1_DCACHE_ASSOC 0 LEVEL1_DCACHE_LINESIZE 0 OR with Nehalem: # getconf -a | grep DCACHE LEVEL1_DCACHE_SIZE 32768 LEVEL1_DCACHE_ASSOC 8 LEVEL1_DCACHE_LINESIZE 64 > > I know it's not the most important problem on earth, but it's still > surprising to meet this when you try to use qemu to emulate itself. > > Regards, > Pierrick > > [1] https://gitlab.com/qemu-project/qemu/-/commit/d135f781405f7c78153aa65e0327b05a4aa72e50 > With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 6+ messages in thread
* GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) 2023-07-03 16:03 qemu-user self emulation broken with default CPU on x86/x64 Pierrick Bouvier 2023-07-03 18:04 ` Daniel P. Berrangé @ 2023-07-03 21:05 ` Daniel P. Berrangé 2023-07-04 17:30 ` Pierrick Bouvier 2023-07-04 17:37 ` Florian Weimer 1 sibling, 2 replies; 6+ messages in thread From: Daniel P. Berrangé @ 2023-07-03 21:05 UTC (permalink / raw) To: Pierrick Bouvier Cc: qemu-devel, Richard Henderson, laurent, Florian Weimer, Sajan Karumanchi On Mon, Jul 03, 2023 at 06:03:08PM +0200, Pierrick Bouvier wrote: > Hi everyone, > > Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default cpu > was updated to "max" instead of qemu32/qemu64. > > This change "broke" qemu self emulation if this new default cpu is used. > > $ ./qemu-x86_64 ./qemu-x86_64 --version > qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion `(isize & > (isize - 1)) == 0' failed. > qemu: uncaught target signal 6 (Aborted) - core dumped > Aborted > > By setting cpu back to qemu64, it works again. > $ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64 --version > qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6) > Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers > > Commenting assert does not work, as qemu aligned malloc fail shortly after. > > I'm willing to fix it, but I'm not sure what is the issue with "max" cpu > exactly. Is it missing CPU cache line, or something else? I've observed GLibC is issuing CPUID leaf 0x8000_001d QEMU 'max' CPU model doesn't defnie xlevel, so QEMU makes it default to the same as min_xlevel, which is calculated to be 0x8000_000a. cpu_x86_cpuid() in QEMU sees CPUID leaf 0x8000_001d is above 0x8000_000a, and so considers it an invaild CPUID and thus forces it to report 0x0000_000d which is supposedly what an invalid CPUID leaf should do. Net result: glibc is asking for 0x8000_001d, but getting back data for 0x0000_000d. This doesn't end happily for obvious reasons, getting garbage for the dcache sizes. The 'qemu64' CPU model also gets CPUID leaf 0x8000_001d capped back to 0x0000_000d, but crucially qemu64 lacks the 'xsave' feature bit, so QEMU returns all-zeroes for CPUID leaf 0x0000_000d. Still not good, but this makes glibc report 0 for DCACHE_*, which in turn avoids tripping up the nested qemu which queries DCACHE sysconf. So the problem is thus more widespread than just 'max' CPU model. Any QEMU CPU model with vendor=AuthenticAMD and the xsave feature, and the xlevel unset, will cause glibc to report garbage for the L1D cache info Any QEMU CPU model with vendor=AuthenticAMD and without the xsave feature, and the xlevel unset, will cause glibc to report zeroes for L1D cache info Neither is good, but the latter at least doesn't trip up the nested QEMU when it queries L1D cache info. I'm unsure if QEMU's behaviour is correct with calculating the default 'xlevel' values for 'max', but I'm assuming the xlevel was correct for Opteron_G4/5 since those are explicitly set in the code for along time. Over to the GLibC side, I see there was a recent change: commit 103a469dc7755fd9e8ccf362f3dd4c55dc761908 Author: Sajan Karumanchi <sajan.karumanchi@amd.com> Date: Wed Jan 18 18:29:04 2023 +0100 x86: Cache computation for AMD architecture. All AMD architectures cache details will be computed based on __cpuid__ `0x8000_001D` and the reference to __cpuid__ `0x8000_0006` will be zeroed out for future architectures. Reviewed-by: Premachandra Mallappa <premachandra.mallappa@amd.com> This introduced the use of CPUID leaf 0x8000_001D. Before this point glibc would use 0x8000_0000 and 0x8000_0005 to calculate the cache size. QEMU worked correctly with this implementation. https://sourceware.org/pipermail/libc-alpha/2023-January/144815.html The reporter said "Though we have done the testing on Zen and pre-Zen architectures, we recommend to carryout the tests from your end too." it is unclear if their testing would have covered Opteron_G4/Opteron_G5 architectures, and I not expecting to have had QEMU testing of course ? I don't have any non-virtual pre-Zen silicon I could verify CPUID behaviour on. I've not found historic versions of the AMD architecture reference to see when they first documented 0x8000_001d as a valid CPUID leaf for getting cache info. IOW it is still unclear to me whether the root cause bug here is in QEMU's emulation of CPUID 0x8000_001d, or whether this was actually a real regression introduced in glibc >= 2.37 I'm tending towards glibc regression though. Copying Florian and the original AMD patch author Brief summary With old glibc 2.36, using QEMU's qemu64/max CPU models: # qemu-x86_64-static -cpu qemu64 /bin/getconf -a | grep DCACHE LEVEL1_DCACHE_SIZE 65536 LEVEL1_DCACHE_ASSOC 2 LEVEL1_DCACHE_LINESIZE 64 # qemu-x86_64-static -cpu Opteron_G4 /bin/getconf -a | grep DCACHE LEVEL1_DCACHE_SIZE 65536 LEVEL1_DCACHE_ASSOC 2 LEVEL1_DCACHE_LINESIZE 64 # qemu-x86_64-static -cpu max /bin/getconf -a | grep DCACHE LEVEL1_DCACHE_SIZE 65536 LEVEL1_DCACHE_ASSOC 2 LEVEL1_DCACHE_LINESIZE 64 With new glibc 2.37: # qemu-x86_64-static -cpu qemu64 /bin/getconf -a | grep DCACHE LEVEL1_DCACHE_SIZE 0 LEVEL1_DCACHE_ASSOC 0 LEVEL1_DCACHE_LINESIZE 0 # qemu-x86_64-static -cpu Opteron_G4 /bin/getconf -a | grep DCACHE LEVEL1_DCACHE_SIZE 693889 LEVEL1_DCACHE_ASSOC 1 LEVEL1_DCACHE_LINESIZE 833 # qemu-x86_64-static -cpu max /bin/getconf -a | grep DCACHE LEVEL1_DCACHE_SIZE 7273809 LEVEL1_DCACHE_ASSOC 1 LEVEL1_DCACHE_LINESIZE 2697 With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) 2023-07-03 21:05 ` GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) Daniel P. Berrangé @ 2023-07-04 17:30 ` Pierrick Bouvier 2023-07-04 17:37 ` Florian Weimer 1 sibling, 0 replies; 6+ messages in thread From: Pierrick Bouvier @ 2023-07-04 17:30 UTC (permalink / raw) To: Daniel P. Berrangé Cc: qemu-devel, Richard Henderson, laurent, Florian Weimer, Sajan Karumanchi Thanks for this deep analysis. Even if the bug is potentially on glibc side, would that be worth to change something in qemu CPU description to avoid it (like changing cpuid)? On 7/3/23 23:05, Daniel P. Berrangé wrote: > On Mon, Jul 03, 2023 at 06:03:08PM +0200, Pierrick Bouvier wrote: >> Hi everyone, >> >> Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default cpu >> was updated to "max" instead of qemu32/qemu64. >> >> This change "broke" qemu self emulation if this new default cpu is used. >> >> $ ./qemu-x86_64 ./qemu-x86_64 --version >> qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion `(isize & >> (isize - 1)) == 0' failed. >> qemu: uncaught target signal 6 (Aborted) - core dumped >> Aborted >> >> By setting cpu back to qemu64, it works again. >> $ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64 --version >> qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6) >> Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers >> >> Commenting assert does not work, as qemu aligned malloc fail shortly after. >> >> I'm willing to fix it, but I'm not sure what is the issue with "max" cpu >> exactly. Is it missing CPU cache line, or something else? > > I've observed GLibC is issuing CPUID leaf 0x8000_001d > > QEMU 'max' CPU model doesn't defnie xlevel, so QEMU makes it default > to the same as min_xlevel, which is calculated to be 0x8000_000a. > > cpu_x86_cpuid() in QEMU sees CPUID leaf 0x8000_001d is above 0x8000_000a, > and so considers it an invaild CPUID and thus forces it to report > 0x0000_000d which is supposedly what an invalid CPUID leaf should do. > > > Net result: glibc is asking for 0x8000_001d, but getting back data > for 0x0000_000d. > > This doesn't end happily for obvious reasons, getting garbage for > the dcache sizes. > > > The 'qemu64' CPU model also gets CPUID leaf 0x8000_001d capped back > to 0x0000_000d, but crucially qemu64 lacks the 'xsave' feature bit, > so QEMU returns all-zeroes for CPUID leaf 0x0000_000d. Still not > good, but this makes glibc report 0 for DCACHE_*, which in turn > avoids tripping up the nested qemu which queries DCACHE sysconf. > > So the problem is thus more widespread than just 'max' CPU model. > > Any QEMU CPU model with vendor=AuthenticAMD and the xsave feature, > and the xlevel unset, will cause glibc to report garbage for the > L1D cache info > > Any QEMU CPU model with vendor=AuthenticAMD and without the xsave > feature, and the xlevel unset, will cause glibc to report zeroes > for L1D cache info > > Neither is good, but the latter at least doesn't trip up the > nested QEMU when it queries L1D cache info. > > I'm unsure if QEMU's behaviour is correct with calculating the > default 'xlevel' values for 'max', but I'm assuming the xlevel > was correct for Opteron_G4/5 since those are explicitly set > in the code for along time. > > Over to the GLibC side, I see there was a recent change: > > commit 103a469dc7755fd9e8ccf362f3dd4c55dc761908 > Author: Sajan Karumanchi <sajan.karumanchi@amd.com> > Date: Wed Jan 18 18:29:04 2023 +0100 > > x86: Cache computation for AMD architecture. > > All AMD architectures cache details will be computed based on > __cpuid__ `0x8000_001D` and the reference to __cpuid__ `0x8000_0006` will be > zeroed out for future architectures. > > Reviewed-by: Premachandra Mallappa <premachandra.mallappa@amd.com> > > > This introduced the use of CPUID leaf 0x8000_001D. Before this point > glibc would use 0x8000_0000 and 0x8000_0005 to calculate the cache > size. QEMU worked correctly with this implementation. > > https://sourceware.org/pipermail/libc-alpha/2023-January/144815.html > > The reporter said > > "Though we have done the testing on Zen and pre-Zen architectures, > we recommend to carryout the tests from your end too." > > it is unclear if their testing would have covered Opteron_G4/Opteron_G5 > architectures, and I not expecting to have had QEMU testing of course ? > > I don't have any non-virtual pre-Zen silicon I could verify CPUID > behaviour on. I've not found historic versions of the AMD architecture > reference to see when they first documented 0x8000_001d as a valid > CPUID leaf for getting cache info. > > IOW it is still unclear to me whether the root cause bug here is in > QEMU's emulation of CPUID 0x8000_001d, or whether this was actually > a real regression introduced in glibc >= 2.37 > > I'm tending towards glibc regression though. > > Copying Florian and the original AMD patch author > > Brief summary > > With old glibc 2.36, using QEMU's qemu64/max CPU models: > > # qemu-x86_64-static -cpu qemu64 /bin/getconf -a | grep DCACHE > LEVEL1_DCACHE_SIZE 65536 > LEVEL1_DCACHE_ASSOC 2 > LEVEL1_DCACHE_LINESIZE 64 > > # qemu-x86_64-static -cpu Opteron_G4 /bin/getconf -a | grep DCACHE > LEVEL1_DCACHE_SIZE 65536 > LEVEL1_DCACHE_ASSOC 2 > LEVEL1_DCACHE_LINESIZE 64 > > # qemu-x86_64-static -cpu max /bin/getconf -a | grep DCACHE > LEVEL1_DCACHE_SIZE 65536 > LEVEL1_DCACHE_ASSOC 2 > LEVEL1_DCACHE_LINESIZE 64 > > > With new glibc 2.37: > > # qemu-x86_64-static -cpu qemu64 /bin/getconf -a | grep DCACHE > LEVEL1_DCACHE_SIZE 0 > LEVEL1_DCACHE_ASSOC 0 > LEVEL1_DCACHE_LINESIZE 0 > > # qemu-x86_64-static -cpu Opteron_G4 /bin/getconf -a | grep DCACHE > LEVEL1_DCACHE_SIZE 693889 > LEVEL1_DCACHE_ASSOC 1 > LEVEL1_DCACHE_LINESIZE 833 > > # qemu-x86_64-static -cpu max /bin/getconf -a | grep DCACHE > LEVEL1_DCACHE_SIZE 7273809 > LEVEL1_DCACHE_ASSOC 1 > LEVEL1_DCACHE_LINESIZE 2697 > > With regards, > Daniel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) 2023-07-03 21:05 ` GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) Daniel P. Berrangé 2023-07-04 17:30 ` Pierrick Bouvier @ 2023-07-04 17:37 ` Florian Weimer 2023-07-05 13:08 ` Karumanchi, Sajan 1 sibling, 1 reply; 6+ messages in thread From: Florian Weimer @ 2023-07-04 17:37 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Pierrick Bouvier, qemu-devel, Richard Henderson, laurent, Sajan Karumanchi * Daniel P. Berrangé: > On Mon, Jul 03, 2023 at 06:03:08PM +0200, Pierrick Bouvier wrote: >> Hi everyone, >> >> Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default cpu >> was updated to "max" instead of qemu32/qemu64. >> >> This change "broke" qemu self emulation if this new default cpu is used. >> >> $ ./qemu-x86_64 ./qemu-x86_64 --version >> qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion `(isize & >> (isize - 1)) == 0' failed. >> qemu: uncaught target signal 6 (Aborted) - core dumped >> Aborted >> >> By setting cpu back to qemu64, it works again. >> $ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64 --version >> qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6) >> Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers >> >> Commenting assert does not work, as qemu aligned malloc fail shortly after. >> >> I'm willing to fix it, but I'm not sure what is the issue with "max" cpu >> exactly. Is it missing CPU cache line, or something else? > > I've observed GLibC is issuing CPUID leaf 0x8000_001d > > QEMU 'max' CPU model doesn't defnie xlevel, so QEMU makes it default > to the same as min_xlevel, which is calculated to be 0x8000_000a. > > cpu_x86_cpuid() in QEMU sees CPUID leaf 0x8000_001d is above 0x8000_000a, > and so considers it an invaild CPUID and thus forces it to report > 0x0000_000d which is supposedly what an invalid CPUID leaf should do. > > > Net result: glibc is asking for 0x8000_001d, but getting back data > for 0x0000_000d. > > This doesn't end happily for obvious reasons, getting garbage for > the dcache sizes. > > > The 'qemu64' CPU model also gets CPUID leaf 0x8000_001d capped back > to 0x0000_000d, but crucially qemu64 lacks the 'xsave' feature bit, > so QEMU returns all-zeroes for CPUID leaf 0x0000_000d. Still not > good, but this makes glibc report 0 for DCACHE_*, which in turn > avoids tripping up the nested qemu which queries DCACHE sysconf. > > So the problem is thus more widespread than just 'max' CPU model. > > Any QEMU CPU model with vendor=AuthenticAMD and the xsave feature, > and the xlevel unset, will cause glibc to report garbage for the > L1D cache info > > Any QEMU CPU model with vendor=AuthenticAMD and without the xsave > feature, and the xlevel unset, will cause glibc to report zeroes > for L1D cache info > > Neither is good, but the latter at least doesn't trip up the > nested QEMU when it queries L1D cache info. > > I'm unsure if QEMU's behaviour is correct with calculating the > default 'xlevel' values for 'max', but I'm assuming the xlevel > was correct for Opteron_G4/5 since those are explicitly set > in the code for along time. We are tracking this as: New AMD cache size computation logic does not work for some CPUs, hypervisors <https://sourceware.org/bugzilla/show_bug.cgi?id=30428> I filed it after we resolved the earlier crashes because the data is clearly not accurate. I was also able to confirm that impacts more than just hypervisors. Sajan posted a first patch: [PATCH] x86: Fix for cache computation on AMD legacy cpus. <https://sourceware.org/pipermail/libc-alpha/2023-June/148763.html> However, it changes the reported cache sizes on some older CPUs compared to what we had before (although the values are no longer zero at least). Thanks, Florian ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) 2023-07-04 17:37 ` Florian Weimer @ 2023-07-05 13:08 ` Karumanchi, Sajan 0 siblings, 0 replies; 6+ messages in thread From: Karumanchi, Sajan @ 2023-07-05 13:08 UTC (permalink / raw) To: Florian Weimer, Daniel P.Berrangé Cc: Pierrick Bouvier, qemu-devel@nongnu.org, Richard Henderson, laurent@vivier.eu, Mallappa, Premachandra [-- Attachment #1: Type: text/plain, Size: 4158 bytes --] [AMD Official Use Only - General] ++ Prem Thanks & Regards, Sajan K. ________________________________ From: Florian Weimer <fweimer@redhat.com> Sent: Tuesday, July 4, 2023 11:07 PM To: Daniel P.Berrangé <berrange@redhat.com> Cc: Pierrick Bouvier <pierrick.bouvier@linaro.org>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; Richard Henderson <richard.henderson@linaro.org>; laurent@vivier.eu <laurent@vivier.eu>; Karumanchi, Sajan <Sajan.Karumanchi@amd.com> Subject: Re: GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding. * Daniel P. Berrangé: > On Mon, Jul 03, 2023 at 06:03:08PM +0200, Pierrick Bouvier wrote: >> Hi everyone, >> >> Recently (in d135f781 [1], between v7.0.0 and v8.0.0), qemu-user default cpu >> was updated to "max" instead of qemu32/qemu64. >> >> This change "broke" qemu self emulation if this new default cpu is used. >> >> $ ./qemu-x86_64 ./qemu-x86_64 --version >> qemu-x86_64: ../util/cacheflush.c:212: init_cache_info: Assertion `(isize & >> (isize - 1)) == 0' failed. >> qemu: uncaught target signal 6 (Aborted) - core dumped >> Aborted >> >> By setting cpu back to qemu64, it works again. >> $ ./qemu-x86_64 -cpu qemu64 ./qemu-x86_64 --version >> qemu-x86_64 version 8.0.50 (v8.0.0-2317-ge125b08ed6) >> Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers >> >> Commenting assert does not work, as qemu aligned malloc fail shortly after. >> >> I'm willing to fix it, but I'm not sure what is the issue with "max" cpu >> exactly. Is it missing CPU cache line, or something else? > > I've observed GLibC is issuing CPUID leaf 0x8000_001d > > QEMU 'max' CPU model doesn't defnie xlevel, so QEMU makes it default > to the same as min_xlevel, which is calculated to be 0x8000_000a. > > cpu_x86_cpuid() in QEMU sees CPUID leaf 0x8000_001d is above 0x8000_000a, > and so considers it an invaild CPUID and thus forces it to report > 0x0000_000d which is supposedly what an invalid CPUID leaf should do. > > > Net result: glibc is asking for 0x8000_001d, but getting back data > for 0x0000_000d. > > This doesn't end happily for obvious reasons, getting garbage for > the dcache sizes. > > > The 'qemu64' CPU model also gets CPUID leaf 0x8000_001d capped back > to 0x0000_000d, but crucially qemu64 lacks the 'xsave' feature bit, > so QEMU returns all-zeroes for CPUID leaf 0x0000_000d. Still not > good, but this makes glibc report 0 for DCACHE_*, which in turn > avoids tripping up the nested qemu which queries DCACHE sysconf. > > So the problem is thus more widespread than just 'max' CPU model. > > Any QEMU CPU model with vendor=AuthenticAMD and the xsave feature, > and the xlevel unset, will cause glibc to report garbage for the > L1D cache info > > Any QEMU CPU model with vendor=AuthenticAMD and without the xsave > feature, and the xlevel unset, will cause glibc to report zeroes > for L1D cache info > > Neither is good, but the latter at least doesn't trip up the > nested QEMU when it queries L1D cache info. > > I'm unsure if QEMU's behaviour is correct with calculating the > default 'xlevel' values for 'max', but I'm assuming the xlevel > was correct for Opteron_G4/5 since those are explicitly set > in the code for along time. We are tracking this as: New AMD cache size computation logic does not work for some CPUs, hypervisors <https://sourceware.org/bugzilla/show_bug.cgi?id=30428> I filed it after we resolved the earlier crashes because the data is clearly not accurate. I was also able to confirm that impacts more than just hypervisors. Sajan posted a first patch: [PATCH] x86: Fix for cache computation on AMD legacy cpus. <https://sourceware.org/pipermail/libc-alpha/2023-June/148763.html> However, it changes the reported cache sizes on some older CPUs compared to what we had before (although the values are no longer zero at least). Thanks, Florian [-- Attachment #2: Type: text/html, Size: 6631 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-07-05 13:42 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-07-03 16:03 qemu-user self emulation broken with default CPU on x86/x64 Pierrick Bouvier 2023-07-03 18:04 ` Daniel P. Berrangé 2023-07-03 21:05 ` GLibC AMD CPUID cache reporting regression (was Re: qemu-user self emulation broken with default CPU on x86/x64) Daniel P. Berrangé 2023-07-04 17:30 ` Pierrick Bouvier 2023-07-04 17:37 ` Florian Weimer 2023-07-05 13:08 ` Karumanchi, Sajan
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).