Re: Slow (higher scheduling latency?) system after S3 - regression 4.17 -> 4.19 ?

["Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2354 bytes --]

On Tue, Apr 07, 2026 at 08:14:42AM +0200, Jan Beulich wrote:
> On 03.04.2026 15:55, Marek Marczykowski-Górecki wrote:
> > On Thu, Apr 02, 2026 at 03:02:15PM +0200, Marek Marczykowski-Górecki wrote:
> >> On Fri, Mar 27, 2026 at 01:47:55PM +0100, Marek Marczykowski-Górecki wrote:
> >>> Some further observation, this time regarding timers:
> >>
> >> In the meantime Roger suggested it might be about C-states. So, I just
> >> tried booting with max_cstate=0. I got this:
> > 
> > We had a design session on this issue, and managed to find (thanks Roger!)
> > that adding:
> > 
> >     tsc_mode='always_emulate'
> > 
> > to the domU config fixes the issue. Now, the question is how to fix it
> > properly...
> 
> Seeing also the remarks on Matrix, what we will hence need to figure is what
> TSC parameters (of the system and/or the VM) change across suspend. As per
> Matrix everything looked at so far was apparently stable.

I added some more logging (based on 
https://sandpile.org/x86/msr.htm | grep TSC) - see attached debug patch.
The output before and after S3:

    ==> msr-before.txt <==
    (XEN) [  118.910886] MSR 00000010: 000000cd6d18d6c1
    (XEN) [  118.911110] MSR 0000003b: 0000000000000000
    (XEN) [  118.911331] MSR c0000103: 0000000000000000
    (XEN) [  118.911559] Fixup #GP[0000]: ffff82d04029ad8d [arch/x86/cpu/intel.c#debug_tsc+0x76/0x84] -> ffff82d04029ad83
    (XEN) [  118.912029] MSR 000006e0: 0000000000000000
    (XEN) [  118.912253] MSR 000000e7: 000000064cf8ec5e
    (XEN) [  118.912473] MSR 000000e8: 000000065d09c019
    (XEN) [  118.912687] MSR 0000064e: 000000052ec83116


    ==> msr-after.txt <==
    (XEN) [  146.396099] MSR 00000010: 00000012fac68ccd
    (XEN) [  146.396329] MSR 0000003b: 0000000000000000
    (XEN) [  146.396553] MSR c0000103: 0000000000000000
    (XEN) [  146.396777] Fixup #GP[0000]: ffff82d04029ad8d [arch/x86/cpu/intel.c#debug_tsc+0x76/0x84] -> ffff82d04029ad83
    (XEN) [  146.397248] MSR 000006e0: 0000000000000000
    (XEN) [  146.397470] MSR 000000e7: 00000001e41a578a
    (XEN) [  146.397690] MSR 000000e8: 000000017dc5b211
    (XEN) [  146.397909] MSR 0000064e: 000000014eddbe8f

Nothing interesting here...

What other values would make sense to compare?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #1.2: 0001-DEBUG-TSC.patch --]
[-- Type: text/plain, Size: 1697 bytes --]

From 2d1bd9c15e605136c7817fe824a7c429cbfb137d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Marek=20Marczykowski-G=C3=B3recki?=
 <marmarek@invisiblethingslab.com>
Date: Tue, 7 Apr 2026 13:55:41 +0200
Subject: [PATCH] DEBUG TSC

---
 xen/arch/x86/cpu/intel.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/xen/arch/x86/cpu/intel.c b/xen/arch/x86/cpu/intel.c
index 51a3d1c4b5f3..846c6a704a3f 100644
--- a/xen/arch/x86/cpu/intel.c
+++ b/xen/arch/x86/cpu/intel.c
@@ -4,6 +4,7 @@
 #include <xen/sched.h>
 #include <xen/smp.h>
 #include <xen/string.h>
+#include <xen/keyhandler.h>
 
 #include <asm/apic.h>
 #include <asm/i387.h>
@@ -675,3 +676,30 @@ void __init intel_init_arat(void)
     if ( opt_arat && cpu_has_arat )
         setup_force_cpu_cap(X86_FEATURE_XEN_ARAT);
 }
+
+static void cf_check debug_tsc(unsigned char key)
+{
+    uint64_t val;
+    int msr_idx[] = { 0x10,         // TSC
+                      0x3b,         // TSC_ADJUST
+                      0xc0000103,   // TSC_AUX
+                      0xc0000104,   // TSC_RATIO
+                      0x06e0,       // TSC_DEADLINE
+                      0xe7,         // MPERF
+                      0xe8,         // APERF
+                      0x64e,        // PPERF
+                      0 };
+
+    for ( int i = 0; msr_idx[i]; i++ )
+    {
+        if ( !rdmsr_safe(msr_idx[i], &val) )
+            printk("MSR %08x: %016lx\n", msr_idx[i], val);
+    }
+}
+
+static int __init cf_check register_debug_tsc(void)
+{
+    register_keyhandler('1', debug_tsc, "Extra TSC debug", false);
+    return 0;
+}
+__initcall(register_debug_tsc);
-- 
2.53.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]