From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 761E5D3941D for ; Thu, 2 Apr 2026 13:02:46 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.1271874.1559868 (Exim 4.92) (envelope-from ) id 1w8Hgk-0001D1-M4; Thu, 02 Apr 2026 13:02:22 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 1271874.1559868; Thu, 02 Apr 2026 13:02:22 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1w8Hgk-0001Cu-JH; Thu, 02 Apr 2026 13:02:22 +0000 Received: by outflank-mailman (input) for mailman id 1271874; Thu, 02 Apr 2026 13:02:20 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1w8Hgi-0001CY-9G for xen-devel@lists.xenproject.org; Thu, 02 Apr 2026 13:02:20 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1w8Hgh-008C1P-1f for xen-devel@lists.xenproject.org; Thu, 02 Apr 2026 15:02:19 +0200 Received: from [10.42.69.12] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 69ce68d8-5cb7-0a2a0a5109dd-0a2a450c930a-16 for ; Thu, 02 Apr 2026 15:02:18 +0200 Received: from [103.168.172.152] (helo=fhigh-a1-smtp.messagingengine.com) by tlsNG-d25034.mxtls.expurgate.net with ESMTPS (eXpurgate 4.56.0) (envelope-from ) id 69ce68d9-f40c-0a2a450c0019-67a8ac98be9d-3 for ; Thu, 02 Apr 2026 15:02:18 +0200 Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfhigh.phl.internal (Postfix) with ESMTP id 319DE14000A4; Thu, 2 Apr 2026 09:02:17 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-02.internal (MEProxy); Thu, 02 Apr 2026 09:02:17 -0400 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 2 Apr 2026 09:02:16 -0400 (EDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=fm1 header.d=invisiblethingslab.com header.i="@invisiblethingslab.com" header.h="Cc:Content-Type:Date:From:In-Reply-To:Message-ID:MIME-Version:References:Subject:To"; dkim=pass header.s=fm2 header.d=messagingengine.com header.i="@messagingengine.com" header.h="Cc:Content-Type:Date:Feedback-ID:From:In-Reply-To:Message-ID:MIME-Version:References:Subject:To:X-ME-Proxy:X-ME-Sender" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm1; t=1775134937; x=1775221337; bh=ym2eMeFIoCCsFKppruamwrkn//7rVgTN9+G1lJzgWJY=; b= PcjLSVGIg//O9ZtDedZ8Sm2/vvgO1KrCfzTcl4XZ+G+PNtQfGSdHS2U6qQ7xz/8m jX1MW91du/DL0j9MzyptdBITcalDp6UFTx2O1zzI3j5XkQmSVlqLTFVPyLI2E/oH GRhcjABMRgGcRlc+5FzdFw+KbC6XmjQG/PsmUEAEOVGDCdi47W4jM9xPzIMbLyr4 bzrrFvGYYkgYHExZ1XdQq24E4dwwHX3phePjnatA2zEbV5MgEzUiP6iWcaIQy4bq 1N+486DUDQW1URhf0YB6YDan0O1/bOZzFFX2qRaqMDNA0OrHw+qlwAfJYhl6Dro2 k75eo0AJFksTcVcxeTWbxQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1775134937; x=1775221337; bh=ym2eMeFIoCCsFKppruamwrkn//7rVgTN9+G 1lJzgWJY=; b=CNc8RMcKq9X20+Pz2d5UpoNOnts65Q6T6c9LxXzYxModTUq6+t0 NklTrTU9NzGbt6H1ZYC+E77TgXSbCbeX81Uxj0n7WU3czJtNPfawOM+7tGlpSq07 Yz2YvOuRZKp7BOEO3wSJErnos6KP5Kbga8fpNGo2bFueD8GHX0fO/vcA+YJcamdb XkoFF/9Y7P4rAMXKUxoFqh3rsEfviwzUni5dpKGR69FgMh/PSP/o+2ioXAc4FZG4 Mc123cTsLIAZ7/LBDpSyGiK8A6u5KONnUgoMswqlLXpq2kFjsByuX1yaxfer7diz 9EFn6WIwHHnuMCInQxrB4iLaet8+G/FKHXQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefhedrtddtgdeiuddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceurghi lhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujfgurh epfffhvfevuffkfhggtggujgesghdtreertddtjeenucfhrhhomhepofgrrhgvkhcuofgr rhgtiiihkhhofihskhhiqdfikphrvggtkhhiuceomhgrrhhmrghrvghksehinhhvihhsih gslhgvthhhihhnghhslhgrsgdrtghomheqnecuggftrfgrthhtvghrnhepgfduleetfeev hfefheeiteeliefhjefhleduveetteekveettddvgeeuteefjedunecuvehluhhsthgvrh fuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghksehinhhv ihhsihgslhgvthhhihhnghhslhgrsgdrtghomhdpnhgspghrtghpthhtohepgedpmhhoug gvpehsmhhtphhouhhtpdhrtghpthhtohepjhgsvghulhhitghhsehsuhhsvgdrtghomhdp rhgtphhtthhopeigvghnqdguvghvvghlsehlihhsthhsrdigvghnphhrohhjvggtthdroh hrghdprhgtphhtthhopehjghhrohhsshesshhushgvrdgtohhmpdhrtghpthhtoheprhho ghgvrhdrphgruhestghithhrihigrdgtohhm X-ME-Proxy: Feedback-ID: i1568416f:Fastmail Date: Thu, 2 Apr 2026 15:02:15 +0200 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Jan Beulich Cc: xen-devel , =?utf-8?B?SsO8cmdlbiBHcm/Dnw==?= , Roger Pau =?utf-8?B?TW9ubsOp?= Subject: Re: Slow (higher scheduling latency?) system after S3 - regression 4.17 -> 4.19 ? Message-ID: References: <29775afa-e17b-4c44-a2a8-4ef4bc7cdf0f@suse.com> <88c65ec7-b7f7-4d8b-a453-d707440b43cb@suse.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="fej7YXlfo2xZKE7n" Content-Disposition: inline In-Reply-To: X-purgate-ID: tlsNG-d25034/1775134938-8254FA3D-92A60490/0/0 X-purgate-type: clean X-purgate-size: 9770 --fej7YXlfo2xZKE7n Content-Type: text/plain; protected-headers=v1; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Date: Thu, 2 Apr 2026 15:02:15 +0200 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Jan Beulich Cc: xen-devel , =?utf-8?B?SsO8cmdlbiBHcm/Dnw==?= , Roger Pau =?utf-8?B?TW9ubsOp?= Subject: Re: Slow (higher scheduling latency?) system after S3 - regression 4.17 -> 4.19 ? On Fri, Mar 27, 2026 at 01:47:55PM +0100, Marek Marczykowski-G=C3=B3recki w= rote: > Some further observation, this time regarding timers: In the meantime Roger suggested it might be about C-states. So, I just tried booting with max_cstate=3D0. I got this: [2026-04-02 11:41:57] (XEN) [ 4.327519] mwait-idle: disabled But then xenpm reports C1 is still used: # xenpm get-cpuidle-states=20 Max possible C-state: C0 cpu id : 0 total C-states : 2 idle time(ms) : 3727115 C0 : transition [ 0] residency [ 3769835 ms] C1 : transition [ 0] residency [ 0 ms] cpu id : 1 total C-states : 2 idle time(ms) : 3763654 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769838 ms] cpu id : 2 total C-states : 2 idle time(ms) : 3729168 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769838 ms] cpu id : 3 total C-states : 2 idle time(ms) : 3750226 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769840 ms] cpu id : 4 total C-states : 2 idle time(ms) : 3726369 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769840 ms] cpu id : 5 total C-states : 2 idle time(ms) : 3755733 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769840 ms] cpu id : 6 total C-states : 2 idle time(ms) : 3725747 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769841 ms] cpu id : 7 total C-states : 2 idle time(ms) : 3730190 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769841 ms] cpu id : 8 total C-states : 2 idle time(ms) : 3716485 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769841 ms] cpu id : 9 total C-states : 2 idle time(ms) : 3710830 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769842 ms] cpu id : 10 total C-states : 2 idle time(ms) : 3704881 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769842 ms] cpu id : 11 total C-states : 2 idle time(ms) : 3719333 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769842 ms] cpu id : 12 total C-states : 2 idle time(ms) : 3715059 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769842 ms] cpu id : 13 total C-states : 2 idle time(ms) : 3711681 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769843 ms] cpu id : 14 total C-states : 2 idle time(ms) : 3709380 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769843 ms] cpu id : 15 total C-states : 2 idle time(ms) : 3720119 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769843 ms] cpu id : 16 total C-states : 2 idle time(ms) : 3737637 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769844 ms] cpu id : 17 total C-states : 2 idle time(ms) : 3737662 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769844 ms] cpu id : 18 total C-states : 2 idle time(ms) : 3729870 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769844 ms] cpu id : 19 total C-states : 2 idle time(ms) : 3745564 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769845 ms] cpu id : 20 total C-states : 2 idle time(ms) : 3717089 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769845 ms] cpu id : 21 total C-states : 2 idle time(ms) : 3590257 C0 : transition [ 1] residency [ 0 ms] C1 : transition [ 1] residency [ 3769845 ms] And the issue still happens... Honestly, I'm confused about the above output. The system is mostly idle, but I wouldn't expect it to be _that_ idle - there is xenstored + xenconsoled running, plus occasional xl create for testing with short-lived domU. Does it really all happen only on CPU0 here? --=20 Best Regards, Marek Marczykowski-G=C3=B3recki Invisible Things Lab --fej7YXlfo2xZKE7n Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmnOaNcACgkQ24/THMrX 1yyp+AgAj4GAG/xT8PTinO9+JGlXbFwmYWi9ykBHcTslns7tHSyNirLLKjlnA1sP TIboxm6BrjOxHIdpQJK5AXfSGVX2Lms3JFCW1J235h7eKSqh+XxVhYR9hzRNTQBo G1FxX3vd00WMx+AoWC2GCCK124j/Yha5Fw8WRbIC4ba1SkluScUmlCytstUkp+Sj c1GBYjaDgOIn2DXnXVxjSwz2QBb92MpJdMQ+RaICtYEbaXGGIfAd4yaBdIdtIYT+ S3ajEsLVWJ8lMHKWF4ejEaWUpc6rkgVkFzslaLz8Y2eUEIfsfpZ+yl+vrkguflaf KRvz9J5Iab4ZoR0Eh3qBqJnaFihz8Q== =HOe2 -----END PGP SIGNATURE----- --fej7YXlfo2xZKE7n--