From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oo1-f53.google.com (mail-oo1-f53.google.com [209.85.161.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 73E99A23 for ; Tue, 27 Jun 2023 02:29:44 +0000 (UTC) Received: by mail-oo1-f53.google.com with SMTP id 006d021491bc7-55e04a83465so2907284eaf.3 for ; Mon, 26 Jun 2023 19:29:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687832983; x=1690424983; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=7s3J7mtj2DOT05cABKB4L4599zUF5rb8Gns79BnadUM=; b=nJqHWenFI/MLjK7BeJH7sTGYYzoSNt10WsYID0G02bMhy1n68WLfT3ZyHqiNFtl9qi GFOr/fTkJg2mcLBQjzczNBZ2i7V1H9NxykzsU8EZqXjlmziF2wGVCBhCyTXiE2vaBaiq sTMNAJXUnmSkGXi110SD1WrOwYYbwX9ns7dLHkJmzCXdir4ExBg3qYcYZZ6otVksIzFz uIPP+lHjhN/sFRS0la+atGEmSABIA+SqMEQ7XN3gFWeGob+nhb2fb2mUCN4LpiOCm6oG 1NRlpcEBzEFz3lP5OIAa1VrYuWnt1Nr1vc7cP7Il21y6PIF0LFpooEZ15CI/q1v6ONue 4NGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687832983; x=1690424983; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=7s3J7mtj2DOT05cABKB4L4599zUF5rb8Gns79BnadUM=; b=YOd+cx04FgYGTP/tDGhA0jluuDGdGhoq1peesKg2ImA+JEUAuJ1/XDPVtwlZX6n3MB YHE1SBglcVG7lPq/LkQz1hI4zFY4i1NK8u+ZTQwL4BOY0XVCzuTblPlZo7MApzwku5y8 V9E7/OB5RREq1eyGlb8mDAXWR2eCKPFpu789h8Q033riMm41wVqMZL+wpzr77Wgx4xJ6 6pLfPbxMfTJm9CIbSeigGaYyRn6Gq1gzL8lmEbtuHv804ytfl/P0k/510FfbAG37PnP0 OO9WsygJ/g+ovoMysJ9+n4tbPGHcgUEsaIEEwUUGv3srCWvseEuu1GIkgaTi0j2miqhG XH2w== X-Gm-Message-State: AC+VfDyEQPmFIzbksLHLwv3I67hktjEkV49B18aZEjXOMhlAPHSXT0Da tfZ+Dl5mLEflhbcMfZpefi0= X-Google-Smtp-Source: ACHHUZ6IwQ9FELesSLU/UHwcmF0hCmBoDWfKnYld0eaSg231u/frP7cFaztMIwvdsHyGV/NyTGXk0w== X-Received: by 2002:a05:6359:62f:b0:127:d8c1:b829 with SMTP id eh47-20020a056359062f00b00127d8c1b829mr20826988rwb.27.1687832983046; Mon, 26 Jun 2023 19:29:43 -0700 (PDT) Received: from debian.me ([103.131.18.64]) by smtp.gmail.com with ESMTPSA id q17-20020a62ae11000000b00679dc747738sm1759832pff.10.2023.06.26.19.29.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Jun 2023 19:29:42 -0700 (PDT) Received: by debian.me (Postfix, from userid 1000) id 92C3781BD58D; Tue, 27 Jun 2023 09:29:37 +0700 (WIB) Date: Tue, 27 Jun 2023 09:29:36 +0700 From: Bagas Sanjaya To: "Steven Rostedt (Google)" , Pablo Neira Ayuso , Jozsef Kadlecsik , Florian Westphal , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Lingutla Chandrasekhar , Frederic Weisbecker , "J. Avila" , Vivek Anand , "Rafael J. Wysocki" , Thomas Renninger , Shuah Khan , Peter Zijlstra , Borislav Petkov , Josh Poimboeuf Cc: Linux Kernel Mailing List , Linux Regressions , Linux Netfilter Development , Netfilter Core Developers , Linux Networking , Linux Power Management Subject: Re: Fwd: High cpu usage caused by kernel process when upgraded to linux 5.19.17 or later Message-ID: References: <01ac399d-f793-49d4-844b-72cd8e0034df@gmail.com> Precedence: bulk X-Mailing-List: regressions@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="90Q4kbwiz0bo494z" Content-Disposition: inline In-Reply-To: <01ac399d-f793-49d4-844b-72cd8e0034df@gmail.com> --90Q4kbwiz0bo494z Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 23, 2023 at 07:58:51AM +0700, Bagas Sanjaya wrote: > Hi, >=20 > I notice a regression report on Bugzilla [1]. Quoting from it: >=20 > > kernel process "kworker/events_power_efficient" uses a lot of cpu power= (100% on ESXI 6.7, ~30% on ESXI 7.0U3 or later) after upgrading from 5.17.= 3 to 5.19.17 or later. > >=20 > > dmesg log: > > [ 2430.973102] > > [ 2430.973131] Sending NMI from CPU 1 to CPUs 0: > > [ 2430.973241] NMI backtrace for cpu 0 > > [ 2430.973247] CPU: 0 PID: 22 Comm: kworker/0:1 Not tainted 6.3.3 #1 > > [ 2430.973254] Hardware name: VMware, Inc. VMware Virtual Platform/440B= X Desktop Reference Platform, BIOS 6.00 11/12/2020 > > [ 2430.973258] Workqueue: events_power_efficient htable_gc [xt_hashlimi= t] > > [ 2430.973275] RIP: 0010:preempt_count_sub+0x2e/0xa0 > > [ 2430.973289] Code: 36 01 85 c9 75 1b 65 8b 15 a7 da f8 5e 89 d1 81 e1= ff ff ff 7f 39 f9 7c 16 81 ff fe 00 00 00 76 3b f7 df 65 01 3d 8a da f8 5e= cc cc cc cc e8 98 aa 25 00 85 c0 74 f2 8b 15 da 71 ed 00 85 d2 > > [ 2430.973294] RSP: 0018:ffffb15ec00dbe58 EFLAGS: 00000297 > > [ 2430.973299] RAX: 0000000000000000 RBX: ffffb15ec12ad000 RCX: 0000000= 000000001 > > [ 2430.973302] RDX: 0000000080000001 RSI: ffffffffa1c3313b RDI: 0000000= 0ffffffff > > [ 2430.973306] RBP: dead000000000122 R08: 0000000000000010 R09: 0000746= e65696369 > > [ 2430.973309] R10: 8080808080808080 R11: 0000000000000018 R12: 0000000= 000000000 > > [ 2430.973312] R13: 0000000000001e2b R14: ffffb15ec12ad048 R15: ffff91c= 279c26a05 > > [ 2430.973316] FS: 0000000000000000(0000) GS:ffff91c279c00000(0000) kn= lGS:0000000000000000 > > [ 2430.973320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 2430.973324] CR2: 000055fc138890e0 CR3: 000000010810e002 CR4: 0000000= 0001706f0 > > [ 2430.973374] Call Trace: > > [ 2430.973388] > > [ 2430.973390] __local_bh_enable_ip+0x32/0x70 > > [ 2430.973413] htable_selective_cleanup+0x95/0xc0 [xt_hashlimit] > > [ 2430.973428] htable_gc+0xf/0x30 [xt_hashlimit] > > [ 2430.973440] process_one_work+0x1d4/0x360 > > [ 2430.973459] ? process_one_work+0x360/0x360 > > [ 2430.973467] worker_thread+0x25/0x3b0 > > [ 2430.973476] ? process_one_work+0x360/0x360 > > [ 2430.973483] kthread+0xe1/0x110 > > [ 2430.973499] ? kthread_complete_and_exit+0x20/0x20 > > [ 2430.973507] ret_from_fork+0x1f/0x30 > > [ 2430.973526] >=20 > See Bugzilla for the full thread and perf output. >=20 > Anyway, I'm tracking it in regzbot so that it doesn't fall through > cracks unnoticed: >=20 > #regzbot introduced: v5.17.3..v5.19.17 https://bugzilla.kernel.org/show_b= ug.cgi?id=3D217586 > #regzbot title: kworker/events_power_efficient utilizes full CPU power af= ter kernel upgrade >=20 The reporter had found the culprit (see Bugzilla for more information), thus telling regzbot: #regzbot introduced: 6ad0ad2bf8a67e #regzbot title: retbleed reporting causes high cpu utilization due to kwork= er/events_power_efficient #regzbot link: https://lore.kernel.org/all/PH0PR05MB8448A203A909959FAC754B7= AAF439@PH0PR05MB8448.namprd05.prod.outlook.com/ #regzbot link: 6ad0ad2bf8a67e Thanks. --=20 An old man doll... just what I always wanted! - Clara --90Q4kbwiz0bo494z Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEABYKAB0WIQSSYQ6Cy7oyFNCHrUH2uYlJVVFOowUCZJpJiwAKCRD2uYlJVVFO o1jRAP4q8brZctIPnGZqlJZqlJEMRnc4sRukxEX4ieIUH5Xn0QEA1g/dJoM/iQbR z2LBNutalMJVqVv8mRZOw176rScpTw0= =xdMq -----END PGP SIGNATURE----- --90Q4kbwiz0bo494z--