From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 82F36EDF04B for ; Thu, 12 Feb 2026 07:03:36 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4fBR924hhwz2yFm; Thu, 12 Feb 2026 18:03:34 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1770879814; cv=none; b=m0UtTt3U43oq3HOZeljMQdDU7ZPI1+UDhwdEciV6gbpdEkgnszOKSvPul/SGUfhJfDG3RTbsays9EAaRSnH1btnz+UePg083+Q5K8zT91pleX57X1XTsde68nYkYg4Ikt2aKsWaN/vMJaF/Q5cW7cZNA0epSd7X30P6rMT3lIo4v7bzoywByl4+v3e5Bc8/0ce9wLMzX/ptn2Vm3gUbDOHl8v6PPShVsFbilit6VOa1Jsg5F/fkhW8257gDAaa85la4/gTEnN78YtoiNZIsiLJE1BB5AJdagXeXgSnOYV2qiojS8N/khHUVaWOZPeOoTtPi/deqKHy3Jh4zQOnHawA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1770879814; c=relaxed/relaxed; bh=poZjhNXE0/uh1v8vuuLiSQGegGURE+K+YLsaDlo9OTg=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=S3ob1EfYnszZ+kq3IcLN4Rc6We9f1zApANBAzvBgvNXzJwTLI5z5z+Xe8mUiy+eUnojeXZU7Uo2hcS2+4nlsWG2IqfbISAoX4UhluWfwYjqGP/p6Ejd/wznpFqV2g9W3qlO6pY9UQdJZMeMRuoKyud5W01iY0qVQN87TprECXDojarvzyWMpHQUSr92a3OSs/8ysOSOGwRXW2GL5DO/KG38S2oueCxIzZmWsHbkpzUBgOxvqQUsUzcfcNCIxHWVzj2Yx26KQyfGbiv6TUwOSIRQ9GLY6IBKNEkukdK2mD8/n3dgEDjFV0Hr4SSo9pK4pxAMGJw6121tHhUoiIJh0ig== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=HGR7OYn0; dkim-atps=neutral; spf=pass (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=sshegde@linux.ibm.com; receiver=lists.ozlabs.org) smtp.mailfrom=linux.ibm.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=HGR7OYn0; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=sshegde@linux.ibm.com; receiver=lists.ozlabs.org) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4fBR911jqrz2xnZ for ; Thu, 12 Feb 2026 18:03:33 +1100 (AEDT) Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 61C1Ose0343472; Thu, 12 Feb 2026 07:02:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=poZjhN XE0/uh1v8vuuLiSQGegGURE+K+YLsaDlo9OTg=; b=HGR7OYn0Otva5nK9+YD49W hOsn2HBVXfQSpoPiOGcHve93HlE6sAO2IkHNlV24LD70O1wbk/Y5QRfnusrwESDo rhgpYH6ImMMQA0rG7Vl3ACyeTxnNs3MreDJYM9TTIBE6VUkd2pVLSgaw9hz0EZg+ QknFPjOTAtyCJT6I5zp4tywTmLhOqxYIGRJJ34fpAhZ09sddzq1gbM4cbqGPY8YH oBDzOAaJwpmJnJRzqBh66oxPIjqsB6Ch4CiPRiKCwjeYrH25/MlVBNJRqMhYBBzL 8aYPgPLZRJmu7vp4VRJAf+e98ZtHAM1hAWuhRMPeIJbmBxFG4QXEU1poCyfL1COw == Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4c696umvdd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 12 Feb 2026 07:02:50 +0000 (GMT) Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 61C3v4pY012996; Thu, 12 Feb 2026 07:02:48 GMT Received: from smtprelay06.fra02v.mail.ibm.com ([9.218.2.230]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4c6h7kh5p8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 12 Feb 2026 07:02:48 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay06.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 61C72iYG13959476 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 12 Feb 2026 07:02:44 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6E66B20040; Thu, 12 Feb 2026 07:02:44 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 259D32004E; Thu, 12 Feb 2026 07:02:37 +0000 (GMT) Received: from [9.109.215.252] (unknown [9.109.215.252]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Thu, 12 Feb 2026 07:02:36 +0000 (GMT) Message-ID: <3b71bdb1-2dbe-420f-8fae-5cb1ab3a4ba9@linux.ibm.com> Date: Thu, 12 Feb 2026 12:32:36 +0530 X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 00/15 v2] tick/sched: Refactor idle cputime accounting To: Frederic Weisbecker Cc: LKML , Vasily Gorbik , Vincent Guittot , Kieran Bingham , Ingo Molnar , Xin Zhao , Joel Fernandes , Neeraj Upadhyay , Sven Schnelle , Boqun Feng , Mel Gorman , Dietmar Eggemann , Ben Segall , Michael Ellerman , "Rafael J. Wysocki" , "Paul E . McKenney" , Anna-Maria Behnsen , Alexander Gordeev , Madhavan Srinivasan , linux-s390@vger.kernel.org, Jan Kiszka , Juri Lelli , "Christophe Leroy (CS GROUP)" , linux-pm@vger.kernel.org, Uladzislau Rezki , Peter Zijlstra , Steven Rostedt , Thomas Gleixner , Nicholas Piggin , Heiko Carstens , linuxppc-dev@lists.ozlabs.org, Christian Borntraeger , Valentin Schneider , Viresh Kumar References: <20260206142245.58987-1-frederic@kernel.org> Content-Language: en-US From: Shrikanth Hegde In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-Reinject: loops=2 maxloops=12 X-Authority-Analysis: v=2.4 cv=YZiwJgRf c=1 sm=1 tr=0 ts=698d7b1a cx=c_pps a=3Bg1Hr4SwmMryq2xdFQyZA==:117 a=3Bg1Hr4SwmMryq2xdFQyZA==:17 a=IkcTkHD0fZMA:10 a=HzLeVaNsDn8A:10 a=VkNPw1HP01LnGYTKEx00:22 a=Mpw57Om8IfrbqaoTuvik:22 a=GgsMoib0sEa3-_RKJdDe:22 a=VwQbUJbxAAAA:8 a=Byx-y9mGAAAA:8 a=jna3-dR-LpdznfkkQtQA:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 X-Proofpoint-ORIG-GUID: ykNMXiJuLMZvY8SHVx1k92NnfL__mZAj X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwMjEyMDA0OCBTYWx0ZWRfXwf0I6im/SFoz UOZhM4tcMuDmndSC7aO0LIN6Sbg1f18jEaekOgqBBncr3g30fHinniII7kb9tqkFq3dwDbqpa0R iQazWnZ1+AcwvTuf8gZQMhXgpGv4dEGT8N55h0QfhCOiW+jU2yAZkMtivl+nRFys0k0iLwIEtYa PjICtoUofjQNMwydGqQ8yS+hm1uVNrjk5bSOFneE8/KGySNj4A9Rp9Rj/TxwCCFtjBX9besOB13 fMIoo0FCrMM3upSaXNejT4ZOyS9GozjE+sJF7a2AE2h3RLQNuac5sGJpKh7+Ws7ETLUCdT56awV V2ITTAHZ5B9LC8Xcbq7iUdYQLWJsTZboEoUhgHc+1mq+XliqFe/o1SLUJ4bEUvkezdYQ5XWdRDK zx/OO7DBfS6fS09IPkWzH8gwmKWhv3MFhTgycMDvo9MhnJYj0gPttfpTFQaLbFdVq2CfMwQPuf9 vVmPxM41fwgtNlUTDCQ== X-Proofpoint-GUID: NCg8jEjEq3HhF8Orr5avhxKfyd3mZDhp X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1121,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-02-12_02,2026-02-11_04,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 spamscore=0 impostorscore=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 adultscore=0 clxscore=1015 lowpriorityscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2601150000 definitions=main-2602120048 On 2/11/26 10:36 PM, Frederic Weisbecker wrote: > Le Wed, Feb 11, 2026 at 07:13:45PM +0530, Shrikanth Hegde a écrit : >> Hi Frederic, >> Gave this series a spin on the same system as v1. >> >> On 2/6/26 7:52 PM, Frederic Weisbecker wrote: >>> Hi, >>> >>> After the issue reported here: >>> >>> https://lore.kernel.org/all/20251210083135.3993562-1-jackzxcui1989@163.com/ >>> >>> It occurs that the idle cputime accounting is a big mess that >>> accumulates within two concurrent statistics, each having their own >>> shortcomings: >>> >>> * The accounting for online CPUs which is based on the delta between >>> tick_nohz_start_idle() and tick_nohz_stop_idle(). >>> >>> Pros: >>> - Works when the tick is off >>> >>> - Has nsecs granularity >>> >>> Cons: >>> - Account idle steal time but doesn't substract it from idle >>> cputime. >>> >>> - Assumes CONFIG_IRQ_TIME_ACCOUNTING by not accounting IRQs but >>> the IRQ time is simply ignored when >>> CONFIG_IRQ_TIME_ACCOUNTING=n >>> >>> - The windows between 1) idle task scheduling and the first call >>> to tick_nohz_start_idle() and 2) idle task between the last >>> tick_nohz_stop_idle() and the rest of the idle time are >>> blindspots wrt. cputime accounting (though mostly insignificant >>> amount) >>> >>> - Relies on private fields outside of kernel stats, with specific >>> accessors. >>> >>> * The accounting for offline CPUs which is based on ticks and the >>> jiffies delta during which the tick was stopped. >>> >>> Pros: >>> - Handles steal time correctly >>> >>> - Handle CONFIG_IRQ_TIME_ACCOUNTING=y and >>> CONFIG_IRQ_TIME_ACCOUNTING=n correctly. >>> >>> - Handles the whole idle task >>> >>> - Accounts directly to kernel stats, without midlayer accumulator. >>> >>> Cons: >>> - Doesn't elapse when the tick is off, which doesn't make it >>> suitable for online CPUs. >>> >>> - Has TICK_NSEC granularity (jiffies) >>> >>> - Needs to track the dyntick-idle ticks that were accounted and >>> substract them from the total jiffies time spent while the tick >>> was stopped. This is an ugly workaround. >>> >>> Having two different accounting for a single context is not the only >>> problem: since those accountings are of different natures, it is >>> possible to observe the global idle time going backward after a CPU goes >>> offline, as reported by Xin Zhao. >>> >>> Clean up the situation with introducing a hybrid approach that stays >>> coherent, fixes the backward jumps and works for both online and offline >>> CPUs: >>> >>> * Tick based or native vtime accounting operate before the tick is >>> stopped and resumes once the tick is restarted. >>> >>> * When the idle loop starts, switch to dynticks-idle accounting as is >>> done currently, except that the statistics accumulate directly to the >>> relevant kernel stat fields. >>> >>> * Private dyntick cputime accounting fields are removed. >>> >>> * Works on both online and offline case. >>> >>> * Move most of the relevant code to the common sched/cputime subsystem >>> >>> * Handle CONFIG_IRQ_TIME_ACCOUNTING=n correctly such that the >>> dynticks-idle accounting still elapses while on IRQs. >>> >>> * Correctly substract idle steal cputime from idle time >>> >>> Changes since v1: >>> >>> - Fix deadlock involving double seq count lock on idle >>> >>> - Fix build breakage on powerpc >>> >>> - Fix build breakage on s390 (Heiko) >>> >>> - Fix broken sysfs s390 idle time file (Heiko) >>> >>> - Convert most ktime usage here into u64 (Peterz) >>> >>> - Add missing (or too implicit) (Peterz) >>> >>> - Fix whole idle time acccounting breakage due to missing TS_FLAG_ set >>> on idle entry (Shrikanth Hegde) >>> >>> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git >>> timers/core-v2 >>> >>> HEAD: 21458b98c80a0567d48131240317b7b73ba34c3c >>> Thanks, >>> Frederic >> >> idle and runtime utilization with mpstat while running stress-ng looks >> correct now. >> >> However, when running hackbench I am noticing the below data. hackbench shows >> severe regressions. >> >> base: tip/master at 9c61ebbdb587a3950072700ab74a9310afe3ad73. >> (nit: patch 7 is already part of tip. so skipped applying it) >> +-----------------------------------------------+-------+---------+-----------+ >> | Test | base | +series | % Diff | >> +-----------------------------------------------+-------+---------+-----------+ >> | HackBench Process 10 groups | 2.23 | 3.05 | -36.77% | >> | HackBench Process 20 groups | 4.17 | 5.82 | -39.57% | >> | HackBench Process 30 groups | 6.04 | 8.49 | -40.56% | >> | HackBench Process 40 groups | 7.90 | 11.10 | -40.51% | >> | HackBench thread 10 | 2.44 | 3.36 | -37.70% | >> | HackBench thread 20 | 4.57 | 6.35 | -38.95% | >> | HackBench Process(Pipe) 10 | 1.76 | 2.29 | -30.11% | >> | HackBench Process(Pipe) 20 | 3.49 | 4.76 | -36.39% | >> | HackBench Process(Pipe) 30 | 5.21 | 7.13 | -36.85% | >> | HackBench Process(Pipe) 40 | 6.89 | 9.31 | -35.12% | >> | HackBench thread(Pipe) 10 | 1.91 | 2.50 | -30.89% | >> | HackBench thread(Pipe) 20 | 3.74 | 5.16 | -37.97% | >> +-----------------------------------------------+-------+---------+-----------+ >> >> I have these in .config and I don't have nohz_full or isolated cpus. >> >> CONFIG_TICK_ONESHOT=y >> CONFIG_NO_HZ_COMMON=y >> # CONFIG_HZ_PERIODIC is not set >> # CONFIG_NO_HZ_IDLE is not set >> CONFIG_NO_HZ_FULL=y >> >> # CPU/Task time and stats accounting >> # >> CONFIG_VIRT_CPU_ACCOUNTING=y >> CONFIG_VIRT_CPU_ACCOUNTING_GEN=y >> CONFIG_IRQ_TIME_ACCOUNTING=y >> CONFIG_HAVE_SCHED_AVG_IRQ=y >> >> I did a git bisect and below is what it says. >> >> git bisect start >> # status: waiting for both good and bad commits >> # bad: [6821315886a3b5267ea31d29dba26fd34647fbbc] sched/cputime: Handle dyntick-idle steal time correctly >> git bisect bad 6821315886a3b5267ea31d29dba26fd34647fbbc >> # status: waiting for good commit(s), bad commit known >> # good: [9c61ebbdb587a3950072700ab74a9310afe3ad73] Merge branch into tip/master: 'x86/sev' >> git bisect good 9c61ebbdb587a3950072700ab74a9310afe3ad73 >> # good: [dc8bb3c84d162f7d9aa6becf9f8392474f92655a] tick/sched: Remove nohz disabled special case in cputime fetch >> git bisect good dc8bb3c84d162f7d9aa6becf9f8392474f92655a >> # good: [5070a778a581cd668f5d717f85fb22b078d8c20c] tick/sched: Account tickless idle cputime only when tick is stopped >> git bisect good 5070a778a581cd668f5d717f85fb22b078d8c20c >> # bad: [1e0ccc25a9a74b188b239c4de716fde279adbf8e] sched/cputime: Provide get_cpu_[idle|iowait]_time_us() off-case >> git bisect bad 1e0ccc25a9a74b188b239c4de716fde279adbf8e >> # bad: [ee7c735b76071000d401869fc2883c451ee3fa61] tick/sched: Consolidate idle time fetching APIs >> git bisect bad ee7c735b76071000d401869fc2883c451ee3fa61 >> # first bad commit: [ee7c735b76071000d401869fc2883c451ee3fa61] tick/sched: >> Consolidate idle time fetching APIs > > I see. Can you try this? (or fetch timers/core-v3 from my tree) > Perhaps that mistake had some impact on cpufreq. > > diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c > index 057fdc00dbc6..08550a6d9469 100644 > --- a/kernel/sched/cputime.c > +++ b/kernel/sched/cputime.c > @@ -524,7 +524,7 @@ static u64 get_cpu_sleep_time_us(int cpu, enum cpu_usage_stat idx, > do_div(res, NSEC_PER_USEC); > > if (last_update_time) > - *last_update_time = res; > + *last_update_time = ktime_to_us(now); > > return res; > } > Yes. This diff helps. Now the data is almost same. +-----------------------------------------------+-------+-------+-----------+ | Test | base | series+ | % Diff | | | | +above diff | +-----------------------------------------------+-------+-------------+-----------+ | HackBench Process 10 groups | 2.23 | 2.25 | -0.90% | | HackBench Process 20 groups | 4.17 | 4.21 | -0.96% | | HackBench Process 30 groups | 6.04 | 6.15 | -1.82% | | HackBench Process 40 groups | 7.90 | 8.06 | -2.03% | | HackBench thread 10 | 2.44 | 2.46 | -0.82% | | HackBench thread 20 | 4.57 | 4.61 | -0.88% | | HackBench Process(Pipe) 10 | 1.76 | 1.73 | 1.70% | | HackBench Process(Pipe) 20 | 3.49 | 3.50 | -0.29% | | HackBench Process(Pipe) 30 | 5.21 | 5.22 | -0.19% | | HackBench Process(Pipe) 40 | 6.89 | 6.96 | -1.02% | | HackBench thread(Pipe) 10 | 1.91 | 1.88 | 1.57% | | HackBench thread(Pipe) 20 | 3.74 | 3.81 | -1.87% | +-----------------------------------------------+-------+-------------+-----------+