From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 12E9D2D2398; Fri, 21 Nov 2025 16:35:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763742957; cv=none; b=f2C72Oa5/jBi2TvEKrAUtYyK8sa4w1ieH+YoT8q32pJtuWrASef9XakpImYIe8hnKjjDwkmRwDsN6kAj54oe+nXdwddSAHzfazSkbrY8o4QyP5CYTobl+hNNvNS1iCbG3kBWddh+cNxeNwGYJN/J2qWE0L/vI4V+l4BXM5VUq74= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763742957; c=relaxed/simple; bh=7+Wch1allgJ/u2F4dLGQvty/3cVdSYDoZQ3qohajjag=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=dP3hmApdnD60lBxhRfj5flI6FBvK4dOoVcPrqX9W7X5eZRhKyRrW4crZ26wW8Q1dsK+hisNBdyUCpNKtnt7LUPT7SlQFRTO5sX+zDL31IFjPxRiBTvkELQNhIwvJNWALAZ0JAyJXqQGlP4CmxvsgJPHKpZqfDirVAJfYY0yfNyM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AE1FE339; Fri, 21 Nov 2025 08:35:40 -0800 (PST) Received: from [10.57.73.129] (unknown [10.57.73.129]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9122B3F73B; Fri, 21 Nov 2025 08:35:45 -0800 (PST) Message-ID: <6e50830f-a1b8-452a-86a7-1621cd3968ce@arm.com> Date: Fri, 21 Nov 2025 16:35:43 +0000 Precedence: bulk X-Mailing-List: stable@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: stable 6.6: commit "sched/cpufreq: Rework schedutil governor performance estimation' causes a regression To: Yu-Che Cheng , Vincent Guittot Cc: Sergey Senozhatsky , "Rafael J. Wysocki" , Viresh Kumar , Greg Kroah-Hartman , Tomasz Figa , stable@vger.kernel.org, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, Lukasz Luba , Dietmar Eggemann References: <86d759a5-9a96-49ff-9f75-8b56e2626d65@arm.com> <2ktr5znjidilpxm2ycixunqlmhu253xwov4tpnb2qablrsqmbv@ysacm5nbcjw7> Content-Language: en-US From: Christian Loehle In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 11/21/25 15:37, Yu-Che Cheng wrote: > Hi Vincent, > > On Fri, Nov 21, 2025 at 10:00 PM Vincent Guittot > wrote: >> >> On Fri, 21 Nov 2025 at 04:55, Sergey Senozhatsky >> wrote: >>> >>> Hi Christian, >>> >>> On (25/11/20 10:15), Christian Loehle wrote: >>>> On 11/20/25 04:45, Sergey Senozhatsky wrote: >>>>> Hi, >>>>> >>>>> We are observing a performance regression on one of our arm64 > boards. >>>>> We tracked it down to the linux-6.6.y commit ada8d7fa0ad4 > ("sched/cpufreq: >> >> You mentioned that you tracked down to linux-6.6.y but which kernel >> are you using ? >> > > We're using ChromeOS 6.6 kernel, which is currently on top of linux-v6.6.99. > But we've tested that the performance regression still happens on exactly > the same scheduler codes (`kernel/sched`) as upstream v6.6.99, compared to > those on v6.6.88. > >>>>> Rework schedutil governor performance estimation"). >>>>> >>>>> UI speedometer benchmark: >>>>> w/commit: 395 +/-38 >>>>> w/o commit: 439 +/-14 >>>>> >>>> >>>> Hi Sergey, >>>> Would be nice to get some details. What board? >>> >>> It's an MT8196 chromebook. >>> >>>> What do the OPPs look like? >>> >>> How do I find that out? >> >> In /sys/kernel/debug/opp/cpu*/ >> or >> /sys/devices/system/cpu/cpufreq/policy*/scaling_available_frequencies >> with related_cpus >> > > The energy model on the device is: > > CPU0-3: > +------------+------------+ > | freq (khz) | power (uw) | > +============+============+ > | 339000 | 34362 | > | 400000 | 42099 | > | 500000 | 52907 | > | 600000 | 63795 | > | 700000 | 74747 | > | 800000 | 88445 | > | 900000 | 101444 | > | 1000000 | 120377 | > | 1100000 | 136859 | > | 1200000 | 154162 | > | 1300000 | 174843 | > | 1400000 | 196833 | > | 1500000 | 217052 | > | 1600000 | 247844 | > | 1700000 | 281464 | > | 1800000 | 321764 | > | 1900000 | 352114 | > | 2000000 | 383791 | > | 2100000 | 421809 | > | 2200000 | 461767 | > | 2300000 | 503648 | > | 2400000 | 540731 | > +------------+------------+ > > CPU4-6: > +------------+------------+ > | freq (khz) | power (uw) | > +============+============+ > | 622000 | 131738 | > | 700000 | 147102 | > | 800000 | 172219 | > | 900000 | 205455 | > | 1000000 | 233632 | > | 1100000 | 254313 | > | 1200000 | 288843 | > | 1300000 | 330863 | > | 1400000 | 358947 | > | 1500000 | 400589 | > | 1600000 | 444247 | > | 1700000 | 497941 | > | 1800000 | 539959 | > | 1900000 | 584011 | > | 2000000 | 657172 | > | 2100000 | 746489 | > | 2200000 | 822854 | > | 2300000 | 904913 | > | 2400000 | 1006581 | > | 2500000 | 1115458 | > | 2600000 | 1205167 | > | 2700000 | 1330751 | > | 2800000 | 1450661 | > | 2900000 | 1596740 | > | 3000000 | 1736568 | > | 3100000 | 1887001 | > | 3200000 | 2048877 | > | 3300000 | 2201141 | > +------------+------------+ > > CPU7: > > +------------+------------+ > | freq (khz) | power (uw) | > +============+============+ > | 798000 | 320028 | > | 900000 | 330714 | > | 1000000 | 358108 | > | 1100000 | 384730 | > | 1200000 | 410669 | > | 1300000 | 438355 | > | 1400000 | 469865 | > | 1500000 | 502740 | > | 1600000 | 531645 | > | 1700000 | 560380 | > | 1800000 | 588902 | > | 1900000 | 617278 | > | 2000000 | 645584 | > | 2100000 | 698653 | > | 2200000 | 744179 | > | 2300000 | 810471 | > | 2400000 | 895816 | > | 2500000 | 985234 | > | 2600000 | 1097802 | > | 2700000 | 1201162 | > | 2800000 | 1332076 | > | 2900000 | 1439847 | > | 3000000 | 1575917 | > | 3100000 | 1741987 | > | 3200000 | 1877346 | > | 3300000 | 2161512 | > | 3400000 | 2437879 | > | 3500000 | 2933742 | > | 3600000 | 3322959 | > | 3626000 | 3486345 | > +------------+------------+ > >>> >>>> Does this system use uclamp during the benchmark? How? >>> >>> How do I find that out? >> >> it can be set per cgroup >> /sys/fs/cgroup/system.slice//cpu.uclam.min|max >> or per task with sched_setattr() >> >> You most probably use it because it's the main reason for ada8d7fa0ad4 >> to remove wrong overestimate of OPP >> > > For the speedometer case, yes, we set the uclamp.min to 20 for the whole > browser and UI (chrome). > There's no system-wide uclamp settings though. (From Sergey's traces) Per-cluster time‑weighted average frequency base => revert: little (cpu0–3, max 2.4 GHz): 0.746 GHz => 1.132 GHz (+51.6%) mid (cpu4–6, max 3.3 GHz): 1.043 GHz => 1.303 GHz (+24.9%) big (cpu7, max 3.626 GHz): 2.563 GHz => 3.116 GHz (+21.6%) And in particular time spent at OPPs (base => revert): Big core at upper 10%: 29.6% => 61.5% little cluster at 339 MHz: 50.1% => 1.0% Interesting that a uclamp.min of 20 (which shouldn't really have much affect on big CPU at all, with or without headroom AFAICS?) makes such a big difference here? > > But we also found other performance regressions in an Android guest VM, > where there's no uclamp for the VM and vCPU processes from the host side. > Particularly, the RAR extraction throughput reduces about 20% in the RAR > app (from RARLAB). > Although it's hard to tell if this is some sort of a side-effect of the UI > regression as the UI is also running at the same time. > I'd be inclined to say that is because of the vastly different DVFS from the UI workload, yes.