From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CC2E21F3FC2; Tue, 11 Feb 2025 10:15:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739268957; cv=none; b=KcUVL6wVqGDB8fLOwBqD4Z1Y+oZogDkILyrWcs87AtLrxuhahg001VX6QVWe5RqZGpYr1UOu0vlRNz7YDe/GM/Sh3Ny0WNpeJYBI6Dl+/8HniOLBbMMfQtMT695+6r2OaHCu6yWQKHqWdcUEZL8ZDswRqjA1fI+aZnJbbXetCk4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739268957; c=relaxed/simple; bh=AANQ4doGqqYCs1ixTUngiq7Ybau1E1MR40bQLIYvdrY=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=V3yHpNqMOo/Ir6UfUq+WLTfpKaTHl1UoaMnRwbo8thpUvnkIHEi0J4XKoFsZzbQrD06inGeYP77EZfy0g5bMk4NP+CQQpwfsoKyn17dm7jNSsq1RvFF+xv6n3iIYbNWFegUmM8yJo2LxnqfC4XQULDNZEaIwelNBzQCOhHzDdb0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 720C61424; Tue, 11 Feb 2025 02:16:16 -0800 (PST) Received: from [10.57.37.40] (unknown [10.57.37.40]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A6E763F6A8; Tue, 11 Feb 2025 02:15:49 -0800 (PST) Message-ID: Date: Tue, 11 Feb 2025 10:15:47 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 3/2] sched/deadline: Check bandwidth overflow earlier for hotplug To: Juri Lelli Cc: Jon Hunter , Dietmar Eggemann , Thierry Reding , Waiman Long , Tejun Heo , Johannes Weiner , Michal Koutny , Ingo Molnar , Peter Zijlstra , Vincent Guittot , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Phil Auld , Qais Yousef , Sebastian Andrzej Siewior , "Joel Fernandes (Google)" , Suleiman Souhlal , Aashish Sharma , Shin Kawamura , Vineeth Remanan Pillai , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "linux-tegra@vger.kernel.org" References: <8572b3bc-46ec-4180-ba55-aa6b9ab7502b@nvidia.com> <5a36a2e8-bd78-4875-9b9e-814468ca6692@arm.com> <8ff19556-a656-4f11-a10c-6f9b92ec9cea@arm.com> Content-Language: en-US From: Christian Loehle In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 2/10/25 17:09, Juri Lelli wrote: > Hi Christian, > > Thanks for taking a look as well. > > On 07/02/25 15:55, Christian Loehle wrote: >> On 2/7/25 14:04, Jon Hunter wrote: >>> >>> >>> On 07/02/2025 13:38, Dietmar Eggemann wrote: >>>> On 07/02/2025 11:38, Jon Hunter wrote: >>>>> >>>>> On 06/02/2025 09:29, Juri Lelli wrote: >>>>>> On 05/02/25 16:56, Jon Hunter wrote: >>>>>> >>>>>> ... >>>>>> >>>>>>> Thanks! That did make it easier :-) >>>>>>> >>>>>>> Here is what I see ... >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Still different from what I can repro over here, so, unfortunately, I >>>>>> had to add additional debug printks. Pushed to the same branch/repo. >>>>>> >>>>>> Could I ask for another run with it? Please also share the complete >>>>>> dmesg from boot, as I would need to check debug output when CPUs are >>>>>> first onlined. >>>> >>>> So you have a system with 2 big and 4 LITTLE CPUs (Denver0 Denver1 A57_0 >>>> A57_1 A57_2 A57_3) in one MC sched domain and (Denver1 and A57_0) are >>>> isol CPUs? >>> >>> I believe that 1-2 are the denvers (even thought they are listed as 0-1 in device-tree). >> >> Interesting, I have yet to reproduce this with equal capacities in isolcpus. >> Maybe I didn't try hard enough yet. >> >>> >>>> This should be easy to set up for me on my Juno-r0 [A53 A57 A57 A53 A53 A53] >>> >>> Yes I think it is similar to this. >>> >>> Thanks! >>> Jon >>> >> >> I could reproduce that on a different LLLLbb with isolcpus=3,4 (Lb) and >> the offlining order: >> echo 0 > /sys/devices/system/cpu/cpu5/online >> echo 0 > /sys/devices/system/cpu/cpu1/online >> echo 0 > /sys/devices/system/cpu/cpu3/online >> echo 0 > /sys/devices/system/cpu/cpu2/online >> echo 0 > /sys/devices/system/cpu/cpu4/online >> >> while the following offlining order succeeds: >> echo 0 > /sys/devices/system/cpu/cpu5/online >> echo 0 > /sys/devices/system/cpu/cpu4/online >> echo 0 > /sys/devices/system/cpu/cpu1/online >> echo 0 > /sys/devices/system/cpu/cpu2/online >> echo 0 > /sys/devices/system/cpu/cpu3/online >> (Both offline an isolcpus last, both have CPU0 online) >> >> The issue only triggers with sugov DL threads (I guess that's obvious, but >> just to mention it). > > It wasn't obvious to me at first :). So thanks for confirming. > >> I'll investigate some more later but wanted to share for now. > > So, problem actually is that I am not yet sure what we should do with > sugovs' bandwidth wrt root domain accounting. W/o isolation it's all > good, as it gets accounted for correctly on the dynamic domains sugov > tasks can run on. But with isolation and sugov affected_cpus that cross > isolation domains (e.g., one BIG one little), we can get into troubles > not knowing if sugov contribution should fall on the DEF or DYN domain. > > Hummm, need to think more about it. That is indeed tricky. I would've found it super appealing to always just have sugov DL tasks activate on this_cpu and not have to worry about all this, but then you have contention amongst CPUs of a cluster and there are energy improvements from always having little cores handle all sugov DL tasks, even for the big CPUs, that's why I introduced commit 93940fbdc468 ("cpufreq/schedutil: Only bind threads if needed") but that really doesn't make this any easier.