From mboxrd@z Thu Jan 1 00:00:00 1970 From: Waiman Long Subject: Re: [PATCH v3] sched: cpuset: Don't rebuild root domains on suspend-resume Date: Tue, 7 Mar 2023 16:13:49 -0500 Message-ID: <1f2cf8ea-a9d7-5245-0f69-eb8be9f64afc@redhat.com> References: <20230206221428.2125324-1-qyousef@layalina.io> Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678223635; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6RuqdEgW86TqgRF087mijC/p9xtmtsdmCeq1juDgkFI=; b=GfqyGDw7FdyPwKRHTDA0jx56vWNqKMUnSjRQ+eXZt/uXJxVCXHIJTMq+nuubuJJkGPvCjZ 8bDa0/gSQ7/tmpqQlHJSEem7OWodjv/D/1+kmj7cDEsTCLXvg8ZR05ZlDi4+MMGipD2CpE q/BDbAho/tgXlZFA1jMCZ+4LBARvXTY= Content-Language: en-US In-Reply-To: List-ID: Content-Type: text/plain; charset="utf-8"; format="flowed" To: Hao Luo Cc: Qais Yousef , Peter Zijlstra , Ingo Molnar , Juri Lelli , Steven Rostedt , tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, luca.abeni-5rdYK369eBLQB0XuIGIEkQ@public.gmane.org, claudio-YOzL5CV4y4YG1A2ADO40+w@public.gmane.org, tommaso.cucinotta-5rdYK369eBLQB0XuIGIEkQ@public.gmane.org, bristot-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, mathieu.poirier-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org, Dietmar Eggemann , cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Vincent Guittot , Wei Wang , Rick Yiu , Quentin Perret , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Sudeep Holla , Zefan Li On 3/7/23 16:06, Hao Luo wrote: > On Tue, Mar 7, 2023 at 12:09 PM Waiman Long wrote: >> On 3/7/23 14:56, Hao Luo wrote: >>> On Mon, Feb 6, 2023 at 2:15 PM Qais Yousef wrote: >>>> Commit f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information") >>>> enabled rebuilding root domain on cpuset and hotplug operations to >>>> correct deadline accounting. >>>> >>>> Rebuilding root domain is a slow operation and we see 10+ of ms delays >>>> on suspend-resume because of that (worst case captures 20ms which >>>> happens often). >>>> >>>> Since nothing is expected to change on suspend-resume operation; skip >>>> rebuilding the root domains to regain the some of the time lost. >>>> >>>> Achieve this by refactoring the code to pass whether dl accoutning needs >>>> an update to rebuild_sched_domains(). And while at it, rename >>>> rebuild_root_domains() to update_dl_rd_accounting() which I believe is >>>> a more representative name since we are not really rebuilding the root >>>> domains, but rather updating dl accounting at the root domain. >>>> >>>> Some users of rebuild_sched_domains() will skip dl accounting update >>>> now: >>>> >>>> * Update sched domains when relaxing the domain level in cpuset >>>> which only impacts searching level in load balance >>>> * update sched domains when cpufreq governor changes and we need >>>> to create the perf domains >>>> >>>> Users in arch/x86 and arch/s390 are left with the old behavior. >>>> >>>> Debugged-by: Rick Yiu >>>> Signed-off-by: Qais Yousef (Google) >>>> --- >>> Hi Qais, >>> >>> Thank you for reporting this. We observed the same issue in our >>> production environment. Rebuild_root_domains() is also called under >>> cpuset_write_resmask, which handles writing to cpuset.cpus. Under >>> production workloads, on a 4.15 kernel, we observed the median latency >>> of writing cpuset.cpus at 3ms, p99 at 7ms. Now the median becomes >>> 60ms, p99 at >100ms. Writing cpuset.cpus is a fairly frequent and >>> critical path in production, but blindly traversing every task in the >>> system is not scalable. And its cost is really unnecessary for users >>> who don't use deadline tasks at all. >> The rebuild_root_domains() function shouldn't be called when updating >> cpuset.cpus unless it is a partition root. Is it? >> > I think it's because we were using the legacy hierarchy. I'm not > familiar with cpuset partition though. In legacy hierarchy, changing cpuset.cpus shouldn't lead to the calling of rebuild_root_domains() unless you play with cpuset.sched_load_balance file by changing it to 0 in the right cpusets. If you are touching cpuset.sched_load_balance, you shouldn't change cpuset.cpus that often. Cheers, Longman