From mboxrd@z Thu Jan 1 00:00:00 1970 From: Waiman Long Subject: Re: [PATCH v3] sched: cpuset: Don't rebuild root domains on suspend-resume Date: Tue, 7 Mar 2023 15:08:52 -0500 Message-ID: References: <20230206221428.2125324-1-qyousef@layalina.io> Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678219740; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aTa3k5Nu5kG9KD5q86aaFluSTkM03P8RErYO6vqZrgw=; b=OVn85nFUXAIRtMsHUmu55vOzqRmJCr+i3qOERp8RdeAZNBwa8SehlG9BLxAD9SBHVljdZh cetxBIBa7aWM6QRLCqWFH9LE4Dy9Z7fhInipD8DUyZ+VHrxb/3Sl8v1AdmEEeGcFsl2+wh YI7r2n8nYB7SB/IXYIflwACNMbazFrg= Content-Language: en-US In-Reply-To: List-ID: Content-Type: text/plain; charset="windows-1252"; format="flowed" To: Hao Luo , Qais Yousef Cc: Peter Zijlstra , Ingo Molnar , Juri Lelli , Steven Rostedt , tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, luca.abeni-5rdYK369eBLQB0XuIGIEkQ@public.gmane.org, claudio-YOzL5CV4y4YG1A2ADO40+w@public.gmane.org, tommaso.cucinotta-5rdYK369eBLQB0XuIGIEkQ@public.gmane.org, bristot-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, mathieu.poirier-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org, Dietmar Eggemann , cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Vincent Guittot , Wei Wang , Rick Yiu , Quentin Perret , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Sudeep Holla , Zefan Li , linux-s390-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, x86@ker On 3/7/23 14:56, Hao Luo wrote: > On Mon, Feb 6, 2023 at 2:15=E2=80=AFPM Qais Yousef wrote: >> Commit f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting i= nformation") >> enabled rebuilding root domain on cpuset and hotplug operations to >> correct deadline accounting. >> >> Rebuilding root domain is a slow operation and we see 10+ of ms delays >> on suspend-resume because of that (worst case captures 20ms which >> happens often). >> >> Since nothing is expected to change on suspend-resume operation; skip >> rebuilding the root domains to regain the some of the time lost. >> >> Achieve this by refactoring the code to pass whether dl accoutning needs >> an update to rebuild_sched_domains(). And while at it, rename >> rebuild_root_domains() to update_dl_rd_accounting() which I believe is >> a more representative name since we are not really rebuilding the root >> domains, but rather updating dl accounting at the root domain. >> >> Some users of rebuild_sched_domains() will skip dl accounting update >> now: >> >> * Update sched domains when relaxing the domain level in cpuset >> which only impacts searching level in load balance >> * update sched domains when cpufreq governor changes and we need >> to create the perf domains >> >> Users in arch/x86 and arch/s390 are left with the old behavior. >> >> Debugged-by: Rick Yiu >> Signed-off-by: Qais Yousef (Google) >> --- > Hi Qais, > > Thank you for reporting this. We observed the same issue in our > production environment. Rebuild_root_domains() is also called under > cpuset_write_resmask, which handles writing to cpuset.cpus. Under > production workloads, on a 4.15 kernel, we observed the median latency > of writing cpuset.cpus at 3ms, p99 at 7ms. Now the median becomes > 60ms, p99 at >100ms. Writing cpuset.cpus is a fairly frequent and > critical path in production, but blindly traversing every task in the > system is not scalable. And its cost is really unnecessary for users > who don't use deadline tasks at all. The rebuild_root_domains() function shouldn't be called when updating=20 cpuset.cpus unless it is a partition root. Is it? Cheers, Longman