From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20F9F376BC2 for ; Thu, 20 Nov 2025 16:12:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763655167; cv=none; b=DnEG5j5ZL0GVTuqgbnE3Uss7sfE2s3bYAFKzK6yX0ZctCgfMP+Fw4HP4u6Opj80UMX0JrYf2MIWKnB072arLTmf0u7RfODxSN9ag5aeVTWSYJGY7seDuUTTuwajgUN2FtUJa/QEzxlD+rudmUm2aShyTzV1sKXr/RuylhImesSk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763655167; c=relaxed/simple; bh=FVbfdCuffvERsxnfI39vefQ9agpwGw4O+4GwEvKUMv0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=JgAVf1Gm4bM7nv5gXWswOcf1BWFoim9mTgAuYiec0M2QQ7q/pv+dt2RJ3s0YmBEl+xLFl5BhLWh2a3I1Xo9Bzr1hDa5ng4i5LfdY2+ZN8j8/O2rBr41oJnzsjOlY07+1W2PYgogsiq6dAHLwqSzWPLPd9hMf/Vbp/0MwEsAiREc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=i7O7m1CA; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="i7O7m1CA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 505DBC4CEF1; Thu, 20 Nov 2025 16:12:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763655166; bh=FVbfdCuffvERsxnfI39vefQ9agpwGw4O+4GwEvKUMv0=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=i7O7m1CApZlAx0ywnzHDRqw5oZDyolvSoqXm/u/7NzXYduvE3l9im+xwybo7jJsPE Z2NA8jJkE6M/7IiW5i+q2n/VX3Zyue/ApuCRS9JHmL8OHnQ0CgajzNfYFGfxcMVSrf Stjip/ZoKArsJ653IqGvP8gQP5zmeUQPSf2LbFQAOSjI6Iyj2/VOFhR0c7/oABLNl1 GnPJXwuM6zRL0ecsLemloVj/TbVonbLDEfiOkiLSl55pjBBTbCSTcwkwKZEK9ShPlx J6L+dK6r6Hd6l3CmUwymMtjvdiVg35WezmJUWG2xqsb3ozFbpvLAFGofWDfu1f/h6p slDpFQeGoo1pA== Date: Thu, 20 Nov 2025 17:12:44 +0100 From: Frederic Weisbecker To: Gabriele Monaco Cc: linux-kernel@vger.kernel.org, Anna-Maria Behnsen , Thomas Gleixner , Waiman Long , "John B. Wyatt IV" , "John B. Wyatt IV" Subject: Re: [PATCH v16 7/7] timers/migration: Exclude isolated cpus from hierarchy Message-ID: References: <20251120145653.296659-1-gmonaco@redhat.com> <20251120145653.296659-8-gmonaco@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20251120145653.296659-8-gmonaco@redhat.com> Le Thu, Nov 20, 2025 at 03:56:53PM +0100, Gabriele Monaco a écrit : > The timer migration mechanism allows active CPUs to pull timers from > idle ones to improve the overall idle time. This is however undesired > when CPU intensive workloads run on isolated cores, as the algorithm > would move the timers from housekeeping to isolated cores, negatively > affecting the isolation. > > Exclude isolated cores from the timer migration algorithm, extend the > concept of unavailable cores, currently used for offline ones, to > isolated ones: > * A core is unavailable if isolated or offline; > * A core is available if non isolated and online; > > A core is considered unavailable as isolated if it belongs to: > * the isolcpus (domain) list > * an isolated cpuset > Except if it is: > * in the nohz_full list (already idle for the hierarchy) > * the nohz timekeeper core (must be available to handle global timers) > > CPUs are added to the hierarchy during late boot, excluding isolated > ones, the hierarchy is also adapted when the cpuset isolation changes. > > Due to how the timer migration algorithm works, any CPU part of the > hierarchy can have their global timers pulled by remote CPUs and have to > pull remote timers, only skipping pulling remote timers would break the > logic. > For this reason, prevent isolated CPUs from pulling remote global > timers, but also the other way around: any global timer started on an > isolated CPU will run there. This does not break the concept of > isolation (global timers don't come from outside the CPU) and, if > considered inappropriate, can usually be mitigated with other isolation > techniques (e.g. IRQ pinning). > > This effect was noticed on a 128 cores machine running oslat on the > isolated cores (1-31,33-63,65-95,97-127). The tool monopolises CPUs, > and the CPU with lowest count in a timer migration hierarchy (here 1 > and 65) appears as always active and continuously pulls global timers, > from the housekeeping CPUs. This ends up moving driver work (e.g. > delayed work) to isolated CPUs and causes latency spikes: > > before the change: > > # oslat -c 1-31,33-63,65-95,97-127 -D 62s > ... > Maximum: 1203 10 3 4 ... 5 (us) > > after the change: > > # oslat -c 1-31,33-63,65-95,97-127 -D 62s > ... > Maximum: 10 4 3 4 3 ... 5 (us) > > The same behaviour was observed on a machine with as few as 20 cores / > 40 threads with isocpus set to: 1-9,11-39 with rtla-osnoise-top. > > Tested-by: John B. Wyatt IV > Tested-by: John B. Wyatt IV > Signed-off-by: Gabriele Monaco Reviewed-by: Frederic Weisbecker Thanks a lot! Looks pretty good now! -- Frederic Weisbecker SUSE Labs