From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from fra-out-008.esa.eu-central-1.outbound.mail-perimeter.amazon.com (fra-out-008.esa.eu-central-1.outbound.mail-perimeter.amazon.com [35.158.23.94]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D5771A704B for ; Fri, 28 Nov 2025 14:37:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=35.158.23.94 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764340663; cv=none; b=m9J850eyM2g3ScrtfGmQZrVEqZ5U5HJ9vpiZzbEhz8US/Hw9/lAO8y+trtiUQwfAny+aM9iVkg3C+0q7Ug9mgFQ0bQQzByaFqf6CriCJ6Bh2szSDA8P6Lu2inzgWcLCruoO+DH5LbvXqm87BIRENWsItV3mnkEK4rmEcW5HdUWk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764340663; c=relaxed/simple; bh=2DAwLbxMAQjKc8R8FLohTOKmDGEIto3B1T1O1AwNSBs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=jWSJNh3FbAPmS+PCMXM/t1+qN5E+71Ctz+5bWzUFQM+8TQa5hoQN/KRM+7tLqGqsJFlNhTkR6K453OoQxNikHwiT5W+tOGKn4VmZtXJL1HjMuVQ0eHjpgbo3XtdSa0HuypfKxq+nZgajx3hrbf6lcW9Y3AZLmucUxhnk5tBlMUI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com; spf=pass smtp.mailfrom=amazon.com; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b=Rs38TClT; arc=none smtp.client-ip=35.158.23.94 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b="Rs38TClT" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazoncorp2; t=1764340662; x=1795876662; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qdDFtvsJO/9+LAaDGYPDXx1E5FOPJ7cabVe2IbUTpPM=; b=Rs38TClTzdtlbiXW1LHImYuzIBjCKTW/sotgf3SOLnN+vTOPuvgMSoQO xaqSJ/7FHNnIDC/LXtI/Jp/ERS5dnUa+SAS0S25r43NCtXWVx1C2Anb9k 6O6Uddebj74RZePetjmpYktJKJ3L4UlK0cNvLzoDtw14ZQF04lcVXwvdd ZBXoMqW+x2EuUqivC+I/ds7kwnJaDJ2jaoyd2oqywA+GZm1cIWX3T7Zah lGJRKETJMZh97Ct5490w5xARk3miALwTritKzs2APX33ZpDSGggK3TlJx m+/zTWzdxBi5MLv4CIbF0EH+9Q6CQ4mBtmmbXx0OpoNd1nHtSf6CvuVRN A==; X-CSE-ConnectionGUID: qsRDu+CuSyeVTzLfq2BNJQ== X-CSE-MsgGUID: Lp70GxdqQlmWc4ElihI9MA== X-IronPort-AV: E=Sophos;i="6.20,234,1758585600"; d="scan'208";a="5951925" Received: from ip-10-6-11-83.eu-central-1.compute.internal (HELO smtpout.naws.eu-central-1.prod.farcaster.email.amazon.dev) ([10.6.11.83]) by internal-fra-out-008.esa.eu-central-1.outbound.mail-perimeter.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2025 14:37:24 +0000 Received: from EX19MTAEUB002.ant.amazon.com [54.240.197.232:30498] by smtpin.naws.eu-central-1.prod.farcaster.email.amazon.dev [10.0.31.35:2525] with esmtp (Farcaster) id 692ccc2e-a52f-4271-aa5f-843669764566; Fri, 28 Nov 2025 14:37:24 +0000 (UTC) X-Farcaster-Flow-ID: 692ccc2e-a52f-4271-aa5f-843669764566 Received: from EX19D003EUB001.ant.amazon.com (10.252.51.97) by EX19MTAEUB002.ant.amazon.com (10.252.51.59) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.29; Fri, 28 Nov 2025 14:37:22 +0000 Received: from u5934974a1cdd59.ant.amazon.com (10.146.13.225) by EX19D003EUB001.ant.amazon.com (10.252.51.97) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.29; Fri, 28 Nov 2025 14:37:15 +0000 From: Fernand Sieber To: CC: , , , , , , , , , , , , Subject: Re: [PATCH] sched/core: Push tasks on force idle Date: Fri, 28 Nov 2025 16:36:51 +0200 Message-ID: <20251128143651.391406-1-sieberf@amazon.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251128133822.GB4067720@noisy.programming.kicks-ass.net> References: <20251128133822.GB4067720@noisy.programming.kicks-ass.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: EX19D031UWC003.ant.amazon.com (10.13.139.252) To EX19D003EUB001.ant.amazon.com (10.252.51.97) Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Fri, Nov 28, 2025 at 02:38:22PM +0100, Peter Zijlstra wrote: > On Fri, Nov 28, 2025 at 03:19:54PM +0200, Fernand Sieber wrote: > > When a cpu enters force idle, it will > > 1) try to steal cookie matching tasks from other CPUs > > 2) do the newidle balance > > > > If the stealing fails, we are out of options to get out of force idle > > properly. New idle balance might decide to pull other tasks, but they > > won't necessarily be matching anyways. > > > > Introduce a step in between where we try to push the runnable tasks > > that are blocked in force idle to a more suitable CPU. > > > > === Testing setup === > > > > Similar setup as in: > > https://lore.kernel.org/lkml/20251127202719.963766-1-sieberf@amazon.com > > > > Testing is aimed at measuring perceived guest noise on hypervisor > > system with time shared scenarios. > > > > Setup is on system where the load is nearing 100% which should allow no > > steal time. The system has 64 CPUs, with 8 VMs, each VM using core > > scheduling with 8 vCPUs per VM, time shared. > > > > 7 VMs are running stressors (`stress-ng --cpu 0`) while the last VM is > > running the hwlat tracer with a width of 100ms, a period of 300ms, and > > a threshold of 100us. Each VM runs a cookied non vCPU VMM process that > > adds a light level of noise which forces some level of load balancing. > > > > The test scenario is ran 10x60s and the average noise is measured (we > > use breaches scaled up to period/width to estimate noise). > > > > === Testing results === > > > > Baseline noise: 1.20% > > After patch noise: 0.66% (-45%) > > This is similar to that other patch, what happens if you combine the > two? Noise results: - Baseline: 1.20% - Force idle aware LB: 0.63% (https://lore.kernel.org/lkml/20251127202719.963766-1-sieberf@amazon.com) - Push force idle tasks: 0.66% (this patch) - Both patches combined: 0.45% Note: I realized I also ran these tests with this patch applied on baseline: "sched/fair: Add more core cookie check in wake up fast path" https://lore.kernel.org/lkml/20251120101955.968586-1-sieberf@amazon.com Ideally I would revert it and compute all improvements independently. Prateek already reviewed that patch, I would appreciate if you could take a look too. I could post all the patches together, though I thought they are fairly independent so it's easier to keep them separate. Additionally, to craft these patches I examined inefficiency opportunities tracked with scheduling ftrace dumps, for which I also relied on a cookie tracepoint proposed here: https://lore.kernel.org/lkml/20250128113410.263994-1-sieberf@amazon.com/ Amazon Development Centre (South Africa) (Proprietary) Limited 29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa Registration Number: 2004 / 034463 / 07