From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A38B4388E61 for ; Thu, 6 Aug 2026 17:44:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.14 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786038261; cv=none; b=N9QIfgIGCS8zBbItRYxU+z8eGS7dcU0g53spKOcgx3wQQWikh2D+2n8c5YAJwufPjMVcTTvMu20X5rfgKq+M1noYcHzdvw0rDWCGB16mV2P14eVvTE3i9jY/um3TSeW4g+GXA2cE8z6O8WYQG6yMbmDW4NedJS1YpKx+W7wTUzQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786038261; c=relaxed/simple; bh=CXdOY6grEJ3x9DmI91pj7sRCgGXqvhZAH3kD8f/qX+U=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=els0FkJ50RbUgwFoIzdLyFhnHgSJFIBnhJEQ9FYpfil5ZJzWw2HnaST9fporxK/x5mCpjnbhjFPKA5p7M1cTkEdQ9BQepySnz7KTBuVpeOGO7vstEAttA2qqKqpHTPXPAELK6nXuYK21frEVP5BGlgna25nOmDBxBUFTCkdt0lw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=W0k9i7bY; arc=none smtp.client-ip=198.175.65.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="W0k9i7bY" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1786038260; x=1817574260; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=CXdOY6grEJ3x9DmI91pj7sRCgGXqvhZAH3kD8f/qX+U=; b=W0k9i7bYDxdH6Xj20/USUsP75j8MMTGmhjgsCmZcJE8eU/w+QAVu7161 3CtgDNFBpwqp7TwMeMyAi4kd2bPzmM1z3VpLY2+/e+/ecW46nHZv+hqOr VwAINstLtq6wtVqi9Ia9T4tenAwBAhJC16mnsC7WcKMlfK+RajPH9lft2 6JTJGG9ZNBmmuTY+ONsrDucyXgwc5bsbiisdXdvRyNsnIXPMYn6t9125n rbSS6W4IabOya5u0fqF4/2lSr0p12lSxtu6BBNtkVjVO0dQ6QPXZArGCc aD4qDMBFgkSFcp5QQieVioDVbbq2DOVciPilddIIKCS9BurO/dJIOZxNE Q==; X-CSE-ConnectionGUID: M3878zc+RqeWUiqXJTMlyA== X-CSE-MsgGUID: 5N+YnP7gSYSD8eqPL5yrAw== X-IronPort-AV: E=McAfee;i="6800,10657,11867"; a="90522071" X-IronPort-AV: E=Sophos;i="6.25,208,1779174000"; d="scan'208";a="90522071" Received: from orviesa003.jf.intel.com ([10.64.159.143]) by orvoesa106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Aug 2026 10:44:19 -0700 X-CSE-ConnectionGUID: o3vHIwMgQgSdV04nsHy48Q== X-CSE-MsgGUID: R3EyZ3pxSRqWYfFH1+71SA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,208,1779174000"; d="scan'208";a="265651861" Received: from vcostago-desk1.jf.intel.com (HELO vcostago-desk1) ([10.88.27.144]) by ORVIESA003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Aug 2026 10:44:20 -0700 From: Vinicius Costa Gomes To: Hillf Danton Cc: Peter Zijlstra , K Prateek Nayak , Christoph Lameter , linux-kernel@vger.kernel.org Subject: Re: [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share In-Reply-To: <20260805021430.957-1-hdanton@sina.com> References: <20260805021430.957-1-hdanton@sina.com> Date: Thu, 06 Aug 2026 10:44:18 -0700 Message-ID: <87mruz2lfx.fsf@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain Hillf Danton writes: > On Tue, 04 Aug 2026 16:14:05 -0700 Vinicius Costa Gomes wrote: >> Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for >> ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too >> strong" and could cause tasks to stack on a busy NUMA node while other >> nodes are relatively idle. >> >> As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for >> ep_poll_callback") improves real workloads a revert is not the answer. >> The fix is to make the WF_SYNC "stack on waker" shortcut take into >> account the load on this and prev's LLC, rejecting the shortcut only >> when the waker (this) LLC is fully loaded and prev's LLC is less >> loaded than the waker's. >> >> Signed-off-by: Vinicius Costa Gomes >> --- >> We received a report of a regression on a openresty based >> workload (the main metric being tail latencies) on a CWF SNC3 single >> socket system, the main symptom that we could measure was one node >> being overloaded while the other nodes were relatively idle. >> >> Further investigation showed that spreading the NIC RX interrupts over >> all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add >> synchronous wakeup support for ep_poll_callback") also helped. >> > The irq approach is prefered because anything that gets the eevdf offloaded > is good, you see it is near to the knowall point, needless to say that they > lie in different layers and from the scheduling-cpu pov irq is a gray rhino > in the room while WF_SYNC is a tiger mosquito in the corner at best in your > case where the mosquito failed to understand your workload. I don't think irq spreading across NUMA nodes is that good of an idea on low loads/by default, as it loses the locality that the kernel (even with irqbalance) try to maintain. I used that as a hackish way of testing "if I spread tasks, does it improve the tail latencies?". Note that the "local-only reproducer" workload (memcached + memtier_benchmark) runs over loopback (no NIC irqs here), I pin memtier (the client) to one NUMA node, leave the server unpinned and I am able to reproduce the issue: with the RFC patch the tail latencies reduce by 2-3x. (on the customer workload the impact is even higher) My expectation was that the scheduler would be able to say: "ugh, even though respecting WF_SYNC is good most of the cases, this isn't one of them". And this is the spirit of the RFC, showing that those cases exist and their impact. Cheers, -- Vinicius