From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A45E412E1DC for ; Tue, 4 Aug 2026 04:41:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785818505; cv=none; b=t8Dw+3x9i7XC5V0KLeFKaNi10Ewauhs+GHbXieaE1DwZu7Q2/pF8mct0bO6jCCTOl5ZDe+ANQ1qMED883JK8UDfQ4t1mQWt/KpQi9vS0GzB73mqNzlD96mka/nyCsAIhaAHrSICydaX//GG5+WgEmBm9DSKyU8C1d41NbFKiBUc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785818505; c=relaxed/simple; bh=wy+IRO579gz0V+Kj4uZla19C87wZ0sR+cBs2cb5JA0w=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=i5lOw+MwuvFJMyZbJW9bKttjbZz3yi1juAxn9rLWLtJzVtbY8LExwPZw0egmq+5+SWL45F/LN8iNzB5LTko2FkxQi76wh+Z835QecDFlhGox/2iEk1PMh/psP999H/aD9Hpq6DPwTvR1B1XsZjweOSkFpBzhHWmgHocu+RKxOUY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=kLH8qRyV; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="kLH8qRyV" Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 6741ITdH3578175; Tue, 4 Aug 2026 04:41:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=2n6vHG ymFJ/jc47YLSVwgvc8OoUFckjdHfWojxvdKXA=; b=kLH8qRyVRDtAnIDCAolN1P moYHPHb/5l+NtAGAtn6affCTvdbRaqy2nACIFMvaHuhfM9qoTBLQUL1J5cj3PSGZ /g2/R9ESrEoF/8SmkaUX7FjTDS6vOtqWKocPVJRrM5MNzbjlat/TK7BeklU7Z7Tt CkRSX7ifP+O6+AGgLTMkbQZpAyngP51t8oePU/5ZBMHrvyuuG8/WvSqTtSQPPmQX CT8uR/cLCa11h/QnsqJMnVd8PHLY5xtqCazVHQGP0RAOb/nPgEe8vjLFiKJvBiOj oFOJ2H86BKSDG9hSHVmd6oxWWC7I3epw8TgddYPvWC1O32BhZaQOatkGmlBXTyhg == Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fs77g3vbe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 04 Aug 2026 04:41:04 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 6744QIgd031912; Tue, 4 Aug 2026 04:41:03 GMT Received: from smtprelay03.fra02v.mail.ibm.com ([9.218.2.224]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fsugw0gqd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 04 Aug 2026 04:41:03 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay03.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 6744f1lH33292590 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 4 Aug 2026 04:41:01 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 25BBD2004F; Tue, 4 Aug 2026 04:41:01 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0C1D12004E; Tue, 4 Aug 2026 04:40:57 +0000 (GMT) Received: from [9.39.27.209] (unknown [9.39.27.209]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Tue, 4 Aug 2026 04:40:56 +0000 (GMT) Message-ID: <98bbe2d7-2401-4713-b49e-12f5dfc36471@linux.ibm.com> Date: Tue, 4 Aug 2026 10:10:55 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4] sched/fair: Preserve wake-affine CPU for non-SMT reciprocal sync wakeups To: "Shubhang Kaushik (Ampere)" , Peter Zijlstra , Vincent Guittot , K Prateek Nayak , Ingo Molnar , Mel Gorman Cc: "Christoph Lameter (Ampere)" , Shubhang Kaushik , linux-kernel@vger.kernel.org, Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Valentin Schneider , Christian Loehle , Madadi Vineeth Reddy References: <20260803-b4-sched-sync-wakeup-v4-1-52333b0cfb79@gentwo.org> Content-Language: en-US From: Shrikanth Hegde In-Reply-To: <20260803-b4-sched-sync-wakeup-v4-1-52333b0cfb79@gentwo.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-Reinject: loops=2 maxloops=12 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwODA0MDAzMCBTYWx0ZWRfX7F1hM1M9TehN Po3IqP9SgEGgc7c6+xen2zpzs/u0W8tBVAlapR7MyafdnAtOBf8053q9JS0CURsNs8VEPaHAWMj 2wPhSSK1+gIlDjL4gGX+cYvxmF0UmtGifPIJvkVgCclqGweJy/JgUT+y1PQkL5RT6kbA0kbFFwD goVElINi/tWmnritA+5OZTa/bYrKg/3sxZooLqm42aQo8HT91C1mt3LzX2fdUIk0FN/FjKJhdp+ QUn5j00bQMJJr1VfdrmBnZWMB96B8SP1TKrbFaPNijJ3DSvEsPDs1pOjZvuduV4UrEZIjZyz9wd 6bH6W4XCoS+8pPCEcX+9gZl/lzhgq+4ud/aqlbxNE2fy4m2uuWCnCLiHI7XYRKx2/M0km2O25y0 QwpiMAeG+IMjydVepjw3wlTNshO/8okIIFMWQhBDVTZeENaV/S6EeZpyt64wirYbBOycW3M4W6I MLv6sJ4s0NeDtsyM5kA== X-Authority-Analysis: v=2.4 cv=WIFPmHsR c=1 sm=1 tr=0 ts=6a716d60 cx=c_pps a=5BHTudwdYE3Te8bg5FgnPg==:117 a=5BHTudwdYE3Te8bg5FgnPg==:17 a=IkcTkHD0fZMA:10 a=Sv0fKeRqtYgA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=V8glGbnc2Ofi9Qvn3v5h:22 a=PuvxfXWCAAAA:8 a=HZjhfAOwq8STxTim-N0A:9 a=QEXdDO2ut3YA:10 a=uAr15Ul7AJ1q7o2wzYQp:22 X-Proofpoint-GUID: xkb7yC6ZcWNfNr96nFXitQtyRWdn-n52 X-Proofpoint-ORIG-GUID: rXnKLJXEc-ao89aDCcn4vYZUAzB91F4U X-Proofpoint-Spam-Info: AW1haW4tMjYwODA0MDAzMCBTYWx0ZWRfX7gwXN0RAoG1G CK0rFwSCrpZdlXF6epIX1gJMki3ij5KyvELmdSYCQu16cHwVkbDDJmSJglQ0DX7wLUyggvE3EBN wTG+mjvUqNJBOvv2qanNyym44HgPves= X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1176,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-08-04_01,2026-08-03_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 lowpriorityscore=0 priorityscore=1501 phishscore=0 malwarescore=0 suspectscore=0 clxscore=1015 impostorscore=0 bulkscore=0 adultscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2608040030 Hi Shubhang. Please give time for discussion/reply for the people looking at your patches. Even before I could respond to your v3, you have sent v4. And your v4 doesn't addresses the concerns raised in v3. On 8/4/26 7:15 AM, Shubhang Kaushik (Ampere) wrote: > For WF_SYNC wakeups, wake_affine() may select the waker CPU, but the CFS > wakeup path still passes that target to select_idle_sibling(). The idle > CPU search can then move the wakee away from the wake-affine target. > > Pipe-style ping-pong workloads expose this because the wakee is handed > back and forth between two tasks. In that case, moving the wakee to > another idle CPU can cost more than preserving the wake-affine waker CPU. > > Use the existing last_wakee and wake_wide() state to identify narrow > reciprocal WF_SYNC wakeups: > > A wakes B > B wakes A > A wakes B > ... > > Handle only this narrow reciprocal case on non-SMT systems. Once the > wake-affine path has selected or kept the waker CPU, preserve that target > when the waker rq has no other runnable fair task. Return the waker CPU > before select_idle_sibling() so the idle CPU search does not move this > handoff away from the wake-affine target. > Why on non-SMT? Why the same problem cannot happen in SMT systems? > This does not define a generic WF_SYNC placement rule. Generic WF_SYNC > wakeups continue through the existing wake_affine() and > select_idle_sibling() behavior. SMT systems also continue through > select_idle_sibling(), where idle sibling/core placement can be handled > with SMT topology visible. > > On asymmetric-capacity systems, still require the wakee to fit on the > waker CPU. > > Signed-off-by: Shubhang Kaushik (Ampere) > --- > Tested on 80-core non-SMT Ampere Altra, tip:sched/core baseline. Where is your LLC? Does it has multiple cores and currently you end up choosing an idle core? Your numbers below pretty much tell the same story. > > perf bench sched pipe -l 1000000, 20 runs: IIUC, sched pipe doesn't do any work apart from ping-pong. > default: > 3.985 -> 3.187 usec/op mean, about 20.0% improvement > 4.026 -> 3.181 usec/op median, about 21.0% improvement > > taskset -c 78,79: > 3.851 -> 3.144 usec/op mean, about 18.4% improvement > 3.804 -> 3.140 usec/op median, about 17.4% improvement > > taskset -c 79: > 3.055 -> 3.113 usec/op mean, about 1.9% slower > 3.045 -> 3.109 usec/op median, about 2.1% slower > Which means you get the best result when it runs on same CPU. The rest of the changes likely enforce that behavior. Then same issue is prevalent in SMT world too. > Hackbench process/thread pipe cases with 1/2/4/8 groups were within > noise, with mean deltas from -1.8% to +3.7% over 10 runs. > > Schbench normal mode at 8/40/80/240 workers and schbench pipe mode at > 1/2/4/8 workers showed no material regression. > > Baseline: tip/sched/core at 5186ef36909c As I said in v3, before we add bells/whistles to sync path, i want to know what is expected of sync behavior today. And that should be documented in Documentation/scheduler/ Be it, - current way of hint only and scheduler can still choose an idle core/idle cpu etc. - Should it be enforcing it to waker cpu if waker cpu has only one task. - Whatever the policy maybe. Current api usage is tricky to use and effect is visible in real life workloads. The case I mentioned in v3 of networking code using sync api leads to strange results due to sync mechanism. - It depends whether waker/wakee are running on same node. - Result of wake_wide. In other end, user sees inconsistent latency/throughput. We can keep on adding minor changes to sync api path, but one benchmark will benefit and one will suffer. Having the behavior documented is a good start. Peter, Ingo, Vincent, Mel, Prateek, What do you guys think?