From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE8F2242D6C for ; Mon, 3 Aug 2026 14:06:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785765975; cv=none; b=oG74MENtsznb4svYnqUBtqeINN9fP05veqUIVCpeDlmQHFOWqmAa0YTz/Z96XXwb6mI+XwCaiMCNgImWkvR2eGz54mJ1Q2UT+ZjHTWfOJ8AQdVWdj+y9Lwnvzf3ta9sEohbGpMrZRWCZ9zkJKT2WTqrWs4q+7k2yJTRaX669/vU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785765975; c=relaxed/simple; bh=Qff/U8JNmZQXk8yKYBe5H/7qkN1liIWtcS7UfnMWgjM=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=hGnhDHAlToXFXfirmmqLsdGwPOzuBdUQPkRf5iVn90mykYS4bRf5ub5QghCj/R6Vd8Np8aVj99lsQKNC4IgIXh/GHBUEcpolyuUbsg4XH4h/TpvUKP0g+TXnj63XVC2yb91SZl6DwRMTLoH5y/Q6Y1vEQxYeUnwOVDkUCQdj/IY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=XqdwiVs/; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="XqdwiVs/" Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 673DlnRH2139521; Mon, 3 Aug 2026 14:05:47 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=yCE4PU 3uPSgtS68Nwr3U65368y356Zy7h/CxXUSqDoU=; b=XqdwiVs/A8yoSeVWfjwuno Rf4YMtfbU0PEjOytDoj+7uwg3J3RU4VuUrwGhh/OzNwz0JUgcCaNRnJM0FpXCtAa uUeZaa0AhvmrIO/JYS59cgtJ5ZgEwNjaBOOHiK2FXg+03O2oFUgUcwnFfyug2KO+ bD3cMtGOMawldkMBTYbCKYHtRLzwmnywU7EXw54/M77pr4jAo9C/UT02ebIPWimc BRmTY+MuZFjj5K5U6dQq4rMsvPDF4+BFVQToVGd9GrOYvZrQ4SCFna/UpgicRIDD 3iDSNqdtxrDpjrkU9C9wzu/PtenM7mW0RFahqUgEpwvu91DFdYySvs7kgGE86vNA == Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fs77g0yw4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 03 Aug 2026 14:05:46 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 673DuO1p004106; Mon, 3 Aug 2026 14:05:45 GMT Received: from smtprelay06.fra02v.mail.ibm.com ([9.218.2.230]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 4fswbg5fun-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 03 Aug 2026 14:05:45 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay06.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 673E5hsu29360590 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 3 Aug 2026 14:05:43 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CDD8220040; Mon, 3 Aug 2026 14:05:43 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 40A0A20043; Mon, 3 Aug 2026 14:05:40 +0000 (GMT) Received: from [9.39.27.22] (unknown [9.39.27.22]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Mon, 3 Aug 2026 14:05:40 +0000 (GMT) Message-ID: Date: Mon, 3 Aug 2026 19:35:39 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3] sched/fair: Prefer waker CPU for non-SMT reciprocal sync wakeups To: "Shubhang Kaushik (Ampere)" , K Prateek Nayak , Madadi Vineeth Reddy , Vincent Guittot , Peter Zijlstra , Ingo Molnar Cc: "Christoph Lameter (Ampere)" , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Christian Loehle , linux-kernel@vger.kernel.org, Shubhang Kaushik References: <20260727-b4-sched-sync-wakeup-v3-1-90cf481dbd85@gentwo.org> Content-Language: en-US From: Shrikanth Hegde In-Reply-To: <20260727-b4-sched-sync-wakeup-v3-1-90cf481dbd85@gentwo.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-Reinject: loops=2 maxloops=12 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwODAzMDEyNCBTYWx0ZWRfX2PzYKvDsuTKq aurIVwzfpfJqp1tGFguY3l+hCU+cZDJh1hg1bOQoDnuwgVOtmB2m68TBF48q/06lcpn6Z0EEAqu Dd0mK5OZWFJyI7uJqr/swzxxvKHcsuQ8YSgnUWpah9yCF5iXO8ul9X5La9963oVKflN49NgsGUx FC0Ioufzvf3x5ruLB+Tsj2EMHiNSew3535S6Tb/lIUNA1q2mHJdkhvEKesZELxyFDXJjMQVQrEQ gjePg27n2q//SuBJ+dk/jJsI3XanKquX29qhJqiDYiPEbZME8TXz1WZfSzYxUC/GF537jqqd1cq 2jdg42cEesImXgIp2B1lDn3hNMYHb9ZmL1e4YCzn/GANmwltoZucQg6WAvtrPtQNbFr6ngpl41g iXY9Mk1jDvFk6lfs6rQjFd4mifyWy7oImsbCNfD3qB3Qye0dP/wrLj8rWgaSBel6NEMu3bUl6D8 pT6H9ZM+ODeL+gaqWpQ== X-Authority-Analysis: v=2.4 cv=WIFPmHsR c=1 sm=1 tr=0 ts=6a70a03b cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=IkcTkHD0fZMA:10 a=Sv0fKeRqtYgA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=V8glGbnc2Ofi9Qvn3v5h:22 a=VwQbUJbxAAAA:8 a=VnNF1IyMAAAA:8 a=mveV7eK2P-lyM_wv-ggA:9 a=QEXdDO2ut3YA:10 X-Proofpoint-GUID: FDj1_ZMyvLWOnEkyNoWMaMd2AnHcQObl X-Proofpoint-ORIG-GUID: 4eMle6sFr6V8hyyQrw64GRdCPDw9g9uW X-Proofpoint-Spam-Info: AW1haW4tMjYwODAzMDEyNCBTYWx0ZWRfX2t6Tqqgq6OfV 9jmLoezeUiDFoNgtkKJRXEOyxpwjM//rUCNVu+V5SIIKIIOzFLJdrJKk/i7wNfvAqQlI0nbv4sx Qusb1pc9l4m8vRlwTJf4kR29LFpDz/k= X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1176,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-08-03_03,2026-08-03_01,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 lowpriorityscore=0 priorityscore=1501 phishscore=0 malwarescore=0 suspectscore=0 clxscore=1011 impostorscore=0 bulkscore=0 adultscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2608030124 Hello. After seeing this and vineeth's patch, https://lore.kernel.org/all/20260801035532.260625-1-vineethr@linux.ibm.com/ I am bit confused on the policy we are trying to do for sync. Find the details below. On 7/28/26 5:28 AM, Shubhang Kaushik (Ampere) wrote: > Pipe-style ping-pong workloads can be dominated by handoff cost. In > such cases, placing the wakee on an idle CPU can be slower than keeping > the pair on the same runqueue. > > Use the existing last_wakee and wake_wide() state to identify narrow > reciprocal WF_SYNC wakeups: > > A wakes B > B wakes A > A wakes B > ... > > When the wake-affine domain allows SD_WAKE_AFFINE, prefer the waker CPU > for these narrow reciprocal handoffs on non-SMT systems. Do so only when > the waker CPU has no other runnable fair task and the wakee fits there on > asymmetric-capacity systems. > > SMT systems, and wakeups that do not match this pattern, continue through > the existing wake_affine() and select_idle_sibling() path. > I think we need to think this on the policy notion rather than a usecase specific. These are api's available to other susystems to make specific call based on its understand of its requirement. i.e wake_up_interruptible_sync_poll vs wake_up, wake_up_interruptible If we look at __wake_up_sync*, It says, /** * __wake_up_sync_key - wake up threads blocked on a waitqueue. * @wq_head: the waitqueue * @mode: which threads * @key: opaque value to be passed to wakeup targets * * The sync wakeup differs that the waker knows that it will schedule * away soon, so while the target thread will be woken up, it will not * be migrated to another CPU - ie. the two threads are 'synchronized' * with each other. This can prevent needless bouncing between CPUs. * * On UP it can prevent extra preemption. * * If this function wakes up a task, it executes a full memory barrier before * accessing the task state. */ void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode, void *key) { if (unlikely(!wq_head)) return; __wake_up_common_lock(wq_head, mode, 1, WF_SYNC, key); } So, with that, we use introduce the notion that, scheduler wakeup will honor the sync behaviour based on underlying arch/hw, how will callers ever know. For example, same SMT system can have all its siblings off, and now it is !smt system. There is already use/abuse of sync api in Networking staff. A recent discussion on it, https://lore.kernel.org/all/amI22o9MwDoGcBMl@linux.ibm.com/ I am assuming there would be more. So, What should sync wakeup should do vs non-sync wakeup? - Should it chose waker's CPU if waker is the only one running. - Should it be always? - Should it be under specific case such !smt, cas specific? - Should it still chose an idle core first, if not chose waker CPU/Sibling? - Should it fallback to waker's LLC vs current LLC. and then choose a CPU in that LLC or choose a recently used cpu, prev_cpu etc? (Current logic) I think we should define the policy for it. (if it is not too late for it) No?