From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 00A00883F for ; Sun, 9 Aug 2026 10:54:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.179 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786272845; cv=none; b=EoH3LwoJfTKe3/VxO3CGA7lK4X4j7zsSqntoPfGAuCFKt6Mz5hzzOSQZvs4nJD2AmFNMwoTIU34n+iBxSfgsXOwKcv0Rmv0Q+Sjj4wIpjJrMiNPJeAEC7pu+x/QeHvE1EANFJAsAVWIvE1W4/r5JAudfdAFr1QgzWH63PC5ROtg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786272845; c=relaxed/simple; bh=3LFL4t+aiXwY/XQpDGHtZc+z9M8Ziw4nJXHNty1OpUc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XQhC735V5R8AsNMqvz7PLcfIt+xFzXXt3VrUOUOfDYxewkwo4k55X2J53Vo5qfBGOfOulL3UVeKtoBSxilYt7WzD+OHUxJBEGvjKUhoi8PKNc8aXqdnhFmF+6AGO3GE5rJKVIGmU/v1HOaLWtOVkQWTYHw29Slu6woF8r6EM34I= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=X+R+Yr68; arc=none smtp.client-ip=209.85.210.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="X+R+Yr68" Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-84862b0d5aeso898056b3a.2 for ; Sun, 09 Aug 2026 03:54:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786272843; x=1786877643; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=NjYP8QIXD8TI/otj75y4h0mjvtlIR/GOS00qdTLY6LY=; b=X+R+Yr68zliGJhPGBgkbHk+OGkRr0I1oTVr5TaYUHdtuPkW3quXBr6GBmPP2X/s5EU aF6kSHPG9ObvHmleGCgM839iuFMgcxnmEL6NY/0UisE8yHp6VRioW9rtaFXHbaJhwsjq df4FXY0qeyKEX7l9FIRbatrHOU0ZU9o4R819p6G4dx8xvuqPfvVVhtEQQt1TCqsDsOsO oon5Nj6cqFTfTP2gWPnq0Xx463Ut41AfTFSGRis1fKa0FKvLYy1v/ZNcj6bmfmT6dZCz ZINEiMu0E74wkbcDfAUyGg46nTdaU/XBriyqc5WU9qAto3f9edY3OtjGgow4S4sYclvM zdZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786272843; x=1786877643; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=NjYP8QIXD8TI/otj75y4h0mjvtlIR/GOS00qdTLY6LY=; b=qV2zW/YaaqdAOVGL/qIA56cH9nuAznLf/PFqjPGVXVYbHj38voomPxgdDgTGGrmM5A o7vRuCjQAHEXDdmbZpeMFZ1AKC4tSn4jTP/31H3G9vEcBV4GBJ+bsAL6TV99mZTlWtS2 uknwL27jwmUMrNrUWNhfcG/m5wzrHnM1TB/nTF0h+nECbzDeypgg3Abfe28FOyAhCHfU p1LNlfogxA+/y8oSi5YX7QRlpp8XUmbK+DijKtgxPBfKnMLjV1MYptotuPp6FtS149wZ ZaKku4qy5GhJziMk8h8Vu0NqoIU1LRhvqc89JRRRFe307vxevym45Qf822nyDOdkQZXq eWTg== X-Forwarded-Encrypted: i=1; AHgh+Rrtl8mz/UEUW37Rr8cvMg3ut2EPnjnyQbYiE2ZrL6bkNUHqQBS4HVtFZBRoysFpOvgQn40GqIbEknW+kD8=@vger.kernel.org X-Gm-Message-State: AOJu0YwSzQ5+jl72R/SbeeoQ+5JnQfz/3cT8+VpVez6hBQskYdVybskS Djf/LDqhCnu7WcpFXpTnvN+W+kus0EFo/rg8uU/SdeGXFe3dGy/Xz9gK X-Gm-Gg: AR+sD12VM+WQLBlZpZMxRDaMLW1o5VZNzU2oZuEMs+NXuzA4opdP6XXfB6v3QTposD8 Ny684ZRHrI/pKGS+3sBaxm3Hz+0EVYg/rvmwebhPptQ8vQ3/uPkgaZhVktL/phH1EyzB0jmctJ1 6H2eeZ9L+xFRK90tO440Rt4Td+hqbVN5O47YvrCKMscuKJCaKfocvyCVq5QCwx7scxeFXqyQlxd KG2/dDcWXfzTjodiw67xFIU99keWWOYD+CbrbU/AxlMWMd0ISA3IQuNHZZkxxiCifPOpeezhO0M 5yLdHAQHcr5IFkU/ffMggUDKS4+ODCsLRBQeD0fg37UDM3sp4nGSaMJPN9S+LnObk1JtmRyyemK LeHgzNCSgBqmNZ7U+HUfYB8VCtU60J6dVjnZIVh4oBRdFDFDJyBTgFxqMOXFb6i/yCADk54GzGx O7XBAxMD6NJ8JRGT6G9Sq4ieppa8RsnxkJoeWBzEP30w8I+4wrwnoIasc5xGxPTWFo5zhzrZBm3 TEaBgmfc1JC1uyo6RM3PldkMQ== X-Received: by 2002:a05:6a00:17aa:b0:84e:e741:174f with SMTP id d2e1a72fcca58-84f2dfc8a36mr34095963b3a.7.1786272843153; Sun, 09 Aug 2026 03:54:03 -0700 (PDT) Received: from PC-2B0BA19500175.company.local ([210.184.73.204]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84f5a58d82asm2700129b3a.57.2026.08.09.03.53.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Aug 2026 03:54:02 -0700 (PDT) From: Lu Wang To: yu.c.chen@intel.com, tim.c.chen@linux.intel.com Cc: peterz@infradead.org, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com, linux-kernel@vger.kernel.org, chen.yu@linux.dev, Lu Wang Subject: [PATCH v2] sched/cache: honor migrate_llc_task semantics in active load balance Date: Sun, 9 Aug 2026 18:53:43 +0800 Message-ID: <20260809105343.1189051-1-wanglu.priv@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260801122252.2476258-1-wanglu.priv@gmail.com> References: <20260801122252.2476258-1-wanglu.priv@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit A passive load-balance pass marks group_llc_balance as migrate_llc_task and queues active balance when it cannot move a task. The CPU stopper callback constructs a fresh lb_env, so select the stopper callback when queueing active balance to preserve the migration semantics across the asynchronous boundary. For CAS-directed active balance, reject a candidate whose preferred LLC does not match the destination LLC. This keeps the fallback from moving a task away from its preferred LLC. Fixes: e4c9a4cb244a (\"sched/cache: Add migrate_llc_task migration type for cache-aware balancing\") Suggested-by: \"Chen, Yu C\" Signed-off-by: Lu Wang --- Changes in v2: - Select the stopper callback at kick time to preserve migrate_llc_task semantics in active load balance without passing migration_type across the stopper, which affects delayed-dequeue tasks. - Link to V1: https://lore.kernel.org/all/20260801122252.2476258-1-wanglu.priv@gmail.com/ kernel/sched/fair.c | 56 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 50 insertions(+), 6 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6..37a470c41 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10254,6 +10254,7 @@ enum migration_type { #define LBF_SOME_PINNED 0x08 #define LBF_ACTIVE_LB 0x10 #define LBF_LLC_PINNED 0x20 +#define LBF_ACTIVE_LB_LLC 0x40 struct lb_env { struct sched_domain *sd; @@ -10645,6 +10646,20 @@ alb_break_llc(struct lb_env *env) return false; } +/* + * Returns true if p's preferred LLC does not match the destination CPU + * under migrate_llc_task semantics. Passive LB passes migrate_llc_task + * in migration_type, while active LB carries it in LBF_ACTIVE_LB_LLC. + */ +static inline bool +migrate_llc_task_wrong_dst(struct task_struct *p, struct lb_env *env) +{ + return sched_cache_enabled() && + (env->migration_type == migrate_llc_task || + env->flags & LBF_ACTIVE_LB_LLC) && + READ_ONCE(p->preferred_llc) != llc_id(env->dst_cpu); +} + /* * Check if migrating task p from env->src_cpu to * env->dst_cpu breaks LLC localiy. @@ -10673,8 +10688,7 @@ static bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env) * run on env->dst_cpu, skip the tasks do not prefer * env->dst_cpu, and find the one that prefers. */ - if (env->migration_type == migrate_llc_task && - READ_ONCE(p->preferred_llc) != llc_id(env->dst_cpu)) + if (migrate_llc_task_wrong_dst(p, env)) return true; if (can_migrate_llc_task(env->src_cpu, @@ -10697,6 +10711,12 @@ alb_break_llc(struct lb_env *env) return false; } +static inline bool +migrate_llc_task_wrong_dst(struct task_struct *p, struct lb_env *env) +{ + return false; +} + static inline bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env) { @@ -10796,7 +10816,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) * 4) too many balance attempts have failed. */ if (env->flags & LBF_ACTIVE_LB) - return 1; + return !migrate_llc_task_wrong_dst(p, env); degrades = migrate_degrades_locality(p, env); if (!degrades) { @@ -13156,6 +13176,20 @@ static int need_active_balance(struct lb_env *env) } static int active_load_balance_cpu_stop(void *data); +static int active_load_balance_llc_cpu_stop(void *data); + +/* + * migration_type is checked elsewhere to decide migration policy, so + * it shouldn't be repurposed just to flag an LLC-directed active + * balance across the stopper. Pick the callback here instead. + */ +static inline cpu_stop_fn_t alb_stop_fn(struct lb_env *env) +{ + if (env->migration_type == migrate_llc_task) + return active_load_balance_llc_cpu_stop; + + return active_load_balance_cpu_stop; +} static int should_we_balance(struct lb_env *env) { @@ -13492,7 +13526,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq, raw_spin_rq_unlock_irqrestore(busiest, flags); if (active_balance) { stop_one_cpu_nowait(cpu_of(busiest), - active_load_balance_cpu_stop, busiest, + alb_stop_fn(&env), busiest, &busiest->active_balance_work); } preempt_enable(); @@ -13603,7 +13637,7 @@ update_next_balance(struct sched_domain *sd, unsigned long *next_balance) * least 1 task to be running on each physical CPU where possible, and * avoids physical / logical imbalances. */ -static int active_load_balance_cpu_stop(void *data) +static int __active_load_balance_cpu_stop(void *data, unsigned int lb_flags) { struct rq *busiest_rq = data; int busiest_cpu = cpu_of(busiest_rq); @@ -13653,7 +13687,7 @@ static int active_load_balance_cpu_stop(void *data) .src_cpu = busiest_rq->cpu, .src_rq = busiest_rq, .idle = CPU_IDLE, - .flags = LBF_ACTIVE_LB, + .flags = LBF_ACTIVE_LB | lb_flags, }; schedstat_inc(sd->alb_count); @@ -13681,6 +13715,16 @@ static int active_load_balance_cpu_stop(void *data) return 0; } +static int active_load_balance_cpu_stop(void *data) +{ + return __active_load_balance_cpu_stop(data, 0); +} + +static int active_load_balance_llc_cpu_stop(void *data) +{ + return __active_load_balance_cpu_stop(data, LBF_ACTIVE_LB_LLC); +} + /* * Scale the max sched_balance_rq interval with the number of CPUs in the system. * This trades load-balance latency on larger machines for less cross talk. -- 2.43.0