From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8D43246A37 for ; Wed, 15 Jan 2025 08:55:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736931335; cv=none; b=MSBY4kq57VjQ4+frndhp75IMVELZR+xZWHREHSSj43+gV9UC7BzYFZms+Nydgs9k7DfywJ3BK7+SWxYpJGWVFpluaKS34LrZMpWFm0h/mLCkCttkls0fqa3wbUg1+nuIEdPS/FqswM/y2Ro7a6NW5xiJ2voxKRS/QEa/6gFSCnA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736931335; c=relaxed/simple; bh=LHC3tjOIk2OtyPXXk53Xl4z+Fclh8r/FxwC58VnrYyc=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=n0Qu4qL8fg1J2kKW0PHiYYocFHu4UBMellIjvKf9o3w5a6zVEKTa0v62++/TZANtbxfG2gU95tB5R2C42spXBcS9fuY0pyymZVJZyyDpTclBRZIHbRmLeYS2UflQAC/JQ8xCslBoLNRY+rFuxgkh/sZD/KtjI9sxHeB0c+Pa4Jc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DS+CpMfR; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DS+CpMfR" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2166022c5caso100800585ad.2 for ; Wed, 15 Jan 2025 00:55:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736931333; x=1737536133; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to:subject :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=nOhVESpKgWXk5uMAciSNClSia3mwFvmYMOqfsWrB2i8=; b=DS+CpMfRihTAJmiWKH1BnF3zpSvz5PbEm865kRy87LfS+3bMzDZQbFqqYSSONwH7Js yD0BxMSG1QqdPWg9tACELwRd+rs7rbEBWh4JsryCCVgU+qeGWeFr1MnXRH0LBgbQvKCb bH2kHPTGheVy7iXDXX/4bR8v/d+fadB5J62JJHw39kKIea/RLGyiIbzEX2J09MKIaENS obCSi4HwPQYbl0dnVDll4QDCuDTHYI+8JsyQPMnFcP+9/8FrKImaPLB9LKOlenVNq5Mm Ivg+/KjBo9eC7NC3NfHiz8+GqoBXD3gxx46sW0ucIMQ97z5MMYkpTYS17je4t6CQgGja Uegw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736931333; x=1737536133; h=content-transfer-encoding:in-reply-to:from:references:cc:to:subject :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=nOhVESpKgWXk5uMAciSNClSia3mwFvmYMOqfsWrB2i8=; b=KwR00fSM/eg/0Hm75y1vdfTCDvjVpLKJRVV4yKC9r0+CQyhRLYCyLPOVTuhkNaS7oz jxlfjkq4HXOXYhWwUbYg0FxsM9LVVQ1LHAatjskibSMjJ8lPgsinG2ch/viADyPgyAOQ oUjSg/+1octXOrMGIYg/cHNsVKDJSZi+/awb7zILfE3KJe/SBVCP9eIFIbi+AkTHyThy xaY+o5qAIfcwrXaEncZt1J48h8EthIE+FLI0PGKJdbgwDMUCXiQ2aJJiEXanS/V++q5u NBwwc0roLIQaH0ZL1t+jjXdm2s/u8P/QOdl6sOYB2xs5uRzOwKkHDtoNce5LU9+nMd7H TNTw== X-Forwarded-Encrypted: i=1; AJvYcCVYA8Xo9/4lDAbLtfqxOqkIFoDiZJNw19z/8QYrvE8iaxljO5avodgBVNcFz2QrzjcR20xHNtD9y2MyLGM=@vger.kernel.org X-Gm-Message-State: AOJu0Yw3KihIkelrgADT4XNydj8saga/UF2012NoxsrZvqMAUlbVvYyL d9G60GGriBw5+ghItQnXcu9j9oXPENDZ1NcmIJso419ndTZIvXAK X-Gm-Gg: ASbGncvT6P8ZAwThkoQGerZEywistENaZEAM16O7hmmjahQzXwnElXs8oEep9M7ER5n ndjJgmzk5N/rAmbQLX3SdofDo3O7xHt0XofirbdnkGRQRlAnoAdCDN3PnpWq3iGmGW/c5BySHqP LDhLWzqqTmekoaKhzvHiQm02PaiTFklAkiKg8uwT/hfnmerhVTxCRIsg5HGCdtgjLjWwnv7HfTw mepaxCTjYrCtk7X35XPilNJzwxWUWwYMKZS8/dEx/mMLEcSMOtNnCloCg/IijuhHofhUgQZdgWW X-Google-Smtp-Source: AGHT+IF202P7jZsCZeHasXKbZfBTyR9O44LVLlvOaTtXUeroVIbXWIbTc1xVVSCYFjeotDfxDG9a2g== X-Received: by 2002:a17:902:ce8a:b0:21a:5501:9d5 with SMTP id d9443c01a7336-21a83fde4femr458933295ad.44.1736931332805; Wed, 15 Jan 2025 00:55:32 -0800 (PST) Received: from [10.125.112.6] ([103.165.80.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f22edbcsm77690175ad.197.2025.01.15.00.55.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 15 Jan 2025 00:55:32 -0800 (PST) Message-ID: Date: Wed, 15 Jan 2025 16:55:26 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Subject: Re: [PATCH v2] sched/core: Prioritize migrating eligible tasks in sched_balance_rq() To: Vincent Guittot Cc: mingo@redhat.com, peterz@infradead.org, mingo@kernel.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, Hao Jia References: <20241223091446.90208-1-jiahao.kernel@gmail.com> <597384e3-6519-b10e-081b-30c3f89b6e3f@gmail.com> From: Hao Jia In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 2025/1/14 16:07, Vincent Guittot wrote: > On Tue, 14 Jan 2025 at 04:18, Hao Jia wrote: >> >> >> >> On 2025/1/14 00:40, Vincent Guittot wrote: >>> On Mon, 13 Jan 2025 at 10:21, Hao Jia wrote: >>>> >>>> Friendly ping... >>>> >>>> >>>> On 2024/12/23 17:14, Hao Jia wrote: >>>>> From: Hao Jia >>>>> >>>>> When the PLACE_LAG scheduling feature is enabled and >>>>> dst_cfs_rq->nr_queued is greater than 1, if a task is >>>>> ineligible (lag < 0) on the source cpu runqueue, it will >>>>> also be ineligible when it is migrated to the destination >>>>> cpu runqueue. Because we will keep the original equivalent >>>>> lag of the task in place_entity(). So if the task was >>>>> ineligible before, it will still be ineligible after >>>>> migration. >>>>> >>>>> So in sched_balance_rq(), we prioritize migrating eligible >>>>> tasks, and we soft-limit ineligible tasks, allowing them >>>>> to migrate only when nr_balance_failed is non-zero to >>>>> avoid load-balancing trying very hard to balance the load. >>> >>> Could you explain why you think it's better to balance eligible tasks >>> in priority and potentially skip a load balance ? >> >> In place_entity(), we maintain the task's original equivalent lag, even >> if we migrate the task to dst_rq, this does not change its eligibility >> attribute. > > Yes, but you don't answer the question why it's better to select an > eligible task vs a non eligible task. > >> >> When there are multiple tasks on src_rq, and the dst_cpu has some >> runnable tasks, migrating ineligible tasks to dst_rq will not allow them >> to run. Therefore, such task migration is inefficient. We should > > Why is it inefficient ? load balance is about evenly balancing the > number of tasks or the load between CPUs, it never says that the newly > migrated task should run immediately My initial thought is that when we need to migrate some tasks during load balancing, at the current point in time, migrating ineligible tasks to dst_cpu means they definitely cannot run there. Therefore, I prefer to keep them on src_cpu to reduce the overhead of dequeueing and enqueueing ineligible tasks. Migrating eligible tasks to dst_cpu does not guarantee that they will run earlier than on src_cpu. it depends on too many factors. > >> prioritize migrating tasks that can run on dst_rq. >> >> In other words, migrating ineligible tasks is merely moving them to >> another runqueue to wait until they become eligible. > > But I don't get why it's a problem. Migrating an eligible task might > delay its scheduling because of its deadline vs other tasks already > eligible on the dst_rq. Eligible and non eligible tasks are all > runnable, it's just how much they have already run. In addition, > migrating an eligible task will clear its positive vlag with > DELAY_ZERO which is unfair IMO Sorry, I'd like to ask you a question that confuses me: Why does migrating eligible task will clear the positive vlag? In detach_task(), the ENQUEUE_DELAYED and DEQUEUE_SLEEP flags are not set, and in dequeue_entity(), eligible tasks will not set sched_delayed, so they will be dequeued normally with se->on_rq being 0. Similarly, attach_task() does not set the ENQUEUE_DELAYED and DEQUEUE_SLEEP flags, and since se->on_rq is 0, it will not call requeue_delayed_entity(). Thanks, Hao > >> >> >>> >>> I can see an interest for idle and newly_idle load balance in order to >>> favor fairness as tasks will become eligible but I don't see why it >>> would be helpful if dst already has some runnable tasks. Furthermore, >>> when a cpu is idle or newly idle, we really want to migrate a task >>> even an non eligible one instead of possibly skipping this load >>> balance round. With your patch, we might end up not pulling any task, >>> increasing the nr_balance_failed and waiting next load balance >>> >> >> If I understand correctly, when the destination CPU is idle, my patch >> does not change the original behavior. it only prevents the migration of >> ineligible tasks when dst_cfs_rq->nr_queued is greater than 1. > > It changes the behavior. My concern is that migrating an eligible task > when dst rq already has runnable tasks, doesn't assure you that it > will give any advantage to this eligible task. > > On an idle or newly idle cpu, any runnable tasks that will be pulled > will immediately start to run whatever it was eligible or not on src > cpu. In such a situation, we could consider that selecting an eligible > task which has less running time (positive lag) than others could be > more fair because the eligible task will immediately run. But this is > true as long as we migrate only 1 task. If you migrate several tasks, > an eligible task on src rq could even become ineligible quicker on dst > cpu than on src cpu has it lost its lag >