From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3003EB64DB for ; Tue, 20 Jun 2023 18:33:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229658AbjFTSdO (ORCPT ); Tue, 20 Jun 2023 14:33:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229567AbjFTSdM (ORCPT ); Tue, 20 Jun 2023 14:33:12 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF4AF1737 for ; Tue, 20 Jun 2023 11:33:01 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-bd5059e5347so12061876276.0 for ; Tue, 20 Jun 2023 11:33:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687285981; x=1689877981; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=sBPwNjci+mVKgvTGGSvenUmGifZY3Myji5qjgOuCzdY=; b=CV163j2PUGaavyM9SvGsrlqZLN2xBHrKdDynwlJOjsbY0BnCYcw8OuXxIYqTcePd/4 75NNiIVKAORJDQqeVbt8R+SXRThRGQvhEqlcna+1G+zhYFac2tqz+Kx++pU6u41NWRnI IzUsvovlGgT4EJdq0PTaZ4Dm0WIXjBtD0wLcnJDoVjgRFlPCmyTU8VsNYLZJDmPqxxCL OT/TGrpF+2O7PaMxGBI6yeul6VwI9pwRx5/fUXqA3PQO4hd7oy3cRJyi66jJCV40Dd2x EjZ+GR+fkHQuIwCMlkwh6W3+frRhMueAcukods0XG7+nZMO2shq9cUUTv+JNlJgKJmLI hdkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687285981; x=1689877981; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=sBPwNjci+mVKgvTGGSvenUmGifZY3Myji5qjgOuCzdY=; b=i8sGZBGR0ZvxFIKuZKApA62FeRzY1pSb3pEHgTwVFhwGGN2Hj0oW/vl4advXGtEYD7 d7Puta3CXQZPhx050yFvJmkjBk/YiYWM5SF1idXAMTmmqmh43eEZfyqKQRBmZ9y9sqSS DsrMin6cmdamF1J4g116jsa70Rrp1qGv+0/Tz3S+tWjaS4FMWwsIULAb3aEejel96Do4 tqh35Cjy9s9q0kgxYsmClk6oIiZQt1ckBw9dNjcQKxhVBg74kcYzpgBxRHOkSnioBIuW DjOxvtFUYIdmn5FzgZOwy1bgysIQIXlCa62jlXocRWxJEloJTuv+5Cth3bN9NyC0Uamk EQOQ== X-Gm-Message-State: AC+VfDzRN7oyVD1DGU3pJYfLYpc9+YO1F9+N9vU1yRZZOjADEJCNJb/A vKS+Akkb/OczqoOd9YukqljJlDtf+ny5 X-Google-Smtp-Source: ACHHUZ7bOatjzQX/Jut8CVloG02AZ3Trw8/m9G2F+ZHZd5l0tbEhR1q1/c7ee6EvEK8TAYW2EztlWHwe8MOW X-Received: from joshdon-desktop.svl.corp.google.com ([2620:15c:2a3:200:7f30:1fa1:c3c6:d39a]) (user=joshdon job=sendgmr) by 2002:a05:6902:561:b0:bad:2b06:da3 with SMTP id a1-20020a056902056100b00bad2b060da3mr4934249ybt.3.1687285981208; Tue, 20 Jun 2023 11:33:01 -0700 (PDT) Date: Tue, 20 Jun 2023 11:32:46 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230620183247.737942-1-joshdon@google.com> Subject: [PATCH v3 1/2] sched: don't account throttle time for empty groups From: Josh Don To: Peter Zijlstra Cc: Tejun Heo , Zefan Li , Johannes Weiner , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, mkoutny@suse.com, Josh Don Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org It is easy for a cfs_rq to become throttled even when it has no enqueued entities (for example, if we have just put_prev()'d the last runnable task of the cfs_rq, and the cfs_rq is out of quota). Avoid accounting this time towards total throttle time, since it otherwise falsely inflates the stats. Note that the dequeue path is special, since we normally disallow migrations when a task is in a throttled hierarchy (see throttled_lb_pair()). Signed-off-by: Josh Don --- v3: fix bug with nested limits kernel/sched/fair.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6189d1a45635..72571d48559a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4787,6 +4787,7 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) } static void check_enqueue_throttle(struct cfs_rq *cfs_rq); +static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq); static inline bool cfs_bandwidth_used(void); @@ -4873,8 +4874,14 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) if (cfs_rq->nr_running == 1) { check_enqueue_throttle(cfs_rq); - if (!throttled_hierarchy(cfs_rq)) + if (!throttled_hierarchy(cfs_rq)) { list_add_leaf_cfs_rq(cfs_rq); + } else { +#ifdef CONFIG_CFS_BANDWIDTH + if (cfs_rq_throttled(cfs_rq) && !cfs_rq->throttled_clock) + cfs_rq->throttled_clock = rq_clock(rq_of(cfs_rq)); +#endif + } } } @@ -5480,7 +5487,9 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) * throttled-list. rq->lock protects completion. */ cfs_rq->throttled = 1; - cfs_rq->throttled_clock = rq_clock(rq); + SCHED_WARN_ON(cfs_rq->throttled_clock); + if (cfs_rq->nr_running) + cfs_rq->throttled_clock = rq_clock(rq); return true; } @@ -5498,7 +5507,10 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) update_rq_clock(rq); raw_spin_lock(&cfs_b->lock); - cfs_b->throttled_time += rq_clock(rq) - cfs_rq->throttled_clock; + if (cfs_rq->throttled_clock) { + cfs_b->throttled_time += rq_clock(rq) - cfs_rq->throttled_clock; + cfs_rq->throttled_clock = 0; + } list_del_rcu(&cfs_rq->throttled_list); raw_spin_unlock(&cfs_b->lock); -- 2.41.0.162.gfafddb0af9-goog