From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on archive.lwn.net X-Spam-Level: X-Spam-Status: No, score=-13.3 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_NONE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.2 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by archive.lwn.net (Postfix) with ESMTP id B58ED7D2F0 for ; Thu, 11 Jul 2019 17:46:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728682AbfGKRqV (ORCPT ); Thu, 11 Jul 2019 13:46:21 -0400 Received: from mail-pf1-f195.google.com ([209.85.210.195]:34191 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728372AbfGKRqV (ORCPT ); Thu, 11 Jul 2019 13:46:21 -0400 Received: by mail-pf1-f195.google.com with SMTP id b13so3104437pfo.1 for ; Thu, 11 Jul 2019 10:46:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=p6qEv8fgTgyqvoOypH+wlalxdPH8SzcdGhJs/o+CWGY=; b=KqgMVRLIbqcIGHDrID8LNbN5FhzJf6sWR5x8kG4hVZMGkWfXDf4dYYgqqc9r74+D+Q wuINI9UEqv8qDAKzJm7CB83/eFNwxqxmhTHeLQRHGOESUpu0vBwWu1oxtA4oCCPSOsSe r3X9iDMN0fhiQ0e+5Kzvg0Y5s1qlC2rBHCD/GPNlEWMvgnuaaSpfgpu/rVdBbyzwvEud kXhE2cDpi24HZpVLDoJFTL5cQbavH/wqeOH18eTMBQFMOhOXXCSDf5tMRrXiX9gepxCr TMVcBc9/J0pn9iK4Cx/Zb+IxnoRu7OGZSSUPMBZ/1QmXnWGAL1wphIaCkt+Giijcu9nw YI7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=p6qEv8fgTgyqvoOypH+wlalxdPH8SzcdGhJs/o+CWGY=; b=QWOJlfPzjv+p5KeRBWBPWIWvVE4IC4avkClM0voPdHOZqiXvpE/lThDrwsqyQJFobu jH7CvzYNmJtzEOv2n9LN9sB0MDphNBdVZH4KGGoegRqe5H4UC3mvG1zwCZ8J6PmY/GSa FEM5mZN7uVLH1N5fPboP1R4zSTsWN/1cRyzgf/tFg3rmRkxKVONZaBRq9I7MRp8oPIEY JKYTtCtmAMpyMFDyi4kTj2NdbPAyt160hs5AqP2R149qIbzPNRHrnZiv3uHkns844jr+ zPaWUBikQtVS2poMaQx0JLrrNWXOXC1DvLyx6sXhySlCUT3IS4ut4A2mNEjTjeZUuFa0 uMNQ== X-Gm-Message-State: APjAAAXZ2StmkiEHr9scd4M9umcOKPgVxj+QdFMDcAXN6982+LRjMDoS W98zKMpa4Y2EEs4Zq1j/p4UNTA== X-Google-Smtp-Source: APXvYqw1QkrRAzsxd2h5YvqzpXFKBpulU11tj9hJY93uhZyX+IEDc6Hma+dKamdBB6NDC4DWsXYMZQ== X-Received: by 2002:a63:1046:: with SMTP id 6mr5833405pgq.111.1562867180047; Thu, 11 Jul 2019 10:46:20 -0700 (PDT) Received: from bsegall-linux.svl.corp.google.com.localhost ([2620:15c:2cd:202:39d7:98b3:2536:e93f]) by smtp.gmail.com with ESMTPSA id 14sm10447986pfj.36.2019.07.11.10.46.18 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Thu, 11 Jul 2019 10:46:19 -0700 (PDT) From: bsegall@google.com To: Peter Zijlstra Cc: Dave Chiluk , Pqhil Auld , Peter Oskolkov , Ingo Molnar , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Brendan Gregg , Kyle Anderson , Gabriel Munos , John Hammond , Cong Wang , Jonathan Corbet , linux-doc@vger.kernel.org, Paul Turner Subject: Re: [PATCH v5 1/1] sched/fair: Fix low cpu usage with high throttling by removing expiration of cpu-local slices References: <1558121424-2914-1-git-send-email-chiluk+linux@indeed.com> <1561664970-1555-1-git-send-email-chiluk+linux@indeed.com> <1561664970-1555-2-git-send-email-chiluk+linux@indeed.com> <20190711095102.GX3402@hirez.programming.kicks-ass.net> Date: Thu, 11 Jul 2019 10:46:18 -0700 In-Reply-To: <20190711095102.GX3402@hirez.programming.kicks-ass.net> (Peter Zijlstra's message of "Thu, 11 Jul 2019 11:51:02 +0200") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain Sender: linux-doc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-doc@vger.kernel.org Peter Zijlstra writes: > FWIW, good to see progress, still waiting for you guys to agree :-) > > On Mon, Jul 01, 2019 at 01:15:44PM -0700, bsegall@google.com wrote: > >> - Taking up-to-every rq->lock is bad and expensive and 5ms may be too >> short a delay for this. I haven't tried microbenchmarks on the cost of >> this vs min_cfs_rq_runtime = 0 vs baseline. > > Yes, that's tricky, SGI/HPE have definite ideas about that. > >> @@ -4781,12 +4790,41 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq) >> */ >> static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b) >> { >> - u64 runtime = 0, slice = sched_cfs_bandwidth_slice(); >> + u64 runtime = 0; >> unsigned long flags; >> u64 expires; >> + struct cfs_rq *cfs_rq, *temp; >> + LIST_HEAD(temp_head); >> + >> + local_irq_save(flags); >> + >> + raw_spin_lock(&cfs_b->lock); >> + cfs_b->slack_started = false; >> + list_splice_init(&cfs_b->slack_cfs_rq, &temp_head); >> + raw_spin_unlock(&cfs_b->lock); >> + >> + >> + /* Gather all left over runtime from all rqs */ >> + list_for_each_entry_safe(cfs_rq, temp, &temp_head, slack_list) { >> + struct rq *rq = rq_of(cfs_rq); >> + struct rq_flags rf; >> + >> + rq_lock(rq, &rf); >> + >> + raw_spin_lock(&cfs_b->lock); >> + list_del_init(&cfs_rq->slack_list); >> + if (!cfs_rq->nr_running && cfs_rq->runtime_remaining > 0 && >> + cfs_rq->runtime_expires == cfs_b->runtime_expires) { >> + cfs_b->runtime += cfs_rq->runtime_remaining; >> + cfs_rq->runtime_remaining = 0; >> + } >> + raw_spin_unlock(&cfs_b->lock); >> + >> + rq_unlock(rq, &rf); >> + } > > But worse still, you take possibly every rq->lock without ever > re-enabling IRQs. > Yeah, I'm not sure why I did that, it isn't correctness.