From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0921C433FE for ; Mon, 21 Nov 2022 12:35:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231181AbiKUMfI (ORCPT ); Mon, 21 Nov 2022 07:35:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230506AbiKUMe4 (ORCPT ); Mon, 21 Nov 2022 07:34:56 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B99CA6A0B for ; Mon, 21 Nov 2022 04:34:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=ZcpLHc2YCyh5+c7me4kst+L3Ex3Jlpt5gOuzGZpHuWw=; b=WCCn/uYNi8EpKuY/HvqyHeglsh LNM0ytvHtQ9+fznRtFFtFHcASvqOJeddeOo+JWYXTUEjYoNoUhBO1SvNfAQ2H9KQS2O6MX4isFhG/ sIGF+Gt+7W4AO+VJ1b1RYuVKUnEnlap7fVQrztp9nBxux/hkvHKXv5+eEOxPidwAr3Rah8xdvLGzG UDOPbC5j9V3H6lO+bTcxRSUttbino5x3VH6Jz1lI8wEclChpbjA86r5/0x9rs/AjCoq9VVhFgV7U/ 0bWhKXKuDodS3w7k6HQx4EnNe5rY+KrRU3KnyNDx6n6EyJuwj1mj6BXQzNbkZxoAYAnQgI6j7F82Z UE8N0h4g==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1ox60Y-005D4w-Hl; Mon, 21 Nov 2022 12:34:42 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 231CD300445; Mon, 21 Nov 2022 13:34:34 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 1000) id F2DFD2D2DAA9D; Mon, 21 Nov 2022 13:34:33 +0100 (CET) Date: Mon, 21 Nov 2022 13:34:33 +0100 From: Peter Zijlstra To: Josh Don Cc: Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , linux-kernel@vger.kernel.org, Tejun Heo , Michal =?iso-8859-1?Q?Koutn=FD?= , Christian Brauner , Zefan Li Subject: Re: [PATCH v3] sched: async unthrottling for cfs bandwidth Message-ID: References: <20221117005418.3499691-1-joshdon@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Nov 18, 2022 at 11:25:09AM -0800, Josh Don wrote: > > Maybe have this function return false when local and save that cfs_rq in > > a local var to process again later, dunno, that might turn messy. > > Maybe something like this? Apologies for inline diff formatting. That looks entirely reasonable, not nearly as horrible as I feared. Let me go make that happen. > Note that one change we definitely want is the extra setting of > throttled = true in the case that cfs_rq->runtime_remaining <= 0, to > catch the case where we run out of runtime to distribute on the last > entity in the list. Done. > > > + > > > + /* Already enqueued */ > > > + if (SCHED_WARN_ON(!list_empty(&cfs_rq->throttled_csd_list))) > > > + return; > > > + > > > + list_add_tail(&cfs_rq->throttled_csd_list, &rq->cfsb_csd_list); > > > + > > > + smp_call_function_single_async(cpu_of(rq), &rq->cfsb_csd); > > > > Hurmph.. so I was expecting something like: > > > > first = list_empty(&rq->cfsb_csd_list); > > list_add_tail(&cfs_rq->throttled_csd_list, &rq->cfsb_csd_list); > > if (first) > > smp_call_function_single_async(cpu_of(rq), &rq->cfsb_csd); > > > > But I suppose I'm remembering the 'old' version. I don't think it is > > broken as written. There's a very narrow window where you'll end up > > sending a second IPI for naught, but meh. > > The CSD doesn't get unlocked until right before we call the func(). > But you're right that that's a (very) narrow window for an extra IPI. > Please feel free to modify the patch with that diff if you like. Since I was manually editing things, I did that too. Please test the final version as found here: https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/commit/?h=sched/core&id=4e3c1b7b489e218dfa576cd6af0680b975b8743e