From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7698C4332F for ; Tue, 20 Dec 2022 06:58:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233221AbiLTG63 (ORCPT ); Tue, 20 Dec 2022 01:58:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233224AbiLTG6Y (ORCPT ); Tue, 20 Dec 2022 01:58:24 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD26A13EA2 for ; Mon, 19 Dec 2022 22:57:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1671519465; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PgjDO6YmPzI6EXFNoVsU/pqgee9rvpcuz0guJ0Z7ptM=; b=WvGLW6H0DDm8XyRwb/+W/aewtZDHfgTI7KUfGk8HUMwfUYRilVJ/TPTKBw3OmtGLsEggFD J8JNqZbo1WdZuQHVdyg+9YiSJ7aq67lo62qOOQLFrzZ0ePm7odluPM+Do9sHumML95IeXE h8+qM9y9JGRvnvsSzM6+Nby4mLYNmq4= Received: from mail-oo1-f72.google.com (mail-oo1-f72.google.com [209.85.161.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-393-k-gfCgR_Olmil1hH184bGw-1; Tue, 20 Dec 2022 01:57:38 -0500 X-MC-Unique: k-gfCgR_Olmil1hH184bGw-1 Received: by mail-oo1-f72.google.com with SMTP id v10-20020a4a860a000000b00480b3e2b5afso5267680ooh.12 for ; Mon, 19 Dec 2022 22:57:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=PgjDO6YmPzI6EXFNoVsU/pqgee9rvpcuz0guJ0Z7ptM=; b=ggPavgYedohHsWAwSr8/VbNhs70sog+sTrA96t4LdLbHv2nUucyUGU681JPB/eMYUD Vv+vztmlCepudvnBXSN/O8BgxNQDKROlGqFZPSrrIonUAtP3s0bRvlfZKQgvebfBrVVS sPJXdfk/OPUOHWjb3tJXYCkh4rzH2t4yFO4OuODurqbNY6bz6Piw1an5jglJeml7YmlP l/rEEHw8zgvSaowQhAmBwP1i64oEr6WJuxoX1Bg4EqRUHymE6gtf0ntnC0WJC3254tw1 +KLw3Dh/2eXvTZ+a6WKw4+OUFWLvHGiE6AakgK5T2oTtf5lgnNNXTmjfyFLO0VHKW9Is 54hw== X-Gm-Message-State: ANoB5pks4WbhkU+ZAjjq4fzQmsWqKXSiEDpGfDTaO0MQC/+DMB1VZdb4 IJY7+rbAj7JoqswHc6qo+xjOuU/q/hS1IgevR31UKTzxUB1HekttRszH85JAZwYFUV0Iqhi2CYX EXNRudhFVuxwaavMQMuZx X-Received: by 2002:aca:d909:0:b0:35b:b20d:53e9 with SMTP id q9-20020acad909000000b0035bb20d53e9mr18301881oig.55.1671519457265; Mon, 19 Dec 2022 22:57:37 -0800 (PST) X-Google-Smtp-Source: AA0mqf7gU8wXwyG7y7JilikoaWDJ51dcAKEUNsDSYXNSc+a5O4RaE++39oKb2zSEmb0O24Awl74Xdw== X-Received: by 2002:aca:d909:0:b0:35b:b20d:53e9 with SMTP id q9-20020acad909000000b0035bb20d53e9mr18301862oig.55.1671519456982; Mon, 19 Dec 2022 22:57:36 -0800 (PST) Received: from ?IPv6:2804:431:c7ec:1f35:e7a0:7a03:cbfa:5430? ([2804:431:c7ec:1f35:e7a0:7a03:cbfa:5430]) by smtp.gmail.com with ESMTPSA id bk22-20020a0568081a1600b003544822f725sm5196652oib.8.2022.12.19.22.57.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Dec 2022 22:57:36 -0800 (PST) Message-ID: Subject: Re: [PATCH v2 3/4] sched/isolation: Add HK_TYPE_WQ to isolcpus=domain From: Leonardo =?ISO-8859-1?Q?Br=E1s?= To: Frederic Weisbecker Cc: Peter Zijlstra , Steffen Klassert , Herbert Xu , "David S. Miller" , Bjorn Helgaas , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Tejun Heo , Lai Jiangshan , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Phil Auld , Antoine Tenart , Christophe JAILLET , Wang Yufen , mtosatti@redhat.com, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, netdev@vger.kernel.org, fweisbec@gmail.com Date: Tue, 20 Dec 2022 03:57:29 -0300 In-Reply-To: <20221129121051.GB1715045@lothringen> References: <20221013184028.129486-1-leobras@redhat.com> <20221013184028.129486-4-leobras@redhat.com> <20221014132410.GA1108603@lothringen> <7249d33e5b3e7d63b1b2a0df2b43e7a6f2082cf9.camel@redhat.com> <20221129121051.GB1715045@lothringen> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.46.2 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On Tue, 2022-11-29 at 13:10 +0100, Frederic Weisbecker wrote: > On Fri, Oct 14, 2022 at 01:27:25PM -0300, Leonardo Br=C3=A1s wrote: > > Hello Frederic, > >=20 > > So, IIUC you are removing all flags composing nohz_full=3D parameter in= favor of a > > unified NOHZ_FULL flag.=20 > >=20 > > I am very new to the code, and I am probably missing the whole picture,= but I > > actually think it's a good approach to keep them split for a couple rea= sons: > > 1 - They are easier to understand in code (IMHO):=C2=A0 > > "This cpu should not do this, because it's not able to do WQ housekeepi= ng" looks > > better than "because it's not in DOMAIN or NOHZ_FULL housekeeping" >=20 > A comment above each site may solve that. Sure, but not having to leave comments would be better. Or am I missing something? >=20 > >=20 > > 2 - They are simpler for using:=C2=A0 > > Suppose we have this function that should run at a WQ, but we want to k= eep them > > out of the isolated cpus. If we have the unified flags, we need to comb= ine both > > DOMAIN and NOHZ_FULL bitmasks, and then combine it again with something= like > > cpu_online_mask. It usually means allocating a new cpumask_t, and also = freeing > > it afterwards. > > If we have a single WQ flag, we can avoid the allocation altogether by = using > > for_each_cpu_and(), making the code much simpler. >=20 > I guess having a specific function for workqueues would arrange for it. You mean keeping a WQ housekeeping bitmap? This could be a solution, but it would affect only the WQ example. >=20 > >=20 > > 3 - It makes easier to compose new isolation modes: > > In case the future requires a new isolation mode that also uses the typ= es of > > isolation we currently have implemented, it would be much easier to jus= t compose > > it with the current HK flags, instead of having to go through all usage= s and do > > a cpumask_and() there. Also, new isolation modes would make (2) worse. >=20 > Actually having a new feature merged in HK_NOHZ_FULL would make it easier= to > handle as it avoids spreading cpumasks. I'm not sure I understand what yo= u > mean. IIUC, your queued patch merges the housekeeping types HK_TYPE_TIMER, HK_TYPE_RCU, HK_TYPE_MISC, HK_TYPE_TICK, HK_TYPE_WQ and HK_TYPE_KTHREAD in = a single HK_TYPE_NOHZ_FULL. Suppose in future we need a new isolation feature in cmdline, say=C2=A0 isol_new=3D, and it works exactly like nohz_full=3D, but = also needs to isolate cpulist against something else, say doing X. How would this get implemented? IIUC, following the same pattern: - A new type HK_TYPE_ISOL_NEW would be created together with a cpumask, - The new cpumask would be used to keep cpulist from doing X - All places that use HK_TYPE_NOHZ_FULL bitmap for isolation would need to = also bitmask_and() the new cpumask. (sometimes needing a local cpumask_t) Ok, there may be shortcuts for this, like keeping an intermediary bitmap, b= ut that can become tricky. Other more complex example: New isolation feature isol_new2=3D beh= aves like nohz_full=3D, keeps cpulist from doing X, but allows unbound = RCU work. Now it's even harder to have shortcuts from previous implementation. What I am trying to defend here is that keeping the HK_type with the idea o= f "things to get cpulist isolated from" works better for future implementatio= ns than a single flag with a lot of responsibilities: - A new type HK_TYPE_X would be created together with a cpumask, - The new cpumask would be used to keep cpulist from doing X - isol_new=3D is composed with the flags for what cpulist is getti= ng isolated. - (No need to touch already implemented isolations.) In fact, I propose that it works better for current implementations also: The current patch (3/4) takes the WQ isolation responsibility from HK_TYPE_DOMAIN and focus it in HK_TYPE_WQ, adding it to isolcpus=3D flags. This avoids some cpumask_and()s, and a cpumask_t kzalloc, and makes = the code less complex to implement when we need to put isolation in further par= ts of the code. (patch 4/4) I am not sure if I am missing some important point here.=C2=A0 Please let me know if it's the case.=20 >=20 > Thanks. >=20 Thank you for replying! Leo