From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D84DC43461 for ; Fri, 21 May 2021 11:27:02 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CCA26613D7 for ; Fri, 21 May 2021 11:27:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CCA26613D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:Cc:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Ujm0YQ5Us2g8oFBpo+6lISCCYKB7cChaMB51Pdwlebs=; b=T6gse3IdNbqrZWvtSV28IEJzNp ZT+3MhJzN+JgAWwwCZoRkVaWuRLTp34rAkB5lrIA3c1APRe8hCnnyWSUnOnNE2IbnSe48FeaRpJtV HuRRhaj2LiMjwDqMP4yOR4o4PAWRPHC2eTTrwdhPzbSWe7LCjM+KdX7CdlxQQpM2QUrIMLFSZHjmd ca/dmiJQH8QvVnHzSN0d9vAscpltPigbOOblrL4bcHxcDfjbX/lYtA5I623OiSt23yQF68xtC1wD2 xk2EMHfve+QTloZrpnlxQ7aRTNau77/zz7FRpp6Ua1ikC22qeaHBwrrDZ/r6CaHZuxnOwLVJypOXZ yZ63587Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lk3GR-0058Mv-CW; Fri, 21 May 2021 11:24:23 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lk3GN-0058MB-44 for linux-arm-kernel@desiato.infradead.org; Fri, 21 May 2021 11:24:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:Cc:To: Subject:Sender:Reply-To:Content-ID:Content-Description; bh=Q1fkq90+cu7eN13nXXAHK/hrxCr3oM3nOGCyDFfqw/Q=; b=x95I9M1yE4M6yClDEgAPrP0Z1U bBbDowSgqhx3Klja2uLLf97lD1COKB7QowU+vz3Jto+8RTayYJ3+Q7625wA6DeaZnCuZKLMNpdIgq XbWTWCr/3u1wpYLEGKCx5FoKgPwHJ1k0axNp/06JP8NeJWn9EXP5zNCptSydyBRUk+IskBXhktje8 +lf582UN8pNhyw02/CHeNVL/KnUCcLgRMo9oE8OEqIVh74E8th2AV3C5ROkpGoYyNqCcCxOYbjxy/ ZwJa7pmW50r8oAGR2+FwrLs/Ad63j25ozDzxdF3TM2gmKPsiqZTrxEDwRuQo65jJQWZzhX/cfb4Wq U+0zbOcA==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lk3GJ-00H4AA-OJ for linux-arm-kernel@lists.infradead.org; Fri, 21 May 2021 11:24:17 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0C3B51424; Fri, 21 May 2021 04:24:11 -0700 (PDT) Received: from [192.168.1.16] (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 99D383F73D; Fri, 21 May 2021 04:24:07 -0700 (PDT) Subject: Re: [PATCH v6 13/21] sched: Admit forcefully-affined tasks into SCHED_DEADLINE To: Will Deacon , Juri Lelli Cc: Quentin Perret , Daniel Bristot de Oliveira , linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, Catalin Marinas , Marc Zyngier , Greg Kroah-Hartman , Peter Zijlstra , Morten Rasmussen , Qais Yousef , Suren Baghdasaryan , Tejun Heo , Johannes Weiner , Ingo Molnar , Vincent Guittot , "Rafael J. Wysocki" , kernel-team@android.com References: <20210518105951.GC7770@willie-the-truck> <20210520101640.GA10065@willie-the-truck> <20210520180138.GA10523@willie-the-truck> <20210521103724.GA11680@willie-the-truck> From: Dietmar Eggemann Message-ID: <3620bad5-2a27-0f9e-f1f0-70036997d33c@arm.com> Date: Fri, 21 May 2021 13:23:55 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <20210521103724.GA11680@willie-the-truck> Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210521_042415_930998_E292E194 X-CRM114-Status: GOOD ( 31.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 21/05/2021 12:37, Will Deacon wrote: > On Fri, May 21, 2021 at 10:39:32AM +0200, Juri Lelli wrote: >> On 21/05/21 08:15, Quentin Perret wrote: >>> On Friday 21 May 2021 at 07:25:51 (+0200), Juri Lelli wrote: >>>> On 20/05/21 19:01, Will Deacon wrote: >>>>> On Thu, May 20, 2021 at 02:38:55PM +0200, Daniel Bristot de Oliveira wrote: >>>>>> On 5/20/21 12:33 PM, Quentin Perret wrote: >>>>>>> On Thursday 20 May 2021 at 11:16:41 (+0100), Will Deacon wrote: >>>>>>>> Ok, thanks for the insight. In which case, I'll go with what we discussed: >>>>>>>> require admission control to be disabled for sched_setattr() but allow >>>>>>>> execve() to a 32-bit task from a 64-bit deadline task with a warning (this >>>>>>>> is probably similar to CPU hotplug?). >>>>>>> >>>>>>> Still not sure that we can let execve go through ... It will break AC >>>>>>> all the same, so it should probably fail as well if AC is on IMO >>>>>>> >>>>>> >>>>>> If the cpumask of the 32-bit task is != of the 64-bit task that is executing it, >>>>>> the admission control needs to be re-executed, and it could fail. So I see this >>>>>> operation equivalent to sched_setaffinity(). This will likely be true for future >>>>>> schedulers that will allow arbitrary affinities (AC should run on affinity >>>>>> change, and could fail). >>>>>> >>>>>> I would vote with Juri: "I'd go with fail hard if AC is on, let it >>>>>> pass if AC is off (supposedly the user knows what to do)," (also hope nobody >>>>>> complains until we add better support for affinity, and use this as a motivation >>>>>> to get back on this front). >>>>> >>>>> I can have a go at implementing it, but I don't think it's a great solution >>>>> and here's why: >>>>> >>>>> Failing an execve() is _very_ likely to be fatal to the application. It's >>>>> also very likely that the task calling execve() doesn't know whether the >>>>> program it's trying to execute is 32-bit or not. Consequently, if we go >>>>> with failing execve() then all that will happen is that people will disable >>>>> admission control altogether. >>> >>> Right, but only on these dumb 32bit asymmetric systems, and only if we >>> care about running 32bits deadline tasks -- which I seriously doubt for >>> the Android use-case. >>> >>> Note that running deadline tasks is also a privileged operation, it >>> can't be done by random apps. >>> >>>>> That has a negative impact on "pure" 64-bit >>>>> applications and so I think we end up with the tail wagging the dog because >>>>> admission control will be disabled for everybody just because there is a >>>>> handful of 32-bit programs which may get executed. I understand that it >>>>> also means that RT throttling would be disabled. >>>> >>>> Completely understand your perplexity. But how can the kernel still give >>>> guarantees to "pure" 64-bit applications if there are 32-bit >>>> applications around that essentially broke admission control when they >>>> were restricted to a subset of cores? >>>> >>>>> Allowing the execve() to continue with a warning is very similar to the >>>>> case in which all the 64-bit CPUs are hot-unplugged at the point of >>>>> execve(), and this is much closer to the illusion that this patch series >>>>> intends to provide. >>>> >>>> So, for hotplug we currently have a check that would make hotplug >>>> operations fail if removing a CPU would mean not enough bandwidth to run >>>> the currently admitted set of DEADLINE tasks. >>> >>> Aha, wasn't aware. Any pointers to that check for my education? >> >> Hotplug ends up calling dl_cpu_busy() (after the cpu being hotplugged out >> got removed), IIRC. So, if that fails the operation in undone. > > Interesting, thanks. Thinking about this some more, it strikes me that with > these silly asymmetric systems there could be an interesting additional > problem with hotplug and deadline tasks. Imagine the following sequence of > events: > > 1. All online CPUs are 32-bit-capable > 2. sched_setattr() admits a 32-bit deadline task > 3. A 64-bit-only CPU is onlined > 4. Some of the 32-bit-capable CPUs are offlined > > I wonder if we can get into a situation where we think we have enough > bandwidth available, but in reality the 32-bit task is in trouble because > it can't make use of the 64-bit-only CPU. > > If so, then it seems to me that admission control is really just > "best-effort" for 32-bit deadline tasks on these systems because it's based > on a snapshot in time of the available resources. IMHO DL AC is per root domain (rd). So if we have e.g. an 8 CPU system with aarch32_el0 eq. [0-3] then we would need 2 exclusive cpusets ([0-3] and [4-7]) to admit 32-bit DL tasks into [0-3] (i.e. to pass the `if (!cpumask_subset(span, p->cpus_ptr) ...` test in __sched_setscheduler(). Trying to admit too many 32-bit DL tasks or trying to hp out a CPU[0-3] would lead to `Device or resource busy` in case the rd bw wouldn't be sufficient anymore for the set of admitted tasks. But the [0-3] DL AC wouldn't care about hp on CPU[4-7]. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel