From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9A9F2C54E71 for ; Fri, 22 Mar 2024 14:25:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CUBQvoSwaRmjUuB+Gkn7+IATLY3QLqhGFDYwvCGTXb4=; b=OdMg9/b5PQJ5c3 o1lO66Bummy4RNtNSJHQmN+/jxdPv8b/aCqdXMsMbOcEbX811ELwlV3+cOaxGIsf6wM1YwX385Qa2 Qv4h8BNmZZ7vl9lkc/JL8BgU8wwXP8hhPdlnFzVmvpZwho42N20WI+v/f+RaBXX4uekmLP0/Snw2P v+8W4ALqBDLuTM6/PmMxv+PC/lAtRYnHIhFnQy6ZYsFafy/z1PcTiKKjBt/jQjZ/H3Fn8B6BCmeSU udgS9634lBhlNF001ZW25829mr1e4HS4V/g2VU06hjOI44vQxgi02aDMeKTLn+wocK/CcWNW3jzIl Z/JWgQPsRixz//IOtDjw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnfpL-00000007WjO-1uLg; Fri, 22 Mar 2024 14:24:59 +0000 Received: from mail-ej1-x62a.google.com ([2a00:1450:4864:20::62a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnfpC-00000007WfS-0pXq for linux-arm-kernel@lists.infradead.org; Fri, 22 Mar 2024 14:24:58 +0000 Received: by mail-ej1-x62a.google.com with SMTP id a640c23a62f3a-a44665605f3so259099466b.2 for ; Fri, 22 Mar 2024 07:24:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711117482; x=1711722282; darn=lists.infradead.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=jpJ4vef/DM0XTXXII9U2wn8MVlRJg1M1abd5hGl+Ct4=; b=lMaxU58NYwXWPsno8D5TRC02EdbUOzfckF+tfmJhTMXBScuuqIK2MocPTBIOuU6po1 G6jgtJcjn/iUsOu5MsoLJfSzt41tZsjdBvSSi0uBP0VINMfpX4cCazIqVNkQx1MA5n4e aSkZBdIswtQjZLmeWZXbHXaS0YNykBb2XmRT8jAQsku//kNvgWBoi6TS3KRfSniUxlKy Ni7DpdCGBJ/YktUpRpymu0iKOx6gJu4SZbKolrByTo3QZDTKvMHhViRlmBf7l17zjdJf 0HERmG2hhlY2JJGhjOaAUdTqO60xj064knXZFyjWM2fYth8h2i25sbQF0tc7mcNF+nZd 8hIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711117482; x=1711722282; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=jpJ4vef/DM0XTXXII9U2wn8MVlRJg1M1abd5hGl+Ct4=; b=N46c2uQkltFhhjv0zLMrAl4vpy2OtIFGfNHkzBJ3wamaepDx9nndQUjtOqc6hzxjeE /rWR5+fq4zFlkybizOXAnE/P9+vCUVBtYjUTe6iuVE6VUQbW0VCQmZ6houvC6CnooKyU ZYcKP/f2w34y836f1fXSPgnq0Z50mjNj2w0PHRCREAbZqM6LafHkacTuXotC/CyEoWI9 abE3jpjmHV4i0l4dz6qu07aOBFB21wP2XGlh7T/ucFaAhJMFTTrhZvdQp9HHjhXNJ5l7 0jsPjzGsPVeR168mMYMJbO/OJJFPm4Bomb1qHuG+78x/d1UB82wqtU+Hnto1h9bSVdZx tXZw== X-Forwarded-Encrypted: i=1; AJvYcCWVpylffNy+cby4KsNAl3Tlw7dKdenbw0dwEriNHO620IsaVVowRYzu21yXYn3MYuDHCH1vCDiE/Oo5glHU5lUySld7gvs8/w6oMqPPcP6+83efEhI= X-Gm-Message-State: AOJu0YwVtrXE63oS0cDiSMCdjZcsOSDJMgaPNO+DQgXrxOhm5sUTEv6x 2h6ySXRQjWYfSrS7GliptI9p+c8mzl0vESpijg4AFkR9x1+OXgB9+h1pdAUTcg== X-Google-Smtp-Source: AGHT+IHV7ZedVCo+7sdEER+uodJailEQLJUTasnzC10mtbi8r3xphXx/fdmPks0AX5p8D0JfI3tpTw== X-Received: by 2002:a17:907:b9ce:b0:a46:af7d:28d3 with SMTP id xa14-20020a170907b9ce00b00a46af7d28d3mr1785337ejc.11.1711117481409; Fri, 22 Mar 2024 07:24:41 -0700 (PDT) Received: from google.com (64.227.90.34.bc.googleusercontent.com. [34.90.227.64]) by smtp.gmail.com with ESMTPSA id r16-20020a170906351000b00a46cddf1efbsm1065100eja.29.2024.03.22.07.24.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Mar 2024 07:24:40 -0700 (PDT) Date: Fri, 22 Mar 2024 14:24:35 +0000 From: Quentin Perret To: Colton Lewis Cc: kvm@vger.kernel.org, Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Paolo Bonzini , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2] KVM: arm64: Add KVM_CAP to control WFx trapping Message-ID: References: <20240319164341.1674863-1-coltonlewis@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240319164341.1674863-1-coltonlewis@google.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240322_072453_666333_E4599E52 X-CRM114-Status: GOOD ( 21.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tuesday 19 Mar 2024 at 16:43:41 (+0000), Colton Lewis wrote: > Add a KVM_CAP to control WFx (WFI or WFE) trapping based on scheduler > runqueue depth. This is so they can be passed through if the runqueue > is shallow or the CPU has support for direct interrupt injection. They > may be always trapped by setting this value to 0. Technically this > means traps will be cleared when the runqueue depth is 0, but that > implies nothing is running anyway so there is no reason to care. The > default value is 1 to preserve previous behavior before adding this > option. I recently discovered that this was enabled by default, but it's not obvious to me everyone will want this enabled, so I'm in favour of figuring out a way to turn it off (in fact we might want to make this feature opt in as the status quo used to be to always trap). There are a few potential issues I see with having this enabled: - a lone vcpu thread on a CPU will completely screw up the host scheduler's load tracking metrics if the vCPU actually spends a significant amount of time in WFI (the PELT signal will no longer be a good proxy for "how much CPU time does this task need"); - the scheduler's decision will impact massively the behaviour of the vcpu task itself. Co-scheduling a task with a vcpu task (or not) will impact massively the perceived behaviour of the vcpu task in a way that is entirely unpredictable to the scheduler; - while the above problems might be OK for some users, I don't think this will always be true, e.g. when running on big.LITTLE systems the above sounds nightmare-ish; - the guest spending long periods of time in WFI prevents the host from being able to enter deeper idle states, which will impact power very negatively; And probably a whole bunch of other things. > Think about his option as a threshold. The instruction will be trapped > if the runqueue depth is higher than the threshold. So talking about the exact interface, I'm not sure exposing this to userspace is really appropriate. The current rq depth is next to impossible for userspace to control well. My gut feeling tells me we might want to gate all of this on PREEMPT_FULL instead, since PREEMPT_FULL is pretty much a way to say "I'm willing to give up scheduler tracking accuracy to gain throughput when I've got a task running alone on a CPU". Thoughts? Thanks, Quentin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel