Re: [PATCH v2] KVM: arm64: Add KVM_CAP to control WFx trapping

From: Quentin Perret <qperret@google.com>
To: Colton Lewis <coltonlewis@google.com>
Cc: kvm@vger.kernel.org, Marc Zyngier <maz@kernel.org>,
	Oliver Upton <oliver.upton@linux.dev>,
	James Morse <james.morse@arm.com>,
	Suzuki K Poulose <suzuki.poulose@arm.com>,
	Zenghui Yu <yuzenghui@huawei.com>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Will Deacon <will@kernel.org>,
	Paolo Bonzini <pbonzini@redhat.com>,
	Ingo Molnar <mingo@redhat.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Juri Lelli <juri.lelli@redhat.com>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
	Daniel Bristot de Oliveira <bristot@redhat.com>,
	Valentin Schneider <vschneid@redhat.com>,
	linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] KVM: arm64: Add KVM_CAP to control WFx trapping
Date: Fri, 22 Mar 2024 14:34:35 +0000	[thread overview]
Message-ID: <Zf2W-8duBlCk5LVm@google.com> (raw)
In-Reply-To: <Zf2Uo1l0JgBEKRAL@google.com>

On Friday 22 Mar 2024 at 14:24:35 (+0000), Quentin Perret wrote:
> On Tuesday 19 Mar 2024 at 16:43:41 (+0000), Colton Lewis wrote:
> > Add a KVM_CAP to control WFx (WFI or WFE) trapping based on scheduler
> > runqueue depth. This is so they can be passed through if the runqueue
> > is shallow or the CPU has support for direct interrupt injection. They
> > may be always trapped by setting this value to 0. Technically this
> > means traps will be cleared when the runqueue depth is 0, but that
> > implies nothing is running anyway so there is no reason to care. The
> > default value is 1 to preserve previous behavior before adding this
> > option.
> 
> I recently discovered that this was enabled by default, but it's not
> obvious to me everyone will want this enabled, so I'm in favour of
> figuring out a way to turn it off (in fact we might want to make this
> feature opt in as the status quo used to be to always trap).
> 
> There are a few potential issues I see with having this enabled:
> 
>  - a lone vcpu thread on a CPU will completely screw up the host
>    scheduler's load tracking metrics if the vCPU actually spends a
>    significant amount of time in WFI (the PELT signal will no longer
>    be a good proxy for "how much CPU time does this task need");
> 
>  - the scheduler's decision will impact massively the behaviour of the
>    vcpu task itself. Co-scheduling a task with a vcpu task (or not) will
>    impact massively the perceived behaviour of the vcpu task in a way
>    that is entirely unpredictable to the scheduler;
> 
>  - while the above problems might be OK for some users, I don't think
>    this will always be true, e.g. when running on big.LITTLE systems the
>    above sounds nightmare-ish;
> 
>  - the guest spending long periods of time in WFI prevents the host from
>    being able to enter deeper idle states, which will impact power very
>    negatively;
> 
> And probably a whole bunch of other things.
> 
> > Think about his option as a threshold. The instruction will be trapped
> > if the runqueue depth is higher than the threshold.
> 
> So talking about the exact interface, I'm not sure exposing this to
> userspace is really appropriate. The current rq depth is next to
> impossible for userspace to control well.
> 
> My gut feeling tells me we might want to gate all of this on
> PREEMPT_FULL instead, since PREEMPT_FULL is pretty much a way to say
> "I'm willing to give up scheduler tracking accuracy to gain throughput
> when I've got a task running alone on a CPU". Thoughts?

And obviously I meant s/PREEMPT_FULL/NOHZ_FULL, but hopefully that was
clear :-)