Re: [PATCH] selftests/rseq: take large C-state exit latency into consideration

[Sean Christopherson <seanjc@google.com>]

On Fri, Mar 22, 2024, Zide Chen wrote:
> Currently, the migration worker delays 1-10 us, assuming that one
> KVM_RUN iteration only takes a few microseconds.  But if C-state exit
> latencies are large enough, for example, hundreds or even thousands
> of microseconds on server CPUs, it may happen that it's not able to
> bring the target CPU out of C-state before the migration worker starts
> to migrate it to the next CPU.
> 
> If the system workload is light, most CPUs could be at a certain level
> of C-state, and the vCPU thread may waste milliseconds before it can
> actually migrate to a new CPU.

Well fudge.  That's definitely not on my bingo sheet.

> Thus, the tests may be inefficient in such systems, and in some cases
> it may fail the migration/KVM_RUN ratio sanity check.
> 
> Since we are not able to turn off the cpuidle sub-system in run time,
> this patch creates an idle thread on every CPU to prevent them from
> entering C-states.

First off, huge thanks for debugging this!  That must have been quite the task
(no pun intended).

While spinning up threads on every CPU is a clever way to ensure they don't go
into a deep sleep state, I'm not exactly excited about the idea of putting every
reachable CPU into a busy loop.  And while this doesn't add _that_ much complexity,
I'm not sure the benefit (preserving the assert for all systems) is worth it.  I
also don't want to arbitrarily prevent idle task (as in, the kernel's idle task)
interactions.  E.g. it's highly (highly) unlikely, but not impossible for there
to be a bug that's unique to idle tasks, or C-states, or other edge case.

Are there any metrics/stats that can be (easily) checked to grant an exception
to the sanity check?  That's a very hand-wavy question, as I'm not even sure what
type of stat we'd want to look at.  Actual runtime of a task, maybe?

If that's not easy, what if we add an off-by-default command line option to skip
the sanity check?  I was resistant to simply deleting the assert in the past, but
that was mainly because I didn't want to delete it without understanding what was
causing problems.  That would allow CI environments to opt-out as needed, while
still keeping the sanity check alive for enough systems to make it useful.