Re: [PATCH kvm-unit-tests v1 4/6] s390x: smp: Create and use a non-waiting CPU stop

[Eric Farman <farman@linux.ibm.com>]

On Mon, 2022-03-07 at 16:30 +0100, Claudio Imbrenda wrote:
> On Thu,  3 Mar 2022 22:04:23 +0100
> Eric Farman <farman@linux.ibm.com> wrote:
> 
> > When stopping a CPU, kvm-unit-tests serializes/waits for everything
> > to finish, in order to get a consistent result whenever those
> > functions are used.
> > 
> > But to test the SIGP STOP itself, these additional measures could
> > mask other problems. For example, did the STOP work, or is the CPU
> > still operating?
> > 
> > Let's create a non-waiting SIGP STOP and use it here, to ensure
> > that
> > the CPU is correctly stopped. A smp_cpu_stopped() call will still
> > be used to see that the SIGP STOP has been processed, and the state
> > of the CPU can be used to determine whether the test passes/fails.
> > 
> > Signed-off-by: Eric Farman <farman@linux.ibm.com>
> > ---
> >  lib/s390x/smp.c | 25 +++++++++++++++++++++++++
> >  lib/s390x/smp.h |  1 +
> >  s390x/smp.c     | 10 ++--------
> >  3 files changed, 28 insertions(+), 8 deletions(-)
> > 
> > diff --git a/lib/s390x/smp.c b/lib/s390x/smp.c
> > index 368d6add..84e536e8 100644
> > --- a/lib/s390x/smp.c
> > +++ b/lib/s390x/smp.c
> > @@ -119,6 +119,31 @@ int smp_cpu_stop(uint16_t idx)
> >  	return rc;
> >  }
> >  
> > +/*
> > + * Functionally equivalent to smp_cpu_stop(), but without the
> > + * elements that wait/serialize matters itself.
> > + * Used to see if KVM itself is serialized correctly.
> > + */
> > +int smp_cpu_stop_nowait(uint16_t idx)
> > +{
> > +	/* refuse to work on the boot CPU */
> > +	if (idx == 0)
> > +		return -1;
> > +
> > +	spin_lock(&lock);
> > +
> > +	/* Don't suppress a CC2 with sigp_retry() */
> > +	if (smp_sigp(idx, SIGP_STOP, 0, NULL)) {
> > +		spin_unlock(&lock);
> > +		return -1;
> > +	}
> > +
> > +	cpus[idx].active = false;
> > +	spin_unlock(&lock);
> > +
> > +	return 0;
> > +}
> > +
> >  int smp_cpu_stop_store_status(uint16_t idx)
> >  {
> >  	int rc;
> > diff --git a/lib/s390x/smp.h b/lib/s390x/smp.h
> > index 1e69a7de..bae03dfd 100644
> > --- a/lib/s390x/smp.h
> > +++ b/lib/s390x/smp.h
> > @@ -44,6 +44,7 @@ bool smp_sense_running_status(uint16_t idx);
> >  int smp_cpu_restart(uint16_t idx);
> >  int smp_cpu_start(uint16_t idx, struct psw psw);
> >  int smp_cpu_stop(uint16_t idx);
> > +int smp_cpu_stop_nowait(uint16_t idx);
> >  int smp_cpu_stop_store_status(uint16_t idx);
> >  int smp_cpu_destroy(uint16_t idx);
> >  int smp_cpu_setup(uint16_t idx, struct psw psw);
> > diff --git a/s390x/smp.c b/s390x/smp.c
> > index 50811bd0..11c2c673 100644
> > --- a/s390x/smp.c
> > +++ b/s390x/smp.c
> > @@ -76,14 +76,8 @@ static void test_restart(void)
> >  
> >  static void test_stop(void)
> >  {
> > -	smp_cpu_stop(1);
> > -	/*
> > -	 * The smp library waits for the CPU to shut down, but let's
> > -	 * also do it here, so we don't rely on the library
> > -	 * implementation
> > -	 */
> > -	while (!smp_cpu_stopped(1)) {}
> > -	report_pass("stop");
> > +	smp_cpu_stop_nowait(1);
> 
> can it happen that the SIGP STOP order is accepted, but the target
> CPU
> is still running (and not even busy)?

Of course. A SIGP that's processed by userspace (which is many of them)
injects a STOP IRQ back to the kernel, which means the CPU might not be
stopped for some time. But...

> 
> > +	report(smp_cpu_stopped(1), "stop");
> 
> e.g. can this ^ check race with the actual stopping of the CPU?

...the smp_cpu_stopped() routine now loops on the CC2 that SIGP SENSE
returns because of that pending IRQ. If SIGP SENSE returns CC0/1, then
the CPU can correctly be identified stopped/operating, and the test can
correctly pass/fail.

> 
> >  }
> >  
> >  static void test_stop_store_status(void)