Re: [Qemu-devel] [PATCH v7 03/12] target/arm: Swap PMU values before/after migrations

[Aaron Lindsay <aaron@os.amperecomputing.com>]

On Nov 16 16:44, Peter Maydell wrote:
> On 16 November 2018 at 16:09, Aaron Lindsay
> <aaron@os.amperecomputing.com> wrote:
> > On Nov 16 14:53, Peter Maydell wrote:
> >> On 5 November 2018 at 18:51, Aaron Lindsay <aaron@os.amperecomputing.com> wrote:
> >> > Because of the PMU's design, many register accesses have side effects
> >> > which are inter-related, meaning that the normal method of saving CP
> >> > registers can result in inconsistent state. These side-effects are
> >> > largely handled in pmu_op_start/finish functions which can be called
> >> > before and after the state is saved/restored. By doing this and adding
> >> > raw read/write functions for the affected registers, we avoid
> >> > migration-related inconsistencies.
> >> >
> >> > Signed-off-by: Aaron Lindsay <aclindsa@gmail.com>
> >>
> >> > --- a/target/arm/machine.c
> >> > +++ b/target/arm/machine.c
> >> > @@ -604,6 +604,8 @@ static int cpu_pre_save(void *opaque)
> >> >  {
> >> >      ARMCPU *cpu = opaque;
> >> >
> >> > +    pmu_op_start(&cpu->env);
> >> > +
> >> >      if (kvm_enabled()) {
> >> >          if (!write_kvmstate_to_list(cpu)) {
> >> >              /* This should never fail */
> >> > @@ -625,6 +627,20 @@ static int cpu_pre_save(void *opaque)
> >> >      return 0;
> >> >  }
> >> >
> >> > +static int cpu_post_save(void *opaque)
> >> > +{
> >> > +    ARMCPU *cpu = opaque;
> >> > +    pmu_op_finish(&cpu->env);
> >> > +    return 0;
> >> > +}
> >> > +
> >> > +static int cpu_pre_load(void *opaque)
> >> > +{
> >> > +    ARMCPU *cpu = opaque;
> >> > +    pmu_op_start(&cpu->env);
> >> > +    return 0;
> >> > +}
> >> > +
> >> >  static int cpu_post_load(void *opaque, int version_id)
> >> >  {
> >> >      ARMCPU *cpu = opaque;
> >> > @@ -672,6 +688,8 @@ static int cpu_post_load(void *opaque, int version_id)
> >> >      hw_breakpoint_update_all(cpu);
> >> >      hw_watchpoint_update_all(cpu);
> >> >
> >> > +    pmu_op_finish(&cpu->env);
> >> > +
> >> >      return 0;
> >> >  }
> >>
> >> This will end up calling pmu_op_start() and pmu_op_finish()
> >> even if the guest is running KVM and we're not using the
> >> TCG code. Is that what you intended?
> >
> > The counters are still stored in their 'difference' format for KVM, so I
> > think this still makes sense. Or is there something I'm missing about
> > how this will interact with KVM? I'm much more familiar with TCG.
> 
> For KVM the counter values are stored in the kernel, until the
> write_kvmstate_to_list() function (which is performed after your
> pmu_op_start() call in cpu_pre_save()) writes them from the
> kernel into the cpreg_vmstate array. Similarly on load they
> go straight from the vmstate array into the kernel registers.
> It's not clear to me what the pmu_op_start()/finish() calls
> are intended to do in the KVM case, and they look at fields
> in the env->cp15 struct which will not have valid values at
> this point.

Oh, I had a fundamental misunderstanding and expected the registers
would be in env->cp15 when this was called, even for KVM.

After looking into this more, it seems like my pmu_op_* calls here
should be contained in `if (!kvm_enabled())` blocks. It doesn't seem
like anything bad would happen otherwise, but it also isn't necessary or
helpful to make those calls.

I'll make this change for v8 if you don't have further feedback.

-Aaron