Re: [PATCH] ppc/spapr: Add support for implement support for H_SCM_HEALTH

[Greg Kurz <groug@kaod.org>]

On Tue, 30 Mar 2021 22:37:06 +0530
Vaibhav Jain <vaibhav@linux.ibm.com> wrote:

> 
> Thanks for looking into this patch Greg. My responses below inline.
> 
> 
> Greg Kurz <groug@kaod.org> writes:
> 
> > Hi Vaibhav,
> >
> > Great to see you around :-)
> 
> :-)
> 
> >
> > On Mon, 29 Mar 2021 21:52:59 +0530
> > Vaibhav Jain <vaibhav@linux.ibm.com> wrote:
> >
> >> Add support for H_SCM_HEALTH hcall described at [1] for spapr
> >> nvdimms. This enables guest to detect the 'unarmed' status of a
> >> specific spapr nvdimm identified by its DRC and if its unarmed, mark
> >> the region backed by the nvdimm as read-only.
> >> 
> >
> > Any chance that you can provide the documentation of this new hcall ?
> >
> H_SCM_HEALTH specifications is already documented in linux kernel
> documentation at
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/powerpc/papr_hcalls.rst
> 
> That documentation was added when kernel support for H_SCM_HEALTH hcall
> support was implemented in 5.9 kernel. 
> 

Oops I skipped that indeed. Maybe even make it:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/powerpc/papr_hcalls.rst#n220

for faster access.

> >> The patch adds h_scm_health() to handle the H_SCM_HEALTH hcall which
> >> returns two 64-bit bitmaps (health bitmap, health bitmap mask) derived
> >> from 'struct nvdimm->unarmed' member.
> >> 
> >> Linux kernel side changes to enable handling of 'unarmed' nvdimms for
> >> ppc64 are proposed at [2].
> >> 
> >> References:
> >> [1] "Hypercall Op-codes (hcalls)"
> >>     https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/powerpc/papr_hcalls.rst
> >> 
> >> [2] "powerpc/papr_scm: Mark nvdimm as unarmed if needed during probe"
> >>     https://lore.kernel.org/linux-nvdimm/20210329113103.476760-1-vaibhav@linux.ibm.com/
> >> 
> >> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> >> ---
> >>  hw/ppc/spapr_nvdimm.c  | 30 ++++++++++++++++++++++++++++++
> >>  include/hw/ppc/spapr.h |  4 ++--
> >>  2 files changed, 32 insertions(+), 2 deletions(-)
> >> 
> >> diff --git a/hw/ppc/spapr_nvdimm.c b/hw/ppc/spapr_nvdimm.c
> >> index b46c36917c..e38740036d 100644
> >> --- a/hw/ppc/spapr_nvdimm.c
> >> +++ b/hw/ppc/spapr_nvdimm.c
> >> @@ -31,6 +31,13 @@
> >>  #include "qemu/range.h"
> >>  #include "hw/ppc/spapr_numa.h"
> >>  
> >> +/* DIMM health bitmap bitmap indicators */
> >> +/* SCM device is unable to persist memory contents */
> >> +#define PAPR_PMEM_UNARMED (1ULL << (63 - 0))
> >
> > This looks like PPC_BIT(0).
> >
> Yes, right. Will update the patch in v2 to use the PPC_BIT macro.
> 

Well, since this was copied from the kernel sources, I guess you can
leave it as is. Maybe just explain where this macro comes from if you
do so.

> >> +
> >> +/* Bits status indicators for health bitmap indicating unarmed dimm */
> >> +#define PAPR_PMEM_UNARMED_MASK (PAPR_PMEM_UNARMED)
> >> +
> >>  bool spapr_nvdimm_validate(HotplugHandler *hotplug_dev, NVDIMMDevice *nvdimm,
> >>                             uint64_t size, Error **errp)
> >>  {
> >> @@ -467,6 +474,28 @@ static target_ulong h_scm_unbind_all(PowerPCCPU *cpu, SpaprMachineState *spapr,
> >>      return H_SUCCESS;
> >>  }
> >>  
> >> +static target_ulong h_scm_health(PowerPCCPU *cpu, SpaprMachineState *spapr,
> >> +                                 target_ulong opcode, target_ulong *args)
> >> +{
> >> +    uint32_t drc_index = args[0];
> >> +    SpaprDrc *drc = spapr_drc_by_index(drc_index);
> >> +    NVDIMMDevice *nvdimm;
> >> +
> >> +    if (drc && spapr_drc_type(drc) != SPAPR_DR_CONNECTOR_TYPE_PMEM) {
> >> +        return H_PARAMETER;
> >> +    }
> >> +
> >> +    nvdimm = NVDIMM(drc->dev);
> >
> > Yeah as already suggested by Shiva, drc->dev should be checked like
> > in h_scm_bind_mem().
> >
> Yes, will send a v2 with this case handled.
> 
> >> +
> >> +    /* Check if the nvdimm is unarmed and send its status via health bitmaps */
> >> +    args[0] = nvdimm->unarmed ? PAPR_PMEM_UNARMED_MASK : 0;
> >> +
> >
> > Shouldn't ^^ use PAPR_PMEM_UNARMED then ?
> >
> >> +    /* health bitmap mask same as the health bitmap */
> >> +    args[1] = args[0];
> >> +
> >
> > If so, it seems that PAPR_PMEM_UNARMED_MASK isn't even needed.
> 
> Definition of these defines are similar to what kernel implementation
> uses at
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/platforms/pseries/papr_scm.c#n53
> 
> Since unarmed condition can also arise due to an unhealthy nvdimm hence
> the kernel implementation uses a mask thats composed of two bits
> PPC_BIT(0) and PPC_BIT(6) being set. Though we arent using PPC_BIT(6)
> right now in qemu, it will change in future when better nvdimm health
> reporting will be done. Hence kept the PPC_BIT(0) define as well as the
> mask to mimic the kernel definitions.
> 

After a deeper look into the kernel documentation and code, it seems that
we really don't need PAPR_PMEM_UNARMED_MASK at all on the QEMU side : it
is _just_ the mask of bits that linux does support (i.e. bit 0 and 6) and
applies to the health bitmap returned by H_SCM_HEALTH.

On the QEMU side, as you say, we just support setting bit 0 from the
time being : args[0] should thus be set to PAPR_PMEM_UNARMED if the
NVDIMM is unarmed, not PAPR_PMEM_UNARMED_MASK.

Also, it seems that the purpose of the health bitmap mask is to tell the
guest about the bits supported by the hypervisor : args[1] should thus
unconditionally be set to PAPR_PMEM_UNARMED, otherwise we'd tell the
guest none of the bits are valid when the NVDIMM isn't unarmed.

> >
> > Having access to the excerpts from the PAPR addendum that describes
> > this hcall would _really_ help in reviewing.
> >
> The kernel documentation for H_SCM_HEALTH mentioned above captures most
> if not all parts of the PAPR addendum for this hcall. I believe it
> contains enough information to review the patch. If you still need more
> info than please let me know.
> 

Sorry again for having missed the link the first time. I guess I have all
the information I need now.

> 
> >> +    return H_SUCCESS;
> >> +}
> >> +
> >>  static void spapr_scm_register_types(void)
> >>  {
> >>      /* qemu/scm specific hcalls */
> >> @@ -475,6 +504,7 @@ static void spapr_scm_register_types(void)
> >>      spapr_register_hypercall(H_SCM_BIND_MEM, h_scm_bind_mem);
> >>      spapr_register_hypercall(H_SCM_UNBIND_MEM, h_scm_unbind_mem);
> >>      spapr_register_hypercall(H_SCM_UNBIND_ALL, h_scm_unbind_all);
> >> +    spapr_register_hypercall(H_SCM_HEALTH, h_scm_health);
> >>  }
> >>  
> >>  type_init(spapr_scm_register_types)
> >> diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
> >> index 47cebaf3ac..18859b9ab2 100644
> >> --- a/include/hw/ppc/spapr.h
> >> +++ b/include/hw/ppc/spapr.h
> >> @@ -538,8 +538,8 @@ struct SpaprMachineState {
> >>  #define H_SCM_BIND_MEM          0x3EC
> >>  #define H_SCM_UNBIND_MEM        0x3F0
> >>  #define H_SCM_UNBIND_ALL        0x3FC
> >> -
> >> -#define MAX_HCALL_OPCODE        H_SCM_UNBIND_ALL
> >> +#define H_SCM_HEALTH            0x400
> >> +#define MAX_HCALL_OPCODE        H_SCM_HEALTH
> >>  
> >>  /* The hcalls above are standardized in PAPR and implemented by pHyp
> >>   * as well.
> >
>