Re: [Qemu-devel] [PATCH v18 1/2] sPAPR: Implement EEH RTAS calls

[Gavin Shan <gwshan@linux.vnet.ibm.com>]

On Wed, Feb 18, 2015 at 07:28:29PM -0600, Michael Roth wrote:
>Quoting Gavin Shan (2015-02-15 23:32:09)
>> On Mon, Feb 16, 2015 at 12:52:48PM +1100, David Gibson wrote:
>> >On Mon, Feb 16, 2015 at 10:16:01AM +1100, Gavin Shan wrote:
>> >> The emulation for EEH RTAS requests from guest isn't covered
>> >> by QEMU yet and the patch implements them.
>> >> 
>> >> The patch defines constants used by EEH RTAS calls and adds
>> >> callbacks sPAPRPHBClass::{eeh_set_option, eeh_get_state, eeh_reset,
>> >> eeh_configure}, which are going to be used as follows:
>> >> 
>> >>   * RTAS calls are received in spapr_pci.c, sanity check is done
>> >>     there.
>> >>   * RTAS handlers handle what they can. If there is something it
>> >>     cannot handle and the corresponding sPAPRPHBClass callback is
>> >>     defined, it is called.
>> >>   * Those callbacks are only implemented for VFIO now. They do ioctl()
>> >>     to the IOMMU container fd to complete the calls. Error codes from
>> >>     that ioctl() are transferred back to the guest.
>
>
>
>> >> 
>> >> [aik: defined RTAS tokens for EEH RTAS calls]
>> >> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> >> ---
>> >>  hw/ppc/spapr_pci.c          | 281 ++++++++++++++++++++++++++++++++++++++++++++
>> >>  include/hw/pci-host/spapr.h |   4 +
>> >>  include/hw/ppc/spapr.h      |  43 ++++++-
>> >>  3 files changed, 326 insertions(+), 2 deletions(-)
>> >> 
>> >> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
>> >> index cebdeb3..29b071d 100644
>> >> --- a/hw/ppc/spapr_pci.c
>> >> +++ b/hw/ppc/spapr_pci.c
>> >> @@ -406,6 +406,268 @@ static void rtas_ibm_query_interrupt_source_number(PowerPCCPU *cpu,
>> >>      rtas_st(rets, 2, 1);/* 0 == level; 1 == edge */
>> >>  }
>> >>  
>> >> +static void rtas_ibm_set_eeh_option(PowerPCCPU *cpu,
>> >> +                                    sPAPREnvironment *spapr,
>> >> +                                    uint32_t token, uint32_t nargs,
>> >> +                                    target_ulong args, uint32_t nret,
>> >> +                                    target_ulong rets)
>> >> +{
>> >> +    sPAPRPHBState *sphb;
>> >> +    sPAPRPHBClass *spc;
>> >> +    uint32_t addr, option;
>> >> +    uint64_t buid;
>> >> +    int ret;
>> >> +
>> >> +    if ((nargs != 4) || (nret != 1)) {
>> >> +        goto param_error_exit;
>> >> +    }
>> >> +
>> >> +    buid = ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2);
>> >> +    addr = rtas_ld(args, 0);
>> >> +    option = rtas_ld(args, 3);
>> >> +
>> >> +    sphb = find_phb(spapr, buid);
>> >> +    if (!sphb) {
>> >> +        goto param_error_exit;
>> >> +    }
>> >> +
>> >> +    spc = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
>> >> +    if (!spc->eeh_set_option) {
>> >> +        goto param_error_exit;
>> >> +    }
>> >> +
>> >> +    /*
>> >> +     * The EEH functionality is enabled on basis of PCI device,
>> >> +     * instead of PE. We need check the validity of the PCI
>> >> +     * device address.
>> >> +     */
>> >> +    if (option == RTAS_EEH_ENABLE &&
>> >> +        !find_dev(spapr, buid, addr)) {
>> >> +        goto param_error_exit;
>> >> +    }
>> >
>> >You're still breaking your layering by doing checks dependent on the
>> >specific option both here and in the callback.
>> >
>> >What I meant by my comments on the previous version was that this
>> >find_dev() test should also move into the eeh_set_option callback.
>> >Obviously that means adding addr into the parameters - but surely if
>> >the addr has any meaning whatsoever, it must be at least potentially
>> >needed by the callback anyway.
>> >
>> 
>> Ok. Either simply dropping the check here, or moving find_dev() to
>> sPAPRPHBClass::eeh_set_option() as you suggested. However, there're more
>> things needed for sPAPRPHBClass::eeh_set_option() to do the check as follows.
>> David, could you help to confirm which way you prefer?
>> 
>> - Rename find_dev() to spapr_find_pci_dev() and make it public. It will be
>>   called in spapr_pci_vfio.c
>> - Add one field sPAPRPHBState::spapr to reference the associated sPAPREnvironment,
>>   which is required by spapr_find_pci_dev(). Otherwise, we have to pass sPAPREnvironment
>>   to sPAPRPHBClass::eeh_set_option().
>
>AFAICT spapr_pci.c:find_dev() only needs sPAPREnvironment to look up the phb
>given a buid, but in your case you already have the phb and pass it on to
>eeh_set_option(), so within eeh_set_option() you can call pci_find_device()
>just like spapr_pci.c:find_dev() does to do the validation.
>

Yeah, it's another option I was thinking about. It would introduce
duplicate code, but it seems it's the best way to go. I'll update
accordingly in next revision. Thanks for your comments.
 
>The validation seems to assume the addr value is a config_addr for the device
>though, isn't it possible we might recieve a pe_addr of the form returned
>by rtas_ibm_get_config_addr_info2? That value would happen to correspond to
>bus:n,device:0,func:0,reg:1, and find_dev in that case would just mask off
>the reg value and verify there's a device in PCI slot 0, instead of whatever
>actually needs to be validated in that situation (which isn't clear to me).
>

Yeah, The address passed to rtas_ibm_set_eeh_option() could be device's
config_addr or PE_addr depending on the options. For option EEH_ENABLE,
it's device's config_addr.

Thanks,
Gavin

>> >Apart from that,
>> >
>> >Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
>> >
>> >-- 
>> >David Gibson                   | I'll have my music baroque, and my code
>> >david AT gibson.dropbear.id.au | minimalist, thank you.  NOT _the_ _other_
>> >                               | _way_ _around_!
>> >http://www.ozlabs.org/~dgibson