From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFA86C6FD20 for ; Fri, 24 Mar 2023 22:03:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232350AbjCXWDs (ORCPT ); Fri, 24 Mar 2023 18:03:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232025AbjCXWDq (ORCPT ); Fri, 24 Mar 2023 18:03:46 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 515A815899 for ; Fri, 24 Mar 2023 15:02:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679695377; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OkdKpUhRJP9VEDGxy2ljm69baU8C0urR7xxEuXUu0S4=; b=i8v/Gu8GMwPi1cIKXu27Hq5KV90yLLmGsHNqFs5TyzzNAogc6gSzG8SnChiMy/xDZrie8x pNoKD29NX6gMry8c3f26J+1Rs4c7LNhjv9Z6yEtWvInhvMvo9lbDctOkmJwlRal8E7fss7 ikVwkHR2tTXMhtSlgmzo7RmTU0+78fM= Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-169-GEPqua-tPY-11wVjBxk0DQ-1; Fri, 24 Mar 2023 18:02:56 -0400 X-MC-Unique: GEPqua-tPY-11wVjBxk0DQ-1 Received: by mail-io1-f70.google.com with SMTP id i189-20020a6b3bc6000000b00758a1ed99c2so1960830ioa.1 for ; Fri, 24 Mar 2023 15:02:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679695374; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OkdKpUhRJP9VEDGxy2ljm69baU8C0urR7xxEuXUu0S4=; b=maUwYjQCOQOUCt94pbCRLD2YGwhA7YaAN3pEKNK6gdvgLhpAjXxldpOzpcL/Vwx8ZY 6bCCCTA63wUmEwhg02qE5t3u0F5ZLexO+sH/IP+auKr3Uakmx4e92nTyIqFujZzHMddR JXZ6dQjrB+dGdiOoicbCz6hXFKgWCCnG6eTfj+8yOjiFVBATe1amyyLinFfCZdgggSkQ sdOxgMVCIC8/SN+en+ai1eMX2n8An6jxNIfvgFBQjRWsospOgCU4z9YLrQJUOQHF4qUb ephSdo7FQoIxCkkIPYU5j4zNXb8EYNuqGIemTrMAO13OjlNaY1EdwXzPXdvxztOBg424 /xDg== X-Gm-Message-State: AAQBX9ebMnokKutwVQK4jOViYapDSm2757DVJZfQ56Tm1FjGdaeDU/E1 WDcs3ssFNGgRY2sAsaTtYI+DcTOy01qN7JyGnUuSYNCVhXgFerUL1/FA2lWgsiBxCT+ThAoVXkr LTAfVW+lFl3yE6zwsLeIG X-Received: by 2002:a92:4b0d:0:b0:325:a550:eb06 with SMTP id m13-20020a924b0d000000b00325a550eb06mr3385273ilg.12.1679695373992; Fri, 24 Mar 2023 15:02:53 -0700 (PDT) X-Google-Smtp-Source: AKy350b+rFxy4zjwUD6vkG3euFUvg4RTywX9D5pFpqRtI1jJWxwOUmi2DNeY3O6WonDJhZcnFqLVcg== X-Received: by 2002:a92:4b0d:0:b0:325:a550:eb06 with SMTP id m13-20020a924b0d000000b00325a550eb06mr3385254ilg.12.1679695373621; Fri, 24 Mar 2023 15:02:53 -0700 (PDT) Received: from redhat.com ([38.15.36.239]) by smtp.gmail.com with ESMTPSA id ce16-20020a0566381a9000b00404f3266fd7sm6938190jab.159.2023.03.24.15.02.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Mar 2023 15:02:53 -0700 (PDT) Date: Fri, 24 Mar 2023 16:02:51 -0600 From: Alex Williamson To: Brett Creeley Cc: , , , , , , , Subject: Re: [PATCH v5 vfio 3/7] vfio/pds: register with the pds_core PF Message-ID: <20230324160251.4014b4e5.alex.williamson@redhat.com> In-Reply-To: <20230322203442.56169-4-brett.creeley@amd.com> References: <20230322203442.56169-1-brett.creeley@amd.com> <20230322203442.56169-4-brett.creeley@amd.com> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.35; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Wed, 22 Mar 2023 13:34:38 -0700 Brett Creeley wrote: > The pds_core driver will supply adminq services, so find the PF > and register with the DSC services. > > Use the following commands to enable a VF: > echo 1 > /sys/bus/pci/drivers/pds_core/$PF_BDF/sriov_numvfs > > Signed-off-by: Brett Creeley > Signed-off-by: Shannon Nelson > --- > drivers/vfio/pci/pds/Makefile | 1 + > drivers/vfio/pci/pds/cmds.c | 67 +++++++++++++++++++++++++++++++++ > drivers/vfio/pci/pds/cmds.h | 12 ++++++ > drivers/vfio/pci/pds/pci_drv.c | 16 +++++++- > drivers/vfio/pci/pds/pci_drv.h | 9 +++++ > drivers/vfio/pci/pds/vfio_dev.c | 5 +++ > drivers/vfio/pci/pds/vfio_dev.h | 2 + > include/linux/pds/pds_lm.h | 12 ++++++ > 8 files changed, 123 insertions(+), 1 deletion(-) > create mode 100644 drivers/vfio/pci/pds/cmds.c > create mode 100644 drivers/vfio/pci/pds/cmds.h > create mode 100644 drivers/vfio/pci/pds/pci_drv.h > create mode 100644 include/linux/pds/pds_lm.h > > diff --git a/drivers/vfio/pci/pds/Makefile b/drivers/vfio/pci/pds/Makefile > index e1a55ae0f079..87581111fa17 100644 > --- a/drivers/vfio/pci/pds/Makefile > +++ b/drivers/vfio/pci/pds/Makefile > @@ -4,5 +4,6 @@ > obj-$(CONFIG_PDS_VFIO_PCI) += pds_vfio.o > > pds_vfio-y := \ > + cmds.o \ > pci_drv.o \ > vfio_dev.o > diff --git a/drivers/vfio/pci/pds/cmds.c b/drivers/vfio/pci/pds/cmds.c > new file mode 100644 > index 000000000000..26e383ec4544 > --- /dev/null > +++ b/drivers/vfio/pci/pds/cmds.c > @@ -0,0 +1,67 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ > + > +#include > +#include > + > +#include > +#include > +#include > +#include > + > +#include "vfio_dev.h" > +#include "cmds.h" > + > +int > +pds_vfio_register_client_cmd(struct pds_vfio_pci_device *pds_vfio) > +{ > + union pds_core_adminq_comp comp = { 0 }; > + union pds_core_adminq_cmd cmd = { 0 }; > + struct device *dev; > + int err, id; > + u16 ci; > + > + id = PCI_DEVID(pds_vfio->pdev->bus->number, > + pci_iov_virtfn_devfn(pds_vfio->pdev, pds_vfio->vf_id)); > + > + dev = &pds_vfio->pdev->dev; > + cmd.client_reg.opcode = PDS_AQ_CMD_CLIENT_REG; > + snprintf(cmd.client_reg.devname, sizeof(cmd.client_reg.devname), > + "%s.%d", PDS_LM_DEV_NAME, id); Does this devname need to be unique, and if so should it factor in pci_domain_nr()? The array seems to be wide enough to easily hold the VF dev_name() but I haven't followed if there are additional constraints. Thanks, Alex > + > + err = pdsc_adminq_post(pds_vfio->pdsc, &cmd, &comp, false); > + if (err) { > + dev_info(dev, "register with DSC failed, status %d: %pe\n", > + comp.status, ERR_PTR(err)); > + return err; > + } > + > + ci = le16_to_cpu(comp.client_reg.client_id); > + if (!ci) { > + dev_err(dev, "%s: device returned null client_id\n", __func__); > + return -EIO; > + } > + pds_vfio->client_id = ci; > + > + return 0; > +} > + > +void > +pds_vfio_unregister_client_cmd(struct pds_vfio_pci_device *pds_vfio) > +{ > + union pds_core_adminq_comp comp = { 0 }; > + union pds_core_adminq_cmd cmd = { 0 }; > + struct device *dev; > + int err; > + > + dev = &pds_vfio->pdev->dev; > + cmd.client_unreg.opcode = PDS_AQ_CMD_CLIENT_UNREG; > + cmd.client_unreg.client_id = cpu_to_le16(pds_vfio->client_id); > + > + err = pdsc_adminq_post(pds_vfio->pdsc, &cmd, &comp, false); > + if (err) > + dev_info(dev, "unregister from DSC failed, status %d: %pe\n", > + comp.status, ERR_PTR(err)); > + > + pds_vfio->client_id = 0; > +} > diff --git a/drivers/vfio/pci/pds/cmds.h b/drivers/vfio/pci/pds/cmds.h > new file mode 100644 > index 000000000000..baf0695b5576 > --- /dev/null > +++ b/drivers/vfio/pci/pds/cmds.h > @@ -0,0 +1,12 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ > + > +#ifndef _CMDS_H_ > +#define _CMDS_H_ > + > +struct pds_vfio_pci_device; > + > +int pds_vfio_register_client_cmd(struct pds_vfio_pci_device *pds_vfio); > +void pds_vfio_unregister_client_cmd(struct pds_vfio_pci_device *pds_vfio); > + > +#endif /* _CMDS_H_ */ > diff --git a/drivers/vfio/pci/pds/pci_drv.c b/drivers/vfio/pci/pds/pci_drv.c > index 5e554420792e..46537afdee2d 100644 > --- a/drivers/vfio/pci/pds/pci_drv.c > +++ b/drivers/vfio/pci/pds/pci_drv.c > @@ -8,9 +8,13 @@ > #include > #include > > +#include > #include > +#include > > #include "vfio_dev.h" > +#include "pci_drv.h" > +#include "cmds.h" > > #define PDS_VFIO_DRV_NAME "pds_vfio" > #define PDS_VFIO_DRV_DESCRIPTION "AMD/Pensando VFIO Device Driver" > @@ -30,13 +34,23 @@ pds_vfio_pci_probe(struct pci_dev *pdev, > > dev_set_drvdata(&pdev->dev, &pds_vfio->vfio_coredev); > pds_vfio->pdev = pdev; > + pds_vfio->pdsc = pdsc_get_pf_struct(pdev); > + > + err = pds_vfio_register_client_cmd(pds_vfio); > + if (err) { > + dev_err(&pdev->dev, "failed to register as client: %pe\n", > + ERR_PTR(err)); > + goto out_put_vdev; > + } > > err = vfio_pci_core_register_device(&pds_vfio->vfio_coredev); > if (err) > - goto out_put_vdev; > + goto out_unreg_client; > > return 0; > > +out_unreg_client: > + pds_vfio_unregister_client_cmd(pds_vfio); > out_put_vdev: > vfio_put_device(&pds_vfio->vfio_coredev.vdev); > return err; > diff --git a/drivers/vfio/pci/pds/pci_drv.h b/drivers/vfio/pci/pds/pci_drv.h > new file mode 100644 > index 000000000000..e79bed12ed14 > --- /dev/null > +++ b/drivers/vfio/pci/pds/pci_drv.h > @@ -0,0 +1,9 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ > + > +#ifndef _PCI_DRV_H > +#define _PCI_DRV_H > + > +#include > + > +#endif /* _PCI_DRV_H */ > diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c > index f1221f14e4f6..592b10a279f0 100644 > --- a/drivers/vfio/pci/pds/vfio_dev.c > +++ b/drivers/vfio/pci/pds/vfio_dev.c > @@ -31,6 +31,11 @@ pds_vfio_init_device(struct vfio_device *vdev) > pds_vfio->vf_id = pci_iov_vf_id(pdev); > pds_vfio->pci_id = PCI_DEVID(pdev->bus->number, pdev->devfn); > > + dev_dbg(&pdev->dev, "%s: PF %#04x VF %#04x (%d) vf_id %d domain %d pds_vfio %p\n", > + __func__, pci_dev_id(pdev->physfn), > + pds_vfio->pci_id, pds_vfio->pci_id, pds_vfio->vf_id, > + pci_domain_nr(pdev->bus), pds_vfio); > + > return 0; > } > > diff --git a/drivers/vfio/pci/pds/vfio_dev.h b/drivers/vfio/pci/pds/vfio_dev.h > index a66f8069b88c..0c7932c6e1e8 100644 > --- a/drivers/vfio/pci/pds/vfio_dev.h > +++ b/drivers/vfio/pci/pds/vfio_dev.h > @@ -10,9 +10,11 @@ > struct pds_vfio_pci_device { > struct vfio_pci_core_device vfio_coredev; > struct pci_dev *pdev; > + struct pdsc *pdsc; > > int vf_id; > int pci_id; > + u16 client_id; > }; > > const struct vfio_device_ops *pds_vfio_ops_info(void); > diff --git a/include/linux/pds/pds_lm.h b/include/linux/pds/pds_lm.h > new file mode 100644 > index 000000000000..2bc2bf79426e > --- /dev/null > +++ b/include/linux/pds/pds_lm.h > @@ -0,0 +1,12 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* Copyright(c) 2022 Pensando Systems, Inc */ > + > +#ifndef _PDS_LM_H_ > +#define _PDS_LM_H_ > + > +#include "pds_common.h" > + > +#define PDS_DEV_TYPE_LM_STR "LM" > +#define PDS_LM_DEV_NAME PDS_CORE_DRV_NAME "." PDS_DEV_TYPE_LM_STR > + > +#endif /* _PDS_LM_H_ */