From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44376C433F5 for ; Wed, 9 Feb 2022 00:08:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239135AbiBIAIH (ORCPT ); Tue, 8 Feb 2022 19:08:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239065AbiBIAID (ORCPT ); Tue, 8 Feb 2022 19:08:03 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 1E282C061576 for ; Tue, 8 Feb 2022 16:08:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1644365282; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lc9Lm5985twS+sRxlWTsNS/BopDzbEB8uOS6Pi7bwTg=; b=HIBb5foFPCbNm0o/D8WxFH1rJByjjVGnjYQ+r1A548stUTXaxT8ewrMbLAUDvncc+mdT3G Y2F+tIsGXHVX7KJQgEqbPd3f74DLGt5g5gLlu+KwJYCKrTHlxL/66I6LCqIPM8a47Emrx0 3Is13CW7BnINFeqaCWLkiEczI84bKic= Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-537-wWdok1PtP-elI8j5jvdWGw-1; Tue, 08 Feb 2022 19:08:01 -0500 X-MC-Unique: wWdok1PtP-elI8j5jvdWGw-1 Received: by mail-io1-f72.google.com with SMTP id b4-20020a05660214c400b00632eb8bff25so641290iow.1 for ; Tue, 08 Feb 2022 16:08:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:organization:mime-version:content-transfer-encoding; bh=Lc9Lm5985twS+sRxlWTsNS/BopDzbEB8uOS6Pi7bwTg=; b=WnBUlYS8Fv497yDxeRFe8hIp2QoYguRaCKWrM28Yahm8YbhPoRm1cTmrEGdl9UuR7O +NPCFUAhIQwJW21lzlSeNe+uW/gee2Q3GcWFit0MtF/gjDAx7YCJWL/BzOSZr2jejs7K f3Fr0SGC/a3dthKIH4J7DxWTEBq4xeF3uOjWc6eHZD3+dcw8Y+EreGWEKxIJGkY/e1wy VvAMZTrSL8bReGg5Sh3HsBYlEqFjOcqMWBCWBLXHY1oGmGG95dcAbDB5kJT9A+EZzmkT F3hOUZdi+bKAgvzQ01eW76sQsTMykQAWc3DOkjehTU43ZRmznMSrxYNcrjYuzcOjz4xe lAiA== X-Gm-Message-State: AOAM5319783N9njJNYGXxr7x01LXeiBL+dsklfNtUuti5cbRslva5/1n praeyQZB59W3AuXE38t4CZNz9lFooQ532MFuEyI3en7invkRNZKGM1xPvJpyNkYj+fwx2Fw6e0w OIBQzXcWOvAYuutVH+26S X-Received: by 2002:a92:c569:: with SMTP id b9mr3319118ilj.140.1644365280633; Tue, 08 Feb 2022 16:08:00 -0800 (PST) X-Google-Smtp-Source: ABdhPJywblXXWaC+On/ejrPQPFKI5TY6fQri38q7QXP5X10t9UODkgWyVQo1w9VqbvT8Kxxbj9HnJg== X-Received: by 2002:a92:c569:: with SMTP id b9mr3319107ilj.140.1644365280395; Tue, 08 Feb 2022 16:08:00 -0800 (PST) Received: from redhat.com ([38.15.36.239]) by smtp.gmail.com with ESMTPSA id i17sm5917685ilq.19.2022.02.08.16.07.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Feb 2022 16:08:00 -0800 (PST) Date: Tue, 8 Feb 2022 17:07:58 -0700 From: Alex Williamson To: Yishai Hadas Cc: , , , , , , , , , , , , , Subject: Re: [PATCH V7 mlx5-next 12/15] vfio/mlx5: Implement vfio_pci driver for mlx5 devices Message-ID: <20220208170758.48c17719.alex.williamson@redhat.com> In-Reply-To: <20220207172216.206415-13-yishaih@nvidia.com> References: <20220207172216.206415-1-yishaih@nvidia.com> <20220207172216.206415-13-yishaih@nvidia.com> Organization: Red Hat MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On Mon, 7 Feb 2022 19:22:13 +0200 Yishai Hadas wrote: > This patch adds support for vfio_pci driver for mlx5 devices. > > It uses vfio_pci_core to register to the VFIO subsystem and then > implements the mlx5 specific logic in the migration area. > > The migration implementation follows the definition from uapi/vfio.h and > uses the mlx5 VF->PF command channel to achieve it. > > This patch implements the suspend/resume flows. > > Signed-off-by: Yishai Hadas > Signed-off-by: Leon Romanovsky > Signed-off-by: Jason Gunthorpe > --- > MAINTAINERS | 6 + > drivers/vfio/pci/Kconfig | 3 + > drivers/vfio/pci/Makefile | 2 + > drivers/vfio/pci/mlx5/Kconfig | 10 + > drivers/vfio/pci/mlx5/Makefile | 4 + > drivers/vfio/pci/mlx5/cmd.h | 1 + > drivers/vfio/pci/mlx5/main.c | 623 +++++++++++++++++++++++++++++++++ > 7 files changed, 649 insertions(+) > create mode 100644 drivers/vfio/pci/mlx5/Kconfig > create mode 100644 drivers/vfio/pci/mlx5/Makefile > create mode 100644 drivers/vfio/pci/mlx5/main.c > > diff --git a/MAINTAINERS b/MAINTAINERS > index ea3e6c914384..5c5216f5e43d 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -20260,6 +20260,12 @@ L: kvm@vger.kernel.org > S: Maintained > F: drivers/vfio/platform/ > > +VFIO MLX5 PCI DRIVER > +M: Yishai Hadas > +L: kvm@vger.kernel.org > +S: Maintained > +F: drivers/vfio/pci/mlx5/ > + > VGA_SWITCHEROO > R: Lukas Wunner > S: Maintained > diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig > index 860424ccda1b..187b9c259944 100644 > --- a/drivers/vfio/pci/Kconfig > +++ b/drivers/vfio/pci/Kconfig > @@ -43,4 +43,7 @@ config VFIO_PCI_IGD > > To enable Intel IGD assignment through vfio-pci, say Y. > endif > + > +source "drivers/vfio/pci/mlx5/Kconfig" > + > endif > diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile > index 349d68d242b4..ed9d6f2e0555 100644 > --- a/drivers/vfio/pci/Makefile > +++ b/drivers/vfio/pci/Makefile > @@ -7,3 +7,5 @@ obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o > vfio-pci-y := vfio_pci.o > vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o > obj-$(CONFIG_VFIO_PCI) += vfio-pci.o > + > +obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ > diff --git a/drivers/vfio/pci/mlx5/Kconfig b/drivers/vfio/pci/mlx5/Kconfig > new file mode 100644 > index 000000000000..29ba9c504a75 > --- /dev/null > +++ b/drivers/vfio/pci/mlx5/Kconfig > @@ -0,0 +1,10 @@ > +# SPDX-License-Identifier: GPL-2.0-only > +config MLX5_VFIO_PCI > + tristate "VFIO support for MLX5 PCI devices" > + depends on MLX5_CORE > + depends on VFIO_PCI_CORE > + help > + This provides migration support for MLX5 devices using the VFIO > + framework. > + > + If you don't know what to do here, say N. > diff --git a/drivers/vfio/pci/mlx5/Makefile b/drivers/vfio/pci/mlx5/Makefile > new file mode 100644 > index 000000000000..689627da7ff5 > --- /dev/null > +++ b/drivers/vfio/pci/mlx5/Makefile > @@ -0,0 +1,4 @@ > +# SPDX-License-Identifier: GPL-2.0-only > +obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5-vfio-pci.o > +mlx5-vfio-pci-y := main.o cmd.o > + > diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h > index 69a1481ed953..1392a11a9cc0 100644 > --- a/drivers/vfio/pci/mlx5/cmd.h > +++ b/drivers/vfio/pci/mlx5/cmd.h > @@ -12,6 +12,7 @@ > struct mlx5_vf_migration_file { > struct file *filp; > struct mutex lock; > + bool disabled; > > struct sg_append_table table; > size_t total_length; > diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c > new file mode 100644 > index 000000000000..acd205bcff70 > --- /dev/null > +++ b/drivers/vfio/pci/mlx5/main.c > @@ -0,0 +1,623 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +/* > + * Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include "cmd.h" > + > +/* Arbitrary to prevent userspace from consuming endless memory */ > +#define MAX_MIGRATION_SIZE (512*1024*1024) > + > +struct mlx5vf_pci_core_device { > + struct vfio_pci_core_device core_device; > + u8 migrate_cap:1; > + /* protect migration state */ > + struct mutex state_mutex; > + enum vfio_device_mig_state mig_state; > + u16 vhca_id; > + struct mlx5_vf_migration_file *resuming_migf; > + struct mlx5_vf_migration_file *saving_migf; > +}; Nit, migrate_cap and vhca_id could minimally be contiguous for better packing of this struct. Thanks, Alex