* [PATCH v5 0/8] Introduce fwctl subystem
@ 2025-02-28 0:26 Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 1/8] fwctl: Add basic structure for a class subsystem with a cdev Jason Gunthorpe
` (9 more replies)
0 siblings, 10 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-02-28 0:26 UTC (permalink / raw)
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
[
We now have the required three drivers on the list so things are
looking probable for reaching this merge window. I will work out some
shared branches with CXL and get it into linux-next once all three drivers
can be assembled and reviews seem to be concluding.
There are couple open notes
- Greg was interested in a new name, Dan offered auxctl
]
fwctl is a new subsystem intended to bring some common rules and order to
the growing pattern of exposing a secure FW interface directly to
userspace. Unlike existing places like RDMA/DRM/VFIO/uacce that are
exposing a device for datapath operations fwctl is focused on debugging,
configuration and provisioning of the device. It will not have the
necessary features like interrupt delivery to support a datapath.
This concept is similar to the long standing practice in the "HW" RAID
space of having a device specific misc device to manage the RAID
controller FW. fwctl generalizes this notion of a companion debug and
management interface that goes along with a dataplane implemented in an
appropriate subsystem.
The need for this has reached a critical point as many users are moving to
run lockdown enabled kernels. Several existing devices have had long
standing tooling for management that relied on /sys/../resource0 or PCI
config space access which is not permitted in lockdown. A major point of
fwctl is to define and document the rules that a device must follow to
expose a lockdown compatible RPC.
Based on some discussion fwctl splits the RPCs into four categories
FWCTL_RPC_CONFIGURATION
FWCTL_RPC_DEBUG_READ_ONLY
FWCTL_RPC_DEBUG_WRITE
FWCTL_RPC_DEBUG_WRITE_FULL
Where the latter two trigger a new TAINT_FWCTL, and the final one requires
CAP_SYS_RAWIO - excluding it from lockdown. The device driver and its FW
would be responsible to restrict RPCs to the requested security scope,
while the core code handles the tainting and CAP checks.
For details see the final patch which introduces the documentation.
The CXL FWCTL driver is now in it own series on v7:
https://lore.kernel.org/r/20250220194438.2281088-1-dave.jiang@intel.com
And a driver for the Pensando DSC (A smart NIC):
https://lore.kernel.org/r/20250211234854.52277-1-shannon.nelson@amd.com
I've got soft commitments for about 7 drivers in total now.
There have been three LWN articles written discussing various aspects of
this proposal:
https://lwn.net/Articles/955001/
https://lwn.net/Articles/969383/
https://lwn.net/Articles/990802/
A really giant ksummit thread preceding a discussion at the Maintainer
Summit:
https://lore.kernel.org/ksummit/668c67a324609_ed99294c0@dwillia2-xfh.jf.intel.com.notmuch/
Several have expressed general support for this concept:
AMD/Pensando - https://lore.kernel.org/linux-rdma/20241205222818.44439-1-shannon.nelson@amd.com
Broadcom Networking - https://lore.kernel.org/r/Zf2n02q0GevGdS-Z@C02YVCJELVCG
Christoph Hellwig - https://lore.kernel.org/r/Zcx53N8lQjkpEu94@infradead.org
Daniel Vetter - https://lore.kernel.org/r/ZrHY2Bds7oF7KRGz@phenom.ffwll.local
Enfabrica - https://lore.kernel.org/r/9cc7127f-8674-43bc-b4d7-b1c4c2d96fed@kernel.org
NVIDIA Networking
Oded Gabbay/Habana - https://lore.kernel.org/r/ZrMl1bkPP-3G9B4N@T14sgabbay.
Oracle Linux - https://lore.kernel.org/r/6lakj6lxlxhdgrewodvj3xh6sxn3d36t5dab6najzyti2navx3@wrge7cyfk6nq
SuSE/Hannes - https://lore.kernel.org/r/2fd48f87-2521-4c34-8589-dbb7e91bb1c8@suse.com
Work is ongoing for userspace, currently the mellanox tool suite has been
ported over:
https://github.com/Mellanox/mstflint
And a more simplified example how to use it:
https://github.com/jgunthorpe/mlx5ctl.git
This is on github: https://github.com/jgunthorpe/linux/commits/fwctl
v5:
- Move hunks between patches to make more sense
- Rename ucmd_buffer to fwctl_ucmd_buffer
- Update comments and commit messages
- Copyright to 2025
- Drop bxnt WIP patches
- Allow a NULL ops->info
- Decode more op codes for mlx5 and the sub-operation for
MLX5_CMD_OP_ACCESS_REG/_USER
v4: https://patch.msgid.link/r/0-v4-0cf4ec3b8143+4995-fwctl_jgg@nvidia.com
- Rebase to v6.14-rc1
- Fine tune comments and rst documentatin
- Adjust cleanup.h usage - remove places that add more ofuscation than
value
- CXL is back to its own independent series
- Increase FWCTL_MAX_DEVICES to 4096, someone hit the limit
- Fix mlx5ctl_validate_rpc() logic around scope checking
- Disable mlx5ctl on SFs
v3: https://patch.msgid.link/r/0-v3-960f17f90f17+516-fwctl_jgg@nvidia.com
- Rebase to v6.11-rc4
- Add a squashed version of David's CXL series as the 2nd driver
- Add missing includes
- Improve comments based on feedback
- Use the kdoc format that puts the member docs inside the struct
- Rewrite fwctl_alloc_device() to be clearer
- Incorporate all remarks for the documentation
v2: https://lore.kernel.org/r/0-v2-940e479ceba9+3821-fwctl_jgg@nvidia.com
- Rebase to v6.10-rc5
- Minor style changes
- Follow the style consensus for the guard stuff
- Documentation grammer/spelling
- Add missed length output for mlx5 get_info
- Add two more missed MLX5 CMD's
- Collect tags
v1: https://lore.kernel.org/r/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
Cc: Andy Gospodarek <andrew.gospodarek@broadcom.com>
Cc: Aron Silverton <aron.silverton@oracle.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: David Ahern <dsahern@kernel.org>
Cc: Itay Avraham <itayavr@nvidia.com>
Cc: Jakub Kicinski <kuba@kernel.org>
Cc: netdev@vger.kernel.org
Cc: Jiri Pirko <jiri@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Leonid Bloch <lbloch@nvidia.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: linux-cxl@vger.kernel.org
Cc: linux-rdma@vger.kernel.org
Cc: "Nelson, Shannon" <shannon.nelson@amd.com>
Cc: Dave Jiang <dave.jiang@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Jason Gunthorpe (6):
fwctl: Add basic structure for a class subsystem with a cdev
fwctl: Basic ioctl dispatch for the character device
fwctl: FWCTL_INFO to return basic information about the device
taint: Add TAINT_FWCTL
fwctl: FWCTL_RPC to execute a Remote Procedure Call to device firmware
fwctl: Add documentation
Saeed Mahameed (2):
fwctl/mlx5: Support for communicating with mlx5 fw
mlx5: Create an auxiliary device for fwctl_mlx5
Documentation/admin-guide/tainted-kernels.rst | 5 +
Documentation/userspace-api/fwctl/fwctl.rst | 285 ++++++++++++
Documentation/userspace-api/fwctl/index.rst | 12 +
Documentation/userspace-api/index.rst | 1 +
.../userspace-api/ioctl/ioctl-number.rst | 1 +
MAINTAINERS | 18 +
drivers/Kconfig | 2 +
drivers/Makefile | 1 +
drivers/fwctl/Kconfig | 23 +
drivers/fwctl/Makefile | 5 +
drivers/fwctl/main.c | 421 ++++++++++++++++++
drivers/fwctl/mlx5/Makefile | 4 +
drivers/fwctl/mlx5/main.c | 411 +++++++++++++++++
drivers/net/ethernet/mellanox/mlx5/core/dev.c | 9 +
include/linux/fwctl.h | 135 ++++++
include/linux/panic.h | 3 +-
include/uapi/fwctl/fwctl.h | 139 ++++++
include/uapi/fwctl/mlx5.h | 36 ++
kernel/panic.c | 1 +
tools/debugging/kernel-chktaint | 8 +
20 files changed, 1519 insertions(+), 1 deletion(-)
create mode 100644 Documentation/userspace-api/fwctl/fwctl.rst
create mode 100644 Documentation/userspace-api/fwctl/index.rst
create mode 100644 drivers/fwctl/Kconfig
create mode 100644 drivers/fwctl/Makefile
create mode 100644 drivers/fwctl/main.c
create mode 100644 drivers/fwctl/mlx5/Makefile
create mode 100644 drivers/fwctl/mlx5/main.c
create mode 100644 include/linux/fwctl.h
create mode 100644 include/uapi/fwctl/fwctl.h
create mode 100644 include/uapi/fwctl/mlx5.h
base-commit: 2014c95afecee3e76ca4a56956a936e23283f05b
--
2.43.0
^ permalink raw reply [flat|nested] 56+ messages in thread
* [PATCH v5 1/8] fwctl: Add basic structure for a class subsystem with a cdev
2025-02-28 0:26 [PATCH v5 0/8] Introduce fwctl subystem Jason Gunthorpe
@ 2025-02-28 0:26 ` Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 2/8] fwctl: Basic ioctl dispatch for the character device Jason Gunthorpe
` (8 subsequent siblings)
9 siblings, 0 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-02-28 0:26 UTC (permalink / raw)
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
Create the class, character device and functions for a fwctl driver to
un/register to the subsystem.
A typical fwctl driver has a sysfs presence like:
$ ls -l /dev/fwctl/fwctl0
crw------- 1 root root 250, 0 Apr 25 19:16 /dev/fwctl/fwctl0
$ ls /sys/class/fwctl/fwctl0
dev device power subsystem uevent
$ ls /sys/class/fwctl/fwctl0/device/infiniband/
ibp0s10f0
$ ls /sys/class/infiniband/ibp0s10f0/device/fwctl/
fwctl0/
$ ls /sys/devices/pci0000:00/0000:00:0a.0/fwctl/fwctl0
dev device power subsystem uevent
Which allows userspace to link all the multi-subsystem driver components
together and learn the subsystem specific names for the device's
components.
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Tested-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
Tested-by: Shannon Nelson <shannon.nelson@amd.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
MAINTAINERS | 7 ++
drivers/Kconfig | 2 +
drivers/Makefile | 1 +
drivers/fwctl/Kconfig | 9 +++
drivers/fwctl/Makefile | 4 +
drivers/fwctl/main.c | 173 +++++++++++++++++++++++++++++++++++++++++
include/linux/fwctl.h | 69 ++++++++++++++++
7 files changed, 265 insertions(+)
create mode 100644 drivers/fwctl/Kconfig
create mode 100644 drivers/fwctl/Makefile
create mode 100644 drivers/fwctl/main.c
create mode 100644 include/linux/fwctl.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 896a307fa06545..1dbf0f216f9936 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9557,6 +9557,13 @@ F: kernel/futex/*
F: tools/perf/bench/futex*
F: tools/testing/selftests/futex/
+FWCTL SUBSYSTEM
+M: Jason Gunthorpe <jgg@nvidia.com>
+M: Saeed Mahameed <saeedm@nvidia.com>
+S: Maintained
+F: drivers/fwctl/
+F: include/linux/fwctl.h
+
GALAXYCORE GC0308 CAMERA SENSOR DRIVER
M: Sebastian Reichel <sre@kernel.org>
L: linux-media@vger.kernel.org
diff --git a/drivers/Kconfig b/drivers/Kconfig
index 7bdad836fc6207..7c556c5ac4fddc 100644
--- a/drivers/Kconfig
+++ b/drivers/Kconfig
@@ -21,6 +21,8 @@ source "drivers/connector/Kconfig"
source "drivers/firmware/Kconfig"
+source "drivers/fwctl/Kconfig"
+
source "drivers/gnss/Kconfig"
source "drivers/mtd/Kconfig"
diff --git a/drivers/Makefile b/drivers/Makefile
index 45d1c3e630f754..b5749cf67044ce 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -135,6 +135,7 @@ obj-y += ufs/
obj-$(CONFIG_MEMSTICK) += memstick/
obj-$(CONFIG_INFINIBAND) += infiniband/
obj-y += firmware/
+obj-$(CONFIG_FWCTL) += fwctl/
obj-$(CONFIG_CRYPTO) += crypto/
obj-$(CONFIG_SUPERH) += sh/
obj-y += clocksource/
diff --git a/drivers/fwctl/Kconfig b/drivers/fwctl/Kconfig
new file mode 100644
index 00000000000000..37147a695add9a
--- /dev/null
+++ b/drivers/fwctl/Kconfig
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menuconfig FWCTL
+ tristate "fwctl device firmware access framework"
+ help
+ fwctl provides a userspace API for restricted access to communicate
+ with on-device firmware. The communication channel is intended to
+ support a wide range of lockdown compatible device behaviors including
+ manipulating device FLASH, debugging, and other activities that don't
+ fit neatly into an existing subsystem.
diff --git a/drivers/fwctl/Makefile b/drivers/fwctl/Makefile
new file mode 100644
index 00000000000000..1cad210f6ba580
--- /dev/null
+++ b/drivers/fwctl/Makefile
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_FWCTL) += fwctl.o
+
+fwctl-y += main.o
diff --git a/drivers/fwctl/main.c b/drivers/fwctl/main.c
new file mode 100644
index 00000000000000..76c4edec20afde
--- /dev/null
+++ b/drivers/fwctl/main.c
@@ -0,0 +1,173 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES
+ */
+#define pr_fmt(fmt) "fwctl: " fmt
+#include <linux/fwctl.h>
+
+#include <linux/container_of.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+
+enum {
+ FWCTL_MAX_DEVICES = 4096,
+};
+static_assert(FWCTL_MAX_DEVICES < (1U << MINORBITS));
+
+static dev_t fwctl_dev;
+static DEFINE_IDA(fwctl_ida);
+
+static int fwctl_fops_open(struct inode *inode, struct file *filp)
+{
+ struct fwctl_device *fwctl =
+ container_of(inode->i_cdev, struct fwctl_device, cdev);
+
+ get_device(&fwctl->dev);
+ filp->private_data = fwctl;
+ return 0;
+}
+
+static int fwctl_fops_release(struct inode *inode, struct file *filp)
+{
+ struct fwctl_device *fwctl = filp->private_data;
+
+ fwctl_put(fwctl);
+ return 0;
+}
+
+static const struct file_operations fwctl_fops = {
+ .owner = THIS_MODULE,
+ .open = fwctl_fops_open,
+ .release = fwctl_fops_release,
+};
+
+static void fwctl_device_release(struct device *device)
+{
+ struct fwctl_device *fwctl =
+ container_of(device, struct fwctl_device, dev);
+
+ ida_free(&fwctl_ida, fwctl->dev.devt - fwctl_dev);
+ kfree(fwctl);
+}
+
+static char *fwctl_devnode(const struct device *dev, umode_t *mode)
+{
+ return kasprintf(GFP_KERNEL, "fwctl/%s", dev_name(dev));
+}
+
+static struct class fwctl_class = {
+ .name = "fwctl",
+ .dev_release = fwctl_device_release,
+ .devnode = fwctl_devnode,
+};
+
+static struct fwctl_device *
+_alloc_device(struct device *parent, const struct fwctl_ops *ops, size_t size)
+{
+ struct fwctl_device *fwctl __free(kfree) = kzalloc(size, GFP_KERNEL);
+ int devnum;
+
+ if (!fwctl)
+ return NULL;
+
+ fwctl->dev.class = &fwctl_class;
+ fwctl->dev.parent = parent;
+
+ devnum = ida_alloc_max(&fwctl_ida, FWCTL_MAX_DEVICES - 1, GFP_KERNEL);
+ if (devnum < 0)
+ return NULL;
+
+ fwctl->dev.devt = fwctl_dev + devnum;
+ fwctl->dev.class = &fwctl_class;
+ fwctl->dev.parent = parent;
+
+ device_initialize(&fwctl->dev);
+ return_ptr(fwctl);
+}
+
+/* Drivers use the fwctl_alloc_device() wrapper */
+struct fwctl_device *_fwctl_alloc_device(struct device *parent,
+ const struct fwctl_ops *ops,
+ size_t size)
+{
+ struct fwctl_device *fwctl __free(fwctl) =
+ _alloc_device(parent, ops, size);
+
+ if (!fwctl)
+ return NULL;
+
+ cdev_init(&fwctl->cdev, &fwctl_fops);
+ /*
+ * The driver module is protected by fwctl_register/unregister(),
+ * unregister won't complete until we are done with the driver's module.
+ */
+ fwctl->cdev.owner = THIS_MODULE;
+
+ if (dev_set_name(&fwctl->dev, "fwctl%d", fwctl->dev.devt - fwctl_dev))
+ return NULL;
+
+ fwctl->ops = ops;
+ return_ptr(fwctl);
+}
+EXPORT_SYMBOL_NS_GPL(_fwctl_alloc_device, "FWCTL");
+
+/**
+ * fwctl_register - Register a new device to the subsystem
+ * @fwctl: Previously allocated fwctl_device
+ *
+ * On return the device is visible through sysfs and /dev, driver ops may be
+ * called.
+ */
+int fwctl_register(struct fwctl_device *fwctl)
+{
+ return cdev_device_add(&fwctl->cdev, &fwctl->dev);
+}
+EXPORT_SYMBOL_NS_GPL(fwctl_register, "FWCTL");
+
+/**
+ * fwctl_unregister - Unregister a device from the subsystem
+ * @fwctl: Previously allocated and registered fwctl_device
+ *
+ * Undoes fwctl_register(). On return no driver ops will be called. The
+ * caller must still call fwctl_put() to free the fwctl.
+ *
+ * The design of fwctl allows this sort of disassociation of the driver from the
+ * subsystem primarily by keeping memory allocations owned by the core subsytem.
+ * The fwctl_device and fwctl_uctx can both be freed without requiring a driver
+ * callback. This allows the module to remain unlocked while FDs are open.
+ */
+void fwctl_unregister(struct fwctl_device *fwctl)
+{
+ cdev_device_del(&fwctl->cdev, &fwctl->dev);
+}
+EXPORT_SYMBOL_NS_GPL(fwctl_unregister, "FWCTL");
+
+static int __init fwctl_init(void)
+{
+ int ret;
+
+ ret = alloc_chrdev_region(&fwctl_dev, 0, FWCTL_MAX_DEVICES, "fwctl");
+ if (ret)
+ return ret;
+
+ ret = class_register(&fwctl_class);
+ if (ret)
+ goto err_chrdev;
+ return 0;
+
+err_chrdev:
+ unregister_chrdev_region(fwctl_dev, FWCTL_MAX_DEVICES);
+ return ret;
+}
+
+static void __exit fwctl_exit(void)
+{
+ class_unregister(&fwctl_class);
+ unregister_chrdev_region(fwctl_dev, FWCTL_MAX_DEVICES);
+}
+
+module_init(fwctl_init);
+module_exit(fwctl_exit);
+MODULE_DESCRIPTION("fwctl device firmware access framework");
+MODULE_LICENSE("GPL");
diff --git a/include/linux/fwctl.h b/include/linux/fwctl.h
new file mode 100644
index 00000000000000..39d5059c9e592b
--- /dev/null
+++ b/include/linux/fwctl.h
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES
+ */
+#ifndef __LINUX_FWCTL_H
+#define __LINUX_FWCTL_H
+#include <linux/device.h>
+#include <linux/cdev.h>
+#include <linux/cleanup.h>
+
+struct fwctl_device;
+struct fwctl_uctx;
+
+struct fwctl_ops {
+};
+
+/**
+ * struct fwctl_device - Per-driver registration struct
+ * @dev: The sysfs (class/fwctl/fwctlXX) device
+ *
+ * Each driver instance will have one of these structs with the driver private
+ * data following immediately after. This struct is refcounted, it is freed by
+ * calling fwctl_put().
+ */
+struct fwctl_device {
+ struct device dev;
+ /* private: */
+ struct cdev cdev;
+ const struct fwctl_ops *ops;
+};
+
+struct fwctl_device *_fwctl_alloc_device(struct device *parent,
+ const struct fwctl_ops *ops,
+ size_t size);
+/**
+ * fwctl_alloc_device - Allocate a fwctl
+ * @parent: Physical device that provides the FW interface
+ * @ops: Driver ops to register
+ * @drv_struct: 'struct driver_fwctl' that holds the struct fwctl_device
+ * @member: Name of the struct fwctl_device in @drv_struct
+ *
+ * This allocates and initializes the fwctl_device embedded in the drv_struct.
+ * Upon success the pointer must be freed via fwctl_put(). Returns a 'drv_struct
+ * \*' on success, NULL on error.
+ */
+#define fwctl_alloc_device(parent, ops, drv_struct, member) \
+ ({ \
+ static_assert(__same_type(struct fwctl_device, \
+ ((drv_struct *)NULL)->member)); \
+ static_assert(offsetof(drv_struct, member) == 0); \
+ (drv_struct *)_fwctl_alloc_device(parent, ops, \
+ sizeof(drv_struct)); \
+ })
+
+static inline struct fwctl_device *fwctl_get(struct fwctl_device *fwctl)
+{
+ get_device(&fwctl->dev);
+ return fwctl;
+}
+static inline void fwctl_put(struct fwctl_device *fwctl)
+{
+ put_device(&fwctl->dev);
+}
+DEFINE_FREE(fwctl, struct fwctl_device *, if (_T) fwctl_put(_T));
+
+int fwctl_register(struct fwctl_device *fwctl);
+void fwctl_unregister(struct fwctl_device *fwctl);
+
+#endif
--
2.43.0
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v5 2/8] fwctl: Basic ioctl dispatch for the character device
2025-02-28 0:26 [PATCH v5 0/8] Introduce fwctl subystem Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 1/8] fwctl: Add basic structure for a class subsystem with a cdev Jason Gunthorpe
@ 2025-02-28 0:26 ` Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 3/8] fwctl: FWCTL_INFO to return basic information about the device Jason Gunthorpe
` (7 subsequent siblings)
9 siblings, 0 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-02-28 0:26 UTC (permalink / raw)
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
Each file descriptor gets a chunk of per-FD driver specific context that
allows the driver to attach a device specific struct to. The core code
takes care of the memory lifetime for this structure.
The ioctl dispatch and design is based on what was built for iommufd. The
ioctls have a struct which has a combined in/out behavior with a typical
'zero pad' scheme for future extension and backwards compatibility.
Like iommufd some shared logic does most of the ioctl marshaling and
compatibility work and table dispatches to some function pointers for
each unique ioctl.
This approach has proven to work quite well in the iommufd and rdma
subsystems.
Allocate an ioctl number space for the subsystem.
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Tested-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
Tested-by: Shannon Nelson <shannon.nelson@amd.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../userspace-api/ioctl/ioctl-number.rst | 1 +
MAINTAINERS | 1 +
drivers/fwctl/main.c | 143 +++++++++++++++++-
include/linux/fwctl.h | 46 ++++++
include/uapi/fwctl/fwctl.h | 38 +++++
5 files changed, 224 insertions(+), 5 deletions(-)
create mode 100644 include/uapi/fwctl/fwctl.h
diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
index 6d1465315df328..3410b020a9d093 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -331,6 +331,7 @@ Code Seq# Include File Comments
0x97 00-7F fs/ceph/ioctl.h Ceph file system
0x99 00-0F 537-Addinboard driver
<mailto:buk@buks.ipn.de>
+0x9A 00-0F include/uapi/fwctl/fwctl.h
0xA0 all linux/sdp/sdp.h Industrial Device Project
<mailto:kenji@bitgate.com>
0xA1 0 linux/vtpm_proxy.h TPM Emulator Proxy Driver
diff --git a/MAINTAINERS b/MAINTAINERS
index 1dbf0f216f9936..1bc1f97934b6b8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9563,6 +9563,7 @@ M: Saeed Mahameed <saeedm@nvidia.com>
S: Maintained
F: drivers/fwctl/
F: include/linux/fwctl.h
+F: include/uapi/fwctl/
GALAXYCORE GC0308 CAMERA SENSOR DRIVER
M: Sebastian Reichel <sre@kernel.org>
diff --git a/drivers/fwctl/main.c b/drivers/fwctl/main.c
index 76c4edec20afde..4bba6109712e42 100644
--- a/drivers/fwctl/main.c
+++ b/drivers/fwctl/main.c
@@ -10,6 +10,8 @@
#include <linux/module.h>
#include <linux/slab.h>
+#include <uapi/fwctl/fwctl.h>
+
enum {
FWCTL_MAX_DEVICES = 4096,
};
@@ -18,20 +20,128 @@ static_assert(FWCTL_MAX_DEVICES < (1U << MINORBITS));
static dev_t fwctl_dev;
static DEFINE_IDA(fwctl_ida);
+struct fwctl_ucmd {
+ struct fwctl_uctx *uctx;
+ void __user *ubuffer;
+ void *cmd;
+ u32 user_size;
+};
+
+/* On stack memory for the ioctl structs */
+union fwctl_ucmd_buffer {
+};
+
+struct fwctl_ioctl_op {
+ unsigned int size;
+ unsigned int min_size;
+ unsigned int ioctl_num;
+ int (*execute)(struct fwctl_ucmd *ucmd);
+};
+
+#define IOCTL_OP(_ioctl, _fn, _struct, _last) \
+ [_IOC_NR(_ioctl) - FWCTL_CMD_BASE] = { \
+ .size = sizeof(_struct) + \
+ BUILD_BUG_ON_ZERO(sizeof(union fwctl_ucmd_buffer) < \
+ sizeof(_struct)), \
+ .min_size = offsetofend(_struct, _last), \
+ .ioctl_num = _ioctl, \
+ .execute = _fn, \
+ }
+static const struct fwctl_ioctl_op fwctl_ioctl_ops[] = {
+};
+
+static long fwctl_fops_ioctl(struct file *filp, unsigned int cmd,
+ unsigned long arg)
+{
+ struct fwctl_uctx *uctx = filp->private_data;
+ const struct fwctl_ioctl_op *op;
+ struct fwctl_ucmd ucmd = {};
+ union fwctl_ucmd_buffer buf;
+ unsigned int nr;
+ int ret;
+
+ nr = _IOC_NR(cmd);
+ if ((nr - FWCTL_CMD_BASE) >= ARRAY_SIZE(fwctl_ioctl_ops))
+ return -ENOIOCTLCMD;
+
+ op = &fwctl_ioctl_ops[nr - FWCTL_CMD_BASE];
+ if (op->ioctl_num != cmd)
+ return -ENOIOCTLCMD;
+
+ ucmd.uctx = uctx;
+ ucmd.cmd = &buf;
+ ucmd.ubuffer = (void __user *)arg;
+ ret = get_user(ucmd.user_size, (u32 __user *)ucmd.ubuffer);
+ if (ret)
+ return ret;
+
+ if (ucmd.user_size < op->min_size)
+ return -EINVAL;
+
+ ret = copy_struct_from_user(ucmd.cmd, op->size, ucmd.ubuffer,
+ ucmd.user_size);
+ if (ret)
+ return ret;
+
+ guard(rwsem_read)(&uctx->fwctl->registration_lock);
+ if (!uctx->fwctl->ops)
+ return -ENODEV;
+ return op->execute(&ucmd);
+}
+
static int fwctl_fops_open(struct inode *inode, struct file *filp)
{
struct fwctl_device *fwctl =
container_of(inode->i_cdev, struct fwctl_device, cdev);
+ int ret;
+
+ guard(rwsem_read)(&fwctl->registration_lock);
+ if (!fwctl->ops)
+ return -ENODEV;
+
+ struct fwctl_uctx *uctx __free(kfree) =
+ kzalloc(fwctl->ops->uctx_size, GFP_KERNEL_ACCOUNT);
+ if (!uctx)
+ return -ENOMEM;
+
+ uctx->fwctl = fwctl;
+ ret = fwctl->ops->open_uctx(uctx);
+ if (ret)
+ return ret;
+
+ scoped_guard(mutex, &fwctl->uctx_list_lock) {
+ list_add_tail(&uctx->uctx_list_entry, &fwctl->uctx_list);
+ }
get_device(&fwctl->dev);
- filp->private_data = fwctl;
+ filp->private_data = no_free_ptr(uctx);
return 0;
}
+static void fwctl_destroy_uctx(struct fwctl_uctx *uctx)
+{
+ lockdep_assert_held(&uctx->fwctl->uctx_list_lock);
+ list_del(&uctx->uctx_list_entry);
+ uctx->fwctl->ops->close_uctx(uctx);
+}
+
static int fwctl_fops_release(struct inode *inode, struct file *filp)
{
- struct fwctl_device *fwctl = filp->private_data;
+ struct fwctl_uctx *uctx = filp->private_data;
+ struct fwctl_device *fwctl = uctx->fwctl;
+ scoped_guard(rwsem_read, &fwctl->registration_lock) {
+ /*
+ * NULL ops means fwctl_unregister() has already removed the
+ * driver and destroyed the uctx.
+ */
+ if (fwctl->ops) {
+ guard(mutex)(&fwctl->uctx_list_lock);
+ fwctl_destroy_uctx(uctx);
+ }
+ }
+
+ kfree(uctx);
fwctl_put(fwctl);
return 0;
}
@@ -40,6 +150,7 @@ static const struct file_operations fwctl_fops = {
.owner = THIS_MODULE,
.open = fwctl_fops_open,
.release = fwctl_fops_release,
+ .unlocked_ioctl = fwctl_fops_ioctl,
};
static void fwctl_device_release(struct device *device)
@@ -48,6 +159,7 @@ static void fwctl_device_release(struct device *device)
container_of(device, struct fwctl_device, dev);
ida_free(&fwctl_ida, fwctl->dev.devt - fwctl_dev);
+ mutex_destroy(&fwctl->uctx_list_lock);
kfree(fwctl);
}
@@ -71,9 +183,6 @@ _alloc_device(struct device *parent, const struct fwctl_ops *ops, size_t size)
if (!fwctl)
return NULL;
- fwctl->dev.class = &fwctl_class;
- fwctl->dev.parent = parent;
-
devnum = ida_alloc_max(&fwctl_ida, FWCTL_MAX_DEVICES - 1, GFP_KERNEL);
if (devnum < 0)
return NULL;
@@ -82,6 +191,10 @@ _alloc_device(struct device *parent, const struct fwctl_ops *ops, size_t size)
fwctl->dev.class = &fwctl_class;
fwctl->dev.parent = parent;
+ init_rwsem(&fwctl->registration_lock);
+ mutex_init(&fwctl->uctx_list_lock);
+ INIT_LIST_HEAD(&fwctl->uctx_list);
+
device_initialize(&fwctl->dev);
return_ptr(fwctl);
}
@@ -132,6 +245,10 @@ EXPORT_SYMBOL_NS_GPL(fwctl_register, "FWCTL");
* Undoes fwctl_register(). On return no driver ops will be called. The
* caller must still call fwctl_put() to free the fwctl.
*
+ * Unregister will return even if userspace still has file descriptors open.
+ * This will call ops->close_uctx() on any open FDs and after return no driver
+ * op will be called. The FDs remain open but all fops will return -ENODEV.
+ *
* The design of fwctl allows this sort of disassociation of the driver from the
* subsystem primarily by keeping memory allocations owned by the core subsytem.
* The fwctl_device and fwctl_uctx can both be freed without requiring a driver
@@ -139,7 +256,23 @@ EXPORT_SYMBOL_NS_GPL(fwctl_register, "FWCTL");
*/
void fwctl_unregister(struct fwctl_device *fwctl)
{
+ struct fwctl_uctx *uctx;
+
cdev_device_del(&fwctl->cdev, &fwctl->dev);
+
+ /* Disable and free the driver's resources for any still open FDs. */
+ guard(rwsem_write)(&fwctl->registration_lock);
+ guard(mutex)(&fwctl->uctx_list_lock);
+ while ((uctx = list_first_entry_or_null(&fwctl->uctx_list,
+ struct fwctl_uctx,
+ uctx_list_entry)))
+ fwctl_destroy_uctx(uctx);
+
+ /*
+ * The driver module may unload after this returns, the op pointer will
+ * not be valid.
+ */
+ fwctl->ops = NULL;
}
EXPORT_SYMBOL_NS_GPL(fwctl_unregister, "FWCTL");
diff --git a/include/linux/fwctl.h b/include/linux/fwctl.h
index 39d5059c9e592b..faa4b2c780e0c6 100644
--- a/include/linux/fwctl.h
+++ b/include/linux/fwctl.h
@@ -11,7 +11,30 @@
struct fwctl_device;
struct fwctl_uctx;
+/**
+ * struct fwctl_ops - Driver provided operations
+ *
+ * fwctl_unregister() will wait until all excuting ops are completed before it
+ * returns. Drivers should be mindful to not let their ops run for too long as
+ * it will block device hot unplug and module unloading.
+ */
struct fwctl_ops {
+ /**
+ * @uctx_size: The size of the fwctl_uctx struct to allocate. The first
+ * bytes of this memory will be a fwctl_uctx. The driver can use the
+ * remaining bytes as its private memory.
+ */
+ size_t uctx_size;
+ /**
+ * @open_uctx: Called when a file descriptor is opened before the uctx
+ * is ever used.
+ */
+ int (*open_uctx)(struct fwctl_uctx *uctx);
+ /**
+ * @close_uctx: Called when the uctx is destroyed, usually when the FD
+ * is closed.
+ */
+ void (*close_uctx)(struct fwctl_uctx *uctx);
};
/**
@@ -26,6 +49,15 @@ struct fwctl_device {
struct device dev;
/* private: */
struct cdev cdev;
+
+ /* Protect uctx_list */
+ struct mutex uctx_list_lock;
+ struct list_head uctx_list;
+ /*
+ * Protect ops, held for write when ops becomes NULL during unregister,
+ * held for read whenever ops is loaded or an ops function is running.
+ */
+ struct rw_semaphore registration_lock;
const struct fwctl_ops *ops;
};
@@ -66,4 +98,18 @@ DEFINE_FREE(fwctl, struct fwctl_device *, if (_T) fwctl_put(_T));
int fwctl_register(struct fwctl_device *fwctl);
void fwctl_unregister(struct fwctl_device *fwctl);
+/**
+ * struct fwctl_uctx - Per user FD context
+ * @fwctl: fwctl instance that owns the context
+ *
+ * Every FD opened by userspace will get a unique context allocation. Any driver
+ * private data will follow immediately after.
+ */
+struct fwctl_uctx {
+ struct fwctl_device *fwctl;
+ /* private: */
+ /* Head at fwctl_device::uctx_list */
+ struct list_head uctx_list_entry;
+};
+
#endif
diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
new file mode 100644
index 00000000000000..8f5fe821cf286e
--- /dev/null
+++ b/include/uapi/fwctl/fwctl.h
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES.
+ */
+#ifndef _UAPI_FWCTL_H
+#define _UAPI_FWCTL_H
+
+#define FWCTL_TYPE 0x9A
+
+/**
+ * DOC: General ioctl format
+ *
+ * The ioctl interface follows a general format to allow for extensibility. Each
+ * ioctl is passed a structure pointer as the argument providing the size of
+ * the structure in the first u32. The kernel checks that any structure space
+ * beyond what it understands is 0. This allows userspace to use the backward
+ * compatible portion while consistently using the newer, larger, structures.
+ *
+ * ioctls use a standard meaning for common errnos:
+ *
+ * - ENOTTY: The IOCTL number itself is not supported at all
+ * - E2BIG: The IOCTL number is supported, but the provided structure has
+ * non-zero in a part the kernel does not understand.
+ * - EOPNOTSUPP: The IOCTL number is supported, and the structure is
+ * understood, however a known field has a value the kernel does not
+ * understand or support.
+ * - EINVAL: Everything about the IOCTL was understood, but a field is not
+ * correct.
+ * - ENOMEM: Out of memory.
+ * - ENODEV: The underlying device has been hot-unplugged and the FD is
+ * orphaned.
+ *
+ * As well as additional errnos, within specific ioctls.
+ */
+enum {
+ FWCTL_CMD_BASE = 0,
+};
+
+#endif
--
2.43.0
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v5 3/8] fwctl: FWCTL_INFO to return basic information about the device
2025-02-28 0:26 [PATCH v5 0/8] Introduce fwctl subystem Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 1/8] fwctl: Add basic structure for a class subsystem with a cdev Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 2/8] fwctl: Basic ioctl dispatch for the character device Jason Gunthorpe
@ 2025-02-28 0:26 ` Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 4/8] taint: Add TAINT_FWCTL Jason Gunthorpe
` (6 subsequent siblings)
9 siblings, 0 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-02-28 0:26 UTC (permalink / raw)
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
Userspace will need to know some details about the fwctl interface being
used to locate the correct userspace code to communicate with the
kernel. Provide a simple device_type enum indicating what the kernel
driver is.
Allow the device to provide a device specific info struct that contains
any additional information that the driver may need to provide to
userspace.
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Tested-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
Tested-by: Shannon Nelson <shannon.nelson@amd.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/fwctl/main.c | 55 ++++++++++++++++++++++++++++++++++++++
include/linux/fwctl.h | 12 +++++++++
include/uapi/fwctl/fwctl.h | 31 +++++++++++++++++++++
3 files changed, 98 insertions(+)
diff --git a/drivers/fwctl/main.c b/drivers/fwctl/main.c
index 4bba6109712e42..bd84c4e5b3b92e 100644
--- a/drivers/fwctl/main.c
+++ b/drivers/fwctl/main.c
@@ -27,8 +27,62 @@ struct fwctl_ucmd {
u32 user_size;
};
+static int ucmd_respond(struct fwctl_ucmd *ucmd, size_t cmd_len)
+{
+ if (copy_to_user(ucmd->ubuffer, ucmd->cmd,
+ min_t(size_t, ucmd->user_size, cmd_len)))
+ return -EFAULT;
+ return 0;
+}
+
+static int copy_to_user_zero_pad(void __user *to, const void *from,
+ size_t from_len, size_t user_len)
+{
+ size_t copy_len;
+
+ copy_len = min(from_len, user_len);
+ if (copy_to_user(to, from, copy_len))
+ return -EFAULT;
+ if (copy_len < user_len) {
+ if (clear_user(to + copy_len, user_len - copy_len))
+ return -EFAULT;
+ }
+ return 0;
+}
+
+static int fwctl_cmd_info(struct fwctl_ucmd *ucmd)
+{
+ struct fwctl_device *fwctl = ucmd->uctx->fwctl;
+ struct fwctl_info *cmd = ucmd->cmd;
+ size_t driver_info_len = 0;
+
+ if (cmd->flags)
+ return -EOPNOTSUPP;
+
+ if (!fwctl->ops->info && cmd->device_data_len) {
+ if (clear_user(u64_to_user_ptr(cmd->out_device_data),
+ cmd->device_data_len))
+ return -EFAULT;
+ } else if (cmd->device_data_len) {
+ void *driver_info __free(kfree) =
+ fwctl->ops->info(ucmd->uctx, &driver_info_len);
+ if (IS_ERR(driver_info))
+ return PTR_ERR(driver_info);
+
+ if (copy_to_user_zero_pad(u64_to_user_ptr(cmd->out_device_data),
+ driver_info, driver_info_len,
+ cmd->device_data_len))
+ return -EFAULT;
+ }
+
+ cmd->out_device_type = fwctl->ops->device_type;
+ cmd->device_data_len = driver_info_len;
+ return ucmd_respond(ucmd, sizeof(*cmd));
+}
+
/* On stack memory for the ioctl structs */
union fwctl_ucmd_buffer {
+ struct fwctl_info info;
};
struct fwctl_ioctl_op {
@@ -48,6 +102,7 @@ struct fwctl_ioctl_op {
.execute = _fn, \
}
static const struct fwctl_ioctl_op fwctl_ioctl_ops[] = {
+ IOCTL_OP(FWCTL_INFO, fwctl_cmd_info, struct fwctl_info, out_device_data),
};
static long fwctl_fops_ioctl(struct file *filp, unsigned int cmd,
diff --git a/include/linux/fwctl.h b/include/linux/fwctl.h
index faa4b2c780e0c6..700a5be940e365 100644
--- a/include/linux/fwctl.h
+++ b/include/linux/fwctl.h
@@ -7,6 +7,7 @@
#include <linux/device.h>
#include <linux/cdev.h>
#include <linux/cleanup.h>
+#include <uapi/fwctl/fwctl.h>
struct fwctl_device;
struct fwctl_uctx;
@@ -19,6 +20,10 @@ struct fwctl_uctx;
* it will block device hot unplug and module unloading.
*/
struct fwctl_ops {
+ /**
+ * @device_type: The drivers assigned device_type number. This is uABI.
+ */
+ enum fwctl_device_type device_type;
/**
* @uctx_size: The size of the fwctl_uctx struct to allocate. The first
* bytes of this memory will be a fwctl_uctx. The driver can use the
@@ -35,6 +40,13 @@ struct fwctl_ops {
* is closed.
*/
void (*close_uctx)(struct fwctl_uctx *uctx);
+ /**
+ * @info: Implement FWCTL_INFO. Return a kmalloc() memory that is copied
+ * to out_device_data. On input length indicates the size of the user
+ * buffer on output it indicates the size of the memory. The driver can
+ * ignore length on input, the core code will handle everything.
+ */
+ void *(*info)(struct fwctl_uctx *uctx, size_t *length);
};
/**
diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
index 8f5fe821cf286e..4052df63f66dc6 100644
--- a/include/uapi/fwctl/fwctl.h
+++ b/include/uapi/fwctl/fwctl.h
@@ -4,6 +4,9 @@
#ifndef _UAPI_FWCTL_H
#define _UAPI_FWCTL_H
+#include <linux/types.h>
+#include <linux/ioctl.h>
+
#define FWCTL_TYPE 0x9A
/**
@@ -33,6 +36,34 @@
*/
enum {
FWCTL_CMD_BASE = 0,
+ FWCTL_CMD_INFO = 0,
};
+enum fwctl_device_type {
+ FWCTL_DEVICE_TYPE_ERROR = 0,
+};
+
+/**
+ * struct fwctl_info - ioctl(FWCTL_INFO)
+ * @size: sizeof(struct fwctl_info)
+ * @flags: Must be 0
+ * @out_device_type: Returns the type of the device from enum fwctl_device_type
+ * @device_data_len: On input the length of the out_device_data memory. On
+ * output the size of the kernel's device_data which may be larger or
+ * smaller than the input. Maybe 0 on input.
+ * @out_device_data: Pointer to a memory of device_data_len bytes. Kernel will
+ * fill the entire memory, zeroing as required.
+ *
+ * Returns basic information about this fwctl instance, particularly what driver
+ * is being used to define the device_data format.
+ */
+struct fwctl_info {
+ __u32 size;
+ __u32 flags;
+ __u32 out_device_type;
+ __u32 device_data_len;
+ __aligned_u64 out_device_data;
+};
+#define FWCTL_INFO _IO(FWCTL_TYPE, FWCTL_CMD_INFO)
+
#endif
--
2.43.0
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v5 4/8] taint: Add TAINT_FWCTL
2025-02-28 0:26 [PATCH v5 0/8] Introduce fwctl subystem Jason Gunthorpe
` (2 preceding siblings ...)
2025-02-28 0:26 ` [PATCH v5 3/8] fwctl: FWCTL_INFO to return basic information about the device Jason Gunthorpe
@ 2025-02-28 0:26 ` Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 5/8] fwctl: FWCTL_RPC to execute a Remote Procedure Call to device firmware Jason Gunthorpe
` (5 subsequent siblings)
9 siblings, 0 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-02-28 0:26 UTC (permalink / raw)
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
Requesting a fwctl scope of access that includes mutating device debug
data will cause the kernel to be tainted. Changing the device operation
through things in the debug scope may cause the device to malfunction in
undefined ways. This should be reflected in the TAINT flags to help any
debuggers understand that something has been done.
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Tested-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
Tested-by: Shannon Nelson <shannon.nelson@amd.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
Documentation/admin-guide/tainted-kernels.rst | 5 +++++
include/linux/panic.h | 3 ++-
kernel/panic.c | 1 +
tools/debugging/kernel-chktaint | 8 ++++++++
4 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/Documentation/admin-guide/tainted-kernels.rst b/Documentation/admin-guide/tainted-kernels.rst
index 700aa72eecb169..a0cc017e44246f 100644
--- a/Documentation/admin-guide/tainted-kernels.rst
+++ b/Documentation/admin-guide/tainted-kernels.rst
@@ -101,6 +101,7 @@ Bit Log Number Reason that got the kernel tainted
16 _/X 65536 auxiliary taint, defined for and used by distros
17 _/T 131072 kernel was built with the struct randomization plugin
18 _/N 262144 an in-kernel test has been run
+ 19 _/J 524288 userspace used a mutating debug operation in fwctl
=== === ====== ========================================================
Note: The character ``_`` is representing a blank in this table to make reading
@@ -184,3 +185,7 @@ More detailed explanation for tainting
build time.
18) ``N`` if an in-kernel test, such as a KUnit test, has been run.
+
+ 19) ``J`` if userpace opened /dev/fwctl/* and performed a FWTCL_RPC_DEBUG_WRITE
+ to use the devices debugging features. Device debugging features could
+ cause the device to malfunction in undefined ways.
diff --git a/include/linux/panic.h b/include/linux/panic.h
index 54d90b6c5f47bd..2494d51707ef42 100644
--- a/include/linux/panic.h
+++ b/include/linux/panic.h
@@ -74,7 +74,8 @@ static inline void set_arch_panic_timeout(int timeout, int arch_default_timeout)
#define TAINT_AUX 16
#define TAINT_RANDSTRUCT 17
#define TAINT_TEST 18
-#define TAINT_FLAGS_COUNT 19
+#define TAINT_FWCTL 19
+#define TAINT_FLAGS_COUNT 20
#define TAINT_FLAGS_MAX ((1UL << TAINT_FLAGS_COUNT) - 1)
struct taint_flag {
diff --git a/kernel/panic.c b/kernel/panic.c
index d8635d5cecb250..0c55eec9e8744a 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -511,6 +511,7 @@ const struct taint_flag taint_flags[TAINT_FLAGS_COUNT] = {
TAINT_FLAG(AUX, 'X', ' ', true),
TAINT_FLAG(RANDSTRUCT, 'T', ' ', true),
TAINT_FLAG(TEST, 'N', ' ', true),
+ TAINT_FLAG(FWCTL, 'J', ' ', true),
};
#undef TAINT_FLAG
diff --git a/tools/debugging/kernel-chktaint b/tools/debugging/kernel-chktaint
index 279be06332be99..e7da0909d09707 100755
--- a/tools/debugging/kernel-chktaint
+++ b/tools/debugging/kernel-chktaint
@@ -204,6 +204,14 @@ else
echo " * an in-kernel test (such as a KUnit test) has been run (#18)"
fi
+T=`expr $T / 2`
+if [ `expr $T % 2` -eq 0 ]; then
+ addout " "
+else
+ addout "J"
+ echo " * fwctl's mutating debug interface was used (#19)"
+fi
+
echo "For a more detailed explanation of the various taint flags see"
echo " Documentation/admin-guide/tainted-kernels.rst in the Linux kernel sources"
echo " or https://kernel.org/doc/html/latest/admin-guide/tainted-kernels.html"
--
2.43.0
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v5 5/8] fwctl: FWCTL_RPC to execute a Remote Procedure Call to device firmware
2025-02-28 0:26 [PATCH v5 0/8] Introduce fwctl subystem Jason Gunthorpe
` (3 preceding siblings ...)
2025-02-28 0:26 ` [PATCH v5 4/8] taint: Add TAINT_FWCTL Jason Gunthorpe
@ 2025-02-28 0:26 ` Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 6/8] fwctl: Add documentation Jason Gunthorpe
` (4 subsequent siblings)
9 siblings, 0 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-02-28 0:26 UTC (permalink / raw)
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
Add the FWCTL_RPC ioctl which allows a request/response RPC call to device
firmware. Drivers implementing this call must follow the security
guidelines under Documentation/userspace-api/fwctl.rst
The core code provides some memory management helpers to get the messages
copied from and back to userspace. The driver is responsible for
allocating the output message memory and delivering the message to the
device.
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Tested-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
Tested-by: Shannon Nelson <shannon.nelson@amd.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/fwctl/main.c | 60 +++++++++++++++++++++++++++++++++
include/linux/fwctl.h | 8 +++++
include/uapi/fwctl/fwctl.h | 69 ++++++++++++++++++++++++++++++++++++++
3 files changed, 137 insertions(+)
diff --git a/drivers/fwctl/main.c b/drivers/fwctl/main.c
index bd84c4e5b3b92e..cb1ac9c4023938 100644
--- a/drivers/fwctl/main.c
+++ b/drivers/fwctl/main.c
@@ -8,17 +8,20 @@
#include <linux/container_of.h>
#include <linux/fs.h>
#include <linux/module.h>
+#include <linux/sizes.h>
#include <linux/slab.h>
#include <uapi/fwctl/fwctl.h>
enum {
FWCTL_MAX_DEVICES = 4096,
+ MAX_RPC_LEN = SZ_2M,
};
static_assert(FWCTL_MAX_DEVICES < (1U << MINORBITS));
static dev_t fwctl_dev;
static DEFINE_IDA(fwctl_ida);
+static unsigned long fwctl_tainted;
struct fwctl_ucmd {
struct fwctl_uctx *uctx;
@@ -80,9 +83,65 @@ static int fwctl_cmd_info(struct fwctl_ucmd *ucmd)
return ucmd_respond(ucmd, sizeof(*cmd));
}
+static int fwctl_cmd_rpc(struct fwctl_ucmd *ucmd)
+{
+ struct fwctl_device *fwctl = ucmd->uctx->fwctl;
+ struct fwctl_rpc *cmd = ucmd->cmd;
+ size_t out_len;
+
+ if (cmd->in_len > MAX_RPC_LEN || cmd->out_len > MAX_RPC_LEN)
+ return -EMSGSIZE;
+
+ switch (cmd->scope) {
+ case FWCTL_RPC_CONFIGURATION:
+ case FWCTL_RPC_DEBUG_READ_ONLY:
+ break;
+
+ case FWCTL_RPC_DEBUG_WRITE_FULL:
+ if (!capable(CAP_SYS_RAWIO))
+ return -EPERM;
+ fallthrough;
+ case FWCTL_RPC_DEBUG_WRITE:
+ if (!test_and_set_bit(0, &fwctl_tainted)) {
+ dev_warn(
+ &fwctl->dev,
+ "%s(%d): has requested full access to the physical device device",
+ current->comm, task_pid_nr(current));
+ add_taint(TAINT_FWCTL, LOCKDEP_STILL_OK);
+ }
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ void *inbuf __free(kvfree) = kvzalloc(cmd->in_len, GFP_KERNEL_ACCOUNT);
+ if (!inbuf)
+ return -ENOMEM;
+ if (copy_from_user(inbuf, u64_to_user_ptr(cmd->in), cmd->in_len))
+ return -EFAULT;
+
+ out_len = cmd->out_len;
+ void *outbuf __free(kvfree) = fwctl->ops->fw_rpc(
+ ucmd->uctx, cmd->scope, inbuf, cmd->in_len, &out_len);
+ if (IS_ERR(outbuf))
+ return PTR_ERR(outbuf);
+ if (outbuf == inbuf) {
+ /* The driver can re-use inbuf as outbuf */
+ inbuf = NULL;
+ }
+
+ if (copy_to_user(u64_to_user_ptr(cmd->out), outbuf,
+ min(cmd->out_len, out_len)))
+ return -EFAULT;
+
+ cmd->out_len = out_len;
+ return ucmd_respond(ucmd, sizeof(*cmd));
+}
+
/* On stack memory for the ioctl structs */
union fwctl_ucmd_buffer {
struct fwctl_info info;
+ struct fwctl_rpc rpc;
};
struct fwctl_ioctl_op {
@@ -103,6 +162,7 @@ struct fwctl_ioctl_op {
}
static const struct fwctl_ioctl_op fwctl_ioctl_ops[] = {
IOCTL_OP(FWCTL_INFO, fwctl_cmd_info, struct fwctl_info, out_device_data),
+ IOCTL_OP(FWCTL_RPC, fwctl_cmd_rpc, struct fwctl_rpc, out),
};
static long fwctl_fops_ioctl(struct file *filp, unsigned int cmd,
diff --git a/include/linux/fwctl.h b/include/linux/fwctl.h
index 700a5be940e365..5d61fc8a687186 100644
--- a/include/linux/fwctl.h
+++ b/include/linux/fwctl.h
@@ -47,6 +47,14 @@ struct fwctl_ops {
* ignore length on input, the core code will handle everything.
*/
void *(*info)(struct fwctl_uctx *uctx, size_t *length);
+ /**
+ * @fw_rpc: Implement FWCTL_RPC. Deliver rpc_in/in_len to the FW and
+ * return the response and set out_len. rpc_in can be returned as the
+ * response pointer. Otherwise the returned pointer is freed with
+ * kvfree().
+ */
+ void *(*fw_rpc)(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
+ void *rpc_in, size_t in_len, size_t *out_len);
};
/**
diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
index 4052df63f66dc6..0bec798790a67a 100644
--- a/include/uapi/fwctl/fwctl.h
+++ b/include/uapi/fwctl/fwctl.h
@@ -37,6 +37,7 @@
enum {
FWCTL_CMD_BASE = 0,
FWCTL_CMD_INFO = 0,
+ FWCTL_CMD_RPC = 1,
};
enum fwctl_device_type {
@@ -66,4 +67,72 @@ struct fwctl_info {
};
#define FWCTL_INFO _IO(FWCTL_TYPE, FWCTL_CMD_INFO)
+/**
+ * enum fwctl_rpc_scope - Scope of access for the RPC
+ *
+ * Refer to fwctl.rst for a more detailed discussion of these scopes.
+ */
+enum fwctl_rpc_scope {
+ /**
+ * @FWCTL_RPC_CONFIGURATION: Device configuration access scope
+ *
+ * Read/write access to device configuration. When configuration
+ * is written to the device it remains in a fully supported state.
+ */
+ FWCTL_RPC_CONFIGURATION = 0,
+ /**
+ * @FWCTL_RPC_DEBUG_READ_ONLY: Read only access to debug information
+ *
+ * Readable debug information. Debug information is compatible with
+ * kernel lockdown, and does not disclose any sensitive information. For
+ * instance exposing any encryption secrets from this information is
+ * forbidden.
+ */
+ FWCTL_RPC_DEBUG_READ_ONLY = 1,
+ /**
+ * @FWCTL_RPC_DEBUG_WRITE: Writable access to lockdown compatible debug information
+ *
+ * Allows write access to data in the device which may leave a fully
+ * supported state. This is intended to permit intensive and possibly
+ * invasive debugging. This scope will taint the kernel.
+ */
+ FWCTL_RPC_DEBUG_WRITE = 2,
+ /**
+ * @FWCTL_RPC_DEBUG_WRITE_FULL: Write access to all debug information
+ *
+ * Allows read/write access to everything. Requires CAP_SYS_RAW_IO, so
+ * it is not required to follow lockdown principals. If in doubt
+ * debugging should be placed in this scope. This scope will taint the
+ * kernel.
+ */
+ FWCTL_RPC_DEBUG_WRITE_FULL = 3,
+};
+
+/**
+ * struct fwctl_rpc - ioctl(FWCTL_RPC)
+ * @size: sizeof(struct fwctl_rpc)
+ * @scope: One of enum fwctl_rpc_scope, required scope for the RPC
+ * @in_len: Length of the in memory
+ * @out_len: Length of the out memory
+ * @in: Request message in device specific format
+ * @out: Response message in device specific format
+ *
+ * Deliver a Remote Procedure Call to the device FW and return the response. The
+ * call's parameters and return are marshaled into linear buffers of memory. Any
+ * errno indicates that delivery of the RPC to the device failed. Return status
+ * originating in the device during a successful delivery must be encoded into
+ * out.
+ *
+ * The format of the buffers matches the out_device_type from FWCTL_INFO.
+ */
+struct fwctl_rpc {
+ __u32 size;
+ __u32 scope;
+ __u32 in_len;
+ __u32 out_len;
+ __aligned_u64 in;
+ __aligned_u64 out;
+};
+#define FWCTL_RPC _IO(FWCTL_TYPE, FWCTL_CMD_RPC)
+
#endif
--
2.43.0
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v5 6/8] fwctl: Add documentation
2025-02-28 0:26 [PATCH v5 0/8] Introduce fwctl subystem Jason Gunthorpe
` (4 preceding siblings ...)
2025-02-28 0:26 ` [PATCH v5 5/8] fwctl: FWCTL_RPC to execute a Remote Procedure Call to device firmware Jason Gunthorpe
@ 2025-02-28 0:26 ` Jason Gunthorpe
2025-03-15 2:53 ` Bagas Sanjaya
2025-02-28 0:26 ` [PATCH v5 7/8] fwctl/mlx5: Support for communicating with mlx5 fw Jason Gunthorpe
` (3 subsequent siblings)
9 siblings, 1 reply; 56+ messages in thread
From: Jason Gunthorpe @ 2025-02-28 0:26 UTC (permalink / raw)
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
Document the purpose and rules for the fwctl subsystem.
Link in kdocs to the doc tree.
Nacked-by: Jakub Kicinski <kuba@kernel.org>
Link: https://lore.kernel.org/r/20240603114250.5325279c@kernel.org
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
https://lore.kernel.org/r/ZrHY2Bds7oF7KRGz@phenom.ffwll.local
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
Documentation/userspace-api/fwctl/fwctl.rst | 284 ++++++++++++++++++++
Documentation/userspace-api/fwctl/index.rst | 12 +
Documentation/userspace-api/index.rst | 1 +
MAINTAINERS | 1 +
4 files changed, 298 insertions(+)
create mode 100644 Documentation/userspace-api/fwctl/fwctl.rst
create mode 100644 Documentation/userspace-api/fwctl/index.rst
diff --git a/Documentation/userspace-api/fwctl/fwctl.rst b/Documentation/userspace-api/fwctl/fwctl.rst
new file mode 100644
index 00000000000000..8c586a8f677d5b
--- /dev/null
+++ b/Documentation/userspace-api/fwctl/fwctl.rst
@@ -0,0 +1,284 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===============
+fwctl subsystem
+===============
+
+:Author: Jason Gunthorpe
+
+Overview
+========
+
+Modern devices contain extensive amounts of FW, and in many cases, are largely
+software-defined pieces of hardware. The evolution of this approach is largely a
+reaction to Moore's Law where a chip tape out is now highly expensive, and the
+chip design is extremely large. Replacing fixed HW logic with a flexible and
+tightly coupled FW/HW combination is an effective risk mitigation against chip
+respin. Problems in the HW design can be counteracted in device FW. This is
+especially true for devices which present a stable and backwards compatible
+interface to the operating system driver (such as NVMe).
+
+The FW layer in devices has grown to incredible size and devices frequently
+integrate clusters of fast processors to run it. For example, mlx5 devices have
+over 30MB of FW code, and big configurations operate with over 1GB of FW managed
+runtime state.
+
+The availability of such a flexible layer has created quite a variety in the
+industry where single pieces of silicon are now configurable software-defined
+devices and can operate in substantially different ways depending on the need.
+Further, we often see cases where specific sites wish to operate devices in ways
+that are highly specialized and require applications that have been tailored to
+their unique configuration.
+
+Further, devices have become multi-functional and integrated to the point they
+no longer fit neatly into the kernel's division of subsystems. Modern
+multi-functional devices have drivers, such as bnxt/ice/mlx5/pds, that span many
+subsystems while sharing the underlying hardware using the auxiliary device
+system.
+
+All together this creates a challenge for the operating system, where devices
+have an expansive FW environment that needs robust device-specific debugging
+support, and FW-driven functionality that is not well suited to “generic”
+interfaces. fwctl seeks to allow access to the full device functionality from
+user space in the areas of debuggability, management, and first-boot/nth-boot
+provisioning.
+
+fwctl is aimed at the common device design pattern where the OS and FW
+communicate via an RPC message layer constructed with a queue or mailbox scheme.
+In this case the driver will typically have some layer to deliver RPC messages
+and collect RPC responses from device FW. The in-kernel subsystem drivers that
+operate the device for its primary purposes will use these RPCs to build their
+drivers, but devices also usually have a set of ancillary RPCs that don't really
+fit into any specific subsystem. For example, a HW RAID controller is primarily
+operated by the block layer but also comes with a set of RPCs to administer the
+construction of drives within the HW RAID.
+
+In the past when devices were more single function, individual subsystems would
+grow different approaches to solving some of these common problems. For instance
+monitoring device health, manipulating its FLASH, debugging the FW,
+provisioning, all have various unique interfaces across the kernel.
+
+fwctl's purpose is to define a common set of limited rules, described below,
+that allow user space to securely construct and execute RPCs inside device FW.
+The rules serve as an agreement between the operating system and FW on how to
+correctly design the RPC interface. As a uAPI the subsystem provides a thin
+layer of discovery and a generic uAPI to deliver the RPCs and collect the
+response. It supports a system of user space libraries and tools which will
+use this interface to control the device using the device native protocols.
+
+Scope of Action
+---------------
+
+fwctl drivers are strictly restricted to being a way to operate the device FW.
+It is not an avenue to access random kernel internals, or other operating system
+SW states.
+
+fwctl instances must operate on a well-defined device function, and the device
+should have a well-defined security model for what scope within the physical
+device the function is permitted to access. For instance, the most complex PCIe
+device today may broadly have several function-level scopes:
+
+ 1. A privileged function with full access to the on-device global state and
+ configuration
+
+ 2. Multiple hypervisor functions with control over itself and child functions
+ used with VMs
+
+ 3. Multiple VM functions tightly scoped within the VM
+
+The device may create a logical parent/child relationship between these scopes.
+For instance a child VM's FW may be within the scope of the hypervisor FW. It is
+quite common in the VFIO world that the hypervisor environment has a complex
+provisioning/profiling/configuration responsibility for the function VFIO
+assigns to the VM.
+
+Further, within the function, devices often have RPC commands that fall within
+some general scopes of action (see enum fwctl_rpc_scope):
+
+ 1. Access to function & child configuration, FLASH, etc. that becomes live at a
+ function reset. Access to function & child runtime configuration that is
+ transparent or non-disruptive to any driver or VM.
+
+ 2. Read-only access to function debug information that may report on FW objects
+ in the function & child, including FW objects owned by other kernel
+ subsystems.
+
+ 3. Write access to function & child debug information strictly compatible with
+ the principles of kernel lockdown and kernel integrity protection. Triggers
+ a kernel Taint.
+
+ 4. Full debug device access. Triggers a kernel Taint, requires CAP_SYS_RAWIO.
+
+User space will provide a scope label on each RPC and the kernel must enforce the
+above CAPs and taints based on that scope. A combination of kernel and FW can
+enforce that RPCs are placed in the correct scope by user space.
+
+Denied behavior
+---------------
+
+There are many things this interface must not allow user space to do (without a
+Taint or CAP), broadly derived from the principles of kernel lockdown. Some
+examples:
+
+ 1. DMA to/from arbitrary memory, hang the system, compromise FW integrity with
+ untrusted code, or otherwise compromise device or system security and
+ integrity.
+
+ 2. Provide an abnormal “back door” to kernel drivers. No manipulation of kernel
+ objects owned by kernel drivers.
+
+ 3. Directly configure or otherwise control kernel drivers. A subsystem kernel
+ driver can react to the device configuration at function reset/driver load
+ time, but otherwise must not be coupled to fwctl.
+
+ 4. Operate the HW in a way that overlaps with the core purpose of another
+ primary kernel subsystem, such as read/write to LBAs, send/receive of
+ network packets, or operate an accelerator's data plane.
+
+fwctl is not a replacement for device direct access subsystems like uacce or
+VFIO.
+
+Operations exposed through fwctl's non-taining interfaces should be fully
+sharable with other users of the device. For instance exposing a RPC through
+fwctl should never prevent a kernel subsystem from also concurrently using that
+same RPC or hardware unit down the road. In such cases fwctl will be less
+important than proper kernel subsystems that eventually emerge. Mistakes in this
+area resulting in clashes will be resolved in favour of a kernel implementation.
+
+fwctl User API
+==============
+
+.. kernel-doc:: include/uapi/fwctl/fwctl.h
+
+sysfs Class
+-----------
+
+fwctl has a sysfs class (/sys/class/fwctl/fwctlNN/) and character devices
+(/dev/fwctl/fwctlNN) with a simple numbered scheme. The character device
+operates the iotcl uAPI described above.
+
+fwctl devices can be related to driver components in other subsystems through
+sysfs::
+
+ $ ls /sys/class/fwctl/fwctl0/device/infiniband/
+ ibp0s10f0
+
+ $ ls /sys/class/infiniband/ibp0s10f0/device/fwctl/
+ fwctl0/
+
+ $ ls /sys/devices/pci0000:00/0000:00:0a.0/fwctl/fwctl0
+ dev device power subsystem uevent
+
+User space Community
+--------------------
+
+Drawing inspiration from nvme-cli, participating in the kernel side must come
+with a user space in a common TBD git tree, at a minimum to usefully operate the
+kernel driver. Providing such an implementation is a pre-condition to merging a
+kernel driver.
+
+The goal is to build user space community around some of the shared problems
+we all have, and ideally develop some common user space programs with some
+starting themes of:
+
+ - Device in-field debugging
+
+ - HW provisioning
+
+ - VFIO child device profiling before VM boot
+
+ - Confidential Compute topics (attestation, secure provisioning)
+
+that stretch across all subsystems in the kernel. fwupd is a great example of
+how an excellent user space experience can emerge out of kernel-side diversity.
+
+fwctl Kernel API
+================
+
+.. kernel-doc:: drivers/fwctl/main.c
+ :export:
+.. kernel-doc:: include/linux/fwctl.h
+
+fwctl Driver design
+-------------------
+
+In many cases a fwctl driver is going to be part of a larger cross-subsystem
+device possibly using the auxiliary_device mechanism. In that case several
+subsystems are going to be sharing the same device and FW interface layer so the
+device design must already provide for isolation and cooperation between kernel
+subsystems. fwctl should fit into that same model.
+
+Part of the driver should include a description of how its scope restrictions
+and security model work. The driver and FW together must ensure that RPCs
+provided by user space are mapped to the appropriate scope. If the validation is
+done in the driver then the validation can read a 'command effects' report from
+the device, or hardwire the enforcement. If the validation is done in the FW,
+then the driver should pass the fwctl_rpc_scope to the FW along with the command.
+
+The driver and FW must cooperate to ensure that either fwctl cannot allocate
+any FW resources, or any resources it does allocate are freed on FD closure. A
+driver primarily constructed around FW RPCs may find that its core PCI function
+and RPC layer belongs under fwctl with auxiliary devices connecting to other
+subsystems.
+
+Each device type must be mindful of Linux's philosophy for stable ABI. The FW
+RPC interface does not have to meet a strictly stable ABI, but it does need to
+meet an expectation that userspace tools that are deployed and in significant
+use don't needlessly break. FW upgrade and kernel upgrade should keep widely
+deployed tooling working.
+
+Development and debugging focused RPCs under more permissive scopes can have
+less stabilitiy if the tools using them are only run under exceptional
+circumstances and not for every day use of the device. Debugging tools may even
+require exact version matching as they may require something similar to DWARF
+debug information from the FW binary.
+
+Security Response
+=================
+
+The kernel remains the gatekeeper for this interface. If violations of the
+scopes, security or isolation principles are found, we have options to let
+devices fix them with a FW update, push a kernel patch to parse and block RPC
+commands or push a kernel patch to block entire firmware versions/devices.
+
+While the kernel can always directly parse and restrict RPCs, it is expected
+that the existing kernel pattern of allowing drivers to delegate validation to
+FW to be a useful design.
+
+Existing Similar Examples
+=========================
+
+The approach described in this document is not a new idea. Direct, or near
+direct device access has been offered by the kernel in different areas for
+decades. With more devices wanting to follow this design pattern it is becoming
+clear that it is not entirely well understood and, more importantly, the
+security considerations are not well defined or agreed upon.
+
+Some examples:
+
+ - HW RAID controllers. This includes RPCs to do things like compose drives into
+ a RAID volume, configure RAID parameters, monitor the HW and more.
+
+ - Baseboard managers. RPCs for configuring settings in the device and more
+
+ - NVMe vendor command capsules. nvme-cli provides access to some monitoring
+ functions that different products have defined, but more exist.
+
+ - CXL also has a NVMe-like vendor command system.
+
+ - DRM allows user space drivers to send commands to the device via kernel
+ mediation
+
+ - RDMA allows user space drivers to directly push commands to the device
+ without kernel involvement
+
+ - Various “raw” APIs, raw HID (SDL2), raw USB, NVMe Generic Interface, etc.
+
+The first 4 are examples of areas that fwctl intends to cover. The latter three
+are examples of denied behavior as they fully overlap with the primary purpose
+of a kernel subsystem.
+
+Some key lessons learned from these past efforts are the importance of having a
+common user space project to use as a pre-condition for obtaining a kernel
+driver. Developing good community around useful software in user space is key to
+getting companies to fund participation to enable their products.
diff --git a/Documentation/userspace-api/fwctl/index.rst b/Documentation/userspace-api/fwctl/index.rst
new file mode 100644
index 00000000000000..06959fbf154743
--- /dev/null
+++ b/Documentation/userspace-api/fwctl/index.rst
@@ -0,0 +1,12 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Firmware Control (FWCTL) Userspace API
+======================================
+
+A framework that define a common set of limited rules that allows user space
+to securely construct and execute RPCs inside device firmware.
+
+.. toctree::
+ :maxdepth: 1
+
+ fwctl
diff --git a/Documentation/userspace-api/index.rst b/Documentation/userspace-api/index.rst
index b1395d94b3fd0a..e8e861f767fd5c 100644
--- a/Documentation/userspace-api/index.rst
+++ b/Documentation/userspace-api/index.rst
@@ -45,6 +45,7 @@ Devices and I/O
accelerators/ocxl
dma-buf-alloc-exchange
+ fwctl/index
gpio/index
iommufd
media/index
diff --git a/MAINTAINERS b/MAINTAINERS
index 1bc1f97934b6b8..319169f7cb7e1c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9561,6 +9561,7 @@ FWCTL SUBSYSTEM
M: Jason Gunthorpe <jgg@nvidia.com>
M: Saeed Mahameed <saeedm@nvidia.com>
S: Maintained
+F: Documentation/userspace-api/fwctl/
F: drivers/fwctl/
F: include/linux/fwctl.h
F: include/uapi/fwctl/
--
2.43.0
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v5 7/8] fwctl/mlx5: Support for communicating with mlx5 fw
2025-02-28 0:26 [PATCH v5 0/8] Introduce fwctl subystem Jason Gunthorpe
` (5 preceding siblings ...)
2025-02-28 0:26 ` [PATCH v5 6/8] fwctl: Add documentation Jason Gunthorpe
@ 2025-02-28 0:26 ` Jason Gunthorpe
2025-03-02 12:11 ` Leon Romanovsky
2025-02-28 0:26 ` [PATCH v5 8/8] mlx5: Create an auxiliary device for fwctl_mlx5 Jason Gunthorpe
` (2 subsequent siblings)
9 siblings, 1 reply; 56+ messages in thread
From: Jason Gunthorpe @ 2025-02-28 0:26 UTC (permalink / raw)
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
From: Saeed Mahameed <saeedm@nvidia.com>
mlx5 FW has a built in security context called UID. Each UID has a set of
permissions controlled by the kernel when it is created and every command
is tagged by the kernel with a particular UID. In general commands cannot
reach objects outside of their UID and commands cannot exceed their UID's
permissions. These restrictions are enforced by FW.
This mechanism has long been used in RDMA for the devx interface where
RDMA will sent commands directly to the FW and the UID limitations
restrict those commands to a ib_device/verbs security domain. For instance
commands that would effect other VFs, or global device resources. The
model is suitable for unprivileged userspace to operate the RDMA
functionality.
The UID has been extended with a "tools resources" permission which allows
additional commands and sub-commands that are intended to match with the
scope limitations set in FWCTL. This is an alternative design to the
"command intent log" where the FW does the enforcement rather than having
the FW report the enforcement the kernel should do.
Consistent with the fwctl definitions the "tools resources" security
context is limited to the FWCTL_RPC_CONFIGURATION,
FWCTL_RPC_DEBUG_READ_ONLY, FWCTL_RPC_DEBUG_WRITE, and
FWCTL_RPC_DEBUG_WRITE_FULL security scopes.
Like RDMA devx, each opened fwctl file descriptor will get a unique UID
associated with each file descriptor.
The fwctl driver is kept simple and we reject commands that can create
objects as the UID mechanism relies on the kernel to track and destroy
objects prior to detroying the UID. Filtering into fwctl sub scopes is
done inside the driver with a switch statement. This substantially limits
what is possible to primarily query functions ad a few limited set
operations.
mlx5 already has a robust infrastructure for delivering RPC messages to
fw. Trivially connect fwctl's RPC mechanism to mlx5_cmd_do(). Enforce the
User Context ID in every RPC header accepted from the FD so the FW knows
the security context of the issuing ID.
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
Documentation/userspace-api/fwctl/fwctl.rst | 1 +
MAINTAINERS | 9 +
drivers/fwctl/Kconfig | 14 +
drivers/fwctl/Makefile | 1 +
drivers/fwctl/mlx5/Makefile | 4 +
drivers/fwctl/mlx5/main.c | 411 ++++++++++++++++++++
include/uapi/fwctl/fwctl.h | 1 +
include/uapi/fwctl/mlx5.h | 36 ++
8 files changed, 477 insertions(+)
create mode 100644 drivers/fwctl/mlx5/Makefile
create mode 100644 drivers/fwctl/mlx5/main.c
create mode 100644 include/uapi/fwctl/mlx5.h
diff --git a/Documentation/userspace-api/fwctl/fwctl.rst b/Documentation/userspace-api/fwctl/fwctl.rst
index 8c586a8f677d5b..04ad78a7cd48bb 100644
--- a/Documentation/userspace-api/fwctl/fwctl.rst
+++ b/Documentation/userspace-api/fwctl/fwctl.rst
@@ -149,6 +149,7 @@ fwctl User API
==============
.. kernel-doc:: include/uapi/fwctl/fwctl.h
+.. kernel-doc:: include/uapi/fwctl/mlx5.h
sysfs Class
-----------
diff --git a/MAINTAINERS b/MAINTAINERS
index 319169f7cb7e1c..93e64a54c56f37 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9558,14 +9558,23 @@ F: tools/perf/bench/futex*
F: tools/testing/selftests/futex/
FWCTL SUBSYSTEM
+M: Dave Jiang <dave.jiang@intel.com>
M: Jason Gunthorpe <jgg@nvidia.com>
M: Saeed Mahameed <saeedm@nvidia.com>
+R: Jonathan Cameron <Jonathan.Cameron@huawei.com>
S: Maintained
F: Documentation/userspace-api/fwctl/
F: drivers/fwctl/
F: include/linux/fwctl.h
F: include/uapi/fwctl/
+FWCTL MLX5 DRIVER
+M: Saeed Mahameed <saeedm@nvidia.com>
+R: Itay Avraham <itayavr@nvidia.com>
+L: linux-kernel@vger.kernel.org
+S: Maintained
+F: drivers/fwctl/mlx5/
+
GALAXYCORE GC0308 CAMERA SENSOR DRIVER
M: Sebastian Reichel <sre@kernel.org>
L: linux-media@vger.kernel.org
diff --git a/drivers/fwctl/Kconfig b/drivers/fwctl/Kconfig
index 37147a695add9a..f802cf5d4951e8 100644
--- a/drivers/fwctl/Kconfig
+++ b/drivers/fwctl/Kconfig
@@ -7,3 +7,17 @@ menuconfig FWCTL
support a wide range of lockdown compatible device behaviors including
manipulating device FLASH, debugging, and other activities that don't
fit neatly into an existing subsystem.
+
+if FWCTL
+config FWCTL_MLX5
+ tristate "mlx5 ConnectX control fwctl driver"
+ depends on MLX5_CORE
+ help
+ MLX5 provides interface for the user process to access the debug and
+ configuration registers of the ConnectX hardware family
+ (NICs, PCI switches and SmartNIC SoCs).
+ This will allow configuration and debug tools to work out of the box on
+ mainstream kernel.
+
+ If you don't know what to do here, say N.
+endif
diff --git a/drivers/fwctl/Makefile b/drivers/fwctl/Makefile
index 1cad210f6ba580..1c535f694d7fe4 100644
--- a/drivers/fwctl/Makefile
+++ b/drivers/fwctl/Makefile
@@ -1,4 +1,5 @@
# SPDX-License-Identifier: GPL-2.0
obj-$(CONFIG_FWCTL) += fwctl.o
+obj-$(CONFIG_FWCTL_MLX5) += mlx5/
fwctl-y += main.o
diff --git a/drivers/fwctl/mlx5/Makefile b/drivers/fwctl/mlx5/Makefile
new file mode 100644
index 00000000000000..139a23e3c7c517
--- /dev/null
+++ b/drivers/fwctl/mlx5/Makefile
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_FWCTL_MLX5) += mlx5_fwctl.o
+
+mlx5_fwctl-y += main.o
diff --git a/drivers/fwctl/mlx5/main.c b/drivers/fwctl/mlx5/main.c
new file mode 100644
index 00000000000000..f93aa0cecdb978
--- /dev/null
+++ b/drivers/fwctl/mlx5/main.c
@@ -0,0 +1,411 @@
+// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
+/*
+ * Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES
+ */
+#include <linux/fwctl.h>
+#include <linux/auxiliary_bus.h>
+#include <linux/mlx5/device.h>
+#include <linux/mlx5/driver.h>
+#include <uapi/fwctl/mlx5.h>
+
+#define mlx5ctl_err(mcdev, format, ...) \
+ dev_err(&mcdev->fwctl.dev, format, ##__VA_ARGS__)
+
+#define mlx5ctl_dbg(mcdev, format, ...) \
+ dev_dbg(&mcdev->fwctl.dev, "PID %u: " format, current->pid, \
+ ##__VA_ARGS__)
+
+struct mlx5ctl_uctx {
+ struct fwctl_uctx uctx;
+ u32 uctx_caps;
+ u32 uctx_uid;
+};
+
+struct mlx5ctl_dev {
+ struct fwctl_device fwctl;
+ struct mlx5_core_dev *mdev;
+};
+DEFINE_FREE(mlx5ctl, struct mlx5ctl_dev *, if (_T) fwctl_put(&_T->fwctl));
+
+struct mlx5_ifc_mbox_in_hdr_bits {
+ u8 opcode[0x10];
+ u8 uid[0x10];
+
+ u8 reserved_at_20[0x10];
+ u8 op_mod[0x10];
+
+ u8 reserved_at_40[0x40];
+};
+
+struct mlx5_ifc_mbox_out_hdr_bits {
+ u8 status[0x8];
+ u8 reserved_at_8[0x18];
+
+ u8 syndrome[0x20];
+
+ u8 reserved_at_40[0x40];
+};
+
+enum {
+ MLX5_UCTX_OBJECT_CAP_TOOLS_RESOURCES = 0x4,
+};
+
+enum {
+ MLX5_CMD_OP_QUERY_DRIVER_VERSION = 0x10c,
+ MLX5_CMD_OP_QUERY_OTHER_HCA_CAP = 0x10e,
+ MLX5_CMD_OP_QUERY_RDB = 0x512,
+ MLX5_CMD_OP_QUERY_PSV = 0x602,
+ MLX5_CMD_OP_QUERY_DC_CNAK_TRACE = 0x716,
+ MLX5_CMD_OP_QUERY_NVMF_BACKEND_CONTROLLER = 0x722,
+ MLX5_CMD_OP_QUERY_NVMF_NAMESPACE_CONTEXT = 0x728,
+ MLX5_CMD_OP_QUERY_BURST_SIZE = 0x813,
+ MLX5_CMD_OP_QUERY_DIAGNOSTIC_PARAMS = 0x819,
+ MLX5_CMD_OP_SET_DIAGNOSTIC_PARAMS = 0x820,
+ MLX5_CMD_OP_QUERY_DIAGNOSTIC_COUNTERS = 0x821,
+ MLX5_CMD_OP_QUERY_DELAY_DROP_PARAMS = 0x911,
+ MLX5_CMD_OP_QUERY_AFU = 0x971,
+ MLX5_CMD_OP_QUERY_CAPI_PEC = 0x981,
+ MLX5_CMD_OP_QUERY_UCTX = 0xa05,
+ MLX5_CMD_OP_QUERY_UMEM = 0xa09,
+ MLX5_CMD_OP_QUERY_NVMF_CC_RESPONSE = 0xb02,
+ MLX5_CMD_OP_QUERY_EMULATED_FUNCTIONS_INFO = 0xb03,
+ MLX5_CMD_OP_QUERY_REGEXP_PARAMS = 0xb05,
+ MLX5_CMD_OP_QUERY_REGEXP_REGISTER = 0xb07,
+ MLX5_CMD_OP_USER_QUERY_XRQ_DC_PARAMS_ENTRY = 0xb08,
+ MLX5_CMD_OP_USER_QUERY_XRQ_ERROR_PARAMS = 0xb0a,
+ MLX5_CMD_OP_ACCESS_REGISTER_USER = 0xb0c,
+ MLX5_CMD_OP_QUERY_EMULATION_DEVICE_EQ_MSIX_MAPPING = 0xb0f,
+ MLX5_CMD_OP_QUERY_MATCH_SAMPLE_INFO = 0xb13,
+ MLX5_CMD_OP_QUERY_CRYPTO_STATE = 0xb14,
+ MLX5_CMD_OP_QUERY_VUID = 0xb22,
+ MLX5_CMD_OP_QUERY_DPA_PARTITION = 0xb28,
+ MLX5_CMD_OP_QUERY_DPA_PARTITIONS = 0xb2a,
+ MLX5_CMD_OP_POSTPONE_CONNECTED_QP_TIMEOUT = 0xb2e,
+ MLX5_CMD_OP_QUERY_EMULATED_RESOURCES_INFO = 0xb2f,
+ MLX5_CMD_OP_QUERY_RSV_RESOURCES = 0x8000,
+ MLX5_CMD_OP_QUERY_MTT = 0x8001,
+ MLX5_CMD_OP_QUERY_SCHED_QUEUE = 0x8006,
+};
+
+static int mlx5ctl_alloc_uid(struct mlx5ctl_dev *mcdev, u32 cap)
+{
+ u32 out[MLX5_ST_SZ_DW(create_uctx_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(create_uctx_in)] = {};
+ void *uctx;
+ int ret;
+ u16 uid;
+
+ uctx = MLX5_ADDR_OF(create_uctx_in, in, uctx);
+
+ mlx5ctl_dbg(mcdev, "%s: caps 0x%x\n", __func__, cap);
+ MLX5_SET(create_uctx_in, in, opcode, MLX5_CMD_OP_CREATE_UCTX);
+ MLX5_SET(uctx, uctx, cap, cap);
+
+ ret = mlx5_cmd_exec(mcdev->mdev, in, sizeof(in), out, sizeof(out));
+ if (ret)
+ return ret;
+
+ uid = MLX5_GET(create_uctx_out, out, uid);
+ mlx5ctl_dbg(mcdev, "allocated uid %u with caps 0x%x\n", uid, cap);
+ return uid;
+}
+
+static void mlx5ctl_release_uid(struct mlx5ctl_dev *mcdev, u16 uid)
+{
+ u32 in[MLX5_ST_SZ_DW(destroy_uctx_in)] = {};
+ struct mlx5_core_dev *mdev = mcdev->mdev;
+ int ret;
+
+ MLX5_SET(destroy_uctx_in, in, opcode, MLX5_CMD_OP_DESTROY_UCTX);
+ MLX5_SET(destroy_uctx_in, in, uid, uid);
+
+ ret = mlx5_cmd_exec_in(mdev, destroy_uctx, in);
+ mlx5ctl_dbg(mcdev, "released uid %u %pe\n", uid, ERR_PTR(ret));
+}
+
+static int mlx5ctl_open_uctx(struct fwctl_uctx *uctx)
+{
+ struct mlx5ctl_uctx *mfd =
+ container_of(uctx, struct mlx5ctl_uctx, uctx);
+ struct mlx5ctl_dev *mcdev =
+ container_of(uctx->fwctl, struct mlx5ctl_dev, fwctl);
+ int uid;
+
+ /*
+ * New FW supports the TOOLS_RESOURCES uid security label
+ * which allows commands to manipulate the global device state.
+ * Otherwise only basic existing RDMA devx privilege are allowed.
+ */
+ if (MLX5_CAP_GEN(mcdev->mdev, uctx_cap) &
+ MLX5_UCTX_OBJECT_CAP_TOOLS_RESOURCES)
+ mfd->uctx_caps |= MLX5_UCTX_OBJECT_CAP_TOOLS_RESOURCES;
+
+ uid = mlx5ctl_alloc_uid(mcdev, mfd->uctx_caps);
+ if (uid < 0)
+ return uid;
+
+ mfd->uctx_uid = uid;
+ return 0;
+}
+
+static void mlx5ctl_close_uctx(struct fwctl_uctx *uctx)
+{
+ struct mlx5ctl_dev *mcdev =
+ container_of(uctx->fwctl, struct mlx5ctl_dev, fwctl);
+ struct mlx5ctl_uctx *mfd =
+ container_of(uctx, struct mlx5ctl_uctx, uctx);
+
+ mlx5ctl_release_uid(mcdev, mfd->uctx_uid);
+}
+
+static void *mlx5ctl_info(struct fwctl_uctx *uctx, size_t *length)
+{
+ struct mlx5ctl_uctx *mfd =
+ container_of(uctx, struct mlx5ctl_uctx, uctx);
+ struct fwctl_info_mlx5 *info;
+
+ info = kzalloc(sizeof(*info), GFP_KERNEL);
+ if (!info)
+ return ERR_PTR(-ENOMEM);
+
+ info->uid = mfd->uctx_uid;
+ info->uctx_caps = mfd->uctx_caps;
+ *length = sizeof(*info);
+ return info;
+}
+
+static bool mlx5ctl_validate_rpc(const void *in, enum fwctl_rpc_scope scope)
+{
+ u16 opcode = MLX5_GET(mbox_in_hdr, in, opcode);
+ u16 op_mod = MLX5_GET(mbox_in_hdr, in, op_mod);
+
+ /*
+ * Currently the driver can't keep track of commands that allocate
+ * objects in the FW, these commands are safe from a security
+ * perspective but nothing will free the memory when the FD is closed.
+ * For now permit only query commands and set commands that don't alter
+ * objects. Also the caps for the scope have not been defined yet,
+ * filter commands manually for now.
+ */
+ switch (opcode) {
+ case MLX5_CMD_OP_POSTPONE_CONNECTED_QP_TIMEOUT:
+ case MLX5_CMD_OP_QUERY_ADAPTER:
+ case MLX5_CMD_OP_QUERY_ESW_FUNCTIONS:
+ case MLX5_CMD_OP_QUERY_HCA_CAP:
+ case MLX5_CMD_OP_QUERY_HCA_VPORT_CONTEXT:
+ case MLX5_CMD_OP_QUERY_OTHER_HCA_CAP:
+ case MLX5_CMD_OP_QUERY_ROCE_ADDRESS:
+ case MLX5_CMD_OPCODE_QUERY_VUID:
+ /*
+ * FW limits SET_HCA_CAP on the tools UID to only the other function
+ * mode which is used for function pre-configuration
+ */
+ case MLX5_CMD_OP_SET_HCA_CAP:
+ return true; /* scope >= FWCTL_RPC_CONFIGURATION; */
+
+ case MLX5_CMD_OP_FPGA_QUERY_QP_COUNTERS:
+ case MLX5_CMD_OP_FPGA_QUERY_QP:
+ case MLX5_CMD_OP_NOP:
+ case MLX5_CMD_OP_QUERY_AFU:
+ case MLX5_CMD_OP_QUERY_BURST_SIZE:
+ case MLX5_CMD_OP_QUERY_CAPI_PEC:
+ case MLX5_CMD_OP_QUERY_CONG_PARAMS:
+ case MLX5_CMD_OP_QUERY_CONG_STATISTICS:
+ case MLX5_CMD_OP_QUERY_CONG_STATUS:
+ case MLX5_CMD_OP_QUERY_CQ:
+ case MLX5_CMD_OP_QUERY_CRYPTO_STATE:
+ case MLX5_CMD_OP_QUERY_DC_CNAK_TRACE:
+ case MLX5_CMD_OP_QUERY_DCT:
+ case MLX5_CMD_OP_QUERY_DELAY_DROP_PARAMS:
+ case MLX5_CMD_OP_QUERY_DIAGNOSTIC_COUNTERS:
+ case MLX5_CMD_OP_QUERY_DIAGNOSTIC_PARAMS:
+ case MLX5_CMD_OP_QUERY_DPA_PARTITION:
+ case MLX5_CMD_OP_QUERY_DPA_PARTITIONS:
+ case MLX5_CMD_OP_QUERY_DRIVER_VERSION:
+ case MLX5_CMD_OP_QUERY_EMULATED_FUNCTIONS_INFO:
+ case MLX5_CMD_OP_QUERY_EMULATED_RESOURCES_INFO:
+ case MLX5_CMD_OP_QUERY_EMULATION_DEVICE_EQ_MSIX_MAPPING:
+ case MLX5_CMD_OP_QUERY_EQ:
+ case MLX5_CMD_OP_QUERY_ESW_VPORT_CONTEXT:
+ case MLX5_CMD_OP_QUERY_FLOW_COUNTER:
+ case MLX5_CMD_OP_QUERY_FLOW_GROUP:
+ case MLX5_CMD_OP_QUERY_FLOW_TABLE_ENTRY:
+ case MLX5_CMD_OP_QUERY_FLOW_TABLE:
+ case MLX5_CMD_OP_QUERY_GENERAL_OBJECT:
+ case MLX5_CMD_OP_QUERY_HCA_VPORT_GID:
+ case MLX5_CMD_OP_QUERY_HCA_VPORT_PKEY:
+ case MLX5_CMD_OP_QUERY_ISSI:
+ case MLX5_CMD_OP_QUERY_L2_TABLE_ENTRY:
+ case MLX5_CMD_OP_QUERY_LAG:
+ case MLX5_CMD_OP_QUERY_MAD_DEMUX:
+ case MLX5_CMD_OP_QUERY_MATCH_SAMPLE_INFO:
+ case MLX5_CMD_OP_QUERY_MKEY:
+ case MLX5_CMD_OP_QUERY_MODIFY_HEADER_CONTEXT:
+ case MLX5_CMD_OP_QUERY_MTT:
+ case MLX5_CMD_OP_QUERY_NIC_VPORT_CONTEXT:
+ case MLX5_CMD_OP_QUERY_NVMF_BACKEND_CONTROLLER:
+ case MLX5_CMD_OP_QUERY_NVMF_CC_RESPONSE:
+ case MLX5_CMD_OP_QUERY_NVMF_NAMESPACE_CONTEXT:
+ case MLX5_CMD_OP_QUERY_PACKET_REFORMAT_CONTEXT:
+ case MLX5_CMD_OP_QUERY_PAGES:
+ case MLX5_CMD_OP_QUERY_PSV:
+ case MLX5_CMD_OP_QUERY_Q_COUNTER:
+ case MLX5_CMD_OP_QUERY_QP:
+ case MLX5_CMD_OP_QUERY_RATE_LIMIT:
+ case MLX5_CMD_OP_QUERY_RDB:
+ case MLX5_CMD_OP_QUERY_REGEXP_PARAMS:
+ case MLX5_CMD_OP_QUERY_REGEXP_REGISTER:
+ case MLX5_CMD_OP_QUERY_RMP:
+ case MLX5_CMD_OP_QUERY_RQ:
+ case MLX5_CMD_OP_QUERY_RQT:
+ case MLX5_CMD_OP_QUERY_RSV_RESOURCES:
+ case MLX5_CMD_OP_QUERY_SCHED_QUEUE:
+ case MLX5_CMD_OP_QUERY_SCHEDULING_ELEMENT:
+ case MLX5_CMD_OP_QUERY_SF_PARTITION:
+ case MLX5_CMD_OP_QUERY_SPECIAL_CONTEXTS:
+ case MLX5_CMD_OP_QUERY_SQ:
+ case MLX5_CMD_OP_QUERY_SRQ:
+ case MLX5_CMD_OP_QUERY_TIR:
+ case MLX5_CMD_OP_QUERY_TIS:
+ case MLX5_CMD_OP_QUERY_UCTX:
+ case MLX5_CMD_OP_QUERY_UMEM:
+ case MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE:
+ case MLX5_CMD_OP_QUERY_VHCA_STATE:
+ case MLX5_CMD_OP_QUERY_VNIC_ENV:
+ case MLX5_CMD_OP_QUERY_VPORT_COUNTER:
+ case MLX5_CMD_OP_QUERY_VPORT_STATE:
+ case MLX5_CMD_OP_QUERY_WOL_ROL:
+ case MLX5_CMD_OP_QUERY_XRC_SRQ:
+ case MLX5_CMD_OP_QUERY_XRQ_DC_PARAMS_ENTRY:
+ case MLX5_CMD_OP_QUERY_XRQ_ERROR_PARAMS:
+ case MLX5_CMD_OP_QUERY_XRQ:
+ case MLX5_CMD_OP_USER_QUERY_XRQ_DC_PARAMS_ENTRY:
+ case MLX5_CMD_OP_USER_QUERY_XRQ_ERROR_PARAMS:
+ return scope >= FWCTL_RPC_DEBUG_READ_ONLY;
+
+ case MLX5_CMD_OP_SET_DIAGNOSTIC_PARAMS:
+ return scope >= FWCTL_RPC_DEBUG_WRITE;
+
+ case MLX5_CMD_OP_ACCESS_REG:
+ case MLX5_CMD_OP_ACCESS_REGISTER_USER:
+ if (op_mod == 0) /* write */
+ return true; /* scope >= FWCTL_RPC_CONFIGURATION; */
+ return scope >= FWCTL_RPC_DEBUG_READ_ONLY;
+ default:
+ return false;
+ }
+}
+
+static void *mlx5ctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
+ void *rpc_in, size_t in_len, size_t *out_len)
+{
+ struct mlx5ctl_dev *mcdev =
+ container_of(uctx->fwctl, struct mlx5ctl_dev, fwctl);
+ struct mlx5ctl_uctx *mfd =
+ container_of(uctx, struct mlx5ctl_uctx, uctx);
+ void *rpc_out;
+ int ret;
+
+ if (in_len < MLX5_ST_SZ_BYTES(mbox_in_hdr) ||
+ *out_len < MLX5_ST_SZ_BYTES(mbox_out_hdr))
+ return ERR_PTR(-EMSGSIZE);
+
+ mlx5ctl_dbg(mcdev, "[UID %d] cmdif: opcode 0x%x inlen %zu outlen %zu\n",
+ mfd->uctx_uid, MLX5_GET(mbox_in_hdr, rpc_in, opcode),
+ in_len, *out_len);
+
+ if (!mlx5ctl_validate_rpc(rpc_in, scope))
+ return ERR_PTR(-EBADMSG);
+
+ /*
+ * mlx5_cmd_do() copies the input message to its own buffer before
+ * executing it, so we can reuse the allocation for the output.
+ */
+ if (*out_len <= in_len) {
+ rpc_out = rpc_in;
+ } else {
+ rpc_out = kvzalloc(*out_len, GFP_KERNEL);
+ if (!rpc_out)
+ return ERR_PTR(-ENOMEM);
+ }
+
+ /* Enforce the user context for the command */
+ MLX5_SET(mbox_in_hdr, rpc_in, uid, mfd->uctx_uid);
+ ret = mlx5_cmd_do(mcdev->mdev, rpc_in, in_len, rpc_out, *out_len);
+
+ mlx5ctl_dbg(mcdev,
+ "[UID %d] cmdif: opcode 0x%x status 0x%x retval %pe\n",
+ mfd->uctx_uid, MLX5_GET(mbox_in_hdr, rpc_in, opcode),
+ MLX5_GET(mbox_out_hdr, rpc_out, status), ERR_PTR(ret));
+
+ /*
+ * -EREMOTEIO means execution succeeded and the out is valid,
+ * but an error code was returned inside out. Everything else
+ * means the RPC did not make it to the device.
+ */
+ if (ret && ret != -EREMOTEIO) {
+ if (rpc_out != rpc_in)
+ kfree(rpc_out);
+ return ERR_PTR(ret);
+ }
+ return rpc_out;
+}
+
+static const struct fwctl_ops mlx5ctl_ops = {
+ .device_type = FWCTL_DEVICE_TYPE_MLX5,
+ .uctx_size = sizeof(struct mlx5ctl_uctx),
+ .open_uctx = mlx5ctl_open_uctx,
+ .close_uctx = mlx5ctl_close_uctx,
+ .info = mlx5ctl_info,
+ .fw_rpc = mlx5ctl_fw_rpc,
+};
+
+static int mlx5ctl_probe(struct auxiliary_device *adev,
+ const struct auxiliary_device_id *id)
+
+{
+ struct mlx5_adev *madev = container_of(adev, struct mlx5_adev, adev);
+ struct mlx5_core_dev *mdev = madev->mdev;
+ struct mlx5ctl_dev *mcdev __free(mlx5ctl) = fwctl_alloc_device(
+ &mdev->pdev->dev, &mlx5ctl_ops, struct mlx5ctl_dev, fwctl);
+ int ret;
+
+ if (!mcdev)
+ return -ENOMEM;
+
+ mcdev->mdev = mdev;
+
+ ret = fwctl_register(&mcdev->fwctl);
+ if (ret)
+ return ret;
+ auxiliary_set_drvdata(adev, no_free_ptr(mcdev));
+ return 0;
+}
+
+static void mlx5ctl_remove(struct auxiliary_device *adev)
+{
+ struct mlx5ctl_dev *mcdev = auxiliary_get_drvdata(adev);
+
+ fwctl_unregister(&mcdev->fwctl);
+ fwctl_put(&mcdev->fwctl);
+}
+
+static const struct auxiliary_device_id mlx5ctl_id_table[] = {
+ {.name = MLX5_ADEV_NAME ".fwctl",},
+ {}
+};
+MODULE_DEVICE_TABLE(auxiliary, mlx5ctl_id_table);
+
+static struct auxiliary_driver mlx5ctl_driver = {
+ .name = "mlx5_fwctl",
+ .probe = mlx5ctl_probe,
+ .remove = mlx5ctl_remove,
+ .id_table = mlx5ctl_id_table,
+};
+
+module_auxiliary_driver(mlx5ctl_driver);
+
+MODULE_IMPORT_NS("FWCTL");
+MODULE_DESCRIPTION("mlx5 ConnectX fwctl driver");
+MODULE_AUTHOR("Saeed Mahameed <saeedm@nvidia.com>");
+MODULE_LICENSE("Dual BSD/GPL");
diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
index 0bec798790a67a..584a5ea8ecee11 100644
--- a/include/uapi/fwctl/fwctl.h
+++ b/include/uapi/fwctl/fwctl.h
@@ -42,6 +42,7 @@ enum {
enum fwctl_device_type {
FWCTL_DEVICE_TYPE_ERROR = 0,
+ FWCTL_DEVICE_TYPE_MLX5 = 1,
};
/**
diff --git a/include/uapi/fwctl/mlx5.h b/include/uapi/fwctl/mlx5.h
new file mode 100644
index 00000000000000..625819180ac6b8
--- /dev/null
+++ b/include/uapi/fwctl/mlx5.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES
+ *
+ * These are definitions for the command interface for mlx5 HW. mlx5 FW has a
+ * User Context mechanism which allows the FW to understand a security scope.
+ * FWCTL binds each FD to a FW user context and then places the User Context ID
+ * (UID) in each command header. The created User Context has a capability set
+ * that is appropriate for FWCTL's security model.
+ *
+ * Command formation should use a copy of the structs in mlx5_ifc.h following
+ * the Programmers Reference Manual. A open release is available here:
+ *
+ * https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf
+ *
+ * The device_type for this file is FWCTL_DEVICE_TYPE_MLX5.
+ */
+#ifndef _UAPI_FWCTL_MLX5_H
+#define _UAPI_FWCTL_MLX5_H
+
+#include <linux/types.h>
+
+/**
+ * struct fwctl_info_mlx5 - ioctl(FWCTL_INFO) out_device_data
+ * @uid: The FW UID this FD is bound to. Each command header will force
+ * this value.
+ * @uctx_caps: The FW capabilities that are enabled for the uid.
+ *
+ * Return basic information about the FW interface available.
+ */
+struct fwctl_info_mlx5 {
+ __u32 uid;
+ __u32 uctx_caps;
+};
+
+#endif
--
2.43.0
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v5 8/8] mlx5: Create an auxiliary device for fwctl_mlx5
2025-02-28 0:26 [PATCH v5 0/8] Introduce fwctl subystem Jason Gunthorpe
` (6 preceding siblings ...)
2025-02-28 0:26 ` [PATCH v5 7/8] fwctl/mlx5: Support for communicating with mlx5 fw Jason Gunthorpe
@ 2025-02-28 0:26 ` Jason Gunthorpe
2025-03-02 12:09 ` Leon Romanovsky
2025-03-04 1:53 ` [PATCH v5 0/8] Introduce fwctl subystem Jakub Kicinski
2025-03-20 23:22 ` Jason Gunthorpe
9 siblings, 1 reply; 56+ messages in thread
From: Jason Gunthorpe @ 2025-02-28 0:26 UTC (permalink / raw)
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
From: Saeed Mahameed <saeedm@nvidia.com>
If the device supports User Context then it can support fwctl. Create an
auxiliary device to allow fwctl to bind to it.
Create a sysfs like:
$ ls /sys/devices/pci0000:00/0000:00:0a.0/mlx5_core.fwctl.0/driver -l
lrwxrwxrwx 1 root root 0 Apr 25 19:46 /sys/devices/pci0000:00/0000:00:0a.0/mlx5_core.fwctl.0/driver -> ../../../../bus/auxiliary/drivers/mlx5_fwctl.mlx5_fwctl
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/net/ethernet/mellanox/mlx5/core/dev.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/dev.c b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
index 9a79674d27f15a..891bbbbfbbf1a4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/dev.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
@@ -228,8 +228,15 @@ enum {
MLX5_INTERFACE_PROTOCOL_VNET,
MLX5_INTERFACE_PROTOCOL_DPLL,
+ MLX5_INTERFACE_PROTOCOL_FWCTL,
};
+static bool is_fwctl_supported(struct mlx5_core_dev *dev)
+{
+ /* fwctl is most useful on PFs, prevent fwctl on SFs for now */
+ return MLX5_CAP_GEN(dev, uctx_cap) && !mlx5_core_is_sf(dev);
+}
+
static const struct mlx5_adev_device {
const char *suffix;
bool (*is_supported)(struct mlx5_core_dev *dev);
@@ -252,6 +259,8 @@ static const struct mlx5_adev_device {
.is_supported = &is_mp_supported },
[MLX5_INTERFACE_PROTOCOL_DPLL] = { .suffix = "dpll",
.is_supported = &is_dpll_supported },
+ [MLX5_INTERFACE_PROTOCOL_FWCTL] = { .suffix = "fwctl",
+ .is_supported = &is_fwctl_supported },
};
int mlx5_adev_idx_alloc(void)
--
2.43.0
^ permalink raw reply related [flat|nested] 56+ messages in thread
* Re: [PATCH v5 8/8] mlx5: Create an auxiliary device for fwctl_mlx5
2025-02-28 0:26 ` [PATCH v5 8/8] mlx5: Create an auxiliary device for fwctl_mlx5 Jason Gunthorpe
@ 2025-03-02 12:09 ` Leon Romanovsky
0 siblings, 0 replies; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-02 12:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, linux-cxl, linux-rdma, netdev, Saeed Mahameed,
Nelson, Shannon
On Thu, Feb 27, 2025 at 08:26:36PM -0400, Jason Gunthorpe wrote:
> From: Saeed Mahameed <saeedm@nvidia.com>
>
> If the device supports User Context then it can support fwctl. Create an
> auxiliary device to allow fwctl to bind to it.
>
> Create a sysfs like:
>
> $ ls /sys/devices/pci0000:00/0000:00:0a.0/mlx5_core.fwctl.0/driver -l
> lrwxrwxrwx 1 root root 0 Apr 25 19:46 /sys/devices/pci0000:00/0000:00:0a.0/mlx5_core.fwctl.0/driver -> ../../../../bus/auxiliary/drivers/mlx5_fwctl.mlx5_fwctl
>
> Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/net/ethernet/mellanox/mlx5/core/dev.c | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
Thanks,
Acked-by: Leon Romanovsky <leonro@nvidia.com>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 7/8] fwctl/mlx5: Support for communicating with mlx5 fw
2025-02-28 0:26 ` [PATCH v5 7/8] fwctl/mlx5: Support for communicating with mlx5 fw Jason Gunthorpe
@ 2025-03-02 12:11 ` Leon Romanovsky
2025-03-04 17:50 ` Jason Gunthorpe
0 siblings, 1 reply; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-02 12:11 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, linux-cxl, linux-rdma, netdev, Saeed Mahameed,
Nelson, Shannon
On Thu, Feb 27, 2025 at 08:26:35PM -0400, Jason Gunthorpe wrote:
> From: Saeed Mahameed <saeedm@nvidia.com>
<...>
> FWCTL SUBSYSTEM
> +M: Dave Jiang <dave.jiang@intel.com>
> M: Jason Gunthorpe <jgg@nvidia.com>
> M: Saeed Mahameed <saeedm@nvidia.com>
> +R: Jonathan Cameron <Jonathan.Cameron@huawei.com>
These are unrelated changes, probably rebase error.
> S: Maintained
> F: Documentation/userspace-api/fwctl/
> F: drivers/fwctl/
> F: include/linux/fwctl.h
> F: include/uapi/fwctl/
>
> +FWCTL MLX5 DRIVER
> +M: Saeed Mahameed <saeedm@nvidia.com>
> +R: Itay Avraham <itayavr@nvidia.com>
> +L: linux-kernel@vger.kernel.org
> +S: Maintained
> +F: drivers/fwctl/mlx5/
Thanks,
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-02-28 0:26 [PATCH v5 0/8] Introduce fwctl subystem Jason Gunthorpe
` (7 preceding siblings ...)
2025-02-28 0:26 ` [PATCH v5 8/8] mlx5: Create an auxiliary device for fwctl_mlx5 Jason Gunthorpe
@ 2025-03-04 1:53 ` Jakub Kicinski
2025-03-04 14:00 ` Jason Gunthorpe
2025-03-20 23:22 ` Jason Gunthorpe
9 siblings, 1 reply; 56+ messages in thread
From: Jakub Kicinski @ 2025-03-04 1:53 UTC (permalink / raw)
To: Jason Gunthorpe, Greg Kroah-Hartman
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, Leon Romanovsky,
linux-cxl, linux-rdma, netdev, Saeed Mahameed, Nelson, Shannon
On Thu, 27 Feb 2025 20:26:28 -0400 Jason Gunthorpe wrote:
> v5:
> - Move hunks between patches to make more sense
> - Rename ucmd_buffer to fwctl_ucmd_buffer
> - Update comments and commit messages
> - Copyright to 2025
> - Drop bxnt WIP patches
> - Allow a NULL ops->info
> - Decode more op codes for mlx5 and the sub-operation for
> MLX5_CMD_OP_ACCESS_REG/_USER
Did you address my feedback? I asked for the mlx5 support to only be
enabled in RDMA is in use. Saeed who wrote the mlx5 parts of this
patchset clearly admitted on v4:
As explained above, netdev doesn't need it
https://lore.kernel.org/all/Z6ZsOMLq7tt3ijX_@x130/
Greg, I've been asking for this interface to be scoped to when RDMA
(/CXL/storage) is in enabled on these NICs since pretty much the first
RFC. Do I need to keep responding?
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-04 1:53 ` [PATCH v5 0/8] Introduce fwctl subystem Jakub Kicinski
@ 2025-03-04 14:00 ` Jason Gunthorpe
2025-03-04 17:59 ` Saeed Mahameed
2025-03-05 0:42 ` Jakub Kicinski
0 siblings, 2 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-03-04 14:00 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, David Ahern, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Leonid Bloch,
Leon Romanovsky, linux-cxl, linux-rdma, netdev, Saeed Mahameed,
Nelson, Shannon
On Mon, Mar 03, 2025 at 05:53:58PM -0800, Jakub Kicinski wrote:
> On Thu, 27 Feb 2025 20:26:28 -0400 Jason Gunthorpe wrote:
> > v5:
> > - Move hunks between patches to make more sense
> > - Rename ucmd_buffer to fwctl_ucmd_buffer
> > - Update comments and commit messages
> > - Copyright to 2025
> > - Drop bxnt WIP patches
> > - Allow a NULL ops->info
> > - Decode more op codes for mlx5 and the sub-operation for
> > MLX5_CMD_OP_ACCESS_REG/_USER
>
> Did you address my feedback? I asked for the mlx5 support to only be
> enabled in RDMA is in use. Saeed who wrote the mlx5 parts of this
> patchset clearly admitted on v4:
I never agreed to that formulation. I suggested that perhaps runtime
configurations where netdev is the only driver using the HW could be
disabled (ie a netdev exclusion, not a rdma inclusion).
However, there is not agreement on this from Saeed who is responsible
for mlx5:
https://lore.kernel.org/all/Z7z0ADkimCkhr7Xz@x130/
I also surveyed other stakeholders on a netdev-exclusion proposal and
did not hear support. You need to convince people this is a good idea.
However, I would agree fwctl should not accept any fwctl drivers for
simple networking devices. However, "smart nics" and RDMA capable
devices are in-scope.
I could also probably agree to using kconfig to disable fwctl drivers
on kernels that statically compile out rdma, vdpa, nvme and related,
though I agree with Saeed that it seems to lack technical merit.
> Greg, I've been asking for this interface to be scoped to when RDMA
> (/CXL/storage) is in enabled on these NICs since pretty much the first
> RFC.
You only started asking for this more limited approach in v4. All your
previous arguments were that fwctl should be entirely killed for any
networking HW.
Jason
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 7/8] fwctl/mlx5: Support for communicating with mlx5 fw
2025-03-02 12:11 ` Leon Romanovsky
@ 2025-03-04 17:50 ` Jason Gunthorpe
0 siblings, 0 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-03-04 17:50 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, linux-cxl, linux-rdma, netdev, Saeed Mahameed,
Nelson, Shannon
On Sun, Mar 02, 2025 at 02:11:44PM +0200, Leon Romanovsky wrote:
> On Thu, Feb 27, 2025 at 08:26:35PM -0400, Jason Gunthorpe wrote:
> > From: Saeed Mahameed <saeedm@nvidia.com>
>
> <...>
>
> > FWCTL SUBSYSTEM
> > +M: Dave Jiang <dave.jiang@intel.com>
> > M: Jason Gunthorpe <jgg@nvidia.com>
> > M: Saeed Mahameed <saeedm@nvidia.com>
> > +R: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>
> These are unrelated changes, probably rebase error.
Got it, thanks
Jason
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-04 14:00 ` Jason Gunthorpe
@ 2025-03-04 17:59 ` Saeed Mahameed
2025-03-05 0:42 ` Jakub Kicinski
1 sibling, 0 replies; 56+ messages in thread
From: Saeed Mahameed @ 2025-03-04 17:59 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Jakub Kicinski, Greg Kroah-Hartman, Andy Gospodarek,
Aron Silverton, Dan Williams, Daniel Vetter, Dave Jiang,
David Ahern, Christoph Hellwig, Itay Avraham, Jiri Pirko,
Jonathan Cameron, Leonid Bloch, Leon Romanovsky, linux-cxl,
linux-rdma, netdev, Saeed Mahameed, Nelson, Shannon
On 04 Mar 10:00, Jason Gunthorpe wrote:
>On Mon, Mar 03, 2025 at 05:53:58PM -0800, Jakub Kicinski wrote:
>> On Thu, 27 Feb 2025 20:26:28 -0400 Jason Gunthorpe wrote:
>> > v5:
>> > - Move hunks between patches to make more sense
>> > - Rename ucmd_buffer to fwctl_ucmd_buffer
>> > - Update comments and commit messages
>> > - Copyright to 2025
>> > - Drop bxnt WIP patches
>> > - Allow a NULL ops->info
>> > - Decode more op codes for mlx5 and the sub-operation for
>> > MLX5_CMD_OP_ACCESS_REG/_USER
>>
>> Did you address my feedback? I asked for the mlx5 support to only be
>> enabled in RDMA is in use. Saeed who wrote the mlx5 parts of this
>> patchset clearly admitted on v4:
When I said fwctl is not needed for netdev, I meant that it will not be used
for netdev object configuration and as I said before FW will block that
anyways. fwctl in mlx5 is not only for RDMA, So I don't know how to address
your comment.
Not to mention that fwctl is a very great tool to debug netdev problems.
>
>I never agreed to that formulation. I suggested that perhaps runtime
>configurations where netdev is the only driver using the HW could be
>disabled (ie a netdev exclusion, not a rdma inclusion).
>
>However, there is not agreement on this from Saeed who is responsible
>for mlx5:
>
> https://lore.kernel.org/all/Z7z0ADkimCkhr7Xz@x130/
>
>I also surveyed other stakeholders on a netdev-exclusion proposal and
>did not hear support. You need to convince people this is a good idea.
>
>However, I would agree fwctl should not accept any fwctl drivers for
>simple networking devices. However, "smart nics" and RDMA capable
>devices are in-scope.
>
>I could also probably agree to using kconfig to disable fwctl drivers
>on kernels that statically compile out rdma, vdpa, nvme and related,
>though I agree with Saeed that it seems to lack technical merit.
>
>> Greg, I've been asking for this interface to be scoped to when RDMA
>> (/CXL/storage) is in enabled on these NICs since pretty much the first
>> RFC.
>
>You only started asking for this more limited approach in v4. All your
>previous arguments were that fwctl should be entirely killed for any
>networking HW.
>
>Jason
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-04 14:00 ` Jason Gunthorpe
2025-03-04 17:59 ` Saeed Mahameed
@ 2025-03-05 0:42 ` Jakub Kicinski
2025-03-05 13:32 ` Jason Gunthorpe
1 sibling, 1 reply; 56+ messages in thread
From: Jakub Kicinski @ 2025-03-05 0:42 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, David Ahern, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Leonid Bloch,
Leon Romanovsky, linux-cxl, linux-rdma, netdev, Saeed Mahameed,
Nelson, Shannon
On Tue, 4 Mar 2025 10:00:36 -0400 Jason Gunthorpe wrote:
> I never agreed to that formulation. I suggested that perhaps runtime
> configurations where netdev is the only driver using the HW could be
> disabled (ie a netdev exclusion, not a rdma inclusion).
I thought you were arguing that me opposing the addition was
"maintainer overreach". As in me telling other parts of the kernel
what is and isn't allowed. Do I not get a say what gets merged under
drivers/net/ now?
> However, I would agree fwctl should not accept any fwctl drivers for
> simple networking devices. However, "smart nics" and RDMA capable
> devices are in-scope.
Please stop with the fake technical arguments.
I worked on SmartNICs for a decade.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 0:42 ` Jakub Kicinski
@ 2025-03-05 13:32 ` Jason Gunthorpe
2025-03-05 13:43 ` Leon Romanovsky
` (2 more replies)
0 siblings, 3 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-03-05 13:32 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, David Ahern, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Leonid Bloch,
Leon Romanovsky, linux-cxl, linux-rdma, netdev, Saeed Mahameed,
Nelson, Shannon
On Tue, Mar 04, 2025 at 04:42:03PM -0800, Jakub Kicinski wrote:
> On Tue, 4 Mar 2025 10:00:36 -0400 Jason Gunthorpe wrote:
> > I never agreed to that formulation. I suggested that perhaps runtime
> > configurations where netdev is the only driver using the HW could be
> > disabled (ie a netdev exclusion, not a rdma inclusion).
>
> I thought you were arguing that me opposing the addition was
> "maintainer overreach". As in me telling other parts of the kernel
> what is and isn't allowed. Do I not get a say what gets merged under
> drivers/net/ now?
The PCI core drivers are a shared resource jointly maintained by all
the subsytems that use them. They are maintained by their respective
maintainers. Saeed/etc in this case.
It would be inappropriate for your preferences to supersede Saeed's
when he is a maintainer of the mlx5_core driver and fwctl. Please try
and get Saeed on board with your plan.
If the placement under drivers/net makes this confusing then we can
certainly change the directory names.
Jason
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 13:32 ` Jason Gunthorpe
@ 2025-03-05 13:43 ` Leon Romanovsky
2025-03-05 15:08 ` Jiri Pirko
2025-03-06 2:16 ` Jakub Kicinski
2 siblings, 0 replies; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-05 13:43 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Jakub Kicinski, Greg Kroah-Hartman, Andy Gospodarek,
Aron Silverton, Dan Williams, Daniel Vetter, Dave Jiang,
David Ahern, Christoph Hellwig, Itay Avraham, Jiri Pirko,
Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
On Wed, Mar 05, 2025 at 09:32:54AM -0400, Jason Gunthorpe wrote:
> On Tue, Mar 04, 2025 at 04:42:03PM -0800, Jakub Kicinski wrote:
> > On Tue, 4 Mar 2025 10:00:36 -0400 Jason Gunthorpe wrote:
> > > I never agreed to that formulation. I suggested that perhaps runtime
> > > configurations where netdev is the only driver using the HW could be
> > > disabled (ie a netdev exclusion, not a rdma inclusion).
> >
> > I thought you were arguing that me opposing the addition was
> > "maintainer overreach". As in me telling other parts of the kernel
> > what is and isn't allowed. Do I not get a say what gets merged under
> > drivers/net/ now?
>
> The PCI core drivers are a shared resource jointly maintained by all
> the subsytems that use them. They are maintained by their respective
> maintainers. Saeed/etc in this case.
>
> It would be inappropriate for your preferences to supersede Saeed's
> when he is a maintainer of the mlx5_core driver and fwctl. Please try
> and get Saeed on board with your plan.
>
> If the placement under drivers/net makes this confusing then we can
> certainly change the directory names.
I suggested it before to Jakub and it looks like he is not opposing to it.
https://lore.kernel.org/all/20250211075553.GF17863@unreal/
Thanks
>
> Jason
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 13:32 ` Jason Gunthorpe
2025-03-05 13:43 ` Leon Romanovsky
@ 2025-03-05 15:08 ` Jiri Pirko
2025-03-05 15:22 ` Leon Romanovsky
2025-03-05 18:17 ` David Ahern
2025-03-06 2:16 ` Jakub Kicinski
2 siblings, 2 replies; 56+ messages in thread
From: Jiri Pirko @ 2025-03-05 15:08 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Jakub Kicinski, Greg Kroah-Hartman, Andy Gospodarek,
Aron Silverton, Dan Williams, Daniel Vetter, Dave Jiang,
David Ahern, Christoph Hellwig, Itay Avraham, Jiri Pirko,
Jonathan Cameron, Leonid Bloch, Leon Romanovsky, linux-cxl,
linux-rdma, netdev, Saeed Mahameed, Nelson, Shannon
Wed, Mar 05, 2025 at 02:32:54PM +0100, jgg@nvidia.com wrote:
>On Tue, Mar 04, 2025 at 04:42:03PM -0800, Jakub Kicinski wrote:
>> On Tue, 4 Mar 2025 10:00:36 -0400 Jason Gunthorpe wrote:
>> > I never agreed to that formulation. I suggested that perhaps runtime
>> > configurations where netdev is the only driver using the HW could be
>> > disabled (ie a netdev exclusion, not a rdma inclusion).
>>
>> I thought you were arguing that me opposing the addition was
>> "maintainer overreach". As in me telling other parts of the kernel
>> what is and isn't allowed. Do I not get a say what gets merged under
>> drivers/net/ now?
>
>The PCI core drivers are a shared resource jointly maintained by all
>the subsytems that use them. They are maintained by their respective
>maintainers. Saeed/etc in this case.
>
>It would be inappropriate for your preferences to supersede Saeed's
>when he is a maintainer of the mlx5_core driver and fwctl. Please try
>and get Saeed on board with your plan.
>
>If the placement under drivers/net makes this confusing then we can
>certainly change the directory names.
According to how mlx5 driver is structured, and the rest of the advanced
drivers in the same area are becoming as well, it would make sense to me
to have mlx5 core in separate core directory, maintained directly by driver
maintainer:
drivers/core/mlx5/
then each of the protocol auxiliary device lands in appropriate
subsystem directory.
It's not always simple to find clear cut though. Like for example
devlink and representors code related to eswitch orchestration.
The benefit would be that lots of core-related non-netdev patch-trafic
would go directly, which would ease the netdev maintainership burden and
would make things more flexible. This per-driver-serialization for
patchset processing bottleneck would become much more bareable.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 15:08 ` Jiri Pirko
@ 2025-03-05 15:22 ` Leon Romanovsky
2025-03-05 15:56 ` Jiri Pirko
2025-03-05 18:17 ` David Ahern
1 sibling, 1 reply; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-05 15:22 UTC (permalink / raw)
To: Jiri Pirko
Cc: Jason Gunthorpe, Jakub Kicinski, Greg Kroah-Hartman,
Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma,
netdev, Saeed Mahameed, Nelson, Shannon
On Wed, Mar 05, 2025 at 04:08:19PM +0100, Jiri Pirko wrote:
> Wed, Mar 05, 2025 at 02:32:54PM +0100, jgg@nvidia.com wrote:
> >On Tue, Mar 04, 2025 at 04:42:03PM -0800, Jakub Kicinski wrote:
> >> On Tue, 4 Mar 2025 10:00:36 -0400 Jason Gunthorpe wrote:
> >> > I never agreed to that formulation. I suggested that perhaps runtime
> >> > configurations where netdev is the only driver using the HW could be
> >> > disabled (ie a netdev exclusion, not a rdma inclusion).
> >>
> >> I thought you were arguing that me opposing the addition was
> >> "maintainer overreach". As in me telling other parts of the kernel
> >> what is and isn't allowed. Do I not get a say what gets merged under
> >> drivers/net/ now?
> >
> >The PCI core drivers are a shared resource jointly maintained by all
> >the subsytems that use them. They are maintained by their respective
> >maintainers. Saeed/etc in this case.
> >
> >It would be inappropriate for your preferences to supersede Saeed's
> >when he is a maintainer of the mlx5_core driver and fwctl. Please try
> >and get Saeed on board with your plan.
> >
> >If the placement under drivers/net makes this confusing then we can
> >certainly change the directory names.
>
> According to how mlx5 driver is structured, and the rest of the advanced
> drivers in the same area are becoming as well, it would make sense to me
> to have mlx5 core in separate core directory, maintained directly by driver
> maintainer:
> drivers/core/mlx5/
> then each of the protocol auxiliary device lands in appropriate
> subsystem directory.
In my vision, the write access to that drivers/core/ will be given to all
relevant subsystem maintainers, so it will operate like shared branch, but
foe everyone.
It means that series for netdev that changes mlx5_core and netdev code
will be sent to netdev and applied by netdev maintainers. In similar
way, series which targets RDMA will be handled by RDMA crew.
It will allow us to make sure that every piece of code in shared
repository is actually used.
Thanks
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 15:22 ` Leon Romanovsky
@ 2025-03-05 15:56 ` Jiri Pirko
0 siblings, 0 replies; 56+ messages in thread
From: Jiri Pirko @ 2025-03-05 15:56 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Jason Gunthorpe, Jakub Kicinski, Greg Kroah-Hartman,
Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma,
netdev, Saeed Mahameed, Nelson, Shannon
Wed, Mar 05, 2025 at 04:22:46PM +0100, leon@kernel.org wrote:
>On Wed, Mar 05, 2025 at 04:08:19PM +0100, Jiri Pirko wrote:
>> Wed, Mar 05, 2025 at 02:32:54PM +0100, jgg@nvidia.com wrote:
>> >On Tue, Mar 04, 2025 at 04:42:03PM -0800, Jakub Kicinski wrote:
>> >> On Tue, 4 Mar 2025 10:00:36 -0400 Jason Gunthorpe wrote:
>> >> > I never agreed to that formulation. I suggested that perhaps runtime
>> >> > configurations where netdev is the only driver using the HW could be
>> >> > disabled (ie a netdev exclusion, not a rdma inclusion).
>> >>
>> >> I thought you were arguing that me opposing the addition was
>> >> "maintainer overreach". As in me telling other parts of the kernel
>> >> what is and isn't allowed. Do I not get a say what gets merged under
>> >> drivers/net/ now?
>> >
>> >The PCI core drivers are a shared resource jointly maintained by all
>> >the subsytems that use them. They are maintained by their respective
>> >maintainers. Saeed/etc in this case.
>> >
>> >It would be inappropriate for your preferences to supersede Saeed's
>> >when he is a maintainer of the mlx5_core driver and fwctl. Please try
>> >and get Saeed on board with your plan.
>> >
>> >If the placement under drivers/net makes this confusing then we can
>> >certainly change the directory names.
>>
>> According to how mlx5 driver is structured, and the rest of the advanced
>> drivers in the same area are becoming as well, it would make sense to me
>> to have mlx5 core in separate core directory, maintained directly by driver
>> maintainer:
>> drivers/core/mlx5/
>> then each of the protocol auxiliary device lands in appropriate
>> subsystem directory.
>
>In my vision, the write access to that drivers/core/ will be given to all
>relevant subsystem maintainers, so it will operate like shared branch, but
>foe everyone.
>
>It means that series for netdev that changes mlx5_core and netdev code
>will be sent to netdev and applied by netdev maintainers. In similar
>way, series which targets RDMA will be handled by RDMA crew.
>
>It will allow us to make sure that every piece of code in shared
>repository is actually used.
Makes perfect sense to me.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 15:08 ` Jiri Pirko
2025-03-05 15:22 ` Leon Romanovsky
@ 2025-03-05 18:17 ` David Ahern
2025-03-05 18:28 ` Leon Romanovsky
1 sibling, 1 reply; 56+ messages in thread
From: David Ahern @ 2025-03-05 18:17 UTC (permalink / raw)
To: Jiri Pirko, Jason Gunthorpe
Cc: Jakub Kicinski, Greg Kroah-Hartman, Andy Gospodarek,
Aron Silverton, Dan Williams, Daniel Vetter, Dave Jiang,
Christoph Hellwig, Itay Avraham, Jiri Pirko, Jonathan Cameron,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
On 3/5/25 8:08 AM, Jiri Pirko wrote:
> Wed, Mar 05, 2025 at 02:32:54PM +0100, jgg@nvidia.com wrote:
>> On Tue, Mar 04, 2025 at 04:42:03PM -0800, Jakub Kicinski wrote:
>>> I thought you were arguing that me opposing the addition was
>>> "maintainer overreach". As in me telling other parts of the kernel
>>> what is and isn't allowed. Do I not get a say what gets merged under
>>> drivers/net/ now?
>>
>> The PCI core drivers are a shared resource jointly maintained by all
>> the subsytems that use them. They are maintained by their respective
>> maintainers. Saeed/etc in this case.
>>
>> It would be inappropriate for your preferences to supersede Saeed's
>> when he is a maintainer of the mlx5_core driver and fwctl. Please try
>> and get Saeed on board with your plan.
>>
>> If the placement under drivers/net makes this confusing then we can
>> certainly change the directory names.
>
> According to how mlx5 driver is structured, and the rest of the advanced
> drivers in the same area are becoming as well, it would make sense to me
> to have mlx5 core in separate core directory, maintained directly by driver
> maintainer:
> drivers/core/mlx5/
> then each of the protocol auxiliary device lands in appropriate
> subsystem directory.
+1
This is how I have structured our drivers -- core driver for owning the
PCI device and hosting the APIs to communicate with hardware, an aux bus
and then smaller subsystem focused drivers for the aux devices that make
the device usable from different contexts.
I think we are ready to start upstreaming, but I am waiting to see how
this falls out - to see if our core driver can land in a non-subsystem
specific location (e.g., drivers/core) or if it needs to go with fwctl
as a generic location.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 18:17 ` David Ahern
@ 2025-03-05 18:28 ` Leon Romanovsky
2025-03-05 20:41 ` Saeed Mahameed
2025-03-13 12:30 ` David Ahern
0 siblings, 2 replies; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-05 18:28 UTC (permalink / raw)
To: David Ahern
Cc: Jiri Pirko, Jason Gunthorpe, Jakub Kicinski, Greg Kroah-Hartman,
Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, Christoph Hellwig, Itay Avraham, Jiri Pirko,
Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
On Wed, Mar 05, 2025 at 11:17:19AM -0700, David Ahern wrote:
> On 3/5/25 8:08 AM, Jiri Pirko wrote:
> > Wed, Mar 05, 2025 at 02:32:54PM +0100, jgg@nvidia.com wrote:
> >> On Tue, Mar 04, 2025 at 04:42:03PM -0800, Jakub Kicinski wrote:
> >>> I thought you were arguing that me opposing the addition was
> >>> "maintainer overreach". As in me telling other parts of the kernel
> >>> what is and isn't allowed. Do I not get a say what gets merged under
> >>> drivers/net/ now?
> >>
> >> The PCI core drivers are a shared resource jointly maintained by all
> >> the subsytems that use them. They are maintained by their respective
> >> maintainers. Saeed/etc in this case.
> >>
> >> It would be inappropriate for your preferences to supersede Saeed's
> >> when he is a maintainer of the mlx5_core driver and fwctl. Please try
> >> and get Saeed on board with your plan.
> >>
> >> If the placement under drivers/net makes this confusing then we can
> >> certainly change the directory names.
> >
> > According to how mlx5 driver is structured, and the rest of the advanced
> > drivers in the same area are becoming as well, it would make sense to me
> > to have mlx5 core in separate core directory, maintained directly by driver
> > maintainer:
> > drivers/core/mlx5/
> > then each of the protocol auxiliary device lands in appropriate
> > subsystem directory.
>
> +1
>
> This is how I have structured our drivers -- core driver for owning the
> PCI device and hosting the APIs to communicate with hardware, an aux bus
> and then smaller subsystem focused drivers for the aux devices that make
> the device usable from different contexts.
>
> I think we are ready to start upstreaming, but I am waiting to see how
> this falls out - to see if our core driver can land in a non-subsystem
> specific location (e.g., drivers/core) or if it needs to go with fwctl
> as a generic location.
Do it right, and push it to drivers/core. I'm aware of at least one
driver from huge company (not Nvidia) which is in preparation phase
before upstreaming, and will fit nicely into this model.
They have separated blocks for PCI, eth, RDMA and GPU.
Thanks
>
>
>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 18:28 ` Leon Romanovsky
@ 2025-03-05 20:41 ` Saeed Mahameed
2025-03-05 23:21 ` Jason Gunthorpe
2025-03-13 12:30 ` David Ahern
1 sibling, 1 reply; 56+ messages in thread
From: Saeed Mahameed @ 2025-03-05 20:41 UTC (permalink / raw)
To: Leon Romanovsky
Cc: David Ahern, Jiri Pirko, Jason Gunthorpe, Jakub Kicinski,
Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma,
netdev, Saeed Mahameed, Nelson, Shannon
On 05 Mar 20:28, Leon Romanovsky wrote:
>On Wed, Mar 05, 2025 at 11:17:19AM -0700, David Ahern wrote:
>> On 3/5/25 8:08 AM, Jiri Pirko wrote:
>> > Wed, Mar 05, 2025 at 02:32:54PM +0100, jgg@nvidia.com wrote:
>> >> On Tue, Mar 04, 2025 at 04:42:03PM -0800, Jakub Kicinski wrote:
>> >>> I thought you were arguing that me opposing the addition was
>> >>> "maintainer overreach". As in me telling other parts of the kernel
>> >>> what is and isn't allowed. Do I not get a say what gets merged under
>> >>> drivers/net/ now?
>> >>
>> >> The PCI core drivers are a shared resource jointly maintained by all
>> >> the subsytems that use them. They are maintained by their respective
>> >> maintainers. Saeed/etc in this case.
>> >>
>> >> It would be inappropriate for your preferences to supersede Saeed's
>> >> when he is a maintainer of the mlx5_core driver and fwctl. Please try
>> >> and get Saeed on board with your plan.
>> >>
>> >> If the placement under drivers/net makes this confusing then we can
>> >> certainly change the directory names.
>> >
>> > According to how mlx5 driver is structured, and the rest of the advanced
>> > drivers in the same area are becoming as well, it would make sense to me
>> > to have mlx5 core in separate core directory, maintained directly by driver
>> > maintainer:
>> > drivers/core/mlx5/
>> > then each of the protocol auxiliary device lands in appropriate
>> > subsystem directory.
>>
>> +1
>>
>> This is how I have structured our drivers -- core driver for owning the
>> PCI device and hosting the APIs to communicate with hardware, an aux bus
>> and then smaller subsystem focused drivers for the aux devices that make
>> the device usable from different contexts.
>>
>> I think we are ready to start upstreaming, but I am waiting to see how
>> this falls out - to see if our core driver can land in a non-subsystem
>> specific location (e.g., drivers/core) or if it needs to go with fwctl
>> as a generic location.
>
>Do it right, and push it to drivers/core. I'm aware of at least one
>driver from huge company (not Nvidia) which is in preparation phase
>before upstreaming, and will fit nicely into this model.
>
How do you imagine this driver/core structure should look like? Who will be
the top dir maintainer? It should be something that is tightly coupled with
aux, currently aux is under drivers/base/auxiliary.c I think it should move
to drivers/aux/auxiliary.c and device drivers should implement their own
aux buses, WH access APIs and probing/init logic under that directory
e.g: drivers/aux/mlx5/..
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 20:41 ` Saeed Mahameed
@ 2025-03-05 23:21 ` Jason Gunthorpe
2025-03-06 7:29 ` Leon Romanovsky
2025-03-11 11:23 ` David Ahern
0 siblings, 2 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-03-05 23:21 UTC (permalink / raw)
To: Saeed Mahameed
Cc: Leon Romanovsky, David Ahern, Jiri Pirko, Jakub Kicinski,
Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma,
netdev, Saeed Mahameed, Nelson, Shannon
On Wed, Mar 05, 2025 at 12:41:35PM -0800, Saeed Mahameed wrote:
> How do you imagine this driver/core structure should look like? Who
> will be the top dir maintainer?
I would set something like this up more like DRM. Every driver
maintainer gets commit rights, some rules about no uAPIs, or at least
other acks before merging uAPI. Use the tree for staging shared
branches.
Driver maintainers with the most commits per cycle does the PR or
something like that.
There is no subsystem or cross-driver entanglement so there is no real
need for gatekeeping.
It would be a good opportunity to help more people engage with the
kernel process and learn the full maintainer flow.
> It should be something that is tightly coupled with aux, currently
> aux is under drivers/base/auxiliary.c I think it should move to
> drivers/aux/auxiliary.c and device drivers should implement their
> own aux buses, WH access APIs and probing/init logic under that
> directory e.g: drivers/aux/mlx5/..
That makes sense to me. I would expect everything in this collection
to be PCI drivers spawing aux devices.
drivers/aux_core/ or something like that, perhaps?
Jason
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 13:32 ` Jason Gunthorpe
2025-03-05 13:43 ` Leon Romanovsky
2025-03-05 15:08 ` Jiri Pirko
@ 2025-03-06 2:16 ` Jakub Kicinski
2 siblings, 0 replies; 56+ messages in thread
From: Jakub Kicinski @ 2025-03-06 2:16 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, David Ahern, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Leonid Bloch,
Leon Romanovsky, linux-cxl, linux-rdma, netdev, Saeed Mahameed,
Nelson, Shannon
On Wed, 5 Mar 2025 09:32:54 -0400 Jason Gunthorpe wrote:
> > I thought you were arguing that me opposing the addition was
> > "maintainer overreach". As in me telling other parts of the kernel
> > what is and isn't allowed. Do I not get a say what gets merged under
> > drivers/net/ now?
>
> The PCI core drivers are a shared resource jointly maintained by all
> the subsytems that use them. They are maintained by their respective
> maintainers. Saeed/etc in this case.
PCI core driver? You need it for RDMA.
Whether you move the code around or spell it backwards, I don't care,
as long as vast majority of the users of this **NIC**, who only use
TCP/IP do not have fwctl access.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 23:21 ` Jason Gunthorpe
@ 2025-03-06 7:29 ` Leon Romanovsky
2025-03-11 11:23 ` David Ahern
1 sibling, 0 replies; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-06 7:29 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Saeed Mahameed, David Ahern, Jiri Pirko, Jakub Kicinski,
Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma,
netdev, Saeed Mahameed, Nelson, Shannon
On Wed, Mar 05, 2025 at 07:21:54PM -0400, Jason Gunthorpe wrote:
> On Wed, Mar 05, 2025 at 12:41:35PM -0800, Saeed Mahameed wrote:
>
> > How do you imagine this driver/core structure should look like? Who
> > will be the top dir maintainer?
>
> I would set something like this up more like DRM. Every driver
> maintainer gets commit rights, some rules about no uAPIs, or at least
> other acks before merging uAPI. Use the tree for staging shared
> branches.
>
> Driver maintainers with the most commits per cycle does the PR or
> something like that.
>
> There is no subsystem or cross-driver entanglement so there is no real
> need for gatekeeping.
Yes, it can be structured like you proposed too or/and combined with my
idea https://lore.kernel.org/netdev/20250303150015.GA1926949@unreal/
The most important part is that it needs to be group of maintainers.
>
> It would be a good opportunity to help more people engage with the
> kernel process and learn the full maintainer flow.
>
> > It should be something that is tightly coupled with aux, currently
> > aux is under drivers/base/auxiliary.c I think it should move to
> > drivers/aux/auxiliary.c and device drivers should implement their
> > own aux buses, WH access APIs and probing/init logic under that
> > directory e.g: drivers/aux/mlx5/..
>
> That makes sense to me. I would expect everything in this collection
> to be PCI drivers spawing aux devices.
>
> drivers/aux_core/ or something like that, perhaps?
I like Saeed's proposal "drivers/aux/", it is more short and catchy.
Thanks
>
> Jason
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 23:21 ` Jason Gunthorpe
2025-03-06 7:29 ` Leon Romanovsky
@ 2025-03-11 11:23 ` David Ahern
2025-03-11 13:59 ` Leon Romanovsky
2025-03-11 14:27 ` Nelson, Shannon
1 sibling, 2 replies; 56+ messages in thread
From: David Ahern @ 2025-03-11 11:23 UTC (permalink / raw)
To: Jason Gunthorpe, Saeed Mahameed
Cc: Leon Romanovsky, Jiri Pirko, Jakub Kicinski, Greg Kroah-Hartman,
Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, Christoph Hellwig, Itay Avraham, Jiri Pirko,
Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
On 3/6/25 12:21 AM, Jason Gunthorpe wrote:
> On Wed, Mar 05, 2025 at 12:41:35PM -0800, Saeed Mahameed wrote:
>
>> How do you imagine this driver/core structure should look like? Who
>> will be the top dir maintainer?
>
> I would set something like this up more like DRM. Every driver
> maintainer gets commit rights, some rules about no uAPIs, or at least
> other acks before merging uAPI. Use the tree for staging shared
> branches.
why no uapi? Core driver can have knowledge of h/w resources across all
use cases. For example, our core driver supports a generid netlink based
dump (no set operations; get and dump only so maybe that should be the
restriction?) of all objects regardless of how created -- netdev, ib,
etc. -- and with much more detail.
>
> Driver maintainers with the most commits per cycle does the PR or
> something like that.
>
> There is no subsystem or cross-driver entanglement so there is no real
> need for gatekeeping.
>
> It would be a good opportunity to help more people engage with the
> kernel process and learn the full maintainer flow.
>
>> It should be something that is tightly coupled with aux, currently
>> aux is under drivers/base/auxiliary.c I think it should move to
>> drivers/aux/auxiliary.c and device drivers should implement their
>> own aux buses, WH access APIs and probing/init logic under that
>> directory e.g: drivers/aux/mlx5/..
>
> That makes sense to me. I would expect everything in this collection
> to be PCI drivers spawing aux devices.
>
> drivers/aux_core/ or something like that, perhaps?
>
drivers/aux_core works for me; removes the 'pci' assumption and makes it
clear the real attribute here is use of the aux bus with subsystem
specific devices. I am still not clear on how such a branch will work -
e.g. We will want multi-vendor review, not just merge the PR and go.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-11 11:23 ` David Ahern
@ 2025-03-11 13:59 ` Leon Romanovsky
2025-03-12 9:31 ` David Ahern
2025-03-11 14:27 ` Nelson, Shannon
1 sibling, 1 reply; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-11 13:59 UTC (permalink / raw)
To: David Ahern
Cc: Jason Gunthorpe, Saeed Mahameed, Jiri Pirko, Jakub Kicinski,
Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma,
netdev, Saeed Mahameed, Nelson, Shannon
On Tue, Mar 11, 2025 at 12:23:19PM +0100, David Ahern wrote:
> On 3/6/25 12:21 AM, Jason Gunthorpe wrote:
> > On Wed, Mar 05, 2025 at 12:41:35PM -0800, Saeed Mahameed wrote:
> >
> >> How do you imagine this driver/core structure should look like? Who
> >> will be the top dir maintainer?
> >
> > I would set something like this up more like DRM. Every driver
> > maintainer gets commit rights, some rules about no uAPIs, or at least
> > other acks before merging uAPI. Use the tree for staging shared
> > branches.
>
> why no uapi? Core driver can have knowledge of h/w resources across all
> use cases. For example, our core driver supports a generid netlink based
> dump (no set operations; get and dump only so maybe that should be the
> restriction?) of all objects regardless of how created -- netdev, ib,
> etc. -- and with much more detail.
Because, we want to make sure that UAPI will be aligned with relevant
subsystems without any way to bypass them.
Thanks
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-11 11:23 ` David Ahern
2025-03-11 13:59 ` Leon Romanovsky
@ 2025-03-11 14:27 ` Nelson, Shannon
1 sibling, 0 replies; 56+ messages in thread
From: Nelson, Shannon @ 2025-03-11 14:27 UTC (permalink / raw)
To: David Ahern, Jason Gunthorpe, Saeed Mahameed
Cc: Leon Romanovsky, Jiri Pirko, Jakub Kicinski, Greg Kroah-Hartman,
Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, Christoph Hellwig, Itay Avraham, Jiri Pirko,
Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma, netdev,
Saeed Mahameed
On 3/11/2025 4:23 AM, David Ahern wrote:
> On 3/6/25 12:21 AM, Jason Gunthorpe wrote:
>>
>>> It should be something that is tightly coupled with aux, currently
>>> aux is under drivers/base/auxiliary.c I think it should move to
>>> drivers/aux/auxiliary.c and device drivers should implement their
>>> own aux buses, WH access APIs and probing/init logic under that
>>> directory e.g: drivers/aux/mlx5/..
>>
>> That makes sense to me. I would expect everything in this collection
>> to be PCI drivers spawing aux devices.
>>
>> drivers/aux_core/ or something like that, perhaps?
>>
>
> drivers/aux_core works for me; removes the 'pci' assumption and makes it
> clear the real attribute here is use of the aux bus with subsystem
> specific devices. I am still not clear on how such a branch will work -
> e.g. We will want multi-vendor review, not just merge the PR and go.
>
At the risk of getting too far into the naming-choosing rat hole... note
that even "aux_core" has the assumption that the thing is auxiliary_bus
oriented, which I don't think is a hard truth. As an example, yes
pds_core supports pds_vdpa through an auxiliary_device, but it also
supports pds_vfio_pci with no aux dev involved.
I could live with drivers/aux_core, but I'd prefer drivers/core.
sln
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-11 13:59 ` Leon Romanovsky
@ 2025-03-12 9:31 ` David Ahern
2025-03-12 10:34 ` Stanislav Fomichev
2025-03-17 12:30 ` Jason Gunthorpe
0 siblings, 2 replies; 56+ messages in thread
From: David Ahern @ 2025-03-12 9:31 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Jason Gunthorpe, Saeed Mahameed, Jiri Pirko, Jakub Kicinski,
Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma,
netdev, Saeed Mahameed, Nelson, Shannon
On 3/11/25 2:59 PM, Leon Romanovsky wrote:
> On Tue, Mar 11, 2025 at 12:23:19PM +0100, David Ahern wrote:
>> On 3/6/25 12:21 AM, Jason Gunthorpe wrote:
>>> On Wed, Mar 05, 2025 at 12:41:35PM -0800, Saeed Mahameed wrote:
>>>
>>>> How do you imagine this driver/core structure should look like? Who
>>>> will be the top dir maintainer?
>>>
>>> I would set something like this up more like DRM. Every driver
>>> maintainer gets commit rights, some rules about no uAPIs, or at least
>>> other acks before merging uAPI. Use the tree for staging shared
>>> branches.
>>
>> why no uapi? Core driver can have knowledge of h/w resources across all
>> use cases. For example, our core driver supports a generid netlink based
>> dump (no set operations; get and dump only so maybe that should be the
>> restriction?) of all objects regardless of how created -- netdev, ib,
>> etc. -- and with much more detail.
>
> Because, we want to make sure that UAPI will be aligned with relevant
> subsystems without any way to bypass them.
>
> Thanks
I hope there will be an open mind on get / dump style introspection apis
here. Devices can work support and work within limited subsystem APIs
and also allow the dumping of full essential and relevant contexts for a
device.
More specifically, I do not see netdev APIs ever recognizing RDMA
concepts like domains and memory regions. For us, everything is relative
to a domain and a region - e.g., whether a queue is created for a netdev
device or an IB QP both use the same common internal APIs. I would
prefer not to use fwctl for something so basic.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-12 9:31 ` David Ahern
@ 2025-03-12 10:34 ` Stanislav Fomichev
2025-03-14 22:34 ` David Ahern
2025-03-17 12:30 ` Jason Gunthorpe
1 sibling, 1 reply; 56+ messages in thread
From: Stanislav Fomichev @ 2025-03-12 10:34 UTC (permalink / raw)
To: David Ahern
Cc: Leon Romanovsky, Jason Gunthorpe, Saeed Mahameed, Jiri Pirko,
Jakub Kicinski, Greg Kroah-Hartman, Andy Gospodarek,
Aron Silverton, Dan Williams, Daniel Vetter, Dave Jiang,
Christoph Hellwig, Itay Avraham, Jiri Pirko, Jonathan Cameron,
Leonid Bloch, linux-cxl, linux-rdma, netdev, Saeed Mahameed,
Nelson, Shannon
On 03/12, David Ahern wrote:
> On 3/11/25 2:59 PM, Leon Romanovsky wrote:
> > On Tue, Mar 11, 2025 at 12:23:19PM +0100, David Ahern wrote:
> >> On 3/6/25 12:21 AM, Jason Gunthorpe wrote:
> >>> On Wed, Mar 05, 2025 at 12:41:35PM -0800, Saeed Mahameed wrote:
> >>>
> >>>> How do you imagine this driver/core structure should look like? Who
> >>>> will be the top dir maintainer?
> >>>
> >>> I would set something like this up more like DRM. Every driver
> >>> maintainer gets commit rights, some rules about no uAPIs, or at least
> >>> other acks before merging uAPI. Use the tree for staging shared
> >>> branches.
> >>
> >> why no uapi? Core driver can have knowledge of h/w resources across all
> >> use cases. For example, our core driver supports a generid netlink based
> >> dump (no set operations; get and dump only so maybe that should be the
> >> restriction?) of all objects regardless of how created -- netdev, ib,
> >> etc. -- and with much more detail.
> >
> > Because, we want to make sure that UAPI will be aligned with relevant
> > subsystems without any way to bypass them.
> >
> > Thanks
>
> I hope there will be an open mind on get / dump style introspection apis
> here. Devices can work support and work within limited subsystem APIs
> and also allow the dumping of full essential and relevant contexts for a
> device.
[..]
> More specifically, I do not see netdev APIs ever recognizing RDMA
> concepts like domains and memory regions. For us, everything is relative
> to a domain and a region - e.g., whether a queue is created for a netdev
> device or an IB QP both use the same common internal APIs. I would
> prefer not to use fwctl for something so basic.
What specifically do you mean here by 'memory regions'? Netdev does
recognize (as of recently, devmem) the concept of memory regions in
the form of dmabufs.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-05 18:28 ` Leon Romanovsky
2025-03-05 20:41 ` Saeed Mahameed
@ 2025-03-13 12:30 ` David Ahern
2025-03-13 12:48 ` Leon Romanovsky
1 sibling, 1 reply; 56+ messages in thread
From: David Ahern @ 2025-03-13 12:30 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Jiri Pirko, Jason Gunthorpe, Jakub Kicinski, Greg Kroah-Hartman,
Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, Christoph Hellwig, Itay Avraham, Jiri Pirko,
Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon, Sinyuk, Konstantin
On 3/5/25 7:28 PM, Leon Romanovsky wrote:
> On Wed, Mar 05, 2025 at 11:17:19AM -0700, David Ahern wrote:
>> On 3/5/25 8:08 AM, Jiri Pirko wrote:
>>> Wed, Mar 05, 2025 at 02:32:54PM +0100, jgg@nvidia.com wrote:
>>>> On Tue, Mar 04, 2025 at 04:42:03PM -0800, Jakub Kicinski wrote:
>>>>> I thought you were arguing that me opposing the addition was
>>>>> "maintainer overreach". As in me telling other parts of the kernel
>>>>> what is and isn't allowed. Do I not get a say what gets merged under
>>>>> drivers/net/ now?
>>>>
>>>> The PCI core drivers are a shared resource jointly maintained by all
>>>> the subsytems that use them. They are maintained by their respective
>>>> maintainers. Saeed/etc in this case.
>>>>
>>>> It would be inappropriate for your preferences to supersede Saeed's
>>>> when he is a maintainer of the mlx5_core driver and fwctl. Please try
>>>> and get Saeed on board with your plan.
>>>>
>>>> If the placement under drivers/net makes this confusing then we can
>>>> certainly change the directory names.
>>>
>>> According to how mlx5 driver is structured, and the rest of the advanced
>>> drivers in the same area are becoming as well, it would make sense to me
>>> to have mlx5 core in separate core directory, maintained directly by driver
>>> maintainer:
>>> drivers/core/mlx5/
>>> then each of the protocol auxiliary device lands in appropriate
>>> subsystem directory.
>>
>> +1
>>
>> This is how I have structured our drivers -- core driver for owning the
>> PCI device and hosting the APIs to communicate with hardware, an aux bus
>> and then smaller subsystem focused drivers for the aux devices that make
>> the device usable from different contexts.
>>
>> I think we are ready to start upstreaming, but I am waiting to see how
>> this falls out - to see if our core driver can land in a non-subsystem
>> specific location (e.g., drivers/core) or if it needs to go with fwctl
>> as a generic location.
>
> Do it right, and push it to drivers/core. I'm aware of at least one
> driver from huge company (not Nvidia) which is in preparation phase
> before upstreaming, and will fit nicely into this model.
>
> They have separated blocks for PCI, eth, RDMA and GPU.
>
Adding that group here after an offlist discussion with that team. If I
understand correctly, their preferred driver breakout is:
┌───────────────────────────────────────────────────────────┐
│ Platform Driver │
│ habanalabs │
└────────▲───────────────────▲────────────────────▲─────────┘
│AUX │AUX │AUX
┌────────┴────────┐ ┌────────┴────────┐ ┌─────────┴─────────┐
│ Accel Driver │ │ Ethernet Driver │ │ InfiniBand Driver │
│ habanalabs_accel│ │ habanalabs_en │ │ habanalabs_ib │
└─────────────────┘ └─────────────────┘ └───────────────────┘
So that means 3 different vendors and 3 different devices looking for a
similar auxbus based hierarchy with a core driver not buried within one
of the subsystems.
I guess at this point we just need to move forward with the proposal and
start sending patches.
Seems like drivers/core is the consensus for the core driver?
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-13 12:30 ` David Ahern
@ 2025-03-13 12:48 ` Leon Romanovsky
2025-03-13 19:59 ` Nelson, Shannon
0 siblings, 1 reply; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-13 12:48 UTC (permalink / raw)
To: David Ahern
Cc: Jiri Pirko, Jason Gunthorpe, Jakub Kicinski, Greg Kroah-Hartman,
Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, Christoph Hellwig, Itay Avraham, Jiri Pirko,
Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon, Sinyuk, Konstantin
On Thu, Mar 13, 2025 at 01:30:52PM +0100, David Ahern wrote:
> On 3/5/25 7:28 PM, Leon Romanovsky wrote:
> > On Wed, Mar 05, 2025 at 11:17:19AM -0700, David Ahern wrote:
> >> On 3/5/25 8:08 AM, Jiri Pirko wrote:
> >>> Wed, Mar 05, 2025 at 02:32:54PM +0100, jgg@nvidia.com wrote:
> >>>> On Tue, Mar 04, 2025 at 04:42:03PM -0800, Jakub Kicinski wrote:
> >>>>> I thought you were arguing that me opposing the addition was
> >>>>> "maintainer overreach". As in me telling other parts of the kernel
> >>>>> what is and isn't allowed. Do I not get a say what gets merged under
> >>>>> drivers/net/ now?
> >>>>
> >>>> The PCI core drivers are a shared resource jointly maintained by all
> >>>> the subsytems that use them. They are maintained by their respective
> >>>> maintainers. Saeed/etc in this case.
> >>>>
> >>>> It would be inappropriate for your preferences to supersede Saeed's
> >>>> when he is a maintainer of the mlx5_core driver and fwctl. Please try
> >>>> and get Saeed on board with your plan.
> >>>>
> >>>> If the placement under drivers/net makes this confusing then we can
> >>>> certainly change the directory names.
> >>>
> >>> According to how mlx5 driver is structured, and the rest of the advanced
> >>> drivers in the same area are becoming as well, it would make sense to me
> >>> to have mlx5 core in separate core directory, maintained directly by driver
> >>> maintainer:
> >>> drivers/core/mlx5/
> >>> then each of the protocol auxiliary device lands in appropriate
> >>> subsystem directory.
> >>
> >> +1
> >>
> >> This is how I have structured our drivers -- core driver for owning the
> >> PCI device and hosting the APIs to communicate with hardware, an aux bus
> >> and then smaller subsystem focused drivers for the aux devices that make
> >> the device usable from different contexts.
> >>
> >> I think we are ready to start upstreaming, but I am waiting to see how
> >> this falls out - to see if our core driver can land in a non-subsystem
> >> specific location (e.g., drivers/core) or if it needs to go with fwctl
> >> as a generic location.
> >
> > Do it right, and push it to drivers/core. I'm aware of at least one
> > driver from huge company (not Nvidia) which is in preparation phase
> > before upstreaming, and will fit nicely into this model.
> >
> > They have separated blocks for PCI, eth, RDMA and GPU.
> >
>
> Adding that group here after an offlist discussion with that team. If I
> understand correctly, their preferred driver breakout is:
>
>
> ┌───────────────────────────────────────────────────────────┐
> │ Platform Driver │
> │ habanalabs │
> └────────▲───────────────────▲────────────────────▲─────────┘
> │AUX │AUX │AUX
> ┌────────┴────────┐ ┌────────┴────────┐ ┌─────────┴─────────┐
> │ Accel Driver │ │ Ethernet Driver │ │ InfiniBand Driver │
> │ habanalabs_accel│ │ habanalabs_en │ │ habanalabs_ib │
> └─────────────────┘ └─────────────────┘ └───────────────────┘
You got it right.
>
> So that means 3 different vendors and 3 different devices looking for a
> similar auxbus based hierarchy with a core driver not buried within one
> of the subsystems.
>
> I guess at this point we just need to move forward with the proposal and
> start sending patches.
>
> Seems like drivers/core is the consensus for the core driver?
Yes, anything that is not aux_core is fine by me.
drivers/core or drivers/aux.
Thanks
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-13 12:48 ` Leon Romanovsky
@ 2025-03-13 19:59 ` Nelson, Shannon
2025-03-14 5:37 ` Greg Kroah-Hartman
2025-03-14 18:09 ` Jacob Keller
0 siblings, 2 replies; 56+ messages in thread
From: Nelson, Shannon @ 2025-03-13 19:59 UTC (permalink / raw)
To: Leon Romanovsky, David Ahern
Cc: Jiri Pirko, Jason Gunthorpe, Jakub Kicinski, Greg Kroah-Hartman,
Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, Christoph Hellwig, Itay Avraham, Jiri Pirko,
Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Sinyuk, Konstantin
On 3/13/2025 5:48 AM, Leon Romanovsky wrote:
> On Thu, Mar 13, 2025 at 01:30:52PM +0100, David Ahern wrote:
>> So that means 3 different vendors and 3 different devices looking for a
>> similar auxbus based hierarchy with a core driver not buried within one
>> of the subsystems.
>>
>> I guess at this point we just need to move forward with the proposal and
>> start sending patches.
>>
>> Seems like drivers/core is the consensus for the core driver?
>
> Yes, anything that is not aux_core is fine by me.
>
> drivers/core or drivers/aux.
Between the two of these I prefer drivers/core - I don't want to see
this tied to aux for the same reasons we don't want it tied to pci or net.
sln
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-13 19:59 ` Nelson, Shannon
@ 2025-03-14 5:37 ` Greg Kroah-Hartman
2025-03-14 18:39 ` Leon Romanovsky
2025-03-14 18:09 ` Jacob Keller
1 sibling, 1 reply; 56+ messages in thread
From: Greg Kroah-Hartman @ 2025-03-14 5:37 UTC (permalink / raw)
To: Nelson, Shannon
Cc: Leon Romanovsky, David Ahern, Jiri Pirko, Jason Gunthorpe,
Jakub Kicinski, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma,
netdev, Saeed Mahameed, Sinyuk, Konstantin
On Thu, Mar 13, 2025 at 12:59:16PM -0700, Nelson, Shannon wrote:
> On 3/13/2025 5:48 AM, Leon Romanovsky wrote:
> > On Thu, Mar 13, 2025 at 01:30:52PM +0100, David Ahern wrote:
>
>
> > > So that means 3 different vendors and 3 different devices looking for a
> > > similar auxbus based hierarchy with a core driver not buried within one
> > > of the subsystems.
> > >
> > > I guess at this point we just need to move forward with the proposal and
> > > start sending patches.
> > >
> > > Seems like drivers/core is the consensus for the core driver?
> >
> > Yes, anything that is not aux_core is fine by me.
> >
> > drivers/core or drivers/aux.
>
> Between the two of these I prefer drivers/core - I don't want to see this
> tied to aux for the same reasons we don't want it tied to pci or net.
Decades ago we tried to add drivers/core/ but I think tools really
didn't like to see "core" as a directory name. Hopefully you all don't
run into that issue here as well :)
Anyway, if you all want me to run that tree as a "neutral" third-party,
I'll be glad to do so.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-13 19:59 ` Nelson, Shannon
2025-03-14 5:37 ` Greg Kroah-Hartman
@ 2025-03-14 18:09 ` Jacob Keller
2025-03-17 12:33 ` Jason Gunthorpe
1 sibling, 1 reply; 56+ messages in thread
From: Jacob Keller @ 2025-03-14 18:09 UTC (permalink / raw)
To: Nelson, Shannon, Leon Romanovsky, David Ahern
Cc: Jiri Pirko, Jason Gunthorpe, Jakub Kicinski, Greg Kroah-Hartman,
Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, Christoph Hellwig, Itay Avraham, Jiri Pirko,
Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Sinyuk, Konstantin
On 3/13/2025 12:59 PM, Nelson, Shannon wrote:
> On 3/13/2025 5:48 AM, Leon Romanovsky wrote:
>> On Thu, Mar 13, 2025 at 01:30:52PM +0100, David Ahern wrote:
>
>
>>> So that means 3 different vendors and 3 different devices looking for a
>>> similar auxbus based hierarchy with a core driver not buried within one
>>> of the subsystems.
>>>
>>> I guess at this point we just need to move forward with the proposal and
>>> start sending patches.
>>>
>>> Seems like drivers/core is the consensus for the core driver?
>>
>> Yes, anything that is not aux_core is fine by me.
>>
>> drivers/core or drivers/aux.
>
> Between the two of these I prefer drivers/core - I don't want to see
> this tied to aux for the same reasons we don't want it tied to pci or net.
>
> sln
>
>
I'd throw my hat in for drivers/core as well, I think it makes the most
sense and is the most subsystem neutral name. Hopefully any issue with
tooling can be solved relatively easily.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-14 5:37 ` Greg Kroah-Hartman
@ 2025-03-14 18:39 ` Leon Romanovsky
0 siblings, 0 replies; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-14 18:39 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Nelson, Shannon, David Ahern, Jiri Pirko, Jason Gunthorpe,
Jakub Kicinski, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma,
netdev, Saeed Mahameed, Sinyuk, Konstantin
On Fri, Mar 14, 2025 at 06:37:11AM +0100, Greg Kroah-Hartman wrote:
> On Thu, Mar 13, 2025 at 12:59:16PM -0700, Nelson, Shannon wrote:
> > On 3/13/2025 5:48 AM, Leon Romanovsky wrote:
> > > On Thu, Mar 13, 2025 at 01:30:52PM +0100, David Ahern wrote:
> >
> >
> > > > So that means 3 different vendors and 3 different devices looking for a
> > > > similar auxbus based hierarchy with a core driver not buried within one
> > > > of the subsystems.
> > > >
> > > > I guess at this point we just need to move forward with the proposal and
> > > > start sending patches.
> > > >
> > > > Seems like drivers/core is the consensus for the core driver?
> > >
> > > Yes, anything that is not aux_core is fine by me.
> > >
> > > drivers/core or drivers/aux.
> >
> > Between the two of these I prefer drivers/core - I don't want to see this
> > tied to aux for the same reasons we don't want it tied to pci or net.
>
> Decades ago we tried to add drivers/core/ but I think tools really
> didn't like to see "core" as a directory name. Hopefully you all don't
> run into that issue here as well :)
>
> Anyway, if you all want me to run that tree as a "neutral" third-party,
> I'll be glad to do so.
Thanks for readiness, we will definitely be glad to see you on board.
Thanks
>
> thanks,
>
> greg k-h
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-12 10:34 ` Stanislav Fomichev
@ 2025-03-14 22:34 ` David Ahern
2025-03-16 7:34 ` Stanislav Fomichev
0 siblings, 1 reply; 56+ messages in thread
From: David Ahern @ 2025-03-14 22:34 UTC (permalink / raw)
To: Stanislav Fomichev
Cc: Leon Romanovsky, Jason Gunthorpe, Saeed Mahameed, Jiri Pirko,
Jakub Kicinski, Greg Kroah-Hartman, Andy Gospodarek,
Aron Silverton, Dan Williams, Daniel Vetter, Dave Jiang,
Christoph Hellwig, Itay Avraham, Jiri Pirko, Jonathan Cameron,
Leonid Bloch, linux-cxl, linux-rdma, netdev, Saeed Mahameed,
Nelson, Shannon
On 3/12/25 11:34 AM, Stanislav Fomichev wrote:
>> More specifically, I do not see netdev APIs ever recognizing RDMA
>> concepts like domains and memory regions. For us, everything is relative
>> to a domain and a region - e.g., whether a queue is created for a netdev
>> device or an IB QP both use the same common internal APIs. I would
>> prefer not to use fwctl for something so basic.
>
> What specifically do you mean here by 'memory regions'? Ne
netdev queues and flows are a subset of RDMA operations, so I mean MRs
as in:
IBV_REG_MR(3) Libibverbs Programmer's Manual
NAME
ibv_reg_mr, ibv_reg_mr_iova, ibv_reg_dmabuf_mr, ibv_dereg_mr -
register or deregister a memory region (MR)
SYNOPSIS
#include <infiniband/verbs.h>
struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr,
size_t length, int access);
struct ibv_mr *ibv_reg_mr_iova(struct ibv_pd *pd, void *addr,
size_t length, uint64_t hca_va,
int access);
struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
size_t length, uint64_t iova,
int fd, int access);
int ibv_dereg_mr(struct ibv_mr *mr);
DESCRIPTION
ibv_reg_mr() registers a memory region (MR) associated with the
protection domain pd. The MR's starting address is addr and its size is
length. The argument access describes the desired mem‐
ory protection attributes; it is either 0 or the bitwise OR of
one or more of the following flags:
...
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 6/8] fwctl: Add documentation
2025-02-28 0:26 ` [PATCH v5 6/8] fwctl: Add documentation Jason Gunthorpe
@ 2025-03-15 2:53 ` Bagas Sanjaya
0 siblings, 0 replies; 56+ messages in thread
From: Bagas Sanjaya @ 2025-03-15 2:53 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
[-- Attachment #1: Type: text/plain, Size: 14379 bytes --]
On Thu, Feb 27, 2025 at 08:26:34PM -0400, Jason Gunthorpe wrote:
> diff --git a/Documentation/userspace-api/fwctl/fwctl.rst b/Documentation/userspace-api/fwctl/fwctl.rst
> new file mode 100644
> index 00000000000000..8c586a8f677d5b
> --- /dev/null
> +++ b/Documentation/userspace-api/fwctl/fwctl.rst
> @@ -0,0 +1,284 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +===============
> +fwctl subsystem
> +===============
> +
> +:Author: Jason Gunthorpe
> +
> +Overview
> +========
> +
> +Modern devices contain extensive amounts of FW, and in many cases, are largely
> +software-defined pieces of hardware. The evolution of this approach is largely a
> +reaction to Moore's Law where a chip tape out is now highly expensive, and the
> +chip design is extremely large. Replacing fixed HW logic with a flexible and
> +tightly coupled FW/HW combination is an effective risk mitigation against chip
> +respin. Problems in the HW design can be counteracted in device FW. This is
> +especially true for devices which present a stable and backwards compatible
> +interface to the operating system driver (such as NVMe).
> +
> +The FW layer in devices has grown to incredible size and devices frequently
> +integrate clusters of fast processors to run it. For example, mlx5 devices have
> +over 30MB of FW code, and big configurations operate with over 1GB of FW managed
> +runtime state.
> +
> +The availability of such a flexible layer has created quite a variety in the
> +industry where single pieces of silicon are now configurable software-defined
> +devices and can operate in substantially different ways depending on the need.
> +Further, we often see cases where specific sites wish to operate devices in ways
> +that are highly specialized and require applications that have been tailored to
> +their unique configuration.
> +
> +Further, devices have become multi-functional and integrated to the point they
> +no longer fit neatly into the kernel's division of subsystems. Modern
> +multi-functional devices have drivers, such as bnxt/ice/mlx5/pds, that span many
> +subsystems while sharing the underlying hardware using the auxiliary device
> +system.
> +
> +All together this creates a challenge for the operating system, where devices
> +have an expansive FW environment that needs robust device-specific debugging
> +support, and FW-driven functionality that is not well suited to “generic”
> +interfaces. fwctl seeks to allow access to the full device functionality from
> +user space in the areas of debuggability, management, and first-boot/nth-boot
> +provisioning.
> +
> +fwctl is aimed at the common device design pattern where the OS and FW
> +communicate via an RPC message layer constructed with a queue or mailbox scheme.
> +In this case the driver will typically have some layer to deliver RPC messages
> +and collect RPC responses from device FW. The in-kernel subsystem drivers that
> +operate the device for its primary purposes will use these RPCs to build their
> +drivers, but devices also usually have a set of ancillary RPCs that don't really
> +fit into any specific subsystem. For example, a HW RAID controller is primarily
> +operated by the block layer but also comes with a set of RPCs to administer the
> +construction of drives within the HW RAID.
> +
> +In the past when devices were more single function, individual subsystems would
> +grow different approaches to solving some of these common problems. For instance
> +monitoring device health, manipulating its FLASH, debugging the FW,
> +provisioning, all have various unique interfaces across the kernel.
> +
> +fwctl's purpose is to define a common set of limited rules, described below,
> +that allow user space to securely construct and execute RPCs inside device FW.
> +The rules serve as an agreement between the operating system and FW on how to
> +correctly design the RPC interface. As a uAPI the subsystem provides a thin
> +layer of discovery and a generic uAPI to deliver the RPCs and collect the
> +response. It supports a system of user space libraries and tools which will
> +use this interface to control the device using the device native protocols.
> +
> +Scope of Action
> +---------------
> +
> +fwctl drivers are strictly restricted to being a way to operate the device FW.
> +It is not an avenue to access random kernel internals, or other operating system
> +SW states.
> +
> +fwctl instances must operate on a well-defined device function, and the device
> +should have a well-defined security model for what scope within the physical
> +device the function is permitted to access. For instance, the most complex PCIe
> +device today may broadly have several function-level scopes:
> +
> + 1. A privileged function with full access to the on-device global state and
> + configuration
> +
> + 2. Multiple hypervisor functions with control over itself and child functions
> + used with VMs
> +
> + 3. Multiple VM functions tightly scoped within the VM
> +
> +The device may create a logical parent/child relationship between these scopes.
> +For instance a child VM's FW may be within the scope of the hypervisor FW. It is
> +quite common in the VFIO world that the hypervisor environment has a complex
> +provisioning/profiling/configuration responsibility for the function VFIO
> +assigns to the VM.
> +
> +Further, within the function, devices often have RPC commands that fall within
> +some general scopes of action (see enum fwctl_rpc_scope):
> +
> + 1. Access to function & child configuration, FLASH, etc. that becomes live at a
> + function reset. Access to function & child runtime configuration that is
> + transparent or non-disruptive to any driver or VM.
> +
> + 2. Read-only access to function debug information that may report on FW objects
> + in the function & child, including FW objects owned by other kernel
> + subsystems.
> +
> + 3. Write access to function & child debug information strictly compatible with
> + the principles of kernel lockdown and kernel integrity protection. Triggers
> + a kernel Taint.
> +
> + 4. Full debug device access. Triggers a kernel Taint, requires CAP_SYS_RAWIO.
> +
> +User space will provide a scope label on each RPC and the kernel must enforce the
> +above CAPs and taints based on that scope. A combination of kernel and FW can
> +enforce that RPCs are placed in the correct scope by user space.
> +
> +Denied behavior
> +---------------
> +
> +There are many things this interface must not allow user space to do (without a
> +Taint or CAP), broadly derived from the principles of kernel lockdown. Some
> +examples:
> +
> + 1. DMA to/from arbitrary memory, hang the system, compromise FW integrity with
> + untrusted code, or otherwise compromise device or system security and
> + integrity.
> +
> + 2. Provide an abnormal “back door” to kernel drivers. No manipulation of kernel
> + objects owned by kernel drivers.
> +
> + 3. Directly configure or otherwise control kernel drivers. A subsystem kernel
> + driver can react to the device configuration at function reset/driver load
> + time, but otherwise must not be coupled to fwctl.
> +
> + 4. Operate the HW in a way that overlaps with the core purpose of another
> + primary kernel subsystem, such as read/write to LBAs, send/receive of
> + network packets, or operate an accelerator's data plane.
> +
> +fwctl is not a replacement for device direct access subsystems like uacce or
> +VFIO.
> +
> +Operations exposed through fwctl's non-taining interfaces should be fully
> +sharable with other users of the device. For instance exposing a RPC through
> +fwctl should never prevent a kernel subsystem from also concurrently using that
> +same RPC or hardware unit down the road. In such cases fwctl will be less
> +important than proper kernel subsystems that eventually emerge. Mistakes in this
> +area resulting in clashes will be resolved in favour of a kernel implementation.
> +
> +fwctl User API
> +==============
> +
> +.. kernel-doc:: include/uapi/fwctl/fwctl.h
> +
> +sysfs Class
> +-----------
> +
> +fwctl has a sysfs class (/sys/class/fwctl/fwctlNN/) and character devices
> +(/dev/fwctl/fwctlNN) with a simple numbered scheme. The character device
> +operates the iotcl uAPI described above.
> +
> +fwctl devices can be related to driver components in other subsystems through
> +sysfs::
> +
> + $ ls /sys/class/fwctl/fwctl0/device/infiniband/
> + ibp0s10f0
> +
> + $ ls /sys/class/infiniband/ibp0s10f0/device/fwctl/
> + fwctl0/
> +
> + $ ls /sys/devices/pci0000:00/0000:00:0a.0/fwctl/fwctl0
> + dev device power subsystem uevent
> +
> +User space Community
> +--------------------
> +
> +Drawing inspiration from nvme-cli, participating in the kernel side must come
> +with a user space in a common TBD git tree, at a minimum to usefully operate the
> +kernel driver. Providing such an implementation is a pre-condition to merging a
> +kernel driver.
> +
> +The goal is to build user space community around some of the shared problems
> +we all have, and ideally develop some common user space programs with some
> +starting themes of:
> +
> + - Device in-field debugging
> +
> + - HW provisioning
> +
> + - VFIO child device profiling before VM boot
> +
> + - Confidential Compute topics (attestation, secure provisioning)
> +
> +that stretch across all subsystems in the kernel. fwupd is a great example of
> +how an excellent user space experience can emerge out of kernel-side diversity.
> +
> +fwctl Kernel API
> +================
> +
> +.. kernel-doc:: drivers/fwctl/main.c
> + :export:
> +.. kernel-doc:: include/linux/fwctl.h
> +
> +fwctl Driver design
> +-------------------
> +
> +In many cases a fwctl driver is going to be part of a larger cross-subsystem
> +device possibly using the auxiliary_device mechanism. In that case several
> +subsystems are going to be sharing the same device and FW interface layer so the
> +device design must already provide for isolation and cooperation between kernel
> +subsystems. fwctl should fit into that same model.
> +
> +Part of the driver should include a description of how its scope restrictions
> +and security model work. The driver and FW together must ensure that RPCs
> +provided by user space are mapped to the appropriate scope. If the validation is
> +done in the driver then the validation can read a 'command effects' report from
> +the device, or hardwire the enforcement. If the validation is done in the FW,
> +then the driver should pass the fwctl_rpc_scope to the FW along with the command.
> +
> +The driver and FW must cooperate to ensure that either fwctl cannot allocate
> +any FW resources, or any resources it does allocate are freed on FD closure. A
> +driver primarily constructed around FW RPCs may find that its core PCI function
> +and RPC layer belongs under fwctl with auxiliary devices connecting to other
> +subsystems.
> +
> +Each device type must be mindful of Linux's philosophy for stable ABI. The FW
> +RPC interface does not have to meet a strictly stable ABI, but it does need to
> +meet an expectation that userspace tools that are deployed and in significant
> +use don't needlessly break. FW upgrade and kernel upgrade should keep widely
> +deployed tooling working.
> +
> +Development and debugging focused RPCs under more permissive scopes can have
> +less stabilitiy if the tools using them are only run under exceptional
> +circumstances and not for every day use of the device. Debugging tools may even
> +require exact version matching as they may require something similar to DWARF
> +debug information from the FW binary.
> +
> +Security Response
> +=================
> +
> +The kernel remains the gatekeeper for this interface. If violations of the
> +scopes, security or isolation principles are found, we have options to let
> +devices fix them with a FW update, push a kernel patch to parse and block RPC
> +commands or push a kernel patch to block entire firmware versions/devices.
> +
> +While the kernel can always directly parse and restrict RPCs, it is expected
> +that the existing kernel pattern of allowing drivers to delegate validation to
> +FW to be a useful design.
> +
> +Existing Similar Examples
> +=========================
> +
> +The approach described in this document is not a new idea. Direct, or near
> +direct device access has been offered by the kernel in different areas for
> +decades. With more devices wanting to follow this design pattern it is becoming
> +clear that it is not entirely well understood and, more importantly, the
> +security considerations are not well defined or agreed upon.
> +
> +Some examples:
> +
> + - HW RAID controllers. This includes RPCs to do things like compose drives into
> + a RAID volume, configure RAID parameters, monitor the HW and more.
> +
> + - Baseboard managers. RPCs for configuring settings in the device and more
> +
> + - NVMe vendor command capsules. nvme-cli provides access to some monitoring
> + functions that different products have defined, but more exist.
> +
> + - CXL also has a NVMe-like vendor command system.
> +
> + - DRM allows user space drivers to send commands to the device via kernel
> + mediation
> +
> + - RDMA allows user space drivers to directly push commands to the device
> + without kernel involvement
> +
> + - Various “raw” APIs, raw HID (SDL2), raw USB, NVMe Generic Interface, etc.
> +
> +The first 4 are examples of areas that fwctl intends to cover. The latter three
> +are examples of denied behavior as they fully overlap with the primary purpose
> +of a kernel subsystem.
> +
> +Some key lessons learned from these past efforts are the importance of having a
> +common user space project to use as a pre-condition for obtaining a kernel
> +driver. Developing good community around useful software in user space is key to
> +getting companies to fund participation to enable their products.
The doc looks good, thanks!
Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>
--
An old man doll... just what I always wanted! - Clara
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-14 22:34 ` David Ahern
@ 2025-03-16 7:34 ` Stanislav Fomichev
0 siblings, 0 replies; 56+ messages in thread
From: Stanislav Fomichev @ 2025-03-16 7:34 UTC (permalink / raw)
To: David Ahern
Cc: Leon Romanovsky, Jason Gunthorpe, Saeed Mahameed, Jiri Pirko,
Jakub Kicinski, Greg Kroah-Hartman, Andy Gospodarek,
Aron Silverton, Dan Williams, Daniel Vetter, Dave Jiang,
Christoph Hellwig, Itay Avraham, Jiri Pirko, Jonathan Cameron,
Leonid Bloch, linux-cxl, linux-rdma, netdev, Saeed Mahameed,
Nelson, Shannon
On 03/14, David Ahern wrote:
> On 3/12/25 11:34 AM, Stanislav Fomichev wrote:
> >> More specifically, I do not see netdev APIs ever recognizing RDMA
> >> concepts like domains and memory regions. For us, everything is relative
> >> to a domain and a region - e.g., whether a queue is created for a netdev
> >> device or an IB QP both use the same common internal APIs. I would
> >> prefer not to use fwctl for something so basic.
> >
> > What specifically do you mean here by 'memory regions'? Ne
>
> netdev queues and flows are a subset of RDMA operations, so I mean MRs
> as in:
>
> IBV_REG_MR(3) Libibverbs Programmer's Manual
>
>
> NAME
> ibv_reg_mr, ibv_reg_mr_iova, ibv_reg_dmabuf_mr, ibv_dereg_mr -
> register or deregister a memory region (MR)
>
> SYNOPSIS
> #include <infiniband/verbs.h>
>
> struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr,
> size_t length, int access);
>
> struct ibv_mr *ibv_reg_mr_iova(struct ibv_pd *pd, void *addr,
> size_t length, uint64_t hca_va,
> int access);
>
> struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
> size_t length, uint64_t iova,
> int fd, int access);
>
> int ibv_dereg_mr(struct ibv_mr *mr);
>
> DESCRIPTION
> ibv_reg_mr() registers a memory region (MR) associated with the
> protection domain pd. The MR's starting address is addr and its size is
> length. The argument access describes the desired mem‐
> ory protection attributes; it is either 0 or the bitwise OR of
> one or more of the following flags:
>
> ...
Sure, and netdev has:
-
name: bind-rx
doc: Bind dmabuf to netdev
attribute-set: dmabuf
flags: [ admin-perm ]
do:
request:
attributes:
- ifindex
- fd
- queues
reply:
attributes:
- id
Which accepts dmabuf fd. Looks very similar to ibv_reg_dmabuf_mr to me.
And I think we can safely ignore ibv_reg_mr_iova which needs things
like proprietary nvidia-peermem to function.
My point is: I don't think netdev is as opposed to memory regions as
you think it is. As long as you come up with sensible new UAPI and
as long as everything is in the open, it's up for discussion.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-12 9:31 ` David Ahern
2025-03-12 10:34 ` Stanislav Fomichev
@ 2025-03-17 12:30 ` Jason Gunthorpe
1 sibling, 0 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-03-17 12:30 UTC (permalink / raw)
To: David Ahern
Cc: Leon Romanovsky, Saeed Mahameed, Jiri Pirko, Jakub Kicinski,
Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma,
netdev, Saeed Mahameed, Nelson, Shannon
On Wed, Mar 12, 2025 at 10:31:51AM +0100, David Ahern wrote:
> I hope there will be an open mind on get / dump style introspection apis
> here. Devices can work support and work within limited subsystem APIs
> and also allow the dumping of full essential and relevant contexts for a
> device.
We have three places to put dumpables like you are talking about,
fwctl, netdev and rdma
rdma already has a robust dumper for these kinds of objects.
netdev drivers have a few options, and people use debugfs there.
fwctl is supposed to let you dump the FW side view of the sytem.
Do you *really* need another one? It sounds excessive to me.
> More specifically, I do not see netdev APIs ever recognizing RDMA
> concepts like domains and memory regions.
Why not? If netdev is going to provide the same detailed view of the
HW state we expect from fwctl it will need to provide ways to dump
the actualy underlying HW objects, whatever they may be.
It is not like building a netdev driver on top of RDMA is anything
new, mlx5 does it as well.
> For us, everything is relative to a domain and a region - e.g.,
> whether a queue is created for a netdev device or an IB QP both use
> the same common internal APIs. I would prefer not to use fwctl for
> something so basic.
It is probably the right answer though.
Jason
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-14 18:09 ` Jacob Keller
@ 2025-03-17 12:33 ` Jason Gunthorpe
2025-03-17 19:00 ` David Ahern
0 siblings, 1 reply; 56+ messages in thread
From: Jason Gunthorpe @ 2025-03-17 12:33 UTC (permalink / raw)
To: Jacob Keller
Cc: Nelson, Shannon, Leon Romanovsky, David Ahern, Jiri Pirko,
Jakub Kicinski, Greg Kroah-Hartman, Andy Gospodarek,
Aron Silverton, Dan Williams, Daniel Vetter, Dave Jiang,
Christoph Hellwig, Itay Avraham, Jiri Pirko, Jonathan Cameron,
Leonid Bloch, linux-cxl, linux-rdma, netdev, Saeed Mahameed,
Sinyuk, Konstantin
On Fri, Mar 14, 2025 at 11:09:47AM -0700, Jacob Keller wrote:
> I'd throw my hat in for drivers/core as well, I think it makes the most
> sense and is the most subsystem neutral name. Hopefully any issue with
> tooling can be solved relatively easily.
Given Greg's point about core dump files sometimes being in .gitignore
maybe "shared_core", or something along that path is a better name?
Jason
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-17 12:33 ` Jason Gunthorpe
@ 2025-03-17 19:00 ` David Ahern
2025-03-17 20:33 ` Keller, Jacob E
0 siblings, 1 reply; 56+ messages in thread
From: David Ahern @ 2025-03-17 19:00 UTC (permalink / raw)
To: Jason Gunthorpe, Jacob Keller
Cc: Nelson, Shannon, Leon Romanovsky, Jiri Pirko, Jakub Kicinski,
Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton, Dan Williams,
Daniel Vetter, Dave Jiang, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch, linux-cxl, linux-rdma,
netdev, Saeed Mahameed, Sinyuk, Konstantin
On 3/17/25 1:33 PM, Jason Gunthorpe wrote:
> On Fri, Mar 14, 2025 at 11:09:47AM -0700, Jacob Keller wrote:
>
>> I'd throw my hat in for drivers/core as well, I think it makes the most
>> sense and is the most subsystem neutral name. Hopefully any issue with
>> tooling can be solved relatively easily.
>
> Given Greg's point about core dump files sometimes being in .gitignore
> maybe "shared_core", or something along that path is a better name?
>
Not seeing the conflict on drivers/core:
$ find . -name .gitignore | xargs grep core
./tools/testing/selftests/powerpc/ptrace/.gitignore:core-pkey
./tools/testing/selftests/cgroup/.gitignore:test_core
./tools/testing/selftests/mincore/.gitignore:mincore_selftest
./arch/mips/crypto/.gitignore:poly1305-core.S
./arch/arm/crypto/.gitignore:aesbs-core.S
./arch/arm/crypto/.gitignore:sha256-core.S
./arch/arm/crypto/.gitignore:sha512-core.S
./arch/arm/crypto/.gitignore:poly1305-core.S
./arch/arm64/crypto/.gitignore:sha256-core.S
./arch/arm64/crypto/.gitignore:sha512-core.S
./arch/arm64/crypto/.gitignore:poly1305-core.S
^ permalink raw reply [flat|nested] 56+ messages in thread
* RE: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-17 19:00 ` David Ahern
@ 2025-03-17 20:33 ` Keller, Jacob E
2025-03-18 13:20 ` Greg Kroah-Hartman
0 siblings, 1 reply; 56+ messages in thread
From: Keller, Jacob E @ 2025-03-17 20:33 UTC (permalink / raw)
To: David Ahern, Jason Gunthorpe
Cc: Nelson, Shannon, Leon Romanovsky, Jiri Pirko, Jakub Kicinski,
Greg Kroah-Hartman, Andy Gospodarek, Aron Silverton,
Williams, Dan J, Daniel Vetter, Jiang, Dave, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Leonid Bloch,
linux-cxl@vger.kernel.org, linux-rdma@vger.kernel.org,
netdev@vger.kernel.org, Saeed Mahameed, Sinyuk, Konstantin
> -----Original Message-----
> From: David Ahern <dsahern@kernel.org>
> Sent: Monday, March 17, 2025 12:01 PM
> To: Jason Gunthorpe <jgg@nvidia.com>; Keller, Jacob E
> <jacob.e.keller@intel.com>
> Cc: Nelson, Shannon <shannon.nelson@amd.com>; Leon Romanovsky
> <leon@kernel.org>; Jiri Pirko <jiri@resnulli.us>; Jakub Kicinski
> <kuba@kernel.org>; Greg Kroah-Hartman <gregkh@linuxfoundation.org>; Andy
> Gospodarek <andrew.gospodarek@broadcom.com>; Aron Silverton
> <aron.silverton@oracle.com>; Williams, Dan J <dan.j.williams@intel.com>; Daniel
> Vetter <daniel.vetter@ffwll.ch>; Jiang, Dave <dave.jiang@intel.com>; Christoph
> Hellwig <hch@infradead.org>; Itay Avraham <itayavr@nvidia.com>; Jiri Pirko
> <jiri@nvidia.com>; Jonathan Cameron <Jonathan.Cameron@huawei.com>;
> Leonid Bloch <lbloch@nvidia.com>; linux-cxl@vger.kernel.org; linux-
> rdma@vger.kernel.org; netdev@vger.kernel.org; Saeed Mahameed
> <saeedm@nvidia.com>; Sinyuk, Konstantin <konstantin.sinyuk@intel.com>
> Subject: Re: [PATCH v5 0/8] Introduce fwctl subystem
>
> On 3/17/25 1:33 PM, Jason Gunthorpe wrote:
> > On Fri, Mar 14, 2025 at 11:09:47AM -0700, Jacob Keller wrote:
> >
> >> I'd throw my hat in for drivers/core as well, I think it makes the most
> >> sense and is the most subsystem neutral name. Hopefully any issue with
> >> tooling can be solved relatively easily.
> >
> > Given Greg's point about core dump files sometimes being in .gitignore
> > maybe "shared_core", or something along that path is a better name?
> >
>
> Not seeing the conflict on drivers/core:
>
> $ find . -name .gitignore | xargs grep core
> ./tools/testing/selftests/powerpc/ptrace/.gitignore:core-pkey
> ./tools/testing/selftests/cgroup/.gitignore:test_core
> ./tools/testing/selftests/mincore/.gitignore:mincore_selftest
> ./arch/mips/crypto/.gitignore:poly1305-core.S
> ./arch/arm/crypto/.gitignore:aesbs-core.S
> ./arch/arm/crypto/.gitignore:sha256-core.S
> ./arch/arm/crypto/.gitignore:sha512-core.S
> ./arch/arm/crypto/.gitignore:poly1305-core.S
> ./arch/arm64/crypto/.gitignore:sha256-core.S
> ./arch/arm64/crypto/.gitignore:sha512-core.S
> ./arch/arm64/crypto/.gitignore:poly1305-core.S
I would guess its people putting core in their own ignore lists, not necessarily what we commit to the kernel tree.
Thanks,
Jake
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-17 20:33 ` Keller, Jacob E
@ 2025-03-18 13:20 ` Greg Kroah-Hartman
2025-03-18 13:25 ` Jason Gunthorpe
0 siblings, 1 reply; 56+ messages in thread
From: Greg Kroah-Hartman @ 2025-03-18 13:20 UTC (permalink / raw)
To: Keller, Jacob E
Cc: David Ahern, Jason Gunthorpe, Nelson, Shannon, Leon Romanovsky,
Jiri Pirko, Jakub Kicinski, Andy Gospodarek, Aron Silverton,
Williams, Dan J, Daniel Vetter, Jiang, Dave, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Leonid Bloch,
linux-cxl@vger.kernel.org, linux-rdma@vger.kernel.org,
netdev@vger.kernel.org, Saeed Mahameed, Sinyuk, Konstantin
On Mon, Mar 17, 2025 at 08:33:04PM +0000, Keller, Jacob E wrote:
>
>
> > -----Original Message-----
> > From: David Ahern <dsahern@kernel.org>
> > Sent: Monday, March 17, 2025 12:01 PM
> > To: Jason Gunthorpe <jgg@nvidia.com>; Keller, Jacob E
> > <jacob.e.keller@intel.com>
> > Cc: Nelson, Shannon <shannon.nelson@amd.com>; Leon Romanovsky
> > <leon@kernel.org>; Jiri Pirko <jiri@resnulli.us>; Jakub Kicinski
> > <kuba@kernel.org>; Greg Kroah-Hartman <gregkh@linuxfoundation.org>; Andy
> > Gospodarek <andrew.gospodarek@broadcom.com>; Aron Silverton
> > <aron.silverton@oracle.com>; Williams, Dan J <dan.j.williams@intel.com>; Daniel
> > Vetter <daniel.vetter@ffwll.ch>; Jiang, Dave <dave.jiang@intel.com>; Christoph
> > Hellwig <hch@infradead.org>; Itay Avraham <itayavr@nvidia.com>; Jiri Pirko
> > <jiri@nvidia.com>; Jonathan Cameron <Jonathan.Cameron@huawei.com>;
> > Leonid Bloch <lbloch@nvidia.com>; linux-cxl@vger.kernel.org; linux-
> > rdma@vger.kernel.org; netdev@vger.kernel.org; Saeed Mahameed
> > <saeedm@nvidia.com>; Sinyuk, Konstantin <konstantin.sinyuk@intel.com>
> > Subject: Re: [PATCH v5 0/8] Introduce fwctl subystem
> >
> > On 3/17/25 1:33 PM, Jason Gunthorpe wrote:
> > > On Fri, Mar 14, 2025 at 11:09:47AM -0700, Jacob Keller wrote:
> > >
> > >> I'd throw my hat in for drivers/core as well, I think it makes the most
> > >> sense and is the most subsystem neutral name. Hopefully any issue with
> > >> tooling can be solved relatively easily.
> > >
> > > Given Greg's point about core dump files sometimes being in .gitignore
> > > maybe "shared_core", or something along that path is a better name?
> > >
> >
> > Not seeing the conflict on drivers/core:
> >
> > $ find . -name .gitignore | xargs grep core
> > ./tools/testing/selftests/powerpc/ptrace/.gitignore:core-pkey
> > ./tools/testing/selftests/cgroup/.gitignore:test_core
> > ./tools/testing/selftests/mincore/.gitignore:mincore_selftest
> > ./arch/mips/crypto/.gitignore:poly1305-core.S
> > ./arch/arm/crypto/.gitignore:aesbs-core.S
> > ./arch/arm/crypto/.gitignore:sha256-core.S
> > ./arch/arm/crypto/.gitignore:sha512-core.S
> > ./arch/arm/crypto/.gitignore:poly1305-core.S
> > ./arch/arm64/crypto/.gitignore:sha256-core.S
> > ./arch/arm64/crypto/.gitignore:sha512-core.S
> > ./arch/arm64/crypto/.gitignore:poly1305-core.S
>
> I would guess its people putting core in their own ignore lists, not necessarily what we commit to the kernel tree.
Yes, note, the issue came up in the 2.5.x kernel days, _WAY_ before we
had git, so this wasn't a git issue. I'm all for "drivers/core/" but
note, that really looks like "the driver core" area of the kernel, so
maybe pick a different name?
Maybe drivers/lib/ as this is common stuff for a variety of different
drivers? I don't know, naming is hard, sorry. I've been defaulting to
just using dutch words for projects these days as they don't seem to be
used much. Hey, that might work here, drivers/kern/ perhaps? Nah...
thanks,
greg k-h
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-18 13:20 ` Greg Kroah-Hartman
@ 2025-03-18 13:25 ` Jason Gunthorpe
2025-03-18 15:39 ` Dave Jiang
2025-03-18 22:07 ` Keller, Jacob E
0 siblings, 2 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-03-18 13:25 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Keller, Jacob E, David Ahern, Nelson, Shannon, Leon Romanovsky,
Jiri Pirko, Jakub Kicinski, Andy Gospodarek, Aron Silverton,
Williams, Dan J, Daniel Vetter, Jiang, Dave, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Leonid Bloch,
linux-cxl@vger.kernel.org, linux-rdma@vger.kernel.org,
netdev@vger.kernel.org, Saeed Mahameed, Sinyuk, Konstantin
On Tue, Mar 18, 2025 at 02:20:45PM +0100, Greg Kroah-Hartman wrote:
> Yes, note, the issue came up in the 2.5.x kernel days, _WAY_ before we
> had git, so this wasn't a git issue. I'm all for "drivers/core/" but
> note, that really looks like "the driver core" area of the kernel, so
> maybe pick a different name?
Yeah, +1. We have lots of places calling what is in drivers/base 'core'.
Jason
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-18 13:25 ` Jason Gunthorpe
@ 2025-03-18 15:39 ` Dave Jiang
2025-03-18 16:06 ` Greg Kroah-Hartman
2025-03-18 22:07 ` Keller, Jacob E
1 sibling, 1 reply; 56+ messages in thread
From: Dave Jiang @ 2025-03-18 15:39 UTC (permalink / raw)
To: Jason Gunthorpe, Greg Kroah-Hartman
Cc: Keller, Jacob E, David Ahern, Nelson, Shannon, Leon Romanovsky,
Jiri Pirko, Jakub Kicinski, Andy Gospodarek, Aron Silverton,
Williams, Dan J, Daniel Vetter, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch,
linux-cxl@vger.kernel.org, linux-rdma@vger.kernel.org,
netdev@vger.kernel.org, Saeed Mahameed, Sinyuk, Konstantin
On 3/18/25 6:25 AM, Jason Gunthorpe wrote:
> On Tue, Mar 18, 2025 at 02:20:45PM +0100, Greg Kroah-Hartman wrote:
>
>> Yes, note, the issue came up in the 2.5.x kernel days, _WAY_ before we
>> had git, so this wasn't a git issue. I'm all for "drivers/core/" but
>> note, that really looks like "the driver core" area of the kernel, so
>> maybe pick a different name?
>
> Yeah, +1. We have lots of places calling what is in drivers/base 'core'.
just throwing in my 2c
drivers/main
drivers/common
drivers/primary
>
> Jason
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-18 15:39 ` Dave Jiang
@ 2025-03-18 16:06 ` Greg Kroah-Hartman
2025-03-19 5:48 ` Przemek Kitszel
2025-03-19 8:17 ` Leon Romanovsky
0 siblings, 2 replies; 56+ messages in thread
From: Greg Kroah-Hartman @ 2025-03-18 16:06 UTC (permalink / raw)
To: Dave Jiang
Cc: Jason Gunthorpe, Keller, Jacob E, David Ahern, Nelson, Shannon,
Leon Romanovsky, Jiri Pirko, Jakub Kicinski, Andy Gospodarek,
Aron Silverton, Williams, Dan J, Daniel Vetter, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Leonid Bloch,
linux-cxl@vger.kernel.org, linux-rdma@vger.kernel.org,
netdev@vger.kernel.org, Saeed Mahameed, Sinyuk, Konstantin
On Tue, Mar 18, 2025 at 08:39:50AM -0700, Dave Jiang wrote:
>
>
> On 3/18/25 6:25 AM, Jason Gunthorpe wrote:
> > On Tue, Mar 18, 2025 at 02:20:45PM +0100, Greg Kroah-Hartman wrote:
> >
> >> Yes, note, the issue came up in the 2.5.x kernel days, _WAY_ before we
> >> had git, so this wasn't a git issue. I'm all for "drivers/core/" but
> >> note, that really looks like "the driver core" area of the kernel, so
> >> maybe pick a different name?
> >
> > Yeah, +1. We have lots of places calling what is in drivers/base 'core'.
>
> just throwing in my 2c
>
> drivers/main
Implies the "driver core"
> drivers/common
lib/ maybe?
> drivers/primary
It's not going to be the primary drivers for my laptop :)
Naming is hard. Let's see some code first...
greg k-h
^ permalink raw reply [flat|nested] 56+ messages in thread
* RE: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-18 13:25 ` Jason Gunthorpe
2025-03-18 15:39 ` Dave Jiang
@ 2025-03-18 22:07 ` Keller, Jacob E
1 sibling, 0 replies; 56+ messages in thread
From: Keller, Jacob E @ 2025-03-18 22:07 UTC (permalink / raw)
To: Jason Gunthorpe, Greg Kroah-Hartman
Cc: David Ahern, Nelson, Shannon, Leon Romanovsky, Jiri Pirko,
Jakub Kicinski, Andy Gospodarek, Aron Silverton, Williams, Dan J,
Daniel Vetter, Jiang, Dave, Christoph Hellwig, Itay Avraham,
Jiri Pirko, Jonathan Cameron, Leonid Bloch,
linux-cxl@vger.kernel.org, linux-rdma@vger.kernel.org,
netdev@vger.kernel.org, Saeed Mahameed, Sinyuk, Konstantin
> -----Original Message-----
> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, March 18, 2025 6:25 AM
> To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Keller, Jacob E <jacob.e.keller@intel.com>; David Ahern
> <dsahern@kernel.org>; Nelson, Shannon <shannon.nelson@amd.com>; Leon
> Romanovsky <leon@kernel.org>; Jiri Pirko <jiri@resnulli.us>; Jakub Kicinski
> <kuba@kernel.org>; Andy Gospodarek <andrew.gospodarek@broadcom.com>;
> Aron Silverton <aron.silverton@oracle.com>; Williams, Dan J
> <dan.j.williams@intel.com>; Daniel Vetter <daniel.vetter@ffwll.ch>; Jiang, Dave
> <dave.jiang@intel.com>; Christoph Hellwig <hch@infradead.org>; Itay Avraham
> <itayavr@nvidia.com>; Jiri Pirko <jiri@nvidia.com>; Jonathan Cameron
> <Jonathan.Cameron@huawei.com>; Leonid Bloch <lbloch@nvidia.com>; linux-
> cxl@vger.kernel.org; linux-rdma@vger.kernel.org; netdev@vger.kernel.org; Saeed
> Mahameed <saeedm@nvidia.com>; Sinyuk, Konstantin
> <konstantin.sinyuk@intel.com>
> Subject: Re: [PATCH v5 0/8] Introduce fwctl subystem
>
> On Tue, Mar 18, 2025 at 02:20:45PM +0100, Greg Kroah-Hartman wrote:
>
> > Yes, note, the issue came up in the 2.5.x kernel days, _WAY_ before we
> > had git, so this wasn't a git issue. I'm all for "drivers/core/" but
> > note, that really looks like "the driver core" area of the kernel, so
> > maybe pick a different name?
>
> Yeah, +1. We have lots of places calling what is in drivers/base 'core'.
>
> Jason
Core is quite often used like "the core of the kernel that is non-driver stuff", i.e. "the networking core", "the PTP core", etc.
Naming things is indeed hard.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-18 16:06 ` Greg Kroah-Hartman
@ 2025-03-19 5:48 ` Przemek Kitszel
2025-03-19 8:14 ` Leon Romanovsky
2025-03-19 8:17 ` Leon Romanovsky
1 sibling, 1 reply; 56+ messages in thread
From: Przemek Kitszel @ 2025-03-19 5:48 UTC (permalink / raw)
To: Greg Kroah-Hartman, Jason Gunthorpe
Cc: Dave Jiang, Keller, Jacob E, David Ahern, Nelson, Shannon,
Leon Romanovsky, Jiri Pirko, Jakub Kicinski, Andy Gospodarek,
Aron Silverton, Williams, Dan J, Daniel Vetter, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Leonid Bloch,
linux-cxl@vger.kernel.org, linux-rdma@vger.kernel.org,
netdev@vger.kernel.org, Saeed Mahameed, Sinyuk, Konstantin
On 3/18/25 17:06, Greg Kroah-Hartman wrote:
> On Tue, Mar 18, 2025 at 08:39:50AM -0700, Dave Jiang wrote:
>>
>>
>> On 3/18/25 6:25 AM, Jason Gunthorpe wrote:
>>> On Tue, Mar 18, 2025 at 02:20:45PM +0100, Greg Kroah-Hartman wrote:
>>>
>>>> Yes, note, the issue came up in the 2.5.x kernel days, _WAY_ before we
>>>> had git, so this wasn't a git issue. I'm all for "drivers/core/" but
>>>> note, that really looks like "the driver core" area of the kernel, so
>>>> maybe pick a different name?
>>>
>>> Yeah, +1. We have lots of places calling what is in drivers/base 'core'.
>>
>> just throwing in my 2c
>>
>> drivers/main
>
> Implies the "driver core"
>
>> drivers/common
>
> lib/ maybe?
>
>> drivers/primary
>
> It's not going to be the primary drivers for my laptop :)
>
> Naming is hard. Let's see some code first...
>
> greg k-h
>
"platfrom" would also suggest a wider thing, so:
"complex"?
anyway, I don't like fwctl, so maybe "fwctl" to at least reveal the real
reason :P
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-19 5:48 ` Przemek Kitszel
@ 2025-03-19 8:14 ` Leon Romanovsky
2025-03-19 10:46 ` Przemek Kitszel
0 siblings, 1 reply; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-19 8:14 UTC (permalink / raw)
To: Przemek Kitszel
Cc: Greg Kroah-Hartman, Jason Gunthorpe, Dave Jiang, Keller, Jacob E,
David Ahern, Nelson, Shannon, Jiri Pirko, Jakub Kicinski,
Andy Gospodarek, Aron Silverton, Williams, Dan J, Daniel Vetter,
Christoph Hellwig, Itay Avraham, Jiri Pirko, Jonathan Cameron,
Leonid Bloch, linux-cxl@vger.kernel.org,
linux-rdma@vger.kernel.org, netdev@vger.kernel.org,
Saeed Mahameed, Sinyuk, Konstantin
On Wed, Mar 19, 2025 at 06:48:48AM +0100, Przemek Kitszel wrote:
> On 3/18/25 17:06, Greg Kroah-Hartman wrote:
> > On Tue, Mar 18, 2025 at 08:39:50AM -0700, Dave Jiang wrote:
> > >
> > >
> > > On 3/18/25 6:25 AM, Jason Gunthorpe wrote:
> > > > On Tue, Mar 18, 2025 at 02:20:45PM +0100, Greg Kroah-Hartman wrote:
> > > >
> > > > > Yes, note, the issue came up in the 2.5.x kernel days, _WAY_ before we
> > > > > had git, so this wasn't a git issue. I'm all for "drivers/core/" but
> > > > > note, that really looks like "the driver core" area of the kernel, so
> > > > > maybe pick a different name?
> > > >
> > > > Yeah, +1. We have lots of places calling what is in drivers/base 'core'.
> > >
> > > just throwing in my 2c
> > >
> > > drivers/main
> >
> > Implies the "driver core"
> >
> > > drivers/common
> >
> > lib/ maybe?
> >
> > > drivers/primary
> >
> > It's not going to be the primary drivers for my laptop :)
> >
> > Naming is hard. Let's see some code first...
> >
> > greg k-h
> >
>
> "platfrom" would also suggest a wider thing, so:
> "complex"?
>
> anyway, I don't like fwctl, so maybe "fwctl" to at least reveal the real
> reason :P
We are discussing where to move XXX_core drivers which historically were
located in drivers/net/ethernet/XXX/, see this idea https://lore.kernel.org/all/20250211075553.GF17863@unreal/
FWCTL is unrelated to this discussion and you are not forced to use it
if you don't like it.
Thanks
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-18 16:06 ` Greg Kroah-Hartman
2025-03-19 5:48 ` Przemek Kitszel
@ 2025-03-19 8:17 ` Leon Romanovsky
1 sibling, 0 replies; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-19 8:17 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Dave Jiang, Jason Gunthorpe, Keller, Jacob E, David Ahern,
Nelson, Shannon, Jiri Pirko, Jakub Kicinski, Andy Gospodarek,
Aron Silverton, Williams, Dan J, Daniel Vetter, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Leonid Bloch,
linux-cxl@vger.kernel.org, linux-rdma@vger.kernel.org,
netdev@vger.kernel.org, Saeed Mahameed, Sinyuk, Konstantin
On Tue, Mar 18, 2025 at 05:06:17PM +0100, Greg Kroah-Hartman wrote:
> On Tue, Mar 18, 2025 at 08:39:50AM -0700, Dave Jiang wrote:
> >
> >
> > On 3/18/25 6:25 AM, Jason Gunthorpe wrote:
> > > On Tue, Mar 18, 2025 at 02:20:45PM +0100, Greg Kroah-Hartman wrote:
> > >
> > >> Yes, note, the issue came up in the 2.5.x kernel days, _WAY_ before we
> > >> had git, so this wasn't a git issue. I'm all for "drivers/core/" but
> > >> note, that really looks like "the driver core" area of the kernel, so
> > >> maybe pick a different name?
> > >
> > > Yeah, +1. We have lots of places calling what is in drivers/base 'core'.
> >
> > just throwing in my 2c
> >
> > drivers/main
>
> Implies the "driver core"
>
> > drivers/common
>
> lib/ maybe?
>
> > drivers/primary
>
> It's not going to be the primary drivers for my laptop :)
>
> Naming is hard. Let's see some code first...
Yes, let's do name contest later when code will come. There are multiple
companies already started to work on it and my hope that it will be ready
for next merge cycle.
Thanks
>
> greg k-h
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-19 8:14 ` Leon Romanovsky
@ 2025-03-19 10:46 ` Przemek Kitszel
2025-03-19 11:22 ` Leon Romanovsky
0 siblings, 1 reply; 56+ messages in thread
From: Przemek Kitszel @ 2025-03-19 10:46 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Greg Kroah-Hartman, Jason Gunthorpe, Dave Jiang, Keller, Jacob E,
David Ahern, Nelson, Shannon, Jiri Pirko, Jakub Kicinski,
Andy Gospodarek, Aron Silverton, Williams, Dan J, Daniel Vetter,
Christoph Hellwig, Itay Avraham, Jiri Pirko, Jonathan Cameron,
Leonid Bloch, linux-cxl@vger.kernel.org,
linux-rdma@vger.kernel.org, netdev@vger.kernel.org,
Saeed Mahameed, Sinyuk, Konstantin
> We are discussing where to move XXX_core drivers which historically were
> located in drivers/net/ethernet/XXX/, see this idea https://lore.kernel.org/all/20250211075553.GF17863@unreal/
yes, I know
> FWCTL is unrelated to this discussion and
it is related,
I see the move as a workaround for "netdev maintainer complains more
than I would like", sorry :P
> you are not forced to use it
> if you don't like it.
I know, time will tell if I would be forced to use it to don't
let competitors to make a short term advantage by going by the
"who cares, we have fwctl" route.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-03-19 10:46 ` Przemek Kitszel
@ 2025-03-19 11:22 ` Leon Romanovsky
0 siblings, 0 replies; 56+ messages in thread
From: Leon Romanovsky @ 2025-03-19 11:22 UTC (permalink / raw)
To: Przemek Kitszel
Cc: Greg Kroah-Hartman, Jason Gunthorpe, Dave Jiang, Keller, Jacob E,
David Ahern, Nelson, Shannon, Jiri Pirko, Jakub Kicinski,
Andy Gospodarek, Aron Silverton, Williams, Dan J, Daniel Vetter,
Christoph Hellwig, Itay Avraham, Jiri Pirko, Jonathan Cameron,
Leonid Bloch, linux-cxl@vger.kernel.org,
linux-rdma@vger.kernel.org, netdev@vger.kernel.org,
Saeed Mahameed, Sinyuk, Konstantin
On Wed, Mar 19, 2025 at 11:46:15AM +0100, Przemek Kitszel wrote:
>
> > We are discussing where to move XXX_core drivers which historically were
> > located in drivers/net/ethernet/XXX/, see this idea https://lore.kernel.org/all/20250211075553.GF17863@unreal/
>
> yes, I know
>
> > FWCTL is unrelated to this discussion and
>
> it is related,
> I see the move as a workaround for "netdev maintainer complains more
> than I would like", sorry :P
No, it is not. There is no technical reason to keep non-netdev code in
netdev subsystem. If you have an explanation why PCI-to-AUX, VDPA, VFIO
and RDMA logic MUST be under netdev subsystem, please speak it now.
Otherwise please keep brigading out of this discussion.
Thanks
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v5 0/8] Introduce fwctl subystem
2025-02-28 0:26 [PATCH v5 0/8] Introduce fwctl subystem Jason Gunthorpe
` (8 preceding siblings ...)
2025-03-04 1:53 ` [PATCH v5 0/8] Introduce fwctl subystem Jakub Kicinski
@ 2025-03-20 23:22 ` Jason Gunthorpe
9 siblings, 0 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2025-03-20 23:22 UTC (permalink / raw)
To: Andy Gospodarek, Aron Silverton, Dan Williams, Daniel Vetter,
Dave Jiang, David Ahern, Greg Kroah-Hartman, Christoph Hellwig,
Itay Avraham, Jiri Pirko, Jonathan Cameron, Jakub Kicinski,
Leonid Bloch, Leon Romanovsky, linux-cxl, linux-rdma, netdev,
Saeed Mahameed, Nelson, Shannon
On Thu, Feb 27, 2025 at 08:26:28PM -0400, Jason Gunthorpe wrote:
> Jason Gunthorpe (6):
> fwctl: Add basic structure for a class subsystem with a cdev
> fwctl: Basic ioctl dispatch for the character device
> fwctl: FWCTL_INFO to return basic information about the device
> taint: Add TAINT_FWCTL
> fwctl: FWCTL_RPC to execute a Remote Procedure Call to device firmware
> fwctl: Add documentation
>
> Saeed Mahameed (2):
> fwctl/mlx5: Support for communicating with mlx5 fw
> mlx5: Create an auxiliary device for fwctl_mlx5
Applied - it has been in linux-next for two weeks already.
Jason
^ permalink raw reply [flat|nested] 56+ messages in thread
end of thread, other threads:[~2025-03-20 23:22 UTC | newest]
Thread overview: 56+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-02-28 0:26 [PATCH v5 0/8] Introduce fwctl subystem Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 1/8] fwctl: Add basic structure for a class subsystem with a cdev Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 2/8] fwctl: Basic ioctl dispatch for the character device Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 3/8] fwctl: FWCTL_INFO to return basic information about the device Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 4/8] taint: Add TAINT_FWCTL Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 5/8] fwctl: FWCTL_RPC to execute a Remote Procedure Call to device firmware Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 6/8] fwctl: Add documentation Jason Gunthorpe
2025-03-15 2:53 ` Bagas Sanjaya
2025-02-28 0:26 ` [PATCH v5 7/8] fwctl/mlx5: Support for communicating with mlx5 fw Jason Gunthorpe
2025-03-02 12:11 ` Leon Romanovsky
2025-03-04 17:50 ` Jason Gunthorpe
2025-02-28 0:26 ` [PATCH v5 8/8] mlx5: Create an auxiliary device for fwctl_mlx5 Jason Gunthorpe
2025-03-02 12:09 ` Leon Romanovsky
2025-03-04 1:53 ` [PATCH v5 0/8] Introduce fwctl subystem Jakub Kicinski
2025-03-04 14:00 ` Jason Gunthorpe
2025-03-04 17:59 ` Saeed Mahameed
2025-03-05 0:42 ` Jakub Kicinski
2025-03-05 13:32 ` Jason Gunthorpe
2025-03-05 13:43 ` Leon Romanovsky
2025-03-05 15:08 ` Jiri Pirko
2025-03-05 15:22 ` Leon Romanovsky
2025-03-05 15:56 ` Jiri Pirko
2025-03-05 18:17 ` David Ahern
2025-03-05 18:28 ` Leon Romanovsky
2025-03-05 20:41 ` Saeed Mahameed
2025-03-05 23:21 ` Jason Gunthorpe
2025-03-06 7:29 ` Leon Romanovsky
2025-03-11 11:23 ` David Ahern
2025-03-11 13:59 ` Leon Romanovsky
2025-03-12 9:31 ` David Ahern
2025-03-12 10:34 ` Stanislav Fomichev
2025-03-14 22:34 ` David Ahern
2025-03-16 7:34 ` Stanislav Fomichev
2025-03-17 12:30 ` Jason Gunthorpe
2025-03-11 14:27 ` Nelson, Shannon
2025-03-13 12:30 ` David Ahern
2025-03-13 12:48 ` Leon Romanovsky
2025-03-13 19:59 ` Nelson, Shannon
2025-03-14 5:37 ` Greg Kroah-Hartman
2025-03-14 18:39 ` Leon Romanovsky
2025-03-14 18:09 ` Jacob Keller
2025-03-17 12:33 ` Jason Gunthorpe
2025-03-17 19:00 ` David Ahern
2025-03-17 20:33 ` Keller, Jacob E
2025-03-18 13:20 ` Greg Kroah-Hartman
2025-03-18 13:25 ` Jason Gunthorpe
2025-03-18 15:39 ` Dave Jiang
2025-03-18 16:06 ` Greg Kroah-Hartman
2025-03-19 5:48 ` Przemek Kitszel
2025-03-19 8:14 ` Leon Romanovsky
2025-03-19 10:46 ` Przemek Kitszel
2025-03-19 11:22 ` Leon Romanovsky
2025-03-19 8:17 ` Leon Romanovsky
2025-03-18 22:07 ` Keller, Jacob E
2025-03-06 2:16 ` Jakub Kicinski
2025-03-20 23:22 ` Jason Gunthorpe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).