Re: [PATCH 0/8] Introduce fwctl subystem

Dan Williams
2024-06-04T23:57:06Z

Jakub Kicinski wrote:
[..]
> I don't begrudge anyone building proprietary options, but leave
> upstream out of it.

So I am of 2 minds here. In general, how is upstream benefited by
requiring every vendor command to be wrapped by a Linux command?

Mind you, I am coming at this from the perspective of being a maintainer
of a subsystem that does *not* allow unrestricted vendor commands. Since
day one, the CXL subsystem has matched netdev's general sentiment and
been more restrictive than NVMe. It places all vendor commands and even
all yet-to-be-Linux-wrapped-standard-commands behind a
CONFIG_CXL_MEM_RAW_COMMANDS option. That default-off option, when
enabled, allows any command to be sent but it taints the kernel with a
WARN(). CXL devices theoretically allow direct manipulation of system
memory without IOMMU protection which is in contrast to NVMe which would
need to work harder to violate kernel-lockdown protections.

The expectation that I laid out here [1] is based on the observation
that a significant portion of the vendor commands these devices support
are for pre-release hardware qualification and debug flows. The
recommendation to device vendors was "if you need wide distribution of
kernels that allow unrestricted vendor passthrough, work with Linux
distributions to enable this option in debug kernels, run those debug
kernels for your pre-release hardware flows, ignore the warnings".

3 years on from that recommendation it seems no vendor has even needed
that level of distribution help. I.e. checking a few distro kernels
(Fedora, openSUSE) shows no uptake for CONFIG_CXL_MEM_RAW_COMMANDS=y in
their debug builds. I can only assume that locally compiled custom
kernel binaries are filling the need.

So all seems quiet with current restriction for CXL endpoint vendor
commands, but this stance was recently challenged in this thread [2] by
CXL switch vendors with an assertion that fabric switch configuration
has need for more and varied vendor flows than endpoint configuration.

While I am not clear on the veracity of that claim, it at least
challenged me to do the thought experiment of "what would it look like
to relax the CXL command restriction?". Maybe we can come up with a
community answer to the "so you want to build a
userpace-to-device-firmware tunnel?" to at least get all the various
concerns documented in one place, and provide guidance for how device
vendors should navigate this space across subsystems. Between NVMe
"allow all the things", CXL "allow all the things only after tainting
the kernel", and the "never allow vendor passthrough" position (I am
sure there are other nuanced positions) it at least seems useful to
document the concerns. Here is a start for that guidance from the CXL
perspective:

* Integrity: Subsystem has a responsibility to meet kernel-lockdown
expectations:

Distros and system owners need to be assured that root's ability to
modify the running kernel image are mitigated. For CXL there are 2 ways
to do this, require Linux wrapper commands for all the low level
commands (status quo), or a new trust the device to publish which
commands have user data effects in something CXL calls the "Command
Effects Log". In that "trust Command Effects" scenario the kernel still
has no idea what the command is actually doing, but it can at least
assert that the device does not claim that the command changes the
contents of system-memory. Now, you might say, "the device can just
lie", but that betrays a conceit of the kernel restriction. A device
could lie that a Linux wrapped command when passed certain payloads does
not in turn proxy to a restricted command. So at some point there is
almost always an out-of-tree way to get around the kernel restriction,
so the question is are we better off giving a blessed path or force
vendors into ugly out-of-tree workarounds?

* Introspection / validation: Subsystem community needs to be able to
audit behavior after the fact.

To me this means even if the kernel is letting a command through based
on the stated Command Effect of "Configuration Change after Cold Reset"
upstream community has a need to be able to read the vendor
specification for that command. I.e. commands might be vendor-specific,
but never vendor-private. I see this as similar to the requirement for
open source userspace for sophisticated accelerators.

* Collaboration: open standards support open driver maintenance.

Without standards we end up with awkward situations like Confidential
Computing where every vendor races to implement the same functionality
in arbitrarily different and vendor specific ways.

For CXL devices, and I believe the devices fwctl is targeting, there
are a whole class of commands for vendor specific configuration and
debug. Commands that the kernel really need not worry about.

Some subsystems may want to allow high-performance science experiments
like what NVMe allows, but it seems worth asking the question if
standardizing device configuration and debug is really the best use of
upstream's limited time?

One of the release valves in the CXL space is openly specified
commands with opaque payloads, like "Read Vendor Debug Log". That is
clear what it does, likely a payload the kernel need never worry
about, and the "Command Effects" is empty. However, going forward there
is a new class of commands called "Set/Get Feature" that allow a wide
range of vendor toggles to be deployed which will need an upstream
response for the driver policy to vendor-specific "Features".

So if fwctl, or something like it, can strike a balance of enforcing
integrity and introspection while encouraging collaboration on the
aspects that are worth upstream collaboration, I think that is a
conversation worth having.

[1]: http://lore.kernel.org/r/CAPcyv4gDShAYih5iWabKg_eTHhuHm54vEAei8ZkcmHnPp3B0cw@mail.gmail.com/
[2]: http://lore.kernel.org/r/20240321174423.00007e0d@Huawei.com