Re: [PATCH 0/8] Introduce fwctl subystem

[Dan Williams <dan.j.williams@intel.com>]

Jakub Kicinski wrote:
[..]
> I don't begrudge anyone building proprietary options, but leave
> upstream out of it.

So I am of 2 minds here. In general, how is upstream benefited by
requiring every vendor command to be wrapped by a Linux command?

Mind you, I am coming at this from the perspective of being a maintainer
of a subsystem that does *not* allow unrestricted vendor commands. Since
day one, the CXL subsystem has matched netdev's general sentiment and
been more restrictive than NVMe. It places all vendor commands and even
all yet-to-be-Linux-wrapped-standard-commands behind a
CONFIG_CXL_MEM_RAW_COMMANDS option. That default-off option, when
enabled, allows any command to be sent but it taints the kernel with a
WARN(). CXL devices theoretically allow direct manipulation of system
memory without IOMMU protection which is in contrast to NVMe which would
need to work harder to violate kernel-lockdown protections.

The expectation that I laid out here [1] is based on the observation
that a significant portion of the vendor commands these devices support
are for pre-release hardware qualification and debug flows. The
recommendation to device vendors was "if you need wide distribution of
kernels that allow unrestricted vendor passthrough, work with Linux
distributions to enable this option in debug kernels, run those debug
kernels for your pre-release hardware flows, ignore the warnings".

3 years on from that recommendation it seems no vendor has even needed
that level of distribution help. I.e. checking a few distro kernels
(Fedora, openSUSE) shows no uptake for CONFIG_CXL_MEM_RAW_COMMANDS=y in
their debug builds. I can only assume that locally compiled custom
kernel binaries are filling the need.

So all seems quiet with current restriction for CXL endpoint vendor
commands, but this stance was recently challenged in this thread [2] by
CXL switch vendors with an assertion that fabric switch configuration
has need for more and varied vendor flows than endpoint configuration.
 
While I am not clear on the veracity of that claim, it at least
challenged me to do the thought experiment of "what would it look like
to relax the CXL command restriction?". Maybe we can come up with a
community answer to the "so you want to build a
userpace-to-device-firmware tunnel?" to at least get all the various
concerns documented in one place, and provide guidance for how device
vendors should navigate this space across subsystems. Between NVMe
"allow all the things", CXL "allow all the things only after tainting
the kernel", and the "never allow vendor passthrough" position (I am
sure there are other nuanced positions) it at least seems useful to
document the concerns. Here is a start for that guidance from the CXL
perspective:

* Integrity: Subsystem has a responsibility to meet kernel-lockdown
  expectations:

  Distros and system owners need to be assured that root's ability to
  modify the running kernel image are mitigated. For CXL there are 2 ways
  to do this, require Linux wrapper commands for all the low level
  commands (status quo), or a new trust the device to publish which
  commands have user data effects in something CXL calls the "Command
  Effects Log". In that "trust Command Effects" scenario the kernel still
  has no idea what the command is actually doing, but it can at least
  assert that the device does not claim that the command changes the
  contents of system-memory. Now, you might say, "the device can just
  lie", but that betrays a conceit of the kernel restriction. A device
  could lie that a Linux wrapped command when passed certain payloads does
  not in turn proxy to a restricted command. So at some point there is
  almost always an out-of-tree way to get around the kernel restriction,
  so the question is are we better off giving a blessed path or force
  vendors into ugly out-of-tree workarounds?

* Introspection / validation: Subsystem community needs to be able to
  audit behavior after the fact.

  To me this means even if the kernel is letting a command through based
  on the stated Command Effect of "Configuration Change after Cold Reset"
  upstream community has a need to be able to read the vendor
  specification for that command. I.e. commands might be vendor-specific,
  but never vendor-private. I see this as similar to the requirement for
  open source userspace for sophisticated accelerators.

* Collaboration: open standards support open driver maintenance.

  Without standards we end up with awkward situations like Confidential
  Computing where every vendor races to implement the same functionality
  in arbitrarily different and vendor specific ways.

  For CXL devices, and I believe the devices fwctl is targeting, there
  are a whole class of commands for vendor specific configuration and
  debug. Commands that the kernel really need not worry about.

  Some subsystems may want to allow high-performance science experiments
  like what NVMe allows, but it seems worth asking the question if
  standardizing device configuration and debug is really the best use of
  upstream's limited time?

  One of the release valves in the CXL space is openly specified
  commands with opaque payloads, like "Read Vendor Debug Log". That is
  clear what it does, likely a payload the kernel need never worry
  about, and the "Command Effects" is empty. However, going forward there
  is a new class of commands called "Set/Get Feature" that allow a wide
  range of vendor toggles to be deployed which will need an upstream
  response for the driver policy to vendor-specific "Features".

So if fwctl, or something like it, can strike a balance of enforcing
integrity and introspection while encouraging collaboration on the
aspects that are worth upstream collaboration, I think that is a
conversation worth having.

[1]: http://lore.kernel.org/r/CAPcyv4gDShAYih5iWabKg_eTHhuHm54vEAei8ZkcmHnPp3B0cw@mail.gmail.com/
[2]: http://lore.kernel.org/r/20240321174423.00007e0d@Huawei.com