Re: [PATCH v4] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Bjorn Helgaas <helgaas@kernel.org>]

On Wed, Nov 13, 2024 at 12:20:20PM +0000, Shijith Thotton wrote:
> >> >> This patch introduces a PCI hotplug controller driver for the OCTEON
> >> >> PCIe device. The OCTEON PCIe device is a multi-function device where the
> >> >> first function serves as the PCI hotplug controller.
> >> >>
> >> >>                +--------------------------------+
> >> >>                |           Root Port            |
> >> >>                +--------------------------------+
> >> >>                                |
> >> >>                               PCIe
> >> >>                                |
> >> >> +---------------------------------------------------------------+
> >> >> |              OCTEON PCIe Multifunction Device                 |
> >> >> +---------------------------------------------------------------+
> >> >>              |                    |              |            |
> >> >>              |                    |              |            |
> >> >> +---------------------+  +----------------+  +-----+  +----------------+
> >> >> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
> >> >> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
> >> >> +---------------------+  +----------------+  +-----+  +----------------+
> >> >>              |
> >> >>              |
> >> >> +-------------------------+
> >> >> |   Controller Firmware   |
> >> >> +-------------------------+
> >> >>
> >> >> The hotplug controller driver enables hotplugging of non-controller
> >> >> functions within the same device. During probing, the driver removes
> >> >> the non-controller functions and registers them as PCI hotplug slots.
> >> >> These slots are added back by the driver, only upon request from the
> >> >> device firmware.
> >> >>
> >> >> The controller uses MSI-X interrupts to notify the host of hotplug
> >> >> events initiated by the OCTEON firmware. Additionally, the driver
> >> >> allows users to enable or disable individual functions via sysfs slot
> >> >> entries, as provided by the PCI hotplug framework.
> >> >
> >> >Can we say something here about what the benefit of this driver is?
> >> >For example, does it save power?
> >>
> >> The driver enables hotplugging of non-controller functions within the device
> >> without requiring a fully implemented switch, reducing both power
> >consumption
> >> and product cost.
> >
> >Reduced product cost is motivation for the hardware design, not for
> >this hotplug driver.
> >
> >You didn't explicitly say that when function 0 hot-removes another
> >function, it reduces overall power consumption.  But I assume that's
> >the case?
> >
> 
> Yes, I will explain it in detail below
> 
> >> >What causes the function 0 firmware to request a hot-add or
> >> >hot-removal of another function?
> >>
> >> The firmware will enable the required number of non-controller
> >> functions based on runtime demand, allowing control over these
> >> functions. For example, in a vDPA scenario, each function could act
> >> as a different type of device (such as net, crypto, or storage)
> >> depending on the firmware configuration.
> >
> >What is the path for this runtime demand?  I assume function 0
> >provides some interface to request a specific kind of functionality
> >(net, crypo, storage, etc)?
> >
> 
> Right now, it done via firmware management console.
> 
> >I don't know anything about vDPA, so if that's important here, it
> >needs a little more context.
> >
> >> Hot removal is useful in cases of live firmware updates.
> >
> >So the idea is that function X is hot-removed, which forces the driver
> >to let go of it, the firmware is updated, and X is hot-added again,
> >and the driver binds to it again?
> >
> 
> I will explain the process in detail, which should also address the questions
> below.
> 
> >And somewhere in there is a reset of function X, and after the reset
> >X is running the new firmware?
> >
> >Who/what initiates this whole path?  Some request to function 0,
> >saying "please remove function X"?
> >
> >But I guess maybe it doesn't go through function 0, since octeon_hp
> >claims function 0, and it doesn't provide that functionality.  Maybe
> >the individual drivers for *other* functions know how to initiate
> >these things, and those functions internally communicate with function
> >0 to ask it to start a hot-remove/hot-add sequence?
> >
> >That wouldn't explain the power reduction plan, though.  A driver for
> >function X could conceivably tell its device "I'm no longer needed"
> >and function X could tell function 0 to remove it.  That might enable
> >some power savings.  But that doesn't have a path to *re-enable*
> >function X, since function X has been removed and there's no driver to
> >ask for it to be hot-added again.
> >
> >Maybe there's some out-of-band management path that can tell function
> >0 to do things, independent of PCIe?
> >
> 
> Our implementation aims to achieve two main objectives:
> 
> 1. Enable changing a function's personality at runtime.
> 2. Reduce power consumption.
> 
> The OCTEON PCI device has multiple ARM cores running Linux, with its firmware
> composed of multiple components. For example, the firmware includes components
> like Virtio-net, NVMe, and Virtio-Crypto, which can be assigned to any function
> at runtime. The device firmware is accessible via a management console, allowing
> components to be started or stopped. For each component, an associated function
> is hot-added on the host to expose its functionality. Initially, after boot, only
> Function 0 and the controller firmware are active.
> 
> Here's a breakdown:
> 
> At Time 0:
> - Linux boots on the device, starting the controller firmware.
> 
> At Time 1:
> - The hotplug driver loads on the host, temporarily removing other functions.
> 
> At Time 2:
> - A network device firmware component starts on an ARM core (initiated through
>   a console command).
> - This component sets up the Function 1 configuration space, data, and other
>   request handlers for network processing.
> - The firmware issues a hot-add request to Function 0 (hotplug driver) on the
>   host to enable Function 1.
> 
> At Time 3:
> - The Function 0 hotplug driver on the host receives the hot-add request and
>   enables Function 1 on the host.
> - A network driver binds to Function 1 based on device class and ID.
> 
> At Time 4:
> - The network device firmware component receives a stop signal.
> - The firmware issues a hot-remove request for Function 1 on the host.
> - The firmware component halts, reducing the device's power consumption.
> 
> At Time 5:
> - The Function 0 hotplug driver on the host receives the hot-remove request and
>   disables Function 1 on the host.
> 
> At Time 6:
> - A crypto device firmware component starts on an ARM core.
> - This component configures the Function 1 configuration space for crypto
>   processing and sets up the required firmware handlers.
> - The firmware issues a hot-add request to enable Function 1 on the host.
> 
> At Time 7:
> - The Function 0 hotplug driver on the host receives the hot-add request and                                                                                                                                                                                                      enables Function 1 on the host.
> - A crypto driver binds to Function 1 based on device class and ID.
> 
> The firmware component for each function only runs and is hot-added when
> needed. Only Function 0 and the controller firmware remain active
> continuously. This dynamic control reduces power usage by keeping unnecessary
> components off. Additionally, a single function can adapt its personality based
> on the associated firmware component, enhancing flexibility. 
>                                                                                                                                                                                                                    
> I hope this clarifies the implementation. Let me know if you have any
> questions.

Thanks very much!  I propose adding text like this to the commit log:

  There is an out-of-band management console interface to firmware
  running on function 0 whereby an administrator can disable functions
  to save power or enable them with one of several personalities
  (virtio-net, virtio-crypto, NVMe, etc) for the other functions.
  Function 0 initiates hotplug events handled by this driver when the
  other functions are enabled or disabled.

I provisionally applied this to pci/hotplug-octeon, but will be happy
to update the text if necessary.

Bjorn