Re: [PATCH v4] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Bjorn Helgaas <helgaas@kernel.org>]

On Tue, Nov 12, 2024 at 09:25:46AM +0000, Shijith Thotton wrote:
> >> This patch introduces a PCI hotplug controller driver for the OCTEON
> >> PCIe device. The OCTEON PCIe device is a multi-function device where the
> >> first function serves as the PCI hotplug controller.
> >>
> >>                +--------------------------------+
> >>                |           Root Port            |
> >>                +--------------------------------+
> >>                                |
> >>                               PCIe
> >>                                |
> >> +---------------------------------------------------------------+
> >> |              OCTEON PCIe Multifunction Device                 |
> >> +---------------------------------------------------------------+
> >>              |                    |              |            |
> >>              |                    |              |            |
> >> +---------------------+  +----------------+  +-----+  +----------------+
> >> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
> >> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
> >> +---------------------+  +----------------+  +-----+  +----------------+
> >>              |
> >>              |
> >> +-------------------------+
> >> |   Controller Firmware   |
> >> +-------------------------+
> >>
> >> The hotplug controller driver enables hotplugging of non-controller
> >> functions within the same device. During probing, the driver removes
> >> the non-controller functions and registers them as PCI hotplug slots.
> >> These slots are added back by the driver, only upon request from the
> >> device firmware.
> >>
> >> The controller uses MSI-X interrupts to notify the host of hotplug
> >> events initiated by the OCTEON firmware. Additionally, the driver
> >> allows users to enable or disable individual functions via sysfs slot
> >> entries, as provided by the PCI hotplug framework.
> >
> >Can we say something here about what the benefit of this driver is?
> >For example, does it save power?
> 
> The driver enables hotplugging of non-controller functions within the device
> without requiring a fully implemented switch, reducing both power consumption
> and product cost.

Reduced product cost is motivation for the hardware design, not for
this hotplug driver.

You didn't explicitly say that when function 0 hot-removes another
function, it reduces overall power consumption.  But I assume that's
the case?

> >What causes the function 0 firmware to request a hot-add or
> >hot-removal of another function?
> 
> The firmware will enable the required number of non-controller
> functions based on runtime demand, allowing control over these
> functions. For example, in a vDPA scenario, each function could act
> as a different type of device (such as net, crypto, or storage)
> depending on the firmware configuration.

What is the path for this runtime demand?  I assume function 0
provides some interface to request a specific kind of functionality
(net, crypo, storage, etc)?

I don't know anything about vDPA, so if that's important here, it
needs a little more context.

> Hot removal is useful in cases of live firmware updates.

So the idea is that function X is hot-removed, which forces the driver
to let go of it, the firmware is updated, and X is hot-added again,
and the driver binds to it again?

And somewhere in there is a reset of function X, and after the reset
X is running the new firmware?

Who/what initiates this whole path?  Some request to function 0,
saying "please remove function X"?

But I guess maybe it doesn't go through function 0, since octeon_hp
claims function 0, and it doesn't provide that functionality.  Maybe
the individual drivers for *other* functions know how to initiate
these things, and those functions internally communicate with function
0 to ask it to start a hot-remove/hot-add sequence?

That wouldn't explain the power reduction plan, though.  A driver for
function X could conceivably tell its device "I'm no longer needed"
and function X could tell function 0 to remove it.  That might enable
some power savings.  But that doesn't have a path to *re-enable*
function X, since function X has been removed and there's no driver to
ask for it to be hot-added again.

Maybe there's some out-of-band management path that can tell function
0 to do things, independent of PCIe?

So confused,
  Bjorn