Re: [PATCH RESEND] PCI: disable runtime PM for PLX switches

[Logan Gunthorpe <logang@deltatee.com>]

On 2019-04-24 8:11 a.m., Bjorn Helgaas wrote:
> [+cc Mika for runtime PM of bridges, Logan for switchtec question]
> 
> On Wed, Apr 24, 2019 at 01:01:02PM +0300, Alexander Fomichev wrote:
>> On Tue, Apr 23, 2019 at 04:53:40PM -0500, Bjorn Helgaas wrote:
>>> On Mon, Apr 15, 2019 at 09:15:54AM -0500, Bjorn Helgaas wrote:
>>>> On Mon, Apr 15, 2019 at 04:59:03PM +0300, Alexander Fomichev wrote:
>>>>> PLX switches have an issue that their internal registers
>>>>> become inaccessible when runtime PM is enabled. Therefore PLX
>>>>> service tools can't communicate with the hardware. A kernel
>>>>> option "pcie_port_pm=off" can be used as a workaround. But it
>>>>> affects all the devices.
>>>>>
>>>>> So this solution is to add PLX switch devices to the quirk
>>>>> list for disabling runtime PM only for them.
>>>>
>>>> I assume the problem is actually that the config space registers
>>>> are inaccessible when the device is in D3hot?
>>>
>>> Reading this again, I realize you said "internal registers".  I
>>> don't know whether that actually means config space registers
>>> (which *should* work even when the device is in D3hot (see the
>>> PCIe reference below and PCI Power Management Spec r1.2, sec
>>> 5.4.1)), or MMIO registers (which are not expected to work while
>>> in D3hot).
>>>
>>> If the service tools read MMIO registers, presumably that goes
>>> through some driver that should be able to manage runtime PM.  Or,
>>> if there's no driver, I think your service tool could prevent
>>> runtime power management by changing
>>> /sys/devices/.../power/control to "on" (see
>>> Documentation/power/runtime_pm.txt).
>>
>> You're right. Config space registers are accessible. The driver
>> can't read/write MMIO registers (Device-Specific Registers as
>> they're called by Broadcom).
> 
> Ah, perfect.  That's exactly what's supposed to happen from a PCI
> hardware point of view.  Unfortunately I don't know much about how
> Linux power management works, but Rafael and Mika do.
> 
> How do your service tools access these MMIO registers?
> 
>    - via a PLX driver that provides read/write/ioctl on a char dev?
>    - read/write on /sys/devices/pci*/.../resource<x> ?
>    - mmap on /sys/devices/pci*/.../resource<x> ?
>    - read/write on /dev/mem?
>    - mmap on /dev/mem?
>    - something else?
> 
> I think there are several ways we might be able to fix this:
> 
>    - Write a driver along the lines of drivers/pci/switch/switchtec.c.
>      I don't see any runtime PM stuff in that driver, so maybe it's
>      magically taken care of by the PM/PCI core?  There might also be
>      an issue if both portdrv and your driver want to claim the same
>      device.  I don't know how switchtec deals with that.

Right, the switchtec device puts all its management MMIO registers 
behind a separate PCIe function which lets us bind a different driver 
and not conflict with the portdrv. I've noticed the PLX parts have an 
extra unused BAR in their upstream port function which means it will be 
an annoying hack to write a driver to use it seeing the portdrv needs to 
be registered to it.

Logan