[PATCH V2 0/4] PCI: Re-evaluate DEV3 14-Bit Tag Requester Enable on link mode changes

[Vidya Sagar <vidyas@nvidia.com>]

DEV3_CTL.14-Bit Tag Requester Enable is only meaningful while the link
operates in Flit Mode.  In Non-Flit Mode the upper tag bits are not
transmitted on the wire, so a requester that still has it set emits
requests whose completions it can no longer match: the completer does
answer, but the completion comes back with the upper tag bits zero.  The
requester therefore logs an Unexpected Completion for it and, once the
timer expires, a Completion Timeout as well.  The completer side needs no
handling, since a completer only reflects the Tag field of the request it
answers and the spec accordingly defines no completer enable.

This is what happens today on the very first config read after a
bridge-mediated reset when the link comes back in Non-Flit Mode.  Two
things are missing in the core:

  - DEV3_CTL is not part of the state the core saves and restores, so its
    contents are lost across a reset even for the devices that do get
    saved and restored, and

  - nothing re-evaluates the 14-Bit Tag Requester Enable when the link
    mode changes.  This matters most for the bridge: it is the requester
    for the config read the core issues to the device below as soon as the
    link is back, but it is not itself saved and restored on these paths,
    so it keeps its stale enable.

The series:

  1/4 adds the DEV3_CAP.14-Bit Tag Requester Supported and
      DEV3_CTL.14-Bit Tag Requester Enable definitions, both bit 2 of
      their respective register.

  2/4 moves __pcie_update_link_speed() out of line, which 4/4 needs.  Pure
      refactor.

  3/4 saves DEV3_CTL and, on restore, drops 14-Bit Tag Requester Enable
      from the saved value when the live LNKSTA2.Flit_Mode and
      DEV3_STA.Segment Captured say Flit Mode is gone.  This covers the
      devices reached via pci_dev_restore().  No other DEV3_CTL bit is
      modified.

  4/4 adds pci_bridge_refresh_14bit_tag(), which fixes the bridge itself
      and then walks its subordinate bus, and calls it from
      pci_bridge_wait_for_secondary_bus() (before the first downstream
      config read after SBR, DPC release, AER bus reset, slot reset or a
      bridge D3cold->D0 resume) and from __pcie_update_link_speed() (which
      covers retrain, bwctrl and hotplug paths that never reach
      pci_bridge_wait_for_secondary_bus()).

3/4 and 4/4 are independent of each other; each is useful on its own.

Tested on an arm64 platform with a Root Port whose link comes back in
Non-Flit Mode after a reset.  Resetting the subordinate bus from sysfs
(reset_subordinate, i.e. pci_reset_bridge() -> pci_try_reset_slot())
previously left the Root Port with 14-Bit Tag Requester Enable set, and
the first config read to the device below failed with a Completion Timeout
plus an Unexpected Completion.  With this series the Root Port is fixed up
first,

  pcieport 0002:80:00.0: cleared 14-Bit Tag Requester Enable: flit mode no longer active (DEV3_STA=0x00000008)

and the reset completes.  Note that DEV3_STA.Segment Captured was still
set at that point while LNKSTA2.Flit_Mode had already dropped, which is
why both are consulted.

V2:
* Split the V1 monolithic patch into 4 patches

Vidya Sagar (4):
  PCI: Add DEV3 14-Bit Tag Requester register definitions
  PCI: Move __pcie_update_link_speed() out of line
  PCI: Save and restore the Device 3 Control register
  PCI: Clear stale 14-Bit Tag Requester Enable when a link leaves Flit
    Mode

 drivers/pci/pci.c             | 208 ++++++++++++++++++++++++++++++++++
 drivers/pci/pci.h             |  17 +--
 drivers/pci/probe.c           |  36 ++++++
 include/uapi/linux/pci_regs.h |   2 +
 4 files changed, 251 insertions(+), 12 deletions(-)

-- 
2.43.0