MMCONFIG violates pci power mgmt spec

MMCONFIG violates pci power mgmt spec

[Michael Chan]

We are encountering a problem with the MMCONFIG code with respect to
power management. Specifically, when pci_mmcfg_write is used to program
a PCI Express device's PMCSR to a different power state, the dummy read
at the end of that routine violates the transition delay specified in
the PCI power management spec.

For example, if the device is transitioning into or out of D3hot, the
spec requires a delay of 10 msec before any accesses can be made to the
device. The dummy read in pci_mmcfg_write violates the delay
requirements even though pci_set_power_state has all the necessary
delays.

I have contacted "Durairaj, Sundarapandian
<sundarapandian.durairaj@intel.com>" but did not get a response, and so
I'm posting to this list. One question I wanted to ask him was whether
the dummy read was necessary. If the Intel chipset treats the mmconfig
write as a non-posted write, the dummy read becomes unnecessary and
removing it will solve the problem. If it is treated as a posted write,
I wonder if there is another way to flush it other than reading from the
target device.

Thanks,
Michael

Re: MMCONFIG violates pci power mgmt spec

[Andi Kleen]

"Michael Chan" <mchan@broadcom.com> writes:

> For example, if the device is transitioning into or out of D3hot, the
> spec requires a delay of 10 msec before any accesses can be made to the
> device. The dummy read in pci_mmcfg_write violates the delay
> requirements even though pci_set_power_state has all the necessary
> delays.

Interesting. What happens? Hangs? 

>
> I have contacted "Durairaj, Sundarapandian
> <sundarapandian.durairaj@intel.com>" but did not get a response, and so
> I'm posting to this list. One question I wanted to ask him was whether
> the dummy read was necessary. If the Intel chipset treats the mmconfig
> write as a non-posted write, the dummy read becomes unnecessary and
> removing it will solve the problem. If it is treated as a posted write,

This was added to keep the new access method to be as compatible
as possible to the old method (which never posts). 

If someone cites the spec that says that it is not allowed I guess
it could be removed.

In the worst case we could add a new pci_read_config_*_relaxed() 
or somesuch.

> I wonder if there is another way to flush it other than reading from the
> target device.

Reading is the official way to flush.

-Andi

Re: MMCONFIG violates pci power mgmt spec

[Roland Dreier]

    Andi> If someone cites the spec that says that it is not allowed I
    Andi> guess it could be removed.

I believe the PCI Express spec says that config writes are never posted.
(I'll check later to be sure)

 - Roland

RE: MMCONFIG violates pci power mgmt spec

[Michael Chan]

> > For example, if the device is transitioning into or out of 
> D3hot, the 
> > spec requires a delay of 10 msec before any accesses can be made to 
> > the device. The dummy read in pci_mmcfg_write violates the delay 
> > requirements even though pci_set_power_state has all the necessary 
> > delays.
> 
> Interesting. What happens? Hangs? 
> 

What happens depends on the system/chipset/bios. Sometimes we see an
NMI, sometimes nothing bad happens.

> 
> Reading is the official way to flush.
> 

The MMCONFIG ECN document from pcisig below gives a few examples on how
the mmconfig write can be flushed. The exact mechanism of course is
implemention specific.

http://www.pcisig.com/specifications/pciexpress/specifications

Michael

RE: MMCONFIG violates pci power mgmt spec

[Michael Chan]

> 
> I believe the PCI Express spec says that config writes are 
> never posted. (I'll check later to be sure)
> 
>  - Roland
> 

Yes, the config write on the PCI Express link is never posted. But the
mmconfig write that gets translated by the chipset to config write on
the PCI Express link may or may not be posted. It is implementation
specific.

Michael