Re: PCIe cycle sequence when updating the msi-x table

[Thomas Gleixner <tglx@linutronix.de>]

On Fri, Apr 07 2023 at 16:17, David Laight wrote:
> The function that updates the MSI-X table currently reads:
>
> Now I'm not 100% sure about the cycle ordering rules.
> I've not got the spec here, and I recall it isn't that easy to
> understand.
> I can't remember whether reads are allowed to overtake writes
> (to different addresses).
> I do remember that reading back the same address was needed to
> flush the cpu store buffer on some space cpus.
> So it might be that the final readl() won't actually flush
> the write to the mask register.
> (The same readback of 'the wrong' address also happens elsewhere.)

It's guaranteed that all writes to a device have reached the device
before a read from the same device which comes after the writes is
issued. The write and read addresses do not matter.

> But there is a bigger problem.
> As the comment says writing the address/data while an entry is
> unmasked must be avoided (because a mixture of the old and new
> values could easily by used for the PCIe write cycle).
>
> But it is also quite likely that that hardware checks the masked
> bit before/after reading the address+data.
>
> So masking the interrupt immediately before the update and/or
> unmasking immediately after could also cause issues.

No it does not, because the writes are strictly ordered.

So the devices gets:

   1) write to control register with MASKBIT set
   2) write to LOWER_ADDRESS
   3) write to UPPER_ADDRESS
   4) write to ENTRY_DATA
   5) write to control register with MASKBIT cleared

#1 disables the vector and the device is not allowed to use the msg data
from the table entry until the mask bit is cleared again.

If the device gets that wrong then that's a bug in the device and not a
kernel problem.

Thanks,

        tglx