RE: PCIe cycle sequence when updating the msi-x table

[David Laight <David.Laight@ACULAB.COM>]

From: Thomas Gleixner
> Sent: 10 April 2023 07:50
...
> Whatever order this reads does not matter. The point is:
> 
>   "Mask Bit - When this bit is Set, the Function is prohibited from
>    sending a message using this MSI-X Table entry."
> 
> So you cannot make this read order dependent at all.
> 
> > Anything fpga based is likely to be using a 32bit memory
> > block for the MSI-X data (possibly even 16bit).
> 
> It's trivial enough to latch the message on unmask into a shadow
> register set and let the state engine work from there.

I could implement the memory block inside the MSI_X logic
and then detect writes.
Anything else uses a lot more fpga resource.

> And no, we are not adding random delays to that code just because.

I was mostly suggesting one be moved.
(Changing the address/data is hardly a hot path.)

Actually I'm trying to work out what the readbacks in the MSI-X
code are actually for (apart from adding a 'random delay').

If you are masking one of the PCI INTx lines, then a readback
delays the cpu until the physical line is de-asserted so that
the IRQ line into the interrupt controlled is de-asserted and
thus the IRQ to the cpu is also de-asserted.
Even if the cpu commits to the interrupt, the interrupt controller
reports 'no vector' (or vector 7 if it is the 8259).
So you can safely do a readback, enable interrupts and know
the interrupt won't happen.
(Although if you have a real 8259 you need to worry about
the cycle recovery time, even a 286 will break it!)

But MSI-X is different, it is very much edge sensitive.
The target MSI-X logic can decide to raise an interrupt
just before the mask bit it set.
It will then request the target's PCIe interface issue
the write TLP, this is probably a 'posted' write in the
internal logic. The PCIe interface could be busy generating
a long write TLP (or two) before actually issuing write for
the interrupt.
The readback will be pretty much back-to-back with the mask write
(as seen on the target).
So maybe the rad can get completed while the write is still
queued.
Now I can't remember whether the data TLP for reads is allowed
to overtake a posted write request, but it really doesn't matter.
Even if the IRQ write arrives at the root hub before the
read completion they then head off in different directions
through the 'cpu' logic - so the read could easily complete
well before the interrupt gets processed.

In fact an interrupt requested well before the mask bit is set
could be pending in the ioapic waiting for the cpu to enable
interrupts.

So I don't see any reason for those readbacks at all.

I've never looked at the cpu end of MSI-X, but I suspect that
stop using an interrupts you need to mask it there first,
then disable the hardware and later make sure you clear
any lurking pending request (probably before unmasking for
later use).

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)