RE: PCIe cycle sequence when updating the msi-x table

[David Laight <David.Laight@ACULAB.COM>]

From: Thomas Gleixner
> Sent: 07 April 2023 20:56
..
> > But there is a bigger problem.
> > As the comment says writing the address/data while an entry is
> > unmasked must be avoided (because a mixture of the old and new
> > values could easily by used for the PCIe write cycle).
> >
> > But it is also quite likely that that hardware checks the masked
> > bit before/after reading the address+data.
> >
> > So masking the interrupt immediately before the update and/or
> > unmasking immediately after could also cause issues.
> 
> No it does not, because the writes are strictly ordered.
> 
> So the devices gets:
> 
>    1) write to control register with MASKBIT set
>    2) write to LOWER_ADDRESS
>    3) write to UPPER_ADDRESS
>    4) write to ENTRY_DATA
>    5) write to control register with MASKBIT cleared
> 
> #1 disables the vector and the device is not allowed to use the msg data
> from the table entry until the mask bit is cleared again.
> 
> If the device gets that wrong then that's a bug in the device and not a
> kernel problem.

Maybe, but the kernel isn't making it easy for a device
state-engine that has to do four separate reads of an
internal 32-bit memory area.
Adding a short delay between #4 and #5 is likely to avoid
some very hard to debug issues if the hardware reads the
values 'mask last', if it reads them 'mask first' you need
a short delay between #1 and #2.

Anything fpga based is likely to be using a 32bit memory
block for the MSI-X data (possibly even 16bit).

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)