Re: PCIe endpoint crosstalk

[Ludwig Petrosyan <ludwig.petrosyan@desy.de>]

On 08/27/2013 06:27 PM, Bjorn Helgaas wrote:
> On Tue, Aug 27, 2013 at 09:53:35AM +0200, Ludwig Petrosyan wrote:
>> So: I use microTCA system with PCIe bus, there are two AMC cards (PCIe
>> endpoints), lets call card A and card B.
>> as well there are two device drivers for A and B. Card B has bug, after
>> PCIe memory write  operation (MWr) the card sends back Completion
>> packet without data (Cpl) (I now it is wrong, but card designed in this
>> way and has to be changed).
>> User process Ua reads data from Card A in loop, everything is OK , but
>> then I start second user process Ub which writes in loop data to card B
>> (bugged card) the Ua gets wrong data. After improving card B the problem
>> was solved, but could be it has to be checked on the PCIe driver level
>> as well.
> PCIe transactions (MWr, MRd, Cpl, etc.) are not directly visible
> to the OS or the driver.
>
> The only thing I can think of that we could do is add a quirk to
> blacklist the broken version of card B.  You can look at existing
> quirks in drivers/pci/quirks.c.  Most of them workaround issues
> that aren't quite as severe as this one, but we could probably
> figure out a way to make the device completely unusable.
>
> Or do you have something else in mind?
>
> Bjorn
We have fixed the bug in card B and now it is OK, but question is open,
what will happen if we got some PCIe endpoint card with the same bug:
read operations from other PCIe devices could be broken. Just I think
this problem should be solved on the OS level (I am not sure)

I will try to explain how things are going on how I think:

User process Ub sends Memory-Write request to card B, this is Posted
request, so just  after sending the request Ub forgets about it,
TLP of this packet contain Requester ID for RootComplex, at the same
time user process Ua (the RootComplex is free now) sends non-Posted
memory read request to card A and waits for Completion packet, but at
the same time the card B (bugged card, it should not send Completion to
Posted memory write request) send to RootComplex Completion Packet
without data and some how Ua get this data as result of his Memory Read
request. Seems the Completer ID (or Tag field) in Completion packet not
checked and completion from one PCIe endpoint returned as completion of
read request from other PCIe endpoint.

I want to say this is only an assumption, just I wont to be sure the
bugged PCIe device won't influence operation of other devices
But could be this problem has to be solved on PCIe Switch or RootComplex
side not in OS side...

with best regards

Ludwig