How to force RC to forward p2p TLPs

How to force RC to forward p2p TLPs

[yu]

Hi all,

We have a PCIE card which has a PEX8732 switch on-board,  and there
are two endpoint SOCs like graphic decoder behind the switch, and by
default the ACS is enabled in 8732.

We use the p2p DMA to transfer data between these two endpoint SOCs,
and if the host server is not enable ACS in BIOS, the p2p works well,
but when ACS is enabled in BIOS, the p2p is always failed. With the
help of a protocol analyzer, we can see that the TLP is redirected to
RC, and RC just discard it.

I tried to find how to make RC forward redirected TLP to its original
target, but nothing found, it seems this is highly related to the RC
vendors.

In the PCIE 4.0 spec, the section of the RC behavior of the p2p
request redirect  said that ''implementation-specific logic within the
RC that determines whether the request is directed towards its
original target, or blocked as an ACS Violation error. the algorithms
and specific controls for making this determination are not
architected by this spec''.


So is there some spec or document to describe how to set the RC? Any
suggestion is appreciated.

Regards,
Eric

Re: How to force RC to forward p2p TLPs

[Bjorn Helgaas]

[+cc Logan, Stephen, Jonathan, Eric, Alex]

On Sat, Dec 22, 2018 at 12:50:19PM +0800, yu wrote:
> Hi all,
> 
> We have a PCIE card which has a PEX8732 switch on-board,  and there
> are two endpoint SOCs like graphic decoder behind the switch, and by
> default the ACS is enabled in 8732.
> 
> We use the p2p DMA to transfer data between these two endpoint SOCs,
> and if the host server is not enable ACS in BIOS, the p2p works well,
> but when ACS is enabled in BIOS, the p2p is always failed. With the
> help of a protocol analyzer, we can see that the TLP is redirected to
> RC, and RC just discard it.
> 
> I tried to find how to make RC forward redirected TLP to its original
> target, but nothing found, it seems this is highly related to the RC
> vendors.
> 
> In the PCIE 4.0 spec, the section of the RC behavior of the p2p
> request redirect  said that ''implementation-specific logic within the
> RC that determines whether the request is directed towards its
> original target, or blocked as an ACS Violation error. the algorithms
> and specific controls for making this determination are not
> architected by this spec''.
> 
> 
> So is there some spec or document to describe how to set the RC? Any
> suggestion is appreciated.

Not that I'm aware of, but the folks I cc'd would know a lot more
about this area.

Bjorn

Re: How to force RC to forward p2p TLPs

[Logan Gunthorpe]

On 2018-12-28 7:29 p.m., Bjorn Helgaas wrote:
>> We use the p2p DMA to transfer data between these two endpoint SOCs,
>> and if the host server is not enable ACS in BIOS, the p2p works well,
>> but when ACS is enabled in BIOS, the p2p is always failed. With the
>> help of a protocol analyzer, we can see that the TLP is redirected to
>> RC, and RC just discard it.
>>
>> I tried to find how to make RC forward redirected TLP to its original
>> target, but nothing found, it seems this is highly related to the RC
>> vendors.

As far as I know, this is not possible with any RCs I've worked with.
For our work, we ensure we disable ACS on paths between peers that want
to communicate. This is why we introduced the 'disable_acs_redir' kernel
parameter[1] to selectively disable ACS for bridges along P2P paths.

>> So is there some spec or document to describe how to set the RC? Any
>> suggestion is appreciated.

If there is, it would be the documentation for your specific RC. Odds
are, the RC simply does not support this.

Logan

[1] https://patchwork.kernel.org/patch/10549279/

RE: How to force RC to forward p2p TLPs

[Eric Wehage]

There is no method to force an RC to forward requests p2p. RC's are also not required to support p2p and whether they do or not is implementation specific. There is also currently no PCIe based standard to determine whether an RC supports p2p or in what capacity. Also, even if p2p is supported, it might not be supported in a system that enables Virtualization (the Virtualization would have to setup p2p tables to allow it).

There is currently a PCIe ECR that has been submitted, but is undergoing review, that defines a PCIe capability for bridge devices (Root Ports and Switch Downstream Ports) that would allow them to advertise what level of p2p is supported by the device.

Eric Wehage
Huawei Principal Engineer, PCIe

-----Original Message-----
From: Bjorn Helgaas [mailto:helgaas@kernel.org] 
Sent: Friday, December 28, 2018 6:29 PM
To: yu <kkndyu@gmail.com>
Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Logan Gunthorpe <logang@deltatee.com>; Stephen Bates <sbates@raithlin.com>; Jonathan Cameron <jonathan.cameron@huawei.com>; Eric Wehage <Eric.Wehage@huawei.com>; Alexander Duyck <alexander.h.duyck@linux.intel.com>
Subject: Re: How to force RC to forward p2p TLPs

[+cc Logan, Stephen, Jonathan, Eric, Alex]

On Sat, Dec 22, 2018 at 12:50:19PM +0800, yu wrote:
> Hi all,
> 
> We have a PCIE card which has a PEX8732 switch on-board,  and there 
> are two endpoint SOCs like graphic decoder behind the switch, and by 
> default the ACS is enabled in 8732.
> 
> We use the p2p DMA to transfer data between these two endpoint SOCs, 
> and if the host server is not enable ACS in BIOS, the p2p works well, 
> but when ACS is enabled in BIOS, the p2p is always failed. With the 
> help of a protocol analyzer, we can see that the TLP is redirected to 
> RC, and RC just discard it.
> 
> I tried to find how to make RC forward redirected TLP to its original 
> target, but nothing found, it seems this is highly related to the RC 
> vendors.
> 
> In the PCIE 4.0 spec, the section of the RC behavior of the p2p 
> request redirect  said that ''implementation-specific logic within the 
> RC that determines whether the request is directed towards its 
> original target, or blocked as an ACS Violation error. the algorithms 
> and specific controls for making this determination are not 
> architected by this spec''.
> 
> 
> So is there some spec or document to describe how to set the RC? Any 
> suggestion is appreciated.

Not that I'm aware of, but the folks I cc'd would know a lot more about this area.

Bjorn

Re: How to force RC to forward p2p TLPs

[Stephen Bates]

Hi Yu

As I understand it you are asking about the case where the two PCIe End Points (EPs) are connected below the same PCIe switch that is connected via it's Upstream Port (UPS) to a Root Port (RP) that is part of a Root  Complex (RC). Do I understand this correctly?

The ECR Eric mentioned aims to implement a platform neutral method for determining if P2P *between* RPs is viable in a system. However I think that is a bit different to your case (since your EPs are connected to the same RP via a PCIe switch).

What I *think* should be happening in your case when ACS is enabled is the the TLPs should be routed to the RP and from there to some translation agent (often an IOMMU). If the IOMMU has an entry for the addresses in  the TLPs then it should attempt to do a translation and forward on the TLP. If the IOMMU does not have a translation (or does not exist) then the behaviour is probably ill-defined and very specific to the platform you are working on.

If the IOMMU does exist and does have a translation entry it still may not support routing the TLP back to the PCI hierarchy. This again would be platform specific.

I guess what I am saying is the same as Logan and Eric. Once TLPs leave the PCIe Root Port and enter the system (CPU or SOC) you leave the world of the PCI/PCIe specification and the behaviour of those TLPs will be dependant  on the specifics of your platform.

Do note that if your platform and your EPs supports ATS then I think the behaviour can be different again because then the P2P TLPs should be able to use a translated address and you can setup ACS to allow for translated  addresses. However we have not had access to any ATS enabled systems to work on this aspect of p2pdma yet. It's on the roadmap.

Cheers

Stephen




From: Eric Wehage <Eric.Wehage@huawei.com>
Sent: December 31, 2018 11:20 PM
To: Bjorn Helgaas; yu
Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Logan Gunthorpe; Stephen Bates; Jonathan Cameron; Alexander Duyck
Subject: RE: How to force RC to forward p2p TLPs
  

There is no method to force an RC to forward requests p2p. RC's are also not required to support p2p and whether they do or not is implementation specific. There is also currently no PCIe based standard to determine whether an RC supports  p2p or in what capacity. Also, even if p2p is supported, it might not be supported in a system that enables Virtualization (the Virtualization would have to setup p2p tables to allow it).

There is currently a PCIe ECR that has been submitted, but is undergoing review, that defines a PCIe capability for bridge devices (Root Ports and Switch Downstream Ports) that would allow them to advertise what level of p2p is supported by the device.

Eric Wehage
Huawei Principal Engineer, PCIe

-----Original Message-----
From: Bjorn Helgaas [mailto:helgaas@kernel.org]
Sent: Friday, December 28, 2018 6:29 PM
To: yu <kkndyu@gmail.com>
Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Logan Gunthorpe <logang@deltatee.com>; Stephen Bates <sbates@raithlin.com>; Jonathan Cameron <jonathan.cameron@huawei.com>; Eric Wehage <Eric.Wehage@huawei.com>; Alexander Duyck <alexander.h.duyck@linux.intel.com>
Subject: Re: How to force RC to forward p2p TLPs

[+cc Logan, Stephen, Jonathan, Eric, Alex]

On Sat, Dec 22, 2018 at 12:50:19PM +0800, yu wrote:
> Hi all,
> 
> We have a PCIE card which has a PEX8732 switch on-board,  and there 
> are two endpoint SOCs like graphic decoder behind the switch, and by 
> default the ACS is enabled in 8732.
> 
> We use the p2p DMA to transfer data between these two endpoint SOCs, 
> and if the host server is not enable ACS in BIOS, the p2p works well, 
> but when ACS is enabled in BIOS, the p2p is always failed. With the 
> help of a protocol analyzer, we can see that the TLP is redirected to 
> RC, and RC just discard it.
> 
> I tried to find how to make RC forward redirected TLP to its original 
> target, but nothing found, it seems this is highly related to the RC 
> vendors.
> 
> In the PCIE 4.0 spec, the section of the RC behavior of the p2p 
> request redirect  said that ''implementation-specific logic within the 
> RC that determines whether the request is directed towards its 
> original target, or blocked as an ACS Violation error. the algorithms 
> and specific controls for making this determination are not 
> architected by this spec''.
> 
> 
> So is there some spec or document to describe how to set the RC? Any 
> suggestion is appreciated.

Not that I'm aware of, but the folks I cc'd would know a lot more about this area.

Bjorn