[RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

[RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

[Jonathan Cameron]

Hi All,

This is perhaps a somewhat unusual RFC as it's to ask for feedback on an ECR
that is currently under discussion in the PCI SIG.  That ECR, at least partly,
came out of the issues highlighted during the discussion of P2PDMA.  In
that patch set, p2pdma is only allowed if there is a switch between the
devices involved ensuring the path does not pass through a root port.
Put simply there is no way to know if a root complex will allow p2p or not.

So we are asking for feedback at a stage where modifications to the
proposal can be made. Eric has agreed to this being shared with mailing list
to hopefully get such feedback.  Note this is under active discussion within
the SIG so inputs in that forum also welcome.

I would recommend reading the ECR and presentation for more details.
Anyone who can't get to Google Drive, email me and I'll send a copy directly.

ECR: https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
Slides: https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCK

My summary probably won't be great, but I'll give it a go to persuade people
to look at the real docs!

Problem Statement:

Linux does not generally enable Peer to Peer because (amongst other reasons)
it does not know which Root Complexes support Peer to Peer.

* Some Root Complexes do not support Peer to Peer
* Some Root Complexes only support Posted Peer to Peer
* Some Root Complexes only support Peer to Peer within one PCIe Segment
* Some Root Complexes only support Address Decode Peer to Peer (no ID routing)
* Some Root Complexes may not support Peer to Peer from Integrated End Points

Related other issue
* MPS must be programmed the same across all devices supporting P2P
* Hot plugged devices might require a smaller MPS than was programmed.

Solution:
* New extended capability and control (though only one bit in the control)
* Advertise what a Peer to Peer Root Port and Switch Down Stream ports can do.
* Define packet splitting scheme for MPS mismatches.
	- Rules on how this must be done if supported.

All feedback welcome.  Let's make sure we get this right.

Thanks,

Jonathan Cameron
Huawei

Re: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

[Logan Gunthorpe]

Hi Jonathan,

Thanks for taking this on.

On 2018-12-10 4:56 a.m., Jonathan Cameron wrote:
> I would recommend reading the ECR and presentation for more details.
> Anyone who can't get to Google Drive, email me and I'll send a copy directly.
> 
> ECR: https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
> Slides: https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCK

I've looked through the ECR now a couple times and haven't really
spotted any major issues with it.

The minimum supported bandwidth requirement is pretty vague and my
biggest concern is that hardware vendors must report consistently
otherwise this is useless. So I'd maybe suggest expanding on what's
required before hardware is allowed to indicate support.

Also, one thought is, it might be better to add another bit indicating
performance. So the hardware can say: these TLPs will be supported but
they will not be supported at high bandwidth.

In terms of defining high bandwidth: what I care about is whether the
bandwidth through the P2P link will be significantly slower than the
bandwidth of going to/from system RAM. If that's the case it probably
means a subset of P2P applications would rather just transfer to RAM
instead of going P2P.

Logan

RE: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

[Eric Wehage]

Hi Logan,

	Thank you for reviewing this.

	Although I mentioned the minimum bandwidth requirement in the introduction of the ECR, I did not include any specification changes that would require this. I recognized that for p2p to be useful, there must be a useful level of bandwidth, however, I could see no easy way to adequately advertise this. Ultimately, bandwidth levels could be seen as part of the platform architecture of each OEM. Performance differences between OEM's is often what makes a buyer choose one OEM's system over another in a competitive environment. So requiring a particular bandwidth level is like requiring an OEM to include a minimum number of cores in a system or a minimum number of PCIe links. So ultimately, I excluded any discussion of bandwidth in the ECR spec changes.

	However, if you see a particular way that this issue could be approached, I am still willing to consider a change.

Eric Wehage
Huawei Principal Engineer, PCIe

-----Original Message-----
From: Logan Gunthorpe [mailto:logang@deltatee.com] 
Sent: Monday, December 10, 2018 9:20 AM
To: Jonathan Cameron <jonathan.cameron@huawei.com>; linux-pci@vger.kernel.org
Cc: bhelgaas@google.com; lorenzo.pieralisi@arm.com; Eric Wehage <Eric.Wehage@huawei.com>; Linuxarm <linuxarm@huawei.com>; Stephen Bates <sbates@raithlin.com>
Subject: Re: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

Hi Jonathan,

Thanks for taking this on.

On 2018-12-10 4:56 a.m., Jonathan Cameron wrote:
> I would recommend reading the ECR and presentation for more details.
> Anyone who can't get to Google Drive, email me and I'll send a copy directly.
> 
> ECR: 
> https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
> Slides: 
> https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCK

I've looked through the ECR now a couple times and haven't really spotted any major issues with it.

The minimum supported bandwidth requirement is pretty vague and my biggest concern is that hardware vendors must report consistently otherwise this is useless. So I'd maybe suggest expanding on what's required before hardware is allowed to indicate support.

Also, one thought is, it might be better to add another bit indicating performance. So the hardware can say: these TLPs will be supported but they will not be supported at high bandwidth.

In terms of defining high bandwidth: what I care about is whether the bandwidth through the P2P link will be significantly slower than the bandwidth of going to/from system RAM. If that's the case it probably means a subset of P2P applications would rather just transfer to RAM instead of going P2P.

Logan

Re: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

[Logan Gunthorpe]

On 2018-12-10 11:02 a.m., Eric Wehage wrote:
> Hi Logan,
> 
> 	Thank you for reviewing this.
> 
> 	Although I mentioned the minimum bandwidth requirement in the introduction of the ECR, I did not include any specification changes that would require this. I recognized that for p2p to be useful, there must be a useful level of bandwidth, however, I could see no easy way to adequately advertise this. Ultimately, bandwidth levels could be seen as part of the platform architecture of each OEM. Performance differences between OEM's is often what makes a buyer choose one OEM's system over another in a competitive environment. So requiring a particular bandwidth level is like requiring an OEM to include a minimum number of cores in a system or a minimum number of PCIe links. So ultimately, I excluded any discussion of bandwidth in the ECR spec changes.
> 
> 	However, if you see a particular way that this issue could be approached, I am still willing to consider a change.

Yeah, per my last email, I don't want to advertise bandwidth. And I gave
a rough rule as to what might qualify as "sufficient" bandwidth (ie. a
boolean indicating whether it is roughly expected to be as fast or
faster than RAM).

I also was trying to get the point across that even if a particular
piece of hardware does not have "sufficient" bandwidth it would still be
valuable to publish whether the link is expected to work for the
different TLP types. Some applications don't care about bandwidth and
just want to know if they can ring doorbells, etc.

Logan

Re: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

[Stephen Bates]

Hi Eric/Jonathan

I'd also like to thank you for taking this work on.

> However, if you see a particular way that this issue could be approached, I am still 
> willing to consider a change.

I would echo Logan's sentiments that a simple Boolean to indicate the p2p performance between RPs is comparable (or better) than RP<->DRAM performance or not would be useful and sufficient. It might be useful to at least add such a field to the ECR so it can be discussed within PCI-SIG.

Cheers
 
Stephen
 

RE: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

[Eric Wehage]

There are 5 TLP types described in each of the three P2P Routes. I'm
not sure that anyone would care about Message Bandwidth, so I
could simply add 3 bits to each Route register indicating whether the
P2P route bandwidth supports the same or more bandwidth than if it
targeted DRAM.

The three RO/HWInit bits for each route register would be:
- P2P Memory Read Bandwidth is equal to or greater than to Main
   Memory
- P2P Memory Write Bandwidth is equal to or greater than to Main
   Memory
- P2P Memory AtomicOp Bandwidth is equal to or greater than to Main
   Memory

Eric Wehage
Huawei Principal Engineer, PCIe

Re: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

[Logan Gunthorpe]

On 2018-12-10 12:36 p.m., Eric Wehage wrote:
> There are 5 TLP types described in each of the three P2P Routes. I'm
> not sure that anyone would care about Message Bandwidth, so I
> could simply add 3 bits to each Route register indicating whether the
> P2P route bandwidth supports the same or more bandwidth than if it
> targeted DRAM.
> 
> The three RO/HWInit bits for each route register would be:
> - P2P Memory Read Bandwidth is equal to or greater than to Main
>    Memory
> - P2P Memory Write Bandwidth is equal to or greater than to Main
>    Memory
> - P2P Memory AtomicOp Bandwidth is equal to or greater than to Main
>    Memory

Yeah! That sounds great. Thanks.

Logan

Re: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

[Jonathan Cameron]

On Mon, 10 Dec 2018 12:39:36 -0700
Logan Gunthorpe <logang@deltatee.com> wrote:

> On 2018-12-10 12:36 p.m., Eric Wehage wrote:
> > There are 5 TLP types described in each of the three P2P Routes. I'm
> > not sure that anyone would care about Message Bandwidth, so I
> > could simply add 3 bits to each Route register indicating whether the
> > P2P route bandwidth supports the same or more bandwidth than if it
> > targeted DRAM.
> > 
> > The three RO/HWInit bits for each route register would be:
> > - P2P Memory Read Bandwidth is equal to or greater than to Main
> >    Memory
> > - P2P Memory Write Bandwidth is equal to or greater than to Main
> >    Memory
> > - P2P Memory AtomicOp Bandwidth is equal to or greater than to Main
> >    Memory  
> 
> Yeah! That sounds great. Thanks.
> 
> Logan

Updated ECR and presentation at:
Presentation: https://drive.google.com/open?id=1fgwz8w32K2ju9aIySP8RXwQwUAcT9icF
ECR: https://drive.google.com/open?id=1cHn4OHgTJpa0KkiVab_CyK1NMjPap3MM

A few small editorial bits and pieces.

The main change is making the Memory Read, Memory Write and Atomic capabilities
in addition to indicating they support peer to peer at all, can now give an
indication of supporting 'same bandwidth as to system memory'.

Note that from a PCI viewpoint system memory really means anything beyond the
host so caches etc are all rolled into that term.

Thanks,

Eric and Jonathan

Re: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

[Logan Gunthorpe]

On 2018-12-13 2:23 a.m., Jonathan Cameron wrote:
> Updated ECR and presentation at:
> Presentation: https://drive.google.com/open?id=1fgwz8w32K2ju9aIySP8RXwQwUAcT9icF
> ECR: https://drive.google.com/open?id=1cHn4OHgTJpa0KkiVab_CyK1NMjPap3MM
> 
> A few small editorial bits and pieces.
> 
> The main change is making the Memory Read, Memory Write and Atomic capabilities
> in addition to indicating they support peer to peer at all, can now give an
> indication of supporting 'same bandwidth as to system memory'.
> 
> Note that from a PCI viewpoint system memory really means anything beyond the
> host so caches etc are all rolled into that term.

Looks good to me. Thanks,

Logan

Re: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

[Alexander Duyck]

On Mon, 2018-12-10 at 11:56 +0000, Jonathan Cameron wrote:
> Hi All,
> 
> This is perhaps a somewhat unusual RFC as it's to ask for feedback on an ECR
> that is currently under discussion in the PCI SIG.  That ECR, at least partly,
> came out of the issues highlighted during the discussion of P2PDMA.  In
> that patch set, p2pdma is only allowed if there is a switch between the
> devices involved ensuring the path does not pass through a root port.
> Put simply there is no way to know if a root complex will allow p2p or not.
> 
> So we are asking for feedback at a stage where modifications to the
> proposal can be made. Eric has agreed to this being shared with mailing list
> to hopefully get such feedback.  Note this is under active discussion within
> the SIG so inputs in that forum also welcome.
> 
> I would recommend reading the ECR and presentation for more details.
> Anyone who can't get to Google Drive, email me and I'll send a copy directly.
> 
> ECR: https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
> Slides: https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCK
> 
> My summary probably won't be great, but I'll give it a go to persuade people
> to look at the real docs!
> 
> Problem Statement:
> 
> Linux does not generally enable Peer to Peer because (amongst other reasons)
> it does not know which Root Complexes support Peer to Peer.
> 
> * Some Root Complexes do not support Peer to Peer
> * Some Root Complexes only support Posted Peer to Peer
> * Some Root Complexes only support Peer to Peer within one PCIe Segment
> * Some Root Complexes only support Address Decode Peer to Peer (no ID routing)
> * Some Root Complexes may not support Peer to Peer from Integrated End Points
> 
> Related other issue
> * MPS must be programmed the same across all devices supporting P2P
> * Hot plugged devices might require a smaller MPS than was programmed.
> 
> Solution:
> * New extended capability and control (though only one bit in the control)
> * Advertise what a Peer to Peer Root Port and Switch Down Stream ports can do.
> * Define packet splitting scheme for MPS mismatches.
> 	- Rules on how this must be done if supported.
> 
> All feedback welcome.  Let's make sure we get this right.
> 
> Thanks,
> 
> Jonathan Cameron
> Huawei

Looking this over some of this seems redundant with the ACS P2P Request
Redirect and Completion Redirect functionality.

I was curious if there was a reason why you couldn't use something like
that to at least get some early enabling going for peer-to-peer root
port pass through?

Also, I have not seen called out anywhere in your documents how this
might interact with something like Address Translation Services (ATS).
I'm assuming there would be scenarios where the translation agent would
need to be a part of the p2p topology.

Thanks.

- Alex