P2PDMA support status for the sappire rapids+

P2PDMA support status for the sappire rapids+

[Damir Chanyshev]

Hello experts!
First thank you so much for everything.

I have a question regarding p2pdma support, I am not an expert in the
kernel subsystems, please pardon my stupidity.
While investigating performance opportunities, I stumbled with dma
between cpu root complex and pcie switch root complex. And found white
list for the Intel platforms [1]

Typical topology looks like this rdma nic<>cpu<>pcie switch<>gpu/xpu,
for each socket.
Questions:
List not updated because new hardware doesn't support this feature?
(For example abandoned for the CXL3+ )
Or just not tested for the new platforms?

[1]
https://lore.kernel.org/all/20220209162801.7647-1-michael.j.ruhl@intel.com/T/#m3f4e4194770274c2873a130ad684a74469038585

P.S.
Pardon my tone, if it sounds harsh, don't mind it, unfortunately
English is my third language.
Thank you in advance, I appreciate all input.


-- 
Thanks,
Damir Chanyshev

Re: P2PDMA support status for the sappire rapids+

[Bjorn Helgaas]

[+cc Logan, Michael]

On Tue, Apr 08, 2025 at 05:48:16PM +0300, Damir Chanyshev wrote:
> Hello experts!
> First thank you so much for everything.
> 
> I have a question regarding p2pdma support, I am not an expert in the
> kernel subsystems, please pardon my stupidity.
> While investigating performance opportunities, I stumbled with dma
> between cpu root complex and pcie switch root complex. And found white
> list for the Intel platforms [1]
> 
> Typical topology looks like this rdma nic<>cpu<>pcie switch<>gpu/xpu,
> for each socket.
> Questions:
> List not updated because new hardware doesn't support this feature?
> (For example abandoned for the CXL3+ )
> Or just not tested for the new platforms?
> 
> [1]
> https://lore.kernel.org/all/20220209162801.7647-1-michael.j.ruhl@intel.com/T/#m3f4e4194770274c2873a130ad684a74469038585

The pci_p2pdma_whitelist[] is only updated when somebody tests a
platform and determines that its RC can route peer-to-peer
transactions between separate Root Ports.  This routing is not
required by the PCIe spec, so we can't assume that all platforms
support it.

From PCIe r6.0, sec 1.3.1:

  The capability to route peer-to-peer transactions between hierarchy
  domains through an RC is optional and implementation dependent.

Bjorn

Re: P2PDMA support status for the sappire rapids+

[Damir Chanyshev]

On Tue, Apr 8, 2025 at 6:06 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc Logan, Michael]
>
>
> The pci_p2pdma_whitelist[] is only updated when somebody tests a
> platform and determines that its RC can route peer-to-peer
> transactions between separate Root Ports.  This routing is not
> required by the PCIe spec, so we can't assume that all platforms
> support it.
>
> From PCIe r6.0, sec 1.3.1:
>
>   The capability to route peer-to-peer transactions between hierarchy
>   domains through an RC is optional and implementation dependent.
>
> Bjorn

Thank you so much for the fast response!

For the testing, is simple rdma ping-pong sufficient? Or a more
specific testing tool?
Unfortunately I don't have a fancy pcie hardware analyzer.

-- 
Thanks,
Damir Chanyshev

Re: P2PDMA support status for the sappire rapids+

[Lukas Wunner]

On Tue, Apr 08, 2025 at 06:14:55PM +0300, Damir Chanyshev wrote:
> On Tue, Apr 8, 2025 at 6:06???PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > The pci_p2pdma_whitelist[] is only updated when somebody tests a
> > platform and determines that its RC can route peer-to-peer
> > transactions between separate Root Ports.  This routing is not
> > required by the PCIe spec, so we can't assume that all platforms
> > support it.
> 
> For the testing, is simple rdma ping-pong sufficient? Or a more
> specific testing tool?
> Unfortunately I don't have a fancy pcie hardware analyzer.

You should test that performance is satisfactory for you needs.

There are claims that older Intel CPUs have poor p2pdma performance:
https://lore.kernel.org/all/e2c56983-0111-4132-ae17-0095aa33fd2d@deltatee.com/

Thanks,

Lukas

Re: P2PDMA support status for the sappire rapids+

[Logan Gunthorpe]

On 2025-04-08 09:28, Lukas Wunner wrote:
> On Tue, Apr 08, 2025 at 06:14:55PM +0300, Damir Chanyshev wrote:
>> On Tue, Apr 8, 2025 at 6:06???PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>>> The pci_p2pdma_whitelist[] is only updated when somebody tests a
>>> platform and determines that its RC can route peer-to-peer
>>> transactions between separate Root Ports.  This routing is not
>>> required by the PCIe spec, so we can't assume that all platforms
>>> support it.
>>
>> For the testing, is simple rdma ping-pong sufficient? Or a more
>> specific testing tool?
>> Unfortunately I don't have a fancy pcie hardware analyzer.
> 
> You should test that performance is satisfactory for you needs.

I couldn't have said it better myself.

If you have any application running on a machine that uses P2P then that
is sufficient testing in and of itself to add the machine to the list.

My expectation is that any modern Intel root complex should work, but we
don't have a concise rule that guarantees this so we are stuck in the
unfortunate situation to add each family as they are tested by someone
who cares about it.

AMD confirmed for us that all modern devices will support P2P and
provided a concise rule that works for all current and future devices.

My limited experience with ARM devices suggest many root complexes do
not support P2P, so they will likely always have to be added on a case
by case basis.

Logan

Re: P2PDMA support status for the sappire rapids+

[Lukas Wunner]

On Tue, Apr 08, 2025 at 05:48:16PM +0300, Damir Chanyshev wrote:
> I have a question regarding p2pdma support, I am not an expert in the
> kernel subsystems, please pardon my stupidity.
> While investigating performance opportunities, I stumbled with dma
> between cpu root complex and pcie switch root complex. And found white
> list for the Intel platforms [1]
> 
> Typical topology looks like this rdma nic<>cpu<>pcie switch<>gpu/xpu,
> for each socket.
> Questions:
> List not updated because new hardware doesn't support this feature?
> (For example abandoned for the CXL3+ )
> Or just not tested for the new platforms?

Basically just lack of resources to do proper testing and amend the list
on a regular basis.

Feel free to submit a patch amending the list for newer platforms if
you got p2pdma working on them.

> Pardon my tone, if it sounds harsh, don't mind it, unfortunately
> English is my third language.

That's fine but it would help us if you could clarify whether you're
associated with an OFAC sanctioned entity.  Unfortunately we're now
forced to perform background checks on contributors before interacting
with them:

https://www.linuxfoundation.org/blog/navigating-global-regulations-and-open-source-us-ofac-sanctions

I note that there's a developer with your name based in Serbia.
If that's you, you're probably not associated with a sanctioned entity:

https://rs.linkedin.com/in/damir-chanyshev

However there is another person based in Saint Petersberg and the
situation is less clear in their case:

https://ru.linkedin.com/in/chanyshv
https://github.com/chanyshv

I am very sorry it has come to this.

Thanks,

Lukas