* Re: Understanding P2P DMA related errors
2022-10-05 15:54 ` Logan Gunthorpe
@ 2022-10-05 17:09 ` Bjorn Helgaas
2022-10-06 2:56 ` Ramesh Errabolu
2022-10-06 19:33 ` Ramesh Errabolu
2 siblings, 0 replies; 6+ messages in thread
From: Bjorn Helgaas @ 2022-10-05 17:09 UTC (permalink / raw)
To: Logan Gunthorpe; +Cc: Ramesh Errabolu, linux-pci
On Wed, Oct 05, 2022 at 09:54:50AM -0600, Logan Gunthorpe wrote:
> On 2022-10-04 22:42, Ramesh Errabolu wrote:
> > Hi,
> >
> > Thanks for taking a look at this. I will see if I can add 0x09A2 to the
> > whitelist and see what happens. But could you clarify my reading of the
> > device tree. In the tree I don't see an AMD device attached to the
> > 0x09A2 device. Is that a misread on my part? Would appreciate it if you
> > could shed light on this aspect.
Ramesh, just FIYI, your emails aren't making it to the mailing list,
probably because they're "too fancy," e.g., they are multi-part or
contain HTML. See http://vger.kernel.org/majordomo-info.html
You can see the effect at
https://lore.kernel.org/all/5d3b257a-c125-fdd6-e29f-229e54679f45@deltatee.com/
The archive contains Logan's responses, but not the emails from you
that Logan is responding to.
Bjorn
^ permalink raw reply [flat|nested] 6+ messages in thread
* Understanding P2P DMA related errors
2022-10-05 15:54 ` Logan Gunthorpe
2022-10-05 17:09 ` Bjorn Helgaas
@ 2022-10-06 2:56 ` Ramesh Errabolu
2022-10-06 15:26 ` Logan Gunthorpe
2022-10-06 19:33 ` Ramesh Errabolu
2 siblings, 1 reply; 6+ messages in thread
From: Ramesh Errabolu @ 2022-10-06 2:56 UTC (permalink / raw)
To: logang; +Cc: linux-pci, ramesh.errabolu
Logan,
You are right about AMD devices connecting to buses [0000:16] and [0000:64].
However I am unable to understand as to how you extend that to mean they
belong to Intel 0x09A2.
Per my understanding I am expecting Root Complex enumerated as a device,
with various other devices hanging off one or more ports/buses. In the
PCIe device tree, I don't see that.
I see the [domain::bus] as the root of the AMD device. Furthermore I see
Intel devices 0x09A2 hanging off the same domain::bus. I will take your
word, but the way the root complex is reported could be less confusing.
If I could make a request, it will be very helpfulf for folks who don't
dabble in this area with a simple cheat sheet plus write explaining with
examples the various root complexes and the variou end-points hanging off
of them.
Let me know if I could help in this effort.
Regards,
Ramesh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Understanding P2P DMA related errors
2022-10-06 2:56 ` Ramesh Errabolu
@ 2022-10-06 15:26 ` Logan Gunthorpe
0 siblings, 0 replies; 6+ messages in thread
From: Logan Gunthorpe @ 2022-10-06 15:26 UTC (permalink / raw)
To: Ramesh Errabolu; +Cc: linux-pci, ramesh.errabolu
On 2022-10-05 20:56, Ramesh Errabolu wrote:
>
> Logan,
>
> You are right about AMD devices connecting to buses [0000:16] and [0000:64].
> However I am unable to understand as to how you extend that to mean they
> belong to Intel 0x09A2.
Well the root bus in your tree is 09A2 and each of the 16 and 64 buses each
have a 09A2. So it's my guess that 09A2 is the root complex it just shows
up multiple times.
> Per my understanding I am expecting Root Complex enumerated as a device,
> with various other devices hanging off one or more ports/buses. In the
> PCIe device tree, I don't see that.
>
> I see the [domain::bus] as the root of the AMD device. Furthermore I see
> Intel devices 0x09A2 hanging off the same domain::bus. I will take your
> word, but the way the root complex is reported could be less confusing.
Yup. Like I said, this is a bit strange.
> If I could make a request, it will be very helpfulf for folks who don't
> dabble in this area with a simple cheat sheet plus write explaining with
> examples the various root complexes and the variou end-points hanging off
> of them.
I don't really know any more than you do here. You'd have to ask Intel what
their newer topologies imply. They keep coming up with new ways to organize
things and its not clear what it means from a P2P perspective.
But really what needs to happen is to verify P2PDMA works between ports and
find a way for the whitelist code to accept it if it does.
Logan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Understanding P2P DMA related errors
2022-10-05 15:54 ` Logan Gunthorpe
2022-10-05 17:09 ` Bjorn Helgaas
2022-10-06 2:56 ` Ramesh Errabolu
@ 2022-10-06 19:33 ` Ramesh Errabolu
2 siblings, 0 replies; 6+ messages in thread
From: Ramesh Errabolu @ 2022-10-06 19:33 UTC (permalink / raw)
To: logang; +Cc: linux-pci, ramesh.errabolu
Logan,
Wanted to thank you for all the time you have given me. I now understand
the PCIe device tree better. The thing that was throwing me off is the
way I was looking at the "lspci -tv" output.
I realized that Root Complex should be understood to mean a set of special
devices hanging of a BUS. The most important member of this set is the device
that acts as "HOST BRIDGE". This becomes apparent when the device tree is sketched
out on a piece of paper. One can then circumscribe this set logically to form
a logical device - "ROOT COMPLEX". I wish I could share my sketch via email.
Including below lspci output of this device set:
localhost:~ #
localhost:~ # lspci -vs 0000:16:00.0
16:00.0 System peripheral: Intel Corporation Device 09a2 (rev 04)
Subsystem: Intel Corporation Device 0000
Flags: fast devsel, NUMA node 0
Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
localhost:~ #
localhost:~ # lspci -vs 0000:16:00.1
16:00.1 System peripheral: Intel Corporation Device 09a4 (rev 04)
Subsystem: Intel Corporation Device 0000
Flags: fast devsel, NUMA node 0
Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
localhost:~ #
localhost:~ # lspci -vs 0000:16:00.2
16:00.2 System peripheral: Intel Corporation Device 09a3 (rev 04)
Subsystem: Intel Corporation Device 0000
Flags: fast devsel, NUMA node 0
Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
localhost:~ #
localhost:~ # lspci -vs 0000:16:00.4
16:00.4 Host bridge: Intel Corporation Device 0998
Subsystem: Intel Corporation Device 0000
Flags: fast devsel, NUMA node 0
Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
localhost:~ #
localhost:~ #
In the above log one can see the device 0000:64:00.4 playing the role of
"HOST BRIDGE' while the remaining devices 0000:64:00.0/1/2 play the role
of devices that act as ENDPOINT. I suspect, not sure, these devices play
a role in Inter-Root complex transactions. If so the whitelist should have
all these devices to support P2P traffic.
Interestingly in a patch I could find I see only 0x09A2 being added. Perhaps
the intention was to support only those systems that have 0x09A2 and not these
other devices such as 0x09A3 and 0x09A4 which may be newer.
Regards,
Ramesh
^ permalink raw reply [flat|nested] 6+ messages in thread