RE: [EXT] Re: [PATCH v13 2/2] PCI: endpoint: pci-epf-vntb: using platform MSI as doorbell

Frank Li
2023-01-09T17:57:02Z

> On Thu, Nov 24 2022 at 00:50, Frank Li wrote:
> > ┌────────────┐ ┌───────────────
> ────────────────────┐ ┌────────
> ────────┐
> > │ │ │ │ │ │
> > │ │ │ PCI Endpoint │ │ PCI Host │
> > │ │ │ │ │ │
> > │ │◄──┤ 1.platform_msi_domain_alloc_irqs()│ │ │
> > │ │ │ │ │ │
> > │ MSI ├──►│ 2.write_msi_msg() ├──►├─BAR<n>
> │
> > │ Controller │ │ update doorbell register address│ │ │
> > │ │ │ for BAR │ │ │
> > │ │ │ │ │ 3. Write BAR<n>│
> > │ │◄──┼──────────────────────
> ─────────────┼───┤ │
> > │ │ │ │ │ │
> > │ ├──►│ 4.Irq Handle │ │ │
> > │ │ │ │ │ │
> > │ │ │ │ │ │
> > └────────────┘ └───────────────
> ────────────────────┘ └────────
> ────────┘
> >
> > Using platform MSI interrupt controller as endpoint(EP)'s doorbell.
>
> Can you please explain what the MSI controller is in this picture? MSI
> controller is not a term which is common in the interrupt handling
> landscape despite the fact that it's pretty wide spread in device tree
> bindings presumably through intensive copy & pasta cargo cult.
>
> > Basic working follow as
> > 1. EP function driver call platform_msi_domain_alloc_irqs() alloc a
> > MSI irq from MSI controller with call back function write_msi_msg();
> > 2. write_msg_msg will config BAR and map to address defined in msi_msg;
> > 3. Host side trigger an IRQ at Endpoint by write to BAR region.
>
> You're explaining what the code does, but fail to explain the underlying
> mechanisms.
>
> Platform MSI is definitely the wrong mechanism here. Why?
>
> This is about a PCIe endpoint, which is usually handled by a PCI/MSI
> interrupt domain. Obviously this usage does not fit into the way how the
> "global" PCI/MSI domains work.
>
> There is upcoming work and at least the generic parts should show up in
> 6.2 which addresses exactly the problem you are trying to solve:
>
@tglx

I finally get a platform, ls1028, which use gic its and also support PCIe EP function.
I port my code into linux-next, (next-20230105) and understand your comments.

Its need a device ID to allocate msi irq for each devices.

Epf device is dynamic created, where are field epc, epc.parent point to PCIe EP
Controller, which have been probed by device-tree.

I use a workaround to get msi irq number.
dev->of_node= ntb->efp->epc->dev.parent->of_node;
Platform_msi_domain_alloc_irqs(dev, ...);

Irq-gic-v3-its-pci-msi use of_node to get a ID.

You said PCI/IMS can resolve this problem. But I still not figure out it yet.
Can you give me sample code, which use PCI/IMS or provide a pseudo code?

Frank

>
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
> ernel.org%2Fall%2F20221124225331.464480443%40linutronix.de&amp;data=0
> 5%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d%7C
> 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638052596904953006%7C
> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Q8jreVGG
> La2M4yhjGO7Njqwdm59XDC0GyLEwkr0k6B0%3D&amp;reserved=0
>
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
> ernel.org%2Fall%2F20221124230505.073418677%40linutronix.de&amp;data=0
> 5%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d%7C
> 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638052596904953006%7C
> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Tc9pXNJ49
> 9ETFgNWQBNLViFk8D5GbvrrwYDlBW%2Bf2qg%3D&amp;reserved=0
>
> plus the prove that the platform MSI mess can be replaced by this:
>
>
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
> ernel.org%2Fall%2F20221121135653.208611233%40linutronix.de&amp;data=0
> 5%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d%7C
> 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638052596904953006%7C
> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=R5KNVfcG
> qxoCam%2FYhY57ihsloWGhGLM3Kh9IkyME4lk%3D&amp;reserved=0
>
> NTB in it's current form should never have happened, but that's water
> down the bridge.
>
> What you really want is:
>
> 1) Convert your platform to the new MSI parent model
>
> 2) Utilize PCI/IMS which is giving you exactly what you need with
> proper PCI semantics
>
> Thanks,
>
> tglx