Re: [early RFC] ARM PCI Passthrough design document

["Edgar E. Iglesias" <edgar.iglesias@xilinx.com>]

On Mon, Feb 13, 2017 at 03:35:19PM +0000, Julien Grall wrote:
> On 02/02/17 15:33, Edgar E. Iglesias wrote:
> >On Wed, Feb 01, 2017 at 07:04:43PM +0000, Julien Grall wrote:
> >>On 31/01/2017 19:06, Edgar E. Iglesias wrote:
> >>>On Tue, Jan 31, 2017 at 05:09:53PM +0000, Julien Grall wrote:
> >>Thank you for the documentation. I am trying to understand if we could move
> >>initialization in Xen as suggested by Stefano. I looked at the driver in
> >>Linux and the code looks simple not many dependencies. However, I was not
> >>able to find where the Gigabit Transceivers are configured. Do you have any
> >>link to the code for that?
> >
> >Hi Julien,
> 
> Hi Edgar,

Hi Julien,

Sorry for the late reply..


> 
> >
> >I suspect that this setup has previously been done by the initial bootloader
> >auto-generated from design configuration tools.
> >
> >Now, this is moving into Linux.
> 
> Do you know why they decide to move the code in Linux? What would be the
> problem to let the bootloader configuring the GT?


No, I'm not sure why this approach was not used. The only thing I can think of
is a runtime configuration approach.


> 
> >There's a specific driver that does that but AFAICS, it has not been upstreamed yet.
> >You can see it here:
> >https://github.com/Xilinx/linux-xlnx/blob/master/drivers/phy/phy-zynqmp.c
> >
> >DTS nodes that need a PHY can then just refer to it, here's an example from SATA:
> >&sata {
> >        phy-names = "sata-phy";
> >        phys = <&lane3 PHY_TYPE_SATA 1 3 150000000>;
> >};
> >
> >I'll see if I can find working examples for PCIe on the ZCU102. Then I'll share
> >DTS, Kernel etc.
> 
> I've found a device tree on the github from the ZCU102: zynqmp-zcu102.dts,
> it looks like there is no use of PHY for the pcie so far.
> 
> Lets imagine in the future, pcie will use the PHY. If we decide to
> initialize the hostbridge in Xen, we would also have to pull the PHY code in
> the hypervisor. Leaving aside the problem to pull more code in Xen, this is
> not nice because the PHY is used by different components (e.g SATA, USB). So
> Xen and DOM0 would have to share the PHY.
> 
> For Xen POV, the best solution would be the bootloader initializing the PHY
> because starting Xen. So we can keep all the hostbridge (initialization +
> access) in Xen.
> 
> If it is not possible, then I would prefer to see the hostbridge
> initialization in DOM0.

Yes, I agree that the GT setup in the bootloader is very attractive.
I don't think hte setup sequence is complicated, we can perhaps even do it
on the commandline in u-boot or xsdb. I'll have to check.


> 
> >
> >If you are looking for a platform to get started, an option could be if I get you a build of
> >our QEMU that includes models for the PCIe controller, MSI and SMMU connections.
> >These models are friendly wrt. PHY configs and initialization sequences, it will
> >accept pretty much any sequence and still work. This would allow you to focus on
> >architectural issues rather than exact details of init sequences (which we can
> >deal with later).
> 
> From my understanding the problem is where the hostbridge should be
> initialized. In an ideal world, I think this is the goal of the bootloader.
> If it is not possible then depending on the complexity, the initialization
> would have to be done either in Xen or DOM0.
> 
> I guess this could be decided on case by case basis. I will suggest
> different possibility in the design document.
> 
> [...]
> 
> >>
> >>From a design point of view, it would make more sense to have the MSI
> >>controller driver in Xen as the hostbridge emulation for guest will also
> >>live there.
> >>
> >>So if we receive MSI in Xen, we need to figure out a way for DOM0 and guest
> >>to receive MSI. The same way would be the best, and I guess non-PV if
> >>possible. I know you are looking to boot unmodified OS in a VM. This would
> >>mean we need to emulate the MSI controller and potentially xilinx PCI
> >>controller. How much are you willing to modify the OS?
> >
> >Today, we have not yet implemented PCIe drivers for our baremetal SDK. So
> >things are very open and we could design with pretty much anything in mind.
> >
> >Yes, we could perhaps include a very small model with most registers dummied.
> >Implementing the MSI read FIFO would allow us to:
> >
> >1. Inject the MSI doorbell SPI into guests. The guest will then see the same
> >   IRQ as on real HW.
> >
> >2. Guest reads host-controller registers (MSI FIFO) to get the signaled MSI.
> 
> The Xilinx PCIe hostbridge is not the only hostbridge having MSI controller
> embedded. So I would like to see a generic solution if possible. This would
> avoid to increase the code required for emulation in Xen.
> 
> My concern with a FIFO is it will require an upper bound to avoid using to
> much memory in Xen. What if the FIFO is full? Will you drop MSI?

The FIFO I'm refering to is a FIFO in the MSI controller itself.
I agree that this wouldn't be generic though....


> 
> >>Regarding the MSI doorbell, I have seen it is configured by the software
> >>using a physical address of a page allocated in the RAM. When the PCI
> >>devices is writing into the doorbell does the access go through the SMMU?
> >
> >That's a good question. On our QEMU model it does, but I'll have to dig a little to see if that is the case on real HW aswell.
> >
> >>Regardless the answer, I think we would need to map the MSI doorbell page in
> >>the guest. Meaning that even if we trap MSI configuration access, a guess
> >>could DMA in the page. So if I am not mistaken, MSI would be insecure in
> >>this case :/.
> >>
> >>Or maybe we could avoid mapping the doorbell in the guest and let Xen
> >>receive an SMMU abort. When receiving the SMMU abort, Xen could sanitize the
> >>value and write into the real MSI doorbell. Not sure if it would works
> >>thought.
> >
> >Yeah, this is a problem.
> >I'm not sure if SMMU aborts would work because I don't think we know the value of the data written when we take the abort.
> >Without the data, I'm not sure how we would distinguish between different MSI's from the same device.
> 
> You are right, you don't get the data written and therefore it is not
> possible to distinguish MSIs. I got confused with the data abort trap.
> 
> >
> >Also, even if the MSI doorbell would be protected by the SMMU, all PCI devices are presented with the same AXI Master ID.
> >BTW, this master-ID SMMU limitation is a showstopper for domU guests isn't it?
> 
> That's limitation is only for your current version of the hardware correct?

Yes :-)


> 
> >Or do you have ideas around that? Perhaps some PV way to request mappings for DMA?
> 
> Guest memory would have to be direct mapped as we do for DOM0. However, it
> means the guest should be able to parse the firmware table (DT, ACPI) in
> order to know where the RAM banks has been positioned.
> 
> Cheers,
> 
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel