From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.free-electrons.com ([94.23.35.102]:60902 "EHLO mail.free-electrons.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753759Ab3A2IAm (ORCPT ); Tue, 29 Jan 2013 03:00:42 -0500 Date: Tue, 29 Jan 2013 09:00:36 +0100 From: Thomas Petazzoni To: Jason Gunthorpe Cc: Bjorn Helgaas , linux-pci@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Jason Cooper , Andrew Lunn , Gregory Clement , Arnd Bergmann , Maen Suleiman , Lior Amsalem , Thierry Reding , Eran Ben-Avi , Nadav Haklai , Shadi Ammouri , Tawfik Bayouk , Stephen Warren , Russell King - ARM Linux Subject: Re: [PATCH v2 19/27] pci: PCIe driver for Marvell Armada 370/XP systems Message-ID: <20130129090036.64c76c25@skate> In-Reply-To: <20130129055508.GA3339@obsidianresearch.com> References: <1359399397-29729-1-git-send-email-thomas.petazzoni@free-electrons.com> <1359399397-29729-20-git-send-email-thomas.petazzoni@free-electrons.com> <20130129055508.GA3339@obsidianresearch.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Sender: linux-pci-owner@vger.kernel.org List-ID: Bjorn, Jason, On Mon, 28 Jan 2013 22:55:08 -0700, Jason Gunthorpe wrote: > > There's no Linux requirement that multiple PCIe interfaces appear to > > be in the same hierarchy. You can just use pci_scan_root_bus() > > separately on each interface. Each interface can be in its own domain > > if necessary. > > What you suggest is basically what the Marvell driver did originally, > the probelm is that Linux requires a pre-assigned aperture for each > PCI domain/root bus, and these new chips have so many PCI-E ports that > they can exhaust the physical address space, and also a limited > internal HW resource for setting address routing. > > Thus they require resource allocation that is sensitive to the devices > present downstream. > > By far the simplest solution is to merge all the physical links into a > single domain and rely on existing PCI resource allocation code to > drive allocation of scarce physical address space and demand allocate > the HW routing resource (specifically there are enough resources to > accomidate MMIO only devices on every bus, but not enough to > accomidate MMIO and IO on every bus). > > > > +/* > > > + * For a given PCIe interface (represented by a mvebu_pcie_port > > > + * structure), we read the PCI configuration space of the > > > + * corresponding PCI-to-PCI bridge in order to find out which range of > > > + * I/O addresses and memory addresses have been assigned to this PCIe > > > + * interface. Using these informations, we set up the appropriate > > > + * address decoding windows so that the physical address are actually > > > + * resolved to the right PCIe interface. > > > + */ > > > > Are you inferring the host bridge apertures by using the resources > > assigned to devices under the bridge, i.e., taking the union of all > > The flow is different, a portion of physical address space is set > aside for use by PCI-E (via DT) and that portion is specified in the > struct resource's ultimately attached to the PCI domain for the bus > scan. You could call that the 'host bridge aperture' though it doesn't > reflect any HW configuration at all. The values come from the device > tree. > > During the bus scan the Linux core code splits up that contiguous > space and assigns to the PCI-PCI bridges and devices under that domain. > > Each physical PCI-E link on the chip is seen by Linux through the SW > emulated PCI-PCI bridge attached to bus 0. When Linux configures the > bridge windows it triggers this code here to copy that window > information from the PCI config space into non-standard internal HW > registers. > > The purpose of the SW PCI-PCI bridge and this code here is to give > the Linux PCI core control over the window (MMIO,IO,busnr) assigned > to the PCI-E link. > > This arrangement with PCI-PCI bridges controlling address routing is > part of the PCI-E standard, in this instance Marvell did not implement > the required config space in HW so the driver is working around that > deficiency. > > Other drivers, like tegra have a similar design, but their hardware > does implement PCI-PCI bridge configuration space and does drive > address decoding through the HW PCI-PCI window registers. > > Having PCI-E links be bridges, not domains/root_bus's is in-line with > the standard and works better with the Linux PCI resource allocator. Thanks a lot Jason for this explanation, I couldn't have explained it as clearly as you did. Bjorn, does Jason's reply answers your questions? Or do you need other details? Thanks! Thomas -- Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com