From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53470) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VUOkF-0002UJ-Mv for qemu-devel@nongnu.org; Thu, 10 Oct 2013 18:30:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VUOk8-00038a-4U for qemu-devel@nongnu.org; Thu, 10 Oct 2013 18:30:11 -0400 Received: from mail-vb0-f49.google.com ([209.85.212.49]:56421) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VUOk7-00036Y-PU for qemu-devel@nongnu.org; Thu, 10 Oct 2013 18:30:04 -0400 Received: by mail-vb0-f49.google.com with SMTP id w16so2158935vbb.36 for ; Thu, 10 Oct 2013 15:30:03 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <87mwmg9313.fsf@pixel.localdomain> References: <87mwmg9313.fsf@pixel.localdomain> Date: Thu, 10 Oct 2013 18:30:03 -0400 Message-ID: From: Mike Day Content-Type: multipart/alternative; boundary=001a11c3bf2602034004e86a8bf8 Subject: Re: [Qemu-devel] [RFC] SPAPR-PCI Hotplug Support in Qemu List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org, qemu-ppc@nongnu.org, Anthony Liguori Cc: Alexey Kardashevskiy , David Gibson --001a11c3bf2602034004e86a8bf8 Content-Type: text/plain; charset=ISO-8859-1 Adding Anthony's corrected address. On Thu, Oct 10, 2013 at 6:25 PM, Mike Day wrote: [RFC] SPAPR-PCI Hotplug Support in Qemu > > Background: > ppc64 has a unique bus structure for PCI slots: each slot is connected > to its PHB by a pci switch. This is true in some IBM hardware as well as > paravirtual hardware (PAPR). > > SLOF firmware normally scans the hardware bus and creates the correct > slot/PCI switch -complex in the open firmware device tree. It also > configures the slot and PCI switch (BARs, etc.) > > For devices set up by platform firmware, each PCI device is attached to > its PHB and correctly configured. > > For Linux hot-plugged devices running under PowerVM today, each device > is created with a PCI switch hanging off the dev->subordinate > pointer. (PowerVM gets this info from the open firmware device tree in > rtas.) > > Problem: > > The Qemu hot-plug path doesn't anticipate a PCI switch being attached > to every PHB slot. > > When hot-plugging a device, Qemu qdev creates the device, which allows > the device to initialize itself. Qemu then passes this initialized > device to the ppc PHB via the hot-plug path.[1] > > The current ppc hot-plug code then creates a device tree node for the > device [2], and allocates resources (BARs etc) for the new device. [3] > > The ppc64 kernel expects each hot-plugged PCI device structure to > point to a subordinate bus dev->subordinate. This assumption is held > throughout the ppc PCI code, and there are numerous opportunities for > panics when the device gets passed to a kernel routine with a > subordinate pointer. [4] > > > Proposed Solutions: > > (1) Create and hook an inert PCI switch to every hot-plugged PCI > device in Qemu. > > (a) After the device has initialized itself, at hot-plug time, create a > new PCI switch, configure the switch, allocate BARs, and attach the > switch to the hot-plugged devices (dev->subordinate). > > (b) create a new device tree node that begins with the PCI switch and > the parent of the hot-plugged device. Add the PCI switch/device > complex to the device tree under the PHB. > > (2) Add each hot-plugged PCI device to its own complex of PHB > (Processor Host Bus) and PCI switch. > > Simplify (1) by creating a new PHB for each hot-plugged device. > > (a) At PHB creation time, create a PCI switch device node for each PHB > slot. > > (b) At hot-plug time, create and configure a new PHB and add the > hot-plugged device to one of the slots. Configure and allocate > resources as normally. > > Comments: > > The current code has only one PHB. We know we need to support more > than one PHB ultimately. Solution #2 is consistent with this approach. > > > [1] https://github.com/mdroth/qemu/blob/spapr-pci-hotplug/hw/ppc/spapr_pci.c > > [2] ibm,rtas_configure_connector: > https://github.com/mdroth/qemu/blob/spapr-pci-hotplug/hw/ppc/spapr_pci.c#L575 > > [3] spapr_phb_add_pci_dt > https://github.com/mdroth/qemu/blob/spapr-pci-hotplug/hw/ppc/spapr_pci.c#L900 > > [4] dlpar_pci_add_bus > http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/drivers/pci/hotplug/rpadlpar_core.c?id=8bf3379a74bc9132751bfa685bad2da318fd59d7#n165 > > > -- > > Mike Day | + 1 919 371-8786 | ncmike@ncultra.org > "Endurance is a Virtue" --001a11c3bf2602034004e86a8bf8 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Adding Anthony's corrected address.

On Thu, Oct 10, 2013 at 6:25 PM, Mike Day <ncmike@ncultra.org> wrote:
[RFC] SPAPR-PCI = Hotplug Support in Qemu
>
> Background:
> ppc64 has a unique bus structure for PCI s= lots: each slot is connected
> to its PHB by a pci switch. This is tr= ue in some IBM hardware as well as
> paravirtual hardware (PAPR).
>
> SLOF firmware normally scans the hardware bus and creates the = correct
> slot/PCI switch -complex in the open firmware device tree. = It also
> configures the slot and PCI switch (BARs, etc.)
>
> For devices set up by platform firmware, each PCI device is attached t= o
> its PHB and correctly configured.
>
> For Linux hot-p= lugged devices running under PowerVM today, each device
> is created = with a PCI switch hanging off the dev->subordinate
> pointer. (PowerVM gets this info from the open firmware device tree in=
> rtas.)
>
> Problem:
>
> The Qemu hot-plug = path doesn't anticipate a PCI switch being attached
> to every PH= B slot.
>
> When hot-plugging a device, Qemu qdev creates the device, whic= h allows
> the device to initialize itself. Qemu then passes this ini= tialized
> device to the ppc PHB via the hot-plug path.[1]
> > The current ppc hot-plug code then creates a device tree node for the<= br>> device [2], and allocates resources (BARs etc) for the new device. = [3]
>
> The ppc64 kernel expects each hot-plugged PCI device st= ructure to
> point to a subordinate bus dev->subordinate. This assumption is hel= d
> throughout the ppc PCI code, and there are numerous opportunities= for
> panics when the device gets passed to a kernel routine with a<= br> > subordinate pointer. [4]
>
>
> Proposed Solutions:>
> (1) Create and hook an inert PCI switch to every hot-plugged= PCI
> device in Qemu.
>
> (a) After the device has initi= alized itself, at hot-plug time, create a
> new PCI switch, configure the switch, allocate BARs, and attach the> switch to the hot-plugged devices (dev->subordinate).
>
&= gt; (b) create a new device tree node that begins with the PCI switch and > the parent of the hot-plugged device. Add the PCI switch/device
>= ; complex to the device tree under the PHB.
>
> (2) Add each ho= t-plugged PCI device to its own complex of PHB
> (Processor Host Bus)= and PCI switch.
>
> Simplify (1) by creating a new PHB for each hot-plugged device= .
>
> (a) At PHB creation time, create a PCI switch device node= for each PHB
> slot.
>
> (b) At hot-plug time, create an= d configure a new PHB and add the
> hot-plugged device to one of the slots. Configure and allocate
>= resources as normally.
>
> Comments:
>
> The curre= nt code has only one PHB. We know we need to support more
> than one = PHB ultimately. Solution #2 is consistent with this approach.
>
>
> [1] https://github.com/mdroth/qemu/blob/spap= r-pci-hotplug/hw/ppc/spapr_pci.c
>
> [2] ibm,rtas_configure= _connector:
> https://github.com/mdroth/qemu/blob/spapr-pci-hotplug/h= w/ppc/spapr_pci.c#L575
>
> [3] spapr_phb_add_pci_dt
>= https://github.com/mdroth/qemu/blob/spapr-pci-hotplug/hw/pp= c/spapr_pci.c#L900
>
> [4] dlpar_pci_add_bus
> http://git= .kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/drivers/pci/= hotplug/rpadlpar_core.c?id=3D8bf3379a74bc9132751bfa685bad2da318fd59d7#n165<= /a>
>
>
> --
>
> Mike Day | + 1 919 371-8786 | ncmike@ncultra.org
> "Endur= ance is a Virtue" --001a11c3bf2602034004e86a8bf8--