From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43470) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b9nZv-0002nL-RS for qemu-devel@nongnu.org; Mon, 06 Jun 2016 02:00:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1b9nZs-0000Ni-0Y for qemu-devel@nongnu.org; Mon, 06 Jun 2016 01:59:58 -0400 Date: Mon, 6 Jun 2016 15:57:59 +1000 From: David Gibson Message-ID: <20160606055759.GL9226@voom.fritz.box> References: <1464771463-37214-1-git-send-email-aik@ozlabs.ru> <201606010901.u518x1wF023568@mx0a-001b2d01.pphosted.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="9JSHP372f+2dzJ8X" Content-Disposition: inline In-Reply-To: <201606010901.u518x1wF023568@mx0a-001b2d01.pphosted.com> Subject: Re: [Qemu-devel] [PATCH qemu v17 11/12] spapr_pci/spapr_pci_vfio: Support Dynamic DMA Windows (DDW) List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexey Kardashevskiy Cc: qemu-devel@nongnu.org, qemu-ppc@nongnu.org, Alexander Graf , Alex Williamson --9JSHP372f+2dzJ8X Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jun 01, 2016 at 06:57:42PM +1000, Alexey Kardashevskiy wrote: > This adds support for Dynamic DMA Windows (DDW) option defined by > the SPAPR specification which allows to have additional DMA window(s) >=20 > The "ddw" property is enabled by default on a PHB but for compatibility > the pseries-2.5 machine (TODO: update version) and older disable it. Looks like your todo is now todone, but you need to update the commit message. > This also creates a single DMA window for the older machines to > maintain backward migration. >=20 > This implements DDW for PHB with emulated and VFIO devices. The host > kernel support is required. The advertised IOMMU page sizes are 4K and > 64K; 16M pages are supported but not advertised by default, in order to > enable them, the user has to specify "pgsz" property for PHB and > enable huge pages for RAM. >=20 > The existing linux guests try creating one additional huge DMA window > with 64K or 16MB pages and map the entire guest RAM to. If succeeded, > the guest switches to dma_direct_ops and never calls TCE hypercalls > (H_PUT_TCE,...) again. This enables VFIO devices to use the entire RAM > and not waste time on map/unmap later. This adds a "dma64_win_addr" > property which is a bus address for the 64bit window and by default > set to 0x800.0000.0000.0000 as this is what the modern POWER8 hardware > uses and this allows having emulated and VFIO devices on the same bus. >=20 > This adds 4 RTAS handlers: > * ibm,query-pe-dma-window > * ibm,create-pe-dma-window > * ibm,remove-pe-dma-window > * ibm,reset-pe-dma-window > These are registered from type_init() callback. >=20 > These RTAS handlers are implemented in a separate file to avoid polluting > spapr_iommu.c with PCI. >=20 > This changes sPAPRPHBState::dma_liobn to an array to allow 2 LIOBNs. >=20 > Signed-off-by: Alexey Kardashevskiy Looks pretty close to ready. There are a handful of nits and one real error noted below. > --- > Changes: > v17: > * fixed: "query" did return non-page-shifted value when memory hotplug is= enabled >=20 > v16: > * s/dma_liobn/dma_liobn[SPAPR_PCI_DMA_MAX_WINDOWS]/ > * s/SPAPR_PCI_LIOBN()/dma_liobn[]/ >=20 > v15: > * moved page mask filtering to PHB realize(), use "-mempath" to know > if there are huge pages > * fixed error reporting in RTAS handlers > * max window size accounts now hotpluggable memory boundaries > --- > hw/ppc/Makefile.objs | 1 + > hw/ppc/spapr.c | 5 + > hw/ppc/spapr_pci.c | 77 +++++++++--- > hw/ppc/spapr_rtas_ddw.c | 293 ++++++++++++++++++++++++++++++++++++++= ++++++ > include/hw/pci-host/spapr.h | 8 +- > include/hw/ppc/spapr.h | 16 ++- > trace-events | 4 + > 7 files changed, 383 insertions(+), 21 deletions(-) > create mode 100644 hw/ppc/spapr_rtas_ddw.c >=20 > diff --git a/hw/ppc/Makefile.objs b/hw/ppc/Makefile.objs > index c1ffc77..986b36f 100644 > --- a/hw/ppc/Makefile.objs > +++ b/hw/ppc/Makefile.objs > @@ -7,6 +7,7 @@ obj-$(CONFIG_PSERIES) +=3D spapr_pci.o spapr_rtc.o spapr_= drc.o spapr_rng.o > ifeq ($(CONFIG_PCI)$(CONFIG_PSERIES)$(CONFIG_LINUX), yyy) > obj-y +=3D spapr_pci_vfio.o > endif > +obj-$(CONFIG_PSERIES) +=3D spapr_rtas_ddw.o > # PowerPC 4xx boards > obj-y +=3D ppc405_boards.o ppc4xx_devs.o ppc405_uc.o ppc440_bamboo.o > obj-y +=3D ppc4xx_pci.o > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c > index 44e401a..6ddcda9 100644 > --- a/hw/ppc/spapr.c > +++ b/hw/ppc/spapr.c > @@ -2366,6 +2366,11 @@ DEFINE_SPAPR_MACHINE(2_6, "2.6", true); > .driver =3D "spapr-vlan", \ > .property =3D "use-rx-buffer-pools", \ > .value =3D "off", \ > + }, \ > + {\ > + .driver =3D TYPE_SPAPR_PCI_HOST_BRIDGE,\ > + .property =3D "ddw",\ > + .value =3D stringify(off),\ > }, > =20 > static void spapr_machine_2_5_instance_options(MachineState *machine) > diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c > index 68de523..bcf0360 100644 > --- a/hw/ppc/spapr_pci.c > +++ b/hw/ppc/spapr_pci.c > @@ -35,6 +35,7 @@ > #include "hw/ppc/spapr.h" > #include "hw/pci-host/spapr.h" > #include "exec/address-spaces.h" > +#include "exec/ram_addr.h" > #include > #include "trace.h" > #include "qemu/error-report.h" > @@ -45,6 +46,7 @@ > #include "hw/ppc/spapr_drc.h" > #include "sysemu/device_tree.h" > #include "sysemu/kvm.h" > +#include "sysemu/hostmem.h" > =20 > #include "hw/vfio/vfio.h" > =20 > @@ -1088,7 +1090,7 @@ static void spapr_phb_add_pci_device(sPAPRDRConnect= or *drc, > int fdt_start_offset =3D 0, fdt_size; > =20 > if (object_dynamic_cast(OBJECT(pdev), "vfio-pci")) { > - sPAPRTCETable *tcet =3D spapr_tce_find_by_liobn(phb->dma_liobn); > + sPAPRTCETable *tcet =3D spapr_tce_find_by_liobn(phb->dma_liobn[0= ]); > =20 > spapr_tce_set_need_vfio(tcet, true); > } Hang on.. I thought you'd got rid of the need for this explicit set_need_vfio() stuff. > @@ -1310,11 +1312,14 @@ static void spapr_phb_realize(DeviceState *dev, E= rror **errp) > PCIBus *bus; > uint64_t msi_window_size =3D 4096; > sPAPRTCETable *tcet; > + const unsigned windows_supported =3D > + sphb->ddw_enabled ? SPAPR_PCI_DMA_MAX_WINDOWS : 1; > =20 > if (sphb->index !=3D (uint32_t)-1) { > hwaddr windows_base; > =20 > - if ((sphb->buid !=3D (uint64_t)-1) || (sphb->dma_liobn !=3D (uin= t32_t)-1) > + if ((sphb->buid !=3D (uint64_t)-1) || (sphb->dma_liobn[0] !=3D (= uint32_t)-1) > + || ((sphb->dma_liobn[1] !=3D (uint32_t)-1) && (windows_suppo= rted > 1)) > || (sphb->mem_win_addr !=3D (hwaddr)-1) > || (sphb->io_win_addr !=3D (hwaddr)-1)) { > error_setg(errp, "Either \"index\" or other parameters must" > @@ -1329,7 +1334,9 @@ static void spapr_phb_realize(DeviceState *dev, Err= or **errp) > } > =20 > sphb->buid =3D SPAPR_PCI_BASE_BUID + sphb->index; > - sphb->dma_liobn =3D SPAPR_PCI_LIOBN(sphb->index, 0); > + for (i =3D 0; i < windows_supported; ++i) { > + sphb->dma_liobn[i] =3D SPAPR_PCI_LIOBN(sphb->index, i); > + } > =20 > windows_base =3D SPAPR_PCI_WINDOW_BASE > + sphb->index * SPAPR_PCI_WINDOW_SPACING; > @@ -1342,8 +1349,9 @@ static void spapr_phb_realize(DeviceState *dev, Err= or **errp) > return; > } > =20 > - if (sphb->dma_liobn =3D=3D (uint32_t)-1) { > - error_setg(errp, "LIOBN not specified for PHB"); > + if ((sphb->dma_liobn[0] =3D=3D (uint32_t)-1) || > + ((sphb->dma_liobn[1] =3D=3D (uint32_t)-1) && (windows_supported = > 1))) { > + error_setg(errp, "LIOBN(s) not specified for PHB"); > return; > } Hrm.. there's a bit of false generality here, since this would break if windows_supported > 2, and dma_liobn[2] was not specified. Not urgent for the initial commit though. > @@ -1461,16 +1469,18 @@ static void spapr_phb_realize(DeviceState *dev, E= rror **errp) > } > } > =20 > - tcet =3D spapr_tce_new_table(DEVICE(sphb), sphb->dma_liobn); > - if (!tcet) { > - error_setg(errp, "Unable to create TCE table for %s", > - sphb->dtbusname); > - return; > + /* DMA setup */ > + for (i =3D 0; i < windows_supported; ++i) { > + tcet =3D spapr_tce_new_table(DEVICE(sphb), sphb->dma_liobn[i]); > + if (!tcet) { > + error_setg(errp, "Creating window#%d failed for %s", > + i, sphb->dtbusname); > + return; > + } > + memory_region_add_subregion_overlap(&sphb->iommu_root, 0, > + spapr_tce_get_iommu(tcet), 0= ); > } > =20 > - memory_region_add_subregion_overlap(&sphb->iommu_root, 0, > - spapr_tce_get_iommu(tcet), 0); > - > sphb->msi =3D g_hash_table_new_full(g_int_hash, g_int_equal, g_free,= g_free); > } > =20 > @@ -1487,13 +1497,19 @@ static int spapr_phb_children_reset(Object *child= , void *opaque) > =20 > void spapr_phb_dma_reset(sPAPRPHBState *sphb) > { > - sPAPRTCETable *tcet =3D spapr_tce_find_by_liobn(sphb->dma_liobn); > + int i; > + sPAPRTCETable *tcet; > =20 > - if (tcet && tcet->nb_table) { > - spapr_tce_table_disable(tcet); > + for (i =3D 0; i < SPAPR_PCI_DMA_MAX_WINDOWS; ++i) { > + tcet =3D spapr_tce_find_by_liobn(sphb->dma_liobn[i]); > + > + if (tcet && tcet->nb_table) { > + spapr_tce_table_disable(tcet); > + } > } > =20 > /* Register default 32bit DMA window */ > + tcet =3D spapr_tce_find_by_liobn(sphb->dma_liobn[0]); > spapr_tce_table_enable(tcet, SPAPR_TCE_PAGE_SHIFT, sphb->dma_win_add= r, > sphb->dma_win_size >> SPAPR_TCE_PAGE_SHIFT); > } > @@ -1515,7 +1531,8 @@ static void spapr_phb_reset(DeviceState *qdev) > static Property spapr_phb_properties[] =3D { > DEFINE_PROP_UINT32("index", sPAPRPHBState, index, -1), > DEFINE_PROP_UINT64("buid", sPAPRPHBState, buid, -1), > - DEFINE_PROP_UINT32("liobn", sPAPRPHBState, dma_liobn, -1), > + DEFINE_PROP_UINT32("liobn", sPAPRPHBState, dma_liobn[0], -1), > + DEFINE_PROP_UINT32("liobn64", sPAPRPHBState, dma_liobn[1], -1), > DEFINE_PROP_UINT64("mem_win_addr", sPAPRPHBState, mem_win_addr, -1), > DEFINE_PROP_UINT64("mem_win_size", sPAPRPHBState, mem_win_size, > SPAPR_PCI_MMIO_WIN_SIZE), > @@ -1527,6 +1544,11 @@ static Property spapr_phb_properties[] =3D { > /* Default DMA window is 0..1GB */ > DEFINE_PROP_UINT64("dma_win_addr", sPAPRPHBState, dma_win_addr, 0), > DEFINE_PROP_UINT64("dma_win_size", sPAPRPHBState, dma_win_size, 0x40= 000000), > + DEFINE_PROP_UINT64("dma64_win_addr", sPAPRPHBState, dma64_window_add= r, > + 0x800000000000000ULL), > + DEFINE_PROP_BOOL("ddw", sPAPRPHBState, ddw_enabled, true), > + DEFINE_PROP_UINT64("pgsz", sPAPRPHBState, page_size_mask, > + (1ULL << 12) | (1ULL << 16)), > DEFINE_PROP_END_OF_LIST(), > }; > =20 > @@ -1603,7 +1625,7 @@ static const VMStateDescription vmstate_spapr_pci = =3D { > .post_load =3D spapr_pci_post_load, > .fields =3D (VMStateField[]) { > VMSTATE_UINT64_EQUAL(buid, sPAPRPHBState), > - VMSTATE_UINT32_EQUAL(dma_liobn, sPAPRPHBState), > + VMSTATE_UNUSED(4), /* dma_liobn */ > VMSTATE_UINT64_EQUAL(mem_win_addr, sPAPRPHBState), > VMSTATE_UINT64_EQUAL(mem_win_size, sPAPRPHBState), > VMSTATE_UINT64_EQUAL(io_win_addr, sPAPRPHBState), > @@ -1780,6 +1802,15 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb, > uint32_t interrupt_map_mask[] =3D { > cpu_to_be32(b_ddddd(-1)|b_fff(0)), 0x0, 0x0, cpu_to_be32(-1)}; > uint32_t interrupt_map[PCI_SLOT_MAX * PCI_NUM_PINS][7]; > + uint32_t ddw_applicable[] =3D { > + cpu_to_be32(RTAS_IBM_QUERY_PE_DMA_WINDOW), > + cpu_to_be32(RTAS_IBM_CREATE_PE_DMA_WINDOW), > + cpu_to_be32(RTAS_IBM_REMOVE_PE_DMA_WINDOW) > + }; > + uint32_t ddw_extensions[] =3D { > + cpu_to_be32(1), > + cpu_to_be32(RTAS_IBM_RESET_PE_DMA_WINDOW) > + }; > sPAPRTCETable *tcet; > PCIBus *bus =3D PCI_HOST_BRIDGE(phb)->bus; > sPAPRFDT s_fdt; > @@ -1804,6 +1835,14 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb, > _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pci-config-space-type", 0x1= )); > _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi", XICS_IRQS)); > =20 > + /* Dynamic DMA window */ > + if (phb->ddw_enabled) { > + _FDT(fdt_setprop(fdt, bus_off, "ibm,ddw-applicable", &ddw_applic= able, > + sizeof(ddw_applicable))); > + _FDT(fdt_setprop(fdt, bus_off, "ibm,ddw-extensions", > + &ddw_extensions, sizeof(ddw_extensions))); > + } > + > /* Build the interrupt-map, this must matches what is done > * in pci_spapr_map_irq > */ > @@ -1827,7 +1866,7 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb, > _FDT(fdt_setprop(fdt, bus_off, "interrupt-map", &interrupt_map, > sizeof(interrupt_map))); > =20 > - tcet =3D spapr_tce_find_by_liobn(phb->dma_liobn); > + tcet =3D spapr_tce_find_by_liobn(phb->dma_liobn[0]); > if (!tcet) { > return -1; > } > diff --git a/hw/ppc/spapr_rtas_ddw.c b/hw/ppc/spapr_rtas_ddw.c > new file mode 100644 > index 0000000..17bbae0 > --- /dev/null > +++ b/hw/ppc/spapr_rtas_ddw.c > @@ -0,0 +1,293 @@ > +/* > + * QEMU sPAPR Dynamic DMA windows support > + * > + * Copyright (c) 2015 Alexey Kardashevskiy, IBM Corporation. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; either version 2 of the License, > + * or (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program; if not, see . > + */ > + > +#include "qemu/osdep.h" > +#include "cpu.h" > +#include "qemu/error-report.h" > +#include "hw/ppc/spapr.h" > +#include "hw/pci-host/spapr.h" > +#include "trace.h" > + > +static int spapr_phb_get_active_win_num_cb(Object *child, void *opaque) > +{ > + sPAPRTCETable *tcet; > + > + tcet =3D (sPAPRTCETable *) object_dynamic_cast(child, TYPE_SPAPR_TCE= _TABLE); > + if (tcet && tcet->nb_table) { > + ++*(unsigned *)opaque; > + } > + return 0; > +} > + > +static unsigned spapr_phb_get_active_win_num(sPAPRPHBState *sphb) > +{ > + unsigned ret =3D 0; > + > + object_child_foreach(OBJECT(sphb), spapr_phb_get_active_win_num_cb, = &ret); > + > + return ret; > +} > + > +static int spapr_phb_get_free_liobn_cb(Object *child, void *opaque) > +{ > + sPAPRTCETable *tcet; > + > + tcet =3D (sPAPRTCETable *) object_dynamic_cast(child, TYPE_SPAPR_TCE= _TABLE); > + if (tcet && !tcet->nb_table) { > + *(uint32_t *)opaque =3D tcet->liobn; > + return 1; > + } > + return 0; > +} > + > +static unsigned spapr_phb_get_free_liobn(sPAPRPHBState *sphb) > +{ > + uint32_t liobn =3D 0; > + > + object_child_foreach(OBJECT(sphb), spapr_phb_get_free_liobn_cb, &lio= bn); > + > + return liobn; > +} > + > +static uint32_t spapr_page_mask_to_query_mask(uint64_t page_mask) > +{ > + int i; > + uint32_t mask =3D 0; > + const struct { int shift; uint32_t mask; } masks[] =3D { > + { 12, RTAS_DDW_PGSIZE_4K }, > + { 16, RTAS_DDW_PGSIZE_64K }, > + { 24, RTAS_DDW_PGSIZE_16M }, > + { 25, RTAS_DDW_PGSIZE_32M }, > + { 26, RTAS_DDW_PGSIZE_64M }, > + { 27, RTAS_DDW_PGSIZE_128M }, > + { 28, RTAS_DDW_PGSIZE_256M }, > + { 34, RTAS_DDW_PGSIZE_16G }, > + }; > + > + for (i =3D 0; i < ARRAY_SIZE(masks); ++i) { > + if (page_mask & (1ULL << masks[i].shift)) { > + mask |=3D masks[i].mask; > + } > + } > + > + return mask; > +} > + > +static void rtas_ibm_query_pe_dma_window(PowerPCCPU *cpu, > + sPAPRMachineState *spapr, > + uint32_t token, uint32_t nargs, > + target_ulong args, > + uint32_t nret, target_ulong ret= s) > +{ > + sPAPRPHBState *sphb; > + uint64_t buid, max_window_size; > + uint32_t avail, addr, pgmask =3D 0; > + MachineState *machine =3D MACHINE(spapr); > + > + if ((nargs !=3D 3) || (nret !=3D 5)) { > + goto param_error_exit; > + } > + > + buid =3D ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2); > + addr =3D rtas_ld(args, 0); > + sphb =3D spapr_pci_find_phb(spapr, buid); > + if (!sphb || !sphb->ddw_enabled) { > + goto param_error_exit; > + } > + > + /* Translate page mask to LoPAPR format */ > + pgmask =3D spapr_page_mask_to_query_mask(sphb->page_size_mask); > + > + /* > + * This is "Largest contiguous block of TCEs allocated specifically > + * for (that is, are reserved for) this PE". > + * Return the maximum number as maximum supported RAM size was in 4K= pages. > + */ > + if (machine->ram_size =3D=3D machine->maxram_size) { > + max_window_size =3D machine->ram_size; > + } else { > + MemoryHotplugState *hpms =3D &spapr->hotplug_memory; > + > + max_window_size =3D hpms->base + memory_region_size(&hpms->mr); > + } > + > + avail =3D SPAPR_PCI_DMA_MAX_WINDOWS - spapr_phb_get_active_win_num(s= phb); > + > + rtas_st(rets, 0, RTAS_OUT_SUCCESS); > + rtas_st(rets, 1, avail); > + rtas_st(rets, 2, max_window_size >> SPAPR_TCE_PAGE_SHIFT); > + rtas_st(rets, 3, pgmask); > + rtas_st(rets, 4, 0); /* DMA migration mask, not supported */ > + > + trace_spapr_iommu_ddw_query(buid, addr, avail, max_window_size, pgma= sk); > + return; > + > +param_error_exit: > + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); > +} > + > +static void rtas_ibm_create_pe_dma_window(PowerPCCPU *cpu, > + sPAPRMachineState *spapr, > + uint32_t token, uint32_t nargs, > + target_ulong args, > + uint32_t nret, target_ulong re= ts) > +{ > + sPAPRPHBState *sphb; > + sPAPRTCETable *tcet =3D NULL; > + uint32_t addr, page_shift, window_shift, liobn; > + uint64_t buid; > + > + if ((nargs !=3D 5) || (nret !=3D 4)) { > + goto param_error_exit; > + } > + > + buid =3D ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2); > + addr =3D rtas_ld(args, 0); > + sphb =3D spapr_pci_find_phb(spapr, buid); > + if (!sphb || !sphb->ddw_enabled) { > + goto param_error_exit; > + } > + > + page_shift =3D rtas_ld(args, 3); > + window_shift =3D rtas_ld(args, 4); > + liobn =3D spapr_phb_get_free_liobn(sphb); > + > + if (!(sphb->page_size_mask & (1ULL << page_shift)) || > + (window_shift < page_shift)) { > + goto param_error_exit; > + } > + > + if (!liobn || !sphb->ddw_enabled || > + spapr_phb_get_active_win_num(sphb) =3D=3D SPAPR_PCI_DMA_MAX_WIND= OWS) { > + goto hw_error_exit; > + } > + > + tcet =3D spapr_tce_find_by_liobn(liobn); > + trace_spapr_iommu_ddw_create(buid, addr, 1ULL << page_shift, > + 1ULL << window_shift, > + tcet ? tcet->bus_offset : 0xbaadf00d, l= iobn); > + if (!tcet) { > + goto hw_error_exit; > + } > + > + spapr_tce_table_enable(tcet, page_shift, sphb->dma64_window_addr, This looks like it's assuming you're creating the second 64-bit window. If the guest removed the default window then tried to recreate it, that might not be the case. > + 1ULL << (window_shift - page_shift)); > + if (!tcet->nb_table) { > + goto hw_error_exit; > + } > + > + rtas_st(rets, 0, RTAS_OUT_SUCCESS); > + rtas_st(rets, 1, liobn); > + rtas_st(rets, 2, tcet->bus_offset >> 32); > + rtas_st(rets, 3, tcet->bus_offset & ((uint32_t) -1)); > + > + return; > + > +hw_error_exit: > + rtas_st(rets, 0, RTAS_OUT_HW_ERROR); > + return; > + > +param_error_exit: > + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); > +} > + > +static void rtas_ibm_remove_pe_dma_window(PowerPCCPU *cpu, > + sPAPRMachineState *spapr, > + uint32_t token, uint32_t nargs, > + target_ulong args, > + uint32_t nret, target_ulong re= ts) > +{ > + sPAPRPHBState *sphb; > + sPAPRTCETable *tcet; > + uint32_t liobn; > + > + if ((nargs !=3D 1) || (nret !=3D 1)) { > + goto param_error_exit; > + } > + > + liobn =3D rtas_ld(args, 0); > + tcet =3D spapr_tce_find_by_liobn(liobn); > + if (!tcet) { > + goto param_error_exit; > + } > + > + sphb =3D SPAPR_PCI_HOST_BRIDGE(OBJECT(tcet)->parent); > + if (!sphb || !sphb->ddw_enabled || !tcet->nb_table) { > + goto param_error_exit; > + } > + > + spapr_tce_table_disable(tcet); > + trace_spapr_iommu_ddw_remove(liobn); > + > + rtas_st(rets, 0, RTAS_OUT_SUCCESS); > + return; > + > +param_error_exit: > + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); > +} > + > +static void rtas_ibm_reset_pe_dma_window(PowerPCCPU *cpu, > + sPAPRMachineState *spapr, > + uint32_t token, uint32_t nargs, > + target_ulong args, > + uint32_t nret, target_ulong ret= s) > +{ > + sPAPRPHBState *sphb; > + uint64_t buid; > + uint32_t addr; > + > + if ((nargs !=3D 3) || (nret !=3D 1)) { > + goto param_error_exit; > + } > + > + buid =3D ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2); > + addr =3D rtas_ld(args, 0); > + sphb =3D spapr_pci_find_phb(spapr, buid); > + if (!sphb || !sphb->ddw_enabled) { > + goto param_error_exit; > + } > + > + spapr_phb_dma_reset(sphb); > + trace_spapr_iommu_ddw_reset(buid, addr); > + > + rtas_st(rets, 0, RTAS_OUT_SUCCESS); > + > + return; > + > +param_error_exit: > + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); > +} > + > +static void spapr_rtas_ddw_init(void) > +{ > + spapr_rtas_register(RTAS_IBM_QUERY_PE_DMA_WINDOW, > + "ibm,query-pe-dma-window", > + rtas_ibm_query_pe_dma_window); > + spapr_rtas_register(RTAS_IBM_CREATE_PE_DMA_WINDOW, > + "ibm,create-pe-dma-window", > + rtas_ibm_create_pe_dma_window); > + spapr_rtas_register(RTAS_IBM_REMOVE_PE_DMA_WINDOW, > + "ibm,remove-pe-dma-window", > + rtas_ibm_remove_pe_dma_window); > + spapr_rtas_register(RTAS_IBM_RESET_PE_DMA_WINDOW, > + "ibm,reset-pe-dma-window", > + rtas_ibm_reset_pe_dma_window); > +} > + > +type_init(spapr_rtas_ddw_init) > diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h > index 7848366..36a370e 100644 > --- a/include/hw/pci-host/spapr.h > +++ b/include/hw/pci-host/spapr.h > @@ -32,6 +32,8 @@ > #define SPAPR_PCI_HOST_BRIDGE(obj) \ > OBJECT_CHECK(sPAPRPHBState, (obj), TYPE_SPAPR_PCI_HOST_BRIDGE) > =20 > +#define SPAPR_PCI_DMA_MAX_WINDOWS 2 > + > typedef struct sPAPRPHBState sPAPRPHBState; > =20 > typedef struct spapr_pci_msi { > @@ -56,7 +58,7 @@ struct sPAPRPHBState { > hwaddr mem_win_addr, mem_win_size, io_win_addr, io_win_size; > MemoryRegion memwindow, iowindow, msiwindow; > =20 > - uint32_t dma_liobn; > + uint32_t dma_liobn[SPAPR_PCI_DMA_MAX_WINDOWS]; > hwaddr dma_win_addr, dma_win_size; > AddressSpace iommu_as; > MemoryRegion iommu_root; > @@ -71,6 +73,10 @@ struct sPAPRPHBState { > spapr_pci_msi_mig *msi_devs; > =20 > QLIST_ENTRY(sPAPRPHBState) list; > + > + bool ddw_enabled; > + uint64_t page_size_mask; > + uint64_t dma64_window_addr; > }; > =20 > #define SPAPR_PCI_MAX_INDEX 255 > diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h > index 971df3d..59fad22 100644 > --- a/include/hw/ppc/spapr.h > +++ b/include/hw/ppc/spapr.h > @@ -412,6 +412,16 @@ int spapr_allocate_irq_block(int num, bool lsi, bool= msi); > #define RTAS_OUT_NOT_AUTHORIZED -9002 > #define RTAS_OUT_SYSPARM_PARAM_ERROR -9999 > =20 > +/* DDW pagesize mask values from ibm,query-pe-dma-window */ > +#define RTAS_DDW_PGSIZE_4K 0x01 > +#define RTAS_DDW_PGSIZE_64K 0x02 > +#define RTAS_DDW_PGSIZE_16M 0x04 > +#define RTAS_DDW_PGSIZE_32M 0x08 > +#define RTAS_DDW_PGSIZE_64M 0x10 > +#define RTAS_DDW_PGSIZE_128M 0x20 > +#define RTAS_DDW_PGSIZE_256M 0x40 > +#define RTAS_DDW_PGSIZE_16G 0x80 > + > /* RTAS tokens */ > #define RTAS_TOKEN_BASE 0x2000 > =20 > @@ -453,8 +463,12 @@ int spapr_allocate_irq_block(int num, bool lsi, bool= msi); > #define RTAS_IBM_SET_SLOT_RESET (RTAS_TOKEN_BASE + 0x23) > #define RTAS_IBM_CONFIGURE_PE (RTAS_TOKEN_BASE + 0x24) > #define RTAS_IBM_SLOT_ERROR_DETAIL (RTAS_TOKEN_BASE + 0x25) > +#define RTAS_IBM_QUERY_PE_DMA_WINDOW (RTAS_TOKEN_BASE + 0x26) > +#define RTAS_IBM_CREATE_PE_DMA_WINDOW (RTAS_TOKEN_BASE + 0x27) > +#define RTAS_IBM_REMOVE_PE_DMA_WINDOW (RTAS_TOKEN_BASE + 0x28) > +#define RTAS_IBM_RESET_PE_DMA_WINDOW (RTAS_TOKEN_BASE + 0x29) > =20 > -#define RTAS_TOKEN_MAX (RTAS_TOKEN_BASE + 0x26) > +#define RTAS_TOKEN_MAX (RTAS_TOKEN_BASE + 0x2A) > =20 > /* RTAS ibm,get-system-parameter token values */ > #define RTAS_SYSPARM_SPLPAR_CHARACTERISTICS 20 > diff --git a/trace-events b/trace-events > index ec32c20..dec80e4 100644 > --- a/trace-events > +++ b/trace-events > @@ -1433,6 +1433,10 @@ spapr_iommu_xlate(uint64_t liobn, uint64_t ioba, u= int64_t tce, unsigned perm, un > spapr_iommu_new_table(uint64_t liobn, void *table, int fd) "liobn=3D%"PR= Ix64" table=3D%p fd=3D%d" > spapr_iommu_pre_save(uint64_t liobn, uint32_t nb, uint64_t offs, uint32_= t ps) "liobn=3D%"PRIx64" %"PRIx32" bus_offset=3D%"PRIx64" ps=3D%"PRIu32 > spapr_iommu_post_load(uint64_t liobn, uint32_t pre_nb, uint32_t post_nb,= uint64_t offs, uint32_t ps) "liobn=3D%"PRIx64" %"PRIx32" =3D> %"PRIx32" bu= s_offset=3D%"PRIx64" ps=3D%"PRIu32 > +spapr_iommu_ddw_query(uint64_t buid, uint32_t cfgaddr, unsigned wa, uint= 64_t win_size, uint32_t pgmask) "buid=3D%"PRIx64" addr=3D%"PRIx32", %u wind= ows available, max window size=3D%"PRIx64", mask=3D%"PRIx32 > +spapr_iommu_ddw_create(uint64_t buid, uint32_t cfgaddr, uint64_t pg_size= , uint64_t req_size, uint64_t start, uint32_t liobn) "buid=3D%"PRIx64" addr= =3D%"PRIx32", page size=3D0x%"PRIx64", requested=3D0x%"PRIx64", start addr= =3D%"PRIx64", liobn=3D%"PRIx32 > +spapr_iommu_ddw_remove(uint32_t liobn) "liobn=3D%"PRIx32 > +spapr_iommu_ddw_reset(uint64_t buid, uint32_t cfgaddr) "buid=3D%"PRIx64"= addr=3D%"PRIx32 > =20 > # hw/ppc/ppc.c > ppc_tb_adjust(uint64_t offs1, uint64_t offs2, int64_t diff, int64_t seco= nds) "adjusted from 0x%"PRIx64" to 0x%"PRIx64", diff %"PRId64" (%"PRId64"s)" --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --9JSHP372f+2dzJ8X Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJXVRDmAAoJEGw4ysog2bOSLqcP/306L/35GeMFr+etI5CGX28T AW3n8nyZgcPDLm7+dK422ODTL+8wENjDWIqZAODKG+/pfwYRkG15FFz4riTu1YuS tmHzWnzUzEBAsd+JBRhzhNSWMi6aX8oZKjXOvj/lMjF2RxPHhxBVkL1TXZMhUBY4 bL3rH9uoL7j8Hd1JGJ/ywvMVgzdKeip0wHwfDpVq5MaQQWgUsgsapRVf4RH3ntGy vZy6fik2p1mPio6zYItTUrljdEMVbUy2w8Udhf9ZAQ5OhlU7JsXtMHTTGWkM0ag6 RnnvPzcuTEF3nC6YZy/nssgIFHCaeZN6jMbvmlH+g0GP/zoVypOvuHAyS7NENSJv p6bSbvwEX/XdEoK0TQqG3bxBMcINkgCd1ZfUXghy6VQnv3EzR/4vAoS9E2JO5qO5 GpSw2dsOAQY0H9pVwGBX4DGydGBGupbBZood17qpPdebewCGq6+9i/urhGLvLdG7 Kx20sYSGh06SfGPOHSeDDCHLzWgAEnU+hOPojanKgjj5WNLtwcWmAHSSMzsCs8N2 ylJN+wmeVuXt+abTYw/fUXj8eeMK6IvZ1ia99lmsfAzHF7KmxEELNM3WqomQizaD qqROmc6D+jbCncch1IQ1YtJlRacYzjum5sevnNPbc1opT9jNRBnlPu2z4POZxZQd 6BnhtrEvJLDZ3jlp+0rp =sLCw -----END PGP SIGNATURE----- --9JSHP372f+2dzJ8X--