From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40053) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YJDhB-0000RS-3z for qemu-devel@nongnu.org; Wed, 04 Feb 2015 23:05:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YJDh7-0001eI-7v for qemu-devel@nongnu.org; Wed, 04 Feb 2015 23:05:37 -0500 Date: Thu, 5 Feb 2015 15:05:54 +1100 From: David Gibson Message-ID: <20150205040554.GL25675@voom.fritz.box> References: <1422523650-2888-1-git-send-email-aik@ozlabs.ru> <1422523650-2888-13-git-send-email-aik@ozlabs.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="qyHYMwAXsHLOQihY" Content-Disposition: inline In-Reply-To: <1422523650-2888-13-git-send-email-aik@ozlabs.ru> Subject: Re: [Qemu-devel] [PATCH v4 12/18] spapr_rtas: Add Dynamic DMA windows (DDW) RTAS handlers List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexey Kardashevskiy Cc: Alex Williamson , qemu-ppc@nongnu.org, qemu-devel@nongnu.org, Alexander Graf --qyHYMwAXsHLOQihY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jan 29, 2015 at 08:27:24PM +1100, Alexey Kardashevskiy wrote: > This adds support for Dynamic DMA Windows (DDW) option defined by > the SPAPR specification which allows to have additional DMA window(s) > which can support page sizes other than 4K. >=20 > The existing implementation of DDW in the guest tries to create one huge > DMA window with 64K or 16MB pages and map the entire guest RAM to. If it > succeeds, the guest switches to dma_direct_ops and never calls > TCE hypercalls (H_PUT_TCE,...) again. This enables VFIO devices to use > the entire RAM and not waste time on map/unmap later. >=20 > This adds 4 RTAS handlers: > * ibm,query-pe-dma-window > * ibm,create-pe-dma-window > * ibm,remove-pe-dma-window > * ibm,reset-pe-dma-window > These are registered from type_init() callback. >=20 > These RTAS handlers are implemented in a separate file to avoid polluting > spapr_iommu.c with PHB. >=20 > Signed-off-by: Alexey Kardashevskiy > --- > Changes: > v3: > * added ibm,reset-pe-dma-window >=20 > v2: > * double loop squashed to spapr_iommu_fixmask() helper > * added @ddw_num counter to PHB, it is used to generate LIOBN for new > window; it is reset on ddw-reset event > * added ULL to constants used in shift operations > * rtas_ibm_reset_pe_dma_window() and rtas_ibm_remove_pe_dma_window() > do not remove windows anymore, the PHB callback has to as it will reuse > the same code in case of guest reboot as well > --- > hw/ppc/Makefile.objs | 3 + > hw/ppc/spapr_rtas_ddw.c | 297 ++++++++++++++++++++++++++++++++++++++++++= ++++++ > trace-events | 4 + > 3 files changed, 304 insertions(+) > create mode 100644 hw/ppc/spapr_rtas_ddw.c >=20 > diff --git a/hw/ppc/Makefile.objs b/hw/ppc/Makefile.objs > index 19d9920..d7fe4fb 100644 > --- a/hw/ppc/Makefile.objs > +++ b/hw/ppc/Makefile.objs > @@ -7,6 +7,9 @@ obj-$(CONFIG_PSERIES) +=3D spapr_pci.o > ifeq ($(CONFIG_PCI)$(CONFIG_PSERIES)$(CONFIG_LINUX), yyy) > obj-y +=3D spapr_pci_vfio.o > endif > +ifeq ($(CONFIG_PCI)$(CONFIG_PSERIES), yy) > +obj-y +=3D spapr_rtas_ddw.o > +endif > # PowerPC 4xx boards > obj-y +=3D ppc405_boards.o ppc4xx_devs.o ppc405_uc.o ppc440_bamboo.o > obj-y +=3D ppc4xx_pci.o > diff --git a/hw/ppc/spapr_rtas_ddw.c b/hw/ppc/spapr_rtas_ddw.c > new file mode 100644 > index 0000000..af70601 > --- /dev/null > +++ b/hw/ppc/spapr_rtas_ddw.c > @@ -0,0 +1,297 @@ > +/* > + * QEMU sPAPR Dynamic DMA windows support > + * > + * Copyright (c) 2014 Alexey Kardashevskiy, IBM Corporation. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; either version 2 of the License, > + * or (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program; if not, see . > + */ > + > +#include "hw/ppc/spapr.h" > +#include "hw/pci-host/spapr.h" > +#include "trace.h" > + > +static uint32_t spapr_iommu_fixmask(struct ppc_one_seg_page_size *sps, > + uint32_t query_mask) This function could do with a comment describing what it's for. Every version of this series I've looked at I've forgotten and had to figure it out again. > +{ > + int i, j; > + uint32_t mask =3D 0; > + const struct { int shift; uint32_t mask; } masks[] =3D { > + { 12, DDW_PGSIZE_4K }, > + { 16, DDW_PGSIZE_64K }, > + { 24, DDW_PGSIZE_16M }, > + { 25, DDW_PGSIZE_32M }, > + { 26, DDW_PGSIZE_64M }, > + { 27, DDW_PGSIZE_128M }, > + { 28, DDW_PGSIZE_256M }, > + { 34, DDW_PGSIZE_16G }, > + }; > + > + for (i =3D 0; i < PPC_PAGE_SIZES_MAX_SZ; i++) { > + for (j =3D 0; j < ARRAY_SIZE(masks); ++j) { > + if ((sps[i].page_shift =3D=3D masks[j].shift) && > + (query_mask & masks[j].mask)) { > + mask |=3D masks[j].mask; > + } > + } > + } > + > + return mask; > +} > + > +static void rtas_ibm_query_pe_dma_window(PowerPCCPU *cpu, > + sPAPREnvironment *spapr, > + uint32_t token, uint32_t nargs, > + target_ulong args, > + uint32_t nret, target_ulong ret= s) > +{ > + CPUPPCState *env =3D &cpu->env; > + sPAPRPHBState *sphb; > + sPAPRPHBClass *spc; > + uint64_t buid; > + uint32_t avail, addr, pgmask =3D 0; > + uint32_t windows_supported =3D 0, page_size_mask =3D 0, dma32_window= _size =3D 0; > + uint64_t dma64_window_size =3D 0; > + unsigned current; > + long ret; > + > + if ((nargs !=3D 3) || (nret !=3D 5)) { > + goto param_error_exit; > + } > + > + buid =3D ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2); > + addr =3D rtas_ld(args, 0); > + sphb =3D spapr_pci_find_phb(spapr, buid); > + if (!sphb || !sphb->ddw_enabled) { > + goto param_error_exit; > + } > + > + spc =3D SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb); > + if (!spc->ddw_query) { > + goto hw_error_exit; > + } > + > + ret =3D spc->ddw_query(sphb, &windows_supported, &page_size_mask, > + &dma32_window_size, &dma64_window_size); > + trace_spapr_iommu_ddw_query(buid, addr, windows_supported, > + page_size_mask, pgmask, ret); > + if (ret) { > + goto hw_error_exit; > + } > + > + current =3D spapr_phb_get_win_num(sphb); > + avail =3D (windows_supported > current) ? (windows_supported - curre= nt) : 0; > + > + /* Work out supported page masks */ > + pgmask =3D spapr_iommu_fixmask(env->sps.sps, page_size_mask); > + > + rtas_st(rets, 0, RTAS_OUT_SUCCESS); > + rtas_st(rets, 1, avail); > + > + /* > + * This is "Largest contiguous block of TCEs allocated specifically > + * for (that is, are reserved for) this PE". > + * Return the maximum number as all RAM was in 4K pages. > + */ > + rtas_st(rets, 2, dma64_window_size >> SPAPR_TCE_PAGE_SHIFT); > + rtas_st(rets, 3, pgmask); > + rtas_st(rets, 4, 0); /* DMA migration mask, not supported */ > + return; > + > +hw_error_exit: > + rtas_st(rets, 0, RTAS_OUT_HW_ERROR); > + return; > + > +param_error_exit: > + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); > +} > + > +static void rtas_ibm_create_pe_dma_window(PowerPCCPU *cpu, > + sPAPREnvironment *spapr, > + uint32_t token, uint32_t nargs, > + target_ulong args, > + uint32_t nret, target_ulong re= ts) > +{ > + sPAPRPHBState *sphb; > + sPAPRPHBClass *spc; > + sPAPRTCETable *tcet =3D NULL; > + uint32_t addr, page_shift, window_shift, liobn; > + uint64_t buid; > + long ret; > + uint32_t windows_supported =3D 0, page_size_mask =3D 0, dma32_window= _size =3D 0; > + uint64_t dma64_window_size =3D 0; > + > + if ((nargs !=3D 5) || (nret !=3D 4)) { > + goto param_error_exit; > + } > + > + buid =3D ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2); > + addr =3D rtas_ld(args, 0); > + sphb =3D spapr_pci_find_phb(spapr, buid); > + if (!sphb || !sphb->ddw_enabled) { > + goto param_error_exit; > + } > + > + spc =3D SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb); > + if (!spc->ddw_create || !spc->ddw_query) { > + goto hw_error_exit; > + } > + > + ret =3D spc->ddw_query(sphb, &windows_supported, &page_size_mask, > + &dma32_window_size, &dma64_window_size); > + if (ret || (spapr_phb_get_win_num(sphb) >=3D windows_supported)) { > + goto hw_error_exit; Is RTAS_OUT_HW_ERROR really the right thing for the case of having allocated all the available windows? > + } > + > + page_shift =3D rtas_ld(args, 3); > + window_shift =3D rtas_ld(args, 4); > + /* Default 32bit window#0 is always there so +1 */ > + liobn =3D SPAPR_PCI_LIOBN(sphb->index, spapr_phb_get_win_num(sphb)); > + > + ret =3D spc->ddw_create(sphb, liobn, page_shift, window_shift, &tcet= ); > + trace_spapr_iommu_ddw_create(buid, addr, 1ULL << page_shift, > + 1ULL << window_shift, > + tcet ? tcet->bus_offset : 0xbaadf00d, > + liobn, ret); > + if (ret || !tcet) { > + goto hw_error_exit; > + } > + > + rtas_st(rets, 0, RTAS_OUT_SUCCESS); > + rtas_st(rets, 1, liobn); > + rtas_st(rets, 2, tcet->bus_offset >> 32); > + rtas_st(rets, 3, tcet->bus_offset & ((uint32_t) -1)); > + > + object_unref(OBJECT(tcet)); > + return; > + > +hw_error_exit: > + rtas_st(rets, 0, RTAS_OUT_HW_ERROR); > + return; > + > +param_error_exit: > + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); > +} > + > +static void rtas_ibm_remove_pe_dma_window(PowerPCCPU *cpu, > + sPAPREnvironment *spapr, > + uint32_t token, uint32_t nargs, > + target_ulong args, > + uint32_t nret, target_ulong re= ts) > +{ > + sPAPRPHBState *sphb; > + sPAPRPHBClass *spc; > + sPAPRTCETable *tcet; > + uint32_t liobn; > + long ret; > + > + if ((nargs !=3D 1) || (nret !=3D 1)) { > + goto param_error_exit; > + } > + > + liobn =3D rtas_ld(args, 0); > + tcet =3D spapr_tce_find_by_liobn(liobn); > + if (!tcet) { > + goto param_error_exit; > + } > + > + sphb =3D SPAPR_PCI_HOST_BRIDGE(OBJECT(tcet)->parent); > + if (!sphb || !sphb->ddw_enabled) { > + goto param_error_exit; > + } > + > + spc =3D SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb); > + if (!spc->ddw_remove) { > + goto hw_error_exit; > + } > + > + ret =3D spc->ddw_remove(sphb, tcet); > + trace_spapr_iommu_ddw_remove(liobn, ret); > + if (ret) { > + goto hw_error_exit; > + } > + > + rtas_st(rets, 0, RTAS_OUT_SUCCESS); > + return; > + > +hw_error_exit: > + rtas_st(rets, 0, RTAS_OUT_HW_ERROR); > + return; > + > +param_error_exit: > + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); > +} > + > +static void rtas_ibm_reset_pe_dma_window(PowerPCCPU *cpu, > + sPAPREnvironment *spapr, > + uint32_t token, uint32_t nargs, > + target_ulong args, > + uint32_t nret, target_ulong ret= s) > +{ > + sPAPRPHBState *sphb; > + sPAPRPHBClass *spc; > + uint64_t buid; > + uint32_t addr; > + long ret; > + > + if ((nargs !=3D 3) || (nret !=3D 1)) { > + goto param_error_exit; > + } > + > + buid =3D ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2); > + addr =3D rtas_ld(args, 0); > + sphb =3D spapr_pci_find_phb(spapr, buid); > + if (!sphb || !sphb->ddw_enabled) { > + goto param_error_exit; > + } > + > + spc =3D SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb); > + if (!spc->ddw_reset) { > + goto hw_error_exit; > + } > + > + ret =3D spc->ddw_reset(sphb); > + trace_spapr_iommu_ddw_reset(buid, addr, ret); > + if (ret) { > + goto hw_error_exit; > + } > + > + rtas_st(rets, 0, RTAS_OUT_SUCCESS); > + > + return; > + > +hw_error_exit: > + rtas_st(rets, 0, RTAS_OUT_HW_ERROR); > + return; > + > +param_error_exit: > + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); > +} > + > +static void spapr_rtas_ddw_init(void) > +{ > + spapr_rtas_register(RTAS_IBM_QUERY_PE_DMA_WINDOW, > + "ibm,query-pe-dma-window", > + rtas_ibm_query_pe_dma_window); > + spapr_rtas_register(RTAS_IBM_CREATE_PE_DMA_WINDOW, > + "ibm,create-pe-dma-window", > + rtas_ibm_create_pe_dma_window); > + spapr_rtas_register(RTAS_IBM_REMOVE_PE_DMA_WINDOW, > + "ibm,remove-pe-dma-window", > + rtas_ibm_remove_pe_dma_window); > + spapr_rtas_register(RTAS_IBM_RESET_PE_DMA_WINDOW, > + "ibm,reset-pe-dma-window", > + rtas_ibm_reset_pe_dma_window); > +} > + > +type_init(spapr_rtas_ddw_init) > diff --git a/trace-events b/trace-events > index 04f5df2..9af53d9 100644 > --- a/trace-events > +++ b/trace-events > @@ -1285,6 +1285,10 @@ spapr_iommu_indirect(uint64_t liobn, uint64_t ioba= , uint64_t tce, uint64_t iobaN > spapr_iommu_stuff(uint64_t liobn, uint64_t ioba, uint64_t tce_value, uin= t64_t npages, uint64_t ret) "liobn=3D%"PRIx64" ioba=3D0x%"PRIx64" tcevalue= =3D0x%"PRIx64" npages=3D%"PRId64" ret=3D%"PRId64 > spapr_iommu_xlate(uint64_t liobn, uint64_t ioba, uint64_t tce, unsigned = perm, unsigned pgsize) "liobn=3D%"PRIx64" 0x%"PRIx64" -> 0x%"PRIx64" perm= =3D%u mask=3D%x" > spapr_iommu_new_table(uint64_t liobn, void *tcet, void *table, int fd) "= liobn=3D%"PRIx64" tcet=3D%p table=3D%p fd=3D%d" > +spapr_iommu_ddw_query(uint64_t buid, uint32_t cfgaddr, uint32_t wa, uint= 32_t pgz, uint32_t pgz_fixed, long ret) "buid=3D%"PRIx64" addr=3D%"PRIx32",= %u windows available, sizes %"PRIx32", fixed %"PRIx32", ret =3D %ld" > +spapr_iommu_ddw_create(uint64_t buid, uint32_t cfgaddr, unsigned long lo= ng pg_size, unsigned long long req_size, uint64_t start, uint32_t liobn, lo= ng ret) "buid=3D%"PRIx64" addr=3D%"PRIx32", page size=3D0x%llx, requested= =3D0x%llx, start addr=3D%"PRIx64", liobn=3D%"PRIx32", ret =3D %ld" > +spapr_iommu_ddw_remove(uint32_t liobn, long ret) "liobn=3D%"PRIx32", ret= =3D %ld" > +spapr_iommu_ddw_reset(uint64_t buid, uint32_t cfgaddr, long ret) "buid= =3D%"PRIx64" addr=3D%"PRIx32", ret =3D %ld" > =20 > # hw/ppc/ppc.c > ppc_tb_adjust(uint64_t offs1, uint64_t offs2, int64_t diff, int64_t seco= nds) "adjusted from 0x%"PRIx64" to 0x%"PRIx64", diff %"PRId64" (%"PRId64"s)" --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --qyHYMwAXsHLOQihY Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJU0uwiAAoJEGw4ysog2bOS/IAQAJnyiIDqP3t1ammN7gvp8myI 14w1GotPWeSimrjk4lulOQTO8Xtx5+PFAt1prA7nc89nkCMQxBwEPdXgtQZDRW7W K/q6I7YcCHtcr+XUImk5AYUEqLvUE0Ih0yOW2cTFEGIAf2YW9gP+Trz0f/KGsJ0K DWWU+D5IoeFR+tCP4ABLNEOxLI6g5le/vhhn9IJKwxr5xxTXEeAoXlWyfzrsZ550 DPZ+W7mC1oGYNx8kusAQK+yh688qhnrS1Ysfy9V7NW3M5LOaF9uMERTh3kmHG2vI F8gQVsVBxy+jJl38yoscPgUVU8CgGDcikZAuMmCAisgHAdWp1LcpYFvl6rXoMQhh rWg5kcNfbKFIY5j4wiETpiatFbLOOBZN9rI7wH4NkND/m4Vybe5GtZzz0VwuIUm9 7KxMTBaiCrVL9JPY1xJNY8D/4JvN6mBnrapRjnYtAeU/Y+n37tnPDg/G8f4QJF6a XHYNubjRzA0k9hQBg0jSCna75UVf0SKO0MukTA8IcAQhrpOlqpkTiMtGGs0u/vz0 XET14a472r4cKF287yQkxphlB2XVPD+mCBeZjpleTV2K4gYUmFnFZmWcpFuApZl7 NFhX9LG1vwN7UEafTb/uNGXaknQu2AyDjQrY31P/qcZtDWjlQJfJjMTf/Rsv6W9A lU8qrVdDsTuuKNxHCMrq =7ObC -----END PGP SIGNATURE----- --qyHYMwAXsHLOQihY--