From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55394) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZCL1X-0002BG-Dv for qemu-devel@nongnu.org; Tue, 07 Jul 2015 01:02:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZCL1U-00086L-6m for qemu-devel@nongnu.org; Tue, 07 Jul 2015 01:02:27 -0400 Date: Tue, 7 Jul 2015 14:58:07 +1000 From: David Gibson Message-ID: <20150707045807.GJ17857@voom.redhat.com> References: <1436148670-6592-1-git-send-email-aik@ozlabs.ru> <1436148670-6592-15-git-send-email-aik@ozlabs.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="HSQ3hISbU3Um6hch" Content-Disposition: inline In-Reply-To: <1436148670-6592-15-git-send-email-aik@ozlabs.ru> Subject: Re: [Qemu-devel] [PATCH qemu v10 14/14] spapr_pci/spapr_pci_vfio: Support Dynamic DMA Windows (DDW) List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexey Kardashevskiy Cc: Michael Roth , Alex Williamson , qemu-ppc@nongnu.org, qemu-devel@nongnu.org, Gavin Shan --HSQ3hISbU3Um6hch Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jul 06, 2015 at 12:11:10PM +1000, Alexey Kardashevskiy wrote: > This adds support for Dynamic DMA Windows (DDW) option defined by > the SPAPR specification which allows to have additional DMA window(s) >=20 > This implements DDW for emulated and VFIO devices. As all TCE root regions > are mapped at 0 and 64bit long (and actual tables are child regions), > this replaces memory_region_add_subregion() with _overlap() to make > QEMU memory API happy. >=20 > This reserves RTAS token numbers for DDW calls. >=20 > This implements helpers to interact with VFIO kernel interface. >=20 > This changes the TCE table migration descriptor to support dynamic > tables as from now on, PHB will create as many stub TCE table objects > as PHB can possibly support but not all of them might be initialized at > the time of migration because DDW might or might not be requested by > the guest. >=20 > The "ddw" property is enabled by default on a PHB but for compatibility > the pseries-2.3 machine and older disable it. >=20 > This implements DDW for VFIO. The host kernel support is required. > This adds a "levels" property to PHB to control the number of levels > in the actual TCE table allocated by the host kernel, 0 is the default > value to tell QEMU to calculate the correct value. Current hardware > supports up to 5 levels. >=20 > The existing linux guests try creating one additional huge DMA window > with 64K or 16MB pages and map the entire guest RAM to. If succeeded, > the guest switches to dma_direct_ops and never calls TCE hypercalls > (H_PUT_TCE,...) again. This enables VFIO devices to use the entire RAM > and not waste time on map/unmap later. This adds a "dma64_win_addr" > property which is a bus address for the 64bit window and by default > set to 0x800.0000.0000.0000 as this is what the modern POWER8 hardware > uses and this allows having emulated and VFIO devices on the same bus. >=20 > This adds 4 RTAS handlers: > * ibm,query-pe-dma-window > * ibm,create-pe-dma-window > * ibm,remove-pe-dma-window > * ibm,reset-pe-dma-window > These are registered from type_init() callback. >=20 > These RTAS handlers are implemented in a separate file to avoid polluting > spapr_iommu.c with PCI. > diff --git a/trace-events b/trace-events > index b300e94..a1234dd 100644 > --- a/trace-events > +++ b/trace-events > @@ -1302,6 +1302,8 @@ spapr_pci_lsi_set(const char *busname, int pin, uin= t32_t irq) "%s PIN%d IRQ %u" > spapr_pci_msi_retry(unsigned config_addr, unsigned req_num, unsigned max= _irqs) "Guest device at %x asked %u, have only %u" > spapr_pci_dma_update(uint64_t liobn, long ret) "liobn=3D%"PRIx64" ret=3D= %ld" > spapr_pci_dma_realloc_update(uint64_t liobn, long ret) "liobn=3D%"PRIx64= " tcet=3D%ld" > +spapr_pci_vfio_init_window(int ps, uint64_t ws, uint64_t off) "pageshift= =3D0x%x winsize=3D0x%"PRIx64" offset=3D0x%"PRIx64 > +spapr_pci_vfio_remove_window(uint64_t off) "offset=3D%"PRIx64 > =20 > # hw/pci/pci.c > pci_update_mappings_del(void *d, uint32_t bus, uint32_t func, uint32_t s= lot, int bar, uint64_t addr, uint64_t size) "d=3D%p %02x:%02x.%x %d,%#"PRIx= 64"+%#"PRIx64 > @@ -1365,6 +1367,10 @@ spapr_iommu_pci_indirect(uint64_t liobn, uint64_t = ioba, uint64_t tce, uint64_t i > spapr_iommu_pci_stuff(uint64_t liobn, uint64_t ioba, uint64_t tce_value,= uint64_t npages, uint64_t ret) "liobn=3D%"PRIx64" ioba=3D0x%"PRIx64" tceva= lue=3D0x%"PRIx64" npages=3D%"PRId64" ret=3D%"PRId64 > spapr_iommu_xlate(uint64_t liobn, uint64_t ioba, uint64_t tce, unsigned = perm, unsigned pgsize) "liobn=3D%"PRIx64" 0x%"PRIx64" -> 0x%"PRIx64" perm= =3D%u mask=3D%x" > spapr_iommu_alloc_table(uint64_t liobn, void *table, int fd) "liobn=3D%"= PRIx64" table=3D%p fd=3D%d" > +spapr_iommu_ddw_query(uint64_t buid, uint32_t cfgaddr, unsigned wa, uint= 64_t win_size, uint32_t pgmask) "buid=3D%"PRIx64" addr=3D%"PRIx32", %u wind= ows available, max window size=3D%"PRIx64", mask=3D%"PRIx32 Turns out the dtrace trace backend barfs on the "long long" here :( $ ./configure --target-list=3Dppc64-softmmu --enable-trace-backends=3Ddtrace [...] $ make GEN config-host.h GEN trace/generated-tracers.h GEN trace/generated-tracers-dtrace.dtrace GEN trace/generated-tracers-dtrace.h Warning: /bin/dtrace:trace/generated-tracers-dtrace.dtrace:2212: syntax err= or near: probe spapr_iommu_ddw_query Warning: Proceeding as if --no-pyparsing was given. GEN trace/generated-tcg-tracers.h GEN trace/generated-helpers-wrappers.h GEN trace/generated-helpers.h CC trace/generated-events.o GEN trace/generated-tracers-dtrace.o Warning: /bin/dtrace:trace/generated-tracers-dtrace.dtrace:2212: syntax err= or near: probe spapr_iommu_ddw_query Warning: Proceeding as if --no-pyparsing was given. --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --HSQ3hISbU3Um6hch Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJVm1xfAAoJEGw4ysog2bOSZpIQAK859zxBYmvgnRVT8IdBvj+P 3poe09+vHLuTzJVofLdkC8lKFwwXF0l/WDfOrkK32Q4CN4KIl6ZbABB/NF56BUVe GBEZ5Fbw88m+hkeT6QDvxbfZRw4dr8TgXQ73KYLvfSUdBpbxG8lHvPvGXnSJVLw+ Mw45sX432Frh+HqxbU1EnXLcOIEkJRcl1HfyNGfyEMMGAOIzrkruv4P8ITa1smO8 A0rSefXz6oUkFmgaon/RtjWsw0ok6eR5KStZvUJApFIW9dFfMQqB/pby93XqWUiG X0GRU+M1ttT3ePO7g2D5mNluVSwwzmEuA4MyU3OLRG4f+5knfH6wmueWlAXWaaU5 /ymb07Rtf+Xf/3weVg5lz8Kri9OE2aSj/2Rb4EOe0k17pU65hV+iDpuOoTeCb7BH xwm2XQSNpuJeLtwvEUzggbLWLiswJ3y0LOBX1n2xGN99BZvnf3wTonrQ7Z59WXfQ 6gY5kpcz3Y0WUc3fI+9uR+9pQi01bccnNeTSPQ/nAcPZm/lub+3MdeST34PxLsf6 DUAXPGS7rgO+T5ffn7ziMs/d5c0xnQPCzcOfCJEpa4Dn6Z2BoIWMmi4Qg/JPGMPd vrvieHDAd3HQBn0IgUFqvMKUsnkxp/zosGRLHhuPKbWDI+cxHu5pbTTPmu9nAuG5 09QSEbNlf1syqVBfPm2W =OncH -----END PGP SIGNATURE----- --HSQ3hISbU3Um6hch--