From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58231) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YkpJq-00025l-Tq for qemu-devel@nongnu.org; Wed, 22 Apr 2015 03:43:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YkpJm-00074H-C0 for qemu-devel@nongnu.org; Wed, 22 Apr 2015 03:43:38 -0400 Date: Wed, 22 Apr 2015 15:53:06 +1000 From: David Gibson Message-ID: <20150422055306.GM31815@voom.redhat.com> References: <1428679484-15451-1-git-send-email-aik@ozlabs.ru> <1428679484-15451-8-git-send-email-aik@ozlabs.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="fNagykWcDoSVAmSd" Content-Disposition: inline In-Reply-To: <1428679484-15451-8-git-send-email-aik@ozlabs.ru> Subject: Re: [Qemu-devel] [PATCH qemu v6 07/15] vfio: spapr: Add SPAPR IOMMU v2 support (DMA memory preregistering) List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexey Kardashevskiy Cc: Alexander Graf , Michael Roth , qemu-devel@nongnu.org, Alex Williamson , qemu-ppc@nongnu.org, Gavin Shan --fNagykWcDoSVAmSd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Apr 11, 2015 at 01:24:36AM +1000, Alexey Kardashevskiy wrote: > This makes use of the new "memory registering" feature. The idea is > to provide the userspace ability to notify the host kernel about pages > which are going to be used for DMA. Having this information, the host > kernel can pin them all once per user process, do locked pages > accounting (once) and not spent time on doing that in real time with > possible failures which cannot be handled nicely in some cases. >=20 > This adds a guest RAM memory listener which notifies a VFIO container > about memory which needs to be pinned/unpinned. VFIO MMIO regions > (i.e. "skip dump" regions) are skipped. >=20 > The feature is only enabled for SPAPR IOMMU v2. The host kernel changes > are required. Since v2 does not need/support VFIO_IOMMU_ENABLE, this does > not call it when v2 is detected and enabled. >=20 > This does not change the guest visible interface. >=20 > Signed-off-by: Alexey Kardashevskiy Reviewed-by: David Gibson Albeit with some nits described below. > --- > Changes: > v6: > * fixed commit log (s/guest/userspace/), added note about no guest visible > change > * fixed error checking if ram registration failed > * added alignment check for section->offset_within_region >=20 > v5: > * simplified the patch > * added trace points > * added round_up() for the size > * SPAPR IOMMU v2 used > --- > hw/vfio/common.c | 26 +++++++++---- > hw/vfio/spapr.c | 88 +++++++++++++++++++++++++++++++++++++= +++++- > include/hw/vfio/vfio-common.h | 5 ++- > trace-events | 1 + > 4 files changed, 109 insertions(+), 11 deletions(-) >=20 > diff --git a/hw/vfio/common.c b/hw/vfio/common.c > index 369e564..9e3e0b0 100644 > --- a/hw/vfio/common.c > +++ b/hw/vfio/common.c > @@ -577,14 +577,18 @@ static int vfio_connect_container(VFIOGroup *group,= AddressSpace *as) > =20 > container->iommu_data.type1.initialized =3D true; > =20 > - } else if (ioctl(fd, VFIO_CHECK_EXTENSION, VFIO_SPAPR_TCE_IOMMU)) { > + } else if (ioctl(fd, VFIO_CHECK_EXTENSION, VFIO_SPAPR_TCE_IOMMU) || > + ioctl(fd, VFIO_CHECK_EXTENSION, VFIO_SPAPR_TCE_v2_IOMMU))= { > + bool v2 =3D !!ioctl(fd, VFIO_CHECK_EXTENSION, VFIO_SPAPR_TCE_v2_= IOMMU); > + > ret =3D ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &fd); > if (ret) { > error_report("vfio: failed to set group container: %m"); > ret =3D -errno; > goto free_container_exit; > } > - ret =3D ioctl(fd, VFIO_SET_IOMMU, VFIO_SPAPR_TCE_IOMMU); > + ret =3D ioctl(fd, VFIO_SET_IOMMU, > + v2 ? VFIO_SPAPR_TCE_v2_IOMMU : VFIO_SPAPR_TCE_IOMMU); > if (ret) { > error_report("vfio: failed to set iommu for container: %m"); > ret =3D -errno; > @@ -596,14 +600,20 @@ static int vfio_connect_container(VFIOGroup *group,= AddressSpace *as) > * when container fd is closed so we do not call it explicitly > * in this file. > */ > - ret =3D ioctl(fd, VFIO_IOMMU_ENABLE); > - if (ret) { > - error_report("vfio: failed to enable container: %m"); > - ret =3D -errno; > - goto free_container_exit; > + if (!v2) { > + ret =3D ioctl(fd, VFIO_IOMMU_ENABLE); > + if (ret) { > + error_report("vfio: failed to enable container: %m"); > + ret =3D -errno; > + goto free_container_exit; > + } > } > =20 > - spapr_memory_listener_register(container); > + ret =3D spapr_memory_listener_register(container, v2 ? 2 : 1); > + if (ret) { > + error_report("vfio: RAM memory listener initialization faile= d for container"); > + goto listener_release_exit; > + } > =20 > } else { > error_report("vfio: No available IOMMU models"); > diff --git a/hw/vfio/spapr.c b/hw/vfio/spapr.c > index 5f79194..31353f1 100644 > --- a/hw/vfio/spapr.c > +++ b/hw/vfio/spapr.c > @@ -17,6 +17,9 @@ > * along with this program; if not, see . > */ > =20 > +#include > +#include > + > #include "hw/vfio/vfio-common.h" > #include "qemu/error-report.h" > #include "trace.h" > @@ -211,16 +214,97 @@ static const MemoryListener vfio_spapr_memory_liste= ner =3D { > .region_del =3D vfio_spapr_listener_region_del, > }; > =20 > +static void vfio_ram_do_region(VFIOContainer *container, > + MemoryRegionSection *section, unsigned lon= g req) > +{ > + int ret; > + struct vfio_iommu_spapr_register_memory reg =3D { .argsz =3D sizeof(= reg) }; > + > + if (!memory_region_is_ram(section->mr) || > + memory_region_is_skip_dump(section->mr)) { > + return; > + } > + > + if (unlikely((section->offset_within_region & (getpagesize() - 1))))= { > + error_report("%s received unaligned region", __func__); > + return; > + } > + > + reg.vaddr =3D (__u64) memory_region_get_ram_ptr(section->mr) + > + section->offset_within_region; > + reg.size =3D ROUND_UP(int128_get64(section->size), TARGET_PAGE_SIZE); > + > + ret =3D ioctl(container->fd, req, ®); > + trace_vfio_ram_register(_IOC_NR(req) - VFIO_BASE, reg.vaddr, reg.siz= e, > + ret ? -errno : 0); > + if (!ret) { > + return; > + } > + > + /* > + * On the initfn path, store the first error in the container so we > + * can gracefully fail. Runtime, there's not much we can do other > + * than throw a hardware error. > + */ > + if (!container->iommu_data.spapr.ram_reg_initialized) { > + if (!container->iommu_data.spapr.ram_reg_error) { > + container->iommu_data.spapr.ram_reg_error =3D -errno; > + } > + } else { > + hw_error("vfio: RAM registering failed, unable to continue"); > + } > +} > + > +static void vfio_spapr_ram_listener_region_add(MemoryListener *listener, > + MemoryRegionSection *sect= ion) > +{ > + VFIOContainer *container =3D container_of(listener, VFIOContainer, > + iommu_data.spapr.ramlistener= ); > + memory_region_ref(section->mr); > + vfio_ram_do_region(container, section, VFIO_IOMMU_SPAPR_REGISTER_MEM= ORY); > +} > + > +static void vfio_spapr_ram_listener_region_del(MemoryListener *listener, > + MemoryRegionSection *sect= ion) > +{ > + VFIOContainer *container =3D container_of(listener, VFIOContainer, > + iommu_data.spapr.ramlistener= ); > + memory_region_unref(section->mr); You're probably safe because of the QBL, but it would be more obviously correct if you did the ioctl unregister before unreferencing the MR. > + vfio_ram_do_region(container, section, VFIO_IOMMU_SPAPR_UNREGISTER_M= EMORY); > +} > + > +static const MemoryListener vfio_spapr_ram_memory_listener =3D { > + .region_add =3D vfio_spapr_ram_listener_region_add, > + .region_del =3D vfio_spapr_ram_listener_region_del, > +}; > + > static void vfio_spapr_listener_release(VFIOContainer *container) > { > memory_listener_unregister(&container->iommu_data.spapr.listener); > } > =20 > -void spapr_memory_listener_register(VFIOContainer *container) > +static void vfio_spapr_listener_release_v2(VFIOContainer *container) > +{ > + memory_listener_unregister(&container->iommu_data.spapr.listener); > + vfio_spapr_listener_release(container); > +} > + > +int spapr_memory_listener_register(VFIOContainer *container, int ver) > { > container->iommu_data.spapr.listener =3D vfio_spapr_memory_listener; > container->iommu_data.release =3D vfio_spapr_listener_release; > - > memory_listener_register(&container->iommu_data.spapr.listener, > container->space->as); > + if (ver < 2) { > + return 0; > + } I wonder if it would make sense to store the IOMMU type value into the VFIOContainer (from non-arch specific code). It would avoid the ad-hoc passing of version here and also allow for some sanity checks. > + > + container->iommu_data.spapr.ramlistener =3D vfio_spapr_ram_memory_li= stener; > + container->iommu_data.release =3D vfio_spapr_listener_release_v2; > + memory_listener_register(&container->iommu_data.spapr.ramlistener, > + &address_space_memory); > + > + container->iommu_data.spapr.ram_reg_initialized =3D true; > + > + return container->iommu_data.spapr.ram_reg_error; > } > diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h > index d0b831c..b5ef446 100644 > --- a/include/hw/vfio/vfio-common.h > +++ b/include/hw/vfio/vfio-common.h > @@ -71,6 +71,9 @@ typedef struct VFIOType1 { > =20 > typedef struct VFIOSPAPR { > MemoryListener listener; > + MemoryListener ramlistener; The names "listener" and "ramlistener" don't exactly give a clear hint as to what the difference between them is. Maybe "register_listener" for the new one since its purpose is to register dma memory. I forget what the purpose of the old one is, and so what a better name for it might be. > + int ram_reg_error; > + bool ram_reg_initialized; > } VFIOSPAPR; > =20 > typedef struct VFIOContainer { > @@ -156,6 +159,6 @@ extern int vfio_dma_unmap(VFIOContainer *container, > hwaddr iova, ram_addr_t size); > bool vfio_listener_skipped_section(MemoryRegionSection *section); > =20 > -extern void spapr_memory_listener_register(VFIOContainer *container); > +extern int spapr_memory_listener_register(VFIOContainer *container, int = ver); > =20 > #endif /* !HW_VFIO_VFIO_COMMON_H */ > diff --git a/trace-events b/trace-events > index 1231ba4..2739140 100644 > --- a/trace-events > +++ b/trace-events > @@ -1563,6 +1563,7 @@ vfio_disconnect_container(int fd) "close container-= >fd=3D%d" > vfio_put_group(int fd) "close group->fd=3D%d" > vfio_get_device(const char * name, unsigned int flags, unsigned int num_= regions, unsigned int num_irqs) "Device %s flags: %u, regions: %u, irqs: %u" > vfio_put_base_device(int fd) "close vdev->fd=3D%d" > +vfio_ram_register(int req, uint64_t va, uint64_t size, int ret) "req=3D%= d va=3D%"PRIx64" size=3D%"PRIx64" ret=3D%d" > =20 > #hw/acpi/memory_hotplug.c > mhp_acpi_invalid_slot_selected(uint32_t slot) "0x%"PRIx32 --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --fNagykWcDoSVAmSd Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJVNzdCAAoJEGw4ysog2bOSR1kP/jV9D6LdUNWJrlJAqCkVz2th 1RTG+PQED4SdAS4hrytrbU6hmfGeSJVyWPxFjqL+8w2SB5cy2IYhjDjrhJ+wcsZi iN2MIhsuE6qnB2tUK701Yp2CUZIrPTZdWv2XgOug7iZmag7aDtGfqm3iws3LLwb7 yCVpb1mEeNv+AP0aidbzSPBt13OSC4YCdFStV3a7wiCNoplucez3dgrnCNfgR+uR czWUWIWVe5eQ1h4IhWcE4S2CZo8eUeuqGehOBnGlgmy1y8DcGzGWlP7j2IUOqqQI qsWxiaVwSfrPxDo6pai5fRwbyMfkMicAHdI6n1byceWWrhAfBmFmqB47P+Ui63sd B+e8Ik27Qc1LXE0US77ei74ngimbwq/wrpr36O7H+H5+DM27d7NxQQku1aCZUJ3g X+juWZkguLV2b4mKm/AyclSDbci+GmO/Z3HbvMQ8Dt//PjVkY8d33mHyukQbel3o z62SjdcSqShM7BOuXga8Z5EjbtU0z2mjiEBD6cFyCS+SI0twabFgUkxi6CPyB75w O+IL0ESzmdout28L7b5tvH4E5ntcYga2JTFpQX1E6yAcCkech61ZuGhT2JiXL9A6 2K27zj0gwyu87CUVD3El5u6sHBz0UQM0baLUNBdabUVl2k5t/LNUDoMgg/kEKChc jpjU+dTb3F+NRfUhKBMP =26mb -----END PGP SIGNATURE----- --fNagykWcDoSVAmSd--