From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53875) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bCmfD-000840-II for qemu-devel@nongnu.org; Tue, 14 Jun 2016 07:37:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bCmf7-0002kP-OJ for qemu-devel@nongnu.org; Tue, 14 Jun 2016 07:37:46 -0400 Received: from mx1.redhat.com ([209.132.183.28]:60024) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bCmf7-0002kK-GT for qemu-devel@nongnu.org; Tue, 14 Jun 2016 07:37:41 -0400 References: <1465813009-21390-1-git-send-email-liang.z.li@intel.com> <1465813009-21390-2-git-send-email-liang.z.li@intel.com> From: Thomas Huth Message-ID: <575FEC80.2070500@redhat.com> Date: Tue, 14 Jun 2016 13:37:36 +0200 MIME-Version: 1.0 In-Reply-To: <1465813009-21390-2-git-send-email-liang.z.li@intel.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [QEMU 1/7] balloon: speed up inflating & deflating process List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Liang Li , qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, mst@redhat.com, lcapitulino@redhat.com, pbonzini@redhat.com, quintela@redhat.com, amit.shah@redhat.com, dgilbert@redhat.com On 13.06.2016 12:16, Liang Li wrote: > The implementation of the current virtio-balloon is not very efficient, > Bellow is test result of time spends on inflating the balloon to 3GB of > a 4GB idle guest: >=20 > a. allocating pages (6.5%, 103ms) > b. sending PFNs to host (68.3%, 787ms) > c. address translation (6.1%, 96ms) > d. madvise (19%, 300ms) >=20 > It takes about 1577ms for the whole inflating process to complete. The > test shows that the bottle neck is the stage b and stage d. >=20 > If using a bitmap to send the page info instead of the PFNs, we can > reduce the overhead spends on stage b quite a lot. Furthermore, it's > possible to do the address translation and do the madvise with a bulk > of pages, instead of the current page per page way, so the overhead of > stage c and stage d can also be reduced a lot. >=20 > This patch is the QEMU side implementation which is intended to speed > up the inflating & deflating process by adding a new feature to the > virtio-balloon device. And now, inflating the balloon to 3GB of a 4GB > idle guest only takes 210ms, it's about 8 times as fast as before. >=20 > TODO: optimize stage a by allocating/freeing a chunk of pages instead > of a single page at a time. >=20 > Signed-off-by: Liang Li > --- > hw/virtio/virtio-balloon.c | 159 ++++++++++++++++= ++++---- > include/standard-headers/linux/virtio_balloon.h | 1 + > 2 files changed, 139 insertions(+), 21 deletions(-) >=20 > diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c > index 8c15e09..8cf74c2 100644 > --- a/hw/virtio/virtio-balloon.c > +++ b/hw/virtio/virtio-balloon.c > @@ -47,6 +47,76 @@ static void balloon_page(void *addr, int deflate) > #endif > } > =20 > +static void do_balloon_bulk_pages(ram_addr_t base_pfn, int page_shift, > + unsigned long len, bool deflate) > +{ > + ram_addr_t size, processed, chunk, base; > + void *addr; > + MemoryRegionSection section =3D {.mr =3D NULL}; > + > + size =3D (len << page_shift); > + base =3D (base_pfn << page_shift); > + > + for (processed =3D 0; processed < size; processed +=3D chunk) { > + chunk =3D size - processed; > + while (chunk >=3D TARGET_PAGE_SIZE) { > + section =3D memory_region_find(get_system_memory(), > + base + processed, chunk); > + if (!section.mr) { > + chunk =3D QEMU_ALIGN_DOWN(chunk / 2, TARGET_PAGE_SIZE)= ; > + } else { > + break; > + } > + } > + > + if (section.mr && > + (int128_nz(section.size) && memory_region_is_ram(section.m= r))) { > + addr =3D section.offset_within_region + > + memory_region_get_ram_ptr(section.mr); > + qemu_madvise(addr, chunk, > + deflate ? QEMU_MADV_WILLNEED : QEMU_MADV_DONT= NEED); > + } else { > + fprintf(stderr, "can't find the chunk, skip\n"); Please try to avoid new fprintf(stderr, ...) in the QEMU sources. Use error_report(...) or in this case maybe rather qemu_log_mask(LOG_GUEST_ERROR, ...) instead, and try to use a more reasonable error message (e.g. that it is clear that the error happened in the balloon code). > + chunk =3D TARGET_PAGE_SIZE; > + } > + } > +} > + > +static void balloon_bulk_pages(ram_addr_t base_pfn, unsigned long *bit= map, > + unsigned long len, int page_shift, bool= deflate) > +{ > +#if defined(__linux__) Why do you need this #if here? > + unsigned long end =3D len * 8; > + unsigned long current =3D 0; > + > + if (!qemu_balloon_is_inhibited() && (!kvm_enabled() || > + kvm_has_sync_mmu())) { > + while (current < end) { > + unsigned long one =3D find_next_bit(bitmap, end, current); > + > + if (one < end) { > + unsigned long zero =3D find_next_zero_bit(bitmap, end,= one + 1); > + unsigned long page_length; > + > + if (zero >=3D end) { > + page_length =3D end - one; > + } else { > + page_length =3D zero - one; > + } > + > + if (page_length) { > + do_balloon_bulk_pages(base_pfn + one, page_shift, > + page_length, deflate); > + } > + current =3D one + page_length; > + } else { > + current =3D one; > + } > + } > + } > +#endif > +} Thomas