[PATCH] hw/virtio: Add ioeventfd option for balloon

[PATCH] hw/virtio: Add ioeventfd option for balloon

[沈哲赟]

Traditional mmio in balloon makes Qemu do balloon inflation in the same
thread as vcpu thread. In a CPU overcommitment scenario, host may run
more than one vcpu threads on one host CPU, which makes
madvise_dontneed_free() wait for a long time due to the function
cond_resched() at host side.

If using SEV/ES and the kernel provided by AMD, the overhead will
become even much larger.(From 90s to 1400s when reclaming 4GB)

With ioeventfd, the thread for host to do balloon inflation will
be separated from the VCPU thread, leading to better performance
for the whole process of balloon inflation.(1400s to 263s 
in SEV CPU overcommitment scenario)

As a para-virtual solution, balloon serves for host so the process
of inflation in host needs to run on a host iothread instead of a guest
VCPU thread.

Signed-off-by: Zheyun Shen <szy0127@sjtu.edu.cn>
---
 hw/virtio/virtio-balloon-pci.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/hw/virtio/virtio-balloon-pci.c b/hw/virtio/virtio-balloon-pci.c
index ce2645b..7195322 100644
--- a/hw/virtio/virtio-balloon-pci.c
+++ b/hw/virtio/virtio-balloon-pci.c
@@ -35,6 +35,12 @@ struct VirtIOBalloonPCI {
     VirtIOBalloon vdev;
 };
 
+static Property virtio_balloon_properties[] = {
+    DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
+            VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, false),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
 static void virtio_balloon_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp)
 {
     VirtIOBalloonPCI *dev = VIRTIO_BALLOON_PCI(vpci_dev);
@@ -51,6 +57,7 @@ static void virtio_balloon_pci_class_init(ObjectClass *klass, void *data)
     PCIDeviceClass *pcidev_k = PCI_DEVICE_CLASS(klass);
     k->realize = virtio_balloon_pci_realize;
     set_bit(DEVICE_CATEGORY_MISC, dc->categories);
+    device_class_set_props(dc, virtio_balloon_properties);
     pcidev_k->vendor_id = PCI_VENDOR_ID_REDHAT_QUMRANET;
     pcidev_k->device_id = PCI_DEVICE_ID_VIRTIO_BALLOON;
     pcidev_k->revision = VIRTIO_PCI_ABI_VERSION;
--
2.34.1

Re: [PATCH] hw/virtio: Add ioeventfd option for balloon

[David Hildenbrand]

On 04.01.24 08:15, 沈哲赟 wrote:
> Traditional mmio in balloon makes Qemu do balloon inflation in the same
> thread as vcpu thread. In a CPU overcommitment scenario, host may run
> more than one vcpu threads on one host CPU, which makes
> madvise_dontneed_free() wait for a long time due to the function
> cond_resched() at host side.
> 
> If using SEV/ES and the kernel provided by AMD, the overhead will
> become even much larger.(From 90s to 1400s when reclaming 4GB)

I recall that encrypted VMs etc are not compatible with ballooning. Are 
there other (i.e., guest kernel) changes required for this setup to 
work? ("provided by AMD")

> 
> With ioeventfd, the thread for host to do balloon inflation will
> be separated from the VCPU thread, leading to better performance
> for the whole process of balloon inflation.(1400s to 263s
> in SEV CPU overcommitment scenario)
> 
> As a para-virtual solution, balloon serves for host so the process
> of inflation in host needs to run on a host iothread instead of a guest
> VCPU thread.
> 
> Signed-off-by: Zheyun Shen <szy0127@sjtu.edu.cn>

If this a resend i.e., v2, please indicate changes below the "---" and 
use versions like "PATCH v2".

https://lkml.kernel.org/r/https://lkml.kernel.org/r/20231212075058.710918-1-szy0127@sjtu.edu.cn

-- 
Cheers,

David / dhildenb

[PATCH] hw/virtio: Add ioeventfd option for balloon

[Zheyun Shen]

Traditional mmio in balloon makes qemu do balloon inflation in the same
thread as vcpu thread.In a CPU overcommitment scenario, host may run
more than one vcpu threads on one host thread, which makes
madvise_dontneed_free() wait for a long time due to the function
cond_resched() at host side.

If using SEV/ES and the kernel provided by AMD, the overhead will
become even much larger.

With ioeventfd, the thread for host to do balloon inflation will
be seperated from the vcpu thread, leading to better performance
for the whole process of balloon inflation.

Signed-off-by: Zheyun Shen <szy0127@sjtu.edu.cn>
---
 hw/virtio/virtio-balloon-pci.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/hw/virtio/virtio-balloon-pci.c b/hw/virtio/virtio-balloon-pci.c
index ce2645b..7195322 100644
--- a/hw/virtio/virtio-balloon-pci.c
+++ b/hw/virtio/virtio-balloon-pci.c
@@ -35,6 +35,12 @@ struct VirtIOBalloonPCI {
     VirtIOBalloon vdev;
 };
 
+static Property virtio_balloon_properties[] = {
+    DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
+            VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, false),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
 static void virtio_balloon_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp)
 {
     VirtIOBalloonPCI *dev = VIRTIO_BALLOON_PCI(vpci_dev);
@@ -51,6 +57,7 @@ static void virtio_balloon_pci_class_init(ObjectClass *klass, void *data)
     PCIDeviceClass *pcidev_k = PCI_DEVICE_CLASS(klass);
     k->realize = virtio_balloon_pci_realize;
     set_bit(DEVICE_CATEGORY_MISC, dc->categories);
+    device_class_set_props(dc, virtio_balloon_properties);
     pcidev_k->vendor_id = PCI_VENDOR_ID_REDHAT_QUMRANET;
     pcidev_k->device_id = PCI_DEVICE_ID_VIRTIO_BALLOON;
     pcidev_k->revision = VIRTIO_PCI_ABI_VERSION;
-- 
2.34.1

Re: [PATCH] hw/virtio: Add ioeventfd option for balloon

[David Hildenbrand]

Please CC the relevant maintainers, I only stumbled over that by luck.

See scripts/get_maintainer.pl


On 12.12.23 08:50, Zheyun Shen wrote:
> Traditional mmio in balloon makes qemu do balloon inflation in the same
> thread as vcpu thread.In a CPU overcommitment scenario, host may run
> more than one vcpu threads on one host thread, which makes
> madvise_dontneed_free() wait for a long time due to the function
> cond_resched() at host side.
> 
> If using SEV/ES and the kernel provided by AMD, the overhead will
> become even much larger.
> 
> With ioeventfd, the thread for host to do balloon inflation will
> be seperated from the vcpu thread, leading to better performance
> for the whole process of balloon inflation.

I didn't really look into ieventfd so far, but from what I understand, 
an iothread will take care of processing these events, instead of the 
vcpu thread.

Do you have some performance numbers?

I'll note that whatever thread calls madvise(), we're holding the BQL in 
that code and will block any other threads that need it.


> 
> Signed-off-by: Zheyun Shen <szy0127@sjtu.edu.cn>
> ---
>   hw/virtio/virtio-balloon-pci.c | 7 +++++++
>   1 file changed, 7 insertions(+)
> 
> diff --git a/hw/virtio/virtio-balloon-pci.c b/hw/virtio/virtio-balloon-pci.c
> index ce2645b..7195322 100644
> --- a/hw/virtio/virtio-balloon-pci.c
> +++ b/hw/virtio/virtio-balloon-pci.c
> @@ -35,6 +35,12 @@ struct VirtIOBalloonPCI {
>       VirtIOBalloon vdev;
>   };
>   
> +static Property virtio_balloon_properties[] = {
> +    DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
> +            VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, false),
> +    DEFINE_PROP_END_OF_LIST(),
> +};
> +
>   static void virtio_balloon_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp)
>   {
>       VirtIOBalloonPCI *dev = VIRTIO_BALLOON_PCI(vpci_dev);
> @@ -51,6 +57,7 @@ static void virtio_balloon_pci_class_init(ObjectClass *klass, void *data)
>       PCIDeviceClass *pcidev_k = PCI_DEVICE_CLASS(klass);
>       k->realize = virtio_balloon_pci_realize;
>       set_bit(DEVICE_CATEGORY_MISC, dc->categories);
> +    device_class_set_props(dc, virtio_balloon_properties);
>       pcidev_k->vendor_id = PCI_VENDOR_ID_REDHAT_QUMRANET;
>       pcidev_k->device_id = PCI_DEVICE_ID_VIRTIO_BALLOON;
>       pcidev_k->revision = VIRTIO_PCI_ABI_VERSION;

-- 
Cheers,

David / dhildenb