* [PATCH v11 0/3] Subject: [PATCH 0/3] virtio-net: Add notification coalescing support
@ 2026-10-17 19:22 Koushik Dutta
2026-10-17 19:22 ` [PATCH v11 1/3] [PATCH 1/3] Introduce virtio_net_handle_tx_dispatch() to unify TX path handling. This dispatcher dynamically selects between timer-based and BH-based TX processing based on configuration Koushik Dutta
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Koushik Dutta @ 2026-10-17 19:22 UTC (permalink / raw)
To: qemu-devel
Cc: Jason Wang, Jason Wang, Michael S. Tsirkin, Zhao Liu,
Stefano Garzarella, Philippe Mathieu-Daudé,
Eugenio Pérez
This series implements VIRTIO_NET_F_NOTF_COAL (bit 53) for virtio-net,
enabling interrupt coalescing to reduce CPU overhead in high-throughput
workloads.
Patch 1 refactors TX path handling by introducing a unified dispatcher
that dynamically selects between timer-based and BH-based processing.
This is preparatory work with no functional changes.
Patch 2 implements the core notification coalescing feature, allowing
guests to configure RX/TX interrupt batching via ethtool -C using
time-based (microseconds) or count-based (packet) thresholds.
Patch 3 adds vDPA backend support, enabling coalescing state to be
restored during live migration when using Shadow VirtQueue.
Testing:
- Functional: ethtool -C configuration in guest with vDPA device
- Migration: VMState persistence across live migration
- Performance: 20% CPU overhead reduction, 30% fewer interrupts
- Compatibility: Feature disabled when tx=timer is configured
Changes in v11:
---------------
- Adds vDPA backend support for notification coalescing
- Resolved review comments
Koushik Dutta (3):
Introduce virtio_net_handle_tx_dispatch() to unify TX path handling.
This dispatcher dynamically selects between timer-based and BH-based
TX processing based on configuration.
virtio-net: Add VIRTIO_NET_F_NOTF_COAL support
vhost-vdpa: Add VIRTIO_NET_F_NOTF_COAL support for vDPA
hw/net/virtio-net.c | 241 ++++++++++++++++++++++++++++++---
include/hw/virtio/virtio-net.h | 11 ++
net/passt.c | 1 +
net/tap.c | 1 +
net/vhost-user.c | 1 +
net/vhost-vdpa.c | 58 ++++++++
6 files changed, 294 insertions(+), 19 deletions(-)
--
2.53.0
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH v11 1/3] [PATCH 1/3] Introduce virtio_net_handle_tx_dispatch() to unify TX path handling. This dispatcher dynamically selects between timer-based and BH-based TX processing based on configuration. 2026-10-17 19:22 [PATCH v11 0/3] Subject: [PATCH 0/3] virtio-net: Add notification coalescing support Koushik Dutta @ 2026-10-17 19:22 ` Koushik Dutta 2026-10-17 19:22 ` [PATCH v11 2/3] [PATCH 2/3] virtio-net: Add VIRTIO_NET_F_NOTF_COAL support Koushik Dutta 2026-10-17 19:22 ` [PATCH v11 3/3] [PATCH 3/3] vhost-vdpa: Add VIRTIO_NET_F_NOTF_COAL support for vDPA Koushik Dutta 2 siblings, 0 replies; 5+ messages in thread From: Koushik Dutta @ 2026-10-17 19:22 UTC (permalink / raw) To: qemu-devel Cc: Jason Wang, Jason Wang, Michael S. Tsirkin, Zhao Liu, Stefano Garzarella, Philippe Mathieu-Daudé, Eugenio Pérez Previously, the tx=timer selected between two completely separate code paths at queue creation time. This refactoring introduces a runtime dispatch mechanism while maintaining identical behavior. This is a preparatory patch with no functional changes, making it easier to add dynamic TX notification coalescing in a subsequent patch. Signed-off-by: Koushik Dutta <kdutta@redhat.com> --- hw/net/virtio-net.c | 42 ++++++++++++++++++++++------------ include/hw/virtio/virtio-net.h | 1 + 2 files changed, 28 insertions(+), 15 deletions(-) diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c index 19f453fb7d..5ca011f890 100644 --- a/hw/net/virtio-net.c +++ b/hw/net/virtio-net.c @@ -3022,6 +3022,22 @@ static void virtio_net_tx_bh(void *opaque) } } +static void virtio_net_handle_tx_dispatch(VirtIODevice *vdev, VirtQueue *vq) +{ + VirtIONet *n = VIRTIO_NET(vdev); + VirtIONetQueue *q = &n->vqs[vq2q(virtio_get_queue_index(vq))]; + + if (n->tx_timer_activate) { + if (q->tx_timer == NULL) { + q->tx_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, + virtio_net_tx_timer, q); + } + virtio_net_handle_tx_timer(vdev, vq); + } else { + virtio_net_handle_tx_bh(vdev, vq); + } +} + static void virtio_net_add_queue(VirtIONet *n, int index) { VirtIODevice *vdev = VIRTIO_DEVICE(n); @@ -3029,21 +3045,13 @@ static void virtio_net_add_queue(VirtIONet *n, int index) n->vqs[index].rx_vq = virtio_add_queue(vdev, n->net_conf.rx_queue_size, virtio_net_handle_rx); - if (n->net_conf.tx && !strcmp(n->net_conf.tx, "timer")) { - n->vqs[index].tx_vq = - virtio_add_queue(vdev, n->net_conf.tx_queue_size, - virtio_net_handle_tx_timer); - n->vqs[index].tx_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, - virtio_net_tx_timer, - &n->vqs[index]); - } else { - n->vqs[index].tx_vq = - virtio_add_queue(vdev, n->net_conf.tx_queue_size, - virtio_net_handle_tx_bh); - n->vqs[index].tx_bh = virtio_bh_new_guarded(DEVICE(vdev), - virtio_net_tx_bh, - &n->vqs[index]); - } + n->vqs[index].tx_vq = + virtio_add_queue(vdev, n->net_conf.tx_queue_size, + virtio_net_handle_tx_dispatch); + + n->vqs[index].tx_bh = + qemu_bh_new_guarded(virtio_net_tx_bh, &n->vqs[index], + &DEVICE(vdev)->mem_reentrancy_guard); n->vqs[index].tx_waiting = 0; n->vqs[index].n = n; @@ -4107,6 +4115,10 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp) error_printf("Defaulting to \"bh\""); } + if (n->net_conf.tx && strcmp(n->net_conf.tx, "timer") == 0) { + n->tx_timer_activate = true; + } + n->net_conf.tx_queue_size = MIN(virtio_net_max_tx_queue_size(n), n->net_conf.tx_queue_size); diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h index 8c967760c2..2e7bc29967 100644 --- a/include/hw/virtio/virtio-net.h +++ b/include/hw/virtio/virtio-net.h @@ -231,6 +231,7 @@ struct VirtIONet { uint32_t nr_ebpf_rss_fds; char **ebpf_rss_fds; bool peers_wait_incoming; + bool tx_timer_activate; }; size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev, -- 2.53.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v11 2/3] [PATCH 2/3] virtio-net: Add VIRTIO_NET_F_NOTF_COAL support 2026-10-17 19:22 [PATCH v11 0/3] Subject: [PATCH 0/3] virtio-net: Add notification coalescing support Koushik Dutta 2026-10-17 19:22 ` [PATCH v11 1/3] [PATCH 1/3] Introduce virtio_net_handle_tx_dispatch() to unify TX path handling. This dispatcher dynamically selects between timer-based and BH-based TX processing based on configuration Koushik Dutta @ 2026-10-17 19:22 ` Koushik Dutta 2026-10-28 13:35 ` Michael S. Tsirkin 2026-10-17 19:22 ` [PATCH v11 3/3] [PATCH 3/3] vhost-vdpa: Add VIRTIO_NET_F_NOTF_COAL support for vDPA Koushik Dutta 2 siblings, 1 reply; 5+ messages in thread From: Koushik Dutta @ 2026-10-17 19:22 UTC (permalink / raw) To: qemu-devel Cc: Jason Wang, Jason Wang, Michael S. Tsirkin, Zhao Liu, Stefano Garzarella, Philippe Mathieu-Daudé, Eugenio Pérez Implement notification coalescing to reduce interrupt overhead. Guests configure time-based (microseconds) or count-based (packets) coalescing via ethtool -C. Coalescing state is migrated via VMState for live migration support. Note: Coalescing is disabled when tx=timer is configured, as timer mode already provides packet batching. Performance impact: 20% reduction in CPU overhead, 30% fewer interrupts under moderate coalescing settings. Signed-off-by: Koushik Dutta <kdutta@redhat.com> --- hw/net/virtio-net.c | 199 ++++++++++++++++++++++++++++++++- include/hw/virtio/virtio-net.h | 10 ++ net/passt.c | 1 + net/tap.c | 1 + net/vhost-user.c | 1 + net/vhost-vdpa.c | 1 + 6 files changed, 209 insertions(+), 4 deletions(-) diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c index 5ca011f890..610cc2b580 100644 --- a/hw/net/virtio-net.c +++ b/hw/net/virtio-net.c @@ -158,6 +158,16 @@ static void flush_or_purge_queued_packets(NetClientState *nc) * - we could suppress RX interrupt if we were so inclined. */ +static void virtio_net_rx_notify(void *opaque) +{ + VirtIONetQueue *q = opaque; + VirtIONet *n = q->n; + VirtIODevice *vdev = VIRTIO_DEVICE(n); + + q->rx_coal.pkt_cnt = 0; + virtio_notify(vdev, q->rx_vq); +} + static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config) { VirtIONet *n = VIRTIO_NET(vdev); @@ -1003,6 +1013,90 @@ static void virtio_net_set_features(VirtIODevice *vdev, } } +static void virtio_net_tx_notification_timer(void *opaque) +{ + VirtIONetQueue *q = opaque; + VirtIONet *n = q->n; + VirtIODevice *vdev = VIRTIO_DEVICE(n); + + q->tx_coal.pkt_cnt = 0; + virtio_notify(vdev, q->tx_vq); +} + +static bool virtio_net_tx_should_notify(VirtIONetQueue *q) +{ + if (!q->tx_coal.timer) { + return true; + } + + q->tx_coal.pkt_cnt++; + + if (q->tx_coal.packets > 0 && q->tx_coal.pkt_cnt >= q->tx_coal.packets) { + q->tx_coal.pkt_cnt = 0; + timer_del(q->tx_coal.timer); + return true; + } + + if (!timer_pending(q->tx_coal.timer)) { + timer_mod(q->tx_coal.timer, + qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + + q->tx_coal.usecs * 1000); + } + + return false; +} + +static int virtio_net_handle_coal(VirtIONet *n, uint8_t cmd, + struct iovec *iov, unsigned int iov_cnt) +{ + struct virtio_net_ctrl_coal coal; + VirtIONetQueue *q; + size_t s; + int i; + + if (n->tx_timer_activate) { + return VIRTIO_NET_ERR; + } + + s = iov_to_buf(iov, iov_cnt, 0, &coal, sizeof(coal)); + if (s != sizeof(coal)) { + return VIRTIO_NET_ERR; + } + + if (cmd == VIRTIO_NET_CTRL_NOTF_COAL_RX_SET) { + for (i = 0; i < n->max_queue_pairs; i++) { + q = &n->vqs[i]; + q->rx_coal.usecs = le32_to_cpu(coal.max_usecs); + q->rx_coal.packets = le32_to_cpu(coal.max_packets); + q->rx_coal.pkt_cnt = 0; + if (q->rx_coal.usecs > 0) { + g_clear_pointer(&q->rx_coal.timer, timer_free); + q->rx_coal.timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, + virtio_net_rx_notify, + q); + } else { + g_clear_pointer(&q->rx_coal.timer, timer_free); + } + } + } else if (cmd == VIRTIO_NET_CTRL_NOTF_COAL_TX_SET) { + for (i = 0; i < n->max_queue_pairs; i++) { + q = &n->vqs[i]; + q->tx_coal.usecs = le32_to_cpu(coal.max_usecs); + q->tx_coal.packets = le32_to_cpu(coal.max_packets); + q->tx_coal.pkt_cnt = 0; + if (q->tx_coal.usecs > 0) { + g_clear_pointer(&q->tx_coal.timer, timer_free); + q->tx_coal.timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, + virtio_net_tx_notification_timer, + q); + } else { + g_clear_pointer(&q->tx_coal.timer, timer_free); + } + } + } + return VIRTIO_NET_OK; +} + static int virtio_net_handle_rx_mode(VirtIONet *n, uint8_t cmd, struct iovec *iov, unsigned int iov_cnt) { @@ -1582,6 +1676,8 @@ size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev, status = virtio_net_handle_mq(n, ctrl.cmd, iov, out_num); } else if (ctrl.class == VIRTIO_NET_CTRL_GUEST_OFFLOADS) { status = virtio_net_handle_offloads(n, ctrl.cmd, iov, out_num); + } else if (ctrl.class == VIRTIO_NET_CTRL_NOTF_COAL) { + status = virtio_net_handle_coal(n, ctrl.cmd, iov, out_num); } s = iov_from_buf(in_sg, in_num, 0, &status, sizeof(status)); @@ -2056,7 +2152,23 @@ static ssize_t virtio_net_receive_rcu(NetClientState *nc, const uint8_t *buf, } virtqueue_flush(q->rx_vq, i); - virtio_notify(vdev, q->rx_vq); + + /* rx coalescing */ + q->rx_coal.pkt_cnt += i; + if (q->rx_coal.usecs == 0 || q->rx_coal.pkt_cnt >= q->rx_coal.packets) { + if (q->rx_coal.timer) { + timer_del(q->rx_coal.timer); + } + virtio_net_rx_notify(q); + } else { + if (q->rx_coal.timer) { + if (!timer_pending(q->rx_coal.timer)) { + timer_mod(q->rx_coal.timer, + qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + + q->rx_coal.usecs * 1000); /* Converted to ns */ + } + } + } return size; @@ -2739,7 +2851,9 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t len) int ret; virtqueue_push(q->tx_vq, q->async_tx.elem, 0); - virtio_notify(vdev, q->tx_vq); + if (virtio_net_tx_should_notify(q)) { + virtio_notify(vdev, q->tx_vq); + } g_free(q->async_tx.elem); q->async_tx.elem = NULL; @@ -2851,7 +2965,10 @@ static int32_t virtio_net_flush_tx(VirtIONetQueue *q) drop: virtqueue_push(q->tx_vq, elem, 0); - virtio_notify(vdev, q->tx_vq); + if (virtio_net_tx_should_notify(q)) { + virtio_notify(vdev, q->tx_vq); + } + g_free(elem); if (++num_packets >= n->tx_burst) { @@ -3054,6 +3171,12 @@ static void virtio_net_add_queue(VirtIONet *n, int index) &DEVICE(vdev)->mem_reentrancy_guard); n->vqs[index].tx_waiting = 0; + n->vqs[index].rx_coal.pkt_cnt = 0; + n->vqs[index].tx_coal.pkt_cnt = 0; + n->vqs[index].rx_coal.usecs = 0; + n->vqs[index].tx_coal.usecs = 0; + n->vqs[index].rx_coal.packets = 0; + n->vqs[index].tx_coal.packets = 0; n->vqs[index].n = n; } @@ -3066,6 +3189,10 @@ static void virtio_net_del_queue(VirtIONet *n, int index) qemu_purge_queued_packets(nc); virtio_del_queue(vdev, index * 2); + + g_clear_pointer(&q->rx_coal.timer, timer_free); + g_clear_pointer(&q->tx_coal.timer, timer_free); + if (q->tx_timer) { timer_free(q->tx_timer); q->tx_timer = NULL; @@ -3156,6 +3283,13 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features, virtio_features_or(features, features, n->host_features_ex); virtio_add_feature_ex(features, VIRTIO_NET_F_MAC); + if (n->tx_timer_activate) { + virtio_clear_feature_ex(features, VIRTIO_NET_F_NOTF_COAL); + } else { + if (!virtio_has_feature(*features, VIRTIO_NET_F_NOTF_COAL)) { + *features &= ~(1ULL << VIRTIO_NET_F_NOTF_COAL); + } + } if (n->peers_wait_incoming) { /* @@ -3336,6 +3470,21 @@ static int virtio_net_post_load_device(void *opaque, int version_id) return -EINVAL; } + for (i = 0; i < n->max_queue_pairs; i++) { + VirtIONetQueue *q = &n->vqs[i]; + if (q->rx_coal.usecs > 0 && !q->rx_coal.timer) { + q->rx_coal.timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, + virtio_net_rx_notify, + q); + } + + if (q->tx_coal.usecs > 0 && !q->tx_coal.timer) { + q->tx_coal.timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, + virtio_net_tx_notification_timer, + q); + } + } + return 0; } @@ -3355,13 +3504,49 @@ static int virtio_net_post_load_virtio(VirtIODevice *vdev) return 0; } +static const VMStateDescription vmstate_virtio_net_coal = { + .name = "virtio-net-coal", + .version_id = 1, + .minimum_version_id = 1, + .fields = (const VMStateField[]) { + VMSTATE_UINT32(usecs, VirtIONetCoal), + VMSTATE_UINT32(packets, VirtIONetCoal), + VMSTATE_UINT32(pkt_cnt, VirtIONetCoal), + VMSTATE_END_OF_LIST() + }, +}; + +static bool virtio_net_queue_notf_coal_needed(void *opaque) +{ + VirtIONetQueue *q = opaque; + return q->rx_coal.usecs > 0 || q->tx_coal.usecs > 0; +} + +static const VMStateDescription vmstate_virtio_net_queue_notf_coal = { + .name = "virtio-net-queue-tx_waiting/notf-coal", + .version_id = 1, + .minimum_version_id = 1, + .needed = virtio_net_queue_notf_coal_needed, + .fields = (const VMStateField[]) { + VMSTATE_STRUCT(rx_coal, VirtIONetQueue, 0, + vmstate_virtio_net_coal, VirtIONetCoal), + VMSTATE_STRUCT(tx_coal, VirtIONetQueue, 0, + vmstate_virtio_net_coal, VirtIONetCoal), + VMSTATE_END_OF_LIST() + }, +}; + /* tx_waiting field of a VirtIONetQueue */ static const VMStateDescription vmstate_virtio_net_queue_tx_waiting = { .name = "virtio-net-queue-tx_waiting", .fields = (const VMStateField[]) { VMSTATE_UINT32(tx_waiting, VirtIONetQueue), VMSTATE_END_OF_LIST() - }, + }, + .subsections = (const VMStateDescription * const []) { + &vmstate_virtio_net_queue_notf_coal, + NULL + } }; static bool max_queue_pairs_gt_1(void *opaque, int version_id) @@ -4117,6 +4302,10 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp) if (n->net_conf.tx && strcmp(n->net_conf.tx, "timer") == 0) { n->tx_timer_activate = true; + if (n->host_features & (1ULL << VIRTIO_NET_F_NOTF_COAL)) { + warn_report("virtio-net: 'vq_notf_coal' is incompatible with 'tx=timer'," + "disabling notification coalescing"); + } } n->net_conf.tx_queue_size = MIN(virtio_net_max_tx_queue_size(n), @@ -4413,6 +4602,8 @@ static const Property virtio_net_properties[] = { VIRTIO_NET_F_GUEST_USO6, true), DEFINE_PROP_BIT64("host_uso", VirtIONet, host_features, VIRTIO_NET_F_HOST_USO, true), + DEFINE_PROP_BIT64("vq_notf_coal", VirtIONet, host_features, + VIRTIO_NET_F_NOTF_COAL, false), DEFINE_PROP_ON_OFF_AUTO_BIT64("hash-ipv4", VirtIONet, rss_data.specified_hash_types, VIRTIO_NET_HASH_REPORT_IPv4 - 1, diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h index 2e7bc29967..c7bb61b3e5 100644 --- a/include/hw/virtio/virtio-net.h +++ b/include/hw/virtio/virtio-net.h @@ -155,6 +155,14 @@ typedef struct VirtioNetRssData { uint16_t default_queue; } VirtioNetRssData; +/* Interrupt notification coalescing parameters */ +typedef struct VirtIONetCoal { + uint32_t usecs; /* Timeout in microseconds */ + uint32_t packets; /* Packet count threshold */ + uint32_t pkt_cnt; /* Current packet count since last notification */ + QEMUTimer *timer; /* Coalescing timer */ +} VirtIONetCoal; + typedef struct VirtIONetQueue { VirtQueue *rx_vq; VirtQueue *tx_vq; @@ -165,6 +173,8 @@ typedef struct VirtIONetQueue { VirtQueueElement *elem; } async_tx; struct VirtIONet *n; + VirtIONetCoal rx_coal; + VirtIONetCoal tx_coal; } VirtIONetQueue; struct VirtIONet { diff --git a/net/passt.c b/net/passt.c index e65198960e..5ce25d0463 100644 --- a/net/passt.c +++ b/net/passt.c @@ -52,6 +52,7 @@ static const int user_feature_bits[] = { VIRTIO_NET_F_GUEST_USO4, VIRTIO_NET_F_GUEST_USO6, VIRTIO_NET_F_HOST_USO, + VIRTIO_NET_F_NOTF_COAL, /* This bit implies RARP isn't sent by QEMU out of band */ VIRTIO_NET_F_GUEST_ANNOUNCE, diff --git a/net/tap.c b/net/tap.c index d71e3dd117..59286e141d 100644 --- a/net/tap.c +++ b/net/tap.c @@ -69,6 +69,7 @@ static const int kernel_feature_bits[] = { VIRTIO_F_NOTIFICATION_DATA, VIRTIO_NET_F_RSC_EXT, VIRTIO_NET_F_HASH_REPORT, + VIRTIO_NET_F_NOTF_COAL, VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO, VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO, VHOST_INVALID_FEATURE_BIT diff --git a/net/vhost-user.c b/net/vhost-user.c index b849de92da..4674ebb5b5 100644 --- a/net/vhost-user.c +++ b/net/vhost-user.c @@ -54,6 +54,7 @@ static const int user_feature_bits[] = { VIRTIO_NET_F_GUEST_USO4, VIRTIO_NET_F_GUEST_USO6, VIRTIO_NET_F_HOST_USO, + VIRTIO_NET_F_NOTF_COAL, /* This bit implies RARP isn't sent by QEMU out of band */ VIRTIO_NET_F_GUEST_ANNOUNCE, diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 1052361a4a..ebd77135da 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -71,6 +71,7 @@ static const int vdpa_feature_bits[] = { VIRTIO_NET_F_CTRL_RX, VIRTIO_NET_F_CTRL_RX_EXTRA, VIRTIO_NET_F_CTRL_VLAN, + VIRTIO_NET_F_NOTF_COAL, VIRTIO_NET_F_CTRL_VQ, VIRTIO_NET_F_GSO, VIRTIO_NET_F_GUEST_CSUM, -- 2.53.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v11 2/3] [PATCH 2/3] virtio-net: Add VIRTIO_NET_F_NOTF_COAL support 2026-10-17 19:22 ` [PATCH v11 2/3] [PATCH 2/3] virtio-net: Add VIRTIO_NET_F_NOTF_COAL support Koushik Dutta @ 2026-10-28 13:35 ` Michael S. Tsirkin 0 siblings, 0 replies; 5+ messages in thread From: Michael S. Tsirkin @ 2026-10-28 13:35 UTC (permalink / raw) To: Koushik Dutta Cc: qemu-devel, Jason Wang, Jason Wang, Zhao Liu, Stefano Garzarella, Philippe Mathieu-Daudé, Eugenio Pérez On Sun, Oct 18, 2026 at 12:52:31AM +0530, Koushik Dutta wrote: > Implement notification coalescing to reduce interrupt overhead. > Guests configure time-based (microseconds) or count-based (packets) > coalescing via ethtool -C. Coalescing state is migrated via VMState > for live migration support. > > Note: Coalescing is disabled when tx=timer is configured, as timer > mode already provides packet batching. > > Performance impact: 20% reduction in CPU overhead, 30% fewer interrupts > under moderate coalescing settings. > > Signed-off-by: Koushik Dutta <kdutta@redhat.com> > --- > hw/net/virtio-net.c | 199 ++++++++++++++++++++++++++++++++- > include/hw/virtio/virtio-net.h | 10 ++ > net/passt.c | 1 + > net/tap.c | 1 + > net/vhost-user.c | 1 + > net/vhost-vdpa.c | 1 + > 6 files changed, 209 insertions(+), 4 deletions(-) > > diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c > index 5ca011f890..610cc2b580 100644 > --- a/hw/net/virtio-net.c > +++ b/hw/net/virtio-net.c > @@ -158,6 +158,16 @@ static void flush_or_purge_queued_packets(NetClientState *nc) > * - we could suppress RX interrupt if we were so inclined. > */ > > +static void virtio_net_rx_notify(void *opaque) > +{ > + VirtIONetQueue *q = opaque; > + VirtIONet *n = q->n; > + VirtIODevice *vdev = VIRTIO_DEVICE(n); > + > + q->rx_coal.pkt_cnt = 0; > + virtio_notify(vdev, q->rx_vq); > +} > + > static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config) > { > VirtIONet *n = VIRTIO_NET(vdev); > @@ -1003,6 +1013,90 @@ static void virtio_net_set_features(VirtIODevice *vdev, > } > } > > +static void virtio_net_tx_notification_timer(void *opaque) > +{ > + VirtIONetQueue *q = opaque; > + VirtIONet *n = q->n; > + VirtIODevice *vdev = VIRTIO_DEVICE(n); > + > + q->tx_coal.pkt_cnt = 0; > + virtio_notify(vdev, q->tx_vq); > +} > + > +static bool virtio_net_tx_should_notify(VirtIONetQueue *q) > +{ > + if (!q->tx_coal.timer) { > + return true; > + } > + > + q->tx_coal.pkt_cnt++; > + > + if (q->tx_coal.packets > 0 && q->tx_coal.pkt_cnt >= q->tx_coal.packets) { > + q->tx_coal.pkt_cnt = 0; > + timer_del(q->tx_coal.timer); > + return true; > + } > + > + if (!timer_pending(q->tx_coal.timer)) { > + timer_mod(q->tx_coal.timer, > + qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + > + q->tx_coal.usecs * 1000); > + } > + > + return false; > +} > + > +static int virtio_net_handle_coal(VirtIONet *n, uint8_t cmd, > + struct iovec *iov, unsigned int iov_cnt) > +{ > + struct virtio_net_ctrl_coal coal; > + VirtIONetQueue *q; > + size_t s; > + int i; > + > + if (n->tx_timer_activate) { > + return VIRTIO_NET_ERR; > + } > + > + s = iov_to_buf(iov, iov_cnt, 0, &coal, sizeof(coal)); > + if (s != sizeof(coal)) { > + return VIRTIO_NET_ERR; > + } > + > + if (cmd == VIRTIO_NET_CTRL_NOTF_COAL_RX_SET) { > + for (i = 0; i < n->max_queue_pairs; i++) { > + q = &n->vqs[i]; > + q->rx_coal.usecs = le32_to_cpu(coal.max_usecs); > + q->rx_coal.packets = le32_to_cpu(coal.max_packets); > + q->rx_coal.pkt_cnt = 0; > + if (q->rx_coal.usecs > 0) { > + g_clear_pointer(&q->rx_coal.timer, timer_free); > + q->rx_coal.timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, > + virtio_net_rx_notify, > + q); > + } else { > + g_clear_pointer(&q->rx_coal.timer, timer_free); > + } > + } > + } else if (cmd == VIRTIO_NET_CTRL_NOTF_COAL_TX_SET) { > + for (i = 0; i < n->max_queue_pairs; i++) { > + q = &n->vqs[i]; > + q->tx_coal.usecs = le32_to_cpu(coal.max_usecs); > + q->tx_coal.packets = le32_to_cpu(coal.max_packets); > + q->tx_coal.pkt_cnt = 0; > + if (q->tx_coal.usecs > 0) { > + g_clear_pointer(&q->tx_coal.timer, timer_free); > + q->tx_coal.timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, > + virtio_net_tx_notification_timer, > + q); > + } else { > + g_clear_pointer(&q->tx_coal.timer, timer_free); > + } > + } > + } > + return VIRTIO_NET_OK; > +} > + > static int virtio_net_handle_rx_mode(VirtIONet *n, uint8_t cmd, > struct iovec *iov, unsigned int iov_cnt) > { > @@ -1582,6 +1676,8 @@ size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev, > status = virtio_net_handle_mq(n, ctrl.cmd, iov, out_num); > } else if (ctrl.class == VIRTIO_NET_CTRL_GUEST_OFFLOADS) { > status = virtio_net_handle_offloads(n, ctrl.cmd, iov, out_num); > + } else if (ctrl.class == VIRTIO_NET_CTRL_NOTF_COAL) { > + status = virtio_net_handle_coal(n, ctrl.cmd, iov, out_num); > } > > s = iov_from_buf(in_sg, in_num, 0, &status, sizeof(status)); > @@ -2056,7 +2152,23 @@ static ssize_t virtio_net_receive_rcu(NetClientState *nc, const uint8_t *buf, > } > > virtqueue_flush(q->rx_vq, i); > - virtio_notify(vdev, q->rx_vq); > + > + /* rx coalescing */ > + q->rx_coal.pkt_cnt += i; > + if (q->rx_coal.usecs == 0 || q->rx_coal.pkt_cnt >= q->rx_coal.packets) { > + if (q->rx_coal.timer) { > + timer_del(q->rx_coal.timer); > + } > + virtio_net_rx_notify(q); > + } else { > + if (q->rx_coal.timer) { > + if (!timer_pending(q->rx_coal.timer)) { > + timer_mod(q->rx_coal.timer, > + qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + > + q->rx_coal.usecs * 1000); /* Converted to ns */ > + } > + } > + } > > return size; > > @@ -2739,7 +2851,9 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t len) > int ret; > > virtqueue_push(q->tx_vq, q->async_tx.elem, 0); > - virtio_notify(vdev, q->tx_vq); > + if (virtio_net_tx_should_notify(q)) { > + virtio_notify(vdev, q->tx_vq); > + } > > g_free(q->async_tx.elem); > q->async_tx.elem = NULL; > @@ -2851,7 +2965,10 @@ static int32_t virtio_net_flush_tx(VirtIONetQueue *q) > > drop: > virtqueue_push(q->tx_vq, elem, 0); > - virtio_notify(vdev, q->tx_vq); > + if (virtio_net_tx_should_notify(q)) { > + virtio_notify(vdev, q->tx_vq); > + } > + > g_free(elem); > > if (++num_packets >= n->tx_burst) { > @@ -3054,6 +3171,12 @@ static void virtio_net_add_queue(VirtIONet *n, int index) > &DEVICE(vdev)->mem_reentrancy_guard); > > n->vqs[index].tx_waiting = 0; > + n->vqs[index].rx_coal.pkt_cnt = 0; > + n->vqs[index].tx_coal.pkt_cnt = 0; > + n->vqs[index].rx_coal.usecs = 0; > + n->vqs[index].tx_coal.usecs = 0; > + n->vqs[index].rx_coal.packets = 0; > + n->vqs[index].tx_coal.packets = 0; > n->vqs[index].n = n; > } > > @@ -3066,6 +3189,10 @@ static void virtio_net_del_queue(VirtIONet *n, int index) > qemu_purge_queued_packets(nc); > > virtio_del_queue(vdev, index * 2); > + > + g_clear_pointer(&q->rx_coal.timer, timer_free); > + g_clear_pointer(&q->tx_coal.timer, timer_free); > + > if (q->tx_timer) { > timer_free(q->tx_timer); > q->tx_timer = NULL; > @@ -3156,6 +3283,13 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features, > virtio_features_or(features, features, n->host_features_ex); > > virtio_add_feature_ex(features, VIRTIO_NET_F_MAC); > + if (n->tx_timer_activate) { > + virtio_clear_feature_ex(features, VIRTIO_NET_F_NOTF_COAL); > + } else { > + if (!virtio_has_feature(*features, VIRTIO_NET_F_NOTF_COAL)) { > + *features &= ~(1ULL << VIRTIO_NET_F_NOTF_COAL); > + } > + } > > if (n->peers_wait_incoming) { > /* > @@ -3336,6 +3470,21 @@ static int virtio_net_post_load_device(void *opaque, int version_id) > return -EINVAL; > } > > + for (i = 0; i < n->max_queue_pairs; i++) { > + VirtIONetQueue *q = &n->vqs[i]; > + if (q->rx_coal.usecs > 0 && !q->rx_coal.timer) { > + q->rx_coal.timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, > + virtio_net_rx_notify, > + q); > + } > + > + if (q->tx_coal.usecs > 0 && !q->tx_coal.timer) { > + q->tx_coal.timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, > + virtio_net_tx_notification_timer, > + q); > + } > + } > + > return 0; > } > > @@ -3355,13 +3504,49 @@ static int virtio_net_post_load_virtio(VirtIODevice *vdev) > return 0; > } > > +static const VMStateDescription vmstate_virtio_net_coal = { > + .name = "virtio-net-coal", > + .version_id = 1, > + .minimum_version_id = 1, > + .fields = (const VMStateField[]) { > + VMSTATE_UINT32(usecs, VirtIONetCoal), > + VMSTATE_UINT32(packets, VirtIONetCoal), > + VMSTATE_UINT32(pkt_cnt, VirtIONetCoal), > + VMSTATE_END_OF_LIST() > + }, > +}; > + > +static bool virtio_net_queue_notf_coal_needed(void *opaque) > +{ > + VirtIONetQueue *q = opaque; > + return q->rx_coal.usecs > 0 || q->tx_coal.usecs > 0; > +} > + > +static const VMStateDescription vmstate_virtio_net_queue_notf_coal = { > + .name = "virtio-net-queue-tx_waiting/notf-coal", > + .version_id = 1, > + .minimum_version_id = 1, > + .needed = virtio_net_queue_notf_coal_needed, > + .fields = (const VMStateField[]) { > + VMSTATE_STRUCT(rx_coal, VirtIONetQueue, 0, > + vmstate_virtio_net_coal, VirtIONetCoal), > + VMSTATE_STRUCT(tx_coal, VirtIONetQueue, 0, > + vmstate_virtio_net_coal, VirtIONetCoal), > + VMSTATE_END_OF_LIST() > + }, > +}; > + > /* tx_waiting field of a VirtIONetQueue */ > static const VMStateDescription vmstate_virtio_net_queue_tx_waiting = { > .name = "virtio-net-queue-tx_waiting", > .fields = (const VMStateField[]) { > VMSTATE_UINT32(tx_waiting, VirtIONetQueue), > VMSTATE_END_OF_LIST() > - }, > + }, > + .subsections = (const VMStateDescription * const []) { > + &vmstate_virtio_net_queue_notf_coal, > + NULL > + } > }; > > static bool max_queue_pairs_gt_1(void *opaque, int version_id) > @@ -4117,6 +4302,10 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp) > > if (n->net_conf.tx && strcmp(n->net_conf.tx, "timer") == 0) { > n->tx_timer_activate = true; > + if (n->host_features & (1ULL << VIRTIO_NET_F_NOTF_COAL)) { > + warn_report("virtio-net: 'vq_notf_coal' is incompatible with 'tx=timer'," > + "disabling notification coalescing"); > + } > } > > n->net_conf.tx_queue_size = MIN(virtio_net_max_tx_queue_size(n), > @@ -4413,6 +4602,8 @@ static const Property virtio_net_properties[] = { > VIRTIO_NET_F_GUEST_USO6, true), > DEFINE_PROP_BIT64("host_uso", VirtIONet, host_features, > VIRTIO_NET_F_HOST_USO, true), > + DEFINE_PROP_BIT64("vq_notf_coal", VirtIONet, host_features, > + VIRTIO_NET_F_NOTF_COAL, false), > DEFINE_PROP_ON_OFF_AUTO_BIT64("hash-ipv4", VirtIONet, > rss_data.specified_hash_types, > VIRTIO_NET_HASH_REPORT_IPv4 - 1, > diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h > index 2e7bc29967..c7bb61b3e5 100644 > --- a/include/hw/virtio/virtio-net.h > +++ b/include/hw/virtio/virtio-net.h > @@ -155,6 +155,14 @@ typedef struct VirtioNetRssData { > uint16_t default_queue; > } VirtioNetRssData; > > +/* Interrupt notification coalescing parameters */ > +typedef struct VirtIONetCoal { > + uint32_t usecs; /* Timeout in microseconds */ > + uint32_t packets; /* Packet count threshold */ > + uint32_t pkt_cnt; /* Current packet count since last notification */ > + QEMUTimer *timer; /* Coalescing timer */ > +} VirtIONetCoal; > + > typedef struct VirtIONetQueue { > VirtQueue *rx_vq; > VirtQueue *tx_vq; > @@ -165,6 +173,8 @@ typedef struct VirtIONetQueue { > VirtQueueElement *elem; > } async_tx; > struct VirtIONet *n; > + VirtIONetCoal rx_coal; > + VirtIONetCoal tx_coal; > } VirtIONetQueue; > > struct VirtIONet { > diff --git a/net/passt.c b/net/passt.c > index e65198960e..5ce25d0463 100644 > --- a/net/passt.c > +++ b/net/passt.c > @@ -52,6 +52,7 @@ static const int user_feature_bits[] = { > VIRTIO_NET_F_GUEST_USO4, > VIRTIO_NET_F_GUEST_USO6, > VIRTIO_NET_F_HOST_USO, > + VIRTIO_NET_F_NOTF_COAL, > > /* This bit implies RARP isn't sent by QEMU out of band */ > VIRTIO_NET_F_GUEST_ANNOUNCE, > diff --git a/net/tap.c b/net/tap.c > index d71e3dd117..59286e141d 100644 > --- a/net/tap.c > +++ b/net/tap.c > @@ -69,6 +69,7 @@ static const int kernel_feature_bits[] = { > VIRTIO_F_NOTIFICATION_DATA, > VIRTIO_NET_F_RSC_EXT, > VIRTIO_NET_F_HASH_REPORT, > + VIRTIO_NET_F_NOTF_COAL, > VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO, > VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO, > VHOST_INVALID_FEATURE_BIT > diff --git a/net/vhost-user.c b/net/vhost-user.c > index b849de92da..4674ebb5b5 100644 > --- a/net/vhost-user.c > +++ b/net/vhost-user.c > @@ -54,6 +54,7 @@ static const int user_feature_bits[] = { > VIRTIO_NET_F_GUEST_USO4, > VIRTIO_NET_F_GUEST_USO6, > VIRTIO_NET_F_HOST_USO, > + VIRTIO_NET_F_NOTF_COAL, > > /* This bit implies RARP isn't sent by QEMU out of band */ > VIRTIO_NET_F_GUEST_ANNOUNCE, > diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c > index 1052361a4a..ebd77135da 100644 > --- a/net/vhost-vdpa.c > +++ b/net/vhost-vdpa.c > @@ -71,6 +71,7 @@ static const int vdpa_feature_bits[] = { > VIRTIO_NET_F_CTRL_RX, > VIRTIO_NET_F_CTRL_RX_EXTRA, > VIRTIO_NET_F_CTRL_VLAN, > + VIRTIO_NET_F_NOTF_COAL, > VIRTIO_NET_F_CTRL_VQ, > VIRTIO_NET_F_GSO, > VIRTIO_NET_F_GUEST_CSUM, which of these backends actually implement the feature? We do not pass-through all bits not because we like code duplication but because we never know what configuration will be required from backend. > -- > 2.53.0 ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v11 3/3] [PATCH 3/3] vhost-vdpa: Add VIRTIO_NET_F_NOTF_COAL support for vDPA 2026-10-17 19:22 [PATCH v11 0/3] Subject: [PATCH 0/3] virtio-net: Add notification coalescing support Koushik Dutta 2026-10-17 19:22 ` [PATCH v11 1/3] [PATCH 1/3] Introduce virtio_net_handle_tx_dispatch() to unify TX path handling. This dispatcher dynamically selects between timer-based and BH-based TX processing based on configuration Koushik Dutta 2026-10-17 19:22 ` [PATCH v11 2/3] [PATCH 2/3] virtio-net: Add VIRTIO_NET_F_NOTF_COAL support Koushik Dutta @ 2026-10-17 19:22 ` Koushik Dutta 2 siblings, 0 replies; 5+ messages in thread From: Koushik Dutta @ 2026-10-17 19:22 UTC (permalink / raw) To: qemu-devel Cc: Jason Wang, Jason Wang, Michael S. Tsirkin, Zhao Liu, Stefano Garzarella, Philippe Mathieu-Daudé, Eugenio Pérez Software mode (svq=on): QEMU implements coalescing using timers in virtio_net_handle_ctrl(). Delays RX/TX notifications based on packet count and microsecond thresholds configured via control virtqueue. Hardware mode (svq=off): Commands passed through to vDPA device. Requires hardware support for VIRTIO_NET_F_NOTF_COAL in silicon Signed-off-by: Koushik Dutta <kdutta@redhat.com> --- net/vhost-vdpa.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index ebd77135da..e8dc79fb23 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -118,6 +118,7 @@ static const uint64_t vdpa_svq_device_features = BIT_ULL(VIRTIO_NET_F_MRG_RXBUF) | BIT_ULL(VIRTIO_NET_F_STATUS) | BIT_ULL(VIRTIO_NET_F_CTRL_VQ) | + BIT_ULL(VIRTIO_NET_F_NOTF_COAL) | BIT_ULL(VIRTIO_NET_F_GSO) | BIT_ULL(VIRTIO_NET_F_CTRL_RX) | BIT_ULL(VIRTIO_NET_F_CTRL_VLAN) | @@ -1233,6 +1234,58 @@ static int vhost_vdpa_net_load_vlan(VhostVDPAState *s, return 0; } +static int vhost_vdpa_net_load_coal(VhostVDPAState *s, + const VirtIONet *n, + struct iovec *out_cursor, + struct iovec *in_cursor) +{ + struct virtio_net_ctrl_coal_rx coal_rx; + struct virtio_net_ctrl_coal_tx coal_tx; + ssize_t dev_written; + + if (!virtio_vdev_has_feature(VIRTIO_DEVICE(n), VIRTIO_NET_F_NOTF_COAL)) { + return 0; + } + + coal_rx.rx_max_packets = cpu_to_le32(n->vqs[0].rx_coal.packets); + coal_rx.rx_usecs = cpu_to_le32(n->vqs[0].rx_coal.usecs); + + const struct iovec data_rx[] = { + { + .iov_base = &coal_rx, + .iov_len = sizeof(coal_rx), + }, + }; + + dev_written = vhost_vdpa_net_load_cmd(s, out_cursor, in_cursor, + VIRTIO_NET_CTRL_NOTF_COAL, + VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, + data_rx, ARRAY_SIZE(data_rx)); + if (unlikely(dev_written < 0)) { + return dev_written; + } + + coal_tx.tx_max_packets = cpu_to_le32(n->vqs[0].tx_coal.packets); + coal_tx.tx_usecs = cpu_to_le32(n->vqs[0].tx_coal.usecs); + + const struct iovec data_tx[] = { + { + .iov_base = &coal_tx, + .iov_len = sizeof(coal_tx), + }, + }; + + dev_written = vhost_vdpa_net_load_cmd(s, out_cursor, in_cursor, + VIRTIO_NET_CTRL_NOTF_COAL, + VIRTIO_NET_CTRL_NOTF_COAL_TX_SET, + data_tx, ARRAY_SIZE(data_tx)); + if (unlikely(dev_written < 0)) { + return dev_written; + } + + return 0; +} + static int vhost_vdpa_net_cvq_load(NetClientState *nc) { VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); @@ -1271,6 +1324,10 @@ static int vhost_vdpa_net_cvq_load(NetClientState *nc) if (unlikely(r)) { return r; } + r = vhost_vdpa_net_load_coal(s, n, &out_cursor, &in_cursor); + if (unlikely(r < 0)) { + return r; + } /* * We need to poll and check all pending device's used buffers. -- 2.53.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-10-28 13:35 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-10-17 19:22 [PATCH v11 0/3] Subject: [PATCH 0/3] virtio-net: Add notification coalescing support Koushik Dutta 2026-10-17 19:22 ` [PATCH v11 1/3] [PATCH 1/3] Introduce virtio_net_handle_tx_dispatch() to unify TX path handling. This dispatcher dynamically selects between timer-based and BH-based TX processing based on configuration Koushik Dutta 2026-10-17 19:22 ` [PATCH v11 2/3] [PATCH 2/3] virtio-net: Add VIRTIO_NET_F_NOTF_COAL support Koushik Dutta 2026-10-28 13:35 ` Michael S. Tsirkin 2026-10-17 19:22 ` [PATCH v11 3/3] [PATCH 3/3] vhost-vdpa: Add VIRTIO_NET_F_NOTF_COAL support for vDPA Koushik Dutta
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.