[PATCH net-next 0/2] Add TX stop/wake counters

[PATCH net-next 0/2] Add TX stop/wake counters

[Daniel Jurgens]

Several drivers provide TX stop and wake counters via ethtool stats. Add
those to the netdev queue stats, and use them in virtio_net.

Daniel Jurgens (2):
  netdev: Add queue stats for TX stop and wake
  virtio_net: Add TX stopped and wake counters

 Documentation/netlink/specs/netdev.yaml | 10 +++++++++
 drivers/net/virtio_net.c                | 28 +++++++++++++++++++++++--
 include/net/netdev_queues.h             |  3 +++
 include/uapi/linux/netdev.h             |  2 ++
 net/core/netdev-genl.c                  |  4 +++-
 tools/include/uapi/linux/netdev.h       |  3 ++-
 6 files changed, 46 insertions(+), 4 deletions(-)

-- 
2.44.0

[PATCH net-next 0/2] Add TX stop/wake counters

[Daniel Jurgens]

Several drivers provide TX stop and wake counters via ethtool stats. Add
those to the netdev queue stats, and use them in virtio_net.

v2:
	- Fixed an accidental line deletion
	- Enhanced documentation

Daniel Jurgens (2):
  netdev: Add queue stats for TX stop and wake
  virtio_net: Add TX stopped and wake counters

 Documentation/netlink/specs/netdev.yaml | 14 +++++++++++++
 drivers/net/virtio_net.c                | 28 +++++++++++++++++++++++--
 include/net/netdev_queues.h             |  3 +++
 include/uapi/linux/netdev.h             |  2 ++
 net/core/netdev-genl.c                  |  4 +++-
 tools/include/uapi/linux/netdev.h       |  2 ++
 6 files changed, 50 insertions(+), 3 deletions(-)

-- 
2.45.0

[PATCH net-next v2 1/2] netdev: Add queue stats for TX stop and wake

[Daniel Jurgens]

TX queue stop and wake are counted by some drivers.
Support reporting these via netdev-genl queue stats.

Signed-off-by: Daniel Jurgens <danielj@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
---
 Documentation/netlink/specs/netdev.yaml | 14 ++++++++++++++
 include/net/netdev_queues.h             |  3 +++
 include/uapi/linux/netdev.h             |  2 ++
 net/core/netdev-genl.c                  |  4 +++-
 tools/include/uapi/linux/netdev.h       |  2 ++
 5 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlink/specs/netdev.yaml
index 2be4b3714d17..11a32373365a 100644
--- a/Documentation/netlink/specs/netdev.yaml
+++ b/Documentation/netlink/specs/netdev.yaml
@@ -439,6 +439,20 @@ attribute-sets:
           Number of the packets dropped by the device due to the transmit
           packets bitrate exceeding the device rate limit.
         type: uint
+      -
+        name: tx-stop
+        doc: |
+          Number of times driver paused accepting new tx packets
+          from the stack to this queue, because the queue was full.
+          Note that if BQL is supported and enabled on the device
+          the networking stack will avoid queuing a lot of data at once.
+        type: uint
+      -
+        name: tx-wake
+        doc: |
+          Number of times driver re-started accepting send
+          requests to this queue from the stack.
+        type: uint
 
 operations:
   list:
diff --git a/include/net/netdev_queues.h b/include/net/netdev_queues.h
index e7b84f018cee..a8a7e48dfa6c 100644
--- a/include/net/netdev_queues.h
+++ b/include/net/netdev_queues.h
@@ -41,6 +41,9 @@ struct netdev_queue_stats_tx {
 	u64 hw_gso_wire_bytes;
 
 	u64 hw_drop_ratelimits;
+
+	u64 stop;
+	u64 wake;
 };
 
 /**
diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h
index cf24f1d9adf8..a8188202413e 100644
--- a/include/uapi/linux/netdev.h
+++ b/include/uapi/linux/netdev.h
@@ -165,6 +165,8 @@ enum {
 	NETDEV_A_QSTATS_TX_HW_GSO_WIRE_PACKETS,
 	NETDEV_A_QSTATS_TX_HW_GSO_WIRE_BYTES,
 	NETDEV_A_QSTATS_TX_HW_DROP_RATELIMITS,
+	NETDEV_A_QSTATS_TX_STOP,
+	NETDEV_A_QSTATS_TX_WAKE,
 
 	__NETDEV_A_QSTATS_MAX,
 	NETDEV_A_QSTATS_MAX = (__NETDEV_A_QSTATS_MAX - 1)
diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c
index 4b5054087309..1f6ae6379e0f 100644
--- a/net/core/netdev-genl.c
+++ b/net/core/netdev-genl.c
@@ -517,7 +517,9 @@ netdev_nl_stats_write_tx(struct sk_buff *rsp, struct netdev_queue_stats_tx *tx)
 	    netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_HW_GSO_BYTES, tx->hw_gso_bytes) ||
 	    netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_HW_GSO_WIRE_PACKETS, tx->hw_gso_wire_packets) ||
 	    netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_HW_GSO_WIRE_BYTES, tx->hw_gso_wire_bytes) ||
-	    netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_HW_DROP_RATELIMITS, tx->hw_drop_ratelimits))
+	    netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_HW_DROP_RATELIMITS, tx->hw_drop_ratelimits) ||
+	    netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_STOP, tx->stop) ||
+	    netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_WAKE, tx->wake))
 		return -EMSGSIZE;
 	return 0;
 }
diff --git a/tools/include/uapi/linux/netdev.h b/tools/include/uapi/linux/netdev.h
index cf24f1d9adf8..a8188202413e 100644
--- a/tools/include/uapi/linux/netdev.h
+++ b/tools/include/uapi/linux/netdev.h
@@ -165,6 +165,8 @@ enum {
 	NETDEV_A_QSTATS_TX_HW_GSO_WIRE_PACKETS,
 	NETDEV_A_QSTATS_TX_HW_GSO_WIRE_BYTES,
 	NETDEV_A_QSTATS_TX_HW_DROP_RATELIMITS,
+	NETDEV_A_QSTATS_TX_STOP,
+	NETDEV_A_QSTATS_TX_WAKE,
 
 	__NETDEV_A_QSTATS_MAX,
 	NETDEV_A_QSTATS_MAX = (__NETDEV_A_QSTATS_MAX - 1)
-- 
2.45.0

[PATCH net-next v2 2/2] virtio_net: Add TX stopped and wake counters

[Daniel Jurgens]

Add a tx queue stop and wake counters, they are useful for debugging.

$ ./tools/net/ynl/cli.py --spec netlink/specs/netdev.yaml \
--dump qstats-get --json '{"scope": "queue"}'
...
 {'ifindex': 13,
  'queue-id': 0,
  'queue-type': 'tx',
  'tx-bytes': 14756682850,
  'tx-packets': 226465,
  'tx-stop': 113208,
  'tx-wake': 113208},
 {'ifindex': 13,
  'queue-id': 1,
  'queue-type': 'tx',
  'tx-bytes': 18167675008,
  'tx-packets': 278660,
  'tx-stop': 8632,
  'tx-wake': 8632}]

Signed-off-by: Daniel Jurgens <danielj@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
---
 drivers/net/virtio_net.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 218a446c4c27..df6121c38a1b 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -95,6 +95,8 @@ struct virtnet_sq_stats {
 	u64_stats_t xdp_tx_drops;
 	u64_stats_t kicks;
 	u64_stats_t tx_timeouts;
+	u64_stats_t stop;
+	u64_stats_t wake;
 };
 
 struct virtnet_rq_stats {
@@ -145,6 +147,8 @@ static const struct virtnet_stat_desc virtnet_rq_stats_desc[] = {
 static const struct virtnet_stat_desc virtnet_sq_stats_desc_qstat[] = {
 	VIRTNET_SQ_STAT_QSTAT("packets", packets),
 	VIRTNET_SQ_STAT_QSTAT("bytes",   bytes),
+	VIRTNET_SQ_STAT_QSTAT("stop",	 stop),
+	VIRTNET_SQ_STAT_QSTAT("wake",	 wake),
 };
 
 static const struct virtnet_stat_desc virtnet_rq_stats_desc_qstat[] = {
@@ -1014,6 +1018,9 @@ static void check_sq_full_and_disable(struct virtnet_info *vi,
 	 */
 	if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
 		netif_stop_subqueue(dev, qnum);
+		u64_stats_update_begin(&sq->stats.syncp);
+		u64_stats_inc(&sq->stats.stop);
+		u64_stats_update_end(&sq->stats.syncp);
 		if (use_napi) {
 			if (unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
 				virtqueue_napi_schedule(&sq->napi, sq->vq);
@@ -1022,6 +1029,9 @@ static void check_sq_full_and_disable(struct virtnet_info *vi,
 			free_old_xmit(sq, false);
 			if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) {
 				netif_start_subqueue(dev, qnum);
+				u64_stats_update_begin(&sq->stats.syncp);
+				u64_stats_inc(&sq->stats.wake);
+				u64_stats_update_end(&sq->stats.syncp);
 				virtqueue_disable_cb(sq->vq);
 			}
 		}
@@ -2322,8 +2332,14 @@ static void virtnet_poll_cleantx(struct receive_queue *rq)
 			free_old_xmit(sq, true);
 		} while (unlikely(!virtqueue_enable_cb_delayed(sq->vq)));
 
-		if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS)
+		if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS) {
+			if (netif_tx_queue_stopped(txq)) {
+				u64_stats_update_begin(&sq->stats.syncp);
+				u64_stats_inc(&sq->stats.wake);
+				u64_stats_update_end(&sq->stats.syncp);
+			}
 			netif_tx_wake_queue(txq);
+		}
 
 		__netif_tx_unlock(txq);
 	}
@@ -2473,8 +2489,14 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget)
 	virtqueue_disable_cb(sq->vq);
 	free_old_xmit(sq, true);
 
-	if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS)
+	if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS) {
+		if (netif_tx_queue_stopped(txq)) {
+			u64_stats_update_begin(&sq->stats.syncp);
+			u64_stats_inc(&sq->stats.wake);
+			u64_stats_update_end(&sq->stats.syncp);
+		}
 		netif_tx_wake_queue(txq);
+	}
 
 	opaque = virtqueue_enable_cb_prepare(sq->vq);
 
@@ -4790,6 +4812,8 @@ static void virtnet_get_base_stats(struct net_device *dev,
 
 	tx->bytes = 0;
 	tx->packets = 0;
+	tx->stop = 0;
+	tx->wake = 0;
 
 	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_BASIC) {
 		tx->hw_drops = 0;
-- 
2.45.0

Re: [PATCH net-next v2 1/2] netdev: Add queue stats for TX stop and wake

[Jason Xing]

On Sat, May 11, 2024 at 4:20 AM Daniel Jurgens <danielj@nvidia.com> wrote:
>
> TX queue stop and wake are counted by some drivers.
> Support reporting these via netdev-genl queue stats.
>
> Signed-off-by: Daniel Jurgens <danielj@nvidia.com>
> Reviewed-by: Jiri Pirko <jiri@nvidia.com>

Good to see this finally :)

Reviewed-by: Jason Xing <kerneljasonxing@gmail.com>

> ---
>  Documentation/netlink/specs/netdev.yaml | 14 ++++++++++++++
>  include/net/netdev_queues.h             |  3 +++
>  include/uapi/linux/netdev.h             |  2 ++
>  net/core/netdev-genl.c                  |  4 +++-
>  tools/include/uapi/linux/netdev.h       |  2 ++
>  5 files changed, 24 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlink/specs/netdev.yaml
> index 2be4b3714d17..11a32373365a 100644
> --- a/Documentation/netlink/specs/netdev.yaml
> +++ b/Documentation/netlink/specs/netdev.yaml
> @@ -439,6 +439,20 @@ attribute-sets:
>            Number of the packets dropped by the device due to the transmit
>            packets bitrate exceeding the device rate limit.
>          type: uint
> +      -
> +        name: tx-stop
> +        doc: |
> +          Number of times driver paused accepting new tx packets
> +          from the stack to this queue, because the queue was full.
> +          Note that if BQL is supported and enabled on the device
> +          the networking stack will avoid queuing a lot of data at once.
> +        type: uint
> +      -
> +        name: tx-wake
> +        doc: |
> +          Number of times driver re-started accepting send
> +          requests to this queue from the stack.
> +        type: uint
>
>  operations:
>    list:
> diff --git a/include/net/netdev_queues.h b/include/net/netdev_queues.h
> index e7b84f018cee..a8a7e48dfa6c 100644
> --- a/include/net/netdev_queues.h
> +++ b/include/net/netdev_queues.h
> @@ -41,6 +41,9 @@ struct netdev_queue_stats_tx {
>         u64 hw_gso_wire_bytes;
>
>         u64 hw_drop_ratelimits;
> +
> +       u64 stop;
> +       u64 wake;
>  };
>
>  /**
> diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h
> index cf24f1d9adf8..a8188202413e 100644
> --- a/include/uapi/linux/netdev.h
> +++ b/include/uapi/linux/netdev.h
> @@ -165,6 +165,8 @@ enum {
>         NETDEV_A_QSTATS_TX_HW_GSO_WIRE_PACKETS,
>         NETDEV_A_QSTATS_TX_HW_GSO_WIRE_BYTES,
>         NETDEV_A_QSTATS_TX_HW_DROP_RATELIMITS,
> +       NETDEV_A_QSTATS_TX_STOP,
> +       NETDEV_A_QSTATS_TX_WAKE,
>
>         __NETDEV_A_QSTATS_MAX,
>         NETDEV_A_QSTATS_MAX = (__NETDEV_A_QSTATS_MAX - 1)
> diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c
> index 4b5054087309..1f6ae6379e0f 100644
> --- a/net/core/netdev-genl.c
> +++ b/net/core/netdev-genl.c
> @@ -517,7 +517,9 @@ netdev_nl_stats_write_tx(struct sk_buff *rsp, struct netdev_queue_stats_tx *tx)
>             netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_HW_GSO_BYTES, tx->hw_gso_bytes) ||
>             netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_HW_GSO_WIRE_PACKETS, tx->hw_gso_wire_packets) ||
>             netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_HW_GSO_WIRE_BYTES, tx->hw_gso_wire_bytes) ||
> -           netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_HW_DROP_RATELIMITS, tx->hw_drop_ratelimits))
> +           netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_HW_DROP_RATELIMITS, tx->hw_drop_ratelimits) ||
> +           netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_STOP, tx->stop) ||
> +           netdev_stat_put(rsp, NETDEV_A_QSTATS_TX_WAKE, tx->wake))
>                 return -EMSGSIZE;
>         return 0;
>  }
> diff --git a/tools/include/uapi/linux/netdev.h b/tools/include/uapi/linux/netdev.h
> index cf24f1d9adf8..a8188202413e 100644
> --- a/tools/include/uapi/linux/netdev.h
> +++ b/tools/include/uapi/linux/netdev.h
> @@ -165,6 +165,8 @@ enum {
>         NETDEV_A_QSTATS_TX_HW_GSO_WIRE_PACKETS,
>         NETDEV_A_QSTATS_TX_HW_GSO_WIRE_BYTES,
>         NETDEV_A_QSTATS_TX_HW_DROP_RATELIMITS,
> +       NETDEV_A_QSTATS_TX_STOP,
> +       NETDEV_A_QSTATS_TX_WAKE,
>
>         __NETDEV_A_QSTATS_MAX,
>         NETDEV_A_QSTATS_MAX = (__NETDEV_A_QSTATS_MAX - 1)
> --
> 2.45.0
>
>

Re: [PATCH net-next v2 2/2] virtio_net: Add TX stopped and wake counters

[Jason Xing]

On Sat, May 11, 2024 at 4:20 AM Daniel Jurgens <danielj@nvidia.com> wrote:
>
> Add a tx queue stop and wake counters, they are useful for debugging.
>
> $ ./tools/net/ynl/cli.py --spec netlink/specs/netdev.yaml \
> --dump qstats-get --json '{"scope": "queue"}'
> ...
>  {'ifindex': 13,
>   'queue-id': 0,
>   'queue-type': 'tx',
>   'tx-bytes': 14756682850,
>   'tx-packets': 226465,
>   'tx-stop': 113208,
>   'tx-wake': 113208},
>  {'ifindex': 13,
>   'queue-id': 1,
>   'queue-type': 'tx',
>   'tx-bytes': 18167675008,
>   'tx-packets': 278660,
>   'tx-stop': 8632,
>   'tx-wake': 8632}]
>
> Signed-off-by: Daniel Jurgens <danielj@nvidia.com>
> Reviewed-by: Jiri Pirko <jiri@nvidia.com>

Reviewed-by: Jason Xing <kerneljasonxing@gmail.com>

Thanks!

> ---
>  drivers/net/virtio_net.c | 28 ++++++++++++++++++++++++++--
>  1 file changed, 26 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 218a446c4c27..df6121c38a1b 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -95,6 +95,8 @@ struct virtnet_sq_stats {
>         u64_stats_t xdp_tx_drops;
>         u64_stats_t kicks;
>         u64_stats_t tx_timeouts;
> +       u64_stats_t stop;
> +       u64_stats_t wake;
>  };
>
>  struct virtnet_rq_stats {
> @@ -145,6 +147,8 @@ static const struct virtnet_stat_desc virtnet_rq_stats_desc[] = {
>  static const struct virtnet_stat_desc virtnet_sq_stats_desc_qstat[] = {
>         VIRTNET_SQ_STAT_QSTAT("packets", packets),
>         VIRTNET_SQ_STAT_QSTAT("bytes",   bytes),
> +       VIRTNET_SQ_STAT_QSTAT("stop",    stop),
> +       VIRTNET_SQ_STAT_QSTAT("wake",    wake),
>  };
>
>  static const struct virtnet_stat_desc virtnet_rq_stats_desc_qstat[] = {
> @@ -1014,6 +1018,9 @@ static void check_sq_full_and_disable(struct virtnet_info *vi,
>          */
>         if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
>                 netif_stop_subqueue(dev, qnum);
> +               u64_stats_update_begin(&sq->stats.syncp);
> +               u64_stats_inc(&sq->stats.stop);
> +               u64_stats_update_end(&sq->stats.syncp);
>                 if (use_napi) {
>                         if (unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
>                                 virtqueue_napi_schedule(&sq->napi, sq->vq);
> @@ -1022,6 +1029,9 @@ static void check_sq_full_and_disable(struct virtnet_info *vi,
>                         free_old_xmit(sq, false);
>                         if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) {
>                                 netif_start_subqueue(dev, qnum);
> +                               u64_stats_update_begin(&sq->stats.syncp);
> +                               u64_stats_inc(&sq->stats.wake);
> +                               u64_stats_update_end(&sq->stats.syncp);
>                                 virtqueue_disable_cb(sq->vq);
>                         }
>                 }
> @@ -2322,8 +2332,14 @@ static void virtnet_poll_cleantx(struct receive_queue *rq)
>                         free_old_xmit(sq, true);
>                 } while (unlikely(!virtqueue_enable_cb_delayed(sq->vq)));
>
> -               if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS)
> +               if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS) {
> +                       if (netif_tx_queue_stopped(txq)) {
> +                               u64_stats_update_begin(&sq->stats.syncp);
> +                               u64_stats_inc(&sq->stats.wake);
> +                               u64_stats_update_end(&sq->stats.syncp);
> +                       }
>                         netif_tx_wake_queue(txq);
> +               }
>
>                 __netif_tx_unlock(txq);
>         }
> @@ -2473,8 +2489,14 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget)
>         virtqueue_disable_cb(sq->vq);
>         free_old_xmit(sq, true);
>
> -       if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS)
> +       if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS) {
> +               if (netif_tx_queue_stopped(txq)) {
> +                       u64_stats_update_begin(&sq->stats.syncp);
> +                       u64_stats_inc(&sq->stats.wake);
> +                       u64_stats_update_end(&sq->stats.syncp);
> +               }
>                 netif_tx_wake_queue(txq);
> +       }
>
>         opaque = virtqueue_enable_cb_prepare(sq->vq);
>
> @@ -4790,6 +4812,8 @@ static void virtnet_get_base_stats(struct net_device *dev,
>
>         tx->bytes = 0;
>         tx->packets = 0;
> +       tx->stop = 0;
> +       tx->wake = 0;
>
>         if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_BASIC) {
>                 tx->hw_drops = 0;
> --
> 2.45.0
>
>

Re: [PATCH net-next 0/2] Add TX stop/wake counters

[patchwork-bot+netdevbpf]

Hello:

This series was applied to netdev/net-next.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Fri, 10 May 2024 23:19:25 +0300 you wrote:
> Several drivers provide TX stop and wake counters via ethtool stats. Add
> those to the netdev queue stats, and use them in virtio_net.
> 
> v2:
> 	- Fixed an accidental line deletion
> 	- Enhanced documentation
> 
> [...]

Here is the summary with links:
  - [net-next,v2,1/2] netdev: Add queue stats for TX stop and wake
    https://git.kernel.org/netdev/net-next/c/b56035101e1c
  - [net-next,v2,2/2] virtio_net: Add TX stopped and wake counters
    https://git.kernel.org/netdev/net-next/c/c39add9b2423

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html