[PATCH] net/tun: expose queue utilization stats via ethtool

[PATCH] net/tun: expose queue utilization stats via ethtool

[Alexander Shalimov]

TUN/TAP devices are heavily used in network virtualization scenarios
such as QEMU/KVM with "-netdev tap" and are commonly paired with virtio-net
or vhost-net backends. Under high network load, queues of the tuntap device
may become saturated, resulting in TX drops.

Existing aggregated drop counters alone are often insufficient during
complex debugging and performance tuning, especially in high-throughput
environments. Visibility of real-time queue utilization is critical for
understanding why guest VMs might be unable to dequeue packets in time.

This patch exposes per-queue utilization statistics via ethtool -S,
allowing on-demand inspection of queue fill levels. Utilization metrics are
captured at the time of the ethtool invocation, providing a snapshot useful
for correlation with guest and host behavior.

Signed-off-by: Alexander Shalimov <alex-shalimov@yandex-team.ru>
---
 drivers/net/tun.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 7babd1e9a378..122327e591a5 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -3537,6 +3537,57 @@ static void tun_get_channels(struct net_device *dev,
 	channels->max_combined = tun->flags & IFF_MULTI_QUEUE ? MAX_TAP_QUEUES : 1;
 }
 
+static void tun_get_strings(struct net_device *dev, u32 stringset, u8 *buf)
+{
+	char *p = (char *)buf;
+	int i;
+
+	switch (stringset) {
+	case ETH_SS_STATS:
+		for (i = 0; i < dev->real_num_tx_queues; i++) {
+			snprintf(p, ETH_GSTRING_LEN, "tx_queue_usage_%u", i);
+			p += ETH_GSTRING_LEN;
+		}
+		break;
+	}
+}
+
+static int tun_get_sset_count(struct net_device *dev, int sset)
+{
+	switch (sset) {
+	case ETH_SS_STATS:
+		return dev->real_num_tx_queues;
+	default:
+		return -EOPNOTSUPP;
+	}
+}
+
+static void tun_get_ethtool_stats(struct net_device *dev,
+				  struct ethtool_stats *stats, u64 *data)
+{
+	struct tun_struct *tun = netdev_priv(dev);
+	struct tun_file *tfile;
+	int i;
+	int producer, consumer, size, usage;
+
+	rcu_read_lock();
+	for (i = 0; i < dev->real_num_tx_queues; i++) {
+		tfile = rcu_dereference(tun->tfiles[i]);
+
+		producer = READ_ONCE(tfile->tx_ring.producer);
+		consumer = READ_ONCE(tfile->tx_ring.consumer_head);
+		size = READ_ONCE(tfile->tx_ring.size);
+
+		if (producer >= consumer)
+			usage = producer - consumer;
+		else
+			usage = size - (consumer - producer);
+
+		data[i] = usage;
+	}
+	rcu_read_unlock();
+}
+
 static const struct ethtool_ops tun_ethtool_ops = {
 	.supported_coalesce_params = ETHTOOL_COALESCE_RX_MAX_FRAMES,
 	.get_drvinfo	= tun_get_drvinfo,
@@ -3549,6 +3600,9 @@ static const struct ethtool_ops tun_ethtool_ops = {
 	.set_coalesce   = tun_set_coalesce,
 	.get_link_ksettings = tun_get_link_ksettings,
 	.set_link_ksettings = tun_set_link_ksettings,
+	.get_strings	    = tun_get_strings,
+	.get_sset_count	    = tun_get_sset_count,
+	.get_ethtool_stats  = tun_get_ethtool_stats,
 };
 
 static int tun_queue_resize(struct tun_struct *tun)
-- 
2.39.5 (Apple Git-154)

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Andrew Lunn]

On Tue, May 06, 2025 at 06:41:17PM +0300, Alexander Shalimov wrote:
> TUN/TAP devices are heavily used in network virtualization scenarios
> such as QEMU/KVM with "-netdev tap" and are commonly paired with virtio-net
> or vhost-net backends. Under high network load, queues of the tuntap device
> may become saturated, resulting in TX drops.
> 
> Existing aggregated drop counters alone are often insufficient during
> complex debugging and performance tuning, especially in high-throughput
> environments. Visibility of real-time queue utilization is critical for
> understanding why guest VMs might be unable to dequeue packets in time.
> 
> This patch exposes per-queue utilization statistics via ethtool -S,
> allowing on-demand inspection of queue fill levels. Utilization metrics are
> captured at the time of the ethtool invocation, providing a snapshot useful
> for correlation with guest and host behavior.

This does not fit the usual statistics pattern, which are simple
incremental counters. Are there any other drivers doing anything like
this?

Maybe devlink resources would be a better API? That would also allow
you to report the size of the ring.

> +static void tun_get_ethtool_stats(struct net_device *dev,
> +				  struct ethtool_stats *stats, u64 *data)
> +{
> +	struct tun_struct *tun = netdev_priv(dev);
> +	struct tun_file *tfile;
> +	int i;
> +	int producer, consumer, size, usage;
> +
> +	rcu_read_lock();
> +	for (i = 0; i < dev->real_num_tx_queues; i++) {
> +		tfile = rcu_dereference(tun->tfiles[i]);
> +
> +		producer = READ_ONCE(tfile->tx_ring.producer);
> +		consumer = READ_ONCE(tfile->tx_ring.consumer_head);
> +		size = READ_ONCE(tfile->tx_ring.size);
> +
> +		if (producer >= consumer)
> +			usage = producer - consumer;
> +		else
> +			usage = size - (consumer - producer);

It seems like this belongs in ptr_ring.h along with all the other
methods which deal with insides of the ring.

	Andrew

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Jacob Keller]

On 5/6/2025 9:07 AM, Andrew Lunn wrote:
> On Tue, May 06, 2025 at 06:41:17PM +0300, Alexander Shalimov wrote:
>> This patch exposes per-queue utilization statistics via ethtool -S,
>> allowing on-demand inspection of queue fill levels. Utilization metrics are
>> captured at the time of the ethtool invocation, providing a snapshot useful
>> for correlation with guest and host behavior.
> 
> This does not fit the usual statistics pattern, which are simple
> incremental counters. Are there any other drivers doing anything like
> this?
I don't recall ever seeing anything like this. If there are, I feel it
is a mistake regardless, and we shouldn't repeat it without good reason.

+1 to looking at another option for reporting.

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Willem de Bruijn]

Jacob Keller wrote:
> 
> 
> On 5/6/2025 9:07 AM, Andrew Lunn wrote:
> > On Tue, May 06, 2025 at 06:41:17PM +0300, Alexander Shalimov wrote:
> >> This patch exposes per-queue utilization statistics via ethtool -S,
> >> allowing on-demand inspection of queue fill levels. Utilization metrics are
> >> captured at the time of the ethtool invocation, providing a snapshot useful
> >> for correlation with guest and host behavior.
> > 
> > This does not fit the usual statistics pattern, which are simple
> > incremental counters. Are there any other drivers doing anything like
> > this?
> I don't recall ever seeing anything like this. If there are, I feel it
> is a mistake regardless, and we shouldn't repeat it without good reason.
> 
> +1 to looking at another option for reporting.

Perhaps bpftrace with a kfunc at a suitable function entry point to
get access to these ring structures.

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Alexander Shalimov]

06.05.2025, 22:32, "Willem de Bruijn" <willemdebruijn.kernel@gmail.com>:
> Perhaps bpftrace with a kfunc at a suitable function entry point to
> get access to these ring structures.

Thank you for your responses!

Initially, we implemented such monitoring using bpftrace but we were
not satisfied with the need to double-check the structure definitions
in tun.c for each new kernel version.

We attached kprobe to the "tun_net_xmit()" function. This function
gets a "struct net_device" as an argument, which is then explicitly
cast to a tun_struct - "struct tun_struct *tun = netdev_priv(dev)".
However, performing such a cast within bpftrace is difficult because
tun_struct is defined in tun.c - meaning the structure definition
cannot be included directly (not a header file). As a result, we were
forced to add fake "struct tun_struct" and "struct tun_file"
definitions, whose maintenance across kernel versions became
cumbersome (see below). The same problems exists even with kfunc and
btf - we are not able to cast properly netdev to tun_struct.

That’s why we decided to add this functionality directly to the kernel.

Here is an example of bpftrace:

#define NET_DEVICE_TUN_OFFSET 0x900

struct tun_net_device {
    unsigned char padding[NET_DEVICE_TUN_OFFSET]; #such calculation is pain
    struct tun_struct tun;
}

kprobe:tun_net_xmit {
    $skb = (struct sk_buff*) arg0;
    $netdev = $skb->dev;
    $tun_dev = (struct tun_net_device *)arg1;
    $tun = $tun_dev->tun;
   ....
}

Could you please recommend the right way to implement such bpftrace script?
Either better place in kernel for the patch.

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Willem de Bruijn]

Alexander Shalimov wrote:
> 06.05.2025, 22:32, "Willem de Bruijn" <willemdebruijn.kernel@gmail.com>:
> > Perhaps bpftrace with a kfunc at a suitable function entry point to
> > get access to these ring structures.
> 
> Thank you for your responses!
> 
> Initially, we implemented such monitoring using bpftrace but we were
> not satisfied with the need to double-check the structure definitions
> in tun.c for each new kernel version.
> 
> We attached kprobe to the "tun_net_xmit()" function. This function
> gets a "struct net_device" as an argument, which is then explicitly
> cast to a tun_struct - "struct tun_struct *tun = netdev_priv(dev)".
> However, performing such a cast within bpftrace is difficult because
> tun_struct is defined in tun.c - meaning the structure definition
> cannot be included directly (not a header file). As a result, we were
> forced to add fake "struct tun_struct" and "struct tun_file"
> definitions, whose maintenance across kernel versions became
> cumbersome (see below). The same problems exists even with kfunc and
> btf - we are not able to cast properly netdev to tun_struct.
> 
> That’s why we decided to add this functionality directly to the kernel.

Let's solve this in bpftrace instead. That's no reason to rever to
hardcoded kernel APIs.

It quite possibly already is. I'm no bpftrace expert. Cc:ing bpf@

There seem to be two parts:

The field lookup in struct tun_struct. This should be captured by BTF:

	$ bpftool btf dump file /sys/kernel/btf/vmlinux | grep tun_struct | wc -l
        1

The cast from netdev_priv to struct tun_struct. Note that in recent
kernels netdev_priv is just args->dev->priv. No need for this manual
struct tun_net_device.

> 
> Here is an example of bpftrace:
> 
> #define NET_DEVICE_TUN_OFFSET 0x900
> 
> struct tun_net_device {
>     unsigned char padding[NET_DEVICE_TUN_OFFSET]; #such calculation is pain
>     struct tun_struct tun;
> }
> 
> kprobe:tun_net_xmit {
>     $skb = (struct sk_buff*) arg0;
>     $netdev = $skb->dev;
>     $tun_dev = (struct tun_net_device *)arg1;
>     $tun = $tun_dev->tun;
>    ....
> }
> 
> Could you please recommend the right way to implement such bpftrace script?
> Either better place in kernel for the patch.

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Daniel Xu]

On Thu, May 15, 2025, at 7:12 AM, Willem de Bruijn wrote:
> Alexander Shalimov wrote:
>> 06.05.2025, 22:32, "Willem de Bruijn" <willemdebruijn.kernel@gmail.com>:
>> > Perhaps bpftrace with a kfunc at a suitable function entry point to
>> > get access to these ring structures.
>> 
>> Thank you for your responses!
>> 
>> Initially, we implemented such monitoring using bpftrace but we were
>> not satisfied with the need to double-check the structure definitions
>> in tun.c for each new kernel version.
>> 
>> We attached kprobe to the "tun_net_xmit()" function. This function
>> gets a "struct net_device" as an argument, which is then explicitly
>> cast to a tun_struct - "struct tun_struct *tun = netdev_priv(dev)".
>> However, performing such a cast within bpftrace is difficult because
>> tun_struct is defined in tun.c - meaning the structure definition
>> cannot be included directly (not a header file). As a result, we were
>> forced to add fake "struct tun_struct" and "struct tun_file"
>> definitions, whose maintenance across kernel versions became
>> cumbersome (see below). The same problems exists even with kfunc and
>> btf - we are not able to cast properly netdev to tun_struct.
>> 
>> That’s why we decided to add this functionality directly to the kernel.
>
> Let's solve this in bpftrace instead. That's no reason to rever to
> hardcoded kernel APIs.
>
> It quite possibly already is. I'm no bpftrace expert. Cc:ing bpf@

Yeah, should be possible. You haven't needed to include header
files to access type information available in BTF for a while now.
This seems to work for me - mind giving this a try?

```
fentry:tun:tun_net_xmit {
    $tun = (struct tun_struct *)args->dev->priv;
    print($tun->numqueues);  // or whatever else you want
}
```

fentry probes are better in general than kprobes if all you're doing
is attaching to the entry of a function.

You could do the same with kprobes like this if you really want, though:

```
kprobe:tun:tun_net_xmit {
    $dev = (struct net_device *)arg1;
    $tun = (struct tun_struct *)$dev->priv;
    print($tun->numqueues);  // or whatever else you want
}
```

Although it looks like there's a bug when you omit the module name
where bpftrace doesn't find the struct definition. I'll look into that.

Thanks,
Daniel

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Willem de Bruijn]

Daniel Xu wrote:
> On Thu, May 15, 2025, at 7:12 AM, Willem de Bruijn wrote:
> > Alexander Shalimov wrote:
> >> 06.05.2025, 22:32, "Willem de Bruijn" <willemdebruijn.kernel@gmail.com>:
> >> > Perhaps bpftrace with a kfunc at a suitable function entry point to
> >> > get access to these ring structures.
> >> 
> >> Thank you for your responses!
> >> 
> >> Initially, we implemented such monitoring using bpftrace but we were
> >> not satisfied with the need to double-check the structure definitions
> >> in tun.c for each new kernel version.
> >> 
> >> We attached kprobe to the "tun_net_xmit()" function. This function
> >> gets a "struct net_device" as an argument, which is then explicitly
> >> cast to a tun_struct - "struct tun_struct *tun = netdev_priv(dev)".
> >> However, performing such a cast within bpftrace is difficult because
> >> tun_struct is defined in tun.c - meaning the structure definition
> >> cannot be included directly (not a header file). As a result, we were
> >> forced to add fake "struct tun_struct" and "struct tun_file"
> >> definitions, whose maintenance across kernel versions became
> >> cumbersome (see below). The same problems exists even with kfunc and
> >> btf - we are not able to cast properly netdev to tun_struct.
> >> 
> >> That’s why we decided to add this functionality directly to the kernel.
> >
> > Let's solve this in bpftrace instead. That's no reason to rever to
> > hardcoded kernel APIs.
> >
> > It quite possibly already is. I'm no bpftrace expert. Cc:ing bpf@
> 
> Yeah, should be possible. You haven't needed to include header
> files to access type information available in BTF for a while now.
> This seems to work for me - mind giving this a try?
> 
> ```
> fentry:tun:tun_net_xmit {
>     $tun = (struct tun_struct *)args->dev->priv;
>     print($tun->numqueues);  // or whatever else you want
> }
> ```
> 
> fentry probes are better in general than kprobes if all you're doing
> is attaching to the entry of a function.
> 
> You could do the same with kprobes like this if you really want, though:
> 
> ```
> kprobe:tun:tun_net_xmit {
>     $dev = (struct net_device *)arg1;
>     $tun = (struct tun_struct *)$dev->priv;
>     print($tun->numqueues);  // or whatever else you want
> }
> ```
> 
> Although it looks like there's a bug when you omit the module name
> where bpftrace doesn't find the struct definition. I'll look into that.

Minor: unless tun is built-in.

Thanks a lot for your response, Daniel. Good to know that we can get
this information without kernel changes. And I learned something new
:) Replicated your examples.

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Daniel Xu]

On Fri, May 16, 2025, at 10:22 AM, Willem de Bruijn wrote:
> Daniel Xu wrote:
>> On Thu, May 15, 2025, at 7:12 AM, Willem de Bruijn wrote:
>> > Alexander Shalimov wrote:
>> >> 06.05.2025, 22:32, "Willem de Bruijn" <willemdebruijn.kernel@gmail.com>:
>> >> > Perhaps bpftrace with a kfunc at a suitable function entry point to
>> >> > get access to these ring structures.
>> >> 
>> >> Thank you for your responses!
>> >> 
>> >> Initially, we implemented such monitoring using bpftrace but we were
>> >> not satisfied with the need to double-check the structure definitions
>> >> in tun.c for each new kernel version.
>> >> 
>> >> We attached kprobe to the "tun_net_xmit()" function. This function
>> >> gets a "struct net_device" as an argument, which is then explicitly
>> >> cast to a tun_struct - "struct tun_struct *tun = netdev_priv(dev)".
>> >> However, performing such a cast within bpftrace is difficult because
>> >> tun_struct is defined in tun.c - meaning the structure definition
>> >> cannot be included directly (not a header file). As a result, we were
>> >> forced to add fake "struct tun_struct" and "struct tun_file"
>> >> definitions, whose maintenance across kernel versions became
>> >> cumbersome (see below). The same problems exists even with kfunc and
>> >> btf - we are not able to cast properly netdev to tun_struct.
>> >> 
>> >> That’s why we decided to add this functionality directly to the kernel.
>> >
>> > Let's solve this in bpftrace instead. That's no reason to rever to
>> > hardcoded kernel APIs.
>> >
>> > It quite possibly already is. I'm no bpftrace expert. Cc:ing bpf@
>> 
>> Yeah, should be possible. You haven't needed to include header
>> files to access type information available in BTF for a while now.
>> This seems to work for me - mind giving this a try?
>> 
>> ```
>> fentry:tun:tun_net_xmit {
>>     $tun = (struct tun_struct *)args->dev->priv;
>>     print($tun->numqueues);  // or whatever else you want
>> }
>> ```
>> 
>> fentry probes are better in general than kprobes if all you're doing
>> is attaching to the entry of a function.
>> 
>> You could do the same with kprobes like this if you really want, though:
>> 
>> ```
>> kprobe:tun:tun_net_xmit {
>>     $dev = (struct net_device *)arg1;
>>     $tun = (struct tun_struct *)$dev->priv;
>>     print($tun->numqueues);  // or whatever else you want
>> }
>> ```
>> 
>> Although it looks like there's a bug when you omit the module name
>> where bpftrace doesn't find the struct definition. I'll look into that.
>
> Minor: unless tun is built-in.

Ah, right.

>
> Thanks a lot for your response, Daniel. Good to know that we can get
> this information without kernel changes. And I learned something new
> :) Replicated your examples.

Nice! Feel free to CC me if you have other stuff in the future.

Bug fix for parsing implicit module BTF up here:
https://github.com/bpftrace/bpftrace/pull/4137

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Alexander Shalimov]

>> Thanks a lot for your response, Daniel. Good to know that we can get
>> this information without kernel changes. And I learned something new
>> :) Replicated your examples.
>
> Nice! Feel free to CC me if you have other stuff in the future.
> 
> Bug fix for parsing implicit module BTF up here:
> https://github.com/bpftrace/bpftrace/pull/4137

Daniel and Willem, I appreciate you help!

My mistake was that we had tun configured as a module (CONFIG_TUN=m), and I
didn't explicitly specify the module name in the kprobe. Also, thank you for
pointing out that recent kernels have added the 'priv' field to net_device.
As a result, the script has now become much more universal and simpler.

Now we will think about how to efficiently implement monitoring on top of
our bpftrace script, which dynamically reports queue utilization.