Re: [PATCH] net/tun: expose queue utilization stats via ethtool

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Willem de Bruijn]

Alexander Shalimov wrote:
> 06.05.2025, 22:32, "Willem de Bruijn" <willemdebruijn.kernel@gmail.com>:
> > Perhaps bpftrace with a kfunc at a suitable function entry point to
> > get access to these ring structures.
> 
> Thank you for your responses!
> 
> Initially, we implemented such monitoring using bpftrace but we were
> not satisfied with the need to double-check the structure definitions
> in tun.c for each new kernel version.
> 
> We attached kprobe to the "tun_net_xmit()" function. This function
> gets a "struct net_device" as an argument, which is then explicitly
> cast to a tun_struct - "struct tun_struct *tun = netdev_priv(dev)".
> However, performing such a cast within bpftrace is difficult because
> tun_struct is defined in tun.c - meaning the structure definition
> cannot be included directly (not a header file). As a result, we were
> forced to add fake "struct tun_struct" and "struct tun_file"
> definitions, whose maintenance across kernel versions became
> cumbersome (see below). The same problems exists even with kfunc and
> btf - we are not able to cast properly netdev to tun_struct.
> 
> That’s why we decided to add this functionality directly to the kernel.

Let's solve this in bpftrace instead. That's no reason to rever to
hardcoded kernel APIs.

It quite possibly already is. I'm no bpftrace expert. Cc:ing bpf@

There seem to be two parts:

The field lookup in struct tun_struct. This should be captured by BTF:

	$ bpftool btf dump file /sys/kernel/btf/vmlinux | grep tun_struct | wc -l
        1

The cast from netdev_priv to struct tun_struct. Note that in recent
kernels netdev_priv is just args->dev->priv. No need for this manual
struct tun_net_device.

> 
> Here is an example of bpftrace:
> 
> #define NET_DEVICE_TUN_OFFSET 0x900
> 
> struct tun_net_device {
>     unsigned char padding[NET_DEVICE_TUN_OFFSET]; #such calculation is pain
>     struct tun_struct tun;
> }
> 
> kprobe:tun_net_xmit {
>     $skb = (struct sk_buff*) arg0;
>     $netdev = $skb->dev;
>     $tun_dev = (struct tun_net_device *)arg1;
>     $tun = $tun_dev->tun;
>    ....
> }
> 
> Could you please recommend the right way to implement such bpftrace script?
> Either better place in kernel for the patch.

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Daniel Xu]

On Thu, May 15, 2025, at 7:12 AM, Willem de Bruijn wrote:
> Alexander Shalimov wrote:
>> 06.05.2025, 22:32, "Willem de Bruijn" <willemdebruijn.kernel@gmail.com>:
>> > Perhaps bpftrace with a kfunc at a suitable function entry point to
>> > get access to these ring structures.
>> 
>> Thank you for your responses!
>> 
>> Initially, we implemented such monitoring using bpftrace but we were
>> not satisfied with the need to double-check the structure definitions
>> in tun.c for each new kernel version.
>> 
>> We attached kprobe to the "tun_net_xmit()" function. This function
>> gets a "struct net_device" as an argument, which is then explicitly
>> cast to a tun_struct - "struct tun_struct *tun = netdev_priv(dev)".
>> However, performing such a cast within bpftrace is difficult because
>> tun_struct is defined in tun.c - meaning the structure definition
>> cannot be included directly (not a header file). As a result, we were
>> forced to add fake "struct tun_struct" and "struct tun_file"
>> definitions, whose maintenance across kernel versions became
>> cumbersome (see below). The same problems exists even with kfunc and
>> btf - we are not able to cast properly netdev to tun_struct.
>> 
>> That’s why we decided to add this functionality directly to the kernel.
>
> Let's solve this in bpftrace instead. That's no reason to rever to
> hardcoded kernel APIs.
>
> It quite possibly already is. I'm no bpftrace expert. Cc:ing bpf@

Yeah, should be possible. You haven't needed to include header
files to access type information available in BTF for a while now.
This seems to work for me - mind giving this a try?

```
fentry:tun:tun_net_xmit {
    $tun = (struct tun_struct *)args->dev->priv;
    print($tun->numqueues);  // or whatever else you want
}
```

fentry probes are better in general than kprobes if all you're doing
is attaching to the entry of a function.

You could do the same with kprobes like this if you really want, though:

```
kprobe:tun:tun_net_xmit {
    $dev = (struct net_device *)arg1;
    $tun = (struct tun_struct *)$dev->priv;
    print($tun->numqueues);  // or whatever else you want
}
```

Although it looks like there's a bug when you omit the module name
where bpftrace doesn't find the struct definition. I'll look into that.

Thanks,
Daniel

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Willem de Bruijn]

Daniel Xu wrote:
> On Thu, May 15, 2025, at 7:12 AM, Willem de Bruijn wrote:
> > Alexander Shalimov wrote:
> >> 06.05.2025, 22:32, "Willem de Bruijn" <willemdebruijn.kernel@gmail.com>:
> >> > Perhaps bpftrace with a kfunc at a suitable function entry point to
> >> > get access to these ring structures.
> >> 
> >> Thank you for your responses!
> >> 
> >> Initially, we implemented such monitoring using bpftrace but we were
> >> not satisfied with the need to double-check the structure definitions
> >> in tun.c for each new kernel version.
> >> 
> >> We attached kprobe to the "tun_net_xmit()" function. This function
> >> gets a "struct net_device" as an argument, which is then explicitly
> >> cast to a tun_struct - "struct tun_struct *tun = netdev_priv(dev)".
> >> However, performing such a cast within bpftrace is difficult because
> >> tun_struct is defined in tun.c - meaning the structure definition
> >> cannot be included directly (not a header file). As a result, we were
> >> forced to add fake "struct tun_struct" and "struct tun_file"
> >> definitions, whose maintenance across kernel versions became
> >> cumbersome (see below). The same problems exists even with kfunc and
> >> btf - we are not able to cast properly netdev to tun_struct.
> >> 
> >> That’s why we decided to add this functionality directly to the kernel.
> >
> > Let's solve this in bpftrace instead. That's no reason to rever to
> > hardcoded kernel APIs.
> >
> > It quite possibly already is. I'm no bpftrace expert. Cc:ing bpf@
> 
> Yeah, should be possible. You haven't needed to include header
> files to access type information available in BTF for a while now.
> This seems to work for me - mind giving this a try?
> 
> ```
> fentry:tun:tun_net_xmit {
>     $tun = (struct tun_struct *)args->dev->priv;
>     print($tun->numqueues);  // or whatever else you want
> }
> ```
> 
> fentry probes are better in general than kprobes if all you're doing
> is attaching to the entry of a function.
> 
> You could do the same with kprobes like this if you really want, though:
> 
> ```
> kprobe:tun:tun_net_xmit {
>     $dev = (struct net_device *)arg1;
>     $tun = (struct tun_struct *)$dev->priv;
>     print($tun->numqueues);  // or whatever else you want
> }
> ```
> 
> Although it looks like there's a bug when you omit the module name
> where bpftrace doesn't find the struct definition. I'll look into that.

Minor: unless tun is built-in.

Thanks a lot for your response, Daniel. Good to know that we can get
this information without kernel changes. And I learned something new
:) Replicated your examples.

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Daniel Xu]

On Fri, May 16, 2025, at 10:22 AM, Willem de Bruijn wrote:
> Daniel Xu wrote:
>> On Thu, May 15, 2025, at 7:12 AM, Willem de Bruijn wrote:
>> > Alexander Shalimov wrote:
>> >> 06.05.2025, 22:32, "Willem de Bruijn" <willemdebruijn.kernel@gmail.com>:
>> >> > Perhaps bpftrace with a kfunc at a suitable function entry point to
>> >> > get access to these ring structures.
>> >> 
>> >> Thank you for your responses!
>> >> 
>> >> Initially, we implemented such monitoring using bpftrace but we were
>> >> not satisfied with the need to double-check the structure definitions
>> >> in tun.c for each new kernel version.
>> >> 
>> >> We attached kprobe to the "tun_net_xmit()" function. This function
>> >> gets a "struct net_device" as an argument, which is then explicitly
>> >> cast to a tun_struct - "struct tun_struct *tun = netdev_priv(dev)".
>> >> However, performing such a cast within bpftrace is difficult because
>> >> tun_struct is defined in tun.c - meaning the structure definition
>> >> cannot be included directly (not a header file). As a result, we were
>> >> forced to add fake "struct tun_struct" and "struct tun_file"
>> >> definitions, whose maintenance across kernel versions became
>> >> cumbersome (see below). The same problems exists even with kfunc and
>> >> btf - we are not able to cast properly netdev to tun_struct.
>> >> 
>> >> That’s why we decided to add this functionality directly to the kernel.
>> >
>> > Let's solve this in bpftrace instead. That's no reason to rever to
>> > hardcoded kernel APIs.
>> >
>> > It quite possibly already is. I'm no bpftrace expert. Cc:ing bpf@
>> 
>> Yeah, should be possible. You haven't needed to include header
>> files to access type information available in BTF for a while now.
>> This seems to work for me - mind giving this a try?
>> 
>> ```
>> fentry:tun:tun_net_xmit {
>>     $tun = (struct tun_struct *)args->dev->priv;
>>     print($tun->numqueues);  // or whatever else you want
>> }
>> ```
>> 
>> fentry probes are better in general than kprobes if all you're doing
>> is attaching to the entry of a function.
>> 
>> You could do the same with kprobes like this if you really want, though:
>> 
>> ```
>> kprobe:tun:tun_net_xmit {
>>     $dev = (struct net_device *)arg1;
>>     $tun = (struct tun_struct *)$dev->priv;
>>     print($tun->numqueues);  // or whatever else you want
>> }
>> ```
>> 
>> Although it looks like there's a bug when you omit the module name
>> where bpftrace doesn't find the struct definition. I'll look into that.
>
> Minor: unless tun is built-in.

Ah, right.

>
> Thanks a lot for your response, Daniel. Good to know that we can get
> this information without kernel changes. And I learned something new
> :) Replicated your examples.

Nice! Feel free to CC me if you have other stuff in the future.

Bug fix for parsing implicit module BTF up here:
https://github.com/bpftrace/bpftrace/pull/4137

Re: [PATCH] net/tun: expose queue utilization stats via ethtool

[Alexander Shalimov]

>> Thanks a lot for your response, Daniel. Good to know that we can get
>> this information without kernel changes. And I learned something new
>> :) Replicated your examples.
>
> Nice! Feel free to CC me if you have other stuff in the future.
> 
> Bug fix for parsing implicit module BTF up here:
> https://github.com/bpftrace/bpftrace/pull/4137

Daniel and Willem, I appreciate you help!

My mistake was that we had tun configured as a module (CONFIG_TUN=m), and I
didn't explicitly specify the module name in the kprobe. Also, thank you for
pointing out that recent kernels have added the 'priv' field to net_device.
As a result, the script has now become much more universal and simpler.

Now we will think about how to efficiently implement monitoring on top of
our bpftrace script, which dynamically reports queue utilization.