Re: [PATCH net-next v2 5/9] tun: use bulk NAPI cache allocation in tun_xdp_one

[Jesper Dangaard Brouer <hawk@kernel.org>]

On 05/12/2025 08.58, Sebastian Andrzej Siewior wrote:
> On 2025-12-03 15:35:24 [+0000], Jon Kohler wrote:
>> Thanks, Sebastian - so if I’m reading this correct, it *is* fine to do
>> the two following patterns, outside of NAPI:
>>
>>     local_bh_disable();
>>     skb = napi_build_skb(buf, len);
>>     local_bh_enable();
>>
>>     local_bh_disable();
>>     napi_consume_skb(skb, 1);
>>     local_bh_enable();
>>
>> If so, I wonder if it would be cleaner to have something like
>>     build_skb_bh(buf, len);
>>
>>     consume_skb_bh(skb, 1);
>>
>> Then have those methods handle the local_bh enable/disable, so that
>> the toggle was a property of a call, not a requirement of the call?
> 
> Having budget = 0 would be for non-NAPI users. So passing the 1 is
> superfluous. You goal seems to be to re-use napi_alloc_cache. Right? And
> this is better than skb_pool?
> 
> There is already napi_alloc_skb() which expects BH to be disabled and
> netdev_alloc_skb() (and friends) which do disable BH if needed. I don't
> see an equivalent for non-NAPI users. Haven't checked if any of these
> could replace your napi_build_skb().
> 
> Historically non-NAPI users would be IRQ users and those can't do
> local_bh_disable(). Therefore there is dev_kfree_skb_irq_reason() for
> them. You need to delay the free for two reasons.
> It seems pure software implementations didn't bother so far.
> 
> It might make sense to do napi_consume_skb() similar to
> __netdev_alloc_skb() so that also budget=0 users fill the pool if this
> is really a benefit.

I'm not convinced that this "optimization" will be an actual benefit on
a busy system.  Let me explain the side-effect of local_bh_enable().

Calling local_bh_enable() is adding a re-scheduling opportunity, e.g.
for processing softirq.  For a benchmark this might not be noticeable as
this is the main workload.  If there isn't any pending softirq this is
also not noticeable.  In a more mixed workload (or packet storm) this
re-scheduling will allow others to "steal" CPU cycles from you.

Thus, you might not actually save any cycles via this short BH-disable
section.  I remember that I was saving around 19ns / 68cycles on a
3.6GHz E5-1650 CPU, by using this SKB recycle cache.  The cost of a re-
scheduling event is like more.

My advice is to use the napi_* function when already running within a
  BH-disabled section, as it makes sense to save those cycles
(essentially reducing the time spend with BH-disabled).  Wrapping these
napi_* function with BH-disabled just to use them outside NAPI feels
wrong in so many ways.

The another reason why these napi_* functions belongs with NAPI is that
netstack NIC drivers will (almost) always do TX completion first, that
will free/consume some SKBs, and afterwards do RX processing that need
to allocate SKBs for the incoming data frames.  Thus, keeping a cache of
SKBs just released/consumed makes sense.  (p.s. in the past we always
bulk free'ed all SKBs in the napi cache when exiting NAPI, as they would
not be cache hot for next round).

--Jesper