Proposal using RCU for callbacks

Proposal using RCU for callbacks

[Stephen Hemminger]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]


The BPF and packet capture Rx/Tx callbacks use a per-queue use counter
so that removal can wait for the datapath. In BPF that costs a locked
instruction per burst. The code would be faster and cleaner if RCU
could be used, but DPDK RCU is limited to the QSBR model, which
requires application opt-in.

The userspace RCU library (liburcu) supports other models, so I did a
brief investigation into whether something similar could be done in
DPDK. As expected, it also turned up pre-existing issues: the counters
are wrong with MT_LOCKFREE Tx, the pdump one is not safe as written,
and cryptodev callback removal never waits.

I used Claude to write it up in more detail (attached). It is all from
code reading; nothing has been reproduced or prototyped yet.
This is too big to go into 26.11 at this point, will try and have something
for 27.03

[-- Attachment #2: rcu-ethdev-callback-removal.html --]
[-- Type: text/html, Size: 22874 bytes --]

Re: Proposal using RCU for callbacks

[Marat Khalili]

On 12/10/2026 19:13, Stephen Hemminger wrote:

> The BPF and packet capture Rx/Tx callbacks use a per-queue use counter
> so that removal can wait for the datapath. In BPF that costs a locked
> instruction per burst. The code would be faster and cleaner if RCU
> could be used, but DPDK RCU is limited to the QSBR model, which
> requires application opt-in.
>
> The userspace RCU library (liburcu) supports other models, so I did a
> brief investigation into whether something similar could be done in
> DPDK. As expected, it also turned up pre-existing issues: the counters
> are wrong with MT_LOCKFREE Tx, the pdump one is not safe as written,
> and cryptodev callback removal never waits.
>
> I used Claude to write it up in more detail (attached). It is all from
> code reading; nothing has been reproduced or prototyped yet.
> This is too big to go into 26.11 at this point, will try and have something
> for 27.03

Maybe it's just me, but the document is a little hard to read without 
knowing all of the context. Nevertheless, there is no doubt the sync 
mechanism in bpf_pkt.c can be improved. It done today, it would probably 
be written differently, very likely with RCU.

I would argue for incremental approach however. I'm currently aware of 
the following problems, in the order of decreasing priority and gain/effort:
* Deprecated rte_smp barrier functions being used.
* BPF inuse counter is not safe when using lockfree transmit (Bug 2051)
* Code needs cleanup and documenting to make discussions more productive.
* Possibly additional safety issues listed above (which I don't yet 
fully understand).
* Missing features: multi-threaded data planes, multiple callbacks, BPF 
program reuse etc.
* On the very last place, performance; as usual, benchmarks are 
mandatory for it to have any meaning.

I think we need to resist the temptation to just rewrite everything, 
especially with AI. Some recent research shows that productivity gains 
from AI are very limited particularly because code review becomes 
bottleneck (tbh it always was). Test coverage is also clearly 
insufficient for any major rewrites of this code. A patch solving 
everything at once risks getting stuck in review for very long time. 
Perhaps we arrive at RCU solution eventually, but I'd start with first 
items in the list.

Re: Proposal using RCU for callbacks

[Stephen Hemminger]

On Mon, 12 Oct 2026 21:27:08 +0100
Marat Khalili <qm2k21@gmail.com> wrote:

> On 12/10/2026 19:13, Stephen Hemminger wrote:
> 
> > The BPF and packet capture Rx/Tx callbacks use a per-queue use counter
> > so that removal can wait for the datapath. In BPF that costs a locked
> > instruction per burst. The code would be faster and cleaner if RCU
> > could be used, but DPDK RCU is limited to the QSBR model, which
> > requires application opt-in.
> >
> > The userspace RCU library (liburcu) supports other models, so I did a
> > brief investigation into whether something similar could be done in
> > DPDK. As expected, it also turned up pre-existing issues: the counters
> > are wrong with MT_LOCKFREE Tx, the pdump one is not safe as written,
> > and cryptodev callback removal never waits.
> >
> > I used Claude to write it up in more detail (attached). It is all from
> > code reading; nothing has been reproduced or prototyped yet.
> > This is too big to go into 26.11 at this point, will try and have something
> > for 27.03  
> 
> Maybe it's just me, but the document is a little hard to read without 
> knowing all of the context. Nevertheless, there is no doubt the sync 
> mechanism in bpf_pkt.c can be improved. It done today, it would probably 
> be written differently, very likely with RCU.
> 
> I would argue for incremental approach however. I'm currently aware of 
> the following problems, in the order of decreasing priority and gain/effort:
> * Deprecated rte_smp barrier functions being used.
> * BPF inuse counter is not safe when using lockfree transmit (Bug 2051)
> * Code needs cleanup and documenting to make discussions more productive.
> * Possibly additional safety issues listed above (which I don't yet 
> fully understand).
> * Missing features: multi-threaded data planes, multiple callbacks, BPF 
> program reuse etc.
> * On the very last place, performance; as usual, benchmarks are 
> mandatory for it to have any meaning.
> 
> I think we need to resist the temptation to just rewrite everything, 
> especially with AI. Some recent research shows that productivity gains 
> from AI are very limited particularly because code review becomes 
> bottleneck (tbh it always was). Test coverage is also clearly 
> insufficient for any major rewrites of this code. A patch solving 
> everything at once risks getting stuck in review for very long time. 
> Perhaps we arrive at RCU solution eventually, but I'd start with first 
> items in the list.
> 

Short form: BPF and pdump could use RCU to manage callback lifetime;
cryptodev already uses QSBR, but incorrectly. I would like to see
one usage pattern across all of DPDK that is fast and correct.

DPDK RCU only does QSBR, which needs the application to report when
each thread has passed a quiescent state. That restriction is why it is underused.

Other RCU flavors avoid this. With membarrier, the reader side needs
no fence, only a compiler barrier, and the writer pays with a syscall.
Reader state can hang off EAL thread registration, so no application
change is needed.

Today the BPF callback does a full memory barrier per burst. RCU would
replace that with a compiler barrier. Benchmarks will be needed to
show it matters.

This is not "let's use AI to rewrite it". AI was useful for the what-if
analysis; the real fixes will be real code, done incrementally.
Agree the barrier conversion and Bug 2051 come first.

See:
 - https://man7.org/linux/man-pages/man2/membarrier.2.html
 - https://liburcu.org/
 - https://github.com/nmraz/simplercu (toy implementation)
 - https://lwn.net/Articles/369018/ (original 2010 RFC)

Re: Proposal using RCU for callbacks

[Konstantin Ananyev]

> On Mon, 12 Oct 2026 21:27:08 +0100
> Marat Khalili <qm2k21@gmail.com> wrote:
>
>> On 12/10/2026 19:13, Stephen Hemminger wrote:
>>
>>> The BPF and packet capture Rx/Tx callbacks use a per-queue use counter
>>> so that removal can wait for the datapath. In BPF that costs a locked
>>> instruction per burst. The code would be faster and cleaner if RCU
>>> could be used, but DPDK RCU is limited to the QSBR model, which
>>> requires application opt-in.
>>>
>>> The userspace RCU library (liburcu) supports other models, so I did a
>>> brief investigation into whether something similar could be done in
>>> DPDK. As expected, it also turned up pre-existing issues: the counters
>>> are wrong with MT_LOCKFREE Tx, the pdump one is not safe as written,
>>> and cryptodev callback removal never waits.
>>>
>>> I used Claude to write it up in more detail (attached). It is all from
>>> code reading; nothing has been reproduced or prototyped yet.
>>> This is too big to go into 26.11 at this point, will try and have something
>>> for 27.03
>> Maybe it's just me, but the document is a little hard to read without
>> knowing all of the context. Nevertheless, there is no doubt the sync
>> mechanism in bpf_pkt.c can be improved. It done today, it would probably
>> be written differently, very likely with RCU.
>>
>> I would argue for incremental approach however. I'm currently aware of
>> the following problems, in the order of decreasing priority and gain/effort:
>> * Deprecated rte_smp barrier functions being used.
>> * BPF inuse counter is not safe when using lockfree transmit (Bug 2051)
>> * Code needs cleanup and documenting to make discussions more productive.
>> * Possibly additional safety issues listed above (which I don't yet
>> fully understand).
>> * Missing features: multi-threaded data planes, multiple callbacks, BPF
>> program reuse etc.
>> * On the very last place, performance; as usual, benchmarks are
>> mandatory for it to have any meaning.
>>
>> I think we need to resist the temptation to just rewrite everything,
>> especially with AI. Some recent research shows that productivity gains
>> from AI are very limited particularly because code review becomes
>> bottleneck (tbh it always was). Test coverage is also clearly
>> insufficient for any major rewrites of this code. A patch solving
>> everything at once risks getting stuck in review for very long time.
>> Perhaps we arrive at RCU solution eventually, but I'd start with first
>> items in the list.
>>
> Short form: BPF and pdump could use RCU to manage callback lifetime;
> cryptodev already uses QSBR, but incorrectly. I would like to see
> one usage pattern across all of DPDK that is fast and correct.
Yep, in general I do agree with your analysis
Yes cryptodev CB add/remove uses RCU incorrectly and needs to be fixed.
ethdev CB add/remove doesn't contain any sync mechanism at all,
which is a problem.
Agree that both should use the same mechanism and then we can simply
remove current one in bpf_pkt.c and rely on one in ethdev.

About further enhancements, i.e. using something better/smarter then our
current QSBR-based RCU ny position is simple: let's fix the problems
above first before starting to do any further enhancements.

In particular, regarding membarrier() - consider using it for BPF CBs
instead of smb_mb, but decided not to. The main reasons:
- with membarrier() - all threads in the process will be affected.
   Also, AFAIK it is implemented via IPI - which is quite heavy.
- current smp_mb (at least on modern x86_64) is quite fast
(it uses lock-ed instruction on thread's stack, and last time I measured it,
takes ¬30-50 cycles). As we usually doing it for a bulk (32+) of packets,
the overhead is quite small: 1-1.5 cycle per packet.

But again, if you have some smart ideas how to improve our RCU lib with 
membarrier() -
would be really interested to hear.
>
> DPDK RCU only does QSBR, which needs the application to report when
> each thread has passed a quiescent state. That restriction is why it is underused.
>
> Other RCU flavors avoid this. With membarrier, the reader side needs
> no fence, only a compiler barrier, and the writer pays with a syscall.
> Reader state can hang off EAL thread registration, so no application
> change is needed.
>
> Today the BPF callback does a full memory barrier per burst. RCU would
> replace that with a compiler barrier. Benchmarks will be needed to
> show it matters.
>
> This is not "let's use AI to rewrite it". AI was useful for the what-if
> analysis; the real fixes will be real code, done incrementally.
> Agree the barrier conversion and Bug 2051 come first.
>
> See:
>   - https://man7.org/linux/man-pages/man2/membarrier.2.html
>   - https://liburcu.org/
>   - https://github.com/nmraz/simplercu (toy implementation)
>   - https://lwn.net/Articles/369018/ (original 2010 RFC)

Re: Proposal using RCU for callbacks

[Stephen Hemminger]

On Wed, 21 Oct 2026 12:51:11 +0100
Konstantin Ananyev <konstantin.v.ananyev@yandex.ru> wrote:

> In particular, regarding membarrier() - consider using it for BPF CBs
> instead of smb_mb, but decided not to. The main reasons:
> - with membarrier() - all threads in the process will be affected.
>    Also, AFAIK it is implemented via IPI - which is quite heavy.
> - current smp_mb (at least on modern x86_64) is quite fast
> (it uses lock-ed instruction on thread's stack, and last time I measured it,
> takes ¬30-50 cycles). As we usually doing it for a bulk (32+) of packets,
> the overhead is quite small: 1-1.5 cycle per packet.
> 
> But again, if you have some smart ideas how to improve our RCU lib with 
> membarrier() -
> would be really interested to hear.

There is a version in current wireshark extcap patch series.
The membarrier is expensive but only happens when capture is stopped.
We can afford a system call there.

The read side (in callback) is:

static inline void
capture_rcu_enter(unsigned int slot)
{
	struct capture_rcu_reader *r = &capture_rcu.reader[slot];
	uint64_t t;

	if (r->depth++ != 0)
		return;

	t = rte_atomic_load_explicit(&capture_rcu.token, rte_memory_order_relaxed);
	rte_atomic_store_explicit(&r->ctr, t, rte_memory_order_relaxed);

	/*
	 * The loads that follow must not move above the store of ctr. The
	 * writer forces that on every CPU with membarrier, so no fence is
	 * needed here; capture is refused if membarrier does not work.
	 */
	rte_compiler_barrier();
}

static inline void
capture_rcu_exit(unsigned int slot)
{
	struct capture_rcu_reader *r = &capture_rcu.reader[slot];

	if (--r->depth != 0)
		return;

	/* release: prior loads complete before the slot reads idle */
	rte_atomic_store_explicit(&r->ctr, 0, rte_memory_order_release);
}

Re: Proposal using RCU for callbacks

[Konstantin Ananyev]

> On Wed, 21 Oct 2026 12:51:11 +0100
> Konstantin Ananyev <konstantin.v.ananyev@yandex.ru> wrote:
>
>> In particular, regarding membarrier() - consider using it for BPF CBs
>> instead of smb_mb, but decided not to. The main reasons:
>> - with membarrier() - all threads in the process will be affected.
>>     Also, AFAIK it is implemented via IPI - which is quite heavy.
>> - current smp_mb (at least on modern x86_64) is quite fast
>> (it uses lock-ed instruction on thread's stack, and last time I measured it,
>> takes ¬30-50 cycles). As we usually doing it for a bulk (32+) of packets,
>> the overhead is quite small: 1-1.5 cycle per packet.
>>
>> But again, if you have some smart ideas how to improve our RCU lib with
>> membarrier() -
>> would be really interested to hear.
> There is a version in current wireshark extcap patch series.
> The membarrier is expensive but only happens when capture is stopped.
> We can afford a system call there.
Probably I didn't explain my concern regarding membarrier() well enough.
Let me try again:
AFAIU, when you issue a membarrier() on one thread, all other threads in 
that process
(in case of MP - on all processes) will be synchronized.
I.E: all of them (all CPUs they run on) will receive an IPI, that they 
have to handle.
So, for each CB add/remove - all DP threads doing packet RX/TX will be 
interrupted.
That probably not what user wants.
Once again - why do you think that current librcu mechanism is not fast 
enough?
As I said before: one thread_fence(seq_cst) per burst doesn't look like 
a huge slowdown to me.

>
> The read side (in callback) is:
>
> static inline void
> capture_rcu_enter(unsigned int slot)
> {
> 	struct capture_rcu_reader *r = &capture_rcu.reader[slot];
> 	uint64_t t;
>
> 	if (r->depth++ != 0)
> 		return;
>
> 	t = rte_atomic_load_explicit(&capture_rcu.token, rte_memory_order_relaxed);
> 	rte_atomic_store_explicit(&r->ctr, t, rte_memory_order_relaxed);
>
> 	/*
> 	 * The loads that follow must not move above the store of ctr. The
> 	 * writer forces that on every CPU with membarrier, so no fence is
> 	 * needed here; capture is refused if membarrier does not work.
> 	 */
> 	rte_compiler_barrier();
> }
>
> static inline void
> capture_rcu_exit(unsigned int slot)
> {
> 	struct capture_rcu_reader *r = &capture_rcu.reader[slot];
>
> 	if (--r->depth != 0)
> 		return;
>
> 	/* release: prior loads complete before the slot reads idle */
> 	rte_atomic_store_explicit(&r->ctr, 0, rte_memory_order_release);
> }