Re: [PATCH bpf 0/4] bpf: remove __rcu annotations from bpf_prog_array

[Stanislav Fomichev <sdf@fomichev.me>]

On 05/08, Stanislav Fomichev wrote:
> On 05/08, Alexei Starovoitov wrote:
> > On Wed, May 08, 2019 at 10:18:41AM -0700, Stanislav Fomichev wrote:
> > > Right now we are not using rcu api correctly: we pass __rcu pointers
> > > to bpf_prog_array_xyz routines but don't use rcu_dereference on them
> > > (see bpf_prog_array_delete_safe and bpf_prog_array_copy in particular).
> > > Instead of sprinkling rcu_dereferences, let's just get rid of those
> > > __rcu annotations and move rcu handling to a higher level.
> > > 
> > > It looks like all those routines are called from the rcu update
> > > side and we can use simple rcu_dereference_protected to get a
> > > reference that is valid as long as we hold a mutex (i.e. no other
> > > updater can change the pointer, no need for rcu read section and
> > > there should not be a use-after-free problem).
> > > 
> > > To be fair, there is currently no issue with the existing approach
> > > since the calls are mutex-protected, pointer values don't change,
> > > __rcu annotations are ignored. But it's still nice to use proper api.
> > > 
> > > The series fixes the following sparse warnings:
> > 
> > Absolutely not.
> > please fix it properly.
> > Removing annotations is not a fix.
> I'm fixing it properly by removing the annotations and moving lifetime
> management to the upper layer. See commits 2-4 where I fix the users, the
> first patch is just the "preparation".
> 
> The users are supposed to do:
> 
> mutex_lock(&x);
> p = rcu_dereference_protected(prog_array, lockdep_is_held(&x))
> // ...
> // call bpf_prog_array helpers while mutex guarantees that
> // the object referenced by p is valid (i.e. no need for bpf_prog_array
> // helpers to care about rcu lifetime)
> // ...
> mutex_unlock(&x);
> 
> What am I missing here?

Just to give you my perspective on why current api with __rcu annotations
is working, but not optimal (even if used from the rcu read section).

Sample code:

	struct bpf_prog_array __rcu *progs = <comes from somewhere>;
	int n;

	rcu_read_lock();
	n = bpf_prog_array_length(progs);
	if (n > 0) {
	  // do something with __rcu progs
	  do_something(progs);
	}
	rcu_read_unlock();

Since progs is __rcu annotated, do_something() would need to do
rcu_dereference again and it might get a different value compared to
whatever bpf_prog_array_free got while doing its dereference.

A better way is not to deal with rcu inside those helpers and let
higher layers do that:

	struct bpf_prog_array __rcu *progs = <comes from somewhere>;
	struct bpf_prog_array *p;
	int n;

	rcu_read_lock();
	p = rcu_dereference(p);
	n = bpf_prog_array_length(p);
	if (n > 0) {
	  do_something(p); // do_something sees the same p as bpf_prog_array_length
	}
	rcu_read_unlock();

What do you think?

If it sounds reasonable, I can follow up with a v2 because I think I can
replace xchg with rcu_swap_protected as well. Or I can resend for bpf-next to
have another round of discussion. Thoughts?