Re: [patch net-next v4 5/9] genetlink: introduce per-sock family private pointer storage

[Jiri Pirko <jiri@resnulli.us>]

Tue, Nov 28, 2023 at 04:11:16PM CET, kuba@kernel.org wrote:
>On Tue, 28 Nov 2023 09:25:21 +0100 Jiri Pirko wrote:
>> >Put the xarray pointer here directly. Plus a lock to protect the init.  
>> 
>> Okay, just to make this clear. You want me to have:
>> 	struct xarray __rcu		*family_privs;
>> 
>> in struct netlink_sock, correct?
>> 
>> 
>> Why I need a lock? If I read things correctly, skbs are processed in
>> serial over one sock. Since this is per-sock, should be safe.
>
>Okay, then add an assertion that the socket lock is held, at least.

No socket lock, but I assumed revcmsg could be called only in parallel.
But I guess that with multiple threads, this assumption is broken.
Okay, will add a spin lock.


>Also, is the socket lock not held yet when the filtering happens?

Nope.


>Maybe the __rcu annotation isn't necessary then either?
>
>> >The size of the per-family struct should be in family definition,
>> >allocation should happen on first get automatically. Family definition  
>> 
>> Yes, I thought about that. But I decided to do this lockless, allocating
>> new priv every time the user sets the filter and replace the xarray item
>> so it could be accessed in rcu read section during notification
>> processing.
>> 
>> What you suggest requires some lock to protect the memory being changed
>> during filter set and suring accessing in in notify. But okay,
>> if you insist.
>
>Not necessarily, you can have a helper which doesn't allocate, too.
>What I'm saying is that the common case for ops will be to access
>the state and allocate if it doesn't exist.
>
>How about genl_sk_family_priv() and genl_sk_has_family_priv() ?

My point is, with what you suggest, it will look something like this:

1) user does DEVLINK_CMD_NOTIFY_FILTER_SET
2) devlink calls into genetlink code for a sk_priv and inserts in xa_array
3) genetlink allocates sk_priv and returns back
4) devlink fills-up the sk_priv

5) user does DEVLINK_CMD_NOTIFY_FILTER_SET, again
6) devlink calls into genetlink code for a sk_priv
7) genetlink returns already exising sk_priv found in xa_array
8) devlink fills-up the sk_priv

Now the notification thread, sees sk_priv zeroed between 3) and 4)
and inconsistent during 4) and 8)

I originally solved that by rcu, DEVLINK_CMD_NOTIFY_FILTER_SET
code always allocates and flips the rcu pointer. Notification thread
always sees sk_priv consistent.

If you want to allocate sk_priv in genetlink code once, hard to use
the rcu mechanism and have to protect the sk_priv memory by a lock.

What am I missing?


>
>> >should also hold a callback to how the data is going to be freed.  
>> 
>> If it is alloceted automatically, why is it needed?
>
>Because priv may be a complex type which has member that need
>individual fields to be destroyed (in fullness of time we also
>need a constructor which can init things like list_head, but
>we can defer that).
>
>I'm guessing in your case the priv will look like this:
>
>struct devlink_sk_priv {
>	const char *nft_fltr_instance_name;
>};
>
>static void devlink_sk_priv_free(void *ptr)
>{
>	struct devlink_sk_priv *priv = ptr;
>
>	kfree(priv->nft_fltr_instance_name);
>}

If genetlink code does the allocation, it should know how to free.
Does not make sense to pass destructor to genetlink code to free memory
it actually allocated :/

If devlink does the allocation, this callback makes sense. I was
thinking about having it, but decided kfree is okay for now and
destructor could be always introduced if needed.

Quoting the patch description:
    Assume that kfree() is good for free of privs for now, as the only user
    introduced by the follow-up patch (devlink) will use kzalloc() for the
    allocation of the memory of the stored pointer. If later on
    this needs to be made custom, a callback is going to be needed.
    Until then (if ever), do this in a simple way.