Re: [Patch bpf-next v3 4/5] skmsg: use skb ext instead of TCP_SKB_CB

[John Fastabend <john.fastabend@gmail.com>]

Cong Wang wrote:
> On Mon, Feb 15, 2021 at 11:20 AM John Fastabend
> <john.fastabend@gmail.com> wrote:
> >
> > Cong Wang wrote:
> > > From: Cong Wang <cong.wang@bytedance.com>
> > >
> > > Currently TCP_SKB_CB() is hard-coded in skmsg code, it certainly
> > > does not work for any other non-TCP protocols. We can move them to
> > > skb ext instead of playing with skb cb, which is harder to make
> > > correct.
> > >
> > > Cc: John Fastabend <john.fastabend@gmail.com>
> > > Cc: Daniel Borkmann <daniel@iogearbox.net>
> > > Cc: Jakub Sitnicki <jakub@cloudflare.com>
> > > Reviewed-by: Lorenz Bauer <lmb@cloudflare.com>
> > > Signed-off-by: Cong Wang <cong.wang@bytedance.com>
> > > ---
> >
> > I'm not seeing the advantage of doing this at the moment. We can
> > continue to use cb[] here, which is simpler IMO and use the ext
> > if needed for the other use cases. This is adding a per packet
> > alloc cost that we don't have at the moment as I understand it.
> 
> Hmm? How can we continue using TCP_SKB_CB() for UDP or
> AF_UNIX?
> 
> I am not sure I get your "at the moment" correctly, do you mean
> I should move this patch to a later patchset, maybe the UDP
> patchset? At least this patch is needed, no matter by which patchset,
> so it should not be dropped.

Agree, the skb_bpf_ext{} pieces are needed for UDP and AF_UNIX. Its
not required for TCP side though. What I'm suggesting is leave the
TCP side as-is, using the cb[] fields. Then use the skb_bpf_ext fields
from UDP and AF_UNIX.

> 
> 
> >
> > [...]
> >
> > > diff --git a/include/linux/skmsg.h b/include/linux/skmsg.h
> > > index e3bb712af257..d5c711ef6d4b 100644
> > > --- a/include/linux/skmsg.h
> > > +++ b/include/linux/skmsg.h
> > > @@ -459,4 +459,44 @@ static inline bool sk_psock_strp_enabled(struct sk_psock *psock)
> > >               return false;
> > >       return !!psock->saved_data_ready;
> > >  }
> > > +
> > > +struct skb_bpf_ext {
> > > +     __u32 flags;
> > > +     struct sock *sk_redir;
> > > +};
> > > +
> > > +#if IS_ENABLED(CONFIG_NET_SOCK_MSG)
> > > +static inline
> > > +bool skb_bpf_ext_ingress(const struct sk_buff *skb)
> > > +{
> > > +     struct skb_bpf_ext *ext = skb_ext_find(skb, SKB_EXT_BPF);
> > > +
> > > +     return ext->flags & BPF_F_INGRESS;
> > > +}
> > > +
> > > +static inline
> > > +void skb_bpf_ext_set_ingress(const struct sk_buff *skb)
> > > +{
> > > +     struct skb_bpf_ext *ext = skb_ext_find(skb, SKB_EXT_BPF);
> > > +
> > > +     ext->flags |= BPF_F_INGRESS;
> > > +}
> > > +
> > > +static inline
> > > +struct sock *skb_bpf_ext_redirect_fetch(struct sk_buff *skb)
> > > +{
> > > +     struct skb_bpf_ext *ext = skb_ext_find(skb, SKB_EXT_BPF);
> > > +
> > > +     return ext->sk_redir;
> > > +}
> > > +
> > > +static inline
> > > +void skb_bpf_ext_redirect_clear(struct sk_buff *skb)
> > > +{
> > > +     struct skb_bpf_ext *ext = skb_ext_find(skb, SKB_EXT_BPF);
> > > +
 > > +     ext->flags = 0;
> > > +     ext->sk_redir = NULL;
> > > +}
> > > +#endif /* CONFIG_NET_SOCK_MSG */
> >
> > So we will have some slight duplication for cb[] variant and ext
> > variant above. I'm OK with that to avoid an allocation.
> 
> Not sure what you mean by "duplication", these are removed from
> TCP_SKB_CB(), so there is clearly no duplication.

In this patch yes, no duplication.  But, I want to leave TCP alone
and have it continue to use cb[] to avoid alloc per packet.

> 
> >
> > [...]
> >
> > > @@ -1003,11 +1008,17 @@ static int sk_psock_verdict_recv(read_descriptor_t *desc, struct sk_buff *skb,
> > >               goto out;
> > >       }
> > >       skb_set_owner_r(skb, sk);
> > > +     if (!skb_ext_add(skb, SKB_EXT_BPF)) {
> > > +             len = 0;
> > > +             kfree_skb(skb);
> > > +             goto out;
> > > +     }
> > > +
> >
> > per packet cost here. Perhaps you can argue small alloc will usually not be
> > noticable in such a large stack, but once we convert over it will be very
> > hard to go back. And I'm looking at optimizing this path now.
> 
> This is a price we need to pay to avoid CB, and skb_ext_add() has been
> used on other fast paths too, for example, tcf_classify_ingress() and
> mptcp_incoming_options(). So, it is definitely acceptable.

For TCP case we can continue to use CB and not pay the price. For UDP
and AF_UNIX we can do the extra alloc.

The use in tcf_classify_ingress is a miss case so not the common path. If
it is/was in the common path I would suggest we rip it out.

> 
> >
> > >       prog = READ_ONCE(psock->progs.skb_verdict);
> > >       if (likely(prog)) {
> > > -             tcp_skb_bpf_redirect_clear(skb);
> > > +             skb_bpf_ext_redirect_clear(skb);
> > >               ret = sk_psock_bpf_run(psock, prog, skb);
> > > -             ret = sk_psock_map_verd(ret, tcp_skb_bpf_redirect_fetch(skb));
> > > +             ret = sk_psock_map_verd(ret, skb_bpf_ext_redirect_fetch(skb));
> > >       }
> > >       sk_psock_verdict_apply(psock, skb, ret);
> >
> > Thanks for the series Cong. Drop this patch and resubmit carry ACKs forward
> > and then lets revisit this later.
> 
> I still believe it is best to stay in this patchset, as it does not change
> any functionality and is a preparation too. And the next patchset will be
> UDP/AF_UNIX changes as you suggested, it is very awkward to put this
> patch into either UDP or AF_UNIX changes.

Disagree. It adds extra code to the TCP side that I think is not needed. Any
reason the TCP implementation can't continue to use cb[]?

> 
> So, let's keep it in this patchset, and I am happy to address any concerns
> and open to other ideas than using skb ext.

The idea here is to just use cb[] in TCP case per above. I only scanned your
other patches, but presumably this can be patch 1 with just the functions

 skb_bpf_ext_ingress()
 skb_bpf_ext_set_ingress()
 skb_bpf_ext_redirect_fetch()
 skb_bpf_ext_redirect_clear()

And none of the removals from TCP side.

.John