Re: [PATCH v5 bpf-next 04/10] bpf: tcp: Guard fast-path bpf_tcp_ops_call() under per-socket flag.

[Stanislav Fomichev <sdf.kernel@gmail.com>]

On 10/08, Kuniyuki Iwashima wrote:
> bpf_tcp_ops.{parse_hdr,hdr_opt_len} are called for every
> incoming / outgoing skb.
> 
> bpf_tcp_ops.rtt is called once per RTT, which is every
> incoming skb in ping-pong workloads like tcp_rr.
> 
> Even attaching NULL callbacks in the fast path hurts performance.
> 
> Let's guard them (and write_hdr_opt) with the new per-socket flags.
> 
> __bpf_tcp_ops_call() and bpf_tcp_ops_call_flag() are added to check
> cgroup_bpf_enabled(CGROUP_TCP_SOCK_OPS) first and avoid accessing
> tp->bpf_tcp_ops_flags when no bpf_tcp_ops is attached.
> 
> Note that bpf_tcp_ops_hdr_opt_len() has 3 callers and previously
> checked the static key both in tcp_established_options() and inside
> bpf_tcp_ops_call().  Now the static key is checked once at the
> beginning of bpf_tcp_ops_hdr_opt_len(), and the redundant check in
> tcp_established_options() is removed.
> 
> tcp_bpf_hdr_opt_len() must use __always_inline, not just
> inline, otherwise Clang inlines bpf_tcp_ops_hdr_opt_len() to
> tcp_bpf_hdr_opt_len() instead.
> 
>   $ llvm-objdump -S -D --disassemble=tcp_established_options vmlinux
>   ...
>   ; 	if (unlikely(BPF_SOCK_OPS_TEST_FLAG(tp,
>   ffffffff8251270e: 41 f6 87 20 0c 00 00 40      	testb	$0x40, 0xc20(%r15)
>   ffffffff82512716: 75 47                	jne	0xffffffff8251275f <tcp_established_options+0x1ff>
>   ; 	asm goto(ARCH_STATIC_BRANCH_ASM("%c0 + %c1", "%l[l_yes]")
>   ffffffff82512718: 0f 1f 44 00 00       	nopl	(%rax,%rax)
>   ; 	return size;
>   ffffffff8251271d: 44 89 e0             	movl	%r12d, %eax
>   ffffffff82512720: 5b                   	popq	%rbx
>   ffffffff82512721: 41 5c                	popq	%r12
>   ffffffff82512723: 41 5e                	popq	%r14
>   ffffffff82512725: 41 5f                	popq	%r15
>   ffffffff82512727: 5d                   	popq	%rbp
>   ffffffff82512728: 2e e9 42 03 43 00    	jmp	0xffffffff82942a70 <__x86_return_thunk>
> 
> bpf_skops_write_hdr_opt() is still not free, but it can be
> optimised later if needed.
> 
> Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
> ---
> v5: Always inline static key to tcp_established_options()

Acked-by: Stanislav Fomichev <sdf@fomichev.me>