Re: [PATCH bpf-next v2 1/3] bpf: Make errors of bpf_tcp_check_syncookie distinguishable

[Maxim Mikityanskiy <maximmi@nvidia.com>]

On 2022-01-25 09:38, John Fastabend wrote:
> Maxim Mikityanskiy wrote:
>> bpf_tcp_check_syncookie returns ambiguous error codes in some cases. The
>> list below shows various error conditions and matching error codes:
>>
>> 1. NULL socket: -EINVAL.
>>
>> 2. Invalid packet: -EINVAL, -ENOENT.
>>
>> 3. Bad cookie: -ENOENT.
>>
>> 4. Cookies are not in use: -EINVAL, -ENOENT.
>>
>> 5. Good cookie: 0.
>>
>> As we see, the same error code may correspond to multiple error
>> conditions, making them undistinguishable, and at the same time one
>> error condition may return different codes, although it's typically
>> handled in the same way.
>>
>> This patch reassigns error codes of bpf_tcp_check_syncookie and
>> documents them:
>>
>> 1. Invalid packet or NULL socket: -EINVAL;
>>
>> 2. Bad cookie: -EACCES.
>>
>> 3. Cookies are not in use: -ENOENT.
>>
>> 4. Good cookie: 0.
>>
>> This change allows XDP programs to make smarter decisions based on error
>> code, because different error conditions are now easily distinguishable.
> 
> I'm missing the point here it still looks a bit like shuffling
> around of code. What do you gain, whats the real bug?

Current error codes are useless. If the caller gets a negative value, 
all it knows is that some error happened. I'm making different error 
conditions distinguishable (invalid input, bad cookie, not expecting a 
cookie).

Use cases could be: statistic counters, debugging, logging. For example, 
the kernel counts LINUX_MIB_SYNCOOKIESFAILED, which only includes the 
"bad cookie" case.

Another use case is replying with RST when not expecting a cookie (a new 
ACK packet could just mean that the server was rebooted, and it's good 
to tell the client that the connection is broken), but dropping packets 
under the flood on receiving bad cookies (to avoid wasting resources on 
sending replies).

> Are you
> trying to differentiate between an overflow condition and a valid
> syncookie? But I don't think you said this anywhere.

I'm not sure what you mean by the "overflow condition", but a valid 
syncookie is easily distinguishable from other cases, it's 0.

> At the moment EINVAL tells me somethings wrong with the input or
> configuration, although really any app that cares checked the
> sysctl flag right?

There is no way to check sysctl from XDP. A single program could be 
useful both with and without syncookies, it could determine its behavior 
in runtime, not to say that the sysctl option could change in runtime. 
The workaround you suggested will work in some cases, but other cases 
just won't work (there are no notification events on sysctl changes that 
a userspace application could monitor and pass to the XDP program; it 
would be not immediate anyway).

> ENOENT tells me either recent overflow or cookie is invalid.

And these cases should be distinguished, as I said above.

> If
> there is no '!ack || rst || syn' then I can either learn that
> directly from the program (why would a real program through
> these at the helper), but it also falls into the incorrect
> cookie in some sense.
> 
>>
>> Backward compatibility shouldn't suffer because of these reasons:
>>
>> 1. The specific error codes weren't documented. The behavior that used
>>     to be documented (0 is good cookie, negative values are errors) still
>>     holds. Anyone who relied on implementation details should have
>>     understood the risks.
> 
> I'll disagree, just because a user facing API doesn't document its
> behavior very well doesn't mean users should some how understand the
> risks. Ideally we would have done better with error codes up front,
> but thats old history. If a user complains that this breaks a real
> application it would likely be reason to revert it.
> 
> At least I would remove this from the commit.
> 
>>
>> 2. Two known usecases (classification of ACKs with cookies that initial
>>     new connections, SYN flood protection) take decisions which don't
>>     depend on specific error codes:
>>
>>       Traffic classification:
>>         ACK packet is new, error == 0: classify as NEW.
>>         ACK packet is new, error < 0: classify as INVALID.
>>
>>       SYN flood protection:
>>         ACK packet is new, error == 0: good cookie, XDP_PASS.
>>         ACK packet is new, error < 0: bad cookie, XDP_DROP.
>>
>>     As Lorenz Bauer confirms, their implementation of traffic classifier
>>     won't break, as well as the kernel selftests.
>>
>> 3. It's hard to imagine that old error codes could be used for any
>>     useful decisions.
>>
>> Signed-off-by: Maxim Mikityanskiy <maximmi@nvidia.com>
>> Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
>> ---
>>   include/uapi/linux/bpf.h       | 18 ++++++++++++++++--
>>   net/core/filter.c              |  6 +++---
>>   tools/include/uapi/linux/bpf.h | 18 ++++++++++++++++--
>>   3 files changed, 35 insertions(+), 7 deletions(-)
>>
>> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
>> index 16a7574292a5..4d2d4a09bf25 100644
>> --- a/include/uapi/linux/bpf.h
>> +++ b/include/uapi/linux/bpf.h
>> @@ -3575,8 +3575,22 @@ union bpf_attr {
>>    * 		*th* points to the start of the TCP header, while *th_len*
>>    * 		contains **sizeof**\ (**struct tcphdr**).
>>    * 	Return
>> - * 		0 if *iph* and *th* are a valid SYN cookie ACK, or a negative
>> - * 		error otherwise.
>> + *		0 if *iph* and *th* are a valid SYN cookie ACK.
>> + *
>> + *		On failure, the returned value is one of the following:
>> + *
>> + *		**-EACCES** if the SYN cookie is not valid.
>> + *
>> + *		**-EINVAL** if the packet or input arguments are invalid.
>> + *
>> + *		**-ENOENT** if SYN cookies are not issued (no SYN flood, or SYN
>> + *		cookies are disabled in sysctl).
>> + *
>> + *		**-EOPNOTSUPP** if the kernel configuration does not enable SYN
>> + *		cookies (CONFIG_SYN_COOKIES is off).
>> + *
>> + *		**-EPROTONOSUPPORT** if the IP version is not 4 or 6 (or 6, but
>> + *		CONFIG_IPV6 is disabled).
>>    *
>>    * long bpf_sysctl_get_name(struct bpf_sysctl *ctx, char *buf, size_t buf_len, u64 flags)
>>    *	Description
>> diff --git a/net/core/filter.c b/net/core/filter.c
>> index a06931c27eeb..18559b5828a3 100644
>> --- a/net/core/filter.c
>> +++ b/net/core/filter.c
>> @@ -6998,10 +6998,10 @@ BPF_CALL_5(bpf_tcp_check_syncookie, struct sock *, sk, void *, iph, u32, iph_len
>>   		return -EINVAL;
>>   
>>   	if (!sock_net(sk)->ipv4.sysctl_tcp_syncookies)
>> -		return -EINVAL;
>> +		return -ENOENT;
> 
> I wouldn't change this.
> 
>>   
>>   	if (!th->ack || th->rst || th->syn)
>> -		return -ENOENT;
>> +		return -EINVAL;
> 
> not sure if this is useful change and it is bpf program visible.
> 
>>   
>>   	if (tcp_synq_no_recent_overflow(sk))
>>   		return -ENOENT;
>> @@ -7032,7 +7032,7 @@ BPF_CALL_5(bpf_tcp_check_syncookie, struct sock *, sk, void *, iph, u32, iph_len
>>   	if (ret > 0)
>>   		return 0;
>>   
>> -	return -ENOENT;
>> +	return -EACCES;
> 
> This one might have a valid argument to differentiate between an
> overflow condition and an invalid cookie. But, curious what do you
> do with this info?
> 
> Thanks,
> John