BPF selftest: mptcp subtest failing

BPF selftest: mptcp subtest failing

[Harshvardhan Jha]

Hi there,
I have explicitly disabled mptpcp by default on my custom kernel and
this seems to be causing the test case to fail. Even after enabling
mtpcp via sysctl command or adding an entry to /etc/sysctl.conf this
fails. I don't think this test should be failing and should account for
cases where mptcp has not been enabled by default?

These are the test logs:

$ sudo tools/testing/selftests/bpf/test_progs -t mptcp
Can't find bpf_testmod.ko kernel module: -2
WARNING! Selftests relying on bpf_testmod.ko will be skipped.

run_test:PASS:bpf_prog_attach 0 nsec
run_test:PASS:connect to fd 0 nsec
verify_tsk:PASS:bpf_map_lookup_elem 0 nsec
verify_tsk:PASS:unexpected invoked count 0 nsec
verify_tsk:PASS:unexpected is_mptcp 0 nsec
test_base:PASS:run_test tcp 0 nsec
(network_helpers.c:107: errno: Protocol not available) Failed to create
server socket
test_base:FAIL:start_mptcp_server unexpected start_mptcp_server: actual
-1 < expected 0
#178/1   mptcp/base:FAIL
test_mptcpify:PASS:test__join_cgroup 0 nsec
create_netns:PASS:ip netns add mptcp_ns 0 nsec
create_netns:PASS:ip -net mptcp_ns link set dev lo up 0 nsec
test_mptcpify:PASS:create_netns 0 nsec
run_mptcpify:PASS:skel_open_load 0 nsec
run_mptcpify:PASS:skel_attach 0 nsec
(network_helpers.c:107: errno: Protocol not available) Failed to create
server socket
run_mptcpify:FAIL:start_server unexpected start_server: actual -1 <
expected 0
test_mptcpify:FAIL:run_mptcpify unexpected error: -5 (errno 92)
#178/2   mptcp/mptcpify:FAIL
#178     mptcp:FAIL

All error logs:
test_base:PASS:test__join_cgroup 0 nsec
create_netns:PASS:ip netns add mptcp_ns 0 nsec
create_netns:PASS:ip -net mptcp_ns link set dev lo up 0 nsec
test_base:PASS:create_netns 0 nsec
test_base:PASS:start_server 0 nsec
run_test:PASS:skel_open_load 0 nsec
run_test:PASS:skel_attach 0 nsec
run_test:PASS:bpf_prog_attach 0 nsec
run_test:PASS:connect to fd 0 nsec
verify_tsk:PASS:bpf_map_lookup_elem 0 nsec
verify_tsk:PASS:unexpected invoked count 0 nsec
verify_tsk:PASS:unexpected is_mptcp 0 nsec
test_base:PASS:run_test tcp 0 nsec
(network_helpers.c:107: errno: Protocol not available) Failed to create
server socket
test_base:FAIL:start_mptcp_server unexpected start_mptcp_server: actual
-1 < expected 0
#178/1   mptcp/base:FAIL
test_mptcpify:PASS:test__join_cgroup 0 nsec
create_netns:PASS:ip netns add mptcp_ns 0 nsec
create_netns:PASS:ip -net mptcp_ns link set dev lo up 0 nsec
test_mptcpify:PASS:create_netns 0 nsec
run_mptcpify:PASS:skel_open_load 0 nsec
run_mptcpify:PASS:skel_attach 0 nsec
(network_helpers.c:107: errno: Protocol not available) Failed to create
server socket
run_mptcpify:FAIL:start_server unexpected start_server: actual -1 <
expected 0
test_mptcpify:FAIL:run_mptcpify unexpected error: -5 (errno 92)
#178/2   mptcp/mptcpify:FAIL
#178     mptcp:FAIL
Summary: 0/0 PASSED, 0 SKIPPED, 1 FAILED

This is the custom patch I had applied on the LTS v6.12.36 kernel and
tested it:

diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c
index dd595d9b5e50c..bdcc4136e92ef 100644
--- a/net/mptcp/ctrl.c
+++ b/net/mptcp/ctrl.c
@@ -89,7 +89,7 @@ const char *mptcp_get_scheduler(const struct net *net)
 
 static void mptcp_pernet_set_defaults(struct mptcp_pernet *pernet)
 {
-	pernet->mptcp_enabled = 1;
+	pernet->mptcp_enabled = 0;
 	pernet->add_addr_timeout = TCP_RTO_MAX;
 	pernet->blackhole_timeout = 3600;
 	atomic_set(&pernet->active_disable_times, 0);
--
Thanks & Regards, Harshvardhan

Re: BPF selftest: mptcp subtest failing

[Matthieu Baerts]

Hi Harshvardhan,

On 07/08/2025 05:50, Harshvardhan Jha wrote:
> Hi there,
> I have explicitly disabled mptpcp by default on my custom kernel and
> this seems to be causing the test case to fail. Even after enabling
> mtpcp via sysctl command or adding an entry to /etc/sysctl.conf this
> fails. I don't think this test should be failing and should account for
> cases where mptcp has not been enabled by default?

It looks like the test is failing because it expects MPTCP to be enabled
by default. Or, said differently, it doesn't expect the kernel to be
modified without adapting the corresponding tests :)

> This is the custom patch I had applied on the LTS v6.12.36 kernel and
> tested it:
> 
> diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c
> index dd595d9b5e50c..bdcc4136e92ef 100644
> --- a/net/mptcp/ctrl.c
> +++ b/net/mptcp/ctrl.c
> @@ -89,7 +89,7 @@ const char *mptcp_get_scheduler(const struct net *net)
>  
>  static void mptcp_pernet_set_defaults(struct mptcp_pernet *pernet)
>  {
> -	pernet->mptcp_enabled = 1;
> +	pernet->mptcp_enabled = 0;
>  	pernet->add_addr_timeout = TCP_RTO_MAX;
>  	pernet->blackhole_timeout = 3600;
>  	atomic_set(&pernet->active_disable_times, 0);

First, I have the same question as the one I asked to RedHat devs: do
you still need to keep MPTCP disabled by default? If I remember well, on
RHEL side, they started to do that when they backported MPTCP on a
previous stable version, as an experimental feature. They left it like
that later mostly for internal process reasons I think. But honestly,
today, it no longer makes sense to do that and annoys users: all other
Linux distributions enable MPTCP by default without patching the kernel
like you did.

If you don't want to revert this patch, I guess you can modify the BPF
selftests in 'prog_tests/mptcp.c' to set 'sysctl net.mptcp.enabled=1' in
each netns created by the test. But again, not changing the default
kernel behaviour sounds like a better solution.

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: BPF selftest: mptcp subtest failing

[Harshvardhan Jha]

Hi Matthieu,

On 07/08/25 7:51 PM, Matthieu Baerts wrote:
> Hi Harshvardhan,
>
> On 07/08/2025 05:50, Harshvardhan Jha wrote:
>> Hi there,
>> I have explicitly disabled mptpcp by default on my custom kernel and
>> this seems to be causing the test case to fail. Even after enabling
>> mtpcp via sysctl command or adding an entry to /etc/sysctl.conf this
>> fails. I don't think this test should be failing and should account for
>> cases where mptcp has not been enabled by default?
> It looks like the test is failing because it expects MPTCP to be enabled
> by default. Or, said differently, it doesn't expect the kernel to be
> modified without adapting the corresponding tests :)
>
>> This is the custom patch I had applied on the LTS v6.12.36 kernel and
>> tested it:
>>
>> diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c
>> index dd595d9b5e50c..bdcc4136e92ef 100644
>> --- a/net/mptcp/ctrl.c
>> +++ b/net/mptcp/ctrl.c
>> @@ -89,7 +89,7 @@ const char *mptcp_get_scheduler(const struct net *net)
>>  
>>  static void mptcp_pernet_set_defaults(struct mptcp_pernet *pernet)
>>  {
>> -	pernet->mptcp_enabled = 1;
>> +	pernet->mptcp_enabled = 0;
>>  	pernet->add_addr_timeout = TCP_RTO_MAX;
>>  	pernet->blackhole_timeout = 3600;
>>  	atomic_set(&pernet->active_disable_times, 0);
> First, I have the same question as the one I asked to RedHat devs: do
> you still need to keep MPTCP disabled by default? If I remember well, on
> RHEL side, they started to do that when they backported MPTCP on a
> previous stable version, as an experimental feature. They left it like
> that later mostly for internal process reasons I think. But honestly,
> today, it no longer makes sense to do that and annoys users: all other
> Linux distributions enable MPTCP by default without patching the kernel
> like you did.

We had observed issues with mptcpd daemon failing before when we had
this enabled by default.
The mtpcpd userspace fix is yet to be integrated. However, shouldn't the
testcase be robust enough to handle that scenario regardless?

>
> If you don't want to revert this patch, I guess you can modify the BPF
> selftests in 'prog_tests/mptcp.c' to set 'sysctl net.mptcp.enabled=1' in
> each netns created by the test. But again, not changing the default
> kernel behaviour sounds like a better solution.

Even after changing /etc/sysctl.conf which is supposed to keep mptcp
enabled across reboots this issue occurs.
I agree with what you have stated, mptcp should be enabled by default
and the userspace fix should be incorporated ideally, however I still
believe that the test case shouldn't be giving a false negative as it is
in this case.

The false negatives seem to be occurring since this commit I believe:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=df8d3ba55b4fa1d6aed8449971ee50757cb0732f
This does the opposite of you have mentioned in certain functions. I
suppose adding all these lines back might do the trick:

ip netns exec $netns sysctl -q net.mptcp.enabled=1

>
> Cheers,
> Matt

Thanks & Regards,
Harshvardhan

Re: BPF selftest: mptcp subtest failing

[Matthieu Baerts]

Hi Harshvardhan,

On 08/08/2025 09:00, Harshvardhan Jha wrote:
> Hi Matthieu,
> 
> On 07/08/25 7:51 PM, Matthieu Baerts wrote:
>> Hi Harshvardhan,
>>
>> On 07/08/2025 05:50, Harshvardhan Jha wrote:
>>> Hi there,
>>> I have explicitly disabled mptpcp by default on my custom kernel and
>>> this seems to be causing the test case to fail. Even after enabling
>>> mtpcp via sysctl command or adding an entry to /etc/sysctl.conf this
>>> fails. I don't think this test should be failing and should account for
>>> cases where mptcp has not been enabled by default?
>> It looks like the test is failing because it expects MPTCP to be enabled
>> by default. Or, said differently, it doesn't expect the kernel to be
>> modified without adapting the corresponding tests :)
>>
>>> This is the custom patch I had applied on the LTS v6.12.36 kernel and
>>> tested it:
>>>
>>> diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c
>>> index dd595d9b5e50c..bdcc4136e92ef 100644
>>> --- a/net/mptcp/ctrl.c
>>> +++ b/net/mptcp/ctrl.c
>>> @@ -89,7 +89,7 @@ const char *mptcp_get_scheduler(const struct net *net)
>>>  
>>>  static void mptcp_pernet_set_defaults(struct mptcp_pernet *pernet)
>>>  {
>>> -	pernet->mptcp_enabled = 1;
>>> +	pernet->mptcp_enabled = 0;
>>>  	pernet->add_addr_timeout = TCP_RTO_MAX;
>>>  	pernet->blackhole_timeout = 3600;
>>>  	atomic_set(&pernet->active_disable_times, 0);
>> First, I have the same question as the one I asked to RedHat devs: do
>> you still need to keep MPTCP disabled by default? If I remember well, on
>> RHEL side, they started to do that when they backported MPTCP on a
>> previous stable version, as an experimental feature. They left it like
>> that later mostly for internal process reasons I think. But honestly,
>> today, it no longer makes sense to do that and annoys users: all other
>> Linux distributions enable MPTCP by default without patching the kernel
>> like you did.
> 
> We had observed issues with mptcpd daemon failing before when we had
> this enabled by default. The mtpcpd userspace fix is yet to be integrated.

If net.mptcp.enabled is set to 0 by default, I guess most mptcpd unit
tests will be skipped. If you have issues when MPTCP is enabled by
default, there might be real issues in mptcpd that would need to be
fixed (or more likely, RHEL devs didn't noticed most tests were skipped,
and there are probably some dependences missing for these tests).

But that's a different topic.

> However, shouldn't the testcase be robust enough to handle that scenario
> regardless?

It should not be needed: these tests run in a dedicated netns where it
expects the kernel to have MPTCP enabled by default. If this behaviour
is changed without adapting the tests, that's normal to have issues.
This is not an issue with the upstream kernel.

>> If you don't want to revert this patch, I guess you can modify the BPF
>> selftests in 'prog_tests/mptcp.c' to set 'sysctl net.mptcp.enabled=1' in
>> each netns created by the test. But again, not changing the default
>> kernel behaviour sounds like a better solution.
> 
> Even after changing /etc/sysctl.conf which is supposed to keep mptcp
> enabled across reboots this issue occurs.

/etc/sysctl.conf needs a userspace daemon to load it and apply the
changes. In these tests, dedicated netns are created, and this file is
not read. That's the whole purpose of using netns: not to have to handle
default config changed on the host, just use the default values from the
kernel. Plus, it is cleaner to use netns: no need to revert changes
after, tests can be executed in parallel without impacting others, etc.

> I agree with what you have stated, mptcp should be enabled by default
> and the userspace fix should be incorporated ideally, however I still
> believe that the test case shouldn't be giving a false negative as it is
> in this case.

I don't think these tests have to handle every possible modifications
done by a custom kernel. If you have to modify the behaviour, then you
also need to adapt everything related to that. That's why I don't
recommend to change the behaviour.

> The false negatives seem to be occurring since this commit I believe:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=df8d3ba55b4fa1d6aed8449971ee50757cb0732f

I don't think so, this commit changes MPTCP selftests, not the BPF ones
you have issues with. I guess you wanted to refer to this commit:

  02d6a057c7be ("selftests/bpf: run mptcp in a dedicated netns")

> This does the opposite of you have mentioned in certain functions.

The MPTCP selftests are setting net.mptcp.enabled=1, because RHEL devs
added them a while ago, while it was making sense, and we lefted them
there, even after the refactoring you mentioned. But we should probably
remove them indeed, because it is not needed.
> I suppose adding all these lines back might do the trick:
> 
> ip netns exec $netns sysctl -q net.mptcp.enabled=1

For your issue, yes. But again, not changing the default kernel
behaviour sounds like a better solution.

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.