CI failures on nipa

CI failures on nipa

[Paolo Abeni]

Hi,

I'm looking at this one:

https://netdev-ctrl.bots.linux.dev/logs/vmksft/mptcp/results/776501/6-mptcp-connect-sh/

the output is strange:

https://netdev-ctrl.bots.linux.dev/logview.html?f=/logs/vmksft/mptcp/results/776501/6-mptcp-connect-sh/stdout#L114

the first nstat dump includes apparently the counters from all the
previous runs, which is unexpected: the nstat_init/nstat_get/pr_nstat
dance should allow showing the data from current run only right?

Also I think mptcp_lib_wait_timeout completes too late (after ~60 secs),
when the listener already bailed out due to accept timeout (after ~30
secs), so it does not show any info on the listener side. I fear it may
also race with the actuall `timeout` command completion showing socket
states after the user-space program completion.

Judging on the final status, I *think*/*guess* the (re)connect is
hanged, possibly due to bad remote address?!? It could be useful to
print out on stderr the reconnect destination address.

/P

Re: CI failures on nipa

[gang.yan]

August 13, 2026 at 3:57 PM, "Paolo Abeni" <pabeni@redhat.com mailto:pabeni@redhat.com?to=%22Paolo%20Abeni%22%20%3Cpabeni%40redhat.com%3E > wrote:


> 
> Hi,
> 
> I'm looking at this one:
> 
> https://netdev-ctrl.bots.linux.dev/logs/vmksft/mptcp/results/776501/6-mptcp-connect-sh/
> 
> the output is strange:
> 
> https://netdev-ctrl.bots.linux.dev/logview.html?f=/logs/vmksft/mptcp/results/776501/6-mptcp-connect-sh/stdout#L114
> 
> the first nstat dump includes apparently the counters from all the
> previous runs, which is unexpected: the nstat_init/nstat_get/pr_nstat
> dance should allow showing the data from current run only right?
> 
> Also I think mptcp_lib_wait_timeout completes too late (after ~60 secs),
> when the listener already bailed out due to accept timeout (after ~30
> secs), so it does not show any info on the listener side. I fear it may
> also race with the actuall `timeout` command completion showing socket
> states after the user-space program completion.
> 
> Judging on the final status, I *think*/*guess* the (re)connect is
> hanged, possibly due to bad remote address?!? It could be useful to
> print out on stderr the reconnect destination address.

Hi Paolo,

That reminds me of an issue when I was reading mptcp_connect.c a long
time ago:

At the end of 'sock_connect_mptcp()', it calls 'freeaddrinfo(addr)',
the 'peer' pointer (which points into 'addr') remains. Later, the main loop
uses this peer pointer for reconnection attempts. If the memory has been freed
and reused, the address data could be overwritten, resulting in an invalid
remote address.

That might cause the hangs or failures you are seeing, right?

Thanks
Gang

> 
> /P
>

Re: CI failures on nipa

[Matthieu Baerts]

Hi Gang,

On 13/08/2026 10:59, gang.yan@linux.dev wrote:
> August 13, 2026 at 3:57 PM, "Paolo Abeni" <pabeni@redhat.com mailto:pabeni@redhat.com?to=%22Paolo%20Abeni%22%20%3Cpabeni%40redhat.com%3E > wrote:

(...)

>> Judging on the final status, I *think*/*guess* the (re)connect is
>> hanged, possibly due to bad remote address?!? It could be useful to
>> print out on stderr the reconnect destination address.
> 
> Hi Paolo,
> 
> That reminds me of an issue when I was reading mptcp_connect.c a long
> time ago:
> 
> At the end of 'sock_connect_mptcp()', it calls 'freeaddrinfo(addr)',
> the 'peer' pointer (which points into 'addr') remains. Later, the main loop
> uses this peer pointer for reconnection attempts. If the memory has been freed
> and reused, the address data could be overwritten, resulting in an invalid
> remote address.
> 
> That might cause the hangs or failures you are seeing, right?

Good catch!

Are you working on a fix for that? I guess we should copy the content,
not just the pointer. Or maybe we don't need to do this copy:
getaddrinfo() should return the same thing, no?

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: CI failures on nipa

[gang.yan]

August 13, 2026 at 5:20 PM, "Matthieu Baerts" <matttbe@kernel.org mailto:matttbe@kernel.org?to=%22Matthieu%20Baerts%22%20%3Cmatttbe%40kernel.org%3E > wrote:


> 
> Hi Gang,
> 
> On 13/08/2026 10:59, gang.yan@linux.dev wrote:
> 
> > 
> > August 13, 2026 at 3:57 PM, "Paolo Abeni" <pabeni@redhat.com mailto:pabeni@redhat.com?to=%22Paolo%20Abeni%22%20%3Cpabeni%40redhat.com%3E > wrote:
> > 
> (...)
> 
> > 
> > > 
> > > Judging on the final status, I *think*/*guess* the (re)connect is
> > >  hanged, possibly due to bad remote address?!? It could be useful to
> > >  print out on stderr the reconnect destination address.
> > > 
> >  
> >  Hi Paolo,
> >  
> >  That reminds me of an issue when I was reading mptcp_connect.c a long
> >  time ago:
> >  
> >  At the end of 'sock_connect_mptcp()', it calls 'freeaddrinfo(addr)',
> >  the 'peer' pointer (which points into 'addr') remains. Later, the main loop
> >  uses this peer pointer for reconnection attempts. If the memory has been freed
> >  and reused, the address data could be overwritten, resulting in an invalid
> >  remote address.
> >  
> >  That might cause the hangs or failures you are seeing, right?
> > 
> Good catch!
> 
> Are you working on a fix for that? I guess we should copy the content,
> not just the pointer. Or maybe we don't need to do this copy:
> getaddrinfo() should return the same thing, no?

Hi Matt

I can working on this right now – and with the help of AI, I believe we can resolve
it quickly. :)

Thanks
Gang

> 
> Cheers,
> Matt
> -- 
> Sponsored by the NGI0 Core fund.
>

Re: CI failures on nipa

[Matthieu Baerts]

Hi Paolo,

On 13/08/2026 09:57, Paolo Abeni wrote:
> I'm looking at this one:
> 
> https://netdev-ctrl.bots.linux.dev/logs/vmksft/mptcp/results/776501/6-mptcp-connect-sh/

I was looking at that too, last night between two mosquitoes :)
> the output is strange:
> 
> https://netdev-ctrl.bots.linux.dev/logview.html?f=/logs/vmksft/mptcp/results/776501/6-mptcp-connect-sh/stdout#L114
> 
> the first nstat dump includes apparently the counters from all the
> previous runs, which is unexpected: the nstat_init/nstat_get/pr_nstat
> dance should allow showing the data from current run only right?

Ah yes, indeed. mptcp_lib_pr_nstat() is not looking at the nstat
history, taking data from the beginning instead of the last time
nstat_init was executed (same for mptcp_lib_get_counter()).

I can send a fix for that.

> Also I think mptcp_lib_wait_timeout completes too late (after ~60 secs),
> when the listener already bailed out due to accept timeout (after ~30
> secs), so it does not show any info on the listener side. I fear it may
> also race with the actuall `timeout` command completion showing socket
> states after the user-space program completion.

Do you mean you would like to have mptcp_lib_wait_timeout wait for less
than the accept timeout?

mptcp_lib_wait_timeout has been introduced to dump the counters when
mptcp_connect stalls, and even the poll timeout didn't help.

What about modifying mptcp_connect.c to dump counters -- only for its
netns -- before exiting in case of timeout?

> Judging on the final status, I *think*/*guess* the (re)connect is
> hanged, possibly due to bad remote address?!? It could be useful to
> print out on stderr the reconnect destination address.
Strange why would it be only visible now? But yes, as Gang said, it
looks like a free is done on the addresses in between.

(When I checked last night, I was initially thinking about an issue with
a Netdev pending patch -- maybe even one related to TCP as it was
visible even with connections from TCP to MPTCP -- because this wasn't
visible on our CI, but maybe not.)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.