NIPA: TDC: delay the start

NIPA: TDC: delay the start

[Matthieu Baerts]

Hi Pedro,

I hope you are doing well.

I'm helping the netdev maintainers with the CI (NIPA), and lately, I
noticed the 'tdc' runners were often marked as either "timeout" or
"pending". It looks like this is because it missed the creation of the
branch which can take more than 2 minutes for various reasons: PW access
is slow, git push needs to be retried in a few seconds, etc.

I opened a PR to delay the start:

  https://github.com/p4tc-dev/tc-executor/pull/2

Another solution is to let the tdc* services running and doing the
monitoring themselves. In other words, you need to remove
"single_shot=true" from the config file. It might require you to update
the NIPA repo you are using to get access to a bunch of fixes.

Cheers,
Matt

Re: NIPA: TDC: delay the start

[Jakub Kicinski]

On Thu, 5 Nov 2026 11:43:27 +0100 Matthieu Baerts wrote:
> Hi Pedro,
> 
> I hope you are doing well.
> 
> I'm helping the netdev maintainers with the CI (NIPA), and lately, I
> noticed the 'tdc' runners were often marked as either "timeout" or
> "pending". It looks like this is because it missed the creation of the
> branch which can take more than 2 minutes for various reasons: PW access
> is slow, git push needs to be retried in a few seconds, etc.
> 
> I opened a PR to delay the start:
> 
>   https://github.com/p4tc-dev/tc-executor/pull/2
> 
> Another solution is to let the tdc* services running and doing the
> monitoring themselves. In other words, you need to remove
> "single_shot=true" from the config file. It might require you to update
> the NIPA repo you are using to get access to a bunch of fixes.

Keeps happening, I dropped TDC from required runners for now.

Re: NIPA: TDC: delay the start

[Jamal Hadi Salim]

,

On Fri, Nov 6, 2026 at 12:28 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Thu, 5 Nov 2026 11:43:27 +0100 Matthieu Baerts wrote:
> > Hi Pedro,
> >
> > I hope you are doing well.
> >
> > I'm helping the netdev maintainers with the CI (NIPA), and lately, I
> > noticed the 'tdc' runners were often marked as either "timeout" or
> > "pending". It looks like this is because it missed the creation of the
> > branch which can take more than 2 minutes for various reasons: PW access
> > is slow, git push needs to be retried in a few seconds, etc.
> >
> > I opened a PR to delay the start:
> >
> >   https://github.com/p4tc-dev/tc-executor/pull/2
> >
> > Another solution is to let the tdc* services running and doing the
> > monitoring themselves. In other words, you need to remove
> > "single_shot=true" from the config file. It might require you to update
> > the NIPA repo you are using to get access to a bunch of fixes.
>
> Keeps happening, I dropped TDC from required runners for now.

This was an issue related to the indeterminate time it takes for the
branch to show up after creation.
Matthieu, Victor responded to you; the overall proposal you made looks
good. I don't know how we can fix this, but since Pedro is currently
away we saw this mention by pure luck: Can we change the contact email
address to hybris@mojatatu.ai? That way if you send an email there,
one of several people will see it and respond.

cheers,
jamal

Re: NIPA: TDC: delay the start

[Matthieu Baerts]

Hi Jamal,

Thank you for your reply!

On 06/11/2026 22:39, Jamal Hadi Salim wrote:
> ,
> 
> On Fri, Nov 6, 2026 at 12:28 PM Jakub Kicinski <kuba@kernel.org> wrote:
>>
>> On Thu, 5 Nov 2026 11:43:27 +0100 Matthieu Baerts wrote:
>>> Hi Pedro,
>>>
>>> I hope you are doing well.
>>>
>>> I'm helping the netdev maintainers with the CI (NIPA), and lately, I
>>> noticed the 'tdc' runners were often marked as either "timeout" or
>>> "pending". It looks like this is because it missed the creation of the
>>> branch which can take more than 2 minutes for various reasons: PW access
>>> is slow, git push needs to be retried in a few seconds, etc.
>>>
>>> I opened a PR to delay the start:
>>>
>>>   https://github.com/p4tc-dev/tc-executor/pull/2
>>>
>>> Another solution is to let the tdc* services running and doing the
>>> monitoring themselves. In other words, you need to remove
>>> "single_shot=true" from the config file. It might require you to update
>>> the NIPA repo you are using to get access to a bunch of fixes.
>>
>> Keeps happening, I dropped TDC from required runners for now.
> 
> This was an issue related to the indeterminate time it takes for the
> branch to show up after creation.

Just to put a bit more context, having no results from a runner means
the CI cannot validate whether the batch being tested was passing all
the tests or not. So we need to wait 3 more hours to get results. It's
not dramatic, but here it happens regularly, and it should be easy to fix.

> Matthieu, Victor responded to you; the overall proposal you made looks
> good. I don't know how we can fix this, but since Pedro is currently
> away we saw this mention by pure luck: Can we change the contact email
> address to hybris@mojatatu.ai? That way if you send an email there,
> one of several people will see it and respond.

Good idea, I just changed that!

Please ping me when it is fixed, so I can re-enable the runner on NIPA's
side.

Cheers,
Matt

Re: NIPA: TDC: delay the start

[Jamal Hadi Salim]

On Mon, Nov 9, 2026 at 11:58 AM Matthieu Baerts <matttbe@kernel.org> wrote:
>
> Hi Jamal,
>
> Thank you for your reply!
>
> On 06/11/2026 22:39, Jamal Hadi Salim wrote:
> > ,
> >
> > On Fri, Nov 6, 2026 at 12:28 PM Jakub Kicinski <kuba@kernel.org> wrote:
> >>
> >> On Thu, 5 Nov 2026 11:43:27 +0100 Matthieu Baerts wrote:
> >>> Hi Pedro,
> >>>
> >>> I hope you are doing well.
> >>>
> >>> I'm helping the netdev maintainers with the CI (NIPA), and lately, I
> >>> noticed the 'tdc' runners were often marked as either "timeout" or
> >>> "pending". It looks like this is because it missed the creation of the
> >>> branch which can take more than 2 minutes for various reasons: PW access
> >>> is slow, git push needs to be retried in a few seconds, etc.
> >>>
> >>> I opened a PR to delay the start:
> >>>
> >>>   https://github.com/p4tc-dev/tc-executor/pull/2
> >>>
> >>> Another solution is to let the tdc* services running and doing the
> >>> monitoring themselves. In other words, you need to remove
> >>> "single_shot=true" from the config file. It might require you to update
> >>> the NIPA repo you are using to get access to a bunch of fixes.
> >>
> >> Keeps happening, I dropped TDC from required runners for now.
> >
> > This was an issue related to the indeterminate time it takes for the
> > branch to show up after creation.
>
> Just to put a bit more context, having no results from a runner means
> the CI cannot validate whether the batch being tested was passing all
> the tests or not. So we need to wait 3 more hours to get results. It's
> not dramatic, but here it happens regularly, and it should be easy to fix.
>

Agreed - we should have caught this sooner. We basically claim
"nothing to see here" because the branch update hasnt fully propagated
(although some of the time we catch the branch propagation)

> > Matthieu, Victor responded to you; the overall proposal you made looks
> > good. I don't know how we can fix this, but since Pedro is currently
> > away we saw this mention by pure luck: Can we change the contact email
> > address to hybris@mojatatu.ai? That way if you send an email there,
> > one of several people will see it and respond.
>
> Good idea, I just changed that!
>
> Please ping me when it is fixed, so I can re-enable the runner on NIPA's
> side.

@Victor Nogueira  is the one fixing it..

cheers,
jamal

Re: NIPA: TDC: delay the start

[Matthieu Baerts]

On 09/11/2026 18:22, Jamal Hadi Salim wrote:
> On Mon, Nov 9, 2026 at 11:58 AM Matthieu Baerts <matttbe@kernel.org> wrote:
>>
>> Hi Jamal,
>>
>> Thank you for your reply!
>>
>> On 06/11/2026 22:39, Jamal Hadi Salim wrote:
>>> ,
>>>
>>> On Fri, Nov 6, 2026 at 12:28 PM Jakub Kicinski <kuba@kernel.org> wrote:
>>>>
>>>> On Thu, 5 Nov 2026 11:43:27 +0100 Matthieu Baerts wrote:
>>>>> Hi Pedro,
>>>>>
>>>>> I hope you are doing well.
>>>>>
>>>>> I'm helping the netdev maintainers with the CI (NIPA), and lately, I
>>>>> noticed the 'tdc' runners were often marked as either "timeout" or
>>>>> "pending". It looks like this is because it missed the creation of the
>>>>> branch which can take more than 2 minutes for various reasons: PW access
>>>>> is slow, git push needs to be retried in a few seconds, etc.
>>>>>
>>>>> I opened a PR to delay the start:
>>>>>
>>>>>   https://github.com/p4tc-dev/tc-executor/pull/2
>>>>>
>>>>> Another solution is to let the tdc* services running and doing the
>>>>> monitoring themselves. In other words, you need to remove
>>>>> "single_shot=true" from the config file. It might require you to update
>>>>> the NIPA repo you are using to get access to a bunch of fixes.
>>>>
>>>> Keeps happening, I dropped TDC from required runners for now.
>>>
>>> This was an issue related to the indeterminate time it takes for the
>>> branch to show up after creation.
>>
>> Just to put a bit more context, having no results from a runner means
>> the CI cannot validate whether the batch being tested was passing all
>> the tests or not. So we need to wait 3 more hours to get results. It's
>> not dramatic, but here it happens regularly, and it should be easy to fix.
>>
> 
> Agreed - we should have caught this sooner. We basically claim
> "nothing to see here" because the branch update hasnt fully propagated
> (although some of the time we catch the branch propagation)

Indeed. Please note that what I proposed in the PR was a quick fix. As I
mentioned earlier, I think the best is probably to let the service doing
the polling regularly, just in case branches are created later than
usual due to other issues. (But then, it is certainly better to switch
to the latest version of these scripts. I guess the update should not
cause issues, but it needs to be tested)

>>> Matthieu, Victor responded to you; the overall proposal you made looks
>>> good. I don't know how we can fix this, but since Pedro is currently
>>> away we saw this mention by pure luck: Can we change the contact email
>>> address to hybris@mojatatu.ai? That way if you send an email there,
>>> one of several people will see it and respond.
>>
>> Good idea, I just changed that!
>>
>> Please ping me when it is fixed, so I can re-enable the runner on NIPA's
>> side.
> 
> @Victor Nogueira  is the one fixing it..

Thank you!

Cheers,
Matt

Re: NIPA: TDC: delay the start

[Victor Nogueira]

Hi Matthieu,

On Mon, Nov 9, 2026 at 2:31 PM Matthieu Baerts <matttbe@kernel.org> wrote:
>
> On 09/11/2026 18:22, Jamal Hadi Salim wrote:
> > On Mon, Nov 9, 2026 at 11:58 AM Matthieu Baerts <matttbe@kernel.org> wrote:
> >>
> >> Hi Jamal,
> >>
> >> Thank you for your reply!
> >>
> >> On 06/11/2026 22:39, Jamal Hadi Salim wrote:
> >>> ,
> >>>
> >>> On Fri, Nov 6, 2026 at 12:28 PM Jakub Kicinski <kuba@kernel.org> wrote:
> >>>>
> >>>> On Thu, 5 Nov 2026 11:43:27 +0100 Matthieu Baerts wrote:
> >>>>> Hi Pedro,
> >>>>>
> >>>>> I hope you are doing well.
> >>>>>
> >>>>> I'm helping the netdev maintainers with the CI (NIPA), and lately, I
> >>>>> noticed the 'tdc' runners were often marked as either "timeout" or
> >>>>> "pending". It looks like this is because it missed the creation of the
> >>>>> branch which can take more than 2 minutes for various reasons: PW access
> >>>>> is slow, git push needs to be retried in a few seconds, etc.
> >>>>>
> >>>>> I opened a PR to delay the start:
> >>>>>
> >>>>>   https://github.com/p4tc-dev/tc-executor/pull/2
>>>>>> [...]
> >>
> >> Good idea, I just changed that!
> >>
> >> Please ping me when it is fixed, so I can re-enable the runner on NIPA's
> >> side.
> >
> > @Victor Nogueira  is the one fixing it..

I applied the change Saturday and it's been running fine since then.
I think it's safe to re-enable now.

cheers,
Victor

Re: NIPA: TDC: delay the start

[Matthieu Baerts]

Hi Victor,

On 09/11/2026 18:39, Victor Nogueira wrote:
> Hi Matthieu,
> 
> On Mon, Nov 9, 2026 at 2:31 PM Matthieu Baerts <matttbe@kernel.org> wrote:
>>
>> On 09/11/2026 18:22, Jamal Hadi Salim wrote:
>>> On Mon, Nov 9, 2026 at 11:58 AM Matthieu Baerts <matttbe@kernel.org> wrote:
>>>>
>>>> Hi Jamal,
>>>>
>>>> Thank you for your reply!
>>>>
>>>> On 06/11/2026 22:39, Jamal Hadi Salim wrote:
>>>>> ,
>>>>>
>>>>> On Fri, Nov 6, 2026 at 12:28 PM Jakub Kicinski <kuba@kernel.org> wrote:
>>>>>>
>>>>>> On Thu, 5 Nov 2026 11:43:27 +0100 Matthieu Baerts wrote:
>>>>>>> Hi Pedro,
>>>>>>>
>>>>>>> I hope you are doing well.
>>>>>>>
>>>>>>> I'm helping the netdev maintainers with the CI (NIPA), and lately, I
>>>>>>> noticed the 'tdc' runners were often marked as either "timeout" or
>>>>>>> "pending". It looks like this is because it missed the creation of the
>>>>>>> branch which can take more than 2 minutes for various reasons: PW access
>>>>>>> is slow, git push needs to be retried in a few seconds, etc.
>>>>>>>
>>>>>>> I opened a PR to delay the start:
>>>>>>>
>>>>>>>   https://github.com/p4tc-dev/tc-executor/pull/2
>>>>>>> [...]
>>>>
>>>> Good idea, I just changed that!
>>>>
>>>> Please ping me when it is fixed, so I can re-enable the runner on NIPA's
>>>> side.
>>>
>>> @Victor Nogueira  is the one fixing it..
> 
> I applied the change Saturday and it's been running fine since then.

Ah good, thank you, I thought I had to wait because the PR was not merged.

> I think it's safe to re-enable now.

Just did!

Cheers,
Matt

Re: NIPA: TDC: delay the start

[Matthieu Baerts]

On 09/11/2026 19:40, Matthieu Baerts wrote:
> Hi Victor,
> 
> On 09/11/2026 18:39, Victor Nogueira wrote:
>> Hi Matthieu,
>>
>> On Mon, Nov 9, 2026 at 2:31 PM Matthieu Baerts <matttbe@kernel.org> wrote:
>>>
>>> On 09/11/2026 18:22, Jamal Hadi Salim wrote:
>>>> On Mon, Nov 9, 2026 at 11:58 AM Matthieu Baerts <matttbe@kernel.org> wrote:
>>>>>
>>>>> Hi Jamal,
>>>>>
>>>>> Thank you for your reply!
>>>>>
>>>>> On 06/11/2026 22:39, Jamal Hadi Salim wrote:
>>>>>> ,
>>>>>>
>>>>>> On Fri, Nov 6, 2026 at 12:28 PM Jakub Kicinski <kuba@kernel.org> wrote:
>>>>>>>
>>>>>>> On Thu, 5 Nov 2026 11:43:27 +0100 Matthieu Baerts wrote:
>>>>>>>> Hi Pedro,
>>>>>>>>
>>>>>>>> I hope you are doing well.
>>>>>>>>
>>>>>>>> I'm helping the netdev maintainers with the CI (NIPA), and lately, I
>>>>>>>> noticed the 'tdc' runners were often marked as either "timeout" or
>>>>>>>> "pending". It looks like this is because it missed the creation of the
>>>>>>>> branch which can take more than 2 minutes for various reasons: PW access
>>>>>>>> is slow, git push needs to be retried in a few seconds, etc.
>>>>>>>>
>>>>>>>> I opened a PR to delay the start:
>>>>>>>>
>>>>>>>>   https://github.com/p4tc-dev/tc-executor/pull/2
>>>>>>>> [...]
>>>>>
>>>>> Good idea, I just changed that!
>>>>>
>>>>> Please ping me when it is fixed, so I can re-enable the runner on NIPA's
>>>>> side.
>>>>
>>>> @Victor Nogueira  is the one fixing it..
>>
>> I applied the change Saturday and it's been running fine since then.
> 
> Ah good, thank you, I thought I had to wait because the PR was not merged.
Oh, sorry, it looks like I missed the last notification, the PR has been
merged! All good then :)

Cheers,
Matt