Re: xen | Failed pipeline for staging-4.18 | 51190865

Re: xen | Failed pipeline for staging-4.18 | 51190865

[Jan Beulich]

On 03.09.2025 17:46, GitLab wrote:
> 
> 
> Pipeline #2019390073 has failed!
> 
> Project: xen ( https://gitlab.com/xen-project/hardware/xen )
> Branch: staging-4.18 ( https://gitlab.com/xen-project/hardware/xen/-/commits/staging-4.18 )
> 
> Commit: 51190865 ( https://gitlab.com/xen-project/hardware/xen/-/commit/51190865a4918c443c310c0478247d5f9caa5dad )
> Commit Message: x86/suspend: unconditionally raise a timer soft...
> Commit Author: Roger Pau Monné
> Committed by: Jan Beulich ( https://gitlab.com/jbeulich )
> 
> 
> Pipeline #2019390073 ( https://gitlab.com/xen-project/hardware/xen/-/pipelines/2019390073 ) triggered by Jan Beulich ( https://gitlab.com/jbeulich )
> had 5 failed jobs.
> 
> Job #11230955404 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955404/raw )
> 
> Stage: test
> Name: adl-suspend-x86-64-gcc-debug
> Job #11230955410 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955410/raw )
> 
> Stage: test
> Name: adl-pci-pv-x86-64-gcc-debug
> Job #11230955417 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955417/raw )
> 
> Stage: test
> Name: adl-pci-hvm-x86-64-gcc-debug
> Job #11233274365 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233274365/raw )
> 
> Stage: test
> Name: adl-smoke-x86-64-gcc-debug
> Job #11233405609 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233405609/raw )
> 
> Stage: test
> Name: adl-smoke-x86-64-dom0pvh-gcc-debug

While the same tests are fine for 4.19 and 4.20, all five show rubbish in the log,
and then fail. No idea what's going on.

Jan

Re: xen | Failed pipeline for staging-4.18 | 51190865

[Marek Marczykowski]

[-- Attachment #1: Type: text/plain, Size: 2121 bytes --]

On Wed, Sep 03, 2025 at 05:58:32PM +0200, Jan Beulich wrote:
> On 03.09.2025 17:46, GitLab wrote:
> > 
> > 
> > Pipeline #2019390073 has failed!
> > 
> > Project: xen ( https://gitlab.com/xen-project/hardware/xen )
> > Branch: staging-4.18 ( https://gitlab.com/xen-project/hardware/xen/-/commits/staging-4.18 )
> > 
> > Commit: 51190865 ( https://gitlab.com/xen-project/hardware/xen/-/commit/51190865a4918c443c310c0478247d5f9caa5dad )
> > Commit Message: x86/suspend: unconditionally raise a timer soft...
> > Commit Author: Roger Pau Monné
> > Committed by: Jan Beulich ( https://gitlab.com/jbeulich )
> > 
> > 
> > Pipeline #2019390073 ( https://gitlab.com/xen-project/hardware/xen/-/pipelines/2019390073 ) triggered by Jan Beulich ( https://gitlab.com/jbeulich )
> > had 5 failed jobs.
> > 
> > Job #11230955404 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955404/raw )
> > 
> > Stage: test
> > Name: adl-suspend-x86-64-gcc-debug
> > Job #11230955410 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955410/raw )
> > 
> > Stage: test
> > Name: adl-pci-pv-x86-64-gcc-debug
> > Job #11230955417 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955417/raw )
> > 
> > Stage: test
> > Name: adl-pci-hvm-x86-64-gcc-debug
> > Job #11233274365 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233274365/raw )
> > 
> > Stage: test
> > Name: adl-smoke-x86-64-gcc-debug
> > Job #11233405609 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233405609/raw )
> > 
> > Stage: test
> > Name: adl-smoke-x86-64-dom0pvh-gcc-debug
> 
> While the same tests are fine for 4.19 and 4.20, all five show rubbish in the log,
> and then fail. No idea what's going on.

The log says "baudrate is    : 115200", but looking at the state after
the test I see 9600. No idea if that was simply switched back after, or
setting to 115200 didn't work. Anyway I suggest to restart (now that
other jobs completed). I set it manually to 115200 now too (not sure if
that will remain there...).

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: xen | Failed pipeline for staging-4.18 | 51190865

[Jan Beulich]

On 03.09.2025 18:03, Marek Marczykowski wrote:
> On Wed, Sep 03, 2025 at 05:58:32PM +0200, Jan Beulich wrote:
>> On 03.09.2025 17:46, GitLab wrote:
>>>
>>>
>>> Pipeline #2019390073 has failed!
>>>
>>> Project: xen ( https://gitlab.com/xen-project/hardware/xen )
>>> Branch: staging-4.18 ( https://gitlab.com/xen-project/hardware/xen/-/commits/staging-4.18 )
>>>
>>> Commit: 51190865 ( https://gitlab.com/xen-project/hardware/xen/-/commit/51190865a4918c443c310c0478247d5f9caa5dad )
>>> Commit Message: x86/suspend: unconditionally raise a timer soft...
>>> Commit Author: Roger Pau Monné
>>> Committed by: Jan Beulich ( https://gitlab.com/jbeulich )
>>>
>>>
>>> Pipeline #2019390073 ( https://gitlab.com/xen-project/hardware/xen/-/pipelines/2019390073 ) triggered by Jan Beulich ( https://gitlab.com/jbeulich )
>>> had 5 failed jobs.
>>>
>>> Job #11230955404 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955404/raw )
>>>
>>> Stage: test
>>> Name: adl-suspend-x86-64-gcc-debug
>>> Job #11230955410 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955410/raw )
>>>
>>> Stage: test
>>> Name: adl-pci-pv-x86-64-gcc-debug
>>> Job #11230955417 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955417/raw )
>>>
>>> Stage: test
>>> Name: adl-pci-hvm-x86-64-gcc-debug
>>> Job #11233274365 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233274365/raw )
>>>
>>> Stage: test
>>> Name: adl-smoke-x86-64-gcc-debug
>>> Job #11233405609 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233405609/raw )
>>>
>>> Stage: test
>>> Name: adl-smoke-x86-64-dom0pvh-gcc-debug
>>
>> While the same tests are fine for 4.19 and 4.20, all five show rubbish in the log,
>> and then fail. No idea what's going on.
> 
> The log says "baudrate is    : 115200", but looking at the state after
> the test I see 9600. No idea if that was simply switched back after, or
> setting to 115200 didn't work. Anyway I suggest to restart (now that
> other jobs completed). I set it manually to 115200 now too (not sure if
> that will remain there...).

The rubbish in the output looks to have gone away, but the adl-* tests fail
as before. I'm retrying two of them another time, but with little hope.

Jan

Re: xen | Failed pipeline for staging-4.18 | 51190865

[Jan Beulich]

On 04.09.2025 07:58, Jan Beulich wrote:
> On 03.09.2025 18:03, Marek Marczykowski wrote:
>> On Wed, Sep 03, 2025 at 05:58:32PM +0200, Jan Beulich wrote:
>>> On 03.09.2025 17:46, GitLab wrote:
>>>>
>>>>
>>>> Pipeline #2019390073 has failed!
>>>>
>>>> Project: xen ( https://gitlab.com/xen-project/hardware/xen )
>>>> Branch: staging-4.18 ( https://gitlab.com/xen-project/hardware/xen/-/commits/staging-4.18 )
>>>>
>>>> Commit: 51190865 ( https://gitlab.com/xen-project/hardware/xen/-/commit/51190865a4918c443c310c0478247d5f9caa5dad )
>>>> Commit Message: x86/suspend: unconditionally raise a timer soft...
>>>> Commit Author: Roger Pau Monné
>>>> Committed by: Jan Beulich ( https://gitlab.com/jbeulich )
>>>>
>>>>
>>>> Pipeline #2019390073 ( https://gitlab.com/xen-project/hardware/xen/-/pipelines/2019390073 ) triggered by Jan Beulich ( https://gitlab.com/jbeulich )
>>>> had 5 failed jobs.
>>>>
>>>> Job #11230955404 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955404/raw )
>>>>
>>>> Stage: test
>>>> Name: adl-suspend-x86-64-gcc-debug
>>>> Job #11230955410 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955410/raw )
>>>>
>>>> Stage: test
>>>> Name: adl-pci-pv-x86-64-gcc-debug
>>>> Job #11230955417 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955417/raw )
>>>>
>>>> Stage: test
>>>> Name: adl-pci-hvm-x86-64-gcc-debug
>>>> Job #11233274365 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233274365/raw )
>>>>
>>>> Stage: test
>>>> Name: adl-smoke-x86-64-gcc-debug
>>>> Job #11233405609 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233405609/raw )
>>>>
>>>> Stage: test
>>>> Name: adl-smoke-x86-64-dom0pvh-gcc-debug
>>>
>>> While the same tests are fine for 4.19 and 4.20, all five show rubbish in the log,
>>> and then fail. No idea what's going on.
>>
>> The log says "baudrate is    : 115200", but looking at the state after
>> the test I see 9600. No idea if that was simply switched back after, or
>> setting to 115200 didn't work. Anyway I suggest to restart (now that
>> other jobs completed). I set it manually to 115200 now too (not sure if
>> that will remain there...).
> 
> The rubbish in the output looks to have gone away, but the adl-* tests fail
> as before. I'm retrying two of them another time, but with little hope.

As opposed to 4.19, where we have this

minimal cmds is: no
!! STDIN is not a TTY !! Continue anyway...
Type [C-a] [C-h] to see available commands
Terminal ready
 Xen 4.19.4-pre
(XEN) [00000043ae35c0c9] Xen version 4.19.4-pre (root@) (gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924) debug=y Wed Sep  3 12:15:19 UTC 2025

for 4.18 things (consistently across the tests) look like this

minimal cmds is: no
!! STDIN is not a TTY !! Continue anyway...
Type [C-a] [C-h] to see available commands
Terminal ready
Accessing Gembird #0 USB device 038
Switched outlet 2 on
Setting boot mode for 2 to gitlabci... done
+ trap 'ssh control@thor.testnet poweroff; : > /tmp/console-stdin' EXIT
+ '[' -n  ]
+ set +x
 Xen 4.18.5
(XEN) Xen version 4.18.5 (root@) (gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924) debug=y Wed Sep  3 12:27:30 UTC 2025

For 4.19 there's no visible delay between "Terminal ready" and the first Xen
message, whereas for 4.18 there is an approx 1:30min (±10s) delay after the
"+ set +x". That's a lot when the timeout is 2min.

Jan

Re: xen | Failed pipeline for staging-4.18 | 51190865

[Marek Marczykowski]

[-- Attachment #1: Type: text/plain, Size: 4170 bytes --]

On Thu, Sep 04, 2025 at 08:27:14AM +0200, Jan Beulich wrote:
> On 04.09.2025 07:58, Jan Beulich wrote:
> > On 03.09.2025 18:03, Marek Marczykowski wrote:
> >> On Wed, Sep 03, 2025 at 05:58:32PM +0200, Jan Beulich wrote:
> >>> On 03.09.2025 17:46, GitLab wrote:
> >>>>
> >>>>
> >>>> Pipeline #2019390073 has failed!
> >>>>
> >>>> Project: xen ( https://gitlab.com/xen-project/hardware/xen )
> >>>> Branch: staging-4.18 ( https://gitlab.com/xen-project/hardware/xen/-/commits/staging-4.18 )
> >>>>
> >>>> Commit: 51190865 ( https://gitlab.com/xen-project/hardware/xen/-/commit/51190865a4918c443c310c0478247d5f9caa5dad )
> >>>> Commit Message: x86/suspend: unconditionally raise a timer soft...
> >>>> Commit Author: Roger Pau Monné
> >>>> Committed by: Jan Beulich ( https://gitlab.com/jbeulich )
> >>>>
> >>>>
> >>>> Pipeline #2019390073 ( https://gitlab.com/xen-project/hardware/xen/-/pipelines/2019390073 ) triggered by Jan Beulich ( https://gitlab.com/jbeulich )
> >>>> had 5 failed jobs.
> >>>>
> >>>> Job #11230955404 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955404/raw )
> >>>>
> >>>> Stage: test
> >>>> Name: adl-suspend-x86-64-gcc-debug
> >>>> Job #11230955410 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955410/raw )
> >>>>
> >>>> Stage: test
> >>>> Name: adl-pci-pv-x86-64-gcc-debug
> >>>> Job #11230955417 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955417/raw )
> >>>>
> >>>> Stage: test
> >>>> Name: adl-pci-hvm-x86-64-gcc-debug
> >>>> Job #11233274365 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233274365/raw )
> >>>>
> >>>> Stage: test
> >>>> Name: adl-smoke-x86-64-gcc-debug
> >>>> Job #11233405609 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233405609/raw )
> >>>>
> >>>> Stage: test
> >>>> Name: adl-smoke-x86-64-dom0pvh-gcc-debug
> >>>
> >>> While the same tests are fine for 4.19 and 4.20, all five show rubbish in the log,
> >>> and then fail. No idea what's going on.
> >>
> >> The log says "baudrate is    : 115200", but looking at the state after
> >> the test I see 9600. No idea if that was simply switched back after, or
> >> setting to 115200 didn't work. Anyway I suggest to restart (now that
> >> other jobs completed). I set it manually to 115200 now too (not sure if
> >> that will remain there...).
> > 
> > The rubbish in the output looks to have gone away, but the adl-* tests fail
> > as before. I'm retrying two of them another time, but with little hope.
> 
> As opposed to 4.19, where we have this
> 
> minimal cmds is: no
> !! STDIN is not a TTY !! Continue anyway...
> Type [C-a] [C-h] to see available commands
> Terminal ready
>  Xen 4.19.4-pre
> (XEN) [00000043ae35c0c9] Xen version 4.19.4-pre (root@) (gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924) debug=y Wed Sep  3 12:15:19 UTC 2025
> 
> for 4.18 things (consistently across the tests) look like this
> 
> minimal cmds is: no
> !! STDIN is not a TTY !! Continue anyway...
> Type [C-a] [C-h] to see available commands
> Terminal ready
> Accessing Gembird #0 USB device 038
> Switched outlet 2 on
> Setting boot mode for 2 to gitlabci... done
> + trap 'ssh control@thor.testnet poweroff; : > /tmp/console-stdin' EXIT
> + '[' -n  ]
> + set +x
>  Xen 4.18.5
> (XEN) Xen version 4.18.5 (root@) (gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924) debug=y Wed Sep  3 12:27:30 UTC 2025
> 
> For 4.19 there's no visible delay between "Terminal ready" and the first Xen
> message, whereas for 4.18 there is an approx 1:30min (±10s) delay after the
> "+ set +x". That's a lot when the timeout is 2min.

That makes no sense to me, at this point there isn't really much
Xen-specific code running yet, so there shouldn't be any difference
between branches. I think this is only different presentation of the
output, because 4.18 doesn't use expect in that script yet.

As for the actual reason, at least one job ends at "Waiting for uevents
to be processed", which sounds like USB devices enumeration. Just in
case I reset that emulated USB now.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: xen | Failed pipeline for staging-4.18 | 51190865

[Jan Beulich]

On 04.09.2025 12:43, Marek Marczykowski wrote:
> On Thu, Sep 04, 2025 at 08:27:14AM +0200, Jan Beulich wrote:
>> On 04.09.2025 07:58, Jan Beulich wrote:
>>> On 03.09.2025 18:03, Marek Marczykowski wrote:
>>>> On Wed, Sep 03, 2025 at 05:58:32PM +0200, Jan Beulich wrote:
>>>>> On 03.09.2025 17:46, GitLab wrote:
>>>>>>
>>>>>>
>>>>>> Pipeline #2019390073 has failed!
>>>>>>
>>>>>> Project: xen ( https://gitlab.com/xen-project/hardware/xen )
>>>>>> Branch: staging-4.18 ( https://gitlab.com/xen-project/hardware/xen/-/commits/staging-4.18 )
>>>>>>
>>>>>> Commit: 51190865 ( https://gitlab.com/xen-project/hardware/xen/-/commit/51190865a4918c443c310c0478247d5f9caa5dad )
>>>>>> Commit Message: x86/suspend: unconditionally raise a timer soft...
>>>>>> Commit Author: Roger Pau Monné
>>>>>> Committed by: Jan Beulich ( https://gitlab.com/jbeulich )
>>>>>>
>>>>>>
>>>>>> Pipeline #2019390073 ( https://gitlab.com/xen-project/hardware/xen/-/pipelines/2019390073 ) triggered by Jan Beulich ( https://gitlab.com/jbeulich )
>>>>>> had 5 failed jobs.
>>>>>>
>>>>>> Job #11230955404 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955404/raw )
>>>>>>
>>>>>> Stage: test
>>>>>> Name: adl-suspend-x86-64-gcc-debug
>>>>>> Job #11230955410 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955410/raw )
>>>>>>
>>>>>> Stage: test
>>>>>> Name: adl-pci-pv-x86-64-gcc-debug
>>>>>> Job #11230955417 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11230955417/raw )
>>>>>>
>>>>>> Stage: test
>>>>>> Name: adl-pci-hvm-x86-64-gcc-debug
>>>>>> Job #11233274365 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233274365/raw )
>>>>>>
>>>>>> Stage: test
>>>>>> Name: adl-smoke-x86-64-gcc-debug
>>>>>> Job #11233405609 ( https://gitlab.com/xen-project/hardware/xen/-/jobs/11233405609/raw )
>>>>>>
>>>>>> Stage: test
>>>>>> Name: adl-smoke-x86-64-dom0pvh-gcc-debug
>>>>>
>>>>> While the same tests are fine for 4.19 and 4.20, all five show rubbish in the log,
>>>>> and then fail. No idea what's going on.
>>>>
>>>> The log says "baudrate is    : 115200", but looking at the state after
>>>> the test I see 9600. No idea if that was simply switched back after, or
>>>> setting to 115200 didn't work. Anyway I suggest to restart (now that
>>>> other jobs completed). I set it manually to 115200 now too (not sure if
>>>> that will remain there...).
>>>
>>> The rubbish in the output looks to have gone away, but the adl-* tests fail
>>> as before. I'm retrying two of them another time, but with little hope.
>>
>> As opposed to 4.19, where we have this
>>
>> minimal cmds is: no
>> !! STDIN is not a TTY !! Continue anyway...
>> Type [C-a] [C-h] to see available commands
>> Terminal ready
>>  Xen 4.19.4-pre
>> (XEN) [00000043ae35c0c9] Xen version 4.19.4-pre (root@) (gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924) debug=y Wed Sep  3 12:15:19 UTC 2025
>>
>> for 4.18 things (consistently across the tests) look like this
>>
>> minimal cmds is: no
>> !! STDIN is not a TTY !! Continue anyway...
>> Type [C-a] [C-h] to see available commands
>> Terminal ready
>> Accessing Gembird #0 USB device 038
>> Switched outlet 2 on
>> Setting boot mode for 2 to gitlabci... done
>> + trap 'ssh control@thor.testnet poweroff; : > /tmp/console-stdin' EXIT
>> + '[' -n  ]
>> + set +x
>>  Xen 4.18.5
>> (XEN) Xen version 4.18.5 (root@) (gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924) debug=y Wed Sep  3 12:27:30 UTC 2025
>>
>> For 4.19 there's no visible delay between "Terminal ready" and the first Xen
>> message, whereas for 4.18 there is an approx 1:30min (±10s) delay after the
>> "+ set +x". That's a lot when the timeout is 2min.
> 
> That makes no sense to me, at this point there isn't really much
> Xen-specific code running yet, so there shouldn't be any difference
> between branches. I think this is only different presentation of the
> output, because 4.18 doesn't use expect in that script yet.

Except that one can observe that 1:30min delay when watching the job progressing.

> As for the actual reason, at least one job ends at "Waiting for uevents
> to be processed", which sounds like USB devices enumeration. Just in
> case I reset that emulated USB now.

Others (e.g. the suspend job) timed out elsewhere, though.

Jan