[PATCH 0/2] CI: new PTL runner

[PATCH 0/2] CI: new PTL runner

[Marek Marczykowski-Górecki]

It's a NUC 16 Pro, with Intel Core Ultra 5 325.

As mentioned on the last community call - the hardware is bought by the Xen
Project, and then hosted together with qubes runners.

Xen itself works pretty well here (except suspend, to be investigated later). I
tried also Qubes installer, and there is some issue between Xorg and the
xe display driver (screen remains blank when Xorg starts). Likely an
issue on the Linux side. Also to be investigated later.

Pipeline:
https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2662549936

Marek Marczykowski-Górecki (2):
  CI: add PTL runner
  CI: enable FRED on PTL runner

 automation/gitlab-ci/test.yaml     | 82 +++++++++++++++++++++++++++++++-
 automation/scripts/qubes-x86_64.sh |  4 +-
 2 files changed, 84 insertions(+), 2 deletions(-)

base-commit: ccde3773c1762d32fb5f6037d6b22780c30895d3
-- 
git-series 0.9.1

[PATCH 1/2] CI: add PTL runner

[Marek Marczykowski-Górecki]

For now run similar set of tests as ADL.

Exclude suspend test for now. While firmware claims S3 is supported,
suspend attempt fails (power LED blinks for a few seconds, and
then system reboots). With no messages on the serial console, even with
`sync_console`.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/gitlab-ci/test.yaml | 81 +++++++++++++++++++++++++++++++++++-
 1 file changed, 81 insertions(+)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 20db71b1c947..7cc52146304e 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -170,6 +170,17 @@
   tags:
     - qubes-hw3
 
+.ptl-x86_64:
+  # it's really similar to the ADL one
+  extends: .adl-x86_64
+  variables:
+    PCIDEV: "56:00.0"
+    PCIDEV_INTR: "MSI-X"
+    CONSOLE_OPTS: "console=com1 com1=115200,8n1"
+    SUT_ADDR: test-22.testnet
+  tags:
+    - qubes-hw22
+
 .zen2-x86_64:
   # it's really similar to the above
   extends: .adl-x86_64
@@ -405,6 +416,76 @@ kbl-tools-tests-pvh-x86_64-gcc-debug:
     - *x86_64-test-needs
     - alpine-3.24-x86_64-gcc-debug
 
+ptl-smoke-x86_64-gcc-debug:
+  extends: .ptl-x86_64
+  script:
+    - ./automation/scripts/qubes-x86_64.sh dom0pv 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86_64-test-needs
+    - alpine-3.24-x86_64-gcc-debug
+
+ptl-smoke-x86_64-dom0pvh-gcc-debug:
+  extends: .ptl-x86_64
+  script:
+    - ./automation/scripts/qubes-x86_64.sh dom0pvh 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86_64-test-needs
+    - alpine-3.24-x86_64-gcc-debug
+
+ptl-smoke-x86_64-dom0pvh-hvm-gcc-debug:
+  extends: .ptl-x86_64
+  script:
+    - ./automation/scripts/qubes-x86_64.sh dom0pvh-hvm 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86_64-test-needs
+    - alpine-3.24-x86_64-gcc-debug
+
+ptl-pci-pv-x86_64-gcc-debug:
+  extends: .ptl-x86_64
+  script:
+    - ./automation/scripts/qubes-x86_64.sh pci-pv 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86_64-test-needs
+    - alpine-3.24-x86_64-gcc-debug
+
+ptl-pci-hvm-x86_64-gcc-debug:
+  extends: .ptl-x86_64
+  script:
+    - ./automation/scripts/qubes-x86_64.sh pci-hvm 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86_64-test-needs
+    - alpine-3.24-x86_64-gcc-debug
+
+ptl-pvshim-x86_64-gcc-debug:
+  extends: .ptl-x86_64
+  script:
+    - ./automation/scripts/qubes-x86_64.sh pvshim 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86_64-test-needs
+    - alpine-3.24-x86_64-gcc-debug
+
+ptl-tools-tests-pv-x86_64-gcc-debug:
+  extends: .ptl-x86_64
+  script:
+    - ./automation/scripts/qubes-x86_64.sh tools-tests-pv 2>&1 | tee ${LOGFILE}
+  artifacts:
+    reports:
+      junit: tests-junit.xml
+  needs:
+    - *x86_64-test-needs
+    - alpine-3.24-x86_64-gcc-debug
+
+ptl-tools-tests-pvh-x86_64-gcc-debug:
+  extends: .ptl-x86_64
+  script:
+    - ./automation/scripts/qubes-x86_64.sh tools-tests-pvh 2>&1 | tee ${LOGFILE}
+  artifacts:
+    reports:
+      junit: tests-junit.xml
+  needs:
+    - *x86_64-test-needs
+    - alpine-3.24-x86_64-gcc-debug
+
 zen2-smoke-x86_64-gcc-debug:
   extends: .zen2-x86_64
   script:
-- 
git-series 0.9.1

[PATCH 2/2] CI: enable FRED on PTL runner

[Marek Marczykowski-Górecki]

Add new job-level EXTRA_XEN_OPTS variable, in addition to CONSOLE_OPTS.
For now, add "fred" to all jobs running on PTL, but separate variable
allows customizing it later on per-job basis.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/gitlab-ci/test.yaml     | 1 +
 automation/scripts/qubes-x86_64.sh | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 7cc52146304e..61adc1baff30 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -177,6 +177,7 @@
     PCIDEV: "56:00.0"
     PCIDEV_INTR: "MSI-X"
     CONSOLE_OPTS: "console=com1 com1=115200,8n1"
+    EXTRA_XEN_OPTS: "fred"
     SUT_ADDR: test-22.testnet
   tags:
     - qubes-hw22
diff --git a/automation/scripts/qubes-x86_64.sh b/automation/scripts/qubes-x86_64.sh
index 7a59fa5f1116..065f03931bea 100755
--- a/automation/scripts/qubes-x86_64.sh
+++ b/automation/scripts/qubes-x86_64.sh
@@ -260,14 +260,14 @@ TFTP=/scratch/gitlab-runner/tftp
 CONTROLLER=control@thor.testnet
 
 echo "
-multiboot2 (http)/gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog $extra_xen_opts
+multiboot2 (http)/gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog ${EXTRA_XEN_OPTS-} $extra_xen_opts
 module2 (http)/gitlab-ci/vmlinuz console=hvc0 root=/dev/ram0 earlyprintk=xen
 module2 --nounzip (http)/gitlab-ci/initrd-dom0
 " > $TFTP/grub.cfg
 
 echo "#!ipxe
 
-kernel /gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog $extra_xen_opts || reboot
+kernel /gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog ${EXTRA_XEN_OPTS-} $extra_xen_opts || reboot
 module /gitlab-ci/vmlinuz console=hvc0 root=/dev/ram0 earlyprintk=xen || reboot
 module /gitlab-ci/initrd-dom0 || reboot
 boot
-- 
git-series 0.9.1

Re: [PATCH 1/2] CI: add PTL runner

[Andrew Cooper]

On 09/07/2026 8:57 pm, Marek Marczykowski-Górecki wrote:
> For now run similar set of tests as ADL.
>
> Exclude suspend test for now. While firmware claims S3 is supported,
> suspend attempt fails (power LED blinks for a few seconds, and
> then system reboots). With no messages on the serial console, even with
> `sync_console`.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

I'll need to configure the runner in the right place(s) first.

Did you sort out the dual NIC problem?

For the patch, Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>

Re: [PATCH 1/2] CI: add PTL runner

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

On Fri, Jul 10, 2026 at 02:11:30PM +0100, Andrew Cooper wrote:
> On 09/07/2026 8:57 pm, Marek Marczykowski-Górecki wrote:
> > For now run similar set of tests as ADL.
> >
> > Exclude suspend test for now. While firmware claims S3 is supported,
> > suspend attempt fails (power LED blinks for a few seconds, and
> > then system reboots). With no messages on the serial console, even with
> > `sync_console`.
> >
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> 
> I'll need to configure the runner in the right place(s) first.

Yes, you should have access to it already, just add it to relevant
projects.

> Did you sort out the dual NIC problem?

Yes, disabled one in firmware :)

> For the patch, Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 2/2] CI: enable FRED on PTL runner

[Andrew Cooper]

On 09/07/2026 8:57 pm, Marek Marczykowski-Górecki wrote:
> Add new job-level EXTRA_XEN_OPTS variable, in addition to CONSOLE_OPTS.

This is stale, but easy to adjust.

> For now, add "fred" to all jobs running on PTL, but separate variable
> allows customizing it later on per-job basis.

"FRED is not yet enabled by default on Intel systems, but is expected to
function.  Explicitly turn it on for added test coverage in the short term."

The final task is to make MSR_SPEC_CTRL virtualisation for PV guests
work under FRED.  Right now, PV guests will lose speculation settings
they believe they have set, hence why FRED isn't supported on Intel yet.

>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
>  automation/gitlab-ci/test.yaml     | 1 +
>  automation/scripts/qubes-x86_64.sh | 4 ++--
>  2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 7cc52146304e..61adc1baff30 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -177,6 +177,7 @@
>      PCIDEV: "56:00.0"
>      PCIDEV_INTR: "MSI-X"
>      CONSOLE_OPTS: "console=com1 com1=115200,8n1"
> +    EXTRA_XEN_OPTS: "fred"
>      SUT_ADDR: test-22.testnet
>    tags:
>      - qubes-hw22
> diff --git a/automation/scripts/qubes-x86_64.sh b/automation/scripts/qubes-x86_64.sh
> index 7a59fa5f1116..065f03931bea 100755
> --- a/automation/scripts/qubes-x86_64.sh
> +++ b/automation/scripts/qubes-x86_64.sh
> @@ -260,14 +260,14 @@ TFTP=/scratch/gitlab-runner/tftp
>  CONTROLLER=control@thor.testnet
>  
>  echo "
> -multiboot2 (http)/gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog $extra_xen_opts
> +multiboot2 (http)/gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog ${EXTRA_XEN_OPTS-} $extra_xen_opts
>  module2 (http)/gitlab-ci/vmlinuz console=hvc0 root=/dev/ram0 earlyprintk=xen
>  module2 --nounzip (http)/gitlab-ci/initrd-dom0
>  " > $TFTP/grub.cfg
>  
>  echo "#!ipxe
>  
> -kernel /gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog $extra_xen_opts || reboot
> +kernel /gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog ${EXTRA_XEN_OPTS-} $extra_xen_opts || reboot
>  module /gitlab-ci/vmlinuz console=hvc0 root=/dev/ram0 earlyprintk=xen || reboot
>  module /gitlab-ci/initrd-dom0 || reboot
>  boot

Wouldn't this be better?

diff --git a/automation/scripts/qubes-x86_64.sh
b/automation/scripts/qubes-x86_64.sh
index 065f03931bea..9f8288cc02e3 100755
--- a/automation/scripts/qubes-x86_64.sh
+++ b/automation/scripts/qubes-x86_64.sh
@@ -15,7 +15,7 @@ set -ex -o pipefail
 test_variant=$1
 
 ### defaults
-extra_xen_opts=
+extra_xen_opts="${EXTRA_XEN_OPTS-}"
 wait_and_wakeup=
 timeout=120
 domU_type="pvh"


Everything is expected to append to this variable.

Re: [PATCH 2/2] CI: enable FRED on PTL runner

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 3543 bytes --]

On Fri, Jul 10, 2026 at 05:15:17PM +0100, Andrew Cooper wrote:
> On 09/07/2026 8:57 pm, Marek Marczykowski-Górecki wrote:
> > Add new job-level EXTRA_XEN_OPTS variable, in addition to CONSOLE_OPTS.
> 
> This is stale, but easy to adjust.

What do you mean stale? It is yet another job variable for adding Xen
options (previously it was only CONSOLE_OPTS).

> > For now, add "fred" to all jobs running on PTL, but separate variable
> > allows customizing it later on per-job basis.
> 
> "FRED is not yet enabled by default on Intel systems, but is expected to
> function.  Explicitly turn it on for added test coverage in the short term."
> 
> The final task is to make MSR_SPEC_CTRL virtualisation for PV guests
> work under FRED.  Right now, PV guests will lose speculation settings
> they believe they have set, hence why FRED isn't supported on Intel yet.
> 
> >
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > ---
> >  automation/gitlab-ci/test.yaml     | 1 +
> >  automation/scripts/qubes-x86_64.sh | 4 ++--
> >  2 files changed, 3 insertions(+), 2 deletions(-)
> >
> > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> > index 7cc52146304e..61adc1baff30 100644
> > --- a/automation/gitlab-ci/test.yaml
> > +++ b/automation/gitlab-ci/test.yaml
> > @@ -177,6 +177,7 @@
> >      PCIDEV: "56:00.0"
> >      PCIDEV_INTR: "MSI-X"
> >      CONSOLE_OPTS: "console=com1 com1=115200,8n1"
> > +    EXTRA_XEN_OPTS: "fred"
> >      SUT_ADDR: test-22.testnet
> >    tags:
> >      - qubes-hw22
> > diff --git a/automation/scripts/qubes-x86_64.sh b/automation/scripts/qubes-x86_64.sh
> > index 7a59fa5f1116..065f03931bea 100755
> > --- a/automation/scripts/qubes-x86_64.sh
> > +++ b/automation/scripts/qubes-x86_64.sh
> > @@ -260,14 +260,14 @@ TFTP=/scratch/gitlab-runner/tftp
> >  CONTROLLER=control@thor.testnet
> >  
> >  echo "
> > -multiboot2 (http)/gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog $extra_xen_opts
> > +multiboot2 (http)/gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog ${EXTRA_XEN_OPTS-} $extra_xen_opts
> >  module2 (http)/gitlab-ci/vmlinuz console=hvc0 root=/dev/ram0 earlyprintk=xen
> >  module2 --nounzip (http)/gitlab-ci/initrd-dom0
> >  " > $TFTP/grub.cfg
> >  
> >  echo "#!ipxe
> >  
> > -kernel /gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog $extra_xen_opts || reboot
> > +kernel /gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot watchdog ${EXTRA_XEN_OPTS-} $extra_xen_opts || reboot
> >  module /gitlab-ci/vmlinuz console=hvc0 root=/dev/ram0 earlyprintk=xen || reboot
> >  module /gitlab-ci/initrd-dom0 || reboot
> >  boot
> 
> Wouldn't this be better?
> 
> diff --git a/automation/scripts/qubes-x86_64.sh
> b/automation/scripts/qubes-x86_64.sh
> index 065f03931bea..9f8288cc02e3 100755
> --- a/automation/scripts/qubes-x86_64.sh
> +++ b/automation/scripts/qubes-x86_64.sh
> @@ -15,7 +15,7 @@ set -ex -o pipefail
>  test_variant=$1
>  
>  ### defaults
> -extra_xen_opts=
> +extra_xen_opts="${EXTRA_XEN_OPTS-}"
>  wait_and_wakeup=
>  timeout=120
>  domU_type="pvh"
> 
> 
> Everything is expected to append to this variable.

Well, currently nothing appends to it, all tests override it...

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 2/2] CI: enable FRED on PTL runner

[Andrew Cooper]

On 10/07/2026 5:23 pm, Marek Marczykowski-Górecki wrote:
> On Fri, Jul 10, 2026 at 05:15:17PM +0100, Andrew Cooper wrote:
>> On 09/07/2026 8:57 pm, Marek Marczykowski-Górecki wrote:
>>> Add new job-level EXTRA_XEN_OPTS variable, in addition to CONSOLE_OPTS.
>> This is stale, but easy to adjust.
> What do you mean stale?

Never mind, I can't read.

>> Wouldn't this be better?
>>
>> diff --git a/automation/scripts/qubes-x86_64.sh
>> b/automation/scripts/qubes-x86_64.sh
>> index 065f03931bea..9f8288cc02e3 100755
>> --- a/automation/scripts/qubes-x86_64.sh
>> +++ b/automation/scripts/qubes-x86_64.sh
>> @@ -15,7 +15,7 @@ set -ex -o pipefail
>>  test_variant=$1
>>  
>>  ### defaults
>> -extra_xen_opts=
>> +extra_xen_opts="${EXTRA_XEN_OPTS-}"
>>  wait_and_wakeup=
>>  timeout=120
>>  domU_type="pvh"
>>
>>
>> Everything is expected to append to this variable.
> Well, currently nothing appends to it, all tests override it...
>

Really can't read...

Lets go with the version you tested.

Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>