[PATCH] automation: restore CR filtering

[PATCH] automation: restore CR filtering

[Stefano Stabellini]

After commit c36efb7fcea6 ("automation: use expect to run QEMU") we lost
the \r filtering introduced by b576497e3b7d ("automation: remove CR
characters from serial output"). This patch reintroduced it.

Fixes: c36efb7fcea6 ("automation: use expect to run QEMU") 
Signed-off-by: Stefano Stabellini <stefano.stabellini@amd.com>

diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
index 5359e0820b..343b71d5ac 100755
--- a/automation/scripts/qemu-alpine-x86_64.sh
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -89,4 +89,4 @@ export QEMU_LOG="smoke.serial"
 export LOG_MSG="Domain-0"
 export PASSED="BusyBox"
 
-./automation/scripts/qemu-key.exp
+./automation/scripts/qemu-key.exp | sed 's/\r//'
diff --git a/automation/scripts/qemu-smoke-dom0-arm32.sh b/automation/scripts/qemu-smoke-dom0-arm32.sh
index bbdcb39f1a..a3ccbbd7f4 100755
--- a/automation/scripts/qemu-smoke-dom0-arm32.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm32.sh
@@ -96,4 +96,4 @@ export QEMU_LOG="${serial_log}"
 export LOG_MSG="Domain-0"
 export PASSED="/ #"
 
-../automation/scripts/qemu-key.exp
+../automation/scripts/qemu-key.exp | sed 's/\r//'
diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
index 0094bfc8e1..a781022d48 100755
--- a/automation/scripts/qemu-smoke-dom0-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
@@ -109,4 +109,4 @@ export QEMU_LOG="smoke.serial"
 export LOG_MSG="Domain-0"
 export PASSED="BusyBox"
 
-./automation/scripts/qemu-key.exp
+./automation/scripts/qemu-key.exp | sed 's/\r//'
diff --git a/automation/scripts/qemu-smoke-dom0less-arm32.sh b/automation/scripts/qemu-smoke-dom0less-arm32.sh
index 68ffbabdb8..3c32a676ff 100755
--- a/automation/scripts/qemu-smoke-dom0less-arm32.sh
+++ b/automation/scripts/qemu-smoke-dom0less-arm32.sh
@@ -149,4 +149,4 @@ export QEMU_LOG="${serial_log}"
 export LOG_MSG="${dom0_prompt}"
 export PASSED="${passed}"
 
-../automation/scripts/qemu-key.exp
+../automation/scripts/qemu-key.exp | sed 's/\r//'
diff --git a/automation/scripts/qemu-smoke-dom0less-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh
index eb25c4af4b..a13bd8ee04 100755
--- a/automation/scripts/qemu-smoke-dom0less-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0less-arm64.sh
@@ -220,4 +220,4 @@ export QEMU_LOG="smoke.serial"
 export LOG_MSG="Welcome to Alpine Linux"
 export PASSED="${passed}"
 
-./automation/scripts/qemu-key.exp
+./automation/scripts/qemu-key.exp | sed 's/\r//'
diff --git a/automation/scripts/qemu-smoke-ppc64le.sh b/automation/scripts/qemu-smoke-ppc64le.sh
index ccb4a576f4..3c02771914 100755
--- a/automation/scripts/qemu-smoke-ppc64le.sh
+++ b/automation/scripts/qemu-smoke-ppc64le.sh
@@ -25,4 +25,4 @@ export QEMU_CMD="qemu-system-ppc64 \
 export QEMU_LOG="${serial_log}"
 export PASSED="Hello, ppc64le!"
 
-./automation/scripts/qemu-key.exp
+./automation/scripts/qemu-key.exp | sed 's/\r//'
diff --git a/automation/scripts/qemu-smoke-riscv64.sh b/automation/scripts/qemu-smoke-riscv64.sh
index 0355c075b7..7ede5a5d8a 100755
--- a/automation/scripts/qemu-smoke-riscv64.sh
+++ b/automation/scripts/qemu-smoke-riscv64.sh
@@ -16,4 +16,4 @@ export QEMU_CMD="qemu-system-riscv64 \
 export QEMU_LOG="smoke.serial"
 export PASSED="All set up"
 
-./automation/scripts/qemu-key.exp
+./automation/scripts/qemu-key.exp | sed 's/\r//'
diff --git a/automation/scripts/qemu-smoke-x86-64.sh b/automation/scripts/qemu-smoke-x86-64.sh
index 37ac10e068..81e7dba329 100755
--- a/automation/scripts/qemu-smoke-x86-64.sh
+++ b/automation/scripts/qemu-smoke-x86-64.sh
@@ -24,4 +24,4 @@ export QEMU_CMD="qemu-system-x86_64 -nographic -kernel binaries/xen \
 export QEMU_LOG="smoke.serial"
 export PASSED="Test result: SUCCESS"
 
-./automation/scripts/qemu-key.exp
+./automation/scripts/qemu-key.exp | sed 's/\r//'
diff --git a/automation/scripts/qemu-xtf-dom0less-arm64.sh b/automation/scripts/qemu-xtf-dom0less-arm64.sh
index 0666f6363e..ed44aab0f0 100755
--- a/automation/scripts/qemu-xtf-dom0less-arm64.sh
+++ b/automation/scripts/qemu-xtf-dom0less-arm64.sh
@@ -65,4 +65,4 @@ export UBOOT_CMD="virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x4000000
 export QEMU_LOG="smoke.serial"
 export PASSED="${passed}"
 
-./automation/scripts/qemu-key.exp
+./automation/scripts/qemu-key.exp | sed 's/\r//'

Re: [PATCH] automation: restore CR filtering

[Michal Orzel]

Hi Stefano,

On 17/08/2024 01:46, Stefano Stabellini wrote:
> 
> 
> After commit c36efb7fcea6 ("automation: use expect to run QEMU") we lost
> the \r filtering introduced by b576497e3b7d ("automation: remove CR
> characters from serial output"). This patch reintroduced it.
> 
> Fixes: c36efb7fcea6 ("automation: use expect to run QEMU")
> Signed-off-by: Stefano Stabellini <stefano.stabellini@amd.com>
> 
> diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
> index 5359e0820b..343b71d5ac 100755
> --- a/automation/scripts/qemu-alpine-x86_64.sh
> +++ b/automation/scripts/qemu-alpine-x86_64.sh
> @@ -89,4 +89,4 @@ export QEMU_LOG="smoke.serial"
>  export LOG_MSG="Domain-0"
>  export PASSED="BusyBox"
> 
> -./automation/scripts/qemu-key.exp
> +./automation/scripts/qemu-key.exp | sed 's/\r//'
> diff --git a/automation/scripts/qemu-smoke-dom0-arm32.sh b/automation/scripts/qemu-smoke-dom0-arm32.sh
> index bbdcb39f1a..a3ccbbd7f4 100755
> --- a/automation/scripts/qemu-smoke-dom0-arm32.sh
> +++ b/automation/scripts/qemu-smoke-dom0-arm32.sh
> @@ -96,4 +96,4 @@ export QEMU_LOG="${serial_log}"
>  export LOG_MSG="Domain-0"
>  export PASSED="/ #"
> 
> -../automation/scripts/qemu-key.exp
> +../automation/scripts/qemu-key.exp | sed 's/\r//'
> diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
> index 0094bfc8e1..a781022d48 100755
> --- a/automation/scripts/qemu-smoke-dom0-arm64.sh
> +++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
> @@ -109,4 +109,4 @@ export QEMU_LOG="smoke.serial"
>  export LOG_MSG="Domain-0"
>  export PASSED="BusyBox"
> 
> -./automation/scripts/qemu-key.exp
> +./automation/scripts/qemu-key.exp | sed 's/\r//'
> diff --git a/automation/scripts/qemu-smoke-dom0less-arm32.sh b/automation/scripts/qemu-smoke-dom0less-arm32.sh
> index 68ffbabdb8..3c32a676ff 100755
> --- a/automation/scripts/qemu-smoke-dom0less-arm32.sh
> +++ b/automation/scripts/qemu-smoke-dom0less-arm32.sh
> @@ -149,4 +149,4 @@ export QEMU_LOG="${serial_log}"
>  export LOG_MSG="${dom0_prompt}"
>  export PASSED="${passed}"
> 
> -../automation/scripts/qemu-key.exp
> +../automation/scripts/qemu-key.exp | sed 's/\r//'
> diff --git a/automation/scripts/qemu-smoke-dom0less-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh
> index eb25c4af4b..a13bd8ee04 100755
> --- a/automation/scripts/qemu-smoke-dom0less-arm64.sh
> +++ b/automation/scripts/qemu-smoke-dom0less-arm64.sh
> @@ -220,4 +220,4 @@ export QEMU_LOG="smoke.serial"
>  export LOG_MSG="Welcome to Alpine Linux"
>  export PASSED="${passed}"
> 
> -./automation/scripts/qemu-key.exp
> +./automation/scripts/qemu-key.exp | sed 's/\r//'
> diff --git a/automation/scripts/qemu-smoke-ppc64le.sh b/automation/scripts/qemu-smoke-ppc64le.sh
> index ccb4a576f4..3c02771914 100755
> --- a/automation/scripts/qemu-smoke-ppc64le.sh
> +++ b/automation/scripts/qemu-smoke-ppc64le.sh
> @@ -25,4 +25,4 @@ export QEMU_CMD="qemu-system-ppc64 \
>  export QEMU_LOG="${serial_log}"
>  export PASSED="Hello, ppc64le!"
> 
> -./automation/scripts/qemu-key.exp
> +./automation/scripts/qemu-key.exp | sed 's/\r//'
> diff --git a/automation/scripts/qemu-smoke-riscv64.sh b/automation/scripts/qemu-smoke-riscv64.sh
> index 0355c075b7..7ede5a5d8a 100755
> --- a/automation/scripts/qemu-smoke-riscv64.sh
> +++ b/automation/scripts/qemu-smoke-riscv64.sh
> @@ -16,4 +16,4 @@ export QEMU_CMD="qemu-system-riscv64 \
>  export QEMU_LOG="smoke.serial"
>  export PASSED="All set up"
> 
> -./automation/scripts/qemu-key.exp
> +./automation/scripts/qemu-key.exp | sed 's/\r//'
> diff --git a/automation/scripts/qemu-smoke-x86-64.sh b/automation/scripts/qemu-smoke-x86-64.sh
> index 37ac10e068..81e7dba329 100755
> --- a/automation/scripts/qemu-smoke-x86-64.sh
> +++ b/automation/scripts/qemu-smoke-x86-64.sh
> @@ -24,4 +24,4 @@ export QEMU_CMD="qemu-system-x86_64 -nographic -kernel binaries/xen \
>  export QEMU_LOG="smoke.serial"
>  export PASSED="Test result: SUCCESS"
> 
> -./automation/scripts/qemu-key.exp
> +./automation/scripts/qemu-key.exp | sed 's/\r//'
> diff --git a/automation/scripts/qemu-xtf-dom0less-arm64.sh b/automation/scripts/qemu-xtf-dom0less-arm64.sh
> index 0666f6363e..ed44aab0f0 100755
> --- a/automation/scripts/qemu-xtf-dom0less-arm64.sh
> +++ b/automation/scripts/qemu-xtf-dom0less-arm64.sh
> @@ -65,4 +65,4 @@ export UBOOT_CMD="virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x4000000
>  export QEMU_LOG="smoke.serial"
>  export PASSED="${passed}"
> 
> -./automation/scripts/qemu-key.exp
> +./automation/scripts/qemu-key.exp | sed 's/\r//'

I compared 3 pipelines:
1) one before c36efb7fcea6 (https://gitlab.com/xen-project/hardware/xen/-/jobs/7566986885)
2) one after c36efb7fcea6 (https://gitlab.com/xen-project/hardware/xen/-/jobs/7603830706)
3) one with this fix (https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7603783403)

In 1), there is Xen log + Linux log in Gitlab web page
In 2), there is no log at all
In 3), there is only Xen log visible

I think we should aim at restoring the original behavior i.e. all logs visible.

~Michal

Re: [PATCH] automation: restore CR filtering

[Anthony PERARD]

On Mon, Aug 19, 2024 at 09:21:22AM +0200, Michal Orzel wrote:
> On 17/08/2024 01:46, Stefano Stabellini wrote:
> > diff --git a/automation/scripts/qemu-xtf-dom0less-arm64.sh b/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > index 0666f6363e..ed44aab0f0 100755
> > --- a/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > +++ b/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > @@ -65,4 +65,4 @@ export UBOOT_CMD="virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x4000000
> >  export QEMU_LOG="smoke.serial"
> >  export PASSED="${passed}"
> > 
> > -./automation/scripts/qemu-key.exp
> > +./automation/scripts/qemu-key.exp | sed 's/\r//'
> 
> I compared 3 pipelines:
> 1) one before c36efb7fcea6 (https://gitlab.com/xen-project/hardware/xen/-/jobs/7566986885)
> 2) one after c36efb7fcea6 (https://gitlab.com/xen-project/hardware/xen/-/jobs/7603830706)
> 3) one with this fix (https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7603783403)
> 
> In 1), there is Xen log + Linux log in Gitlab web page
> In 2), there is no log at all
> In 3), there is only Xen log visible

It's nice that you can select uboot/Xen logs or Linux logs based on the
number of '\r' at the end of a line (output cat -A):
    U-Boot 2023.01+dfsg-2+deb12u1 (Apr 18 2024 - 22:00:21 +0000)^M^M$
    (XEN) [    0.013864] Xen version 4.20-unstable (root@) (gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924) debug=n Sat Aug 17 00:54:57 UTC 2024^M^M$
    [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]^M^M^M$

But to display to GitLab's job logs, we want: sed 's/\r\+$//'

Also, do you have to edit every single script to overcome a shortcoming
from the "expect" script? Can't you write a bit of Tcl and edit the line
in the script instead?

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH] automation: restore CR filtering

[Stefano Stabellini]

On Mon, 19 Aug 2024, Anthony PERARD wrote:
> On Mon, Aug 19, 2024 at 09:21:22AM +0200, Michal Orzel wrote:
> > On 17/08/2024 01:46, Stefano Stabellini wrote:
> > > diff --git a/automation/scripts/qemu-xtf-dom0less-arm64.sh b/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > > index 0666f6363e..ed44aab0f0 100755
> > > --- a/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > > +++ b/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > > @@ -65,4 +65,4 @@ export UBOOT_CMD="virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x4000000
> > >  export QEMU_LOG="smoke.serial"
> > >  export PASSED="${passed}"
> > > 
> > > -./automation/scripts/qemu-key.exp
> > > +./automation/scripts/qemu-key.exp | sed 's/\r//'
> > 
> > I compared 3 pipelines:
> > 1) one before c36efb7fcea6 (https://gitlab.com/xen-project/hardware/xen/-/jobs/7566986885)
> > 2) one after c36efb7fcea6 (https://gitlab.com/xen-project/hardware/xen/-/jobs/7603830706)
> > 3) one with this fix (https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7603783403)
> > 
> > In 1), there is Xen log + Linux log in Gitlab web page
> > In 2), there is no log at all
> > In 3), there is only Xen log visible
> 
> It's nice that you can select uboot/Xen logs or Linux logs based on the
> number of '\r' at the end of a line (output cat -A):
>     U-Boot 2023.01+dfsg-2+deb12u1 (Apr 18 2024 - 22:00:21 +0000)^M^M$
>     (XEN) [    0.013864] Xen version 4.20-unstable (root@) (gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924) debug=n Sat Aug 17 00:54:57 UTC 2024^M^M$
>     [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]^M^M^M$
> 
> But to display to GitLab's job logs, we want: sed 's/\r\+$//'
> 
> Also, do you have to edit every single script to overcome a shortcoming
> from the "expect" script? Can't you write a bit of Tcl and edit the line
> in the script instead?

The sed route is not perfect but it works :-)

I did try using expect but the logs were mangled. I think I missed that
there can be multilple \r. I managed to get close to the wanted behavior
with the below, but the Xen logs are still missing and I don't know why. 

https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7617161552

diff --git a/automation/scripts/qemu-key.exp b/automation/scripts/qemu-key.exp
index 35eb903a31..38061f2deb 100755
--- a/automation/scripts/qemu-key.exp
+++ b/automation/scripts/qemu-key.exp
@@ -9,7 +9,8 @@ match_max 10000
 eval spawn $env(QEMU_CMD)
 
 expect_after {
-    -re "(.*)\r" {
+    -re "(.*)\r+$" {
+        puts $expect_out(1,string)
         exp_continue
     }
     timeout {send_error "ERROR-Timeout!\n"; exit 1}

Re: [PATCH] automation: restore CR filtering

[Anthony PERARD]

On Mon, Aug 19, 2024 at 06:56:47PM -0700, Stefano Stabellini wrote:
> On Mon, 19 Aug 2024, Anthony PERARD wrote:
> > On Mon, Aug 19, 2024 at 09:21:22AM +0200, Michal Orzel wrote:
> > > On 17/08/2024 01:46, Stefano Stabellini wrote:
> > > > diff --git a/automation/scripts/qemu-xtf-dom0less-arm64.sh b/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > > > index 0666f6363e..ed44aab0f0 100755
> > > > --- a/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > > > +++ b/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > > > @@ -65,4 +65,4 @@ export UBOOT_CMD="virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x4000000
> > > >  export QEMU_LOG="smoke.serial"
> > > >  export PASSED="${passed}"
> > > > 
> > > > -./automation/scripts/qemu-key.exp
> > > > +./automation/scripts/qemu-key.exp | sed 's/\r//'
> > > 
> > > I compared 3 pipelines:
> > > 1) one before c36efb7fcea6 (https://gitlab.com/xen-project/hardware/xen/-/jobs/7566986885)
> > > 2) one after c36efb7fcea6 (https://gitlab.com/xen-project/hardware/xen/-/jobs/7603830706)
> > > 3) one with this fix (https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7603783403)
> > > 
> > > In 1), there is Xen log + Linux log in Gitlab web page
> > > In 2), there is no log at all
> > > In 3), there is only Xen log visible
> > 
> > It's nice that you can select uboot/Xen logs or Linux logs based on the
> > number of '\r' at the end of a line (output cat -A):
> >     U-Boot 2023.01+dfsg-2+deb12u1 (Apr 18 2024 - 22:00:21 +0000)^M^M$
> >     (XEN) [    0.013864] Xen version 4.20-unstable (root@) (gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924) debug=n Sat Aug 17 00:54:57 UTC 2024^M^M$
> >     [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]^M^M^M$
> > 
> > But to display to GitLab's job logs, we want: sed 's/\r\+$//'
> > 
> > Also, do you have to edit every single script to overcome a shortcoming
> > from the "expect" script? Can't you write a bit of Tcl and edit the line
> > in the script instead?
> 
> The sed route is not perfect but it works :-)
> 
> I did try using expect but the logs were mangled. I think I missed that
> there can be multilple \r. I managed to get close to the wanted behavior
> with the below, but the Xen logs are still missing and I don't know why. 
> 
> https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7617161552

Well, it just looks like the output is duplicated, if you look at the
raw output:
https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7617161552/raw

So I don't know if it's possible to edit the output before `expect`
prints it. It probably is, but not easy to do.

I did try to edit the command line to change QEMU's output:
    -eval spawn $env(QEMU_CMD)
    +spawn sh -c "$env(QEMU_CMD) | sed s/\\\\r\\\\+//"

But then many failure, probably because expect can't interact with qemu
anymore.

So I guess `sed` the output of `expect` will do. Maybe put that in a
script that also call the expect script? (To avoid duplication, and help
with maintainability of the whole thing.)

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH] automation: restore CR filtering

[Stefano Stabellini]

On Tue, 20 Aug 2024, Anthony PERARD wrote:
> On Mon, Aug 19, 2024 at 06:56:47PM -0700, Stefano Stabellini wrote:
> > On Mon, 19 Aug 2024, Anthony PERARD wrote:
> > > On Mon, Aug 19, 2024 at 09:21:22AM +0200, Michal Orzel wrote:
> > > > On 17/08/2024 01:46, Stefano Stabellini wrote:
> > > > > diff --git a/automation/scripts/qemu-xtf-dom0less-arm64.sh b/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > > > > index 0666f6363e..ed44aab0f0 100755
> > > > > --- a/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > > > > +++ b/automation/scripts/qemu-xtf-dom0less-arm64.sh
> > > > > @@ -65,4 +65,4 @@ export UBOOT_CMD="virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x4000000
> > > > >  export QEMU_LOG="smoke.serial"
> > > > >  export PASSED="${passed}"
> > > > > 
> > > > > -./automation/scripts/qemu-key.exp
> > > > > +./automation/scripts/qemu-key.exp | sed 's/\r//'
> > > > 
> > > > I compared 3 pipelines:
> > > > 1) one before c36efb7fcea6 (https://gitlab.com/xen-project/hardware/xen/-/jobs/7566986885)
> > > > 2) one after c36efb7fcea6 (https://gitlab.com/xen-project/hardware/xen/-/jobs/7603830706)
> > > > 3) one with this fix (https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7603783403)
> > > > 
> > > > In 1), there is Xen log + Linux log in Gitlab web page
> > > > In 2), there is no log at all
> > > > In 3), there is only Xen log visible
> > > 
> > > It's nice that you can select uboot/Xen logs or Linux logs based on the
> > > number of '\r' at the end of a line (output cat -A):
> > >     U-Boot 2023.01+dfsg-2+deb12u1 (Apr 18 2024 - 22:00:21 +0000)^M^M$
> > >     (XEN) [    0.013864] Xen version 4.20-unstable (root@) (gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924) debug=n Sat Aug 17 00:54:57 UTC 2024^M^M$
> > >     [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]^M^M^M$
> > > 
> > > But to display to GitLab's job logs, we want: sed 's/\r\+$//'
> > > 
> > > Also, do you have to edit every single script to overcome a shortcoming
> > > from the "expect" script? Can't you write a bit of Tcl and edit the line
> > > in the script instead?
> > 
> > The sed route is not perfect but it works :-)
> > 
> > I did try using expect but the logs were mangled. I think I missed that
> > there can be multilple \r. I managed to get close to the wanted behavior
> > with the below, but the Xen logs are still missing and I don't know why. 
> > 
> > https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7617161552
> 
> Well, it just looks like the output is duplicated, if you look at the
> raw output:
> https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7617161552/raw
> 
> So I don't know if it's possible to edit the output before `expect`
> prints it. It probably is, but not easy to do.
> 
> I did try to edit the command line to change QEMU's output:
>     -eval spawn $env(QEMU_CMD)
>     +spawn sh -c "$env(QEMU_CMD) | sed s/\\\\r\\\\+//"
> 
> But then many failure, probably because expect can't interact with qemu
> anymore.
> 
> So I guess `sed` the output of `expect` will do. Maybe put that in a
> script that also call the expect script? (To avoid duplication, and help
> with maintainability of the whole thing.)

I tried a couple more times with expect but couldn't get it to work.
With sed it seems to behave correctly. Can you guys spot any issues with
it? If not, I'll send the full patch.

https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7627376496

diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
index 0094bfc8e1..e0cea742b0 100755
--- a/automation/scripts/qemu-smoke-dom0-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
@@ -109,4 +109,4 @@ export QEMU_LOG="smoke.serial"
 export LOG_MSG="Domain-0"
 export PASSED="BusyBox"
 
-./automation/scripts/qemu-key.exp
+./automation/scripts/qemu-key.exp | sed 's/\r\+$//'

Re: [PATCH] automation: restore CR filtering

[Anthony PERARD]

On Tue, Aug 20, 2024 at 08:48:55PM -0700, Stefano Stabellini wrote:
> On Tue, 20 Aug 2024, Anthony PERARD wrote:
> > So I guess `sed` the output of `expect` will do. Maybe put that in a
> > script that also call the expect script? (To avoid duplication, and help
> > with maintainability of the whole thing.)
> 
> I tried a couple more times with expect but couldn't get it to work.
> With sed it seems to behave correctly. Can you guys spot any issues with
> it? If not, I'll send the full patch.
> 
> https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7627376496

That job looks fine to me.

> diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
> index 0094bfc8e1..e0cea742b0 100755
> --- a/automation/scripts/qemu-smoke-dom0-arm64.sh
> +++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
> @@ -109,4 +109,4 @@ export QEMU_LOG="smoke.serial"
>  export LOG_MSG="Domain-0"
>  export PASSED="BusyBox"
>  
> -./automation/scripts/qemu-key.exp
> +./automation/scripts/qemu-key.exp | sed 's/\r\+$//'

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech