[PATCH] gitlab-ci: Use artifacts instead of dumping logs in the Cirrus-CI jobs

[PATCH] gitlab-ci: Use artifacts instead of dumping logs in the Cirrus-CI jobs

[Thomas Huth]

The meson log files can get very big, especially if running the tests in
verbose mode. So dumping those logs to the console was a bad idea, since
gitlab truncates the output if it is getting too big. Let's publish the
logs as artifacts instead. This has the disadvantage that you have to
look up the logs on cirrus-ci.com now instead, but that's still better
than not having the important part of the log at all since it got
truncated.

Fixes: 998f334722 ("gitlab: show testlog.txt contents ...")
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 Note: I also tried to publish the junit xml files as artifacts
 instead, but while the cirrus-ci docs claim to support it, I only
 got unreadable XML output in my browser that way, so the .txt
 files look like the better option to me.

 .gitlab-ci.d/cirrus/build.yml | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
index 7ef6af8d33..a9444902ec 100644
--- a/.gitlab-ci.d/cirrus/build.yml
+++ b/.gitlab-ci.d/cirrus/build.yml
@@ -32,6 +32,9 @@ build_task:
     - $MAKE -j$(sysctl -n hw.ncpu)
     - for TARGET in $TEST_TARGETS ;
       do
-        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1
-        || { cat meson-logs/testlog.txt; exit 1; } ;
+        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
       done
+  always:
+    build_result_artifacts:
+      path: build/meson-logs/*log.txt
+      type: text/plain
-- 
2.31.1

Re: [PATCH] gitlab-ci: Use artifacts instead of dumping logs in the Cirrus-CI jobs

[Philippe Mathieu-Daudé]

Hi Thomas,

On 15/2/23 15:25, Thomas Huth wrote:
> The meson log files can get very big, especially if running the tests in
> verbose mode. So dumping those logs to the console was a bad idea, since
> gitlab truncates the output if it is getting too big. Let's publish the
> logs as artifacts instead. This has the disadvantage that you have to
> look up the logs on cirrus-ci.com now instead, but that's still better
> than not having the important part of the log at all since it got
> truncated.
> 
> Fixes: 998f334722 ("gitlab: show testlog.txt contents ...")
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   Note: I also tried to publish the junit xml files as artifacts
>   instead, but while the cirrus-ci docs claim to support it, I only
>   got unreadable XML output in my browser that way, so the .txt
>   files look like the better option to me.
> 
>   .gitlab-ci.d/cirrus/build.yml | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> index 7ef6af8d33..a9444902ec 100644
> --- a/.gitlab-ci.d/cirrus/build.yml
> +++ b/.gitlab-ci.d/cirrus/build.yml
> @@ -32,6 +32,9 @@ build_task:
>       - $MAKE -j$(sysctl -n hw.ncpu)
>       - for TARGET in $TEST_TARGETS ;
>         do
> -        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1
> -        || { cat meson-logs/testlog.txt; exit 1; } ;
> +        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
>         done
> +  always:
> +    build_result_artifacts:
> +      path: build/meson-logs/*log.txt
> +      type: text/plain

How long is that log kept available?

Re: [PATCH] gitlab-ci: Use artifacts instead of dumping logs in the Cirrus-CI jobs

[Thomas Huth]

On 15/02/2023 15.57, Philippe Mathieu-Daudé wrote:
> Hi Thomas,
> 
> On 15/2/23 15:25, Thomas Huth wrote:
>> The meson log files can get very big, especially if running the tests in
>> verbose mode. So dumping those logs to the console was a bad idea, since
>> gitlab truncates the output if it is getting too big. Let's publish the
>> logs as artifacts instead. This has the disadvantage that you have to
>> look up the logs on cirrus-ci.com now instead, but that's still better
>> than not having the important part of the log at all since it got
>> truncated.
>>
>> Fixes: 998f334722 ("gitlab: show testlog.txt contents ...")
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   Note: I also tried to publish the junit xml files as artifacts
>>   instead, but while the cirrus-ci docs claim to support it, I only
>>   got unreadable XML output in my browser that way, so the .txt
>>   files look like the better option to me.
>>
>>   .gitlab-ci.d/cirrus/build.yml | 7 +++++--
>>   1 file changed, 5 insertions(+), 2 deletions(-)
>>
>> diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
>> index 7ef6af8d33..a9444902ec 100644
>> --- a/.gitlab-ci.d/cirrus/build.yml
>> +++ b/.gitlab-ci.d/cirrus/build.yml
>> @@ -32,6 +32,9 @@ build_task:
>>       - $MAKE -j$(sysctl -n hw.ncpu)
>>       - for TARGET in $TEST_TARGETS ;
>>         do
>> -        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1
>> -        || { cat meson-logs/testlog.txt; exit 1; } ;
>> +        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
>>         done
>> +  always:
>> +    build_result_artifacts:
>> +      path: build/meson-logs/*log.txt
>> +      type: text/plain
> 
> How long is that log kept available?

No clue, I haven't seen anything related in the docs:

  https://cirrus-ci.org/guide/writing-tasks/#artifacts-instruction

... but I assume they will at least be available for a couple of days so 
you've got enough time to look at them in case a job failed.

  Thomas

Re: [PATCH] gitlab-ci: Use artifacts instead of dumping logs in the Cirrus-CI jobs

[Philippe Mathieu-Daudé]

On 15/2/23 16:45, Thomas Huth wrote:
> On 15/02/2023 15.57, Philippe Mathieu-Daudé wrote:
>> Hi Thomas,
>>
>> On 15/2/23 15:25, Thomas Huth wrote:
>>> The meson log files can get very big, especially if running the tests in
>>> verbose mode. So dumping those logs to the console was a bad idea, since
>>> gitlab truncates the output if it is getting too big. Let's publish the
>>> logs as artifacts instead. This has the disadvantage that you have to
>>> look up the logs on cirrus-ci.com now instead, but that's still better
>>> than not having the important part of the log at all since it got
>>> truncated.
>>>
>>> Fixes: 998f334722 ("gitlab: show testlog.txt contents ...")
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
>>>   Note: I also tried to publish the junit xml files as artifacts
>>>   instead, but while the cirrus-ci docs claim to support it, I only
>>>   got unreadable XML output in my browser that way, so the .txt
>>>   files look like the better option to me.
>>>
>>>   .gitlab-ci.d/cirrus/build.yml | 7 +++++--
>>>   1 file changed, 5 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/.gitlab-ci.d/cirrus/build.yml 
>>> b/.gitlab-ci.d/cirrus/build.yml
>>> index 7ef6af8d33..a9444902ec 100644
>>> --- a/.gitlab-ci.d/cirrus/build.yml
>>> +++ b/.gitlab-ci.d/cirrus/build.yml
>>> @@ -32,6 +32,9 @@ build_task:
>>>       - $MAKE -j$(sysctl -n hw.ncpu)
>>>       - for TARGET in $TEST_TARGETS ;
>>>         do
>>> -        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1
>>> -        || { cat meson-logs/testlog.txt; exit 1; } ;
>>> +        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
>>>         done
>>> +  always:
>>> +    build_result_artifacts:
>>> +      path: build/meson-logs/*log.txt
>>> +      type: text/plain
>>
>> How long is that log kept available?
> 
> No clue, I haven't seen anything related in the docs:
> 
>   https://cirrus-ci.org/guide/writing-tasks/#artifacts-instruction
> 
> ... but I assume they will at least be available for a couple of days so 
> you've got enough time to look at them in case a job failed.

OK thanks.

Re: [PATCH] gitlab-ci: Use artifacts instead of dumping logs in the Cirrus-CI jobs

[Daniel P. Berrangé]

On Wed, Feb 15, 2023 at 03:25:03PM +0100, Thomas Huth wrote:
> The meson log files can get very big, especially if running the tests in
> verbose mode. So dumping those logs to the console was a bad idea, since
> gitlab truncates the output if it is getting too big. Let's publish the
> logs as artifacts instead. This has the disadvantage that you have to
> look up the logs on cirrus-ci.com now instead, but that's still better
> than not having the important part of the log at all since it got
> truncated.

Having to go over to cirrus-ci.com is pretty awful user experiance,
especially as there's no direct link.

> Fixes: 998f334722 ("gitlab: show testlog.txt contents ...")
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  Note: I also tried to publish the junit xml files as artifacts
>  instead, but while the cirrus-ci docs claim to support it, I only
>  got unreadable XML output in my browser that way, so the .txt
>  files look like the better option to me.
> 
>  .gitlab-ci.d/cirrus/build.yml | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> index 7ef6af8d33..a9444902ec 100644
> --- a/.gitlab-ci.d/cirrus/build.yml
> +++ b/.gitlab-ci.d/cirrus/build.yml
> @@ -32,6 +32,9 @@ build_task:
>      - $MAKE -j$(sysctl -n hw.ncpu)
>      - for TARGET in $TEST_TARGETS ;
>        do
> -        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1
> -        || { cat meson-logs/testlog.txt; exit 1; } ;
> +        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
>        done
> +  always:
> +    build_result_artifacts:
> +      path: build/meson-logs/*log.txt
> +      type: text/plain

Does it have to be either/or, can't we do both ?

95% of the time the truncated testlog.txt is sufficient on its own.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] gitlab-ci: Use artifacts instead of dumping logs in the Cirrus-CI jobs

[Thomas Huth]

On 15/02/2023 15.58, Daniel P. Berrangé wrote:
> On Wed, Feb 15, 2023 at 03:25:03PM +0100, Thomas Huth wrote:
>> The meson log files can get very big, especially if running the tests in
>> verbose mode. So dumping those logs to the console was a bad idea, since
>> gitlab truncates the output if it is getting too big. Let's publish the
>> logs as artifacts instead. This has the disadvantage that you have to
>> look up the logs on cirrus-ci.com now instead, but that's still better
>> than not having the important part of the log at all since it got
>> truncated.
> 
> Having to go over to cirrus-ci.com is pretty awful user experiance,
> especially as there's no direct link.

It's not that bad, see e.g.:

  https://gitlab.com/thuth/qemu/-/jobs/3775523498

The log shows a link to cirrus-ci.com at the end:

  Build failed: https://cirrus-ci.com/build/4811273133621248

If you click on that URL, you've just got to go into the "build" and click 
through the build artifacts to get to the log that you want.

>> +      type: text/plain
> 
> Does it have to be either/or, can't we do both ?
> 
> 95% of the time the truncated testlog.txt is sufficient on its own.

I'd say 95% of the time the normal console output (without the "cat 
.../testlog.txt") is already enough since meson prints the stdout and stderr 
of the failing tests to the console already.

FWIW, this is the test run with the truncated output that Peter complained 
about earlier today in IRC:

  https://gitlab.com/qemu-project/qemu/-/jobs/3768540680

Even if you click on the "complete raw" link there, you'll have a hard time 
finding the information that you are interested in. So I'd really prefer to 
not dump testlog.txt by default and only have it via the artifacts instead.

  Thomas

Re: [PATCH] gitlab-ci: Use artifacts instead of dumping logs in the Cirrus-CI jobs

[Daniel P. Berrangé]

On Wed, Feb 15, 2023 at 04:33:42PM +0100, Thomas Huth wrote:
> On 15/02/2023 15.58, Daniel P. Berrangé wrote:
> > On Wed, Feb 15, 2023 at 03:25:03PM +0100, Thomas Huth wrote:
> > > The meson log files can get very big, especially if running the tests in
> > > verbose mode. So dumping those logs to the console was a bad idea, since
> > > gitlab truncates the output if it is getting too big. Let's publish the
> > > logs as artifacts instead. This has the disadvantage that you have to
> > > look up the logs on cirrus-ci.com now instead, but that's still better
> > > than not having the important part of the log at all since it got
> > > truncated.
> > 
> > Having to go over to cirrus-ci.com is pretty awful user experiance,
> > especially as there's no direct link.
> 
> It's not that bad, see e.g.:
> 
>  https://gitlab.com/thuth/qemu/-/jobs/3775523498
> 
> The log shows a link to cirrus-ci.com at the end:
> 
>  Build failed: https://cirrus-ci.com/build/4811273133621248
> 
> If you click on that URL, you've just got to go into the "build" and click
> through the build artifacts to get to the log that you want.
> 
> > > +      type: text/plain
> > 
> > Does it have to be either/or, can't we do both ?
> > 
> > 95% of the time the truncated testlog.txt is sufficient on its own.
> 
> I'd say 95% of the time the normal console output (without the "cat
> .../testlog.txt") is already enough since meson prints the stdout and stderr
> of the failing tests to the console already.
> 
> FWIW, this is the test run with the truncated output that Peter complained
> about earlier today in IRC:
> 
>  https://gitlab.com/qemu-project/qemu/-/jobs/3768540680
> 
> Even if you click on the "complete raw" link there, you'll have a hard time
> finding the information that you are interested in. So I'd really prefer to
> not dump testlog.txt by default and only have it via the artifacts instead.

Ok, you convinced me.

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] gitlab-ci: Use artifacts instead of dumping logs in the Cirrus-CI jobs

[Peter Maydell]

On Wed, 15 Feb 2023 at 15:33, Thomas Huth <thuth@redhat.com> wrote:
>
> On 15/02/2023 15.58, Daniel P. Berrangé wrote:
> > On Wed, Feb 15, 2023 at 03:25:03PM +0100, Thomas Huth wrote:
> >> The meson log files can get very big, especially if running the tests in
> >> verbose mode. So dumping those logs to the console was a bad idea, since
> >> gitlab truncates the output if it is getting too big. Let's publish the
> >> logs as artifacts instead. This has the disadvantage that you have to
> >> look up the logs on cirrus-ci.com now instead, but that's still better
> >> than not having the important part of the log at all since it got
> >> truncated.
> >
> > Having to go over to cirrus-ci.com is pretty awful user experiance,
> > especially as there's no direct link.
>
> It's not that bad, see e.g.:
>
>   https://gitlab.com/thuth/qemu/-/jobs/3775523498
>
> The log shows a link to cirrus-ci.com at the end:
>
>   Build failed: https://cirrus-ci.com/build/4811273133621248
>
> If you click on that URL, you've just got to go into the "build" and click
> through the build artifacts to get to the log that you want.

Could somebody write up some documentation for how to get from
"top level gitlab CI pipeline page" to "detailed logs for a
job failure", please? I'm finding that the UI for this (both
Cirrus and gitlab) is very non-obvious, so a writeup like
the above of "follow this link from the log, click this button,
go look at build artefact X" would be very helpful.
Somewhere in https://wiki.qemu.org/Merges is probably a
convenient place for thsi.

thanks
-- PMM

Re: [PATCH] gitlab-ci: Use artifacts instead of dumping logs in the Cirrus-CI jobs

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> The meson log files can get very big, especially if running the tests in
> verbose mode. So dumping those logs to the console was a bad idea, since
> gitlab truncates the output if it is getting too big. Let's publish the
> logs as artifacts instead. This has the disadvantage that you have to
> look up the logs on cirrus-ci.com now instead, but that's still better
> than not having the important part of the log at all since it got
> truncated.
>
> Fixes: 998f334722 ("gitlab: show testlog.txt contents ...")
> Signed-off-by: Thomas Huth <thuth@redhat.com>

Queued to testing/next, thanks.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro