Re: [OE-core][RFC] tests artifacts retriever

[Alexandre Belloni <alexandre.belloni@bootlin.com>]

On 23/05/2023 17:37:25+0200, Alexis Lothor� wrote:
> Hello Alexander, thanks for the feedback
> 
> On 5/23/23 11:50, Alexander Kanavin wrote:
> > I think having to write a custom artifacts.conf for each ptest is not
> > a good direction, as it's both too much work, and requires good
> > understanding what is important in the output and what isn't :) When a
> > ptest fails, it would be enough to simply make a tarball of
> > /usr/lib/component/ptest/ with everything in it. Maybe /tmp and
> > /var/log as well.
> 
> Indeed, it can be tedious to write an artifacts conf file for each ptest.
> If it is tolerable to get an archive of the whole ptest directory (+ possibly
> /var/log and /tmp), we can go for it and fine-tune later if needed. In this
> case, I could either keep artifacts.conf to list the directories we want for all
> ptests, or completely drop it.
> 
> I will let this RFC run for more time before starting to implement something,
> but if no one come back on this, I will apply your suggestion.
> 

I think the requirement that I didn't convey properly is that there
should be one global conf file for the build.

> Kind regards,
> 
> > 
> > Alex
> > 
> > On Mon, 22 May 2023 at 16:58, Alexis Lothor� via
> > lists.openembedded.org
> > <alexis.lothore=bootlin.com@lists.openembedded.org> wrote:
> >>
> >> On 5/22/23 16:52, Alexis Lothor� via lists.openembedded.org wrote:
> >>> Hello everyone,
> >>> I have been briefed about the need to add an infrastructure to be able to
> >>> retrieve some test files (test output, logs, etc) in Yocto/CI
> >>> automation to ease debugging of some hard-to-reproduce issues. Before
> >>> starting to implement a solution, I would like to get some feedback about
> >>> what I have understood from the original issue and what I propose to
> >>> implement to improve this. Please find below my reflexions, feel free
> >>> to comment, correct, enrich and/or approve.
> >>>
> >>> # Motivation
> >>>
> >>> Some test failures appearing in automatic testing can be hard to fix, based
> >>> on too few informations from failing tests (see
> >>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901 ). Sometimes
> >>> packages/binaries under test are able to generate more detailed output in a
> >>> dedicated file, but there is actually no standardized way of retrieving
> >>> such output files. There has been some custom efforts done to solve this issue
> >>> for specific recipes, for example for lttng-tools tests :
> >>> https://git.yoctoproject.org/poky/commit/?id=2ea27691aa57951aaba3cc1714a080a112d15408
> >>>
> >>> The goal of this RFC is to propose an architecture for a generic system which
> >>> allows to retrieve an arbitrary list of files from a test run.
> >>>
> >>> # Use case example
> >>>
> >>> lttng-tools recipe provides LTTNG tools tests as ptests (see
> >>> meta/recipes-kernel/lttng/lttng-tools/run-ptest) and generate some log
> >>> files (error.log, test-suite.log), it would be convenient to retrieve those
> >>> files.
> >>>
> >>> # Proposal
> >>>
> >>> - Introduce an "artifacts.conf" file
> >>>   - this file contains a "files of interest" list to be retrieved onto
> >>>     host after running ptests (exact format to be defined)
> >>>   - artifacts.conf is specific to a ptest package and is stored next to
> >>>     run-ptest script in package
> >>>   - artifacts.conf contains, for each ptest package, the following
> >>>     informations:
> >>>     - name of file to retrieve
> >>>   - artifacts.conf will be installed in target image (as for
> >>>     package-specific run-ptest script)
> >>>   - each ptest package MAY have an artifacts.conf file
> >>> - When running ptests, for each available ptest, ptest-runner (on the
> >>>   target) will read artifacts.conf if it exists, and retrieve
> >>>   corresponding artifacts file in an artifacts directory, with one
> >>>   subdirectory per ptest.
> >>>   - Artifacts will be aggregated only if at least one test has a status
> >>>     different from "PASS"
> >>> - Once tests are finished, tests manager on host (ptest.py) will fetch
> >>>   artifacts directory. It will put this directory alongside testresults
> >>>   file on host
> >>> - This mechanism will work for both real targets and emulated targets
> >>>   (Qemu), and will based on SSH to transfer artifacts back to host
> >>>
> >>> # Additional questions
> >>>
> >>> - once retrieved from target, aside from default test results directory
> >>>   (build/tmp/log/oeqa/<...>), where should we store artifacts ? For
> >>>   automated testing, should we publish them alongside other test and
> >>>   regression reports (in web server directory) ?
> >>> - current proposal is focused on ptests: should I broaden the scope and
> >>>   include all kind of runtime tests (as found in
> >>>   meta/lib/oeqa/runtime/cases ?)
> >>> - are there examples of tests producing files to retrieve with non-static
> >>>   names (which will then require the artifacts.conf to be smarter) ?
> >>>
> >>
> >> Fixing Alexandre Belloni's address in CC, sorry for the noise
> >>
> >> --
> >> Alexis Lothor�, Bootlin
> >> Embedded Linux and Kernel engineering
> >> https://bootlin.com
> >>
> >>
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >> Links: You receive all messages sent to this group.
> >> View/Reply Online (#181597): https://lists.openembedded.org/g/openembedded-core/message/181597
> >> Mute This Topic: https://lists.openembedded.org/mt/99066378/1686489
> >> Group Owner: openembedded-core+owner@lists.openembedded.org
> >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >>
> 
> -- 
> Alexis Lothor�, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
> 

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com