Re: QA: Monitor Linux log messages as port of release (candidate) testing

[Guenter Roeck <linux@roeck-us.net>]

Hi Paul,

On Tue, Sep 07, 2021 at 03:50:39PM +0200, Paul Menzel wrote:
> Dear Guenter,
> 
> 
> Am 07.09.21 um 14:53 schrieb Guenter Roeck:
> > On Tue, Sep 07, 2021 at 10:40:31AM +0200, Paul Menzel wrote:
> 
> > > Thank you for testing release candidates and releases [1]. Is your test
> > > setup documented somewhere?
> > > 
> > Not really, except its source is available at github:
> > 	https://github.com/groeck/linux-build-test
> 
> Thank you.
> 
> > > If not happening already, could the Linux messages (at least up to log level
> > > warning) also be monitored? For example, in Linux 5.14, a new warning snuck
> > > in by cefc7ca462 (ACPI: PRM: implement OperationRegion handler for the
> > > PlatformRtMechanism subtype) [2], which could have been caught early on, and
> > > fixed before the release.
> > > 
> > > The test summaries would then also notify about possible behavior change.
> > 
> > Logs are available and can be examined at kerneltests.org/builders.
> 
> Sorry for being blind. Under *qemu-tests*, looking at build #1831 [1],
> clicking on *stdio* [2] under *Steps and Logfiles*, I do not see any Linux
> logs.
> 
> > Reports are generated manually, so it would be way too much effort to add
> > build warnings to those. Besides, logs are way too noisy to be useful in a
> > summary e-mail.
> 
> Just to avoid misunderstandings, it’s about the Linux run-time logs.
> 
Run-time logs are only provided if there are errors or runtime issues
(crashes, warning tracebacks, or test failures).

> > Also, Geert's build reports already provide build warnings and errors.
> > The same applies to reports sent by 0-day. Indeed, I do see at least
> > one 0-day report against commit cefc7ca46235.
> 
> How can I find that report?
> 
I just searched for the SHA.

https://www.spinics.net/lists/linux-acpi/msg101721.html

> > What would be the point of adding yet another report of build
> > warnings on top of that ?
> If the functionality already exists, great. But to be clear, it’s about the
> runtime logs.
> 

If there are (new) runtime issues (crashes, warnings, or other test failures),
I usually analyze, bisect, and report the problem against the patch introducing
the problem unless it was already reported elsewhere.

For example, there is currently a backtrace in arm64 tests:

BUG: sleeping function called from invalid context at kernel/locking/semaphore.c:163

which is due to ACPI code being called from the wrong context.
Another example is the cirular locking problem reported in various
mips tests, from mtd code. Both problems have been fixed in -next,
and the fixes will hopefully be pushed upstream soon.

The qemu tests do not log build warnings. I used to do that, but it
was way too noisy (some builds used to produce hundreds of build
warnings). Also, there is no logging data if there are neither
crashes nor warnings or other test failures, for the same reason.
If you are looking for complete boot logs, I'd suggest looking at
test results from kernelci.org.

Thanks,
Guenter