Re: GRUB release schedule?

[Bruce Dubbs <bruce.dubbs@gmail.com>]

Konrad Rzeszutek Wilk wrote:
> On Fri, Aug 21, 2015 at 09:24:33PM +0300, Andrei Borzenkov wrote:

>> GRUB includes comprehensive amount of regression tests. Just run "make
>> check". The practical problems are
>>
>> - many tests require additional tools (filesystem tests need at least mkfs
>> for respective file system, LVM etc)
>>
>> - each platform must be built separately; that requires either native system
>> or cross tools (which itself may not be trivial). So I e.g. am limited to
>> x86
>>
>> - tests are not really formalized, you get PASS/FAIL but what failed is up
>> to human to understand
>>
>> - some tests require server part, e.g. to run anything involving HTTP server
>> must be available
>>
>> - some tests are pretty heavy hit; it is better now when I have new hardware
>> still I cannot dream running them continuously on my notebook ...
>>
>> Of course addition to regression testing is always welcome.
>
> Lets start with a list of priorities:
>   - What are the most important platforms after x86?
>   - What are the most important tests that MUST PASS all the time?
>   - Which ones have been FAILing for years?
>
> Surely if we weed out the most important cases that cover 99% that will
> give the foundation for going out with a release?

Although tests are very useful, not all packages ship tests.  One prime 
example is the linux kernel, but there are many more.

Tests depend on external programs and specific setups used by the 
developers, but often are not available on builder's systems.  For 
example, in grub-2.02~beta2, I get:

============================================================================
Testsuite summary for GRUB 2.02~beta2
============================================================================
# TOTAL: 78
# PASS:  12
# SKIP:  18
# XFAIL: 0
# FAIL:  48
# XPASS: 0
# ERROR: 0

That doesn't mean that the build is bad on a x86_64 system.  It works 
quite well (although grub-mkconfig always produces an unusable 
configuration for us.)


A lot of the FAILs are due to things like:

FAIL: iso9660_test
==================

cp: cannot stat '/usr/share/dict/linux.words': No such file or directory

although I have other dictionaries.

FAIL: pata_test
===============

tar: Removing leading `/' from member names
timeout: failed to run command 'qemu-system-i386': No such file or directory

Although I have /usr/bin/qemu -> qemu-system-x86_64

This corresponds to 45 of the 48 "FAILs" above.  Creating a symlink 
qemu-system-i386 to qemu-system-x86_64 allows most ot the tests to pass, 
but hangs after test_unset.

++++++++++++

In other words, the tests are highly sensitive to the user's system.

Please do not let the tests shipped in the tarball hold up a release. 
Perfect is the enemy of good enough.

   -- Bruce Dubbs
      linuxfromscratch.org