From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from list by lists.gnu.org with archive (Exim 4.71) id 1ZSsQ3-00089r-5y for mharc-grub-devel@gnu.org; Fri, 21 Aug 2015 15:56:07 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49805) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZSsQ0-00088Z-8E for grub-devel@gnu.org; Fri, 21 Aug 2015 15:56:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZSsPw-0002FB-7M for grub-devel@gnu.org; Fri, 21 Aug 2015 15:56:04 -0400 Received: from mail-oi0-x22c.google.com ([2607:f8b0:4003:c06::22c]:34281) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZSsPw-0002E1-2I for grub-devel@gnu.org; Fri, 21 Aug 2015 15:56:00 -0400 Received: by oiey141 with SMTP id y141so47882200oie.1 for ; Fri, 21 Aug 2015 12:55:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=phlCHekOP4NNsAb0zY/OJ7h/jUnlQgvKxLs27BoPYwk=; b=tpAvLBA9IM0JPQzuVPJ39CT/iQGjGtUlVnsqc0P0SpT5jtVdfrkGHnBmeZ12kbii3w nLucjeJrfkDQ1Dt0KUyMbxXugiB3ReW3Ga+UHS89jgZYoWmoOpOFofr2ezH2nbZTlmLR OkRUsOO54jRPHQHVgDaqSMLA5/RX2IlnWKP2WaWjYr1VHokmYoz/NYio6rfDwmw/yOaD 4hNpIH51lNGbO7TpofyyRQCXwOx+9bz8tjgF2BHOEEkPQKn9ofx+yFQYmuFctVn0dBgU Jx0CiPfxICgn7cbSSSHcTT/7miLugqRSLEUohQyZzWGbU22Dbc7OyrjOw47gaI+5fh1g v24Q== X-Received: by 10.202.217.215 with SMTP id q206mr2644598oig.110.1440186956381; Fri, 21 Aug 2015 12:55:56 -0700 (PDT) Received: from [192.168.0.75] (cpe-70-123-244-133.satx.res.rr.com. [70.123.244.133]) by smtp.gmail.com with ESMTPSA id h200sm5150605oic.9.2015.08.21.12.55.55 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Aug 2015 12:55:55 -0700 (PDT) Message-ID: <55D7824B.7010008@gmail.com> Date: Fri, 21 Aug 2015 14:55:55 -0500 From: Bruce Dubbs User-Agent: Mozilla/5.0 (X11; Linux i686; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: The development of GNU GRUB Subject: Re: GRUB release schedule? References: <20150720182245.GC14894@redhat.com> <55D7585B.7090508@fb.com> <20150821171142.GM26663@l.oracle.com> <55D76CE1.6040204@gmail.com> <20150821184157.GB1948@l.oracle.com> In-Reply-To: <20150821184157.GB1948@l.oracle.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2607:f8b0:4003:c06::22c X-BeenThere: grub-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: The development of GNU GRUB List-Id: The development of GNU GRUB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Aug 2015 19:56:05 -0000 Konrad Rzeszutek Wilk wrote: > On Fri, Aug 21, 2015 at 09:24:33PM +0300, Andrei Borzenkov wrote: >> GRUB includes comprehensive amount of regression tests. Just run "make >> check". The practical problems are >> >> - many tests require additional tools (filesystem tests need at least mkfs >> for respective file system, LVM etc) >> >> - each platform must be built separately; that requires either native system >> or cross tools (which itself may not be trivial). So I e.g. am limited to >> x86 >> >> - tests are not really formalized, you get PASS/FAIL but what failed is up >> to human to understand >> >> - some tests require server part, e.g. to run anything involving HTTP server >> must be available >> >> - some tests are pretty heavy hit; it is better now when I have new hardware >> still I cannot dream running them continuously on my notebook ... >> >> Of course addition to regression testing is always welcome. > > Lets start with a list of priorities: > - What are the most important platforms after x86? > - What are the most important tests that MUST PASS all the time? > - Which ones have been FAILing for years? > > Surely if we weed out the most important cases that cover 99% that will > give the foundation for going out with a release? Although tests are very useful, not all packages ship tests. One prime example is the linux kernel, but there are many more. Tests depend on external programs and specific setups used by the developers, but often are not available on builder's systems. For example, in grub-2.02~beta2, I get: ============================================================================ Testsuite summary for GRUB 2.02~beta2 ============================================================================ # TOTAL: 78 # PASS: 12 # SKIP: 18 # XFAIL: 0 # FAIL: 48 # XPASS: 0 # ERROR: 0 That doesn't mean that the build is bad on a x86_64 system. It works quite well (although grub-mkconfig always produces an unusable configuration for us.) A lot of the FAILs are due to things like: FAIL: iso9660_test ================== cp: cannot stat '/usr/share/dict/linux.words': No such file or directory although I have other dictionaries. FAIL: pata_test =============== tar: Removing leading `/' from member names timeout: failed to run command 'qemu-system-i386': No such file or directory Although I have /usr/bin/qemu -> qemu-system-x86_64 This corresponds to 45 of the 48 "FAILs" above. Creating a symlink qemu-system-i386 to qemu-system-x86_64 allows most ot the tests to pass, but hangs after test_unset. ++++++++++++ In other words, the tests are highly sensitive to the user's system. Please do not let the tests shipped in the tarball hold up a release. Perfect is the enemy of good enough. -- Bruce Dubbs linuxfromscratch.org