From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1mO3iK-0003Tj-Cl for mharc-grub-devel@gnu.org; Wed, 08 Sep 2021 15:58:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33374) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mO3iH-0003TL-1a for grub-devel@gnu.org; Wed, 08 Sep 2021 15:58:29 -0400 Received: from mail-pj1-x1034.google.com ([2607:f8b0:4864:20::1034]:35679) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mO3iE-0002Wz-Nv for grub-devel@gnu.org; Wed, 08 Sep 2021 15:58:28 -0400 Received: by mail-pj1-x1034.google.com with SMTP id f3-20020a17090a638300b00199097ddf1aso217016pjj.0 for ; Wed, 08 Sep 2021 12:58:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficientek-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=UOykc3R1lcwPh0Gmtwv9824PskEVR1z2c9tNYLuiFdI=; b=MYzoa2jKjQCQ+yux5kGGn0AM6eRgjr2B0ekbUbMXvNRfjmRkIYcn/MMo/+k6qIpzdS EOy5oZlXR6w0Dd+r2jz9EcgUdJte4nhZKv5Fawdrumlf87xJsx19o1+7EJxlvtZWX/ey 82/z3lY+ahzkt7uTxOf4vetqHKmyuDH6jcFPxTaEElLysE6+pxzGYCu7355ZAeZKvmyl z3+ZMb4upQuTjpWQDqA0MzC3wzqy0Rjmw4sDhKxyD+WFzz7SQaggYT8/qakt7zOAnte0 ndTQVZUzXj4x8XQMe/nx0hf2g21k/Bf/9hDS6MBHSxZcWw0RKs2jN0suGyAesHi6U2Xj v/Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=UOykc3R1lcwPh0Gmtwv9824PskEVR1z2c9tNYLuiFdI=; b=365sbrhW93zs2t26fOA9/xy+/CNZHAGEUtic9M8mmgVnKWs91pOr68EdoMDjGoDAK3 iinmqD3j89UKFPynEuACXLs0rPduavqa33cOLO04e8dQoqux9T/KSA5Vuce7Ap3i85MI usIKAj7VJ4xSyDB8tIROJvhaSn4B1pXFWPj6+wDlZXPhCHjLlfLLDJS5D2nQOkTm7a5k /DTUW7kMjX/nbjNUR8m4kpfcHoTv2gnzk/2vBPAREDhVrE7jHW2KiCvudF0Z2nl4+4QN lEdf1bXaCjiS7L4G7sf3p5iakWcYR/vJmzBXRjZsU0sIaXuy6UuSJKhjfbH+up9cbVqf TSIw== X-Gm-Message-State: AOAM531lupKF5VzWeh8w8FV3cvFgKjeuxS37ea+2K/1RCte21jjZbiby l5hIO/0bfLpWRNsxSbi4x9qvxw== X-Google-Smtp-Source: ABdhPJxfPRPd1SwWJ9hvw20eFu4xrmvksJq0V0838cI6h99yZ7sR5woEMb/R6oVomcng7n+7WTTf2g== X-Received: by 2002:a17:902:aa02:b0:13a:6c8f:407f with SMTP id be2-20020a170902aa0200b0013a6c8f407fmr435560plb.59.1631131104770; Wed, 08 Sep 2021 12:58:24 -0700 (PDT) Received: from ubuntu ([185.220.103.12]) by smtp.gmail.com with ESMTPSA id n79sm38788pfd.167.2021.09.08.12.58.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Sep 2021 12:58:24 -0700 (PDT) Date: Wed, 8 Sep 2021 19:57:53 +0000 From: Glenn Washburn To: Daniel Kiper Cc: Carlos Maiolino , The development of GNU GRUB , Erwan Velu , Erwan Velu , javierm@redhat.com Subject: Re: Where is the testing? (was: Re: [PATCH] fs/xfs: Avoid unreadble filesystem if V4 superblock) Message-ID: <20210908195753.5f388dad@ubuntu> In-Reply-To: <20210908160350.farvryv6xyrodua7@tomti.i.net-space.pl> References: <20210825133152.1165682-1-e.velu@criteo.com> <20210826132616.dhhiwdowapcolrdr@omega.lan> <20210830114850.kemzuknbe4nwzkip@omega.lan> <20210901124057.qqmqkjh7k3dkgnsv@tomti.i.net-space.pl> <20210902085649.zucibvxhnez2meys@omega.lan> <20210908012220.60a34a2a@ubuntu> <20210908160350.farvryv6xyrodua7@tomti.i.net-space.pl> Reply-To: development@efficientek.com X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=2607:f8b0:4864:20::1034; envelope-from=development@efficientek.com; helo=mail-pj1-x1034.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: grub-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: The development of GNU GRUB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2021 19:58:29 -0000 On Wed, 8 Sep 2021 18:03:50 +0200 Daniel Kiper wrote: > On Wed, Sep 08, 2021 at 01:22:20AM +0000, Glenn Washburn wrote: > > It looks like the xfs_test test succeeds with tag grub-2.06-rc1a, > > fails with tag grub-2.06, and succeeds with current master. Yes, as > > expected. However, what this tells me is that no "make check" tests > > were done before finalizing the 2.06 release. I was under the > > impression that that was part of the release procedure. If its not, > > it would seem that we're not using the tests at a time when they > > would have the most impact. > > Currently I run build tests for all architectures and platforms and > Coverity for x86_64 EFI before every push and release. I know this is > not enough but I tried "make check" at least once and got the > impression that the tests are not reliable. Sadly I have not time to > dive deeper and fix this and that. If someone, you?, wants to verify > all tests and fix broken ones I will be more than happy to start > using it (it seems to me your "[PATCH v2 0/8] Various > fixes/improvements for tests" patch set fixes some issues but I am > not sure if all of them). I suspect the problem you're running into with the tests is more a matter of setting up the expected environment than the tests themselves. From my experience, most of the tests work once everything is setup right. My gitlab patch shows how to do this on Ubuntu 20.04. Due to not being able to load kernel modules on the shared gitlab runners, I was not able to run all the tests either, and there are some tests that fail. The failures that come to mind are on Sparc and some of the functional tests, and I've notified the list, but no one came forward to fix them. So what I've done is have a list of known failing tests and ignore those in determining whether the test suite as a whole has succeeded. I'm not convinced that that behavior is something we should include in the test framework itself. So to my mind, I have verified (nearly) all tests on most platforms. I have fixed the ones I could. I think the biggest low hanging fruit (for me), is to get the tests running on a system with all the needed kernel fs drivers loaded. Perhaps to your point, the filesystem tests failing due to not having a kernel module loaded should be marked as skipped because the test isn't really run. Here is an example of a successfully completed pipeline https://gitlab.com/gwashburn/grub/-/pipelines/258561619 And here is the raw log for the x86_64-efi build and test (search for "Testsuite summary" and look at the lines above for individual test pass/fail): https://storage.googleapis.com/gitlab-gprd-artifacts/0e/0d/0e0d2ccc27a1e9a121b3d4ef831c56b5fd81cc8b565a3dbc5fa09b9a58edbcb7/2021_02_19/1042905700/1137549157/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&GoogleAccessId=gitlab-object-storage-prd@gitlab-production.iam.gserviceaccount.com&Signature=ewpRWBqWWorOXak6hFkb5kUEfefU1biAtu6xY2Rtyds3%2BduM6q0kiFIKe4A5%0A8wS%2FPbd8Al3AwF45Q22KcpYyZ87UBkryQHjispAj%2B3tBQrnnyOYSRKGNV%2Bbz%0A4iaKAdYn4onIcJM9Ro4RkJ%2FCAY2tBxWmmLoEZ%2FQzWLvbhH%2BJSmuNqtEZXfuD%0AI1tXD1ZKgoLcYCCLNz8iaMFpgB32NfR2W6sDDuFb24ZFSy0X7H02KRVAMIor%0Akhj5QXdXGqoqFFXXMM9r8ZGv34Kh4GdDbVjMmJ%2FaDhvLPgmZF3UKnhcDYJoB%0AiMj%2BbimYNoiQW0SLg4hHA11PkKNn4KPAFqhLVscsCw%3D%3D&Expires=1631130462 This is all to show that the tests are (mostly) working. There's some post-processing on the pass/fail status of some tests to ignore expected failures (eg. zfs_test). Occasionally there are spurious test failures, I think due to the VM getting starved at times (eg. why grub_cmd_sleep fails and is ignored). No as far as you personally using the tests, based on the above what do we need to get that to happen? I don't know the setup that you plan on doing the tests on (is it debian based?). Do you want a script that sets up the environment for you? Perhaps a docker or lxd setup (probably better in the long run)? A word about why I haven't verified the fs tests that require kernel modules not loaded in gitlab shared runner, I could run these on my system. However, I've had some strange experiences in the past with the fs tests not cleaning up well and causing some issues with the rest of the system. So I'm not keen on running all the fs tests on my daily driver. Perhaps in a container (a virtual machine would work, but I suspect really slow, because many of the tests use virtual machines for testing, so there would be VMs in a VM). I'd be curious to know what specific issues you had with the test, if you can remember or reproduce them. > > It is my understanding that we have travis-ci tests that get run (at > > some point?), however they are only build tests and so would not > > have caught this. It was precisely this scenario that I hoped to > > avoid by doing more thorough continuous integration, which runs the > > extensive array of "make check" tests, when I submitted the > > Gitlab-CI patch series (which would've caught this automatically if > > it had been merged). To me this scenario is the poster child for > > taking action on this front. Can we make this a priority? > > I think I can take a closer look at patch set mentioned above in a > week or two. If "make check" works as expected and can be run locally > then we can think about automating the testing. I think the requirement to "run locally" should be more precisely defined. Is that all tests for all architectures? Running it reliably on an local machine might be difficult due to the variability of configurations. Perhaps in a docker or lxd container would be the best route for that. Thoughts? Glenn