From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1mfzwV-0006HY-Oj for mharc-grub-devel@gnu.org; Thu, 28 Oct 2021 03:35:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52146) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mfzwT-0006H4-UE for grub-devel@gnu.org; Thu, 28 Oct 2021 03:35:18 -0400 Received: from mail-qt1-x82c.google.com ([2607:f8b0:4864:20::82c]:36733) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mfzwR-0005iC-J2 for grub-devel@gnu.org; Thu, 28 Oct 2021 03:35:17 -0400 Received: by mail-qt1-x82c.google.com with SMTP id h14so4116502qtb.3 for ; Thu, 28 Oct 2021 00:35:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficientek-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Gk1ITD46hGY0lMOMdqsOpCHLikQR9YiXuFwt1mMTeM0=; b=Z/INAIFWNbJOb+kIVHbY2nXOi4gUB9FC9ed46Tk7/YfPX7pyIG5O2Ro5njJm5wJCJU oZDTppDhEOAT8CIc9RQmKJ6jYOYilJ+SHao8IR3EpDQ/5XR9sm7vCi335fnkyqYKNqKK T8/tmCJoehLTJ7FWXICPupBWqR/LpIL39Jseqs6QPL1CVVpmNgvCOiAwyfywG53wsFN4 aMhMsmNUuxhq6ZRkVxBC5eR3btBRe27DgZIKCWeIdwCsyadSbQwEMA5imac+1pygdm2Z k0DFfz9DiNMKbF2HQYXt1LtjJsyloV/T5S58yoZgoTvKGt0ZxPH0ZF0JwqKfmogvGlAr GviA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Gk1ITD46hGY0lMOMdqsOpCHLikQR9YiXuFwt1mMTeM0=; b=qoV8pO+C5cWManSz8CO+Js5VVNil5MyIdQzBtrmsrckMABW/rvEN1dLZ0Gi6rAYAbB d0S8UBUSjlxRM9eqRihGXADJSgx+g2I+Losyj+65Ed3vUwsX+D0rxLHdaUy12dFHNcaK mcY00/PsUsVVMYpwbxu0u6Mo3zBVUcxjxS8FCnrML1C81NMr918vQP+1LNzq/wdeIQQp xH7Yo9VfBLGFbYbOU/wWRN+FFWZShr1xwZ+ESDK/hhlpgaMEsD05EoDyofl5kM6MFF5U s2+gWZn1prMOipsPLxk8gqE27KwxEKzOJDpbi3VzU26yk+Z5GXL8YqqBsEGkjuqXnXPy qW5A== X-Gm-Message-State: AOAM531BHvdalsXSsHqQYnLyoEVS6zq2HN77j25g8lhsFLR3asntgfFW 5GWClMWYK3CBJi9aJCOzwD5zAAByK/X0FA== X-Google-Smtp-Source: ABdhPJzWWVs9EtlYtBZ7yhRLrxbk3ELZHoxBagPOc0f5NJkAex8JT3wmSUUzZQB2ORIhMaDjjVpjDg== X-Received: by 2002:a05:622a:4:: with SMTP id x4mr2779599qtw.185.1635406512882; Thu, 28 Oct 2021 00:35:12 -0700 (PDT) Received: from localhost.localdomain ([37.218.244.251]) by smtp.gmail.com with ESMTPSA id k14sm1441086qko.120.2021.10.28.00.35.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Oct 2021 00:35:12 -0700 (PDT) From: Glenn Washburn To: grub-devel@gnu.org, Daniel Kiper Cc: Glenn Washburn Subject: [LOCAL-CI 0/3] Add support for local automated testing Date: Thu, 28 Oct 2021 02:34:44 -0500 Message-Id: X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2607:f8b0:4864:20::82c; envelope-from=development@efficientek.com; helo=mail-qt1-x82c.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: grub-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: The development of GNU GRUB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Oct 2021 07:35:18 -0000 This patch series aims to make automated testing of all supported targets easy. Not all targets are actually tested for a variety of reasons, but this series covers a significant portion of the target space and should make it easy to add more targets as we figure out how do run their tests. All supported targets (that I know of) are build tested. The main work horse is the scripts/local-tester.sh script which ties everything together. It is very configurable, but defaults to building and testing as much as it can. To start it just run "./scripts/local-tester.sh" from the root of the repository. By default a directory named 'grub-tests' in the current working directory is created and everything is put in there. See the beginning of ./scripts/ci/function.sh for available environment variables that can be used to configure how you want it to run. The file ./scripts/ci/function.sh has a bunch of functions most of which are for stages in the automated testing. It is intended to be sourced by config files for other CI systems so that they can reuse this code. One might ask, why not just have all CI systems use local-tester.sh and put everything in there? The issue is that currently local-tester.sh does not do things in parallel (the make subprocess can be made more parallel with the JOBS environment variable). So in many CI systems, one could have all targets building and testing at the same time, which local-tester.sh only does this serially. This makes local-tester.sh a lot slower than it needs to be. Also other CI systems allow the CI pipeline to e broken into many stages, each of which could be run on a different machine, with the ability to cache the output of certain stages. These scripts have been written and tested on debian based systems, specifically the reference system, Debian 11. Some of the stages are debian or debian- derivative specific, such as the package install stage which uses apt and dpkg. Most of the stages, I believe, are fairly distro agnostic and the ones that aren't should be able to be adapted for a specific distro fairly easily. Also, this patch series is meant to be used on top of the grub-shell patch series already submitted to the list. It will run without that series, but some of the features may not work or work as well. Noteably, the QEMU tests for targets i386-efi, arm-efi and arm64-efi will fail. Of particular note, there are some knobs that can provide a lot debugging output and save the intermediate files of failed tests for later analysis. local-tester.sh will try to download and install all packages it needs to function. Obviously, this will not work when not running as a privileged user. A further patch series is intended, which will add support for running the system successfully completely as an unprivileged user. If local-tester.sh is run as an unprivileged user, it will skip running of privileged commands, like the package installer. This means it can continue past the package install phase, but it assumes that the needed packages are already installed. Patch 1: Add general scripts for automated building/testing. Patch 2: Add automated building/testing stages and respective configuration variables. Patch 3: Add consumer of those stages, local-tester.sh, for running the automated building/testing locally. I'm not particularly in love with the name "local-tester.sh", so suggestions on alternative names are welcome. The same goes for the output, which I think could look better. Glenn Glenn Washburn (3): scripts: Add general scripts to aid automated testing scripts: Add functions for CI stages and default variables to functions.sh scripts: Add local-tester.sh script to run local CI tests scripts/ci/build.sh | 67 +++ scripts/ci/functions.local.sh | 37 ++ scripts/ci/functions.sh | 930 ++++++++++++++++++++++++++++++++++ scripts/ci/make-images.sh | 86 ++++ scripts/ci/process-tests.sh | 111 ++++ scripts/ci/test.sh | 120 +++++ scripts/local-tester.sh | 39 ++ 7 files changed, 1390 insertions(+) create mode 100755 scripts/ci/build.sh create mode 100644 scripts/ci/functions.local.sh create mode 100644 scripts/ci/functions.sh create mode 100755 scripts/ci/make-images.sh create mode 100755 scripts/ci/process-tests.sh create mode 100755 scripts/ci/test.sh create mode 100755 scripts/local-tester.sh -- 2.27.0