From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67976135A53 for ; Thu, 25 Jan 2024 20:35:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706214915; cv=none; b=TQg0ZvMPTWuVLHHDStzisPshI6ZUNzSlWOPcyPzKpEmNAJd1GjuRkHTVMs56m8dhLxJMXqryIk5rd5ACBrVIGE5yr+fKavelNofsI2l7zkO+B4AwyzbqOggGNiyQZmJ0XmZcw/PkJwtb8ahyo2PZQ3Qrf4Be0GdmkvqdFLNcGqE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706214915; c=relaxed/simple; bh=CLi0KgUZeRLn7UV5NFKlLkCvyM2E+h/V5hB/+Q4N6vo=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=rAiQlIl1F3XSpBFx/PVz4N10M5fw2U99PsSBWkqJPUBwzIC3QYs7X9A+DYyKa3f35jHmoj6Ua1Syss8Ikmj654L3KH+oC4p7T4Fg0XkQwEBnMidodF6WuZY+wSC2ui8k59dEBCx/wIdHFtqSzcHNa7XFlpLZe1iu97/GHghkqrk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=cJMYpdJH; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="cJMYpdJH" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-Type: Content-ID:Content-Description:In-Reply-To:References; bh=on86OTRlSBJ5vlT40z3d1+K2TXZuifc/HMqVLE/FOW0=; b=cJMYpdJHlyjQ7VXtjTujzQnCZj naPhZrpvI0Wb2dtvd8/7XAMNTMtBa7VJnrysnaGV9naU/EdBZYrHa9RTM/P4xNx6Cu2rIt0I7PoLa PgJ8EvYGUoGjWlAQBWXAu2GUJsm/Ti95iJVHxOIM1OmgOV/IZMRPNV22XlV5lHMOpZKsEo7ywMRCx 5++yacgLVVWFYHwL0Do4ck/dPjtcrCpxCi2XTOFpBzU165h7dU9ENblRxwgk/hzc59xAEYlMwdNme dzCH8sIza6TY4S63QQ6vr5Y9uqLGsr86jbyPZH8R+KU5ojvUDPecVTLgPdeH8gpErm+Jgiwc5+vmY jwE51Aew==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rT6RK-00000001ntb-0dUF; Thu, 25 Jan 2024 20:35:10 +0000 From: Luis Chamberlain To: kdevops@lists.linux.dev, da.gomez@samsung.com, p.raghav@samsung.com Cc: Luis Chamberlain Subject: [PATCH 0/6] Fix ordering for systemd remote journal support Date: Thu, 25 Jan 2024 12:35:01 -0800 Message-ID: <20240125203507.430113-1-mcgrof@kernel.org> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain For those that may have missed it, in December I added support for systemd remote journals. This allows us to: 1) reduce the workflow watchdog time, since it is now on the localhost 2) always lets us collect node logs in case they crash and the node is gone after so we can't insepct it 3) let's us use journalctl to look for data on the target nodes The implementation however was to address a series of issues when one is using and enabling SOAK_DURATION of 2.5 hours (so that is CONFIG_FSTESTS_SOAK_DURATION=9900), and so it was designed to cure an existing running system I was fet up with not being able to collect logs for, as crashes are much more frequent to the point you really do often loose access to the system / console. Since I implemented support for a running kdevops system I simply togggled on CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE=y and ran 'make' and then: make journal-server make journal-client And then use it. The other commands are: journal-restart - Restart client upload service journal-status - Ensure systemd-journal-remote works journal-ls - List journals available and sizes journal-ln - Add symlinks with hostnames The issue with this is I had not tested a new cluster bringup. After Daniel Gomez struggled with that, I now looked into that and have identified the issues. The fix is to enhance our semantics on bringup by defining clearly what a provider is and also add clear semantics to allow us to install things *prior* to running the generic devconfig playbook with all the bells and whistles. The last few patches should demonstrate how easy it is now to add new install targets for things we need in the pipeline for provisioning. This should bring using CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE closer to a reality for most folks. The only nagging thing I don't like about it is, is that we require sudo on the local system to query the client logs, and that pollutes your systems's logs with a ton of messages with sudo. So the next thing is to verify we can remove sudo from the watchdog calls, since we already took care of making the systemd remote journal directory with a sticky bit, and then ensured our user is part of the group systemd-journal-remote so to allow our user to read the remote journals without sudo. We just gotta test this a bit more wider. So other than patch review, it's now a good time to ask for wider testing with CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE=y and so we can default CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE=y later. Please let me know what you think. Luis Chamberlain (6): mirror: add a smart git check bringup: split provisioning into 2 steps bringup: share bringup method targe and agument by two steps provision: move all provisioning things to its own Makefile bringup: move journal-server setup early journal-server: fix by adjusting ordering .gitignore | 2 +- Makefile | 31 ++-------- kconfigs/Kconfig.mirror | 24 ++++++-- playbooks/roles/devconfig/tasks/main.yml | 1 + scripts/bringup.Makefile | 56 ------------------ scripts/guestfs.Makefile | 19 ++---- scripts/journal-server.Makefile | 56 ++++++++++++++++++ scripts/provision.Makefile | 74 ++++++++++++++++++++++++ scripts/terraform.Makefile | 10 ++-- scripts/test_git_firewall.sh | 17 ++++++ scripts/vagrant.Makefile | 19 ++---- 11 files changed, 187 insertions(+), 122 deletions(-) create mode 100644 scripts/journal-server.Makefile create mode 100644 scripts/provision.Makefile create mode 100755 scripts/test_git_firewall.sh -- 2.42.0