Openembedded Core Discussions
 help / color / mirror / Atom feed
From: "Alexis Lothoré" <alexis.lothore@bootlin.com>
To: <Openembedded-core@lists.openembedded.org>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>,
	Alexandre Belloni <alexandre.belloni@bootlin.com>
Subject: [PATCH 0/2] oeqa/utils/postactions: fix disk storage consumption
Date: Fri,  5 Jul 2024 16:46:36 +0200	[thread overview]
Message-ID: <20240705144638.441976-1-alexis.lothore@bootlin.com> (raw)

Hello,
this small series aims to fix Bugzilla issue 15536 [1], which results in a
big disk storage consumption and/or inode max count issue. The main root
cause is that in its current form, the artifacts retriever convert symlinks
to the targeted files. This series changes the way those files are
retrieved to make sure the symlinks are preserved. To do so, it replaces
the series of scp performed on each file by a single tar command which
output the generated compressed archive on its stdout. It is captured and
returned as part of an ssh process, and it it then fed to another tar
command run this time on host, to extract the artifacts, at the same place
as before (tmp/log/oeqa/artifacts)

The first commit is a small update to SSH target in ssh.py to allow to
retrieve raw output (ie raw bytes, without any decoding or formatting) from
a command run on the target. This allows to get the compressed archive
directly as an output from the remote command.
The second commit uses this new feature to generate, retrieve and extract
this archive on the fly, without ever storing it onto the target nor the
host.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=15536

Alexis Lothoré (2):
  oeqa/ssh: allow to retrieve raw, unformatted ouput
  oeqa/utils/postactions: transfer whole archive over ssh instead of
    doing individual copies

 meta/lib/oeqa/core/target/ssh.py   | 16 ++++++++--------
 meta/lib/oeqa/utils/postactions.py | 19 +++++++++----------
 2 files changed, 17 insertions(+), 18 deletions(-)

-- 
2.45.2



             reply	other threads:[~2024-07-05 14:47 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-07-05 14:46 Alexis Lothoré [this message]
2024-07-05 14:46 ` [PATCH 1/2] oeqa/ssh: allow to retrieve raw, unformatted ouput Alexis Lothoré
2024-07-05 14:46 ` [PATCH 2/2] oeqa/utils/postactions: transfer whole archive over ssh instead of doing individual copies Alexis Lothoré
2024-07-05 15:06   ` Patchtest results for " patchtest
2024-07-07  8:10   ` [OE-core] " Richard Purdie
2024-07-08 21:10     ` Alexis Lothoré
2024-07-08 21:37       ` Richard Purdie
2024-07-09  8:06         ` Alexis Lothoré

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240705144638.441976-1-alexis.lothore@bootlin.com \
    --to=alexis.lothore@bootlin.com \
    --cc=Openembedded-core@lists.openembedded.org \
    --cc=alexandre.belloni@bootlin.com \
    --cc=thomas.petazzoni@bootlin.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox