All of lore.kernel.org
 help / color / mirror / Atom feed
From: Max Yu <yumx@amazon.com>
To: <openembedded-core@lists.openembedded.org>
Cc: Max Yu <yumx@amazon.com>, Przemyslaw Sobon <psobon@amazon.com>
Subject: [PATCH] sstate: remove corrupted artifacts from local mirror
Date: Sat, 5 Oct 2024 00:44:51 +0000	[thread overview]
Message-ID: <20241005004451.79575-1-yumx@amazon.com> (raw)

We observe sstate cache corruptions sometimes which cause rebuilds. That is not
a fatal error as the package has to be rebuilt and updated artifact needs to be
pushed to remote sstate cache mirror. Currently, Yocto does not handle
corruptions properly, where the corrupted artifact is not deleted or
renamed. Later, after the package is built the same corrupted artifact is pushed
to remote mirror and the same procedure is circled again and again.

This change verifies the outcome of the unpacking action and renames the
artifact if a fatal error occurred ("tar" tool returns error 2). In such case we
rename the artifact what causes that a proper one is created and uploaded
overwriting the exisiting one - the corrupted one - in the remote mirror. That
way we break the loop of uploading corrupted file again and again.

Suggested-by: Przemyslaw Sobon <psobon@amazon.com>
Signed-off-by: Max Yu <yumx@amazon.com>
---
 meta/classes-global/sstate.bbclass | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass
index 11bb892a42..5a7ce35341 100644
--- a/meta/classes-global/sstate.bbclass
+++ b/meta/classes-global/sstate.bbclass
@@ -937,7 +937,12 @@ sstate_unpack_package () {
 		ZSTD="pzstd -p ${ZSTD_THREADS}"
 	fi
 
-	tar -I "$ZSTD" -xvpf ${SSTATE_PKG}
+	if ! tar -I "$ZSTD" -xvpf ${SSTATE_PKG}; then
+		echo "Fatal error extracting sstate cache artifacts, file might be corrupted or truncated, renaming"
+		mv ${SSTATE_PKG} ${SSTATE_PKG}.unpack_error
+		exit 2
+	fi
+
 	# update .siginfo atime on local/NFS mirror if it is a symbolic link
 	[ ! -h ${SSTATE_PKG}.siginfo ] || [ ! -e ${SSTATE_PKG}.siginfo ] || touch -a ${SSTATE_PKG}.siginfo 2>/dev/null || true
 	# update each symbolic link instead of any referenced file
-- 
2.40.1



             reply	other threads:[~2024-10-05  0:45 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-10-05  0:44 Max Yu [this message]
2024-10-07 11:57 ` [OE-core] [PATCH] sstate: remove corrupted artifacts from local mirror Alexander Kanavin
2024-10-07 20:09   ` Yu, Max
2024-10-07 20:38     ` Alexander Kanavin
2024-10-08  9:37     ` Alexander Kanavin
  -- strict thread matches above, loose matches on Subject: below --
2024-10-05  0:16 Max Yu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20241005004451.79575-1-yumx@amazon.com \
    --to=yumx@amazon.com \
    --cc=openembedded-core@lists.openembedded.org \
    --cc=psobon@amazon.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.