[RFC PATCH 0/1] classes: add workdir_save class

[RFC PATCH 0/1] classes: add workdir_save class

[Paul Eggleton]

This is fairly trivial but I've found it useful in my setup - I'd like to
know if others would as well. Improvement suggestions welcome.


Please review the following changes for suitability for inclusion. If you have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

The following changes since commit b3c96103a5063eeefb0c537227eab3f77616b9c0:

  libnl: Extend for native/nativesdk (2020-07-08 10:56:11 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/workdir_save
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/workdir_save

Paul Eggleton (1):
  classes: add workdir_save class

 meta/classes/workdir_save.bbclass | 54 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
 create mode 100644 meta/classes/workdir_save.bbclass

-- 
1.8.3.1

[RFC PATCH 1/1] classes: add workdir_save class

[Paul Eggleton]

If you are running your builds inside an environment where you don't
have access to the build tree (e.g. an autobuilder where you can only
download final artifacts such as images), then debugging build failures
can be difficult - you can't examine log files, the source tree or output
files. When enabled, this class triggers on task failure and saves a
tarball of the work directory for the task's recipe and puts it in a
configurable location, where it can be picked up by a separate process
and made available as a downloadable artifact.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
---
 meta/classes/workdir_save.bbclass | 54 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
 create mode 100644 meta/classes/workdir_save.bbclass

diff --git a/meta/classes/workdir_save.bbclass b/meta/classes/workdir_save.bbclass
new file mode 100644
index 0000000..7007041
--- /dev/null
+++ b/meta/classes/workdir_save.bbclass
@@ -0,0 +1,54 @@
+# Save the work directory for a recipe when one of its tasks fails.
+# Useful in cases where the environment in which builds are run is
+# ephemeral or otherwise inaccessible for examination during
+# debugging.
+#
+# To enable, simply add the following to your configuration:
+#
+# INHERIT += "workdir_save"
+#
+# Notes:
+# * For this to work you also need corresponding logic in your build
+#   orchestration tool to pick up any files written out to WORKDIR_SAVE_DIR
+#   (with the other assumption being that no files are present there at
+#   the start of the build).
+# * Work directories can be quite large, so saving them can take some time
+#   and of course space.
+#
+# Copyright (c) 2020 Microsoft Corporation
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+WORKDIR_SAVE_DIR ?= "${TMPDIR}/workdir_save"
+WORKDIR_SAVE_ENABLED ?= "1"
+
+addhandler workdir_save_handler
+workdir_save_handler[eventmask] = "bb.build.TaskFailed"
+
+python workdir_save_handler() {
+    import datetime
+    if d.getVar('WORKDIR_SAVE_ENABLED') != '1':
+        return
+
+    base_workdir = d.getVar('BASE_WORKDIR')
+    workdir = d.getVar('WORKDIR')
+    outdir = d.getVar('WORKDIR_SAVE_DIR')
+    bb.utils.mkdirhier(outdir)
+    pn = d.getVar('PN')
+    tstamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
+    tarfn = 'workdir_%s_%s.tar.gz' % (pn, tstamp)
+    tarfp = os.path.join(outdir, tarfn)
+    taskname = d.getVar('BB_CURRENTTASK')
+    bb.plain('NOTE: Saving workdir for failed task %s.do_%s to %s' % (pn, taskname, tarfp))
+    try:
+        bb.process.run(['tar', 'czf', tarfp,
+                os.path.relpath(workdir, base_workdir)], cwd=base_workdir)
+    except bb.process.ExecutionError as e:
+        # It is possible for other tasks to be writing to the workdir
+        # while we are tarring it up, in which case tar will return 1,
+        # but we don't care in this situation (tar returns 2 for other
+        # errors so we we will see those)
+        if e.exitcode != 1:
+            bb.warn('workdir save error: %s' % str(e))
+}
-- 
1.8.3.1

Re: [OE-core] [RFC PATCH 1/1] classes: add workdir_save class

[Richard Purdie]

On Wed, 2020-07-08 at 11:02 -0700, Paul Eggleton wrote:
> If you are running your builds inside an environment where you don't
> have access to the build tree (e.g. an autobuilder where you can only
> download final artifacts such as images), then debugging build
> failures
> can be difficult - you can't examine log files, the source tree or
> output
> files. When enabled, this class triggers on task failure and saves a
> tarball of the work directory for the task's recipe and puts it in a
> configurable location, where it can be picked up by a separate
> process
> and made available as a downloadable artifact.
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>

Seems like a useful idea. I think the name may need "failed" in the
name to make it clearer what it does. We also need a section for the
classes chapter in the manual.

As another data point, OEQA is using OEQA_DEBUGGING_SAVED_OUTPUT to
trigger saving information for reproducible builds...

Cheers,

Richard

Re: [OE-core] [RFC PATCH 1/1] classes: add workdir_save class

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 8 juli 2020 20:12
> To: Paul Eggleton <paul.eggleton@linux.microsoft.com>; openembedded-
> core@lists.openembedded.org
> Subject: Re: [OE-core] [RFC PATCH 1/1] classes: add workdir_save class
> 
> On Wed, 2020-07-08 at 11:02 -0700, Paul Eggleton wrote:
> > If you are running your builds inside an environment where you don't
> > have access to the build tree (e.g. an autobuilder where you can only
> > download final artifacts such as images), then debugging build
> > failures
> > can be difficult - you can't examine log files, the source tree or
> > output
> > files. When enabled, this class triggers on task failure and saves a
> > tarball of the work directory for the task's recipe and puts it in a
> > configurable location, where it can be picked up by a separate
> > process
> > and made available as a downloadable artifact.
> >
> > Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
> 
> Seems like a useful idea. I think the name may need "failed" in the
> name to make it clearer what it does. We also need a section for the
> classes chapter in the manual.
> 
> As another data point, OEQA is using OEQA_DEBUGGING_SAVED_OUTPUT to
> trigger saving information for reproducible builds...
> 
> Cheers,
> 
> Richard

How about "save_workdir_on_failure.bbclass"?

//Peter

Re: [OE-core] [RFC PATCH 1/1] classes: add workdir_save class

[Paul Eggleton]

On 10/07/2020 5:02 am, Peter Kjellerstedt wrote:
 >Richard Purdie wrote:
>> Seems like a useful idea. I think the name may need "failed" in the
>> name to make it clearer what it does. We also need a section for the
>> classes chapter in the manual.
>>
>> As another data point, OEQA is using OEQA_DEBUGGING_SAVED_OUTPUT to
>> trigger saving information for reproducible builds...
>>
> How about "save_workdir_on_failure.bbclass"?

I'd like to keep the name short if possible, it'll end up in the 
variable names as well. Thinking ahead we might want to save other 
information in future beyond the workdir, so we could possibly drop that 
from the name. What about "failure_save" or "failure_saver"? Also, with 
underscores or without?

Paul