[PATCH] image: Attempt to work around do_image_complete race

[PATCH] image: Attempt to work around do_image_complete race

[Mike Crowe]

If two do_image_complete tasks run in parallel they risk both trying to put
their image into ${DEPLOY_DIR_IMAGE} at the same time. Both will contain a
README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt file. In theory this
should be safe because "cp -alf" will just cause one to overwrite the
other. Unfortunately, coreutils cp also has a race which means that if one
copy creates the file at just the wrong point the other will fail with:

 cp: cannot create hard link ‘..../tmp-glibc/deploy/images/pantera/README_-_DO_NOT_DELETE_FILES_IN_THIS_D.txt’ to ‘..../tmp-glibc/work/rage_against-oe-linux-gnueabi/my-own-image/1.0-r0/deploy-my-own-image-complete/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt’: File exists

The coreutils bug has been reported and fixed upstream as
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25680 , but it will take
a while for host distributions to catch up.

In the meantime, let's try and avoid the race altogether by causing
sstate.bbclass:sstate_install to take a lock file in
${DEPLOY_DIR_IMAGE} whilst writing. This lock is probably a bit broad,
but it's better than the build failing.

Signed-off-by: Mike Crowe <mac@mcrowe.com>
---
 meta/classes/image.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index b5a4fb4a33..5c2f183446 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -298,6 +298,10 @@ SSTATE_SKIP_CREATION_task-image-complete = '1'
 do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
 do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_image_complete[stamp-extra-info] = "${MACHINE}"
+# Defend against copying two different images to ${DEPLOY_DIR_IMAGE}
+# at the same time because they risk both trying to create
+# README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt at the same time.
+do_image_complete[sstate-lockfile-shared] = "${DEPLOY_DIR_IMAGE}/image-complete-output.lock"
 addtask do_image_complete after do_image before do_build
 
 # Add image-level QA/sanity checks to IMAGE_QA_COMMANDS
-- 
2.12.0.rc0

Re: [PATCH] image: Attempt to work around do_image_complete race

[Khem Raj]

On Mon, Feb 13, 2017 at 4:41 AM, Mike Crowe <mac@mcrowe.com> wrote:
> If two do_image_complete tasks run in parallel they risk both trying to put
> their image into ${DEPLOY_DIR_IMAGE} at the same time. Both will contain a
> README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt file. In theory this
> should be safe because "cp -alf" will just cause one to overwrite the
> other. Unfortunately, coreutils cp also has a race which means that if one
> copy creates the file at just the wrong point the other will fail with:
>
>  cp: cannot create hard link ‘..../tmp-glibc/deploy/images/pantera/README_-_DO_NOT_DELETE_FILES_IN_THIS_D.txt’ to ‘..../tmp-glibc/work/rage_against-oe-linux-gnueabi/my-own-image/1.0-r0/deploy-my-own-image-complete/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt’: File exists
>
> The coreutils bug has been reported and fixed upstream as
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25680 , but it will take
> a while for host distributions to catch up.
>
> In the meantime, let's try and avoid the race altogether by causing
> sstate.bbclass:sstate_install to take a lock file in
> ${DEPLOY_DIR_IMAGE} whilst writing. This lock is probably a bit broad,
> but it's better than the build failing.
>

will this lock serialize writing other files too ? then this will slow
down parallel builds. Perhaps, writing common files can be spun into a
task of its own which is then common 1 task across all images building
in parallel.

> Signed-off-by: Mike Crowe <mac@mcrowe.com>
> ---
>  meta/classes/image.bbclass | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index b5a4fb4a33..5c2f183446 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -298,6 +298,10 @@ SSTATE_SKIP_CREATION_task-image-complete = '1'
>  do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
>  do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
>  do_image_complete[stamp-extra-info] = "${MACHINE}"
> +# Defend against copying two different images to ${DEPLOY_DIR_IMAGE}
> +# at the same time because they risk both trying to create
> +# README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt at the same time.
> +do_image_complete[sstate-lockfile-shared] = "${DEPLOY_DIR_IMAGE}/image-complete-output.lock"
>  addtask do_image_complete after do_image before do_build
>
>  # Add image-level QA/sanity checks to IMAGE_QA_COMMANDS
> --
> 2.12.0.rc0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] image: Attempt to work around do_image_complete race

[Mike Crowe]

On Monday 13 February 2017 at 09:38:04 -0800, Khem Raj wrote:
> On Mon, Feb 13, 2017 at 4:41 AM, Mike Crowe <mac@mcrowe.com> wrote:
> > If two do_image_complete tasks run in parallel they risk both trying to put
> > their image into ${DEPLOY_DIR_IMAGE} at the same time. Both will contain a
> > README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt file. In theory this
> > should be safe because "cp -alf" will just cause one to overwrite the
> > other. Unfortunately, coreutils cp also has a race which means that if one
> > copy creates the file at just the wrong point the other will fail with:
> >
> >  cp: cannot create hard link ‘..../tmp-glibc/deploy/images/pantera/README_-_DO_NOT_DELETE_FILES_IN_THIS_D.txt’ to ‘..../tmp-glibc/work/rage_against-oe-linux-gnueabi/my-own-image/1.0-r0/deploy-my-own-image-complete/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt’: File exists
> >
> > The coreutils bug has been reported and fixed upstream as
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25680 , but it will take
> > a while for host distributions to catch up.
> >
> > In the meantime, let's try and avoid the race altogether by causing
> > sstate.bbclass:sstate_install to take a lock file in
> > ${DEPLOY_DIR_IMAGE} whilst writing. This lock is probably a bit broad,
> > but it's better than the build failing.
> >
> 
> will this lock serialize writing other files too ? then this will slow
> down parallel builds. Perhaps, writing common files can be spun into a
> task of its own which is then common 1 task across all images building
> in parallel.

That sounds like a great idea.

Or, we could just lose the common README file altogether. :)

Mike.

Re: [PATCH] image: Attempt to work around do_image_complete race

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 240 bytes --]

On 13 February 2017 at 18:21, Mike Crowe <mac@mcrowe.com> wrote:

> Or, we could just lose the common README file altogether. :)
>

I'd look very favourable on a patch to remove the README entirely, and I
think RP would too.

Ross

[-- Attachment #2: Type: text/html, Size: 631 bytes --]