* [PATCH 0/1] Fix random kernel unpack failure
@ 2011-01-04 11:36 Yu Ke
2011-01-04 11:36 ` [PATCH 1/1] base.bbclass: add lock file for do_unpack task Yu Ke
2011-01-04 19:51 ` [PATCH 0/1] Fix random kernel unpack failure Saul Wold
0 siblings, 2 replies; 6+ messages in thread
From: Yu Ke @ 2011-01-04 11:36 UTC (permalink / raw)
To: poky
autobuilder recently is blocked by random unpack failure of linux-libc-headers-yocto
and linux-yocto. this patch fix this issue. the patch description has the detailed
root cause and fix.
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/kernel-unpack-fix
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/kernel-unpack-fix
Thanks,
Yu Ke <ke.yu@intel.com>
---
Yu Ke (1):
base.bbclass: add lock file for do_unpack task
meta/classes/base.bbclass | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 1/1] base.bbclass: add lock file for do_unpack task
2011-01-04 11:36 [PATCH 0/1] Fix random kernel unpack failure Yu Ke
@ 2011-01-04 11:36 ` Yu Ke
2011-01-04 15:45 ` Darren Hart
2011-01-04 19:51 ` [PATCH 0/1] Fix random kernel unpack failure Saul Wold
1 sibling, 1 reply; 6+ messages in thread
From: Yu Ke @ 2011-01-04 11:36 UTC (permalink / raw)
To: poky
This patch intend to fix the random unpack failure of linux-libc-headers-yocto
and linux-yocto.
The root cause of the unpack failure is that: these two recpies has the same URL, thus
has the same dest file during the fetch and unpack phase:
do_fetch : create tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz
do_unpack : extract tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz
fetch phase is protected by lockfile, so it works fine. but unpack phase is not lock protected,
thus there is race condition like: when linux-yocto do_unpack is extracting the tar ball,
linux-libc-headers-yocto do_fetch starts to create tar ball thus overwrite linux-yocto's
tar ball and cause linux-yocto do_unpack failure
To fix this issue, do_unpack also need to be protected by lock
Signed-off-by: Yu Ke <ke.yu@intel.com>
---
meta/classes/base.bbclass | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 54a1dcd..f8ce123 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -242,6 +242,8 @@ python base_do_unpack() {
localdata = bb.data.createCopy(d)
bb.data.update_data(localdata)
+ urldata = bb.fetch.init([], localdata, True)
+
src_uri = bb.data.getVar('SRC_URI', localdata, True)
if not src_uri:
return
@@ -253,7 +255,9 @@ python base_do_unpack() {
if local is None:
continue
local = os.path.realpath(local)
+ lf = bb.utils.lockfile(urldata[url].lockfile)
ret = oe_unpack_file(local, localdata, url)
+ bb.utils.unlockfile(lf)
if not ret:
raise bb.build.FuncFailed("oe_unpack_file failed with return value %s" % ret)
}
--
1.7.0.4
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH 1/1] base.bbclass: add lock file for do_unpack task
2011-01-04 11:36 ` [PATCH 1/1] base.bbclass: add lock file for do_unpack task Yu Ke
@ 2011-01-04 15:45 ` Darren Hart
2011-01-05 2:18 ` Cui, Dexuan
0 siblings, 1 reply; 6+ messages in thread
From: Darren Hart @ 2011-01-04 15:45 UTC (permalink / raw)
To: Yu Ke; +Cc: poky
On 01/04/2011 03:36 AM, Yu Ke wrote:
> This patch intend to fix the random unpack failure of linux-libc-headers-yocto
> and linux-yocto.
>
> The root cause of the unpack failure is that: these two recpies has the same URL, thus
> has the same dest file during the fetch and unpack phase:
> do_fetch : create tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz
> do_unpack : extract tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz
> fetch phase is protected by lockfile, so it works fine. but unpack phase is not lock protected,
> thus there is race condition like: when linux-yocto do_unpack is extracting the tar ball,
> linux-libc-headers-yocto do_fetch starts to create tar ball thus overwrite linux-yocto's
> tar ball and cause linux-yocto do_unpack failure
>
> To fix this issue, do_unpack also need to be protected by lock
Nice work! My out-for-review trace-cmd and kernelshark recipes will
suffer the same issue.
--
Darren Hart
Yocto Linux Kernel
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH 1/1] base.bbclass: add lock file for do_unpack task
2011-01-04 15:45 ` Darren Hart
@ 2011-01-05 2:18 ` Cui, Dexuan
0 siblings, 0 replies; 6+ messages in thread
From: Cui, Dexuan @ 2011-01-05 2:18 UTC (permalink / raw)
To: 'Darren Hart', Yu, Ke; +Cc: poky@yoctoproject.org
Darren Hart wrote:
> On 01/04/2011 03:36 AM, Yu Ke wrote:
>> This patch intend to fix the random unpack failure of
>> linux-libc-headers-yocto
>> and linux-yocto.
>>
>> The root cause of the unpack failure is that: these two recpies has
>> the same URL, thus
>> has the same dest file during the fetch and unpack phase:
>> do_fetch : create tar ball
>> ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz do_unpack
>> : extract tar ball
>> ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz
>> fetch phase is protected by lockfile, so it works fine. but unpack
>> phase is not lock protected, thus there is race condition like: when
>> linux-yocto do_unpack is extracting the tar ball,
>> linux-libc-headers-yocto do_fetch starts to create tar ball thus
>> overwrite linux-yocto's
>> tar ball and cause linux-yocto do_unpack failure
>>
>> To fix this issue, do_unpack also need to be protected by lock
>
> Nice work! My out-for-review trace-cmd and kernelshark recipes will
> suffer the same issue.
Thanks to Ke's fix!
Previously I met with the linux-libc-headers unpack failure everytime... Now the failure has gone with the fix.
Thanks,
-- Dexuan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 0/1] Fix random kernel unpack failure
2011-01-04 11:36 [PATCH 0/1] Fix random kernel unpack failure Yu Ke
2011-01-04 11:36 ` [PATCH 1/1] base.bbclass: add lock file for do_unpack task Yu Ke
@ 2011-01-04 19:51 ` Saul Wold
2011-01-05 2:00 ` Yu Ke
1 sibling, 1 reply; 6+ messages in thread
From: Saul Wold @ 2011-01-04 19:51 UTC (permalink / raw)
To: Yu Ke; +Cc: poky
On 01/04/2011 03:36 AM, Yu Ke wrote:
> autobuilder recently is blocked by random unpack failure of linux-libc-headers-yocto
> and linux-yocto. this patch fix this issue. the patch description has the detailed
> root cause and fix.
>
> Pull URL: git://git.pokylinux.org/poky-contrib.git
> Branch: kyu3/kernel-unpack-fix
> Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/kernel-unpack-fix
>
> Thanks,
> Yu Ke<ke.yu@intel.com>
> ---
>
>
> Yu Ke (1):
> base.bbclass: add lock file for do_unpack task
>
> meta/classes/base.bbclass | 4 ++++
> 1 files changed, 4 insertions(+), 0 deletions(-)
>
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>
Ke,
This has been pulled into master as a workaround for the existing issue,
please ensure that this issue gets addressed properly once the fetch
work is completed to a point that this can be done.
Thanks
Sau!
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 0/1] Fix random kernel unpack failure
2011-01-04 19:51 ` [PATCH 0/1] Fix random kernel unpack failure Saul Wold
@ 2011-01-05 2:00 ` Yu Ke
0 siblings, 0 replies; 6+ messages in thread
From: Yu Ke @ 2011-01-05 2:00 UTC (permalink / raw)
To: Saul Wold; +Cc: poky
On Jan 04, 11:51, Saul Wold wrote:
> On 01/04/2011 03:36 AM, Yu Ke wrote:
> >autobuilder recently is blocked by random unpack failure of linux-libc-headers-yocto
> >and linux-yocto. this patch fix this issue. the patch description has the detailed
> >root cause and fix.
> >
> >Pull URL: git://git.pokylinux.org/poky-contrib.git
> > Branch: kyu3/kernel-unpack-fix
> > Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/kernel-unpack-fix
> >
> >Thanks,
> > Yu Ke<ke.yu@intel.com>
> >---
> >
> >
> >Yu Ke (1):
> > base.bbclass: add lock file for do_unpack task
> >
> > meta/classes/base.bbclass | 4 ++++
> > 1 files changed, 4 insertions(+), 0 deletions(-)
> >
> >_______________________________________________
> >poky mailing list
> >poky@yoctoproject.org
> >https://lists.yoctoproject.org/listinfo/poky
> >
>
> Ke,
>
> This has been pulled into master as a workaround for the existing
> issue, please ensure that this issue gets addressed properly once
> the fetch work is completed to a point that this can be done.
>
> Thanks
> Sau!
Yes, with the new fetcher, do_fetch will not create the tar ball for git repo, instead, do_unpack will directly checkout the code from the git repo, thus there will be no race condition and the lock can be safely removed.
I will take care of it in the new fetcher.
Regards
Ke
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-01-05 2:18 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-01-04 11:36 [PATCH 0/1] Fix random kernel unpack failure Yu Ke
2011-01-04 11:36 ` [PATCH 1/1] base.bbclass: add lock file for do_unpack task Yu Ke
2011-01-04 15:45 ` Darren Hart
2011-01-05 2:18 ` Cui, Dexuan
2011-01-04 19:51 ` [PATCH 0/1] Fix random kernel unpack failure Saul Wold
2011-01-05 2:00 ` Yu Ke
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.