* [Patch] locking in do_package_tar
@ 2010-04-23 10:04 Jens Erdmann
2010-04-23 15:55 ` Denys Dmytriyenko
0 siblings, 1 reply; 4+ messages in thread
From: Jens Erdmann @ 2010-04-23 10:04 UTC (permalink / raw)
To: openembedded-devel
[-- Attachment #1: Type: text/plain, Size: 608 bytes --]
Hey folks,
I would like to introduce locking in do_package_tar. I had a case here
were do_package_tar and do_package_ipk where running parallel. Because
of this "tar: .: file changed as we read it" occurs. I fixed it by
introduce locking in do_package_tar. What do you think about it?
--
------- ROAD ...the handyPC Company - - - ) ) )
Jens Erdmann
Software Development
ROAD GmbH
Bennigsenstr. 14 | 12159 Berlin | Germany
fon: +49 (30) 230069 - 64 | fax: +49 (30) 230069 - 69
url: www.road.de
Amtsgericht Charlottenburg: HRB 96688 B
Managing director: Hans-Peter Constien
[-- Attachment #2: oe_tar_lock.patch --]
[-- Type: text/x-diff, Size: 1255 bytes --]
diff --git a/classes/package_tar.bbclass b/classes/package_tar.bbclass
index b905e17..47bf2cc 100644
--- a/classes/package_tar.bbclass
+++ b/classes/package_tar.bbclass
@@ -58,6 +58,7 @@ python do_package_tar () {
localdata = bb.data.createCopy(d)
pkgdest = bb.data.getVar('PKGDEST', d, 1)
root = "%s/%s" % (pkgdest, pkg)
+ lf = bb.utils.lockfile(root + ".lock")
bb.data.setVar('ROOT', '', localdata)
bb.data.setVar('ROOT_%s' % pkg, root, localdata)
@@ -65,6 +66,7 @@ python do_package_tar () {
overrides = bb.data.getVar('OVERRIDES', localdata)
if not overrides:
+ bb.utils.unlockfile(lf)
raise bb.build.FuncFailed('OVERRIDES not defined')
overrides = bb.data.expand(overrides, localdata)
bb.data.setVar('OVERRIDES', '%s:%s' % (overrides, pkg), localdata)
@@ -82,10 +84,12 @@ python do_package_tar () {
from glob import glob
if not glob('*') + glob('.[!.]*'):
bb.note("Not creating empty archive for %s-%s" % (pkg, bb.data.expand('${PV}-${PR}${DISTRO_PR}', d, True)))
+ bb.utils.unlockfile(lf)
continue
ret = os.system("tar -czf %s %s" % (tarfn, '.'))
if ret != 0:
bb.error("Creation of tar %s failed." % tarfn)
+ bb.utils.unlockfile(lf)
}
python () {
^ permalink raw reply related [flat|nested] 4+ messages in thread* Re: [Patch] locking in do_package_tar
2010-04-23 10:04 [Patch] locking in do_package_tar Jens Erdmann
@ 2010-04-23 15:55 ` Denys Dmytriyenko
2010-04-24 16:56 ` Denys Dmytriyenko
0 siblings, 1 reply; 4+ messages in thread
From: Denys Dmytriyenko @ 2010-04-23 15:55 UTC (permalink / raw)
To: openembedded-devel
On Fri, Apr 23, 2010 at 12:04:44PM +0200, Jens Erdmann wrote:
> Hey folks,
>
> I would like to introduce locking in do_package_tar. I had a case here were
> do_package_tar and do_package_ipk where running parallel. Because of this
> "tar: .: file changed as we read it" occurs. I fixed it by introduce
> locking in do_package_tar. What do you think about it?
Ah, I've been suffering from this problem for quite some time now, when making
parallel builds with BB_NUMBER_THREADS. I was trying to debug it and noticed
it would fail when kernel recipe would try to do_deploy at the same time
another package would try to do_package_stage, so I was digging in the
packaged staging direction. Tom Rini even made a patch to add an extra lock in
there, which didn't help with my issue...
So, let me try your patch and see if this problem is gone for good! Thanks.
--
Denys
> --
> ------- ROAD ...the handyPC Company - - - ) ) )
>
> Jens Erdmann
> Software Development
>
> ROAD GmbH
> Bennigsenstr. 14 | 12159 Berlin | Germany
> fon: +49 (30) 230069 - 64 | fax: +49 (30) 230069 - 69
> url: www.road.de
>
> Amtsgericht Charlottenburg: HRB 96688 B
> Managing director: Hans-Peter Constien
>
> diff --git a/classes/package_tar.bbclass b/classes/package_tar.bbclass
> index b905e17..47bf2cc 100644
> --- a/classes/package_tar.bbclass
> +++ b/classes/package_tar.bbclass
> @@ -58,6 +58,7 @@ python do_package_tar () {
> localdata = bb.data.createCopy(d)
> pkgdest = bb.data.getVar('PKGDEST', d, 1)
> root = "%s/%s" % (pkgdest, pkg)
> + lf = bb.utils.lockfile(root + ".lock")
>
> bb.data.setVar('ROOT', '', localdata)
> bb.data.setVar('ROOT_%s' % pkg, root, localdata)
> @@ -65,6 +66,7 @@ python do_package_tar () {
>
> overrides = bb.data.getVar('OVERRIDES', localdata)
> if not overrides:
> + bb.utils.unlockfile(lf)
> raise bb.build.FuncFailed('OVERRIDES not defined')
> overrides = bb.data.expand(overrides, localdata)
> bb.data.setVar('OVERRIDES', '%s:%s' % (overrides, pkg), localdata)
> @@ -82,10 +84,12 @@ python do_package_tar () {
> from glob import glob
> if not glob('*') + glob('.[!.]*'):
> bb.note("Not creating empty archive for %s-%s" % (pkg, bb.data.expand('${PV}-${PR}${DISTRO_PR}', d, True)))
> + bb.utils.unlockfile(lf)
> continue
> ret = os.system("tar -czf %s %s" % (tarfn, '.'))
> if ret != 0:
> bb.error("Creation of tar %s failed." % tarfn)
> + bb.utils.unlockfile(lf)
> }
>
> python () {
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [Patch] locking in do_package_tar
2010-04-23 15:55 ` Denys Dmytriyenko
@ 2010-04-24 16:56 ` Denys Dmytriyenko
2010-04-26 8:57 ` Jens Erdmann
0 siblings, 1 reply; 4+ messages in thread
From: Denys Dmytriyenko @ 2010-04-24 16:56 UTC (permalink / raw)
To: openembedded-devel
On Fri, Apr 23, 2010 at 11:55:15AM -0400, Denys Dmytriyenko wrote:
> On Fri, Apr 23, 2010 at 12:04:44PM +0200, Jens Erdmann wrote:
> > Hey folks,
> >
> > I would like to introduce locking in do_package_tar. I had a case here were
> > do_package_tar and do_package_ipk where running parallel. Because of this
> > "tar: .: file changed as we read it" occurs. I fixed it by introduce
> > locking in do_package_tar. What do you think about it?
>
> Ah, I've been suffering from this problem for quite some time now, when making
> parallel builds with BB_NUMBER_THREADS. I was trying to debug it and noticed
> it would fail when kernel recipe would try to do_deploy at the same time
> another package would try to do_package_stage, so I was digging in the
> packaged staging direction. Tom Rini even made a patch to add an extra lock in
> there, which didn't help with my issue...
> So, let me try your patch and see if this problem is gone for good! Thanks.
It may solve some other similar issue, but unfortunately, it didn't seem to
work for me, as it failed again the same way with the patch applied:
NOTE: Running task 2011 of 2173 (ID: 953, /OE/arago-oe-dev/recipes/freetype/freetype_2.3.9.bb, do_package_stage)
/OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/
/OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/
/OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/
/OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/
ERROR: function staging_packager failed
ERROR: log data follows (/OE/arago-tmp/work/dm3730-am3715-evm-none-linux-gnueabi/linux-omap-psp-2.6.32-r60+gitr7b8926aa626991fa087b00f6bbc1fb6b0e8269b0/temp/log.staging_packager.19650)
| tar: .: file changed as we read it
NOTE: Task failed: /OE/arago-tmp/work/dm3730-am3715-evm-none-linux-gnueabi/linux-omap-psp-2.6.32-r60+gitr7b8926aa626991fa087b00f6bbc1fb6b0e8269b0/temp/log.staging_packager.19650
NOTE: oestats: task failed, see http://tinderbox.openembedded.net/packages/556734/
ERROR: TaskFailed event exception, aborting
ERROR: Build of /OE/arago-oe-dev/recipes/linux/linux-omap-psp_2.6.32.bb do_package_stage failed
ERROR: Task 1009 (/OE/arago-oe-dev/recipes/linux/linux-omap-psp_2.6.32.bb, do_package_stage) failed
NOTE: Waiting for 3 active tasks to finish
NOTE: 1: /OE/arago-oe-dev/recipes/freetype/freetype_2.3.9.bb, do_package_stage (21747)
So, it looks like my problem is actually still packaged staging related, as is
fails when 2 recipes run do_package_stage task at the same time (in the
example above, it's freetype and linux-omap-psp)...
--
Denys
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Patch] locking in do_package_tar
2010-04-24 16:56 ` Denys Dmytriyenko
@ 2010-04-26 8:57 ` Jens Erdmann
0 siblings, 0 replies; 4+ messages in thread
From: Jens Erdmann @ 2010-04-26 8:57 UTC (permalink / raw)
To: openembedded-devel
On 04/24/2010 06:56 PM, Denys Dmytriyenko wrote:
> On Fri, Apr 23, 2010 at 11:55:15AM -0400, Denys Dmytriyenko wrote:
>> On Fri, Apr 23, 2010 at 12:04:44PM +0200, Jens Erdmann wrote:
>>> Hey folks,
>>>
>>> I would like to introduce locking in do_package_tar. I had a case here were
>>> do_package_tar and do_package_ipk where running parallel. Because of this
>>> "tar: .: file changed as we read it" occurs. I fixed it by introduce
>>> locking in do_package_tar. What do you think about it?
>>
>> Ah, I've been suffering from this problem for quite some time now, when making
>> parallel builds with BB_NUMBER_THREADS. I was trying to debug it and noticed
>> it would fail when kernel recipe would try to do_deploy at the same time
>> another package would try to do_package_stage, so I was digging in the
>> packaged staging direction. Tom Rini even made a patch to add an extra lock in
>> there, which didn't help with my issue...
>> So, let me try your patch and see if this problem is gone for good! Thanks.
>
> It may solve some other similar issue, but unfortunately, it didn't seem to
> work for me, as it failed again the same way with the patch applied:
>
> NOTE: Running task 2011 of 2173 (ID: 953, /OE/arago-oe-dev/recipes/freetype/freetype_2.3.9.bb, do_package_stage)
> /OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/
> /OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/
> /OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/
> /OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/
> ERROR: function staging_packager failed
> ERROR: log data follows (/OE/arago-tmp/work/dm3730-am3715-evm-none-linux-gnueabi/linux-omap-psp-2.6.32-r60+gitr7b8926aa626991fa087b00f6bbc1fb6b0e8269b0/temp/log.staging_packager.19650)
> | tar: .: file changed as we read it
> NOTE: Task failed: /OE/arago-tmp/work/dm3730-am3715-evm-none-linux-gnueabi/linux-omap-psp-2.6.32-r60+gitr7b8926aa626991fa087b00f6bbc1fb6b0e8269b0/temp/log.staging_packager.19650
> NOTE: oestats: task failed, see http://tinderbox.openembedded.net/packages/556734/
> ERROR: TaskFailed event exception, aborting
> ERROR: Build of /OE/arago-oe-dev/recipes/linux/linux-omap-psp_2.6.32.bb do_package_stage failed
> ERROR: Task 1009 (/OE/arago-oe-dev/recipes/linux/linux-omap-psp_2.6.32.bb, do_package_stage) failed
> NOTE: Waiting for 3 active tasks to finish
> NOTE: 1: /OE/arago-oe-dev/recipes/freetype/freetype_2.3.9.bb, do_package_stage (21747)
>
> So, it looks like my problem is actually still packaged staging related, as is
> fails when 2 recipes run do_package_stage task at the same time (in the
> example above, it's freetype and linux-omap-psp)...
>
This looks more packaged-staging.bbclass related. Maybe you could try to
make the lock in line 444 two lines earlier so that staging_packager and
staging_helper will be locked. too.
To my patch: Is locking in tar a good idea?
--
------- ROAD ...the handyPC Company - - - ) ) )
Jens Erdmann
Software Development
ROAD GmbH
Bennigsenstr. 14 | 12159 Berlin | Germany
fon: +49 (30) 230069 - 64 | fax: +49 (30) 230069 - 69
url: www.road.de
Amtsgericht Charlottenburg: HRB 96688 B
Managing director: Hans-Peter Constien
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2010-04-26 9:00 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-04-23 10:04 [Patch] locking in do_package_tar Jens Erdmann
2010-04-23 15:55 ` Denys Dmytriyenko
2010-04-24 16:56 ` Denys Dmytriyenko
2010-04-26 8:57 ` Jens Erdmann
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.