Oddness regarding file locks in package.bbclass

Oddness regarding file locks in package.bbclass

[Peter Kjellerstedt]

I just stumbled upon something odd in package.bbclass. In commit 
ede381d5 from January 2011 (the code hasn't changed since), the 
use of the ${PACKAGELOCK} lock file was changed to shared to 
improve parallelism. However, when looking at the actual change 
it becomes confusing. I have included it below for reference.

> commit ede381d56b180b384fdad98d445e5430819cfade
> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date:   Wed Jan 19 11:04:15 2011 +0000
> 
>     package.bbclass: Take a shared lock when reading to improve do_package parallelism
>     
>     Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index d39c694de5..8e7fa26f72 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -497,7 +497,8 @@ python emit_pkgdata() {
>  	pkgdest = bb.data.getVar('PKGDEST', d, 1)
>  	pkgdatadir = bb.data.getVar('PKGDESTWORK', d, True)
>  
> -	lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}", d))
> +	# Take shared lock since we're only reading, not writing
> +	lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}", d), True)

Here the lock is changed to shared as per the commit message.

>  
>  	data_file = pkgdatadir + bb.data.expand("/${PN}" , d)
>  	f = open(data_file, 'w')
> @@ -649,6 +650,7 @@ python package_do_shlibs() {
>  	shlibs_dir = bb.data.getVar('SHLIBSDIR', d, True)
>  	shlibswork_dir = bb.data.getVar('SHLIBSWORKDIR', d, True)
>  
> +	# Take shared lock since we're only reading, not writing
>  	lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}", d))

Here, however, it is not changed, even though a comment is added to 
say that it is. Was this intentional, or just an oversight?

>  
>  	def linux_so(root, path, file):
> @@ -878,6 +880,7 @@ python package_do_pkgconfig () {
>  							if hdr == 'Requires':
>  								pkgconfig_needed[pkg] += exp.replace(',', ' ').split()
>  
> +	# Take shared lock since we're only reading, not writing
>  	lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}", d))

Here again a comment is added, but the code is not changed to match.

>  
>  	for pkg in packages.split():

Also, what is the ${PACKAGELOCK} lock file actually protecting? With 
the exception of the two questionable cases above, I cannot see that 
the lock is taken privately anywhere else. And since it looks as the 
code in package_do_shlibs() and package_do_pkgconfig() is not what 
needs protection (based on the added comments above), what is?

//Peter

Re: Oddness regarding file locks in package.bbclass

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

On 3 October 2017 at 19:17, Peter Kjellerstedt <peter.kjellerstedt@axis.com>
wrote:

> > +     # Take shared lock since we're only reading, not writing
> >       lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}", d))
>
> Here, however, it is not changed, even though a comment is added to
> say that it is. Was this intentional, or just an oversight?
>
> > +     # Take shared lock since we're only reading, not writing
> >       lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}", d))
>
> Here again a comment is added, but the code is not changed to match.


I'm no expert on this piece of code but that definitely looks like an
oversight.


> Also, what is the ${PACKAGELOCK} lock file actually protecting? With
>
the exception of the two questionable cases above, I cannot see that
> the lock is taken privately anywhere else. And since it looks as the
> code in package_do_shlibs() and package_do_pkgconfig() is not what
> needs protection (based on the added comments above), what is?
>

Good point.  The sstate locks are also shared.

Richard was involved in all of these changes so lets wait for him to return
from his holiday...

Ross

[-- Attachment #2: Type: text/html, Size: 2086 bytes --]

Re: Oddness regarding file locks in package.bbclass

[Trevor Woerner]

[-- Attachment #1: Type: text/plain, Size: 1573 bytes --]

This item doesn't appear to have a follow-up.

On Wed, Oct 4, 2017 at 11:10 AM, Burton, Ross <ross.burton@intel.com> wrote:

> On 3 October 2017 at 19:17, Peter Kjellerstedt <
> peter.kjellerstedt@axis.com> wrote:
>
>> > +     # Take shared lock since we're only reading, not writing
>> >       lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}", d))
>>
>> Here, however, it is not changed, even though a comment is added to
>> say that it is. Was this intentional, or just an oversight?
>>
>> > +     # Take shared lock since we're only reading, not writing
>> >       lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}", d))
>>
>> Here again a comment is added, but the code is not changed to match.
>
>
> I'm no expert on this piece of code but that definitely looks like an
> oversight.
>
>
>> Also, what is the ${PACKAGELOCK} lock file actually protecting? With
>>
> the exception of the two questionable cases above, I cannot see that
>> the lock is taken privately anywhere else. And since it looks as the
>> code in package_do_shlibs() and package_do_pkgconfig() is not what
>> needs protection (based on the added comments above), what is?
>>
>
> Good point.  The sstate locks are also shared.
>
> Richard was involved in all of these changes so lets wait for him to
> return from his holiday...
>
> Ross
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>

[-- Attachment #2: Type: text/html, Size: 3075 bytes --]

Re: Oddness regarding file locks in package.bbclass

[Richard Purdie]

On Tue, 2017-10-03 at 18:17 +0000, Peter Kjellerstedt wrote:
> I just stumbled upon something odd in package.bbclass. In commit 
> ede381d5 from January 2011 (the code hasn't changed since), the 
> use of the ${PACKAGELOCK} lock file was changed to shared to 
> improve parallelism. However, when looking at the actual change 
> it becomes confusing. I have included it below for reference.
> 
> > 
> > commit ede381d56b180b384fdad98d445e5430819cfade
> > Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> > Date:   Wed Jan 19 11:04:15 2011 +0000
> > 
> >     package.bbclass: Take a shared lock when reading to improve
> > do_package parallelism
> >     
> >     Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.o
> > rg>
> > 
> > diff --git a/meta/classes/package.bbclass
> > b/meta/classes/package.bbclass
> > index d39c694de5..8e7fa26f72 100644
> > --- a/meta/classes/package.bbclass
> > +++ b/meta/classes/package.bbclass
> > @@ -497,7 +497,8 @@ python emit_pkgdata() {
> >  	pkgdest = bb.data.getVar('PKGDEST', d, 1)
> >  	pkgdatadir = bb.data.getVar('PKGDESTWORK', d, True)
> >  
> > -	lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}",
> > d))
> > +	# Take shared lock since we're only reading, not writing
> > +	lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}",
> > d), True)
> Here the lock is changed to shared as per the commit message.
> 
> > 
> >  
> >  	data_file = pkgdatadir + bb.data.expand("/${PN}" , d)
> >  	f = open(data_file, 'w')
> > @@ -649,6 +650,7 @@ python package_do_shlibs() {
> >  	shlibs_dir = bb.data.getVar('SHLIBSDIR', d, True)
> >  	shlibswork_dir = bb.data.getVar('SHLIBSWORKDIR', d, True)
> >  
> > +	# Take shared lock since we're only reading, not writing
> >  	lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}",
> > d))
> Here, however, it is not changed, even though a comment is added to 
> say that it is. Was this intentional, or just an oversight?
> 
> > 
> >  
> >  	def linux_so(root, path, file):
> > @@ -878,6 +880,7 @@ python package_do_pkgconfig () {
> >  							if hdr ==
> > 'Requires':
> >  								pk
> > gconfig_needed[pkg] += exp.replace(',', ' ').split()
> >  
> > +	# Take shared lock since we're only reading, not writing
> >  	lf = bb.utils.lockfile(bb.data.expand("${PACKAGELOCK}",
> > d))
> Here again a comment is added, but the code is not changed to match.
> 
> > 
> >  
> >  	for pkg in packages.split():
> Also, what is the ${PACKAGELOCK} lock file actually protecting? With 
> the exception of the two questionable cases above, I cannot see that 
> the lock is taken privately anywhere else. And since it looks as the 
> code in package_do_shlibs() and package_do_pkgconfig() is not what 
> needs protection (based on the added comments above), what is?

Sorry for not replying sooner, this was brought to my attention again.

PACKAGELOCK is there for PKGDATA_DIR which is defined as
${TMPDIR}/pkgdata/${MACHINE}, i.e. not recipe specific.

If something is reading files from pkgdata and something else
writes/changes them, we used to see build failures. The lock therefore
does have a purpose in guarding against this.

Looking at the code, something has gotten lost with the addition of
recipe specific sysroots and the separation of do_packagedata from
do_package.

I suspect it needs:

-do_packagedata[sstate-lockfile-shared] = "${PACKAGELOCK}"
+do_packagedata[sstate-lockfile] = "${PACKAGELOCK}"

and the other sites you mention should be shared locks, then this would
all make a lot more sense.

Cheers,

Richard