summary about recent do_populate_sdk failures

summary about recent do_populate_sdk failures

[Tian, Kevin]

Lianhao and I are looking into recent master instability, more specifically starting
from reported do_populate_sdk failures. There have been 3 problems revealed:

1) unexpected do_package/do_package_write* rebuild
2) change PACKAGE_CLASSES causes more do_package rebuilds
3) do_populate_sdk finally exits due to eglibc not matching expected version

We've root caused 1), and the patch has been sent out:
http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/master1&id=68ad8ead1a83161afb8c2a65a28dfc205181d80e

We're not sure whether 3) is caused by too many unexpected do_package rebuilds,
and is now testing whether 3) will disappear with the fix for 1). We'll also look into
original failure to see whether real cause of 3) may be hidden with the fix.

Now the remaining open is about 2). We got the reason, but would like to welcome
Comments on an elegant solution.

The phenomenon is that do_package checksum changes with switching the order
In PACKAGE_CLASSES, e.g:

From
PACKAGE_CLASSES = "package_rpm package_ipk"
To
PACKAGE_CLASSES = "package_ipk package_rpm"

This is undesired since the order only matters for final rootfs generation. The actual
cause is related to how do_package is generated, which depends on when 
package.bbclass is first brought in.

Take package_rpm.bbclass for example, which inherit package.bbclass. Bitbake
generates below wrappers implicitly:

Do_package () {
	bb.build.exec_func('package_rpm_do_package', d)
}

Package_rpm_do_package() {
	bb.build.exec_func('package_do_package', d)
}

Above is implicitly created when bitbake parses class files based on inheritance tree,
and once a class file is parsed it's then saved to the cache. When later package_ipk is
parsed, bitbake will simply read from the cache for package.bbclass and then
do_package remains tying to package_rpm_do_package.

If package_ipk is the 1st in PACKAGE_CLASSES, then do_package becomes:

Do_package () {
	bb.build.exec_func('package_ipk_do_package', d)
}

This sure generates different checksum to break sstate installation, but in reality it's
just name difference, no actual functional change. We considered to use vardeps and 
varexcldeps which don't help though since it's the do_package itself changed.

One possible option is to sort the INHERIT list, which ensures we always parse
Inherited class files in same order if no content change. We'll test whether this works,
and also hear your comments.

Thanks
Kevin

Re: summary about recent do_populate_sdk failures

[Tian, Kevin]

> From: Tian, Kevin
> Sent: Monday, January 31, 2011 3:18 PM
> 
> One possible option is to sort the INHERIT list, which ensures we always parse
> Inherited class files in same order if no content change. We'll test whether this
> works,
> and also hear your comments.

This doesn't help, since the order really matters to decide which type rootfs will
Use... :-/

Thanks
Kevin

Re: summary about recent do_populate_sdk failures

[Tian, Kevin]

> From: Tian, Kevin
> Sent: Monday, January 31, 2011 3:21 PM
> 
> > From: Tian, Kevin
> > Sent: Monday, January 31, 2011 3:18 PM
> >
> > One possible option is to sort the INHERIT list, which ensures we always
> parse
> > Inherited class files in same order if no content change. We'll test whether
> this
> > works,
> > and also hear your comments.
> 
> This doesn't help, since the order really matters to decide which type rootfs will
> Use... :-/
> 

We come up another try at:

http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/tmp

regarding to the fact that one bbclass is only parsed once even when it's inherited
by multiple files. If original reason to include the 2nd class name is to avoid possible
name confliction, this then won't happen.

Not sure whether this will break things. The test is ongoing and at least the parse
phase doesn't report errors.

Thanks
Kevin

Re: summary about recent do_populate_sdk failures

[Richard Purdie]

On Mon, 2011-01-31 at 15:18 +0800, Tian, Kevin wrote:
> Lianhao and I are looking into recent master instability, more specifically starting
> from reported do_populate_sdk failures. There have been 3 problems revealed:
> 
> 1) unexpected do_package/do_package_write* rebuild
> 2) change PACKAGE_CLASSES causes more do_package rebuilds
> 3) do_populate_sdk finally exits due to eglibc not matching expected version
> 
> We've root caused 1), and the patch has been sent out:
> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/master1&id=68ad8ead1a83161afb8c2a65a28dfc205181d80e
> 
> We're not sure whether 3) is caused by too many unexpected do_package rebuilds,
> and is now testing whether 3) will disappear with the fix for 1). We'll also look into
> original failure to see whether real cause of 3) may be hidden with the fix.
> 
> Now the remaining open is about 2). We got the reason, but would like to welcome
> Comments on an elegant solution.
> 
> The phenomenon is that do_package checksum changes with switching the order
> In PACKAGE_CLASSES, e.g:
> 
> From
> PACKAGE_CLASSES = "package_rpm package_ipk"
> To
> PACKAGE_CLASSES = "package_ipk package_rpm"
> 
> This is undesired since the order only matters for final rootfs generation. The actual
> cause is related to how do_package is generated, which depends on when 
> package.bbclass is first brought in.
> 
> Take package_rpm.bbclass for example, which inherit package.bbclass. Bitbake
> generates below wrappers implicitly:
> 
> Do_package () {
> 	bb.build.exec_func('package_rpm_do_package', d)
> }
> 
> Package_rpm_do_package() {
> 	bb.build.exec_func('package_do_package', d)
> }
> 
> Above is implicitly created when bitbake parses class files based on inheritance tree,
> and once a class file is parsed it's then saved to the cache. When later package_ipk is
> parsed, bitbake will simply read from the cache for package.bbclass and then
> do_package remains tying to package_rpm_do_package.
> 
> If package_ipk is the 1st in PACKAGE_CLASSES, then do_package becomes:
> 
> Do_package () {
> 	bb.build.exec_func('package_ipk_do_package', d)
> }
> 
> This sure generates different checksum to break sstate installation, but in reality it's
> just name difference, no actual functional change. We considered to use vardeps and 
> varexcldeps which don't help though since it's the do_package itself changed.
> 
> One possible option is to sort the INHERIT list, which ensures we always parse
> Inherited class files in same order if no content change. We'll test whether this works,
> and also hear your comments.

For 2), I suggest we drop the EXPORT_FUNCTIONS for do_package since we
never actually use that functionality, there are other ways to customise
packaging. The patch would be:

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 7e1f3f0..856858c 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1065,7 +1065,7 @@ PACKAGEFUNCS ?= "perform_packagecopy \
                package_depchains \
                emit_pkgdata"
 
-python package_do_package () {
+python do_package () {
        packages = (bb.data.getVar('PACKAGES', d, True) or "").split()
        if len(packages) < 1:
                bb.debug(1, "No packages to build, skipping do_package")
@@ -1110,8 +1110,6 @@ do_package_write[noexec] = "1"
 do_build[recrdeptask] += "do_package_write"
 addtask package_write before do_build after do_package
 
-EXPORT_FUNCTIONS do_package do_package_write
-
 #
 # Helper functions for the package writing classes
 #

Re: summary about recent do_populate_sdk failures

[Richard Purdie]

On Mon, 2011-01-31 at 15:18 +0800, Tian, Kevin wrote:
> Lianhao and I are looking into recent master instability, more specifically starting
> from reported do_populate_sdk failures. There have been 3 problems revealed:
> 
> 1) unexpected do_package/do_package_write* rebuild
> 2) change PACKAGE_CLASSES causes more do_package rebuilds
> 3) do_populate_sdk finally exits due to eglibc not matching expected version
> 
> We've root caused 1), and the patch has been sent out:
> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/master1&id=68ad8ead1a83161afb8c2a65a28dfc205181d80e
> 
> We're not sure whether 3) is caused by too many unexpected do_package rebuilds,
> and is now testing whether 3) will disappear with the fix for 1). We'll also look into
> original failure to see whether real cause of 3) may be hidden with the fix.

I think there is a race and we need to adjust the packaging process for
libgcc/libc as I mentioned in the other email.

To reproduce I'm guessing something like:

"bitbake libgcc virtual/libc -c package -f" will consistently break
rootfs builds after that command is run.

(This relies on the fact that libc will take much longer than libgcc to
package.)

Cheers,

Richard

Re: summary about recent do_populate_sdk failures

[Lu, Lianhao]

Richard Purdie wrote on 2011-01-31:
> On Mon, 2011-01-31 at 15:18 +0800, Tian, Kevin wrote:
>> Lianhao and I are looking into recent master instability, more
>> specifically starting from reported do_populate_sdk failures. There
>> have been 3 problems revealed:
>> 
>> 1) unexpected do_package/do_package_write* rebuild
>> 2) change PACKAGE_CLASSES causes more do_package rebuilds
>> 3) do_populate_sdk finally exits due to eglibc not matching expected
>> version
>> 
>> We've root caused 1), and the patch has been sent out:
>> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/mas
>> te r1&id=68ad8ead1a83161afb8c2a65a28dfc205181d80e
>> 
>> We're not sure whether 3) is caused by too many unexpected
>> do_package rebuilds, and is now testing whether 3) will disappear
>> with the fix for 1). We'll also look into original failure to see
>> whether real cause of 3) may
> be hidden with the fix.
> 
> I think there is a race and we need to adjust the packaging process
> for libgcc/libc as I mentioned in the other email.
> 
> To reproduce I'm guessing something like:
> 
> "bitbake libgcc virtual/libc -c package -f" will consistently break
> rootfs builds after that command is run.
> 
> (This relies on the fact that libc will take much longer than libgcc to
> package.)
> 

Yes, we think the cause 1) & 2) mentioned above somehow exposed the race problem to us. We guess it might be that the libgcc's do_package_write_xxx task started before the eglibc's do_package task finished. Richard, I'll try to reproduce this using your method tonight, but I have to build from scratch, because my building environment is totally messed up(I even saw a "do_install" error for libgcc when trying the 'bitbake libgcc virtual/libc -c package -f').

Best Regards,
Lianhao