hash generation/PR service problem with xuser-account and other packages

hash generation/PR service problem with xuser-account and other packages

[Steffen Sledz]

As i mentioned in another thread before we're investigating some problems related with package versions going backwards using a PR service.

Now i have some more information. But i'm not able to understand and fix the problem for myself.

Assume we have a clean workspace and i bitbake the xuser-account package with this command:

  MACHINE="foo" bitbake xuser-account

This generates these packages (the 207 comes from our PR service).

  tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.207_all.ipk
  tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.207_all.ipk
  tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.207_all.ipk

If i call the same bitbake command again, the same packages are generated. Fine!

Now i call:

  MACHINE="bar" bitbake xuser-account

The generated packages now get a new number from the PR server (so they have a different hash i believe).

  tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.208_all.ipk
  tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.208_all.ipk
  tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.208_all.ipk

But this package does not seem to be machine dependent!?

If the next build for machine foo is a clean build again (e.g. on our Jenkins continuous integration server which makes clean builds once a week) the PR service generated number wents back to 207 which results in:

  "ERROR: QA Issue: Package version for package xuser-account went backwards which would break package feeds"

Can someone confirm this behaviour?

Is this a bug? I think so.

Can someone give some details what went wrong here? Also after studying <http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#checksums> i was not able to fully understand the background, not to speak of fixing something.

BTW: We see the same behaviour for the cpufreq-tweaks, linux-firmware, run-postinsts, and some of our own packages.

Thanx for any help,
Steffen

PS: We're working on the daisy branches using Angstrom with some more own layers.

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: hash generation/PR service problem with xuser-account and other packages

[Richard Purdie]

On Tue, 2014-09-23 at 14:07 +0200, Steffen Sledz wrote:
> As i mentioned in another thread before we're investigating some problems related with package versions going backwards using a PR service.
> 
> Now i have some more information. But i'm not able to understand and fix the problem for myself.
> 
> Assume we have a clean workspace and i bitbake the xuser-account package with this command:
> 
>   MACHINE="foo" bitbake xuser-account
> 
> This generates these packages (the 207 comes from our PR service).
> 
>   tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.207_all.ipk
>   tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.207_all.ipk
>   tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.207_all.ipk
> 
> If i call the same bitbake command again, the same packages are generated. Fine!
> 
> Now i call:
> 
>   MACHINE="bar" bitbake xuser-account
> 
> The generated packages now get a new number from the PR server (so they have a different hash i believe).
> 
>   tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.208_all.ipk
>   tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.208_all.ipk
>   tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.208_all.ipk
> 
> But this package does not seem to be machine dependent!?
> 
> If the next build for machine foo is a clean build again (e.g. on our Jenkins continuous integration server which makes clean builds once a week) the PR service generated number wents back to 207 which results in:
> 
>   "ERROR: QA Issue: Package version for package xuser-account went backwards which would break package feeds"
> 
> Can someone confirm this behaviour?
> 
> Is this a bug? I think so.
> 
> Can someone give some details what went wrong here? Also after studying <http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#checksums> i was not able to fully understand the background, not to speak of fixing something.
> 
> BTW: We see the same behaviour for the cpufreq-tweaks, linux-firmware, run-postinsts, and some of our own packages.

I just took a look at this on master. There are some changes between
master and daisy and I want to get this resolved in master before we
think about daisy. I thought walking through debugging this may be
helpful, so, I did:

MACHINE=qemux86 bitbake xuser-account
MACHINE=qemuarm bitbake xuser-account

and sure enough, you are correct, things rebuild and they should not for
an allarch package like this. Looking at the list of packages being
built, its pulling in the toolchain. I checked:

MACHINE=qemuarm bitbake xuser-account -e | grep INHIBIT_DEFAULT_DEPS

and sure enough, that is set (as per allarch.bbclass) so why is it
building the toolchain?! It has to be a dependency, so:

MACHINE=qemuarm bitbake xuser-account -e | grep DEPENDS=

and in there we see:

DEPENDS="  base-files base-passwd shadow-native shadow-sysroot shadow"

base-files is whitelisted in conf/layer.conf as a dependency we don't
need to checksum. The checksum of shadow-native isn't an issue, the
others however are a problem. Since they have well behaved APIs, we can
add them to the list in layer.conf.

What this does mean is that if shadow rebuilds, xuser-account will not
rebuild automatically. I don't believe that should be an issue.

After doing this, I reran the bitbake commands and then:

ls tmp/stamps/all-poky-linux/xuser-account/

shows things like:

0.1-r0.do_populate_sysroot.978b6f9bfd5282121de701a24dc52c80.qemuarm
0.1-r0.do_populate_sysroot.978b6f9bfd5282121de701a24dc52c80.qemux86


basically, the stamp hashes should not change between the two bitbake
commands. SO the patch needed to fix xuser is:

diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf
index 9143bdb..4deee89 100644
--- a/meta/conf/layer.conf
+++ b/meta/conf/layer.conf
@@ -31,6 +31,9 @@ SIGGEN_EXCLUDERECIPES_ABISAFE += " \
   packagegroup-x11-xserver \
   systemd-serialgetty \
   initscripts \
+  shadow \
+  shadow-sysroot \
+  base-passwd \
 "
 
 SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \

The other recipes will need investigating since I doubt its this issue
for them.

Cheers,

Richard

Re: hash generation/PR service problem with xuser-account and other packages

[Steffen Sledz]

On 23.09.2014 15:21, Richard Purdie wrote:
> On Tue, 2014-09-23 at 14:07 +0200, Steffen Sledz wrote:
>> As i mentioned in another thread before we're investigating some problems related with package versions going backwards using a PR service.
>>
>> Now i have some more information. But i'm not able to understand and fix the problem for myself.
>>
>> Assume we have a clean workspace and i bitbake the xuser-account package with this command:
>>
>>   MACHINE="foo" bitbake xuser-account
>>
>> This generates these packages (the 207 comes from our PR service).
>>
>>   tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.207_all.ipk
>>   tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.207_all.ipk
>>   tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.207_all.ipk
>>
>> If i call the same bitbake command again, the same packages are generated. Fine!
>>
>> Now i call:
>>
>>   MACHINE="bar" bitbake xuser-account
>>
>> The generated packages now get a new number from the PR server (so they have a different hash i believe).
>>
>>   tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.208_all.ipk
>>   tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.208_all.ipk
>>   tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.208_all.ipk
>>
>> But this package does not seem to be machine dependent!?
>>
>> If the next build for machine foo is a clean build again (e.g. on our Jenkins continuous integration server which makes clean builds once a week) the PR service generated number wents back to 207 which results in:
>>
>>   "ERROR: QA Issue: Package version for package xuser-account went backwards which would break package feeds"
>>
>> Can someone confirm this behaviour?
>>
>> Is this a bug? I think so.
>>
>> Can someone give some details what went wrong here? Also after studying <http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#checksums> i was not able to fully understand the background, not to speak of fixing something.
>>
>> BTW: We see the same behaviour for the cpufreq-tweaks, linux-firmware, run-postinsts, and some of our own packages.
> 
> I just took a look at this on master. There are some changes between
> master and daisy and I want to get this resolved in master before we
> think about daisy. I thought walking through debugging this may be
> helpful, so, I did:
> 
> MACHINE=qemux86 bitbake xuser-account
> MACHINE=qemuarm bitbake xuser-account
> 
> and sure enough, you are correct, things rebuild and they should not for
> an allarch package like this. Looking at the list of packages being
> built, its pulling in the toolchain. I checked:
> 
> MACHINE=qemuarm bitbake xuser-account -e | grep INHIBIT_DEFAULT_DEPS
> 
> and sure enough, that is set (as per allarch.bbclass) so why is it
> building the toolchain?! It has to be a dependency, so:
> 
> MACHINE=qemuarm bitbake xuser-account -e | grep DEPENDS=
> 
> and in there we see:
> 
> DEPENDS="  base-files base-passwd shadow-native shadow-sysroot shadow"
> 
> base-files is whitelisted in conf/layer.conf as a dependency we don't
> need to checksum. The checksum of shadow-native isn't an issue, the
> others however are a problem. Since they have well behaved APIs, we can
> add them to the list in layer.conf.
> 
> What this does mean is that if shadow rebuilds, xuser-account will not
> rebuild automatically. I don't believe that should be an issue.
> 
> After doing this, I reran the bitbake commands and then:
> 
> ls tmp/stamps/all-poky-linux/xuser-account/
> 
> shows things like:
> 
> 0.1-r0.do_populate_sysroot.978b6f9bfd5282121de701a24dc52c80.qemuarm
> 0.1-r0.do_populate_sysroot.978b6f9bfd5282121de701a24dc52c80.qemux86
> 
> 
> basically, the stamp hashes should not change between the two bitbake
> commands. SO the patch needed to fix xuser is:
> 
> diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf
> index 9143bdb..4deee89 100644
> --- a/meta/conf/layer.conf
> +++ b/meta/conf/layer.conf
> @@ -31,6 +31,9 @@ SIGGEN_EXCLUDERECIPES_ABISAFE += " \
>    packagegroup-x11-xserver \
>    systemd-serialgetty \
>    initscripts \
> +  shadow \
> +  shadow-sysroot \
> +  base-passwd \
>  "
>  
>  SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \

Hi Richard, thanx for this detailled explanation. It is another step on my way to understand all this. ;-)

> The other recipes will need investigating since I doubt its this issue
> for them.

Is there someone who can do this work in a similar brightening way Richard did?

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: hash generation/PR service problem with xuser-account and other packages

[Richard Purdie]

On Tue, 2014-09-23 at 16:51 +0200, Steffen Sledz wrote:
> Hi Richard, thanx for this detailled explanation. It is another step on my way to understand all this. ;-)
> 
> > The other recipes will need investigating since I doubt its this issue
> > for them.
> 
> Is there someone who can do this work in a similar brightening way Richard did?

I had a look at run-postinsts:

MACHINE=qemux86 bitbake run-postinsts
MACHINE=qemuarm bitbake run-postinsts

ls tmp/stamps/all-poky-linux/run-postinsts/

Hmm, two different sigdata files for do_package.

bitbake-diffsigs tmp/stamps/all-poky-linux/run-postinsts/1.0-r9.do_package.sigdata.*
(assumes only two sigdata files there, idea is to compare them)

Difference is the OVERRIDES variable. We therefore dump one of the files:

bitbake-diffsigs tmp/stamps/all-poky-linux/run-postinsts/1.0-r9.do_package.sigdata.X | grep OVERRIDES

and see that OVERRIDES is depended on in two places:

List of dependencies for variable systemd_populate_packages is set(['systemd_prerm', 'D', 'exec_prefix', 'OVERRIDES', 'sysconfdir', 'SYSTEMD_PACKAGES', 'systemd_postinst', 'nonarch_base_libdir', 'PACKAGES'])
List of dependencies for variable populate_packages_updatercd is set(['D', 'updatercd_postrm', 'OVERRIDES', 'INITSCRIPT_NAME', 'INHIBIT_UPDATERCD_BBCLASS', 'MLPREFIX', 'updatercd_postinst', 'updatercd_preinst', 'updatercd_prerm', 'INIT_D_DIR', 'PACKAGES', 'INITSCRIPT_PACKAGES', 'UPDATERCPN'])

So the patch to fix this is:

diff --git a/meta/classes/systemd.bbclass b/meta/classes/systemd.bbclass
index a6ad723..708ba89 100644
--- a/meta/classes/systemd.bbclass
+++ b/meta/classes/systemd.bbclass
@@ -55,6 +55,8 @@ fi
 
 
 systemd_populate_packages[vardeps] += "systemd_prerm systemd_postinst"
+systemd_populate_packages[vardepsexclude] += "OVERRIDES"
+
 
 python systemd_populate_packages() {
     if not bb.utils.contains('DISTRO_FEATURES', 'systemd', True, False, d):
diff --git a/meta/classes/update-rc.d.bbclass b/meta/classes/update-rc.d.bbclass
index 3c5414b..bc1aa7d 100644
--- a/meta/classes/update-rc.d.bbclass
+++ b/meta/classes/update-rc.d.bbclass
@@ -72,6 +72,7 @@ PACKAGESPLITFUNCS_prepend = "populate_packages_updatercd "
 PACKAGESPLITFUNCS_remove_class-nativesdk = "populate_packages_updatercd "
 
 populate_packages_updatercd[vardeps] += "updatercd_prerm updatercd_postrm updatercd_preinst updatercd_postinst"
+populate_packages_updatercd[vardepsexclude] += "OVERRIDES"
 
 python populate_packages_updatercd () {
     def update_rcd_auto_depend(pkg):


since OVERRIDES isn't a needed dependency here, its effect is all that matters, not its contents.

Cheers,

Richard

Re: hash generation/PR service problem with xuser-account and other packages

[Steffen Sledz]

On 23.09.2014 18:16, Richard Purdie wrote:
> On Tue, 2014-09-23 at 16:51 +0200, Steffen Sledz wrote:
>> Hi Richard, thanx for this detailled explanation. It is another step on my way to understand all this. ;-)
>>
>>> The other recipes will need investigating since I doubt its this issue
>>> for them.
>>
>> Is there someone who can do this work in a similar brightening way Richard did?
> 
> I had a look at run-postinsts:
> 
> MACHINE=qemux86 bitbake run-postinsts
> MACHINE=qemuarm bitbake run-postinsts
> 
> ls tmp/stamps/all-poky-linux/run-postinsts/
> 
> Hmm, two different sigdata files for do_package.
> 
> bitbake-diffsigs tmp/stamps/all-poky-linux/run-postinsts/1.0-r9.do_package.sigdata.*
> (assumes only two sigdata files there, idea is to compare them)
> 
> Difference is the OVERRIDES variable. We therefore dump one of the files:
> 
> bitbake-diffsigs tmp/stamps/all-poky-linux/run-postinsts/1.0-r9.do_package.sigdata.X | grep OVERRIDES
> 
> and see that OVERRIDES is depended on in two places:
> 
> List of dependencies for variable systemd_populate_packages is set(['systemd_prerm', 'D', 'exec_prefix', 'OVERRIDES', 'sysconfdir', 'SYSTEMD_PACKAGES', 'systemd_postinst', 'nonarch_base_libdir', 'PACKAGES'])
> List of dependencies for variable populate_packages_updatercd is set(['D', 'updatercd_postrm', 'OVERRIDES', 'INITSCRIPT_NAME', 'INHIBIT_UPDATERCD_BBCLASS', 'MLPREFIX', 'updatercd_postinst', 'updatercd_preinst', 'updatercd_prerm', 'INIT_D_DIR', 'PACKAGES', 'INITSCRIPT_PACKAGES', 'UPDATERCPN'])
> 
> So the patch to fix this is:
> 
> diff --git a/meta/classes/systemd.bbclass b/meta/classes/systemd.bbclass
> index a6ad723..708ba89 100644
> --- a/meta/classes/systemd.bbclass
> +++ b/meta/classes/systemd.bbclass
> @@ -55,6 +55,8 @@ fi
>  
>  
>  systemd_populate_packages[vardeps] += "systemd_prerm systemd_postinst"
> +systemd_populate_packages[vardepsexclude] += "OVERRIDES"
> +
>  
>  python systemd_populate_packages() {
>      if not bb.utils.contains('DISTRO_FEATURES', 'systemd', True, False, d):
> diff --git a/meta/classes/update-rc.d.bbclass b/meta/classes/update-rc.d.bbclass
> index 3c5414b..bc1aa7d 100644
> --- a/meta/classes/update-rc.d.bbclass
> +++ b/meta/classes/update-rc.d.bbclass
> @@ -72,6 +72,7 @@ PACKAGESPLITFUNCS_prepend = "populate_packages_updatercd "
>  PACKAGESPLITFUNCS_remove_class-nativesdk = "populate_packages_updatercd "
>  
>  populate_packages_updatercd[vardeps] += "updatercd_prerm updatercd_postrm updatercd_preinst updatercd_postinst"
> +populate_packages_updatercd[vardepsexclude] += "OVERRIDES"
>  
>  python populate_packages_updatercd () {
>      def update_rcd_auto_depend(pkg):
> 
> 
> since OVERRIDES isn't a needed dependency here, its effect is all that matters, not its contents.

This patch fixes the cpufreq-tweaks (from meta-angstrom) problem too. :)

So there's still linux-firmware pending from the packages i mentioned.

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: hash generation/PR service problem with xuser-account and other packages

[Richard Purdie]

On Wed, 2014-09-24 at 15:22 +0200, Steffen Sledz wrote:
> This patch fixes the cpufreq-tweaks (from meta-angstrom) problem too. :)
> 
> So there's still linux-firmware pending from the packages i mentioned.

Actually, I pushed a patch for that last night. It was the same debug
approach as the second case I mentioned, diff the sigs and the issue
became clear.

Cheers,

Richard