[PATCH 0/1] toaster: Y9137 get packages from sstate

[PATCH 0/1] toaster: Y9137 get packages from sstate

[brian avery]

This addresses the missing packages part of Y9137.  This patch allows us to see
packages that came from a setscene.
Testing it is a little hard:
1) You can add the following to your local.conf in a toaster dir:
SSTATE_MIRRORS="file://.* http://downloads.yoctoproject.org/tools/support/toaster/sstate-test/master/sstate-cache/PATH"
This might also work:
SSTATE_MIRRORS="file://.* http://sstate.yoctoproject.org/dev/PATH"
and build dropbear in a LOCAL project. You should see ~322 pkgs built.

The hard way to test this is to:
1) start toaster
2) build dropbear as a local project
3) stop toaster
4) rm everything except sstate-cache/ and conf/
5) restart toaster
6) build dropbearas a local project.  it should come from sstate and downloads should remain empty.

Note, there is still a big difference between the pkgs we get from a build ~2844 and the ones we get from setscening.
The setscening doesn't package:
*locales
glibc-charmap*
glibc-gconv*
perl-module*
ncurses-lib*
as they seem to be unnecessary in order to be able to package dropbear.

---
This does not address the need to change "packages built" to something more generic like packages in the UI.


-b
an Intel employee

The following changes since commit 325a35c43df83d4268fc54a8955ee785f6a1f6fe:

  toaster: update the meta-yocto toaster configuration file (2016-03-07 21:36:03 -0800)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib bavery/toaster/getPkgsFromSstate-Y9137V2
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/toaster/getPkgsFromSstate-Y9137V2

brian avery (1):
  toaster: show packages that were setscened into existence too

 meta/classes/toaster.bbclass | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

--
1.9.1

[PATCH 1/1] toaster: show packages that were setscened into existence too

[brian avery]

        We were previously ignoring pkgs that came in from a setscene (like
        from an sstate mirror).  With this patch we can use pkgs than come
        from sstate for image customisation as well.
        [YOCTO #9137]

Signed-off-by: brian avery <brian.avery@intel.com>
---
 meta/classes/toaster.bbclass | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass
index 51a4c74..4afaf1c 100644
--- a/meta/classes/toaster.bbclass
+++ b/meta/classes/toaster.bbclass
@@ -112,6 +112,26 @@ def _toaster_load_pkgdatafile(dirpath, filepath):
                 pass    # ignore lines without valid key: value pairs
     return pkgdata
 
+python toaster_package_dumpdata_setscene() {
+    """
+    Dumps the data created by package_setscene
+    """
+    # replicate variables from the package.bbclass
+    packages = d.getVar('PACKAGES', True)
+    pkgdatadir = d.getVar('PKGDATA_DIR', True)
+    # scan and send data for each package
+    lpkgdata = {}
+    for pkg in packages.split():
+        try:
+          lpkgdata = _toaster_load_pkgdatafile(pkgdatadir + "/runtime/", pkg)
+        except:
+          # these are typically foo-locale which actually point into foo-locale-<language> in runtime-rprovides
+          bb.note("toaster_package_dumpdata_setscene: failed to load pkg information for: %s:%s"%(pkg,sys.exc_info()[0]))
+        # Fire an event containing the pkg data
+        bb.event.fire(bb.event.MetadataEvent("SinglePackageInfo", lpkgdata), d)
+
+}
+
 
 python toaster_package_dumpdata() {
     """
@@ -120,8 +140,6 @@ python toaster_package_dumpdata() {
     # replicate variables from the package.bbclass
 
     packages = d.getVar('PACKAGES', True)
-    pkgdest = d.getVar('PKGDEST', True)
-
     pkgdatadir = d.getVar('PKGDESTWORK', True)
 
     # scan and send data for each package
@@ -362,6 +380,9 @@ toaster_collect_task_stats[eventmask] = "bb.event.BuildCompleted bb.build.TaskSu
 addhandler toaster_buildhistory_dump
 toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted"
 
+do_packagedata_setscene[postfuncs] += "toaster_package_dumpdata_setscene "
+do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata_setscene "
+
 do_package[postfuncs] += "toaster_package_dumpdata "
 do_package[vardepsexclude] += "toaster_package_dumpdata "
 
-- 
1.9.1

Re: [PATCH 0/1] toaster: Y9137 get packages from sstate

[Barros Pena, Belen]

On 08/03/2016 05:50, "toaster-bounces@yoctoproject.org on behalf of brian
avery" <toaster-bounces@yoctoproject.org on behalf of
avery.brian@gmail.com> wrote:

>This addresses the missing packages part of Y9137.  This patch allows us
>to see
>packages that came from a setscene.

This works for me, which is great. But in my innocence, I somehow assumed
it would also perform the miracle in those builds where no sstate is used
(the task output from a previous build is used. We call those tasks
'Prebuilt' in Toaster, although I am not sure how they are called in the
non-Toaster world). The kind of build you do when, after step 6 below, you
rebuild dropbear yet again.

This might be completely unrealistic though.

Cheers

Belén


>Testing it is a little hard:
>1) You can add the following to your local.conf in a toaster dir:
>SSTATE_MIRRORS="file://.*
>http://downloads.yoctoproject.org/tools/support/toaster/sstate-test/master
>/sstate-cache/PATH"
>This might also work:
>SSTATE_MIRRORS="file://.* http://sstate.yoctoproject.org/dev/PATH"
>and build dropbear in a LOCAL project. You should see ~322 pkgs built.
>
>The hard way to test this is to:
>1) start toaster
>2) build dropbear as a local project
>3) stop toaster
>4) rm everything except sstate-cache/ and conf/
>5) restart toaster
>6) build dropbearas a local project.  it should come from sstate and
>downloads should remain empty.
>
>Note, there is still a big difference between the pkgs we get from a
>build ~2844 and the ones we get from setscening.
>The setscening doesn't package:
>*locales
>glibc-charmap*
>glibc-gconv*
>perl-module*
>ncurses-lib*
>as they seem to be unnecessary in order to be able to package dropbear.
>
>---
>This does not address the need to change "packages built" to something
>more generic like packages in the UI.
>
>
>-b
>an Intel employee
>
>The following changes since commit
>325a35c43df83d4268fc54a8955ee785f6a1f6fe:
>
>  toaster: update the meta-yocto toaster configuration file (2016-03-07
>21:36:03 -0800)
>
>are available in the git repository at:
>
>  git://git.yoctoproject.org/poky-contrib
>bavery/toaster/getPkgsFromSstate-Y9137V2
>  
>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/toaster/ge
>tPkgsFromSstate-Y9137V2
>
>brian avery (1):
>  toaster: show packages that were setscened into existence too
>
> meta/classes/toaster.bbclass | 25 +++++++++++++++++++++++--
> 1 file changed, 23 insertions(+), 2 deletions(-)
>
>--
>1.9.1
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster

Re: [PATCH 0/1] toaster: Y9137 get packages from sstate

[Brian Avery]

in those cases, bitbake itself doesn't do anything, so the 0 is
correct, albeit perhaps confusing. however, since it did do something
once before the package is available for ic. My main fear was having
packages be available but toaster being unable to add them into a CI.
-b
an intel employee

On Tue, Mar 8, 2016 at 7:07 AM, Barros Pena, Belen
<belen.barros.pena@intel.com> wrote:
>
>
> On 08/03/2016 05:50, "toaster-bounces@yoctoproject.org on behalf of brian
> avery" <toaster-bounces@yoctoproject.org on behalf of
> avery.brian@gmail.com> wrote:
>
>>This addresses the missing packages part of Y9137.  This patch allows us
>>to see
>>packages that came from a setscene.
>
> This works for me, which is great. But in my innocence, I somehow assumed
> it would also perform the miracle in those builds where no sstate is used
> (the task output from a previous build is used. We call those tasks
> 'Prebuilt' in Toaster, although I am not sure how they are called in the
> non-Toaster world). The kind of build you do when, after step 6 below, you
> rebuild dropbear yet again.
>
> This might be completely unrealistic though.
>
> Cheers
>
> Belén
>
>
>>Testing it is a little hard:
>>1) You can add the following to your local.conf in a toaster dir:
>>SSTATE_MIRRORS="file://.*
>>http://downloads.yoctoproject.org/tools/support/toaster/sstate-test/master
>>/sstate-cache/PATH"
>>This might also work:
>>SSTATE_MIRRORS="file://.* http://sstate.yoctoproject.org/dev/PATH"
>>and build dropbear in a LOCAL project. You should see ~322 pkgs built.
>>
>>The hard way to test this is to:
>>1) start toaster
>>2) build dropbear as a local project
>>3) stop toaster
>>4) rm everything except sstate-cache/ and conf/
>>5) restart toaster
>>6) build dropbearas a local project.  it should come from sstate and
>>downloads should remain empty.
>>
>>Note, there is still a big difference between the pkgs we get from a
>>build ~2844 and the ones we get from setscening.
>>The setscening doesn't package:
>>*locales
>>glibc-charmap*
>>glibc-gconv*
>>perl-module*
>>ncurses-lib*
>>as they seem to be unnecessary in order to be able to package dropbear.
>>
>>---
>>This does not address the need to change "packages built" to something
>>more generic like packages in the UI.
>>
>>
>>-b
>>an Intel employee
>>
>>The following changes since commit
>>325a35c43df83d4268fc54a8955ee785f6a1f6fe:
>>
>>  toaster: update the meta-yocto toaster configuration file (2016-03-07
>>21:36:03 -0800)
>>
>>are available in the git repository at:
>>
>>  git://git.yoctoproject.org/poky-contrib
>>bavery/toaster/getPkgsFromSstate-Y9137V2
>>
>>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/toaster/ge
>>tPkgsFromSstate-Y9137V2
>>
>>brian avery (1):
>>  toaster: show packages that were setscened into existence too
>>
>> meta/classes/toaster.bbclass | 25 +++++++++++++++++++++++--
>> 1 file changed, 23 insertions(+), 2 deletions(-)
>>
>>--
>>1.9.1
>>--
>>_______________________________________________
>>toaster mailing list
>>toaster@yoctoproject.org
>>https://lists.yoctoproject.org/listinfo/toaster
>

Re: [PATCH 1/1] toaster: show packages that were setscened into existence too

[Michael Wood]

Thanks sent with some commit messages fixes to openembedded-core 
http://lists.openembedded.org/pipermail/openembedded-core/2016-March/118796.html 
and pushed to toaster-next

On 08/03/16 05:50, brian avery wrote:
>          We were previously ignoring pkgs that came in from a setscene (like
>          from an sstate mirror).  With this patch we can use pkgs than come
>          from sstate for image customisation as well.
>          [YOCTO #9137]
>
> Signed-off-by: brian avery <brian.avery@intel.com>
> ---
>   meta/classes/toaster.bbclass | 25 +++++++++++++++++++++++--
>   1 file changed, 23 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass
> index 51a4c74..4afaf1c 100644
> --- a/meta/classes/toaster.bbclass
> +++ b/meta/classes/toaster.bbclass
> @@ -112,6 +112,26 @@ def _toaster_load_pkgdatafile(dirpath, filepath):
>                   pass    # ignore lines without valid key: value pairs
>       return pkgdata
>   
> +python toaster_package_dumpdata_setscene() {
> +    """
> +    Dumps the data created by package_setscene
> +    """
> +    # replicate variables from the package.bbclass
> +    packages = d.getVar('PACKAGES', True)
> +    pkgdatadir = d.getVar('PKGDATA_DIR', True)
> +    # scan and send data for each package
> +    lpkgdata = {}
> +    for pkg in packages.split():
> +        try:
> +          lpkgdata = _toaster_load_pkgdatafile(pkgdatadir + "/runtime/", pkg)
> +        except:
> +          # these are typically foo-locale which actually point into foo-locale-<language> in runtime-rprovides
> +          bb.note("toaster_package_dumpdata_setscene: failed to load pkg information for: %s:%s"%(pkg,sys.exc_info()[0]))
> +        # Fire an event containing the pkg data
> +        bb.event.fire(bb.event.MetadataEvent("SinglePackageInfo", lpkgdata), d)
> +
> +}
> +
>   
>   python toaster_package_dumpdata() {
>       """
> @@ -120,8 +140,6 @@ python toaster_package_dumpdata() {
>       # replicate variables from the package.bbclass
>   
>       packages = d.getVar('PACKAGES', True)
> -    pkgdest = d.getVar('PKGDEST', True)
> -
>       pkgdatadir = d.getVar('PKGDESTWORK', True)
>   
>       # scan and send data for each package
> @@ -362,6 +380,9 @@ toaster_collect_task_stats[eventmask] = "bb.event.BuildCompleted bb.build.TaskSu
>   addhandler toaster_buildhistory_dump
>   toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted"
>   
> +do_packagedata_setscene[postfuncs] += "toaster_package_dumpdata_setscene "
> +do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata_setscene "
> +
>   do_package[postfuncs] += "toaster_package_dumpdata "
>   do_package[vardepsexclude] += "toaster_package_dumpdata "
>