[meta-oe][PATCH 0/3] Add bits to save/restore URI headrevs

[meta-oe][PATCH 0/3] Add bits to save/restore URI headrevs

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This is useful to support BB_NO_NETWORK with AUTOREV, by letting someone ship
dumped headrevs to the user, who then inherit this class, which ensures that
the cached headrevs are used, and upstream is not contacted at parse time.
BB_SRCREV_POLICY will be set to "cache" as well, if it's not already set, as
otherwise bitbake will contact upstream to update the cached values.

This is helpful when shipping downloads to someone when there's a desire to
support both BB_NO_NETWORK and AUTOREV.

The restore_headrevs.bbclass will restore dumped headrevs at config parse time
(add to INHERIT), and the oe.headrevs python module provides a function one
can call to dump the headrevs.

The following changes since commit 62bfd1f93f8873611e818c7cc8c13a761d629502:

  wireshark: update to version 2.0.0 (2015-11-30 14:39:59 -0500)

are available in the git repository at:

  git://github.com/kergoth/meta-openembedded.git headrevs
  https://github.com/kergoth/meta-openembedded/tree/headrevs

Christopher Larson (3):
  oe: add python namespace package
  oe.headrevs: save and restore the bitbake URI headrevs
  restore_headrevs.bbclass: add

 meta-oe/classes/restore_headrevs.bbclass | 32 +++++++++++++++++++++++++++
 meta-oe/lib/oe/__init__.py               |  2 ++
 meta-oe/lib/oe/headrevs.py               | 38 ++++++++++++++++++++++++++++++++
 3 files changed, 72 insertions(+)
 create mode 100644 meta-oe/classes/restore_headrevs.bbclass
 create mode 100644 meta-oe/lib/oe/__init__.py
 create mode 100644 meta-oe/lib/oe/headrevs.py

-- 
2.2.1

[meta-oe][PATCH 1/3] oe: add python namespace package

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta-oe/lib/oe/__init__.py | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 meta-oe/lib/oe/__init__.py

diff --git a/meta-oe/lib/oe/__init__.py b/meta-oe/lib/oe/__init__.py
new file mode 100644
index 0000000..3ad9513
--- /dev/null
+++ b/meta-oe/lib/oe/__init__.py
@@ -0,0 +1,2 @@
+from pkgutil import extend_path
+__path__ = extend_path(__path__, __name__)
-- 
2.2.1

[meta-oe][PATCH 2/3] oe.headrevs: save and restore the bitbake URI headrevs

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This module provides functions to save and restore the BB_URI_HEADREVS from/to
the persist_data database. The BB_URI_HEADREVS are the mappings between git
refs and the revisions they refer to.

This is useful to fully support BB_NO_NETWORK when AUTOREV is involved, as we
can use the dumped mappings rather than contacting upstream at parse time.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta-oe/lib/oe/headrevs.py | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 meta-oe/lib/oe/headrevs.py

diff --git a/meta-oe/lib/oe/headrevs.py b/meta-oe/lib/oe/headrevs.py
new file mode 100644
index 0000000..9245300
--- /dev/null
+++ b/meta-oe/lib/oe/headrevs.py
@@ -0,0 +1,38 @@
+"""Save and restore the bitbake URI headrevs.
+
+This module provides functions to save and restore the BB_URI_HEADREVS from/to
+the persist_data database. The BB_URI_HEADREVS are the mappings between git
+refs and the revisions they refer to.
+
+This is useful to fully support BB_NO_NETWORK when AUTOREV is involved, as we
+can use the dumped mappings rather than contacting upstream at parse time. See
+also restore_headrevs.bbclass.
+
+By default, any existing mapping will not be overwritten, as we assume the
+user's mappings are more current than any dumped content.
+"""
+
+
+def copy_persist_domain(d, domain, other_db_path, restore=False, overwrite=False):
+    import contextlib
+
+    source_table = bb.persist_data.persist(domain, d)
+    dest_table = bb.persist_data.SQLTable(other_db_path, domain)
+    if restore:
+        source_table, dest_table = dest_table, source_table
+
+    with contextlib.nested(source_table, dest_table):
+        if overwrite:
+            dest_table.update(source_table)
+        else:
+            for key in source_table:
+                if key not in dest_table:
+                    dest_table[key] = source_table[key]
+
+
+def dump_headrevs(d, dump_db_path):
+    copy_persist_domain(d, 'BB_URI_HEADREVS', dump_db_path)
+
+
+def restore_headrevs(d, dump_db_path):
+    copy_persist_domain(d, 'BB_URI_HEADREVS', dump_db_path, restore=True)
-- 
2.2.1

[meta-oe][PATCH 3/3] restore_headrevs.bbclass: add

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

If BB_NO_NETWORK=1, this class restores saved bitbake URI headrevs.

This is useful to support BB_NO_NETWORK with AUTOREV, by letting someone ship
dumped headrevs to the user, who then inherit this class, which ensures that
the cached headrevs are used, and upstream is not contacted at parse time.
BB_SRCREV_POLICY will be set to "cache" as well, if it's not already set, as
otherwise bitbake will contact upstream to update the cached values.

This only handles restoring dumped headrevs. Using the python module to dump
to this database will likely be done at release time, elsewhere.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta-oe/classes/restore_headrevs.bbclass | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 meta-oe/classes/restore_headrevs.bbclass

diff --git a/meta-oe/classes/restore_headrevs.bbclass b/meta-oe/classes/restore_headrevs.bbclass
new file mode 100644
index 0000000..0d83428
--- /dev/null
+++ b/meta-oe/classes/restore_headrevs.bbclass
@@ -0,0 +1,32 @@
+# If BB_NO_NETWORK=1, this class restores saved bitbake URI headrevs.
+#
+# This is useful to support BB_NO_NETWORK with AUTOREV, by letting someone
+# ship dumped headrevs to the user, who then inherit this class, which ensures
+# that the cached headrevs are used, and upstream is not contacted at parse
+# time. BB_SRCREV_POLICY will be set to "cache" as well, if it's not already
+# set, as otherwise bitbake will contact upstream to update the cached values.
+#
+# See also lib/oe/headrevs.py.
+#
+# This only handles restoring dumped headrevs. Using the python module to dump
+# to this database will likely be done at release time, elsewhere.
+
+HEADREVS_RESTORE_STAMP ?= "${PERSISTENT_DIR}/headrevs_restored"
+DUMPED_HEADREVS_DB ?= "${COREBASE}/headrevs.db"
+
+python restore_dumped_headrevs() {
+    dump_db_path = d.getVar('DUMPED_HEADREVS_DB', True)
+    if os.path.exists(dump_db_path) and d.getVar('BB_NO_NETWORK', True) == '1':
+        if 'BB_SRCREV_POLICY' not in d:
+            d.setVar('BB_SRCREV_POLICY', 'cache')
+
+        stamp_path = d.getVar('HEADREVS_RESTORE_STAMP', True)
+        if (not os.path.exists(stamp_path) or
+                os.path.getmtime(dump_db_path) > os.path.getmtime(stamp_path)):
+            import oe.headrevs
+            oe.headrevs.restore_headrevs(d, dump_db_path)
+            bb.utils.mkdirhier(os.path.dirname(stamp_path))
+            open(stamp_path, 'w').close()
+}
+restore_dumped_headrevs[eventmask] = "bb.event.ConfigParsed"
+addhandler restore_dumped_headrevs
-- 
2.2.1

Re: [meta-oe][PATCH 0/3] Add bits to save/restore URI headrevs

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2691 bytes --]

On Tue, Dec 15, 2015 at 12:37:44PM -0700, Christopher Larson wrote:
> From: Christopher Larson <chris_larson@mentor.com>
> 
> This is useful to support BB_NO_NETWORK with AUTOREV, by letting someone ship
> dumped headrevs to the user, who then inherit this class, which ensures that
> the cached headrevs are used, and upstream is not contacted at parse time.
> BB_SRCREV_POLICY will be set to "cache" as well, if it's not already set, as
> otherwise bitbake will contact upstream to update the cached values.
> 
> This is helpful when shipping downloads to someone when there's a desire to
> support both BB_NO_NETWORK and AUTOREV.
> 
> The restore_headrevs.bbclass will restore dumped headrevs at config parse time
> (add to INHERIT), and the oe.headrevs python module provides a function one
> can call to dump the headrevs.

Maybe I'm missing something, but what's advantage of shipping something
with AUTOREV which is then used together with BB_NO_NETWORK?

We already have some implementation in bitbake which dumps latest
SRCREVs used by all recipes (including AUTOREV) onces so I would expect
that the release will ship this .inc files with locked revisions and if
the user of such release wants to use BB_NO_NETWORK he will also include
that .inc file with locked revisions.

Is it because it's easier of faster to dump them from the headrefs db
you already ship with release and you don't want to generate .inc from
buildhistory?

Cheers,

> 
> The following changes since commit 62bfd1f93f8873611e818c7cc8c13a761d629502:
> 
>   wireshark: update to version 2.0.0 (2015-11-30 14:39:59 -0500)
> 
> are available in the git repository at:
> 
>   git://github.com/kergoth/meta-openembedded.git headrevs
>   https://github.com/kergoth/meta-openembedded/tree/headrevs
> 
> Christopher Larson (3):
>   oe: add python namespace package
>   oe.headrevs: save and restore the bitbake URI headrevs
>   restore_headrevs.bbclass: add
> 
>  meta-oe/classes/restore_headrevs.bbclass | 32 +++++++++++++++++++++++++++
>  meta-oe/lib/oe/__init__.py               |  2 ++
>  meta-oe/lib/oe/headrevs.py               | 38 ++++++++++++++++++++++++++++++++
>  3 files changed, 72 insertions(+)
>  create mode 100644 meta-oe/classes/restore_headrevs.bbclass
>  create mode 100644 meta-oe/lib/oe/__init__.py
>  create mode 100644 meta-oe/lib/oe/headrevs.py
> 
> -- 
> 2.2.1
> 
> -- 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: [meta-oe][PATCH 0/3] Add bits to save/restore URI headrevs

[Christopher Larson]

On Tue, Dec 15, 2015 at 12:49 PM, Martin Jansa <martin.jansa@gmail.com>
wrote:

> On Tue, Dec 15, 2015 at 12:37:44PM -0700, Christopher Larson wrote:
> > From: Christopher Larson <chris_larson@mentor.com>
> >
> > This is useful to support BB_NO_NETWORK with AUTOREV, by letting someone
> ship
> > dumped headrevs to the user, who then inherit this class, which ensures
> that
> > the cached headrevs are used, and upstream is not contacted at parse
> time.
> > BB_SRCREV_POLICY will be set to "cache" as well, if it's not already
> set, as
> > otherwise bitbake will contact upstream to update the cached values.
> >
> > This is helpful when shipping downloads to someone when there's a desire
> to
> > support both BB_NO_NETWORK and AUTOREV.
> >
> > The restore_headrevs.bbclass will restore dumped headrevs at config
> parse time
> > (add to INHERIT), and the oe.headrevs python module provides a function
> one
> > can call to dump the headrevs.
>
> Maybe I'm missing something, but what's advantage of shipping something
> with AUTOREV which is then used together with BB_NO_NETWORK?
>

It's quite useful to not have to go through and manually lock down all your
kernels recipes when doing a release. All our releases ship with sources
and are BB_NO_NETWORK-capable, as that's a requirement for a lot of folks.


> We already have some implementation in bitbake which dumps latest
> SRCREVs used by all recipes (including AUTOREV) onces so I would expect
> that the release will ship this .inc files with locked revisions and if
> the user of such release wants to use BB_NO_NETWORK he will also include
> that .inc file with locked revisions.
>
> Is it because it's easier of faster to dump them from the headrefs db
> you already ship with release and you don't want to generate .inc from
> buildhistory?


Actually, this was in my patch queue from before buildhistory existed, and
I somehow managed to forget about buildhistory-collect-srcrevs :) Feel free
to ignore this, in that case.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics