[PATCH v4 0/8] Support signed RPM package feeds

[PATCH v4 0/8] Support signed RPM package feeds

[Markus Lehtonen]

Fourth iteration of the patchset enabling signed RPM package feeds.
Changes since v3:
- new patch for the 'signing-keys' recipe, fixing a likely failure in the dnf
  package feed selftest
- one comment line added in "dnf: expand dnf selftest to test signed package feeds"
  patch

The following changes since commit 5a25ed1071f0d9b7d95edcc2b5b4545f960d5f95:

  image-prelink: Disable for musl images (2017-08-11 08:14:19 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib marquiz/fixes-11209
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-11209

Markus Lehtonen (8):
  lib/oe/package_manager: re-implement rpm feed signing
  dnf: rrecommend gnupg
  signing-keys: delete unused tasks
  sign_package_feed.bbclass: install signing key into rootfs
  package_manager.py: enable dnf's repo_gpgcheck if feed signing is
    enabled
  oeqa: fix dnf tests
  oeqa: fix temp file handling in dnf package feed test
  dnf: expand dnf selftest to test signed package feeds

 meta-selftest/files/signing/key.passphrase         |  1 +
 .../cases/{dnf-runtime.py => dnf_runtime.py}       | 13 ++++++++----
 meta/classes/sign_package_feed.bbclass             |  3 +++
 meta/lib/oe/package_manager.py                     | 24 +++++++++++++++++++---
 meta/lib/oeqa/selftest/cases/runtime_test.py       | 14 +++++++++++--
 meta/recipes-core/meta/signing-keys.bb             |  8 ++++++++
 meta/recipes-devtools/dnf/dnf_2.6.3.bb             |  2 ++
 7 files changed, 56 insertions(+), 9 deletions(-)
 create mode 100644 meta-selftest/files/signing/key.passphrase
 rename meta-selftest/lib/oeqa/runtime/cases/{dnf-runtime.py => dnf_runtime.py} (77%)

-- 
2.12.3

[PATCH v4 1/8] lib/oe/package_manager: re-implement rpm feed signing

[Markus Lehtonen]

[YOCTO #11209]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oe/package_manager.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 518cf8dbe3..942f2dd903 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -104,13 +104,25 @@ class Indexer(object, metaclass=ABCMeta):
 class RpmIndexer(Indexer):
     def write_index(self):
         if self.d.getVar('PACKAGE_FEED_SIGN') == '1':
-            raise NotImplementedError('Package feed signing not yet implementd for rpm')
+            signer = get_signer(self.d, self.d.getVar('PACKAGE_FEED_GPG_BACKEND'))
+        else:
+            signer = None
 
         createrepo_c = bb.utils.which(os.environ['PATH'], "createrepo_c")
         result = create_index("%s --update -q %s" % (createrepo_c, self.deploy_dir))
         if result:
             bb.fatal(result)
 
+        # Sign repomd
+        if signer:
+            sig_type = self.d.getVar('PACKAGE_FEED_GPG_SIGNATURE_TYPE')
+            is_ascii_sig = (sig_type.upper() != "BIN")
+            signer.detach_sign(os.path.join(self.deploy_dir, 'repodata', 'repomd.xml'),
+                               self.d.getVar('PACKAGE_FEED_GPG_NAME'),
+                               self.d.getVar('PACKAGE_FEED_GPG_PASSPHRASE_FILE'),
+                               armor=is_ascii_sig)
+
+
 class OpkgIndexer(Indexer):
     def write_index(self):
         arch_vars = ["ALL_MULTILIB_PACKAGE_ARCHS",
-- 
2.12.3

[PATCH v4 2/8] dnf: rrecommend gnupg

[Markus Lehtonen]

This makes it possible to enable 'repo_gpgcheck' in dnf.conf. That is, do
GPG signature check on repository metadata. Without gnupg dnf fails with
"error: Invalid crypto engine."

[YOCTO #11209]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/recipes-devtools/dnf/dnf_2.6.3.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-devtools/dnf/dnf_2.6.3.bb b/meta/recipes-devtools/dnf/dnf_2.6.3.bb
index ade55ea0a8..51072901e4 100644
--- a/meta/recipes-devtools/dnf/dnf_2.6.3.bb
+++ b/meta/recipes-devtools/dnf/dnf_2.6.3.bb
@@ -26,6 +26,8 @@ EXTRA_OECMAKE = " -DWITH_MAN=0 -DPYTHON_INSTALL_DIR=${PYTHON_SITEPACKAGES_DIR} -
 
 BBCLASSEXTEND = "native nativesdk"
 RDEPENDS_${PN}_class-target += "python3-core python3-codecs python3-netclient python3-email python3-threading python3-distutils librepo python3-shell python3-subprocess libcomps libdnf python3-sqlite3 python3-compression python3-rpm python3-iniparse python3-json python3-importlib python3-curses python3-argparse python3-misc python3-gpg"
+# Recommend gnupg so that GPG signature check on repository metadata is possible
+RRECOMMENDS_${PN}_class-target += "gnupg"
 
 # Create a symlink called 'dnf' as 'make install' does not do it, but
 # .spec file in dnf source tree does (and then Fedora and dnf documentation
-- 
2.12.3

[PATCH v4 3/8] signing-keys: delete unused tasks

[Markus Lehtonen]

In particular, do_unpack needs to be deleted as it was causing missing
subpackages. Do_unpack cleans the builddir and if it was run after
do_get_public_keys it cleaned up all keys causing no packages to be
built.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/recipes-core/meta/signing-keys.bb | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/recipes-core/meta/signing-keys.bb b/meta/recipes-core/meta/signing-keys.bb
index aaa01d0c34..ebfb08c433 100644
--- a/meta/recipes-core/meta/signing-keys.bb
+++ b/meta/recipes-core/meta/signing-keys.bb
@@ -71,3 +71,11 @@ do_deploy[cleandirs] = "${DEPLOYDIR}"
 # clear stamp-extra-info since MACHINE is normally put there by deploy.bbclass
 do_deploy[stamp-extra-info] = ""
 addtask deploy after do_get_public_keys
+
+# Delete unnecessary tasks. In particular, "do_unpack" _must_ be deleted because
+# it cleans ${B} and will wipe any keys exported by do_get_public_keys.
+deltask do_fetch
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
-- 
2.12.3

[PATCH v4 4/8] sign_package_feed.bbclass: install signing key into rootfs

[Markus Lehtonen]

If package-management is enabled.

[YOCTO #11209]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/classes/sign_package_feed.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/sign_package_feed.bbclass b/meta/classes/sign_package_feed.bbclass
index 71df03bab3..f03c4802d0 100644
--- a/meta/classes/sign_package_feed.bbclass
+++ b/meta/classes/sign_package_feed.bbclass
@@ -28,6 +28,9 @@ PACKAGE_FEED_SIGN = '1'
 PACKAGE_FEED_GPG_BACKEND ?= 'local'
 PACKAGE_FEED_GPG_SIGNATURE_TYPE ?= 'ASC'
 
+# Make feed signing key to be present in rootfs
+FEATURE_PACKAGES_package-management_append = " signing-keys-packagefeed"
+
 python () {
     # Check sanity of configuration
     for var in ('PACKAGE_FEED_GPG_NAME', 'PACKAGE_FEED_GPG_PASSPHRASE_FILE'):
-- 
2.12.3

[PATCH v4 5/8] package_manager.py: enable dnf's repo_gpgcheck if feed signing is enabled

[Markus Lehtonen]

If package feed signing is enabled enable repo gpg signature check for
rpm repositories added via PACKAGE_FEED_URIS. This has the implication
that all repositories added via this mechanism must be signed with the
same key.

[YOCTO #11209]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oe/package_manager.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 942f2dd903..d43d729203 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -559,6 +559,12 @@ class RpmPM(PackageManager):
         if feed_uris == "":
             return
 
+        if self.d.getVar('PACKAGE_FEED_SIGN') == '1':
+            gpg_opts = 'repo_gpgcheck=1\n'
+            gpg_opts += 'gpgkey=file://%s/pki/packagefeed-gpg/PACKAGEFEED-GPG-KEY-%s\n' % (self.d.getVar('sysconfdir'), self.d.getVar('DISTRO_VERSION'))
+        else:
+            gpg_opts = ''
+
         bb.utils.mkdirhier(oe.path.join(self.target_rootfs, "etc", "yum.repos.d"))
         remote_uris = self.construct_uris(feed_uris.split(), feed_base_paths.split())
         for uri in remote_uris:
@@ -569,12 +575,12 @@ class RpmPM(PackageManager):
                     repo_id   = "oe-remote-repo"  + "-".join(urlparse(repo_uri).path.split("/"))
                     repo_name = "OE Remote Repo:" + " ".join(urlparse(repo_uri).path.split("/"))
                     open(oe.path.join(self.target_rootfs, "etc", "yum.repos.d", repo_base + ".repo"), 'a').write(
-                             "[%s]\nname=%s\nbaseurl=%s\n\n" % (repo_id, repo_name, repo_uri))
+                             "[%s]\nname=%s\nbaseurl=%s\n%s\n" % (repo_id, repo_name, repo_uri, gpg_opts))
             else:
                 repo_name = "OE Remote Repo:" + " ".join(urlparse(uri).path.split("/"))
                 repo_uri = uri
                 open(oe.path.join(self.target_rootfs, "etc", "yum.repos.d", repo_base + ".repo"), 'w').write(
-                             "[%s]\nname=%s\nbaseurl=%s\n" % (repo_base, repo_name, repo_uri))
+                             "[%s]\nname=%s\nbaseurl=%s\n%s" % (repo_base, repo_name, repo_uri, gpg_opts))
 
     def _prepare_pkg_transaction(self):
         os.environ['D'] = self.target_rootfs
-- 
2.12.3

[PATCH v4 6/8] oeqa: fix dnf tests

[Markus Lehtonen]

Rename one dnf runtime test that it will recognized as a python module
and thus also found by the oe test loader. Also, fix value of
TEST_SUITES in dnf selftest so that all test dependencies are satisfied
and the runtime test may be successfully run from there.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta-selftest/lib/oeqa/runtime/cases/{dnf-runtime.py => dnf_runtime.py} | 0
 meta/lib/oeqa/selftest/cases/runtime_test.py                            | 2 +-
 2 files changed, 1 insertion(+), 1 deletion(-)
 rename meta-selftest/lib/oeqa/runtime/cases/{dnf-runtime.py => dnf_runtime.py} (100%)

diff --git a/meta-selftest/lib/oeqa/runtime/cases/dnf-runtime.py b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
similarity index 100%
rename from meta-selftest/lib/oeqa/runtime/cases/dnf-runtime.py
rename to meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 72f906deea..27cf584ea5 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -138,7 +138,7 @@ class TestImage(OESelftestTestCase):
             self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
 
         features = 'INHERIT += "testimage"\n'
-        features += 'TEST_SUITES = "ping ssh dnf-runtime"\n'
+        features += 'TEST_SUITES = "ping ssh dnf_runtime dnf.DnfBasicTest.test_dnf_help"\n'
         # We don't yet know what the server ip and port will be - they will be patched
         # in at the start of the on-image test
         features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
-- 
2.12.3

[PATCH v4 7/8] oeqa: fix temp file handling in dnf package feed test

[Markus Lehtonen]

Prevent stale temp files and a possible (if unlikely) race in tempfile
usage.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
index 123e7259f1..68e56f2c5e 100644
--- a/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
+++ b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
@@ -6,6 +6,8 @@ class DnfSelftest(DnfTest):
 
     @classmethod
     def setUpClass(cls):
+        import tempfile
+        cls.temp_dir = tempfile.TemporaryDirectory(prefix="oeqa-remotefeeds-")
         cls.repo_server = HTTPService(os.path.join(cls.tc.td['WORKDIR'], 'oe-rootfs-repo'),
                                       cls.tc.target.server_ip)
         cls.repo_server.start()
@@ -13,6 +15,7 @@ class DnfSelftest(DnfTest):
     @classmethod
     def tearDownClass(cls):
         cls.repo_server.stop()
+        cls.temp_dir.cleanup()
 
     @OETestDepends(['dnf.DnfBasicTest.test_dnf_help'])
     def test_verify_package_feeds(self):
@@ -25,11 +28,11 @@ class DnfSelftest(DnfTest):
         """
         # When we created an image, we had to supply fake ip and port
         # for the feeds. Now we can patch the real ones into the config file.
-        import tempfile
-        temp_file = tempfile.TemporaryDirectory(prefix="oeqa-remotefeeds-").name
+        temp_file = os.path.join(self.temp_dir.name, 'tmp.repo')
         self.tc.target.copyFrom("/etc/yum.repos.d/oe-remote-repo.repo", temp_file)
         fixed_config = open(temp_file, "r").read().replace("bogus_ip", self.tc.target.server_ip).replace("bogus_port", str(self.repo_server.port))
-        open(temp_file, "w").write(fixed_config)
+        with open(temp_file, "w") as f:
+            f.write(fixed_config)
         self.tc.target.copyTo(temp_file, "/etc/yum.repos.d/oe-remote-repo.repo")
 
         import re
-- 
2.12.3

[PATCH v4 8/8] dnf: expand dnf selftest to test signed package feeds

[Markus Lehtonen]

[YOCTO #11209]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta-selftest/files/signing/key.passphrase          |  1 +
 meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py |  4 +++-
 meta/lib/oeqa/selftest/cases/runtime_test.py        | 12 +++++++++++-
 3 files changed, 15 insertions(+), 2 deletions(-)
 create mode 100644 meta-selftest/files/signing/key.passphrase

diff --git a/meta-selftest/files/signing/key.passphrase b/meta-selftest/files/signing/key.passphrase
new file mode 100644
index 0000000000..5271a52680
--- /dev/null
+++ b/meta-selftest/files/signing/key.passphrase
@@ -0,0 +1 @@
+test123
diff --git a/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
index 68e56f2c5e..6742e8c080 100644
--- a/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
+++ b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
@@ -36,7 +36,9 @@ class DnfSelftest(DnfTest):
         self.tc.target.copyTo(temp_file, "/etc/yum.repos.d/oe-remote-repo.repo")
 
         import re
-        output_makecache = self.dnf('makecache')
+        # Use '-y' for non-interactive mode: automatically import the feed signing key
+        output_makecache = self.dnf('-y makecache')
+        self.assertTrue(re.match(r".*Failed to synchronize cache", output_makecache, re.DOTALL) is None, msg = "dnf makecache failed to synchronize repo: %s" %(output_makecache))
         self.assertTrue(re.match(r".*Metadata cache created", output_makecache, re.DOTALL) is not None, msg = "dnf makecache failed: %s" %(output_makecache))
 
         output_repoinfo = self.dnf('repoinfo')
diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 27cf584ea5..673b1998ac 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -3,6 +3,7 @@ from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqem
 from oeqa.core.decorator.oeid import OETestID
 import os
 import re
+import tempfile
 
 class TestExport(OESelftestTestCase):
 
@@ -143,7 +144,16 @@ class TestImage(OESelftestTestCase):
         # in at the start of the on-image test
         features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
         features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
-        features += 'PACKAGE_CLASSES = "package_rpm"'
+        features += 'PACKAGE_CLASSES = "package_rpm"\n'
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.TemporaryDirectory(prefix="oeqa-feed-sign-")
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpg --batch --homedir %s --import %s' % (self.gpg_home.name, os.path.join(signing_key_dir, 'key.secret')))
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home.name
         self.write_config(features)
 
         # Build core-image-sato and testimage
-- 
2.12.3