[PATCH 0/3] Sign packages in RPM feeds

[PATCH 0/3] Sign packages in RPM feeds

[Markus Lehtonen]

Implement simple scheme of signing RPM packages and RPM package feeds locally
in the builder host. RPM package signing is implemented in a new bbclass. This
could be extended/replaced to enable more sophisticated schemes like using a
signing server.

[YOCTO #8134]

Markus Lehtonen (3):
  package_rpm: support signing of rpm packages
  os-release: add the public package-signing key
  package_manager: support for signed RPM package feeds

 meta/classes/package_rpm.bbclass           |  5 +++
 meta/classes/sign_rpm.bbclass              | 58 ++++++++++++++++++++++++++++++
 meta/lib/oe/package_manager.py             | 48 +++++++++++++++++++++++++
 meta/recipes-core/os-release/os-release.bb |  9 +++++
 4 files changed, 120 insertions(+)
 create mode 100644 meta/classes/sign_rpm.bbclass

-- 
2.1.4

[PATCH 1/3] package_rpm: support signing of rpm packages

[Markus Lehtonen]

This patch adds a new bbclass for generating rpm packages that are
signed with a user defined key. The packages are signed as part of the
"package_write_rpm" task.

In order to enable the feature you need to
1. 'INHERIT += " sign_rpm"' in bitbake config (e.g. local or
   distro)
2. Create a file that contains the passphrase to your gpg secret key
3. 'RPM_GPG_PASSPHRASE_FILE = "<path_to_file>" in bitbake config,
   pointing to the passphrase file created in 2.
4. Define GPG key name to use by either defining
   'RPM_GPG_NAME = "<key_id>" in bitbake config OR by defining
   %_gpg_name <key_id> in your ~/.oerpmmacros file
5. 'RPM_GPG_PUBKEY = "<path_to_pubkey>" in bitbake config pointing to
   the public key (in "armor" format)

The sign_rpm.bbclass implements a simple scenario of locally signing the
packages. It could be replaced by a more advanced class that would
utilize a separate signing server for signing the packages, for example.

[YOCTO #8134]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/classes/package_rpm.bbclass |  5 ++++
 meta/classes/sign_rpm.bbclass    | 58 ++++++++++++++++++++++++++++++++++++++++
 meta/lib/oe/package_manager.py   | 28 +++++++++++++++++++
 3 files changed, 91 insertions(+)
 create mode 100644 meta/classes/sign_rpm.bbclass

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 8fd0685..3e933ef 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -695,6 +695,8 @@ python do_package_rpm () {
     else:
         d.setVar('PACKAGE_ARCH_EXTEND', package_arch)
     pkgwritedir = d.expand('${PKGWRITEDIRRPM}/${PACKAGE_ARCH_EXTEND}')
+    d.setVar('RPM_PKGWRITEDIR', pkgwritedir)
+    bb.debug(1, 'PKGWRITEDIR: %s' % d.getVar('RPM_PKGWRITEDIR', True))
     pkgarch = d.expand('${PACKAGE_ARCH_EXTEND}${HOST_VENDOR}-${HOST_OS}')
     magicfile = d.expand('${STAGING_DIR_NATIVE}${datadir_native}/misc/magic.mgc')
     bb.utils.mkdirhier(pkgwritedir)
@@ -730,6 +732,9 @@ python do_package_rpm () {
     d.setVar('BUILDSPEC', cmd + "\n")
     d.setVarFlag('BUILDSPEC', 'func', '1')
     bb.build.exec_func('BUILDSPEC', d)
+
+    if d.getVar('RPM_SIGN_PACKAGES', True) == '1':
+        bb.build.exec_func("sign_rpm", d)
 }
 
 python () {
diff --git a/meta/classes/sign_rpm.bbclass b/meta/classes/sign_rpm.bbclass
new file mode 100644
index 0000000..ddf6c3b
--- /dev/null
+++ b/meta/classes/sign_rpm.bbclass
@@ -0,0 +1,58 @@
+inherit sanity
+
+RPM_SIGN_PACKAGES='1'
+
+
+_check_gpg_name () {
+    macrodef=`rpm -E '%_gpg_name'`
+    [ "$macrodef" == "%_gpg_name" ] && return 1 || return 0
+}
+
+
+def rpmsign_wrapper(d, files, passphrase, gpg_name=None):
+    import pexpect
+
+    # Find the correct rpm binary
+    rpm_bin_path = d.getVar('STAGING_BINDIR_NATIVE', True) + '/rpm'
+    cmd = rpm_bin_path + " --addsign "
+    if gpg_name:
+        cmd += "--define '%%_gpg_name %s' " % gpg_name
+    else:
+        try:
+            bb.build.exec_func('_check_gpg_name', d)
+        except bb.build.FuncFailed:
+            raise_sanity_error("You need to define RPM_GPG_NAME in bitbake "
+                               "config or the %_gpg_name RPM macro defined "
+                               "(e.g. in  ~/.oerpmmacros", d)
+    cmd += ' '.join(files)
+
+    # Need to use pexpect for feeding the passphrase
+    proc = pexpect.spawn(cmd)
+    try:
+        proc.expect_exact('Enter pass phrase:', timeout=15)
+        proc.sendline(passphrase)
+        proc.expect(pexpect.EOF, timeout=900)
+        proc.close()
+    except pexpect.TIMEOUT as err:
+        bb.debug('rpmsign timeout: %s' % err)
+        proc.terminate()
+    return proc.exitstatus
+
+
+python sign_rpm () {
+    import glob
+
+    rpm_gpg_pass_file = (d.getVar("RPM_GPG_PASSPHRASE_FILE", True) or "")
+    if rpm_gpg_pass_file:
+        with open(rpm_gpg_pass_file) as fobj:
+            rpm_gpg_passphrase = fobj.readlines()[0].rstrip('\n')
+    else:
+        raise_sanity_error("You need to define RPM_GPG_PASSPHRASE_FILE in the config", d)
+
+    rpm_gpg_name = (d.getVar("RPM_GPG_NAME", True) or "")
+
+    rpms = glob.glob(d.getVar('RPM_PKGWRITEDIR', True) + '/*')
+
+    if rpmsign_wrapper(d, rpms, rpm_gpg_passphrase, rpm_gpg_name) != 0:
+        raise bb.build.FuncFailed("RPM signing failed")
+}
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 2ab1d78..753b3eb 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -108,7 +108,14 @@ class RpmIndexer(Indexer):
         archs = archs.union(set(sdk_pkg_archs))
 
         rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
+        rpm_bin = bb.utils.which(os.getenv('PATH'), "rpm")
+        if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
+            rpm_pubkey = self.d.getVar('RPM_GPG_PUBKEY', True)
+        else:
+            rpm_pubkey = None
+
         index_cmds = []
+        key_import_cmds = []
         rpm_dirs_found = False
         for arch in archs:
             dbpath = os.path.join(self.d.getVar('WORKDIR', True), 'rpmdb', arch)
@@ -118,6 +125,9 @@ class RpmIndexer(Indexer):
             if not os.path.isdir(arch_dir):
                 continue
 
+            if rpm_pubkey:
+                key_import_cmds.append("%s --define '_dbpath %s' --import %s" %
+                                   (rpm_bin, dbpath, rpm_pubkey))
             index_cmds.append("%s --dbpath %s --update -q %s" % \
                              (rpm_createrepo, dbpath, arch_dir))
 
@@ -127,9 +137,18 @@ class RpmIndexer(Indexer):
             bb.note("There are no packages in %s" % self.deploy_dir)
             return
 
+        # Import GPG key to all temporary RPMDBs
+        result = oe.utils.multiprocess_exec(key_import_cmds, create_index)
+        if result:
+            bb.fatal('%s' % ('\n'.join(result)))
+        # Create repodata
         result = oe.utils.multiprocess_exec(index_cmds, create_index)
         if result:
             bb.fatal('%s' % ('\n'.join(result)))
+        # Copy pubkey to repo
+        if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
+            shutil.copy2(self.d.getVar('RPM_GPG_PUBKEY', True),
+                         os.path.join(self.deploy_dir, 'RPM-GPG-KEY-oe'))
 
 
 class OpkgIndexer(Indexer):
@@ -352,6 +371,9 @@ class RpmPkgsList(PkgsList):
             pkg = line.split()[0]
             arch = line.split()[1]
             ver = line.split()[2]
+            # Skip GPG keys
+            if pkg == 'gpg-pubkey':
+                continue
             if self.rpm_version == 4:
                 pkgorigin = "unknown"
             else:
@@ -864,6 +886,12 @@ class RpmPM(PackageManager):
         except subprocess.CalledProcessError as e:
             bb.fatal("Create rpm database failed. Command '%s' "
                      "returned %d:\n%s" % (cmd, e.returncode, e.output))
+        # Import GPG key to RPM database of the target system
+        if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
+            pubkey_path = self.d.getVar('RPM_GPG_PUBKEY', True)
+            cmd = "%s --root %s --dbpath /var/lib/rpm --import %s > /dev/null" % (
+                  self.rpm_cmd, self.target_rootfs, pubkey_path)
+            subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True)
 
         # Configure smart
         bb.note("configuring Smart settings")
-- 
2.1.4

[PATCH 2/3] os-release: add the public package-signing key

[Markus Lehtonen]

Adds the public package-signing key into this package. It will be
installed under /etc/pki/rpm-gpg if the RPM signing feature is used. The
key file is not currently directly used by anything in the target
system. It is merely there for possible later use.

[YOCTO #8134]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/recipes-core/os-release/os-release.bb | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/meta/recipes-core/os-release/os-release.bb b/meta/recipes-core/os-release/os-release.bb
index 87fea6f..542cf56 100644
--- a/meta/recipes-core/os-release/os-release.bb
+++ b/meta/recipes-core/os-release/os-release.bb
@@ -23,15 +23,24 @@ PRETTY_NAME = "${DISTRO_NAME} ${VERSION}"
 BUILD_ID ?= "${DATETIME}"
 
 python do_compile () {
+    import shutil
     with open(d.expand('${B}/os-release'), 'w') as f:
         for field in d.getVar('OS_RELEASE_FIELDS', True).split():
             value = d.getVar(field, True)
             if value:
                 f.write('{0}={1}\n'.format(field, value))
+    if d.getVar('RPM_SIGN_PACKAGES', True) == '1':
+        rpm_gpg_pubkey = d.getVar('RPM_GPG_PUBKEY', True)
+        shutil.copy2(rpm_gpg_pubkey, d.expand('${B}/RPM-GPG-KEY-default'))
 }
 do_compile[vardeps] += "${OS_RELEASE_FIELDS}"
 
 do_install () {
     install -d ${D}${sysconfdir}
     install -m 0644 os-release ${D}${sysconfdir}/
+
+    if [ -f "RPM-GPG-KEY-default" ]; then
+        install -d "${D}${sysconfdir}/pki/rpm-gpg"
+        install -m 0644 "RPM-GPG-KEY-oe" "${D}${sysconfdir}/pki/rpm-gpg/"
+    fi
 }
-- 
2.1.4

[PATCH 3/3] package_manager: support for signed RPM package feeds

[Markus Lehtonen]

This change makes it possible to create GPG signed RPM package feeds -
i.e. package feed with GPG signed metadata (repodata). All deployed RPM
repositories will be signed and the GPG public key is copied to the rpm
deployment directory.

In order to enable the new feature one needs to define four variables in
bitbake configuration.
1. 'PACKAGE_FEED_SIGN = "1"' enabling the feature
2. 'PACKAGE_FEED_GPG_NAME = "<key_id>"' defining the GPG key to use for
   signing
3. 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "<path_to_file>"' pointing to a
   file containing the passphrase for the secret signing key
4. 'PACKAGE_FEED_GPG_PUBKEY = "<path_to_pubkey>"' pointing to the
   corresponding public key (in "armor" format)

[YOCTO #8134]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oe/package_manager.py | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 753b3eb..5d7ef54 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -113,8 +113,15 @@ class RpmIndexer(Indexer):
             rpm_pubkey = self.d.getVar('RPM_GPG_PUBKEY', True)
         else:
             rpm_pubkey = None
+        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
+            pkgfeed_gpg_name = self.d.getVar('PACKAGE_FEED_GPG_NAME', True)
+            pkgfeed_gpg_pass = self.d.getVar('PACKAGE_FEED_GPG_PASSPHRASE_FILE', True)
+        else:
+            pkgfeed_gpg_name = None
+            pkgfeed_gpg_pass = None
 
         index_cmds = []
+        repo_sign_cmds = []
         key_import_cmds = []
         rpm_dirs_found = False
         for arch in archs:
@@ -126,10 +133,16 @@ class RpmIndexer(Indexer):
                 continue
 
             if rpm_pubkey:
-                key_import_cmds.append("%s --define '_dbpath %s' --import %s" %
+                key_import_cmds.append("%s --dbpath '%s' --import %s" %
                                    (rpm_bin, dbpath, rpm_pubkey))
             index_cmds.append("%s --dbpath %s --update -q %s" % \
                              (rpm_createrepo, dbpath, arch_dir))
+            if pkgfeed_gpg_name:
+                repomd_file = os.path.join(arch_dir, 'repodata', 'repomd.xml')
+                gpg_cmd = "gpg2 --detach-sign --armor --batch --no-tty --yes " \
+                          "--passphrase-file '%s' -u '%s' %s" % \
+                          (pkgfeed_gpg_pass, pkgfeed_gpg_name, repomd_file)
+                repo_sign_cmds.append(gpg_cmd)
 
             rpm_dirs_found = True
 
@@ -145,10 +158,17 @@ class RpmIndexer(Indexer):
         result = oe.utils.multiprocess_exec(index_cmds, create_index)
         if result:
             bb.fatal('%s' % ('\n'.join(result)))
-        # Copy pubkey to repo
+        # Sign repomd
+        result = oe.utils.multiprocess_exec(repo_sign_cmds, create_index)
+        if result:
+            bb.fatal('%s' % ('\n'.join(result)))
+        # Copy pubkey(s) to repo
         if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
             shutil.copy2(self.d.getVar('RPM_GPG_PUBKEY', True),
                          os.path.join(self.deploy_dir, 'RPM-GPG-KEY-oe'))
+        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
+            shutil.copy2(self.d.getVar('PACKAGE_FEED_GPG_PUBKEY', True),
+                         os.path.join(self.deploy_dir, 'REPODATA-GPG-KEY'))
 
 
 class OpkgIndexer(Indexer):
-- 
2.1.4

Re: [PATCH 1/3] package_rpm: support signing of rpm packages

[Mark Hatle]

On 8/26/15 6:18 AM, Markus Lehtonen wrote:
> This patch adds a new bbclass for generating rpm packages that are
> signed with a user defined key. The packages are signed as part of the
> "package_write_rpm" task.
> 
> In order to enable the feature you need to
> 1. 'INHERIT += " sign_rpm"' in bitbake config (e.g. local or
>    distro)
> 2. Create a file that contains the passphrase to your gpg secret key
> 3. 'RPM_GPG_PASSPHRASE_FILE = "<path_to_file>" in bitbake config,
>    pointing to the passphrase file created in 2.
> 4. Define GPG key name to use by either defining
>    'RPM_GPG_NAME = "<key_id>" in bitbake config OR by defining
>    %_gpg_name <key_id> in your ~/.oerpmmacros file
> 5. 'RPM_GPG_PUBKEY = "<path_to_pubkey>" in bitbake config pointing to
>    the public key (in "armor" format)
> 
> The sign_rpm.bbclass implements a simple scenario of locally signing the
> packages. It could be replaced by a more advanced class that would
> utilize a separate signing server for signing the packages, for example.
> 
> [YOCTO #8134]
> 
> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> ---
>  meta/classes/package_rpm.bbclass |  5 ++++
>  meta/classes/sign_rpm.bbclass    | 58 ++++++++++++++++++++++++++++++++++++++++
>  meta/lib/oe/package_manager.py   | 28 +++++++++++++++++++
>  3 files changed, 91 insertions(+)
>  create mode 100644 meta/classes/sign_rpm.bbclass
> 
> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
> index 8fd0685..3e933ef 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -695,6 +695,8 @@ python do_package_rpm () {
>      else:
>          d.setVar('PACKAGE_ARCH_EXTEND', package_arch)
>      pkgwritedir = d.expand('${PKGWRITEDIRRPM}/${PACKAGE_ARCH_EXTEND}')
> +    d.setVar('RPM_PKGWRITEDIR', pkgwritedir)
> +    bb.debug(1, 'PKGWRITEDIR: %s' % d.getVar('RPM_PKGWRITEDIR', True))
>      pkgarch = d.expand('${PACKAGE_ARCH_EXTEND}${HOST_VENDOR}-${HOST_OS}')
>      magicfile = d.expand('${STAGING_DIR_NATIVE}${datadir_native}/misc/magic.mgc')
>      bb.utils.mkdirhier(pkgwritedir)
> @@ -730,6 +732,9 @@ python do_package_rpm () {
>      d.setVar('BUILDSPEC', cmd + "\n")
>      d.setVarFlag('BUILDSPEC', 'func', '1')
>      bb.build.exec_func('BUILDSPEC', d)
> +
> +    if d.getVar('RPM_SIGN_PACKAGES', True) == '1':
> +        bb.build.exec_func("sign_rpm", d)
>  }
>  
>  python () {
> diff --git a/meta/classes/sign_rpm.bbclass b/meta/classes/sign_rpm.bbclass
> new file mode 100644
> index 0000000..ddf6c3b
> --- /dev/null
> +++ b/meta/classes/sign_rpm.bbclass
> @@ -0,0 +1,58 @@
> +inherit sanity
> +
> +RPM_SIGN_PACKAGES='1'
> +
> +
> +_check_gpg_name () {
> +    macrodef=`rpm -E '%_gpg_name'`
> +    [ "$macrodef" == "%_gpg_name" ] && return 1 || return 0
> +}
> +
> +
> +def rpmsign_wrapper(d, files, passphrase, gpg_name=None):
> +    import pexpect
> +
> +    # Find the correct rpm binary
> +    rpm_bin_path = d.getVar('STAGING_BINDIR_NATIVE', True) + '/rpm'
> +    cmd = rpm_bin_path + " --addsign "
> +    if gpg_name:
> +        cmd += "--define '%%_gpg_name %s' " % gpg_name
> +    else:
> +        try:
> +            bb.build.exec_func('_check_gpg_name', d)
> +        except bb.build.FuncFailed:
> +            raise_sanity_error("You need to define RPM_GPG_NAME in bitbake "
> +                               "config or the %_gpg_name RPM macro defined "
> +                               "(e.g. in  ~/.oerpmmacros", d)
> +    cmd += ' '.join(files)
> +
> +    # Need to use pexpect for feeding the passphrase
> +    proc = pexpect.spawn(cmd)
> +    try:
> +        proc.expect_exact('Enter pass phrase:', timeout=15)
> +        proc.sendline(passphrase)
> +        proc.expect(pexpect.EOF, timeout=900)
> +        proc.close()
> +    except pexpect.TIMEOUT as err:
> +        bb.debug('rpmsign timeout: %s' % err)
> +        proc.terminate()
> +    return proc.exitstatus
> +
> +
> +python sign_rpm () {
> +    import glob
> +
> +    rpm_gpg_pass_file = (d.getVar("RPM_GPG_PASSPHRASE_FILE", True) or "")
> +    if rpm_gpg_pass_file:
> +        with open(rpm_gpg_pass_file) as fobj:
> +            rpm_gpg_passphrase = fobj.readlines()[0].rstrip('\n')
> +    else:
> +        raise_sanity_error("You need to define RPM_GPG_PASSPHRASE_FILE in the config", d)
> +
> +    rpm_gpg_name = (d.getVar("RPM_GPG_NAME", True) or "")
> +
> +    rpms = glob.glob(d.getVar('RPM_PKGWRITEDIR', True) + '/*')
> +
> +    if rpmsign_wrapper(d, rpms, rpm_gpg_passphrase, rpm_gpg_name) != 0:
> +        raise bb.build.FuncFailed("RPM signing failed")
> +}
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index 2ab1d78..753b3eb 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -108,7 +108,14 @@ class RpmIndexer(Indexer):
>          archs = archs.union(set(sdk_pkg_archs))
>  
>          rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
> +        rpm_bin = bb.utils.which(os.getenv('PATH'), "rpm")
> +        if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
> +            rpm_pubkey = self.d.getVar('RPM_GPG_PUBKEY', True)
> +        else:
> +            rpm_pubkey = None
> +
>          index_cmds = []
> +        key_import_cmds = []
>          rpm_dirs_found = False
>          for arch in archs:
>              dbpath = os.path.join(self.d.getVar('WORKDIR', True), 'rpmdb', arch)
> @@ -118,6 +125,9 @@ class RpmIndexer(Indexer):
>              if not os.path.isdir(arch_dir):
>                  continue
>  
> +            if rpm_pubkey:
> +                key_import_cmds.append("%s --define '_dbpath %s' --import %s" %
> +                                   (rpm_bin, dbpath, rpm_pubkey))
>              index_cmds.append("%s --dbpath %s --update -q %s" % \
>                               (rpm_createrepo, dbpath, arch_dir))
>  
> @@ -127,9 +137,18 @@ class RpmIndexer(Indexer):
>              bb.note("There are no packages in %s" % self.deploy_dir)
>              return
>  
> +        # Import GPG key to all temporary RPMDBs
> +        result = oe.utils.multiprocess_exec(key_import_cmds, create_index)
> +        if result:
> +            bb.fatal('%s' % ('\n'.join(result)))
> +        # Create repodata
>          result = oe.utils.multiprocess_exec(index_cmds, create_index)
>          if result:
>              bb.fatal('%s' % ('\n'.join(result)))
> +        # Copy pubkey to repo
> +        if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
> +            shutil.copy2(self.d.getVar('RPM_GPG_PUBKEY', True),
> +                         os.path.join(self.deploy_dir, 'RPM-GPG-KEY-oe'))

Do you really need to do the above hunk?

createrepo can very easily be modified to ignore the key in a signed package
from being validated as the repository information is created.

The only reason why I think you'd want to do the above is simply to have
createrepo verify the (externally signed) package has not been
modified/corrupted before createrepo runs.  (Even if it has, the test would fail
for people using the package feed... so I think it's unlikely to be an issue.)

Without pasting the whole patch:

--- createrepo-0.4.11.orig/dumpMetadata.py
+++ createrepo-0.4.11/dumpMetadata.py
@@ -92,7 +92,7 @@ def returnHdr(ts, package):
-    ts.setVSFlags((rpm.RPMVSF_NOMD5|rpm.RPMVSF_NEEDPAYLOAD))
+
ts.setVSFlags((rpm.RPMVSF_NOMD5|rpm.RPMVSF_NEEDPAYLOAD|rpm.RPMVSF_NODSA|rpm.RPMVSF_NORSA|rpm.RPMVSF_NODSAHEADER|rpm.RPMVSF_NORSAHEADER))

I can send up this change if you think it's useful in this case (and would
eliminate these steps.)

(The reason I question the steps is purely because we've seen in the past these
temporary RPM databases seem to be fragile at times.  So anything we can do to
avoid that is probably good.)

>  class OpkgIndexer(Indexer):
> @@ -352,6 +371,9 @@ class RpmPkgsList(PkgsList):
>              pkg = line.split()[0]
>              arch = line.split()[1]
>              ver = line.split()[2]
> +            # Skip GPG keys
> +            if pkg == 'gpg-pubkey':
> +                continue
>              if self.rpm_version == 4:
>                  pkgorigin = "unknown"
>              else:
> @@ -864,6 +886,12 @@ class RpmPM(PackageManager):
>          except subprocess.CalledProcessError as e:
>              bb.fatal("Create rpm database failed. Command '%s' "
>                       "returned %d:\n%s" % (cmd, e.returncode, e.output))
> +        # Import GPG key to RPM database of the target system
> +        if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
> +            pubkey_path = self.d.getVar('RPM_GPG_PUBKEY', True)
> +            cmd = "%s --root %s --dbpath /var/lib/rpm --import %s > /dev/null" % (
> +                  self.rpm_cmd, self.target_rootfs, pubkey_path)
> +            subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True)
>  
>          # Configure smart
>          bb.note("configuring Smart settings")
> 

Re: [PATCH 3/3] package_manager: support for signed RPM package feeds

[Mark Hatle]

On 8/26/15 6:18 AM, Markus Lehtonen wrote:
> This change makes it possible to create GPG signed RPM package feeds -
> i.e. package feed with GPG signed metadata (repodata). All deployed RPM
> repositories will be signed and the GPG public key is copied to the rpm
> deployment directory.
> 
> In order to enable the new feature one needs to define four variables in
> bitbake configuration.
> 1. 'PACKAGE_FEED_SIGN = "1"' enabling the feature
> 2. 'PACKAGE_FEED_GPG_NAME = "<key_id>"' defining the GPG key to use for
>    signing
> 3. 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "<path_to_file>"' pointing to a
>    file containing the passphrase for the secret signing key
> 4. 'PACKAGE_FEED_GPG_PUBKEY = "<path_to_pubkey>"' pointing to the
>    corresponding public key (in "armor" format)
> 
> [YOCTO #8134]
> 
> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> ---
>  meta/lib/oe/package_manager.py | 24 ++++++++++++++++++++++--
>  1 file changed, 22 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index 753b3eb..5d7ef54 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -113,8 +113,15 @@ class RpmIndexer(Indexer):
>              rpm_pubkey = self.d.getVar('RPM_GPG_PUBKEY', True)
>          else:
>              rpm_pubkey = None
> +        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
> +            pkgfeed_gpg_name = self.d.getVar('PACKAGE_FEED_GPG_NAME', True)
> +            pkgfeed_gpg_pass = self.d.getVar('PACKAGE_FEED_GPG_PASSPHRASE_FILE', True)
> +        else:
> +            pkgfeed_gpg_name = None
> +            pkgfeed_gpg_pass = None
>  
>          index_cmds = []
> +        repo_sign_cmds = []
>          key_import_cmds = []
>          rpm_dirs_found = False
>          for arch in archs:
> @@ -126,10 +133,16 @@ class RpmIndexer(Indexer):
>                  continue
>  
>              if rpm_pubkey:
> -                key_import_cmds.append("%s --define '_dbpath %s' --import %s" %
> +                key_import_cmds.append("%s --dbpath '%s' --import %s" %
>                                     (rpm_bin, dbpath, rpm_pubkey))
>              index_cmds.append("%s --dbpath %s --update -q %s" % \
>                               (rpm_createrepo, dbpath, arch_dir))
> +            if pkgfeed_gpg_name:
> +                repomd_file = os.path.join(arch_dir, 'repodata', 'repomd.xml')
> +                gpg_cmd = "gpg2 --detach-sign --armor --batch --no-tty --yes " \
> +                          "--passphrase-file '%s' -u '%s' %s" % \
> +                          (pkgfeed_gpg_pass, pkgfeed_gpg_name, repomd_file)
> +                repo_sign_cmds.append(gpg_cmd)

I've had problems in the past hard coding 'gpg' or 'gpg2' as the name to use.

Can we get this to be dynamic.. even if it's a system level define for what
GPG/PGP program to use?

Also I'd forgotten about it until there.  RPM has a similar variable to define
the GPG program to use.  So using that variable (_signature) and defaulting to
the same item would be a good idea.

(One such reason to do this is to write a wrapper that uses an alternative
keychain for these keys....)

>  
>              rpm_dirs_found = True
>  
> @@ -145,10 +158,17 @@ class RpmIndexer(Indexer):
>          result = oe.utils.multiprocess_exec(index_cmds, create_index)
>          if result:
>              bb.fatal('%s' % ('\n'.join(result)))
> -        # Copy pubkey to repo
> +        # Sign repomd
> +        result = oe.utils.multiprocess_exec(repo_sign_cmds, create_index)
> +        if result:
> +            bb.fatal('%s' % ('\n'.join(result)))
> +        # Copy pubkey(s) to repo
>          if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
>              shutil.copy2(self.d.getVar('RPM_GPG_PUBKEY', True),
>                           os.path.join(self.deploy_dir, 'RPM-GPG-KEY-oe'))
> +        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
> +            shutil.copy2(self.d.getVar('PACKAGE_FEED_GPG_PUBKEY', True),
> +                         os.path.join(self.deploy_dir, 'REPODATA-GPG-KEY'))

I didn't notice this before..  but we shouldn't hardcode RPM-GPG-KEY-oe, it
should use a value such as 'DISTRO' to allow different distributions to have
non-conflicting keys.  The repository keys I would think would be similar as
well.. since you may have multiple repositories from different sources.  So
naming the key ending in -${DISTRO} might be a good idea there as well.
(Extending it to ${DISTRO_VERSION} might be make sense... since these will be
used for long-term upgradable systems.)

--Mark

>  
>  
>  class OpkgIndexer(Indexer):
> 

Re: [PATCH 1/3] package_rpm: support signing of rpm packages

[Markus Lehtonen]

Hi,

On 26/08/15 18:04, "Mark Hatle" <mark.hatle@windriver.com> wrote:

>On 8/26/15 6:18 AM, Markus Lehtonen wrote:
>> This patch adds a new bbclass for generating rpm packages that are
>> signed with a user defined key. The packages are signed as part of the
>> "package_write_rpm" task.
>> 
>> In order to enable the feature you need to
>> 1. 'INHERIT += " sign_rpm"' in bitbake config (e.g. local or
>>    distro)
>> 2. Create a file that contains the passphrase to your gpg secret key
>> 3. 'RPM_GPG_PASSPHRASE_FILE = "<path_to_file>" in bitbake config,
>>    pointing to the passphrase file created in 2.
>> 4. Define GPG key name to use by either defining
>>    'RPM_GPG_NAME = "<key_id>" in bitbake config OR by defining
>>    %_gpg_name <key_id> in your ~/.oerpmmacros file
>> 5. 'RPM_GPG_PUBKEY = "<path_to_pubkey>" in bitbake config pointing to
>>    the public key (in "armor" format)
>> 
>> The sign_rpm.bbclass implements a simple scenario of locally signing the
>> packages. It could be replaced by a more advanced class that would
>> utilize a separate signing server for signing the packages, for example.
>> 
>> [YOCTO #8134]
>> 
>> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
>> ---
>>  meta/classes/package_rpm.bbclass |  5 ++++
>>  meta/classes/sign_rpm.bbclass    | 58
>>++++++++++++++++++++++++++++++++++++++++
>>  meta/lib/oe/package_manager.py   | 28 +++++++++++++++++++
>>  3 files changed, 91 insertions(+)
>>  create mode 100644 meta/classes/sign_rpm.bbclass
>> 
>> diff --git a/meta/classes/package_rpm.bbclass
>>b/meta/classes/package_rpm.bbclass
>> index 8fd0685..3e933ef 100644
>> --- a/meta/classes/package_rpm.bbclass
>> +++ b/meta/classes/package_rpm.bbclass
>> @@ -695,6 +695,8 @@ python do_package_rpm () {
>>      else:
>>          d.setVar('PACKAGE_ARCH_EXTEND', package_arch)
>>      pkgwritedir = d.expand('${PKGWRITEDIRRPM}/${PACKAGE_ARCH_EXTEND}')
>> +    d.setVar('RPM_PKGWRITEDIR', pkgwritedir)
>> +    bb.debug(1, 'PKGWRITEDIR: %s' % d.getVar('RPM_PKGWRITEDIR', True))
>>      pkgarch = 
>>d.expand('${PACKAGE_ARCH_EXTEND}${HOST_VENDOR}-${HOST_OS}')
>>      magicfile =
>>d.expand('${STAGING_DIR_NATIVE}${datadir_native}/misc/magic.mgc')
>>      bb.utils.mkdirhier(pkgwritedir)
>> @@ -730,6 +732,9 @@ python do_package_rpm () {
>>      d.setVar('BUILDSPEC', cmd + "\n")
>>      d.setVarFlag('BUILDSPEC', 'func', '1')
>>      bb.build.exec_func('BUILDSPEC', d)
>> +
>> +    if d.getVar('RPM_SIGN_PACKAGES', True) == '1':
>> +        bb.build.exec_func("sign_rpm", d)
>>  }
>>  
>>  python () {
>> diff --git a/meta/classes/sign_rpm.bbclass
>>b/meta/classes/sign_rpm.bbclass
>> new file mode 100644
>> index 0000000..ddf6c3b
>> --- /dev/null
>> +++ b/meta/classes/sign_rpm.bbclass
>> @@ -0,0 +1,58 @@
>> +inherit sanity
>> +
>> +RPM_SIGN_PACKAGES='1'
>> +
>> +
>> +_check_gpg_name () {
>> +    macrodef=`rpm -E '%_gpg_name'`
>> +    [ "$macrodef" == "%_gpg_name" ] && return 1 || return 0
>> +}
>> +
>> +
>> +def rpmsign_wrapper(d, files, passphrase, gpg_name=None):
>> +    import pexpect
>> +
>> +    # Find the correct rpm binary
>> +    rpm_bin_path = d.getVar('STAGING_BINDIR_NATIVE', True) + '/rpm'
>> +    cmd = rpm_bin_path + " --addsign "
>> +    if gpg_name:
>> +        cmd += "--define '%%_gpg_name %s' " % gpg_name
>> +    else:
>> +        try:
>> +            bb.build.exec_func('_check_gpg_name', d)
>> +        except bb.build.FuncFailed:
>> +            raise_sanity_error("You need to define RPM_GPG_NAME in
>>bitbake "
>> +                               "config or the %_gpg_name RPM macro
>>defined "
>> +                               "(e.g. in  ~/.oerpmmacros", d)
>> +    cmd += ' '.join(files)
>> +
>> +    # Need to use pexpect for feeding the passphrase
>> +    proc = pexpect.spawn(cmd)
>> +    try:
>> +        proc.expect_exact('Enter pass phrase:', timeout=15)
>> +        proc.sendline(passphrase)
>> +        proc.expect(pexpect.EOF, timeout=900)
>> +        proc.close()
>> +    except pexpect.TIMEOUT as err:
>> +        bb.debug('rpmsign timeout: %s' % err)
>> +        proc.terminate()
>> +    return proc.exitstatus
>> +
>> +
>> +python sign_rpm () {
>> +    import glob
>> +
>> +    rpm_gpg_pass_file = (d.getVar("RPM_GPG_PASSPHRASE_FILE", True) or
>>"")
>> +    if rpm_gpg_pass_file:
>> +        with open(rpm_gpg_pass_file) as fobj:
>> +            rpm_gpg_passphrase = fobj.readlines()[0].rstrip('\n')
>> +    else:
>> +        raise_sanity_error("You need to define RPM_GPG_PASSPHRASE_FILE
>>in the config", d)
>> +
>> +    rpm_gpg_name = (d.getVar("RPM_GPG_NAME", True) or "")
>> +
>> +    rpms = glob.glob(d.getVar('RPM_PKGWRITEDIR', True) + '/*')
>> +
>> +    if rpmsign_wrapper(d, rpms, rpm_gpg_passphrase, rpm_gpg_name) != 0:
>> +        raise bb.build.FuncFailed("RPM signing failed")
>> +}
>> diff --git a/meta/lib/oe/package_manager.py
>>b/meta/lib/oe/package_manager.py
>> index 2ab1d78..753b3eb 100644
>> --- a/meta/lib/oe/package_manager.py
>> +++ b/meta/lib/oe/package_manager.py
>> @@ -108,7 +108,14 @@ class RpmIndexer(Indexer):
>>          archs = archs.union(set(sdk_pkg_archs))
>>  
>>          rpm_createrepo = bb.utils.which(os.getenv('PATH'),
>>"createrepo")
>> +        rpm_bin = bb.utils.which(os.getenv('PATH'), "rpm")
>> +        if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
>> +            rpm_pubkey = self.d.getVar('RPM_GPG_PUBKEY', True)
>> +        else:
>> +            rpm_pubkey = None
>> +
>>          index_cmds = []
>> +        key_import_cmds = []
>>          rpm_dirs_found = False
>>          for arch in archs:
>>              dbpath = os.path.join(self.d.getVar('WORKDIR', True),
>>'rpmdb', arch)
>> @@ -118,6 +125,9 @@ class RpmIndexer(Indexer):
>>              if not os.path.isdir(arch_dir):
>>                  continue
>>  
>> +            if rpm_pubkey:
>> +                key_import_cmds.append("%s --define '_dbpath %s'
>>--import %s" %
>> +                                   (rpm_bin, dbpath, rpm_pubkey))
>>              index_cmds.append("%s --dbpath %s --update -q %s" % \
>>                               (rpm_createrepo, dbpath, arch_dir))
>>  
>> @@ -127,9 +137,18 @@ class RpmIndexer(Indexer):
>>              bb.note("There are no packages in %s" % self.deploy_dir)
>>              return
>>  
>> +        # Import GPG key to all temporary RPMDBs
>> +        result = oe.utils.multiprocess_exec(key_import_cmds,
>>create_index)
>> +        if result:
>> +            bb.fatal('%s' % ('\n'.join(result)))
>> +        # Create repodata
>>          result = oe.utils.multiprocess_exec(index_cmds, create_index)
>>          if result:
>>              bb.fatal('%s' % ('\n'.join(result)))
>> +        # Copy pubkey to repo
>> +        if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
>> +            shutil.copy2(self.d.getVar('RPM_GPG_PUBKEY', True),
>> +                         os.path.join(self.deploy_dir,
>>'RPM-GPG-KEY-oe'))
>
>Do you really need to do the above hunk?
>
>createrepo can very easily be modified to ignore the key in a signed
>package
>from being validated as the repository information is created.
>
>The only reason why I think you'd want to do the above is simply to have
>createrepo verify the (externally signed) package has not been
>modified/corrupted before createrepo runs.  (Even if it has, the test
>would fail
>for people using the package feed... so I think it's unlikely to be an
>issue.)

Yes, I was thinking this as a verification step. But, you're right, the
signature is checked in any case when the package is used (i.e. when
creating an image or installing packages from the feed). Thus, I think
you're suggestion is preferable.


>Without pasting the whole patch:
>
>--- createrepo-0.4.11.orig/dumpMetadata.py
>+++ createrepo-0.4.11/dumpMetadata.py
>@@ -92,7 +92,7 @@ def returnHdr(ts, package):
>-    ts.setVSFlags((rpm.RPMVSF_NOMD5|rpm.RPMVSF_NEEDPAYLOAD))
>+
>ts.setVSFlags((rpm.RPMVSF_NOMD5|rpm.RPMVSF_NEEDPAYLOAD|rpm.RPMVSF_NODSA|rp
>m.RPMVSF_NORSA|rpm.RPMVSF_NODSAHEADER|rpm.RPMVSF_NORSAHEADER))
>
>I can send up this change if you think it's useful in this case (and would
>eliminate these steps.)

I can introduce a separate patch in the next version of this patchset.




>
>(The reason I question the steps is purely because we've seen in the past
>these
>temporary RPM databases seem to be fragile at times.  So anything we can
>do to
>avoid that is probably good.)

Yes.


Thanks,
  Markus

Re: [PATCH 3/3] package_manager: support for signed RPM package feeds

[Markus Lehtonen]

Hi Mark,

On 26/08/15 18:10, "Mark Hatle" <mark.hatle@windriver.com> wrote:

>On 8/26/15 6:18 AM, Markus Lehtonen wrote:
>> This change makes it possible to create GPG signed RPM package feeds -
>> i.e. package feed with GPG signed metadata (repodata). All deployed RPM
>> repositories will be signed and the GPG public key is copied to the rpm
>> deployment directory.
>> 
>> In order to enable the new feature one needs to define four variables in
>> bitbake configuration.
>> 1. 'PACKAGE_FEED_SIGN = "1"' enabling the feature
>> 2. 'PACKAGE_FEED_GPG_NAME = "<key_id>"' defining the GPG key to use for
>>    signing
>> 3. 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "<path_to_file>"' pointing to a
>>    file containing the passphrase for the secret signing key
>> 4. 'PACKAGE_FEED_GPG_PUBKEY = "<path_to_pubkey>"' pointing to the
>>    corresponding public key (in "armor" format)
>> 
>> [YOCTO #8134]
>> 
>> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
>> ---
>>  meta/lib/oe/package_manager.py | 24 ++++++++++++++++++++++--
>>  1 file changed, 22 insertions(+), 2 deletions(-)
>> 
>> diff --git a/meta/lib/oe/package_manager.py
>>b/meta/lib/oe/package_manager.py
>> index 753b3eb..5d7ef54 100644
>> --- a/meta/lib/oe/package_manager.py
>> +++ b/meta/lib/oe/package_manager.py
>> @@ -113,8 +113,15 @@ class RpmIndexer(Indexer):
>>              rpm_pubkey = self.d.getVar('RPM_GPG_PUBKEY', True)
>>          else:
>>              rpm_pubkey = None
>> +        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
>> +            pkgfeed_gpg_name = self.d.getVar('PACKAGE_FEED_GPG_NAME',
>>True)
>> +            pkgfeed_gpg_pass =
>>self.d.getVar('PACKAGE_FEED_GPG_PASSPHRASE_FILE', True)
>> +        else:
>> +            pkgfeed_gpg_name = None
>> +            pkgfeed_gpg_pass = None
>>  
>>          index_cmds = []
>> +        repo_sign_cmds = []
>>          key_import_cmds = []
>>          rpm_dirs_found = False
>>          for arch in archs:
>> @@ -126,10 +133,16 @@ class RpmIndexer(Indexer):
>>                  continue
>>  
>>              if rpm_pubkey:
>> -                key_import_cmds.append("%s --define '_dbpath %s'
>>--import %s" %
>> +                key_import_cmds.append("%s --dbpath '%s' --import %s" %
>>                                     (rpm_bin, dbpath, rpm_pubkey))
>>              index_cmds.append("%s --dbpath %s --update -q %s" % \
>>                               (rpm_createrepo, dbpath, arch_dir))
>> +            if pkgfeed_gpg_name:
>> +                repomd_file = os.path.join(arch_dir, 'repodata',
>>'repomd.xml')
>> +                gpg_cmd = "gpg2 --detach-sign --armor --batch --no-tty
>>--yes " \
>> +                          "--passphrase-file '%s' -u '%s' %s" % \
>> +                          (pkgfeed_gpg_pass, pkgfeed_gpg_name,
>>repomd_file)
>> +                repo_sign_cmds.append(gpg_cmd)
>
>I've had problems in the past hard coding 'gpg' or 'gpg2' as the name to
>use.
>
>Can we get this to be dynamic.. even if it's a system level define for
>what
>GPG/PGP program to use?

OK, I can introduce a new variable for defining this.


>Also I'd forgotten about it until there.  RPM has a similar variable to
>define
>the GPG program to use.  So using that variable (_signature) and
>defaulting to
>the same item would be a good idea.

I think this is not feasible as we're actually using the host's gpg(2)
here and rpm might not even be available.


Thanks,
   Markus



>(One such reason to do this is to write a wrapper that uses an alternative
>keychain for these keys....)
>
>>  
>>              rpm_dirs_found = True
>>  
>> @@ -145,10 +158,17 @@ class RpmIndexer(Indexer):
>>          result = oe.utils.multiprocess_exec(index_cmds, create_index)
>>          if result:
>>              bb.fatal('%s' % ('\n'.join(result)))
>> -        # Copy pubkey to repo
>> +        # Sign repomd
>> +        result = oe.utils.multiprocess_exec(repo_sign_cmds,
>>create_index)
>> +        if result:
>> +            bb.fatal('%s' % ('\n'.join(result)))
>> +        # Copy pubkey(s) to repo
>>          if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
>>              shutil.copy2(self.d.getVar('RPM_GPG_PUBKEY', True),
>>                           os.path.join(self.deploy_dir,
>>'RPM-GPG-KEY-oe'))
>> +        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
>> +            shutil.copy2(self.d.getVar('PACKAGE_FEED_GPG_PUBKEY',
>>True),
>> +                         os.path.join(self.deploy_dir,
>>'REPODATA-GPG-KEY'))
>
>I didn't notice this before..  but we shouldn't hardcode RPM-GPG-KEY-oe,
>it
>should use a value such as 'DISTRO' to allow different distributions to
>have
>non-conflicting keys.  The repository keys I would think would be similar
>as
>well.. since you may have multiple repositories from different sources.
>So
>naming the key ending in -${DISTRO} might be a good idea there as well.
>(Extending it to ${DISTRO_VERSION} might be make sense... since these
>will be
>used for long-term upgradable systems.)
>
>--Mark
>
>>  
>>  
>>  class OpkgIndexer(Indexer):
>> 
>

Re: [PATCH 1/3] package_rpm: support signing of rpm packages

[Mark Hatle]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

On 8/26/15 10:11 PM, Markus Lehtonen wrote:
>> Without pasting the whole patch:
>>
>> --- createrepo-0.4.11.orig/dumpMetadata.py
>> +++ createrepo-0.4.11/dumpMetadata.py
>> @@ -92,7 +92,7 @@ def returnHdr(ts, package):
>> -    ts.setVSFlags((rpm.RPMVSF_NOMD5|rpm.RPMVSF_NEEDPAYLOAD))
>> +
>> ts.setVSFlags((rpm.RPMVSF_NOMD5|rpm.RPMVSF_NEEDPAYLOAD|rpm.RPMVSF_NODSA|rp
>> m.RPMVSF_NORSA|rpm.RPMVSF_NODSAHEADER|rpm.RPMVSF_NORSAHEADER))
>>
>> I can send up this change if you think it's useful in this case (and would
>> eliminate these steps.)
> 
> I can introduce a separate patch in the next version of this patchset.
> 

The patch we use is attached.  (it's just a patch, not a proper git commit..)

> 
> 
>>
>> (The reason I question the steps is purely because we've seen in the past
>> these
>> temporary RPM databases seem to be fragile at times.  So anything we can
>> do to
>> avoid that is probably good.)
> 
> Yes.
> 
> 
> Thanks,
>   Markus
> 
> 


[-- Attachment #2: createrepo-skipsigned.patch --]
[-- Type: text/plain, Size: 1149 bytes --]

createrepo: Disable GPG signature validation

If the packages are signed, and the signature is not in the rpm-native
RPMDB, an error will occur.  We want to avoid this failure mode 
when building the native version of createrepo.

This only affects the feed generation and will not change any later
validations during use/install from the package feed.

Upstream-status: Inappropriate [no longer maintained version]

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>

Index: createrepo-0.4.11/dumpMetadata.py
===================================================================
--- createrepo-0.4.11.orig/dumpMetadata.py
+++ createrepo-0.4.11/dumpMetadata.py
@@ -92,7 +92,7 @@ def returnHdr(ts, package):
             fdno = package # let's assume this is an fdno and go with it :)
     except OSError:
         raise MDError, "Error opening file"
-    ts.setVSFlags((rpm.RPMVSF_NOMD5|rpm.RPMVSF_NEEDPAYLOAD))
+    ts.setVSFlags((rpm.RPMVSF_NOMD5|rpm.RPMVSF_NEEDPAYLOAD|rpm.RPMVSF_NODSA|rpm.RPMVSF_NORSA|rpm.RPMVSF_NODSAHEADER|rpm.RPMVSF_NORSAHEADER))
     try:
         hdr = ts.hdrFromFdno(fdno)
     except rpm.error:

Re: [PATCH 3/3] package_manager: support for signed RPM package feeds

[Mark Hatle]

On 8/26/15 11:27 PM, Markus Lehtonen wrote:
> Hi Mark,
> 
> On 26/08/15 18:10, "Mark Hatle" <mark.hatle@windriver.com> wrote:
> 
>> On 8/26/15 6:18 AM, Markus Lehtonen wrote:
>>> This change makes it possible to create GPG signed RPM package feeds -
>>> i.e. package feed with GPG signed metadata (repodata). All deployed RPM
>>> repositories will be signed and the GPG public key is copied to the rpm
>>> deployment directory.
>>>
>>> In order to enable the new feature one needs to define four variables in
>>> bitbake configuration.
>>> 1. 'PACKAGE_FEED_SIGN = "1"' enabling the feature
>>> 2. 'PACKAGE_FEED_GPG_NAME = "<key_id>"' defining the GPG key to use for
>>>    signing
>>> 3. 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "<path_to_file>"' pointing to a
>>>    file containing the passphrase for the secret signing key
>>> 4. 'PACKAGE_FEED_GPG_PUBKEY = "<path_to_pubkey>"' pointing to the
>>>    corresponding public key (in "armor" format)
>>>
>>> [YOCTO #8134]
>>>
>>> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
>>> ---
>>>  meta/lib/oe/package_manager.py | 24 ++++++++++++++++++++++--
>>>  1 file changed, 22 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/meta/lib/oe/package_manager.py
>>> b/meta/lib/oe/package_manager.py
>>> index 753b3eb..5d7ef54 100644
>>> --- a/meta/lib/oe/package_manager.py
>>> +++ b/meta/lib/oe/package_manager.py
>>> @@ -113,8 +113,15 @@ class RpmIndexer(Indexer):
>>>              rpm_pubkey = self.d.getVar('RPM_GPG_PUBKEY', True)
>>>          else:
>>>              rpm_pubkey = None
>>> +        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
>>> +            pkgfeed_gpg_name = self.d.getVar('PACKAGE_FEED_GPG_NAME',
>>> True)
>>> +            pkgfeed_gpg_pass =
>>> self.d.getVar('PACKAGE_FEED_GPG_PASSPHRASE_FILE', True)
>>> +        else:
>>> +            pkgfeed_gpg_name = None
>>> +            pkgfeed_gpg_pass = None
>>>  
>>>          index_cmds = []
>>> +        repo_sign_cmds = []
>>>          key_import_cmds = []
>>>          rpm_dirs_found = False
>>>          for arch in archs:
>>> @@ -126,10 +133,16 @@ class RpmIndexer(Indexer):
>>>                  continue
>>>  
>>>              if rpm_pubkey:
>>> -                key_import_cmds.append("%s --define '_dbpath %s'
>>> --import %s" %
>>> +                key_import_cmds.append("%s --dbpath '%s' --import %s" %
>>>                                     (rpm_bin, dbpath, rpm_pubkey))
>>>              index_cmds.append("%s --dbpath %s --update -q %s" % \
>>>                               (rpm_createrepo, dbpath, arch_dir))
>>> +            if pkgfeed_gpg_name:
>>> +                repomd_file = os.path.join(arch_dir, 'repodata',
>>> 'repomd.xml')
>>> +                gpg_cmd = "gpg2 --detach-sign --armor --batch --no-tty
>>> --yes " \
>>> +                          "--passphrase-file '%s' -u '%s' %s" % \
>>> +                          (pkgfeed_gpg_pass, pkgfeed_gpg_name,
>>> repomd_file)
>>> +                repo_sign_cmds.append(gpg_cmd)
>>
>> I've had problems in the past hard coding 'gpg' or 'gpg2' as the name to
>> use.
>>
>> Can we get this to be dynamic.. even if it's a system level define for
>> what
>> GPG/PGP program to use?
> 
> OK, I can introduce a new variable for defining this.
> 
> 
>> Also I'd forgotten about it until there.  RPM has a similar variable to
>> define
>> the GPG program to use.  So using that variable (_signature) and
>> defaulting to
>> the same item would be a good idea.
> 
> I think this is not feasible as we're actually using the host's gpg(2)
> here and rpm might not even be available.

Sorry I listed the wrong variable..  What I was referring to was the gpg
program.  See below..

What I'm asking for is similar to the above of replacing:

gpg_cmd = "gpg2 --detach-sign --armor --batch --no-tty --yes "

with something like:

gpg_cmd = d.getVar("GPG", True) + "--detach-sign --armor --batch --no-tty --yes "

In the sections where you setup the RPM macros you would define signature in the
same way:

(patch 1/3)

    if gpg_name:
        cmd += "--define '%%_gpg_name %s' " % gpg_name

cmd += "--define '__gpg %s' --define '%%_gpg_name %s' " % (d.getVar("GPG",
True), gpg_name)

--Mark

> 
> Thanks,
>    Markus
> 
> 
> 
>> (One such reason to do this is to write a wrapper that uses an alternative
>> keychain for these keys....)
>>
>>>  
>>>              rpm_dirs_found = True
>>>  
>>> @@ -145,10 +158,17 @@ class RpmIndexer(Indexer):
>>>          result = oe.utils.multiprocess_exec(index_cmds, create_index)
>>>          if result:
>>>              bb.fatal('%s' % ('\n'.join(result)))
>>> -        # Copy pubkey to repo
>>> +        # Sign repomd
>>> +        result = oe.utils.multiprocess_exec(repo_sign_cmds,
>>> create_index)
>>> +        if result:
>>> +            bb.fatal('%s' % ('\n'.join(result)))
>>> +        # Copy pubkey(s) to repo
>>>          if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
>>>              shutil.copy2(self.d.getVar('RPM_GPG_PUBKEY', True),
>>>                           os.path.join(self.deploy_dir,
>>> 'RPM-GPG-KEY-oe'))
>>> +        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
>>> +            shutil.copy2(self.d.getVar('PACKAGE_FEED_GPG_PUBKEY',
>>> True),
>>> +                         os.path.join(self.deploy_dir,
>>> 'REPODATA-GPG-KEY'))
>>
>> I didn't notice this before..  but we shouldn't hardcode RPM-GPG-KEY-oe,
>> it
>> should use a value such as 'DISTRO' to allow different distributions to
>> have
>> non-conflicting keys.  The repository keys I would think would be similar
>> as
>> well.. since you may have multiple repositories from different sources.
>> So
>> naming the key ending in -${DISTRO} might be a good idea there as well.
>> (Extending it to ${DISTRO_VERSION} might be make sense... since these
>> will be
>> used for long-term upgradable systems.)
>>
>> --Mark
>>
>>>  
>>>  
>>>  class OpkgIndexer(Indexer):
>>>
>>
> 
> 

Re: [PATCH 3/3] package_manager: support for signed RPM package feeds

[Markus Lehtonen]

Hi,

On 27/08/15 15:03, "Mark Hatle" <mark.hatle@windriver.com> wrote:

>On 8/26/15 11:27 PM, Markus Lehtonen wrote:
>> Hi Mark,
>> 
>> On 26/08/15 18:10, "Mark Hatle" <mark.hatle@windriver.com> wrote:
>> 
>>> On 8/26/15 6:18 AM, Markus Lehtonen wrote:
>>>> This change makes it possible to create GPG signed RPM package feeds -
>>>> i.e. package feed with GPG signed metadata (repodata). All deployed
>>>>RPM
>>>> repositories will be signed and the GPG public key is copied to the
>>>>rpm
>>>> deployment directory.
>>>>
>>>> In order to enable the new feature one needs to define four variables
>>>>in
>>>> bitbake configuration.
>>>> 1. 'PACKAGE_FEED_SIGN = "1"' enabling the feature
>>>> 2. 'PACKAGE_FEED_GPG_NAME = "<key_id>"' defining the GPG key to use
>>>>for
>>>>    signing
>>>> 3. 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "<path_to_file>"' pointing to a
>>>>    file containing the passphrase for the secret signing key
>>>> 4. 'PACKAGE_FEED_GPG_PUBKEY = "<path_to_pubkey>"' pointing to the
>>>>    corresponding public key (in "armor" format)
>>>>
>>>> [YOCTO #8134]
>>>>
>>>> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
>>>> ---
>>>>  meta/lib/oe/package_manager.py | 24 ++++++++++++++++++++++--
>>>>  1 file changed, 22 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/meta/lib/oe/package_manager.py
>>>> b/meta/lib/oe/package_manager.py
>>>> index 753b3eb..5d7ef54 100644
>>>> --- a/meta/lib/oe/package_manager.py
>>>> +++ b/meta/lib/oe/package_manager.py
>>>> @@ -113,8 +113,15 @@ class RpmIndexer(Indexer):
>>>>              rpm_pubkey = self.d.getVar('RPM_GPG_PUBKEY', True)
>>>>          else:
>>>>              rpm_pubkey = None
>>>> +        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
>>>> +            pkgfeed_gpg_name = self.d.getVar('PACKAGE_FEED_GPG_NAME',
>>>> True)
>>>> +            pkgfeed_gpg_pass =
>>>> self.d.getVar('PACKAGE_FEED_GPG_PASSPHRASE_FILE', True)
>>>> +        else:
>>>> +            pkgfeed_gpg_name = None
>>>> +            pkgfeed_gpg_pass = None
>>>>  
>>>>          index_cmds = []
>>>> +        repo_sign_cmds = []
>>>>          key_import_cmds = []
>>>>          rpm_dirs_found = False
>>>>          for arch in archs:
>>>> @@ -126,10 +133,16 @@ class RpmIndexer(Indexer):
>>>>                  continue
>>>>  
>>>>              if rpm_pubkey:
>>>> -                key_import_cmds.append("%s --define '_dbpath %s'
>>>> --import %s" %
>>>> +                key_import_cmds.append("%s --dbpath '%s' --import
>>>>%s" %
>>>>                                     (rpm_bin, dbpath, rpm_pubkey))
>>>>              index_cmds.append("%s --dbpath %s --update -q %s" % \
>>>>                               (rpm_createrepo, dbpath, arch_dir))
>>>> +            if pkgfeed_gpg_name:
>>>> +                repomd_file = os.path.join(arch_dir, 'repodata',
>>>> 'repomd.xml')
>>>> +                gpg_cmd = "gpg2 --detach-sign --armor --batch
>>>>--no-tty
>>>> --yes " \
>>>> +                          "--passphrase-file '%s' -u '%s' %s" % \
>>>> +                          (pkgfeed_gpg_pass, pkgfeed_gpg_name,
>>>> repomd_file)
>>>> +                repo_sign_cmds.append(gpg_cmd)
>>>
>>> I've had problems in the past hard coding 'gpg' or 'gpg2' as the name
>>>to
>>> use.
>>>
>>> Can we get this to be dynamic.. even if it's a system level define for
>>> what
>>> GPG/PGP program to use?
>> 
>> OK, I can introduce a new variable for defining this.
>> 
>> 
>>> Also I'd forgotten about it until there.  RPM has a similar variable to
>>> define
>>> the GPG program to use.  So using that variable (_signature) and
>>> defaulting to
>>> the same item would be a good idea.
>> 
>> I think this is not feasible as we're actually using the host's gpg(2)
>> here and rpm might not even be available.
>
>Sorry I listed the wrong variable..  What I was referring to was the gpg
>program.  See below..
>
>What I'm asking for is similar to the above of replacing:
>
>gpg_cmd = "gpg2 --detach-sign --armor --batch --no-tty --yes "
>
>with something like:
>
>gpg_cmd = d.getVar("GPG", True) + "--detach-sign --armor --batch --no-tty
>--yes "
>
>In the sections where you setup the RPM macros you would define signature
>in the
>same way:
>
>(patch 1/3)
>
>    if gpg_name:
>        cmd += "--define '%%_gpg_name %s' " % gpg_name
>
>cmd += "--define '__gpg %s' --define '%%_gpg_name %s' " % (d.getVar("GPG",
>True), gpg_name)

I got the point and did something along these lines in my v2 patchset.
Although the variable name I used was GPG_BIN.

Thanks for your comments,
   Markus




>--Mark
>
>> 
>> Thanks,
>>    Markus
>> 
>> 
>> 
>>> (One such reason to do this is to write a wrapper that uses an
>>>alternative
>>> keychain for these keys....)
>>>
>>>>  
>>>>              rpm_dirs_found = True
>>>>  
>>>> @@ -145,10 +158,17 @@ class RpmIndexer(Indexer):
>>>>          result = oe.utils.multiprocess_exec(index_cmds, create_index)
>>>>          if result:
>>>>              bb.fatal('%s' % ('\n'.join(result)))
>>>> -        # Copy pubkey to repo
>>>> +        # Sign repomd
>>>> +        result = oe.utils.multiprocess_exec(repo_sign_cmds,
>>>> create_index)
>>>> +        if result:
>>>> +            bb.fatal('%s' % ('\n'.join(result)))
>>>> +        # Copy pubkey(s) to repo
>>>>          if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
>>>>              shutil.copy2(self.d.getVar('RPM_GPG_PUBKEY', True),
>>>>                           os.path.join(self.deploy_dir,
>>>> 'RPM-GPG-KEY-oe'))
>>>> +        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
>>>> +            shutil.copy2(self.d.getVar('PACKAGE_FEED_GPG_PUBKEY',
>>>> True),
>>>> +                         os.path.join(self.deploy_dir,
>>>> 'REPODATA-GPG-KEY'))
>>>
>>> I didn't notice this before..  but we shouldn't hardcode
>>>RPM-GPG-KEY-oe,
>>> it
>>> should use a value such as 'DISTRO' to allow different distributions to
>>> have
>>> non-conflicting keys.  The repository keys I would think would be
>>>similar
>>> as
>>> well.. since you may have multiple repositories from different sources.
>>> So
>>> naming the key ending in -${DISTRO} might be a good idea there as well.
>>> (Extending it to ${DISTRO_VERSION} might be make sense... since these
>>> will be
>>> used for long-term upgradable systems.)
>>>
>>> --Mark
>>>
>>>>  
>>>>  
>>>>  class OpkgIndexer(Indexer):
>>>>
>>>
>> 
>> 
>