[PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Ferry Toth]

From: Xavier Berger <xavier.berger@bio-logic.net>

output_suffix: If defined, add output_suffix as file name extension.
use_sha256: If True, use sha256 for gpg as digest algorithm

Signed-off-by: Xavier Berger <xavier.berger@bio-logic.net>
Signed-off-by: Ferry Toth <ftoth@exalondelft.nl>
---
 meta/lib/oe/gpg_sign.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py
index 1bce6cb792..aa9bb49f2c 100644
--- a/meta/lib/oe/gpg_sign.py
+++ b/meta/lib/oe/gpg_sign.py
@@ -58,7 +58,7 @@ class LocalSigner(object):
         for i in range(0, len(files), sign_chunk):
             subprocess.check_output(shlex.split(cmd + ' '.join(files[i:i+sign_chunk])), stderr=subprocess.STDOUT)
 
-    def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True):
+    def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True, output_suffix=None, use_sha256=False):
         """Create a detached signature of a file"""
 
         if passphrase_file and passphrase:
@@ -71,6 +71,10 @@ class LocalSigner(object):
             cmd += ['--homedir', self.gpg_path]
         if armor:
             cmd += ['--armor']
+        if output_suffix:
+            cmd += ['-o', input_file + "." + output_suffix]
+        if use_sha256:
+            cmd += ['--digest-algo', "SHA256"]
 
         #gpg > 2.1 supports password pipes only through the loopback interface
         #gpg < 2.1 errors out if given unknown parameters
-- 
2.32.0

[PATCH v1 2/2] package_manager: sign DEB package feeds

[Ferry Toth]

From: Ferry Toth <ftoth@exalondelft.nl>

Implement debian package repository signature.
For each Release file created in repository subdirectory, a signature
Release.gpg is created.

Signature is performed using gpg backend when the following variables
are set in local.conf:
PACKAGE_CLASSES += "sign_package_feed"
PACKAGE_FEED_GPG_NAME = "<Id of GPG key>"
PACKAGE_FEED_GPG_PASSPHRASE_FILE="<path to password file>"

Signed-off-by: Xavier Berger <xavier.berger@bio-logic.net>
Signed-off-by: Ferry Toth <ftoth@exalondelft.nl>
---
 meta/lib/oe/package_manager/deb/__init__.py | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/package_manager/deb/__init__.py b/meta/lib/oe/package_manager/deb/__init__.py
index 9f112ae25b..86ddb130ad 100644
--- a/meta/lib/oe/package_manager/deb/__init__.py
+++ b/meta/lib/oe/package_manager/deb/__init__.py
@@ -53,6 +53,7 @@ class DpkgIndexer(Indexer):
 
         index_cmds = []
         deb_dirs_found = False
+        index_sign_files = set()
         for arch in arch_list:
             arch_dir = os.path.join(self.deploy_dir, arch)
             if not os.path.isdir(arch_dir):
@@ -62,7 +63,10 @@ class DpkgIndexer(Indexer):
 
             cmd += "%s -fcn Packages > Packages.gz;" % gzip
 
-            with open(os.path.join(arch_dir, "Release"), "w+") as release:
+            release_file = os.path.join(arch_dir, "Release")
+            index_sign_files.add(release_file)
+
+            with open(release_file, "w+") as release:
                 release.write("Label: %s\n" % arch)
 
             cmd += "PSEUDO_UNLOAD=1 %s release . >> Release" % apt_ftparchive
@@ -76,8 +80,17 @@ class DpkgIndexer(Indexer):
             return
 
         oe.utils.multiprocess_launch(create_index, index_cmds, self.d)
-        if self.d.getVar('PACKAGE_FEED_SIGN') == '1':
-            raise NotImplementedError('Package feed signing not implementd for dpkg')
+        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
+            signer = get_signer(self.d, self.d.getVar('PACKAGE_FEED_GPG_BACKEND', True))
+        else:
+            signer = None
+        if signer:
+            for f in index_sign_files:
+                signer.detach_sign(f,
+                                   self.d.getVar('PACKAGE_FEED_GPG_NAME', True),
+                                   self.d.getVar('PACKAGE_FEED_GPG_PASSPHRASE_FILE', True),
+                                   output_suffix="gpg",
+                                   use_sha256=True)
 
 class PMPkgsList(PkgsList):
 
-- 
2.32.0

Re: [PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Ferry Toth]

Hi Richard,

I forgot to add a cover letter, sorry for that. The 2 patches  together 
implement DEB repository signing.

This is necessary since Gatesgarth |apt| (1.8.2) has become more strict 
and doesn’t allow unsigned repositories by default.

It is possible to override this behavior |but||| is more work then to 
enable signed DEB repositories. These patches makes DEB a first class 
citizen as IPK and RPM.

Patches have been in use in meta-intel-edison since Gatesgarth, see 
https://edison-fw.github.io/meta-intel-edison/5.0-Creating-a-deb-repository.html\

Ferry

Op 22-03-2022 om 22:19 schreef Ferry Toth:
> From: Xavier Berger <xavier.berger@bio-logic.net>
>
> output_suffix: If defined, add output_suffix as file name extension.
> use_sha256: If True, use sha256 for gpg as digest algorithm
>
> Signed-off-by: Xavier Berger <xavier.berger@bio-logic.net>
> Signed-off-by: Ferry Toth <ftoth@exalondelft.nl>
> ---
>   meta/lib/oe/gpg_sign.py | 6 +++++-
>   1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py
> index 1bce6cb792..aa9bb49f2c 100644
> --- a/meta/lib/oe/gpg_sign.py
> +++ b/meta/lib/oe/gpg_sign.py
> @@ -58,7 +58,7 @@ class LocalSigner(object):
>           for i in range(0, len(files), sign_chunk):
>               subprocess.check_output(shlex.split(cmd + ' '.join(files[i:i+sign_chunk])), stderr=subprocess.STDOUT)
>   
> -    def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True):
> +    def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True, output_suffix=None, use_sha256=False):
>           """Create a detached signature of a file"""
>   
>           if passphrase_file and passphrase:
> @@ -71,6 +71,10 @@ class LocalSigner(object):
>               cmd += ['--homedir', self.gpg_path]
>           if armor:
>               cmd += ['--armor']
> +        if output_suffix:
> +            cmd += ['-o', input_file + "." + output_suffix]
> +        if use_sha256:
> +            cmd += ['--digest-algo', "SHA256"]
>   
>           #gpg > 2.1 supports password pipes only through the loopback interface
>           #gpg < 2.1 errors out if given unknown parameters

Re: [PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Richard Purdie]

On Wed, 2022-03-23 at 19:34 +0100, Ferry Toth wrote:
> Hi Richard,
> 
> I forgot to add a cover letter, sorry for that. The 2 patches  together 
> implement DEB repository signing.
> 
> This is necessary since Gatesgarth |apt| (1.8.2) has become more strict 
> and doesn’t allow unsigned repositories by default.
> 
> It is possible to override this behavior |but||| is more work then to 
> enable signed DEB repositories. These patches makes DEB a first class 
> citizen as IPK and RPM.
> 
> Patches have been in use in meta-intel-edison since Gatesgarth, see 
> https://edison-fw.github.io/meta-intel-edison/5.0-Creating-a-deb-repository.html\

What puzzles me is that we can build root filesystems using apt, we test this on
the autobuilder. Saying repositories are broken since gatesgarth therefore seems
confusing in the commit message.

I guess we must configure apt to override that during the rootfs process and
likely an end user with a remote feed could do the same, possibly with a warning
from apt?

I'm also worried that there isn't any automated testing of this change. The
reason I worry is that since we don't show any testing failures right now, there
is clearly a hole in our automated testing coverage and there is no guarantee
that this feature will keep working. It is these smaller corner case issues
which tend to make or break the project's experience as if a feature is present,
people expect it to work. Can we improve the testing situation?

Cheers,

Richard

Re: [PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Ferry Toth]

[-- Attachment #1: Type: text/plain, Size: 2735 bytes --]

Hi,

Op 24-03-2022 om 09:12 schreef Richard Purdie:
> On Wed, 2022-03-23 at 19:34 +0100, Ferry Toth wrote:
>> Hi Richard,
>>
>> I forgot to add a cover letter, sorry for that. The 2 patches  together
>> implement DEB repository signing.
>>
>> This is necessary since Gatesgarth |apt| (1.8.2) has become more strict
>> and doesn’t allow unsigned repositories by default.
>>
>> It is possible to override this behavior |but||| is more work then to
>> enable signed DEB repositories. These patches makes DEB a first class
>> citizen as IPK and RPM.
>>
>> Patches have been in use in meta-intel-edison since Gatesgarth, see
>> https://edison-fw.github.io/meta-intel-edison/5.0-Creating-a-deb-repository.html\
> What puzzles me is that we can build root filesystems using apt, we test this on
> the autobuilder. Saying repositories are broken since gatesgarth therefore seems
> confusing in the commit message.

Good question. When I (meta-intel-edison) build the rootfs using DEB's 
it just works.

Could it be that during rootfs build dpkg is used and not apt? I think I 
have seen that in the logs.

Of course apt uses dpkg to install a package as well, but it refuses to 
download the package from a repo when it's not signed.

> I guess we must configure apt to override that during the rootfs process and
> likely an end user with a remote feed could do the same, possibly with a warning
> from apt?
I believe there is no issue during rootfs generation.
> I'm also worried that there isn't any automated testing of this change. The
> reason I worry is that since we don't show any testing failures right now, there
> is clearly a hole in our automated testing coverage and there is no guarantee
> that this feature will keep working. It is these smaller corner case issues
> which tend to make or break the project's experience as if a feature is present,
> people expect it to work. Can we improve the testing situation?

It doesn't seem to be a particularly volatile area in the code. I 
refreshed Xavier's patches for Gatesgarth, and am actively using 
unchanged patch on Honisiter.

I don't know how the automated testing is working but I guess for RPM a 
repo is generated using a small layer? And then tested on a qemu running 
the rootfs?

Should be almost same for deb/apt, maybe could be modified from rpm test?

Point is: currently deb is documented as a feasible package format to 
generate a repo. But it really is not without these signing patches. So 
we could either deprecate deb's (no, no please don't) or fix it.

These patches fix it. With or without automated testing, it is already 
better then the current situation.

> Cheers,
>
> Richard
>
>
>

[-- Attachment #2: Type: text/html, Size: 4070 bytes --]

Re: [PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Richard Purdie]

On Thu, 2022-03-24 at 12:23 +0100, Ferry Toth wrote:
> > On Wed, 2022-03-23 at 19:34 +0100, Ferry Toth wrote:
> > > 
> > > I forgot to add a cover letter, sorry for that. The 2 patches  together 
> > > implement DEB repository signing.
> > > 
> > > This is necessary since Gatesgarth |apt| (1.8.2) has become more strict 
> > > and doesn’t allow unsigned repositories by default.
> > > 
> > > It is possible to override this behavior |but||| is more work then to 
> > > enable signed DEB repositories. These patches makes DEB a first class 
> > > citizen as IPK and RPM.
> > > 
> > > Patches have been in use in meta-intel-edison since Gatesgarth, see 
> > > https://edison-fw.github.io/meta-intel-edison/5.0-Creating-a-deb-
> > > repository.html\
> > What puzzles me is that we can build root filesystems using apt, we test
> > this on
> > the autobuilder. Saying repositories are broken since gatesgarth therefore
> > seems
> > confusing in the commit message.
> Good question. When I (meta-intel-edison) build the rootfs using DEB's it just
> works.
> Could it be that during rootfs build dpkg is used and not apt? I think I have
> seen that in the logs.

It definitely uses apt.

> Of course apt uses dpkg to install a package as well, but it refuses to
> download the package from a repo when it's not signed. 

Perhaps the difference is the packages are local and not remote?

> > I guess we must configure apt to override that during the rootfs process and
> > likely an end user with a remote feed could do the same, possibly with a
> > warning
> > from apt?
>  I believe there is no issue during rootfs generation.
>  
> > I'm also worried that there isn't any automated testing of this change. The
> > reason I worry is that since we don't show any testing failures right now,
> > there
> > is clearly a hole in our automated testing coverage and there is no
> > guarantee
> > that this feature will keep working. It is these smaller corner case issues
> > which tend to make or break the project's experience as if a feature is
> > present,
> > people expect it to work. Can we improve the testing situation?
> It doesn't seem to be a particularly volatile area in the code. I refreshed
> Xavier's patches for Gatesgarth, and am actively using unchanged patch on
> Honisiter.
> I don't know how the automated testing is working but I guess for RPM a repo
> is generated using a small layer? And then tested on a qemu running the
> rootfs?
> Should be almost same for deb/apt, maybe could be modified from rpm test?

I think the rpm test is test_testimage_dnf in
meta/lib/oeqa/selftest/cases/runtime_test.py. You'd run it with:

oe-selftest -r runtime_test.TestImage.test_testimage_dnf

> Point is: currently deb is documented as a feasible package format to generate
> a repo. But it really is not without these signing patches. So we could either
> deprecate deb's (no, no please don't) or fix it.
> These patches fix it. With or without automated testing, it is already better
> then the current situation.

I'm not sure it is. The project gains yet another set of config options which we
don't have tests for and the maintainer stress levels in trying to keep it all
working just get worse.

We have a huge number of ways people can configure the builds. The only way the
project manages to keep all of this working is through automated testing, there
is simply no other way to do it. I appreciate adding tests is a pain and nobody
likes doing it. It does however let the project stay functional for everyone's
diverse use cases.

There are all kinds of patches I could take because the improve some corner
case. In most cases we don't know what else they might break or whether that
code continues to be used or stays working. I therefore do really want tests for
new configurations.

Who should write those tests? If you don't/can't as some actively using the
feature, it means someone else has to, but who?

I've not decided what to do with the patches to be honest but the merge and not
bother with tests argument doesn't sit well with me.

Cheers,

Richard

Re: [PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Ferry Toth]

[-- Attachment #1: Type: text/plain, Size: 4428 bytes --]

Hi

Op 24-03-2022 om 13:03 schreef Richard Purdie:
> On Thu, 2022-03-24 at 12:23 +0100, Ferry Toth wrote:
>>> On Wed, 2022-03-23 at 19:34 +0100, Ferry Toth wrote:
>>>> I forgot to add a cover letter, sorry for that. The 2 patches  together
>>>> implement DEB repository signing.
>>>>
>>>> This is necessary since Gatesgarth |apt| (1.8.2) has become more strict
>>>> and doesn’t allow unsigned repositories by default.
>>>>
>>>> It is possible to override this behavior |but||| is more work then to
>>>> enable signed DEB repositories. These patches makes DEB a first class
>>>> citizen as IPK and RPM.
>>>>
>>>> Patches have been in use in meta-intel-edison since Gatesgarth, see
>>>> https://edison-fw.github.io/meta-intel-edison/5.0-Creating-a-deb-
>>>> repository.html\
>>> What puzzles me is that we can build root filesystems using apt, we test
>>> this on
>>> the autobuilder. Saying repositories are broken since gatesgarth therefore
>>> seems
>>> confusing in the commit message.
>> Good question. When I (meta-intel-edison) build the rootfs using DEB's it just
>> works.
>> Could it be that during rootfs build dpkg is used and not apt? I think I have
>> seen that in the logs.
> It definitely uses apt.
>
>> Of course apt uses dpkg to install a package as well, but it refuses to
>> download the package from a repo when it's not signed.
> Perhaps the difference is the packages are local and not remote?
>
>>> I guess we must configure apt to override that during the rootfs process and
>>> likely an end user with a remote feed could do the same, possibly with a
>>> warning
>>> from apt?
>>   I believe there is no issue during rootfs generation.
>>   
>>> I'm also worried that there isn't any automated testing of this change. The
>>> reason I worry is that since we don't show any testing failures right now,
>>> there
>>> is clearly a hole in our automated testing coverage and there is no
>>> guarantee
>>> that this feature will keep working. It is these smaller corner case issues
>>> which tend to make or break the project's experience as if a feature is
>>> present,
>>> people expect it to work. Can we improve the testing situation?
>> It doesn't seem to be a particularly volatile area in the code. I refreshed
>> Xavier's patches for Gatesgarth, and am actively using unchanged patch on
>> Honisiter.
>> I don't know how the automated testing is working but I guess for RPM a repo
>> is generated using a small layer? And then tested on a qemu running the
>> rootfs?
>> Should be almost same for deb/apt, maybe could be modified from rpm test?
> I think the rpm test is test_testimage_dnf in
> meta/lib/oeqa/selftest/cases/runtime_test.py. You'd run it with:
>
> oe-selftest -r runtime_test.TestImage.test_testimage_dnf
I'll have a look.
>> Point is: currently deb is documented as a feasible package format to generate
>> a repo. But it really is not without these signing patches. So we could either
>> deprecate deb's (no, no please don't) or fix it.
>> These patches fix it. With or without automated testing, it is already better
>> then the current situation.
> I'm not sure it is. The project gains yet another set of config options which we
> don't have tests for and the maintainer stress levels in trying to keep it all
> working just get worse.
>
> We have a huge number of ways people can configure the builds. The only way the
> project manages to keep all of this working is through automated testing, there
> is simply no other way to do it. I appreciate adding tests is a pain and nobody
> likes doing it. It does however let the project stay functional for everyone's
> diverse use cases.
>
> There are all kinds of patches I could take because the improve some corner
> case. In most cases we don't know what else they might break or whether that
> code continues to be used or stays working. I therefore do really want tests for
> new configurations.
It's not an improvement but a fix for an existing config. Even if it 
would break in the future it can't be worse then it is now.
> Who should write those tests? If you don't/can't as some actively using the
> feature, it means someone else has to, but who?
I do understand.
> I've not decided what to do with the patches to be honest but the merge and not
> bother with tests argument doesn't sit well with me.
>
> Cheers,
>
> Richard
>

[-- Attachment #2: Type: text/html, Size: 6440 bytes --]

Re: [PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Ferry Toth]

Hi

Op 24-03-2022 om 16:36 schreef Ferry Toth:
>
> Hi
>
> Op 24-03-2022 om 13:03 schreef Richard Purdie:
>> On Thu, 2022-03-24 at 12:23 +0100, Ferry Toth wrote:
>>>> On Wed, 2022-03-23 at 19:34 +0100, Ferry Toth wrote:
>>>>> I forgot to add a cover letter, sorry for that. The 2 patches  together
>>>>> implement DEB repository signing.
>>>>>
>>>>> This is necessary since Gatesgarth |apt| (1.8.2) has become more strict
>>>>> and doesn’t allow unsigned repositories by default.
>>>>>
>>>>> It is possible to override this behavior |but||| is more work then to
>>>>> enable signed DEB repositories. These patches makes DEB a first class
>>>>> citizen as IPK and RPM.
>>>>>
>>>>> Patches have been in use in meta-intel-edison since Gatesgarth, see
>>>>> https://edison-fw.github.io/meta-intel-edison/5.0-Creating-a-deb-
>>>>> repository.html\
>>>> What puzzles me is that we can build root filesystems using apt, we test
>>>> this on
>>>> the autobuilder. Saying repositories are broken since gatesgarth therefore
>>>> seems
>>>> confusing in the commit message.

The Development Task manual in 3.22.4.3.3 names it a repository, as I 
believe is common for a distributions server holding pre-built packages. 
But I could change the name to package feed if that would be better.

What I meant to say is that apt requires signed package feeds since 
1.8.2 and disabling that is a workaround.

>>> Good question. When I (meta-intel-edison) build the rootfs using DEB's it just
>>> works.
>>> Could it be that during rootfs build dpkg is used and not apt? I think I have
>>> seen that in the logs.
>> It definitely uses apt.
Checking logs I see you are correct.
>>> Of course apt uses dpkg to install a package as well, but it refuses to
>>> download the package from a repo when it's not signed.
>> Perhaps the difference is the packages are local and not remote?

No, for generating the rootfs it appears the sources list file has:

deb [trusted=yes] file:/path.../oe-rootfs-repo/edison/ ./

So apparently has been taken care by disabling the signing requirement. 
[trusted=yes] is part of that workaround.

>>>> I guess we must configure apt to override that during the rootfs process and
>>>> likely an end user with a remote feed could do the same, possibly with a
>>>> warning
>>>> from apt?
Yes. But it is a pain to google to find how to do that.
>>>   I believe there is no issue during rootfs generation.
>>>   
>>>> I'm also worried that there isn't any automated testing of this change. The
>>>> reason I worry is that since we don't show any testing failures right now,
>>>> there
>>>> is clearly a hole in our automated testing coverage and there is no
>>>> guarantee
>>>> that this feature will keep working. It is these smaller corner case issues
>>>> which tend to make or break the project's experience as if a feature is
>>>> present,
>>>> people expect it to work. Can we improve the testing situation?
>>> It doesn't seem to be a particularly volatile area in the code. I refreshed
>>> Xavier's patches for Gatesgarth, and am actively using unchanged patch on
>>> Honisiter.
>>> I don't know how the automated testing is working but I guess for RPM a repo
>>> is generated using a small layer? And then tested on a qemu running the
>>> rootfs?
>>> Should be almost same for deb/apt, maybe could be modified from rpm test?
>> I think the rpm test is test_testimage_dnf in
>> meta/lib/oeqa/selftest/cases/runtime_test.py. You'd run it with:
>>
>> oe-selftest -r runtime_test.TestImage.test_testimage_dnf
> I'll have a look.

Pfff. These tests are O(1) more then the signing code they test. I'm 
trying this original test (on rpm) now to see if I can get that to run 
and understand how it works.

Am I right that meta/lib/oeqa/selftest/cases/signing.py tests the actual 
package feed signing?

>>> Point is: currently deb is documented as a feasible package format to generate
>>> a repo. But it really is not without these signing patches. So we could either
>>> deprecate deb's (no, no please don't) or fix it.
>>> These patches fix it. With or without automated testing, it is already better
>>> then the current situation.
>> I'm not sure it is. The project gains yet another set of config options which we
>> don't have tests for and the maintainer stress levels in trying to keep it all
>> working just get worse.
>>
>> We have a huge number of ways people can configure the builds. The only way the
>> project manages to keep all of this working is through automated testing, there
>> is simply no other way to do it. I appreciate adding tests is a pain and nobody
>> likes doing it. It does however let the project stay functional for everyone's
>> diverse use cases.
>>
>> There are all kinds of patches I could take because the improve some corner
>> case. In most cases we don't know what else they might break or whether that
>> code continues to be used or stays working. I therefore do really want tests for
>> new configurations.
> It's not an improvement but a fix for an existing config. Even if it 
> would break in the future it can't be worse then it is now.
>> Who should write those tests? If you don't/can't as some actively using the
>> feature, it means someone else has to, but who?
> I do understand.
>> I've not decided what to do with the patches to be honest but the merge and not
>> bother with tests argument doesn't sit well with me.
>>
>> Cheers,
>>
>> Richard
>>

Re: [PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Richard Purdie]

[-- Attachment #1: Type: text/plain, Size: 4942 bytes --]

On Thu, 2022-03-24 at 23:11 +0100, Ferry Toth wrote:
> Op 24-03-2022 om 16:36 schreef Ferry Toth:
> > Op 24-03-2022 om 13:03 schreef Richard Purdie:
> > > On Thu, 2022-03-24 at 12:23 +0100, Ferry Toth wrote:
> > > > > On Wed, 2022-03-23 at 19:34 +0100, Ferry Toth wrote:
> > > > > > I forgot to add a cover letter, sorry for that. The 2 patches  together
> > > > > > implement DEB repository signing.
> > > > > > 
> > > > > > This is necessary since Gatesgarth |apt| (1.8.2) has become more strict
> > > > > > and doesn’t allow unsigned repositories by default.
> > > > > > 
> > > > > > It is possible to override this behavior |but||| is more work then to
> > > > > > enable signed DEB repositories. These patches makes DEB a first class
> > > > > > citizen as IPK and RPM.
> > > > > > 
> > > > > > Patches have been in use in meta-intel-edison since Gatesgarth, see
> > > > > > https://edison-fw.github.io/meta-intel-edison/5.0-Creating-a-deb-
> > > > > > repository.html\
> > > > > What puzzles me is that we can build root filesystems using apt, we test
> > > > > this on
> > > > > the autobuilder. Saying repositories are broken since gatesgarth therefore
> > > > > seems
> > > > > confusing in the commit message.
> 
> The Development Task manual in 3.22.4.3.3 names it a repository, as I 
> believe is common for a distributions server holding pre-built packages. 
> But I could change the name to package feed if that would be better.
> 
> What I meant to say is that apt requires signed package feeds since 
> 1.8.2 and disabling that is a workaround.

I mean to distinguish between the use of repositories at rootfs creation time
compared to repositories used later to update an image. Since I know
repositories are used to build the rootfs, the commit message did confuse me at
first since rootfs construction works.



> 
> > > > Good question. When I (meta-intel-edison) build the rootfs using DEB's it just
> > > > works.
> > > > Could it be that during rootfs build dpkg is used and not apt? I think I have
> > > > seen that in the logs.
> > > It definitely uses apt.
> Checking logs I see you are correct.
> > > > Of course apt uses dpkg to install a package as well, but it refuses to
> > > > download the package from a repo when it's not signed.
> > > Perhaps the difference is the packages are local and not remote?
> 
> No, for generating the rootfs it appears the sources list file has:
> 
> deb [trusted=yes] file:/path.../oe-rootfs-repo/edison/ ./
> 
> So apparently has been taken care by disabling the signing requirement. 
> [trusted=yes] is part of that workaround.

That does give another way to work around this then...


> 
> > > > > I guess we must configure apt to override that during the rootfs process and
> > > > > likely an end user with a remote feed could do the same, possibly with a
> > > > > warning
> > > > > from apt?
> Yes. But it is a pain to google to find how to do that.
> > > >   I believe there is no issue during rootfs generation.
> > > >   
> > > > > I'm also worried that there isn't any automated testing of this change. The
> > > > > reason I worry is that since we don't show any testing failures right now,
> > > > > there
> > > > > is clearly a hole in our automated testing coverage and there is no
> > > > > guarantee
> > > > > that this feature will keep working. It is these smaller corner case issues
> > > > > which tend to make or break the project's experience as if a feature is
> > > > > present,
> > > > > people expect it to work. Can we improve the testing situation?
> > > > It doesn't seem to be a particularly volatile area in the code. I refreshed
> > > > Xavier's patches for Gatesgarth, and am actively using unchanged patch on
> > > > Honisiter.
> > > > I don't know how the automated testing is working but I guess for RPM a repo
> > > > is generated using a small layer? And then tested on a qemu running the
> > > > rootfs?
> > > > Should be almost same for deb/apt, maybe could be modified from rpm test?
> > > I think the rpm test is test_testimage_dnf in
> > > meta/lib/oeqa/selftest/cases/runtime_test.py. You'd run it with:
> > > 
> > > oe-selftest -r runtime_test.TestImage.test_testimage_dnf
> > I'll have a look.
> 
> Pfff. These tests are O(1) more then the signing code they test. I'm 
> trying this original test (on rpm) now to see if I can get that to run 
> and understand how it works.

What you're asking to work is a complex scenario so the test isn't going to be
simple.

FWIW there is also oeqa/runtime/cases/apt.py. That test checks if a package can
be installed from a remote package feed (no signing). It is doing:

echo deb [ allow-insecure=yes ] %s ./ > sources.list

to work around the signing issue.

> Am I right that meta/lib/oeqa/selftest/cases/signing.py tests the actual 
> package feed signing?

No. That tests that signed packages in rpm works rather than a signed remote
repository which is different.

Cheers,

Richard

[-- Attachment #2: Type: text/html, Size: 8228 bytes --]

Re: [PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Ferry Toth]

Hi,

Op 24-03-2022 om 23:20 schreef Richard Purdie:
> On Thu, 2022-03-24 at 23:11 +0100, Ferry Toth wrote:
>> Op 24-03-2022 om 16:36 schreef Ferry Toth:
>>> Op 24-03-2022 om 13:03 schreef Richard Purdie:
>>>> On Thu, 2022-03-24 at 12:23 +0100, Ferry Toth wrote:
>>>>>> On Wed, 2022-03-23 at 19:34 +0100, Ferry Toth wrote:
>>>>>>> I forgot to add a cover letter, sorry for that. The 2 patches  together
>>>>>>> implement DEB repository signing.
>>>>>>> This is necessary since Gatesgarth |apt| (1.8.2) has become more strict
>>>>>>> and doesn’t allow unsigned repositories by default.
>>>>>>> It is possible to override this behavior |but||| is more work then to
>>>>>>> enable signed DEB repositories. These patches makes DEB a first class
>>>>>>> citizen as IPK and RPM.
>>>>>>> Patches have been in use in meta-intel-edison since Gatesgarth, see
>>>>>>> https://edison-fw.github.io/meta-intel-edison/5.0-Creating-a-deb-
>>>>>>> repository.html\
>>>>>>
<snip>


>>>>>> I think the rpm test is test_testimage_dnf in
>>>>>> meta/lib/oeqa/selftest/cases/runtime_test.py. You'd run it with:
>>>>>> oe-selftest -r runtime_test.TestImage.test_testimage_dnf
>>> I'll have a look.
>>
I got the rpm test to run and with the help of Alexander I can restart 
that test to complete in about 6 minutes, which is good.

Right now, I have the modified the recipe so the test image is built 
from deb packages and then runs on qemu.

So next I need the workaround for apt disabled, auto install the gpg key 
on the testimage, run apt and parse the results.

I'm not really fluent with python, but might be able to get this done. 
If Alexander would be willing to give me a few pointers if I get stuck?

Let me play with this a few days (evenings) and then I'll post a WIP patch.


>
>
> Cheers,
>
> Richard

Re: [PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Ferry Toth]

[-- Attachment #1: Type: text/plain, Size: 982 bytes --]

Hi,

Op 24-03-2022 om 23:11 schreef Ferry Toth:
....

> It doesn't seem to be a particularly volatile area in the code. I 
> refreshed
>>>> Xavier's patches for Gatesgarth, and am actively using unchanged 
>>>> patch on
>>>> Honisiter.
>>>> I don't know how the automated testing is working but I guess for 
>>>> RPM a repo
>>>> is generated using a small layer? And then tested on a qemu running 
>>>> the
>>>> rootfs?
>>>> Should be almost same for deb/apt, maybe could be modified from rpm 
>>>> test?
>>> I think the rpm test is test_testimage_dnf in
>>> meta/lib/oeqa/selftest/cases/runtime_test.py. You'd run it with:
>>>
>>> oe-selftest -r runtime_test.TestImage.test_testimage_dnf
>> I'll have a look.

Grumble. I ran this test but it failed due to needing to run sudo to 
setup a tap interface (I ran it under screen).

Is there a way to continue this test without first clearing the build-st 
directory? It take about 2 hours to get to this failure point.

Thanks,

Ferry

[-- Attachment #2: Type: text/html, Size: 1872 bytes --]

Re: [OE-core] [PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Alexander Kanavin]

You can copy the sstate cache out of build-st somewhere where it will
be reused (set up local.conf accordingly).

Alex

On Fri, 25 Mar 2022 at 12:28, Ferry Toth <fntoth@gmail.com> wrote:
>
> Hi,
>
> Op 24-03-2022 om 23:11 schreef Ferry Toth:
> ....
>
> It doesn't seem to be a particularly volatile area in the code. I refreshed
>
> Xavier's patches for Gatesgarth, and am actively using unchanged patch on
> Honisiter.
> I don't know how the automated testing is working but I guess for RPM a repo
> is generated using a small layer? And then tested on a qemu running the
> rootfs?
> Should be almost same for deb/apt, maybe could be modified from rpm test?
>
> I think the rpm test is test_testimage_dnf in
> meta/lib/oeqa/selftest/cases/runtime_test.py. You'd run it with:
>
> oe-selftest -r runtime_test.TestImage.test_testimage_dnf
>
> I'll have a look.
>
> Grumble. I ran this test but it failed due to needing to run sudo to setup a tap interface (I ran it under screen).
>
> Is there a way to continue this test without first clearing the build-st directory? It take about 2 hours to get to this failure point.
>
> Thanks,
>
> Ferry
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#163636): https://lists.openembedded.org/g/openembedded-core/message/163636
> Mute This Topic: https://lists.openembedded.org/mt/89962631/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [PATCH v1 1/2] gpg-sign: Add parameters to gpg signature function

[Ferry Toth]

[-- Attachment #1: Type: text/plain, Size: 416 bytes --]

Hi Alexander,

I see your answer on gmane, but not by e-mail, strange.

But thanks for the tip, it worked and I got oe-selftest -r runtime_test.TestImage.test_testimage_dnf to run successfully.

(for future readers: by following "Yocto Project Development Tasks" manual section "Performing Automated Runtime Testing" https://docs.yoctoproject.org/dev-manual/common-tasks.html#enabling-runtime-tests-on-qemu )

[-- Attachment #2: Type: text/html, Size: 580 bytes --]