[Buildroot] [PATCH] utils/genrandconfig: use --no-check-certificate in wget by default

[Buildroot] [PATCH] utils/genrandconfig: use --no-check-certificate in wget by default

[Thomas Petazzoni]

A number of autobuilder failures are due to the fact that autobuilder
instances use old distributions, with old SSL certificates, and
therefore wget aborts with an error "The certificate of `xyz.org' is
not trusted.".

In order to avoid such failures that are not very interesting in the
context of the autobuilders, we pass --no-check-certificate to
wget. The integrity of the downloaded files is anyway verified by the
hashes, and this is only meant to be used in the context of
testing/CI, not in production.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 utils/genrandconfig | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/utils/genrandconfig b/utils/genrandconfig
index a67d46fad9..5ce6015765 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -345,6 +345,9 @@ def gen_config(args):
     with open(minimalconfigfile) as minimalf:
         configlines += minimalf.readlines()
 
+    # Allow hosts with old certificates to download over https
+    configlines.append("BR2_WGET=\"wget --passive-ftp -nd -t 3 --no-check-certificate\"")
+
     # Amend the configuration with a few things.
     if randint(0, 20) == 0:
         configlines.append("BR2_ENABLE_DEBUG=y\n")
-- 
2.13.5

[Buildroot] [PATCH] utils/genrandconfig: use --no-check-certificate in wget by default

[Thomas Petazzoni]

Hello,

On Sat,  2 Sep 2017 23:29:38 +0200, Thomas Petazzoni wrote:
> A number of autobuilder failures are due to the fact that autobuilder
> instances use old distributions, with old SSL certificates, and
> therefore wget aborts with an error "The certificate of `xyz.org' is
> not trusted.".
> 
> In order to avoid such failures that are not very interesting in the
> context of the autobuilders, we pass --no-check-certificate to
> wget. The integrity of the downloaded files is anyway verified by the
> hashes, and this is only meant to be used in the context of
> testing/CI, not in production.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Would it be possible to get some feedback on this patch?

We recently bump dbus to 1.10.24, and look how the autobuilders are
"polluted" by this certificate issue:
http://autobuild.buildroot.net/?reason=dbus-1.10.24.

We've got two options: either we do it in utils/genrandconfig as
proposed, or we do it in the autobuild-run script that runs on the
autobuilder slaves.

Comments ?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH] utils/genrandconfig: use --no-check-certificate in wget by default

[Arnout Vandecappelle]

On 10-10-17 22:25, Thomas Petazzoni wrote:
> Hello,
> 
> On Sat,  2 Sep 2017 23:29:38 +0200, Thomas Petazzoni wrote:
>> A number of autobuilder failures are due to the fact that autobuilder
>> instances use old distributions, with old SSL certificates, and
>> therefore wget aborts with an error "The certificate of `xyz.org' is
>> not trusted.".

 I don't understand how this can happen. The autobuilders should all be running
supported (not EOL) distros, right? And in a supported distro, I'd expect
ca-certificates get updated. Otherwise the distro becomes next to useless.

 But I seem to remember that you have a CentOS 5 autobuilder running, and CentOS
5 went EOL on March 31, 2017. So perhaps it's time to switch to CentOS 6?


>> In order to avoid such failures that are not very interesting in the
>> context of the autobuilders

 I think they *are* interesting (not very, but still interesting), because
actual users *will* hit these problems.

>> , we pass --no-check-certificate to
>> wget. The integrity of the downloaded files is anyway verified by the
>> hashes, and this is only meant to be used in the context of
>> testing/CI, not in production.
>>
>> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> 
> Would it be possible to get some feedback on this patch?
> 
> We recently bump dbus to 1.10.24, and look how the autobuilders are
> "polluted" by this certificate issue:
> http://autobuild.buildroot.net/?reason=dbus-1.10.24.

 But once Peter updates sources.buildroot.org that should be OK again, no?

 Regards,
 Arnout

> 
> We've got two options: either we do it in utils/genrandconfig as
> proposed, or we do it in the autobuild-run script that runs on the
> autobuilder slaves.
> 
> Comments ?
> 
> Thanks,
> 
> Thomas
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] utils/genrandconfig: use --no-check-certificate in wget by default

[Thomas Petazzoni]

Hello,

On Tue, 10 Oct 2017 23:25:03 +0200, Arnout Vandecappelle wrote:

>  I don't understand how this can happen. The autobuilders should all be running
> supported (not EOL) distros, right?

Absolutely not. I'm intentionally running an old Debian in order to
catch build issues related to old systems. Running non-supported (i.e
EOL distros) is very common in enterprise environments, so I want to
keep testing this. Just to be clear: I don't personally run EOL
distros, and we don't use EOL distros at Free Electrons so it's not
something that I personally care about. But I believe that a lot of
companies do use such old distros on build machines, and therefore it
makes sense to keep testing with a fairly old system.

And therefore the certificates are often out of date.

>  But I seem to remember that you have a CentOS 5 autobuilder running, and CentOS
> 5 went EOL on March 31, 2017. So perhaps it's time to switch to CentOS 6?

Me running CentOS ? Crazy you. I'm running a Debian system for the
autobuilders.

> >> In order to avoid such failures that are not very interesting in the
> >> context of the autobuilders  
> 
>  I think they *are* interesting (not very, but still interesting), because
> actual users *will* hit these problems.

Not really: by the time we make a release, sources.buildroot.net will
have fetched the tarball, and therefore our users will simply see
nothing, except that their download gracefully falls back to
sources.buildroot.net.

So those build failures only pollute the autobuild.buildroot.net
results in the time window between a package being bumped, and the
tarball being grabbed by sources.buildroot.net. If you look today at
autobuild.buildroot.net, there are no more dbus-1.10.24 build failures,
because the "old" autobuilder instances download the tarball from
sources.b.n.

> > We recently bump dbus to 1.10.24, and look how the autobuilders are
> > "polluted" by this certificate issue:
> > http://autobuild.buildroot.net/?reason=dbus-1.10.24.  
> 
>  But once Peter updates sources.buildroot.org that should be OK again, no?

Yes, but this pollution happens again and again and again every time we
bump a package that is fetched from https://, with a certificate too
recent for those old systems. This creates useless noise in the
autobuilders. And this noise is useless because we are really not going
to do anything about this.

But perhaps it's an issue more unique to my autobuilder instance, and
therefore something I should fix locally? Perhaps just an autobuild-run
feature that allows to append a custom config option to the config file
being built, so that people can locally add some tweaks?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH] utils/genrandconfig: use --no-check-certificate in wget by default

[Arnout Vandecappelle]

On 11-10-17 08:53, Thomas Petazzoni wrote:
> Hello,
> 
> On Tue, 10 Oct 2017 23:25:03 +0200, Arnout Vandecappelle wrote:
[snip]
>>> We recently bump dbus to 1.10.24, and look how the autobuilders are
>>> "polluted" by this certificate issue:
>>> http://autobuild.buildroot.net/?reason=dbus-1.10.24.  
>>
>>  But once Peter updates sources.buildroot.org that should be OK again, no?
> 
> Yes, but this pollution happens again and again and again every time we
> bump a package that is fetched from https://, with a certificate too
> recent for those old systems. This creates useless noise in the
> autobuilders. And this noise is useless because we are really not going
> to do anything about this.

 Even though this problem hasn't occurred again in the last half year AFAIK,
your explanation does make sense so I've applied to master, thanks.

 Regards,
 Arnout

> But perhaps it's an issue more unique to my autobuilder instance, and
> therefore something I should fix locally? Perhaps just an autobuild-run
> feature that allows to append a custom config option to the config file
> being built, so that people can locally add some tweaks?

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF