[Buildroot] [PATCH 0/5] support/docker: generate reproducible base images

[Buildroot] [PATCH 0/5] support/docker: generate reproducible base images

[Yann E. MORIN]

Hello All!

Currently, we only ever refer to the "latest" version of the image.
This is not nice because:
  1. this is not reproducible;
  2. we can't have different images between master and the various
     maintenance branches.

So, this series makes the generated image reproducible, and explains how
to gnerate and push a tagged image.

Once this series has been applied, we'll have to generate such a tagged
image and update our .gitlab-ci.yml accordingly.


Regards,
Yann E. MORIN.


The following changes since commit 651af57cf011e6d8764d11932ce87fc064cd9a9b

  dash: enable line editting if libedit is selected (2018-02-04 12:09:09 +0100)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to c7619a7bdc91c8c2cb9d486859ddad8a8e89e78a

  support/dockker: create and push tagged images (2018-02-04 15:43:06 +0100)


----------------------------------------------------------------
Yann E. MORIN (5):
      support/docker: limit the number of layers
      support/docker: don't be silent when setting up
      support/docker: use a known base distribution
      support/docker: use a fixed Debian snapshot
      support/dockker: create and push tagged images

 support/docker/Dockerfile       | 45 ++++++++++++++++++++++++-----------------
 support/docker/apt-sources.list |  4 ++++
 2 files changed, 30 insertions(+), 19 deletions(-)
 create mode 100644 support/docker/apt-sources.list

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/5] support/docker: limit the number of layers

[Yann E. MORIN]

The official documentation [0] suggests limiting the number of layers
generated from a dockerfile. A layer is created for each RUN (and COPY
and ADD) command. But we are only ever interested in the final image,
so the intermediate layers are useless to us.

Limit the number of RUN commands to limit the number of generated
layers.

[0] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#minimize-the-number-of-layers

Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
 support/docker/Dockerfile | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
index ebb471f7e5..23018585eb 100644
--- a/support/docker/Dockerfile
+++ b/support/docker/Dockerfile
@@ -11,25 +11,25 @@ description="Container with everything needed to run Buildroot"
 # Setup environment
 ENV DEBIAN_FRONTEND noninteractive
 
-RUN dpkg --add-architecture i386
 # The container has no package lists, so need to update first
-RUN apt-get update -y -qq
-RUN apt-get install -y -qq --no-install-recommends \
-    build-essential cmake libc6:i386 gcc-multilib \
-    bc ca-certificates file locales rsync \
-    cvs bzr git mercurial subversion wget \
-    cpio unzip \
-    libncurses5-dev \
-    python-nose2 python-pexpect qemu-system-arm qemu-system-x86
-RUN apt-get -q -y autoremove
-RUN apt-get -q -y clean
+RUN dpkg --add-architecture i386 && \
+    apt-get update -y -qq && \
+    apt-get install -y -qq --no-install-recommends \
+        build-essential cmake libc6:i386 gcc-multilib \
+        bc ca-certificates file locales rsync \
+        cvs bzr git mercurial subversion wget \
+        cpio unzip \
+        libncurses5-dev \
+        python-nose2 python-pexpect qemu-system-arm qemu-system-x86 && \
+    apt-get -q -y autoremove && \
+    apt-get -q -y clean
 
 # To be able to generate a toolchain with locales, enable one UTF-8 locale
-RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
-RUN /usr/sbin/locale-gen
+RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
+    /usr/sbin/locale-gen
 
-RUN useradd -ms /bin/bash br-user
-RUN chown -R br-user:br-user /home/br-user
+RUN useradd -ms /bin/bash br-user && \
+    chown -R br-user:br-user /home/br-user
 
 USER br-user
 WORKDIR /home/br-user
-- 
2.14.1

[Buildroot] [PATCH 2/5] support/docker: don't be silent when setting up

[Yann E. MORIN]

This image is not built very often, and when it is, it is important to
see what's going on, so don't be silent when installing packages from
the distro, and since that can take a bit of time it thus serves as
progress report...

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
 support/docker/Dockerfile | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
index 23018585eb..aa2b38fcc9 100644
--- a/support/docker/Dockerfile
+++ b/support/docker/Dockerfile
@@ -13,16 +13,16 @@ ENV DEBIAN_FRONTEND noninteractive
 
 # The container has no package lists, so need to update first
 RUN dpkg --add-architecture i386 && \
-    apt-get update -y -qq && \
-    apt-get install -y -qq --no-install-recommends \
+    apt-get update -y && \
+    apt-get install -y --no-install-recommends \
         build-essential cmake libc6:i386 gcc-multilib \
         bc ca-certificates file locales rsync \
         cvs bzr git mercurial subversion wget \
         cpio unzip \
         libncurses5-dev \
         python-nose2 python-pexpect qemu-system-arm qemu-system-x86 && \
-    apt-get -q -y autoremove && \
-    apt-get -q -y clean
+    apt-get -y autoremove && \
+    apt-get -y clean
 
 # To be able to generate a toolchain with locales, enable one UTF-8 locale
 RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
-- 
2.14.1

[Buildroot] [PATCH 3/5] support/docker: use a known base distribution

[Yann E. MORIN]

Currently, we are using debian:stable, which is subject to change with
time, as new stable versions of Debian are released/updated.

Use the latest tagged stable release, stretch-20171210 as of today, as
the base distribution to use.

This will ease reproducible builds in the future.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
---
 support/docker/Dockerfile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
index aa2b38fcc9..49b74b88bd 100644
--- a/support/docker/Dockerfile
+++ b/support/docker/Dockerfile
@@ -3,7 +3,7 @@
 #   sudo docker build -t buildroot/base support/docker
 #   sudo docker push buildroot/base
 
-FROM debian:stable
+FROM debian:stretch-20171210
 LABEL maintainer="Buildroot mailing list <buildroot@buildroot.org>" \
       vendor="Buildroot" \
 description="Container with everything needed to run Buildroot"
-- 
2.14.1

[Buildroot] [PATCH 4/5] support/docker: use a fixed Debian snapshot

[Yann E. MORIN]

Since we're now using a specific base image tag, we need to also use a
specific, stable repository to get additional packages from for this
image.

As such, use the Debian snapshot that matches the base image.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
---
 support/docker/Dockerfile       | 7 +++++++
 support/docker/apt-sources.list | 4 ++++
 2 files changed, 11 insertions(+)
 create mode 100644 support/docker/apt-sources.list

diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
index 49b74b88bd..f18d195f37 100644
--- a/support/docker/Dockerfile
+++ b/support/docker/Dockerfile
@@ -3,7 +3,11 @@
 #   sudo docker build -t buildroot/base support/docker
 #   sudo docker push buildroot/base
 
+# We use a specific tag for the base image *and* the corresponding date
+# for the repository., so do not forget to update the apt-sources.list
+# file that is shipped next to this Dockerfile.
 FROM debian:stretch-20171210
+
 LABEL maintainer="Buildroot mailing list <buildroot@buildroot.org>" \
       vendor="Buildroot" \
 description="Container with everything needed to run Buildroot"
@@ -11,6 +15,9 @@ description="Container with everything needed to run Buildroot"
 # Setup environment
 ENV DEBIAN_FRONTEND noninteractive
 
+# This repository can be a bit slow at times. Don't panic...
+COPY apt-sources.list /etc/apt/sources.list
+
 # The container has no package lists, so need to update first
 RUN dpkg --add-architecture i386 && \
     apt-get update -y && \
diff --git a/support/docker/apt-sources.list b/support/docker/apt-sources.list
new file mode 100644
index 0000000000..789fb8fc17
--- /dev/null
+++ b/support/docker/apt-sources.list
@@ -0,0 +1,4 @@
+# Latest just before 20171210T000000Z:
+deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch main
+deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch-updates main
+deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/20171209T224618Z/ stretch/updates main
-- 
2.14.1

[Buildroot] [PATCH 5/5] support/dockker: create and push tagged images

[Yann E. MORIN]

Currently, we refer to the latest version of the image, which means we
can't guarantee any reproducibility. Also, it measn we can't have a
separate images for the maintenance branches (especially the LTS) and
master.

Update the comment in the Dockerfile to create and push tagged images.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>

---
Note: once this is applied, we can push such a tagged image and then
update our .gitlab-ci.yml accordingly.
---
 support/docker/Dockerfile | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
index f18d195f37..474e073c61 100644
--- a/support/docker/Dockerfile
+++ b/support/docker/Dockerfile
@@ -1,7 +1,7 @@
 # This Dockerfile generates the docker image that gets used by Gitlab CI
-# To build it:
-#   sudo docker build -t buildroot/base support/docker
-#   sudo docker push buildroot/base
+# To build it (YYYYMMDD.HHMM is the current date and time in UTC):
+#   sudo docker build -t buildroot/base:YYYYMMDD.HHMM support/docker
+#   sudo docker push buildroot/base:YYYYMMDD.HHMM
 
 # We use a specific tag for the base image *and* the corresponding date
 # for the repository., so do not forget to update the apt-sources.list
-- 
2.14.1

[Buildroot] [PATCH 0/5] support/docker: generate reproducible base images

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Hello All!
 > Currently, we only ever refer to the "latest" version of the image.
 > This is not nice because:
 >   1. this is not reproducible;
 >   2. we can't have different images between master and the various
 >      maintenance branches.

 > So, this series makes the generated image reproducible, and explains how
 > to gnerate and push a tagged image.

 > Once this series has been applied, we'll have to generate such a tagged
 > image and update our .gitlab-ci.yml accordingly.

Committed series, thanks - You are now officially our Docker expert ;)

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 0/5] support/docker: generate reproducible base images

[Yann E. MORIN]

Peter, All,

On 2018-02-04 22:00 +0100, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
>  > Currently, we only ever refer to the "latest" version of the image.
>  > This is not nice because:
>  >   1. this is not reproducible;
>  >   2. we can't have different images between master and the various
>  >      maintenance branches.
>  > So, this series makes the generated image reproducible, and explains how
>  > to gnerate and push a tagged image.
>  > Once this series has been applied, we'll have to generate such a tagged
>  > image and update our .gitlab-ci.yml accordingly.
> 
> Committed series, thanks - You are now officially our Docker expert ;)

Oh, now, we do have Cam Hutchison as an expert! ;-)

I'll regenerate the image now, push it, and update our .gitlab-ci.yml
now. Expect patch any minute now...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 4/5] support/docker: use a fixed Debian snapshot

[Arnout Vandecappelle]

On 04-02-18 15:44, Yann E. MORIN wrote:
> Since we're now using a specific base image tag, we need to also use a
> specific, stable repository to get additional packages from for this
> image.
> 
> As such, use the Debian snapshot that matches the base image.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Peter Korsgaard <peter@korsgaard.com>
> ---
>  support/docker/Dockerfile       | 7 +++++++
>  support/docker/apt-sources.list | 4 ++++
>  2 files changed, 11 insertions(+)
>  create mode 100644 support/docker/apt-sources.list
> 
> diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
> index 49b74b88bd..f18d195f37 100644
> --- a/support/docker/Dockerfile
> +++ b/support/docker/Dockerfile
> @@ -3,7 +3,11 @@
>  #   sudo docker build -t buildroot/base support/docker
>  #   sudo docker push buildroot/base
>  
> +# We use a specific tag for the base image *and* the corresponding date
> +# for the repository., so do not forget to update the apt-sources.list
> +# file that is shipped next to this Dockerfile.
>  FROM debian:stretch-20171210
> +
>  LABEL maintainer="Buildroot mailing list <buildroot@buildroot.org>" \
>        vendor="Buildroot" \
>  description="Container with everything needed to run Buildroot"
> @@ -11,6 +15,9 @@ description="Container with everything needed to run Buildroot"
>  # Setup environment
>  ENV DEBIAN_FRONTEND noninteractive
>  
> +# This repository can be a bit slow at times. Don't panic...
> +COPY apt-sources.list /etc/apt/sources.list
> +
>  # The container has no package lists, so need to update first
>  RUN dpkg --add-architecture i386 && \
>      apt-get update -y && \
> diff --git a/support/docker/apt-sources.list b/support/docker/apt-sources.list
> new file mode 100644
> index 0000000000..789fb8fc17
> --- /dev/null
> +++ b/support/docker/apt-sources.list
> @@ -0,0 +1,4 @@
> +# Latest just before 20171210T000000Z:
> +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch main

 AFAIU, snapshot should resolve any date to the latest available snapshot, so
http://snapshot.debian.org/archive/debian/20171210T000000Z/ should work as well
here. It should redirect to exactly the same URLs. Note thought that this
doesn't work for the top-level, only for a specific release (because they are
snapshotted at different times). So just clicking on the URL above won't work,
but if you put dists/stretch after it it redirects to
http://snapshot.debian.org/archive/debian/20171209T220346Z/dists/stretch/

 Anyway, OK for now, but something to keep in mind when it is bumped.

 Regards,
 Arnout

> +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch-updates main
> +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/20171209T224618Z/ stretch/updates main
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 4/5] support/docker: use a fixed Debian snapshot

[Yann E. MORIN]

Arnout, All,

On 2018-02-06 09:52 +0100, Arnout Vandecappelle spake thusly:
> On 04-02-18 15:44, Yann E. MORIN wrote:
> > Since we're now using a specific base image tag, we need to also use a
> > specific, stable repository to get additional packages from for this
> > image.
[--SNIP--]
> > diff --git a/support/docker/apt-sources.list b/support/docker/apt-sources.list
> > new file mode 100644
> > index 0000000000..789fb8fc17
> > --- /dev/null
> > +++ b/support/docker/apt-sources.list
> > @@ -0,0 +1,4 @@
> > +# Latest just before 20171210T000000Z:
> > +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch main
> 
>  AFAIU, snapshot should resolve any date to the latest available snapshot, so
> http://snapshot.debian.org/archive/debian/20171210T000000Z/ should work as well
> here. It should redirect to exactly the same URLs.

Yeah, I knew about the latest-prior-to-given-date rule, but see below...

> Note thought that this
> doesn't work for the top-level, only for a specific release (because they are
> snapshotted at different times). So just clicking on the URL above won't work,
> but if you put dists/stretch after iti

For me, I even got http 500, and apt-get update would also fail from
time to time (about once for three).

That is indeed something I should have mentioned in the commit log.

> it redirects to
> http://snapshot.debian.org/archive/debian/20171209T220346Z/dists/stretch/

nd what's even weird ios that it now works, even for the debian-security
suite, which is not tagged at the same date:
    http://snapshot.debian.org/archive/debian-security/20171209T224618Z/dists/stretch/updates/

>  Anyway, OK for now, but something to keep in mind when it is bumped.

Yeah, I also would have preferred it, but as I said, it was sometimes
failing for me... :-/

Thanks for the feedback! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'