[layerindex-web] RFC: layer index docker fixes

[layerindex-web] RFC: layer index docker fixes

[Paul Eggleton]

Hi folks

I've been trying to get the OE layer index example docker setup in shape,
based upon improvements from myself, Michael and Konrad. For the moment I've
taken most of what Michael and I did (with some fixes and tweaks, and broken up
into smaller patches) and a few pieces of the Dockerfile from Konrad's recent
patch and put it here:

  http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/dockerfixes

Konrad, I know you were a little bit more ambitious with your changes but I 
wanted to get something closer to what Michael is using and then we can build 
upon it. In particular I haven't yet explored docker-compose as you have,
although it looks like it would make deploying all of this a bit easier. 

Let me know what you think.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [layerindex-web] RFC: layer index docker fixes

[Konrad Scherer]

[-- Attachment #1: Type: text/plain, Size: 1414 bytes --]

On 07/10/2018 12:17 PM, Paul Eggleton wrote:
> Hi folks
> 
> I've been trying to get the OE layer index example docker setup in shape,
> based upon improvements from myself, Michael and Konrad. For the moment I've
> taken most of what Michael and I did (with some fixes and tweaks, and broken up
> into smaller patches) and a few pieces of the Dockerfile from Konrad's recent
> patch and put it here:
> 
>    http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/dockerfixes
> 
> Konrad, I know you were a little bit more ambitious with your changes but I
> wanted to get something closer to what Michael is using and then we can build
> upon it. In particular I haven't yet explored docker-compose as you have,
> although it looks like it would make deploying all of this a bit easier.
> 
> Let me know what you think.

Hello Paul,

I have attached a patch to reduce the layerindex-web image size further. 
Docker image layers are like git objects and RUN commands can only hide 
files, not remove them once they have been committed. The pip upgrade 
caused the following 'pip install' to fail and was removed.

I recommend docker-compose because much of the information in 
docker/README can be captured in a compose yaml file. It handles the 
network setup and makes managing shared volumes much simpler.

-- 
Konrad Scherer, MTS, Linux Products Group, Wind River

[-- Attachment #2: 0001-Dockerfile-Reduce-image-size-by-merging-RUN-stages-t.patch --]
[-- Type: text/x-diff, Size: 2182 bytes --]

From 22274ea8298b4cb0c8ea3e545367a9c2c4c379c4 Mon Sep 17 00:00:00 2001
From: Konrad Scherer <Konrad.Scherer@windriver.com>
Date: Wed, 11 Jul 2018 12:19:15 -0400
Subject: [PATCH] Dockerfile: Reduce image size by merging RUN stages together

Docker commits changes after each RUN stage and commands that delete
files must part of the sane RUN command to actually reduce image size.

Debian stretch contains pip 9.0.3 and the upgrade to 10 caused pip
install to fail.

Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com>
---
 Dockerfile | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/Dockerfile b/Dockerfile
index 523280d..0aff2ec 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -12,8 +12,8 @@ ENV PYTHONUNBUFFERED=1 \
 #ENV https_proxy https://your.proxy.server:port
 
 COPY requirements.txt /
-RUN apt-get update
-RUN apt-get install -y --no-install-recommends \
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
 	autoconf \
 	g++ \
 	gcc \
@@ -31,20 +31,20 @@ RUN apt-get install -y --no-install-recommends \
 	locales \
 	netcat-openbsd \
 	curl \
-	vim
-RUN echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
+	vim \
+    && echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
 	&& locale-gen en_US.UTF-8 \
-	&& update-locale
-RUN pip install --upgrade pip
-RUN pip3 install gunicorn
-RUN pip install setuptools
-RUN pip3 install setuptools
-RUN pip install -r /requirements.txt
-RUN pip3 install -r /requirements.txt
-RUN apt-get purge -y autoconf g++ gcc make python3-dev libjpeg-dev libmariadbclient-dev \
+	&& update-locale \
+    && pip3 install gunicorn \
+    && pip install setuptools \
+    && pip3 install setuptools \
+    && pip install -r /requirements.txt \
+    && pip3 install -r /requirements.txt \
+    && apt-get purge -y autoconf g++ gcc make python-dev python3-dev libjpeg-dev libmariadbclient-dev \
 	&& apt-get autoremove -y \
 	&& rm -rf /var/lib/apt/lists/* \
 	&& apt-get clean
+
 COPY . /opt/layerindex
 COPY docker/settings.py /opt/layerindex/settings.py
 COPY docker/refreshlayers.sh /opt/refreshlayers.sh
-- 
2.18.0

Re: [layerindex-web] RFC: layer index docker fixes

[Paul Eggleton]

Hi Konrad

On Wednesday, 11 July 2018 7:03:28 PM CEST Konrad Scherer wrote:
> I have attached a patch to reduce the layerindex-web image size further. 
> Docker image layers are like git objects and RUN commands can only hide 
> files, not remove them once they have been committed. The pip upgrade 
> caused the following 'pip install' to fail and was removed.

Right, overall you don't save any storage space with separate stages - but the 
downside of merging these is that you lose the ability to incrementally update 
the container within those stages; any time anything before changes the whole 
block will need to be re-run. Having said that we still have the separation 
between requirements.txt and everything else which is probably sufficient.

Oddly I did not see a failure from pip when I built this yesterday, but I 
guess it's probably not strictly necessary to upgrade it anyway.

I've pushed your patch to the paule/dockerfixes branch.

> I recommend docker-compose because much of the information in 
> docker/README can be captured in a compose yaml file. It handles the 
> network setup and makes managing shared volumes much simpler.

Right, we'd just need to make sure everything we now have in the README is 
covered by the compose file (everything that's practical to cover, anyway).

Cheers,
Pau;

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [layerindex-web] RFC: layer index docker fixes

[Konrad Scherer]

On 07/12/2018 02:34 AM, Paul Eggleton wrote:
> Hi Konrad
> 
> On Wednesday, 11 July 2018 7:03:28 PM CEST Konrad Scherer wrote:
>> I have attached a patch to reduce the layerindex-web image size further.
>> Docker image layers are like git objects and RUN commands can only hide
>> files, not remove them once they have been committed. The pip upgrade
>> caused the following 'pip install' to fail and was removed.
> 
> Right, overall you don't save any storage space with separate stages - but the
> downside of merging these is that you lose the ability to incrementally update
> the container within those stages; any time anything before changes the whole
> block will need to be re-run. Having said that we still have the separation
> between requirements.txt and everything else which is probably sufficient.
> 
> Oddly I did not see a failure from pip when I built this yesterday, but I
> guess it's probably not strictly necessary to upgrade it anyway.
> 
> I've pushed your patch to the paule/dockerfixes branch.

Thanks, I just noticed that the image doesn't contain git. Was this 
deliberate or an oversight?

>> I recommend docker-compose because much of the information in
>> docker/README can be captured in a compose yaml file. It handles the
>> network setup and makes managing shared volumes much simpler.
> 
> Right, we'd just need to make sure everything we now have in the README is
> covered by the compose file (everything that's practical to cover, anyway).

I am heading out on vacation soon. I will rework my docker-compose 
configuration with the new image and make some documentation when I get 
back.

-- 
Konrad Scherer, MTS, Linux Products Group, Wind River

Re: [layerindex-web] RFC: layer index docker fixes

[Paul Eggleton]

On Thursday, 12 July 2018 10:16:12 PM CEST you wrote:
> I just noticed that the image doesn't contain git. Was this 
> deliberate or an oversight?

Ah, no, it wasn't deliberate - perhaps it was there earlier with buildpack-
deps? It doesn't appear that we ever explicitly had it in there. I've added it 
and pushed the branch again.
 
> >> I recommend docker-compose because much of the information in
> >> docker/README can be captured in a compose yaml file. It handles the
> >> network setup and makes managing shared volumes much simpler.
> > 
> > Right, we'd just need to make sure everything we now have in the README is
> > covered by the compose file (everything that's practical to cover
> > , anyway).
> 
> I am heading out on vacation soon. I will rework my docker-compose 
> configuration with the new image and make some documentation when I get 
> back.

OK great, thanks.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [layerindex-web] RFC: layer index docker fixes

[Konrad Scherer]

On 07/13/2018 03:29 AM, Paul Eggleton wrote:
> On Thursday, 12 July 2018 10:16:12 PM CEST you wrote:
>> I just noticed that the image doesn't contain git. Was this
>> deliberate or an oversight?
> 
> Ah, no, it wasn't deliberate - perhaps it was there earlier with buildpack-
> deps? It doesn't appear that we ever explicitly had it in there. I've added it
> and pushed the branch again.

I recently noticed that if gcc is removed in the 'apt-get purge' step, 
the bitbake sanity checker will fail due to missing 'gcc' when doing a 
layer update.

-- 
Konrad Scherer, MTS, Linux Products Group, Wind River

Re: [layerindex-web] RFC: layer index docker fixes

[Paul Eggleton]

On Monday, 16 July 2018 8:24:53 PM CEST Konrad Scherer wrote:
> On 07/13/2018 03:29 AM, Paul Eggleton wrote:
> > On Thursday, 12 July 2018 10:16:12 PM CEST you wrote:
> >> I just noticed that the image doesn't contain git. Was this
> >> deliberate or an oversight?
> > 
> > Ah, no, it wasn't deliberate - perhaps it was there earlier with
> > buildpack-deps? It doesn't appear that we ever explicitly had it in there. 
> > I've added it and pushed the branch again.
> 
> I recently noticed that if gcc is removed in the 'apt-get purge' step, 
> the bitbake sanity checker will fail due to missing 'gcc' when doing a 
> layer update.

Hmm, well that's a different problem - those sanity checks shouldn't be 
running. We don't need those tools since we're not building anything. I will 
have a look into this.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [layerindex-web] RFC: layer index docker fixes

[Paul Eggleton]

On Monday, 16 July 2018 10:16:55 PM CEST Paul Eggleton wrote:
> On Monday, 16 July 2018 8:24:53 PM CEST Konrad Scherer wrote:
> > I recently noticed that if gcc is removed in the 'apt-get purge' step, 
> > the bitbake sanity checker will fail due to missing 'gcc' when doing a 
> > layer update.
> 
> Hmm, well that's a different problem - those sanity checks shouldn't be 
> running. We don't need those tools since we're not building anything. I will 
> have a look into this.

I was wrong - I remembered that we explicitly set HOSTTOOLS = "gcc" in our own 
conf/local.conf because there are other places in the metadata where we check 
for gcc which we do not override, and thus it's expected to be there. I've 
fixed and pushed the branch again, thanks for pointing out the issue (this 
time I actually tested running an update ;).

Cheers,
Paul


-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [layerindex-web] RFC: layer index docker fixes

[Konrad Scherer]

On 07/17/2018 06:16 AM, Paul Eggleton wrote:
> On Monday, 16 July 2018 10:16:55 PM CEST Paul Eggleton wrote:
>> On Monday, 16 July 2018 8:24:53 PM CEST Konrad Scherer wrote:
>>> I recently noticed that if gcc is removed in the 'apt-get purge' step,
>>> the bitbake sanity checker will fail due to missing 'gcc' when doing a
>>> layer update.
>>
>> Hmm, well that's a different problem - those sanity checks shouldn't be
>> running. We don't need those tools since we're not building anything. I will
>> have a look into this.
> 
> I was wrong - I remembered that we explicitly set HOSTTOOLS = "gcc" in our own
> conf/local.conf because there are other places in the metadata where we check
> for gcc which we do not override, and thus it's expected to be there. I've
> fixed and pushed the branch again, thanks for pointing out the issue (this
> time I actually tested running an update ;).

I am also guilty of not running a full update before I sent the last 
patch. I leave on vacation tomorrow and will work on a docker-compose 
setup when I get back.

Have a great summer!

-- 
Konrad Scherer, MTS, Linux Products Group, Wind River

Re: [layerindex-web] RFC: layer index docker fixes

[Paul Eggleton]

Hi folks

I know Konrad promised to update the docker-compose setup to be closer
to what we'd done recently with the docker setup, but I needed something
urgently and wanted to learn how to use docker-compose anyway, so I've
put this together:

  http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/docker-compose

Let me know what you think - it works well enough for me here, although the
setup commands are now a bit ugly on their own.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [layerindex-web] RFC: layer index docker fixes

[Paul Eggleton]

On Wednesday, 29 August 2018 10:46:27 PM NZST Paul Eggleton wrote:
> Hi folks
> 
> I know Konrad promised to update the docker-compose setup to be closer
> to what we'd done recently with the docker setup, but I needed something
> urgently and wanted to learn how to use docker-compose anyway, so I've
> put this together:
> 
>   http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/docker-compose
> 
> Let me know what you think - it works well enough for me here, although the
> setup commands are now a bit ugly on their own.

I realised this morning that "docker-compose run" can simplify a lot of the
docker run commands, so I've just converted most of the commands in 
docker/README and pushed the branch again.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [layerindex-web] RFC: layer index docker fixes

[Paul Eggleton]

On Thursday, 30 August 2018 11:44:10 AM NZST Paul Eggleton wrote:
> On Wednesday, 29 August 2018 10:46:27 PM NZST Paul Eggleton wrote:
> > I know Konrad promised to update the docker-compose setup to be closer
> > to what we'd done recently with the docker setup, but I needed something
> > urgently and wanted to learn how to use docker-compose anyway, so I've
> > put this together:
> > 
> >   http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/docker-compose
> > 
> > Let me know what you think - it works well enough for me here, although the
> > setup commands are now a bit ugly on their own.
> 
> I realised this morning that "docker-compose run" can simplify a lot of the
> docker run commands, so I've just converted most of the commands in 
> docker/README and pushed the branch again.

So, any comments?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [layerindex-web] RFC: layer index docker fixes

[Konrad Scherer]

On 09/05/2018 08:04 AM, Paul Eggleton wrote:
> On Thursday, 30 August 2018 11:44:10 AM NZST Paul Eggleton wrote:
>> On Wednesday, 29 August 2018 10:46:27 PM NZST Paul Eggleton wrote:
>>> I know Konrad promised to update the docker-compose setup to be closer
>>> to what we'd done recently with the docker setup, but I needed something
>>> urgently and wanted to learn how to use docker-compose anyway, so I've
>>> put this together:
>>>
>>>    http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/docker-compose
>>>
>>> Let me know what you think - it works well enough for me here, although the
>>> setup commands are now a bit ugly on their own.
>>
>> I realised this morning that "docker-compose run" can simplify a lot of the
>> docker run commands, so I've just converted most of the commands in
>> docker/README and pushed the branch again.
> 
> So, any comments?

Sorry for the delay. I just rebased our internal layerindex to the 
latest and made some updates to the Dockerfiles and other files.

https://github.com/WindRiver-OpenSourceLabs/layerindex-web/tree/rebase-20181002

Dockerfile:
- added the wheel package to avoid errors in pip package installation
- Used ADD --chown to add all the files with the correct permissions to 
avoid a second chown RUN step
- Used an entrypoint.sh script instead of CMD to handle things like db 
initialization, creating a superuser, overriding default configuration 
with environment variables and collecting static files. The same 
entrypoint is used to start either the celery-worker or gunicorn.

https://github.com/WindRiver-OpenSourceLabs/layerindex-web/blob/rebase-20181002/docker/entrypoint.sh

docker/docker-compose.yaml:
- separating the gunicorn and the celery-worker is a good idea
- the blacklabelops/nginx image generates the nginx config from env 
variables which makes development easier. Changing an env var in 
docker-compose.yaml is easier than rebuilding the image or bind-mounting 
in a new version.
- One thing I didn't put in passing in the DB username and password 
because I am using this as a test instance, not a production one. Should 
be a simple change.

I had problems setting up a reverse proxy for the layerindex-web app 
when using a subpath (ie. http://<host>/li/). In urls.py, the leading 
slash on the last redirect works as a redirect to an absolute path and 
breaks the redirect when using a subpath . Fortunately the fix is simple 
and doesn't break usage without a subpath.

https://github.com/WindRiver-OpenSourceLabs/layerindex-web/commit/4586ff3f4fa477daa6e0bee6c6e9e0602105c8d2#diff-de6dd4b4c889fe0882cfd3f6df5aa451L29

Thanks!

-- 
Konrad Scherer, MTS, Linux Products Group, Wind River