[Qemu-devel] [PATCH RFC v4 0/4] docker: Support building qemu-user powered docker test images

[Qemu-devel] [PATCH RFC v4 0/4] docker: Support building qemu-user powered docker test images

[Fam Zheng]

Alex,

This is the result of my fiddling around docker + qemu-user in the weekend. It
can do most of the work except the injection of qemu-user binary from host
build.  We can try to integrate your "docker.py update" into Makefile to do
that, but an open question is how to handle the dependency cleanly: we
intentionally allow "make docker-foo" w/o configure or build, but the qemu-user
case is very different.

The major change is using "FROM debian" and build thing in the container so
that qemu-arm, fakeroot and debootstrap are not required on the system (the
docker file installs qemu-user-static). This way the pre script is not needed.

The upside is debootstrap can make use of docker cache, so updating is easy,
but we have to handle chroot in run script, before running the test command.

This seems cleaner in host side dependencies to me, what do you think?

Fam

Fam Zheng (4):
  docker: More sensible run script
  docker: Fix exit code if $CMD failed
  docker: Support "QEMU_CHROOT" in dockerfiles
  docker: Add debootstrap-arm image

 tests/docker/Makefile.include                   |  5 ++--
 tests/docker/dockerfiles/debootstrap-arm.docker | 35 +++++++++++++++++++++++++
 tests/docker/run                                | 28 +++++++++++++++++---
 3 files changed, 62 insertions(+), 6 deletions(-)
 create mode 100644 tests/docker/dockerfiles/debootstrap-arm.docker

-- 
2.7.4

[Qemu-devel] [PATCH RFC v4 1/4] docker: More sensible run script

[Fam Zheng]

It is very easy to figure out current directory and bash option from the
execution, so do less in the Makefile invocation command line, and
figure both options in the script.

This makes the next patch easier.

Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/Makefile.include |  4 +---
 tests/docker/run              | 12 +++++++++---
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index f88c0a7..c5546ee 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -114,10 +114,8 @@ docker-run-%: docker-qemu-src
 				-e CCACHE_DIR=/var/tmp/ccache \
 				-v $$(realpath $(DOCKER_SRC_COPY)):/var/tmp/qemu:z$(COMMA)ro \
 				-v $(DOCKER_CCACHE_DIR):/var/tmp/ccache:z \
-				-w /var/tmp/qemu \
 				qemu:$(IMAGE) \
-				$(if $V,/bin/bash -x ,) \
-				./run \
+				/var/tmp/qemu/run \
 				$(CMD); \
 			, "  RUN $(CMD) in $(IMAGE)")))
 
diff --git a/tests/docker/run b/tests/docker/run
index ec3d119..575e732 100755
--- a/tests/docker/run
+++ b/tests/docker/run
@@ -11,6 +11,12 @@
 # or (at your option) any later version. See the COPYING file in
 # the top-level directory.
 
+if test -n "$V"; then
+    set -x
+fi
+
+BASE="$(dirname $(realpath $0))"
+
 # Prepare the environment
 . /etc/profile || true
 export PATH=/usr/lib/ccache:$PATH
@@ -24,10 +30,10 @@ export TEST_DIR=/tmp/qemu-test
 mkdir -p $TEST_DIR/{src,build,install}
 
 # Extract the source tarballs
-tar -C $TEST_DIR/src -xzf qemu.tgz
+tar -C $TEST_DIR/src -xzf $BASE/qemu.tgz
 for p in dtc pixman; do
-    if test -f $p.tgz; then
-        tar -C $TEST_DIR/src/$p -xzf $p.tgz
+    if test -f $BASE/$p.tgz; then
+        tar -C $TEST_DIR/src/$p -xzf $BASE/$p.tgz
         export FEATURES="$FEATURES $p"
     fi
 done
-- 
2.7.4

[Qemu-devel] [PATCH RFC v4 2/4] docker: Fix exit code if $CMD failed

[Fam Zheng]

Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/run | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/docker/run b/tests/docker/run
index 575e732..38ce789 100755
--- a/tests/docker/run
+++ b/tests/docker/run
@@ -11,6 +11,8 @@
 # or (at your option) any later version. See the COPYING file in
 # the top-level directory.
 
+set -e
+
 if test -n "$V"; then
     set -x
 fi
@@ -61,4 +63,6 @@ elif test -n "$DEBUG"; then
     echo
     # Force error after shell exits
     $SHELL && exit 1
+else
+    exit 1
 fi
-- 
2.7.4

Re: [Qemu-devel] [PATCH RFC v4 2/4] docker: Fix exit code if $CMD failed

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

On 07/10/2016 09:20 PM, Fam Zheng wrote:
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  tests/docker/run | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/tests/docker/run b/tests/docker/run
> index 575e732..38ce789 100755
> --- a/tests/docker/run
> +++ b/tests/docker/run
> @@ -11,6 +11,8 @@
>  # or (at your option) any later version. See the COPYING file in
>  # the top-level directory.
>  
> +set -e
> +

'set -e' is a crutch that often does not do what you naively expect. In
particular, it interacts very poorly with shell functions; if 'f' is a
shell function, running 'f' is different than running 'f || alternate',
in whether the body of 'f' will exit early.  I'd much rather script
without having to think about whether 'set -e' is doing the right thing,
because it usually isn't.

>  if test -n "$V"; then
>      set -x
>  fi
> @@ -61,4 +63,6 @@ elif test -n "$DEBUG"; then
>      echo
>      # Force error after shell exits
>      $SHELL && exit 1
> +else
> +    exit 1
>  fi
> 

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

[Qemu-devel] [PATCH RFC v4 3/4] docker: Support "QEMU_CHROOT" in dockerfiles

[Fam Zheng]

This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
indicate that the test execution should be done in a chroot in the
container.

Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
testing scripts.

The SYS_ADMIN is a required capability for mount, add it to the
docker run command line.

Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/Makefile.include |  1 +
 tests/docker/run              | 12 ++++++++++++
 2 files changed, 13 insertions(+)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index c5546ee..e9821ba 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -107,6 +107,7 @@ docker-run-%: docker-qemu-src
 		$(call quiet-command,\
 			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
 				-t \
+				--cap-add SYS_ADMIN \
 				$(if $(DEBUG),-i,--net=none) \
 				-e TARGET_LIST=$(TARGET_LIST) \
 				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
diff --git a/tests/docker/run b/tests/docker/run
index 38ce789..4e80cc3 100755
--- a/tests/docker/run
+++ b/tests/docker/run
@@ -19,6 +19,18 @@ fi
 
 BASE="$(dirname $(realpath $0))"
 
+# cp files into the chroot and execute there
+if test -n "$QEMU_CHROOT"; then
+    mkdir -p $QEMU_CHROOT/$BASE
+    cp $BASE/* $QEMU_CHROOT/$BASE
+    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
+    for bp in dev sys proc; do
+        mount --bind /$bp $QEMU_CHROOT/$bp
+    done
+    QEMU_CHROOT="" chroot $QEMU_CHROOT_SAVE $BASE/run "$@"
+    exit 0
+fi
+
 # Prepare the environment
 . /etc/profile || true
 export PATH=/usr/lib/ccache:$PATH
-- 
2.7.4

Re: [Qemu-devel] [PATCH RFC v4 3/4] docker: Support "QEMU_CHROOT" in dockerfiles

[Paolo Bonzini]

On 11/07/2016 05:20, Fam Zheng wrote:
> This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
> indicate that the test execution should be done in a chroot in the
> container.
> 
> Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
> testing scripts.
> 
> The SYS_ADMIN is a required capability for mount, add it to the
> docker run command line.
> 
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  tests/docker/Makefile.include |  1 +
>  tests/docker/run              | 12 ++++++++++++
>  2 files changed, 13 insertions(+)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index c5546ee..e9821ba 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -107,6 +107,7 @@ docker-run-%: docker-qemu-src
>  		$(call quiet-command,\
>  			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
>  				-t \
> +				--cap-add SYS_ADMIN \
>  				$(if $(DEBUG),-i,--net=none) \
>  				-e TARGET_LIST=$(TARGET_LIST) \
>  				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
> diff --git a/tests/docker/run b/tests/docker/run
> index 38ce789..4e80cc3 100755
> --- a/tests/docker/run
> +++ b/tests/docker/run
> @@ -19,6 +19,18 @@ fi
>  
>  BASE="$(dirname $(realpath $0))"
>  
> +# cp files into the chroot and execute there
> +if test -n "$QEMU_CHROOT"; then
> +    mkdir -p $QEMU_CHROOT/$BASE
> +    cp $BASE/* $QEMU_CHROOT/$BASE
> +    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
> +    for bp in dev sys proc; do
> +        mount --bind /$bp $QEMU_CHROOT/$bp

Can you ask docker to do these bind mounts instead?

Thanks,

Paolo

> +    done
> +    QEMU_CHROOT="" chroot $QEMU_CHROOT_SAVE $BASE/run "$@"
> +    exit 0
> +fi
> +
>  # Prepare the environment
>  . /etc/profile || true
>  export PATH=/usr/lib/ccache:$PATH
> 

Re: [Qemu-devel] [PATCH RFC v4 3/4] docker: Support "QEMU_CHROOT" in dockerfiles

[Alex Bennée]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 11/07/2016 05:20, Fam Zheng wrote:
>> This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
>> indicate that the test execution should be done in a chroot in the
>> container.
>>
>> Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
>> testing scripts.
>>
>> The SYS_ADMIN is a required capability for mount, add it to the
>> docker run command line.
>>
>> Signed-off-by: Fam Zheng <famz@redhat.com>
>> ---
>>  tests/docker/Makefile.include |  1 +
>>  tests/docker/run              | 12 ++++++++++++
>>  2 files changed, 13 insertions(+)
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index c5546ee..e9821ba 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -107,6 +107,7 @@ docker-run-%: docker-qemu-src
>>  		$(call quiet-command,\
>>  			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
>>  				-t \
>> +				--cap-add SYS_ADMIN \
>>  				$(if $(DEBUG),-i,--net=none) \
>>  				-e TARGET_LIST=$(TARGET_LIST) \
>>  				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
>> diff --git a/tests/docker/run b/tests/docker/run
>> index 38ce789..4e80cc3 100755
>> --- a/tests/docker/run
>> +++ b/tests/docker/run
>> @@ -19,6 +19,18 @@ fi
>>
>>  BASE="$(dirname $(realpath $0))"
>>
>> +# cp files into the chroot and execute there
>> +if test -n "$QEMU_CHROOT"; then
>> +    mkdir -p $QEMU_CHROOT/$BASE
>> +    cp $BASE/* $QEMU_CHROOT/$BASE
>> +    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
>> +    for bp in dev sys proc; do
>> +        mount --bind /$bp $QEMU_CHROOT/$bp
>
> Can you ask docker to do these bind mounts instead?

AFAICT docker's various mount directives are all focused on tasks like mounting data
volumes from the host into the container.

It's a bit of a shame having to do this as the original approach was to
use docker to avoid having fancy bind mounts on my hosts system. We are
now getting to inception levels of nesting here. But the benefit is not
requiring the host having the pre-requisites to bootstrap the system.

That said looking at the debootstrap requirements I've seen instructions
that start with download the deb, ar extract and then run the script by
hand so maybe this is over complicating things?

Is it possible to bootstrap a Fedora rootfs with a similar script?

>
> Thanks,
>
> Paolo
>
>> +    done
>> +    QEMU_CHROOT="" chroot $QEMU_CHROOT_SAVE $BASE/run "$@"
>> +    exit 0
>> +fi
>> +
>>  # Prepare the environment
>>  . /etc/profile || true
>>  export PATH=/usr/lib/ccache:$PATH
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH RFC v4 3/4] docker: Support "QEMU_CHROOT" in dockerfiles

[Paolo Bonzini]

On 11/07/2016 12:08, Alex Bennée wrote:
> 
> Paolo Bonzini <pbonzini@redhat.com> writes:
> 
>> On 11/07/2016 05:20, Fam Zheng wrote:
>>> This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
>>> indicate that the test execution should be done in a chroot in the
>>> container.
>>>
>>> Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
>>> testing scripts.
>>>
>>> The SYS_ADMIN is a required capability for mount, add it to the
>>> docker run command line.
>>>
>>> Signed-off-by: Fam Zheng <famz@redhat.com>
>>> ---
>>>  tests/docker/Makefile.include |  1 +
>>>  tests/docker/run              | 12 ++++++++++++
>>>  2 files changed, 13 insertions(+)
>>>
>>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>>> index c5546ee..e9821ba 100644
>>> --- a/tests/docker/Makefile.include
>>> +++ b/tests/docker/Makefile.include
>>> @@ -107,6 +107,7 @@ docker-run-%: docker-qemu-src
>>>  		$(call quiet-command,\
>>>  			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
>>>  				-t \
>>> +				--cap-add SYS_ADMIN \
>>>  				$(if $(DEBUG),-i,--net=none) \
>>>  				-e TARGET_LIST=$(TARGET_LIST) \
>>>  				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
>>> diff --git a/tests/docker/run b/tests/docker/run
>>> index 38ce789..4e80cc3 100755
>>> --- a/tests/docker/run
>>> +++ b/tests/docker/run
>>> @@ -19,6 +19,18 @@ fi
>>>
>>>  BASE="$(dirname $(realpath $0))"
>>>
>>> +# cp files into the chroot and execute there
>>> +if test -n "$QEMU_CHROOT"; then
>>> +    mkdir -p $QEMU_CHROOT/$BASE
>>> +    cp $BASE/* $QEMU_CHROOT/$BASE
>>> +    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
>>> +    for bp in dev sys proc; do
>>> +        mount --bind /$bp $QEMU_CHROOT/$bp
>>
>> Can you ask docker to do these bind mounts instead?
> 
> AFAICT docker's various mount directives are all focused on tasks like mounting data
> volumes from the host into the container.
> 
> It's a bit of a shame having to do this as the original approach was to
> use docker to avoid having fancy bind mounts on my hosts system. We are
> now getting to inception levels of nesting here. But the benefit is not
> requiring the host having the pre-requisites to bootstrap the system.
> 
> That said looking at the debootstrap requirements I've seen instructions
> that start with download the deb, ar extract and then run the script by
> hand so maybe this is over complicating things?
> 
> Is it possible to bootstrap a Fedora rootfs with a similar script?

In theory "yum" is all that you need to bootstrap a Fedora rootfs.

Paolo

Re: [Qemu-devel] [PATCH RFC v4 3/4] docker: Support "QEMU_CHROOT" in dockerfiles

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
> indicate that the test execution should be done in a chroot in the
> container.
>
> Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
> testing scripts.
>
> The SYS_ADMIN is a required capability for mount, add it to the
> docker run command line.
>
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  tests/docker/Makefile.include |  1 +
>  tests/docker/run              | 12 ++++++++++++
>  2 files changed, 13 insertions(+)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index c5546ee..e9821ba 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -107,6 +107,7 @@ docker-run-%: docker-qemu-src
>  		$(call quiet-command,\
>  			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
>  				-t \
> +				--cap-add SYS_ADMIN \
>  				$(if $(DEBUG),-i,--net=none) \
>  				-e TARGET_LIST=$(TARGET_LIST) \
>  				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
> diff --git a/tests/docker/run b/tests/docker/run
> index 38ce789..4e80cc3 100755
> --- a/tests/docker/run
> +++ b/tests/docker/run
> @@ -19,6 +19,18 @@ fi
>
>  BASE="$(dirname $(realpath $0))"
>
> +# cp files into the chroot and execute there
> +if test -n "$QEMU_CHROOT"; then
> +    mkdir -p $QEMU_CHROOT/$BASE
> +    cp $BASE/* $QEMU_CHROOT/$BASE
> +    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
> +    for bp in dev sys proc; do
> +        mount --bind /$bp $QEMU_CHROOT/$bp

So this misses out /dev/pts which is a subdir of dev which leads to
messages like:

debconf: delaying package configuration, since apt-utils is not
installed
Fetched 103 kB in 0s (133 kB/s)
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such
file or directory)


> +    done
> +    QEMU_CHROOT="" chroot $QEMU_CHROOT_SAVE $BASE/run "$@"
> +    exit 0
> +fi
> +
>  # Prepare the environment
>  . /etc/profile || true
>  export PATH=/usr/lib/ccache:$PATH


--
Alex Bennée

Re: [Qemu-devel] [PATCH RFC v4 3/4] docker: Support "QEMU_CHROOT" in dockerfiles

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> This allows a docker file to say "ENV QEMU_CHROOT /path/to/new/root" to
> indicate that the test execution should be done in a chroot in the
> container.
>
> Bind mount dev,sys,proc into QEMU_CHROOT to make them avaiable for
> testing scripts.
>
> The SYS_ADMIN is a required capability for mount, add it to the
> docker run command line.
>
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  tests/docker/Makefile.include |  1 +
>  tests/docker/run              | 12 ++++++++++++
>  2 files changed, 13 insertions(+)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index c5546ee..e9821ba 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -107,6 +107,7 @@ docker-run-%: docker-qemu-src
>  		$(call quiet-command,\
>  			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
>  				-t \
> +				--cap-add SYS_ADMIN \
>  				$(if $(DEBUG),-i,--net=none) \
>  				-e TARGET_LIST=$(TARGET_LIST) \
>  				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
> diff --git a/tests/docker/run b/tests/docker/run
> index 38ce789..4e80cc3 100755
> --- a/tests/docker/run
> +++ b/tests/docker/run
> @@ -19,6 +19,18 @@ fi
>
>  BASE="$(dirname $(realpath $0))"
>
> +# cp files into the chroot and execute there
> +if test -n "$QEMU_CHROOT"; then
> +    mkdir -p $QEMU_CHROOT/$BASE
> +    cp $BASE/* $QEMU_CHROOT/$BASE
> +    QEMU_CHROOT_SAVE="$QEMU_CHROOT"
> +    for bp in dev sys proc; do
> +        mount --bind /$bp $QEMU_CHROOT/$bp
> +    done
> +    QEMU_CHROOT="" chroot $QEMU_CHROOT_SAVE $BASE/run "$@"
> +    exit 0
> +fi
> +

Running:

make docker-test TEST="test-quick" IMAGES="debootstrap-arm" V=1 J=9

Hmm another failure:

Image is up to date.
/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py run  -t --cap-add SYS_ADMIN --net=none -e TARGET_LIST= -e EXTRA_CONFIGURE_OPTS= -e V=1 -e J=9 -e DEBUG= -e CCACHE_DIR=/var/tmp/ccache -v $(realpath docker-src.2016-07-11-13.16.09.23334):/var/tmp/qemu:z,ro -v $HOME/.cache/qemu-docker-ccache:/var/tmp/ccache:z qemu:debootstrap-arm /var/tmp/qemu/run test-clang;
+++ realpath /var/tmp/qemu/run
++ dirname /var/tmp/qemu/run
+ BASE=/var/tmp/qemu
+ test -n /debootstrap-arm
+ mkdir -p /debootstrap-arm//var/tmp/qemu
+ cp /var/tmp/qemu/dtc.tgz /var/tmp/qemu/pixman.tgz /var/tmp/qemu/qemu.tgz /var/tmp/qemu/run /debootstrap-arm//var/tmp/qemu
+ QEMU_CHROOT_SAVE=/debootstrap-arm
+ for bp in dev sys proc
+ mount --bind /dev /debootstrap-arm/dev
mount: mount /dev on /debootstrap-arm/dev failed: Permission denied
/home/alex/lsrc/qemu/qemu.git/tests/docker/Makefile.include:102: recipe for target 'docker-run-test-clang@debootstrap-arm' failed
make: *** [docker-run-test-clang@debootstrap-arm] Error 32
You have new mail in /var/mail/alex

>  # Prepare the environment
>  . /etc/profile || true
>  export PATH=/usr/lib/ccache:$PATH


--
Alex Bennée

[Qemu-devel] [PATCH RFC v4 4/4] docker: Add debootstrap-arm image

[Fam Zheng]

Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/dockerfiles/debootstrap-arm.docker | 35 +++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debootstrap-arm.docker

diff --git a/tests/docker/dockerfiles/debootstrap-arm.docker b/tests/docker/dockerfiles/debootstrap-arm.docker
new file mode 100644
index 0000000..cb15f2f
--- /dev/null
+++ b/tests/docker/dockerfiles/debootstrap-arm.docker
@@ -0,0 +1,35 @@
+FROM debian:testing
+
+RUN apt-get update
+RUN apt-get install -y fakeroot debootstrap qemu-user-static
+
+RUN mkdir /debootstrap-arm
+
+RUN cd /debootstrap-arm && fakeroot debootstrap --variant=buildd --foreign \
+    --arch=armhf testing . http://httpredir.debian.org/debian
+
+RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' \
+    /debootstrap-arm/debootstrap/functions
+
+RUN mkdir -p /debootstrap-arm/usr/local/bin
+
+RUN ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/bin/qemu-arm && \
+    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/bin/qemu-arm-static && \
+    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/local/bin/qemu-arm && \
+    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/local/bin/qemu-arm-static
+
+# Run stage 2
+RUN if ! chroot /debootstrap-arm /debootstrap/debootstrap --second-stage; then \
+        echo "Failed to chroot and do stage 2"; \
+        echo "Please set up binfmt_misc to point arm binary to one of:"; \
+        echo "  /usr/bin/qemu-arm"; \
+        echo "  /usr/bin/qemu-arm-static"; \
+        echo "  /usr/local/bin/qemu-arm"; \
+        echo "  /usr/local/bin/qemu-arm-static"; \
+        exit 1; \
+    fi
+RUN chroot /debootstrap-arm sh -c 'cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list'
+RUN chroot /debootstrap-arm apt-get update
+RUN chroot /debootstrap-arm apt-get build-dep -y qemu
+RUN chroot /debootstrap-arm apt-get install -y ccache
+ENV QEMU_CHROOT /debootstrap-arm
-- 
2.7.4

Re: [Qemu-devel] [PATCH RFC v4 4/4] docker: Add debootstrap-arm image

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  tests/docker/dockerfiles/debootstrap-arm.docker | 35 +++++++++++++++++++++++++
>  1 file changed, 35 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/debootstrap-arm.docker
>
> diff --git a/tests/docker/dockerfiles/debootstrap-arm.docker b/tests/docker/dockerfiles/debootstrap-arm.docker
> new file mode 100644
> index 0000000..cb15f2f
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debootstrap-arm.docker
> @@ -0,0 +1,35 @@
> +FROM debian:testing
> +
> +RUN apt-get update
> +RUN apt-get install -y fakeroot debootstrap qemu-user-static
> +
> +RUN mkdir /debootstrap-arm
> +
> +RUN cd /debootstrap-arm && fakeroot debootstrap --variant=buildd --foreign \
> +    --arch=armhf testing . http://httpredir.debian.org/debian
> +
> +RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' \
> +    /debootstrap-arm/debootstrap/functions
> +
> +RUN mkdir -p /debootstrap-arm/usr/local/bin
> +
> +RUN ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/bin/qemu-arm && \
> +    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/bin/qemu-arm-static && \
> +    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/local/bin/qemu-arm && \
> +    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/local/bin/qemu-arm-static
> +
> +# Run stage 2
> +RUN if ! chroot /debootstrap-arm /debootstrap/debootstrap --second-stage; then \
> +        echo "Failed to chroot and do stage 2"; \
> +        echo "Please set up binfmt_misc to point arm binary to one of:"; \
> +        echo "  /usr/bin/qemu-arm"; \
> +        echo "  /usr/bin/qemu-arm-static"; \
> +        echo "  /usr/local/bin/qemu-arm"; \
> +        echo "  /usr/local/bin/qemu-arm-static"; \
> +        exit 1; \
> +    fi
> +RUN chroot /debootstrap-arm sh -c 'cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list'
> +RUN chroot /debootstrap-arm apt-get update
> +RUN chroot /debootstrap-arm apt-get build-dep -y qemu
> +RUN chroot /debootstrap-arm apt-get install -y ccache
> +ENV QEMU_CHROOT /debootstrap-arm

OK I've done some more experimenting and two things are apparent:

  debootstrap is widely packaged for various distros

And

  The script it fairly portable so we can always run it directly

I hacked up the .pre script to do the following and tested on my Arch
VM:

#!/bin/sh
#
# Simple wrapper for debootstrap, run in the docker build context
#
FAKEROOT=`which fakeroot 2> /dev/null`
DEBOOTSTRAP=`which debootstrap 2> /dev/null`
DEBOOTSTRAP_SOURCE=https://anonscm.debian.org/git/d-i/debootstrap.git

if [ -z $FAKEROOT ]; then
    echo "Please install fakeroot to enable bootstraping"
    exit 1
fi

if [ -z $DEBOOTSTRAP ]; then
    echo "No debootstrap installed, attempting to install from SCM"
    git clone ${DEBOOTSTRAP_SOURCE} ./debootstrap.git
    export DEBOOTSTRAP_DIR=./debootstrap.git
    DEBOOTSTRAP=./debootstrap.git/debootstrap
fi

echo "Building a rootfs using ${FAKEROOT} and ${DEBOOTSTRAP}"

${FAKEROOT} /bin/sh -x ${DEBOOTSTRAP} --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian
exit 0

So I think it is feasible to but the pre-requisite checking and work
around in the pre script and be done with it. It seems neater than the
chroot within a container approach.

Thoughts?

--
Alex Bennée

Re: [Qemu-devel] [PATCH RFC v4 4/4] docker: Add debootstrap-arm image

[Fam Zheng]

On Mon, 07/11 20:06, Alex Bennée wrote:
> 
> Fam Zheng <famz@redhat.com> writes:
> 
> > Signed-off-by: Fam Zheng <famz@redhat.com>
> > ---
> >  tests/docker/dockerfiles/debootstrap-arm.docker | 35 +++++++++++++++++++++++++
> >  1 file changed, 35 insertions(+)
> >  create mode 100644 tests/docker/dockerfiles/debootstrap-arm.docker
> >
> > diff --git a/tests/docker/dockerfiles/debootstrap-arm.docker b/tests/docker/dockerfiles/debootstrap-arm.docker
> > new file mode 100644
> > index 0000000..cb15f2f
> > --- /dev/null
> > +++ b/tests/docker/dockerfiles/debootstrap-arm.docker
> > @@ -0,0 +1,35 @@
> > +FROM debian:testing
> > +
> > +RUN apt-get update
> > +RUN apt-get install -y fakeroot debootstrap qemu-user-static
> > +
> > +RUN mkdir /debootstrap-arm
> > +
> > +RUN cd /debootstrap-arm && fakeroot debootstrap --variant=buildd --foreign \
> > +    --arch=armhf testing . http://httpredir.debian.org/debian
> > +
> > +RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' \
> > +    /debootstrap-arm/debootstrap/functions
> > +
> > +RUN mkdir -p /debootstrap-arm/usr/local/bin
> > +
> > +RUN ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/bin/qemu-arm && \
> > +    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/bin/qemu-arm-static && \
> > +    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/local/bin/qemu-arm && \
> > +    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/local/bin/qemu-arm-static
> > +
> > +# Run stage 2
> > +RUN if ! chroot /debootstrap-arm /debootstrap/debootstrap --second-stage; then \
> > +        echo "Failed to chroot and do stage 2"; \
> > +        echo "Please set up binfmt_misc to point arm binary to one of:"; \
> > +        echo "  /usr/bin/qemu-arm"; \
> > +        echo "  /usr/bin/qemu-arm-static"; \
> > +        echo "  /usr/local/bin/qemu-arm"; \
> > +        echo "  /usr/local/bin/qemu-arm-static"; \
> > +        exit 1; \
> > +    fi
> > +RUN chroot /debootstrap-arm sh -c 'cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list'
> > +RUN chroot /debootstrap-arm apt-get update
> > +RUN chroot /debootstrap-arm apt-get build-dep -y qemu
> > +RUN chroot /debootstrap-arm apt-get install -y ccache
> > +ENV QEMU_CHROOT /debootstrap-arm
> 
> OK I've done some more experimenting and two things are apparent:
> 
>   debootstrap is widely packaged for various distros
> 
> And
> 
>   The script it fairly portable so we can always run it directly
> 
> I hacked up the .pre script to do the following and tested on my Arch
> VM:
> 
> #!/bin/sh
> #
> # Simple wrapper for debootstrap, run in the docker build context
> #
> FAKEROOT=`which fakeroot 2> /dev/null`
> DEBOOTSTRAP=`which debootstrap 2> /dev/null`
> DEBOOTSTRAP_SOURCE=https://anonscm.debian.org/git/d-i/debootstrap.git
> 
> if [ -z $FAKEROOT ]; then
>     echo "Please install fakeroot to enable bootstraping"
>     exit 1
> fi
> 
> if [ -z $DEBOOTSTRAP ]; then
>     echo "No debootstrap installed, attempting to install from SCM"
>     git clone ${DEBOOTSTRAP_SOURCE} ./debootstrap.git
>     export DEBOOTSTRAP_DIR=./debootstrap.git
>     DEBOOTSTRAP=./debootstrap.git/debootstrap
> fi
> 
> echo "Building a rootfs using ${FAKEROOT} and ${DEBOOTSTRAP}"
> 
> ${FAKEROOT} /bin/sh -x ${DEBOOTSTRAP} --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian
> exit 0
> 
> So I think it is feasible to but the pre-requisite checking and work
> around in the pre script and be done with it. It seems neater than the
> chroot within a container approach.
> 
> Thoughts?

Yes, makes sense to me.  Two more questions:

Can we have "make docker-images" to update the docker image if pre script has
been updated?

Can we skip debootstrap (i.e. skip .pre script) if it's only the dockerfile
that has changed?

Thanks,

Fam

Re: [Qemu-devel] [PATCH RFC v4 4/4] docker: Add debootstrap-arm image

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Mon, 07/11 20:06, Alex Bennée wrote:
>>
>> Fam Zheng <famz@redhat.com> writes:
>>
>> > Signed-off-by: Fam Zheng <famz@redhat.com>
>> > ---
>> >  tests/docker/dockerfiles/debootstrap-arm.docker | 35 +++++++++++++++++++++++++
>> >  1 file changed, 35 insertions(+)
>> >  create mode 100644 tests/docker/dockerfiles/debootstrap-arm.docker
>> >
>> > diff --git a/tests/docker/dockerfiles/debootstrap-arm.docker b/tests/docker/dockerfiles/debootstrap-arm.docker
>> > new file mode 100644
>> > index 0000000..cb15f2f
>> > --- /dev/null
>> > +++ b/tests/docker/dockerfiles/debootstrap-arm.docker
>> > @@ -0,0 +1,35 @@
>> > +FROM debian:testing
>> > +
>> > +RUN apt-get update
>> > +RUN apt-get install -y fakeroot debootstrap qemu-user-static
>> > +
>> > +RUN mkdir /debootstrap-arm
>> > +
>> > +RUN cd /debootstrap-arm && fakeroot debootstrap --variant=buildd --foreign \
>> > +    --arch=armhf testing . http://httpredir.debian.org/debian
>> > +
>> > +RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' \
>> > +    /debootstrap-arm/debootstrap/functions
>> > +
>> > +RUN mkdir -p /debootstrap-arm/usr/local/bin
>> > +
>> > +RUN ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/bin/qemu-arm && \
>> > +    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/bin/qemu-arm-static && \
>> > +    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/local/bin/qemu-arm && \
>> > +    ln /usr/bin/qemu-arm-static /debootstrap-arm/usr/local/bin/qemu-arm-static
>> > +
>> > +# Run stage 2
>> > +RUN if ! chroot /debootstrap-arm /debootstrap/debootstrap --second-stage; then \
>> > +        echo "Failed to chroot and do stage 2"; \
>> > +        echo "Please set up binfmt_misc to point arm binary to one of:"; \
>> > +        echo "  /usr/bin/qemu-arm"; \
>> > +        echo "  /usr/bin/qemu-arm-static"; \
>> > +        echo "  /usr/local/bin/qemu-arm"; \
>> > +        echo "  /usr/local/bin/qemu-arm-static"; \
>> > +        exit 1; \
>> > +    fi
>> > +RUN chroot /debootstrap-arm sh -c 'cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list'
>> > +RUN chroot /debootstrap-arm apt-get update
>> > +RUN chroot /debootstrap-arm apt-get build-dep -y qemu
>> > +RUN chroot /debootstrap-arm apt-get install -y ccache
>> > +ENV QEMU_CHROOT /debootstrap-arm
>>
>> OK I've done some more experimenting and two things are apparent:
>>
>>   debootstrap is widely packaged for various distros
>>
>> And
>>
>>   The script it fairly portable so we can always run it directly
>>
>> I hacked up the .pre script to do the following and tested on my Arch
>> VM:
>>
>> #!/bin/sh
>> #
>> # Simple wrapper for debootstrap, run in the docker build context
>> #
>> FAKEROOT=`which fakeroot 2> /dev/null`
>> DEBOOTSTRAP=`which debootstrap 2> /dev/null`
>> DEBOOTSTRAP_SOURCE=https://anonscm.debian.org/git/d-i/debootstrap.git
>>
>> if [ -z $FAKEROOT ]; then
>>     echo "Please install fakeroot to enable bootstraping"
>>     exit 1
>> fi
>>
>> if [ -z $DEBOOTSTRAP ]; then
>>     echo "No debootstrap installed, attempting to install from SCM"
>>     git clone ${DEBOOTSTRAP_SOURCE} ./debootstrap.git
>>     export DEBOOTSTRAP_DIR=./debootstrap.git
>>     DEBOOTSTRAP=./debootstrap.git/debootstrap
>> fi
>>
>> echo "Building a rootfs using ${FAKEROOT} and ${DEBOOTSTRAP}"
>>
>> ${FAKEROOT} /bin/sh -x ${DEBOOTSTRAP} --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian
>> exit 0
>>
>> So I think it is feasible to but the pre-requisite checking and work
>> around in the pre script and be done with it. It seems neater than the
>> chroot within a container approach.
>>
>> Thoughts?
>
> Yes, makes sense to me.  Two more questions:
>
> Can we have "make docker-images" to update the docker image if pre script has
> been updated?

We could, I'm not sure what the dependency should be for a file that may
or may not exist?

The larger problem is what to do about the fact that debootstrap is a
multi-arch script? Currently I can manually build the
qmu:debian-bootstrap image from the command line for any linux-user
architecture.

Once this is done I can then run all the tests under that single
emulated architecture. The question is should the "make docker" expand
debian-bootstrap into all the various architecture targets and build
them all or just assume what ever one is set-up by the user is the one
that gets tested?

Currently:

  make docker-image-debian-bootstrap DEB_ARCH=arm64 DEB_TYPE=testing V=1 J=9

Will fail because the build machinery doesn't know what
--include-executable to pass to the docker.py script.

However after that everything should work.

My current state is at:

   https://github.com/stsquad/qemu/tree/misc/docker-linux-user-v4

I'll post v5 shortly after I've done some basic testing.

>
> Can we skip debootstrap (i.e. skip .pre script) if it's only the dockerfile
> that has changed?

No, because a running a dockerfile implies re-building the container
from scratch and the .pre script needs to pre-seed the build context.

>
> Thanks,
>
> Fam


--
Alex Bennée

Re: [Qemu-devel] [PATCH RFC v4 0/4] docker: Support building qemu-user powered docker test images

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> Alex,
>
> This is the result of my fiddling around docker + qemu-user in the weekend. It
> can do most of the work except the injection of qemu-user binary from host
> build.  We can try to integrate your "docker.py update" into Makefile to do
> that, but an open question is how to handle the dependency cleanly: we
> intentionally allow "make docker-foo" w/o configure or build, but the qemu-user
> case is very different.

I think it is certainly worth keeping the ability to inject qemu into
the docker container. While for a random build-on-other-arch type setup
we can rely on the docker distro's implementation for builds the other
use case is testing the current qemu still works for a linux-user type
setup.

> The major change is using "FROM debian" and build thing in the container so
> that qemu-arm, fakeroot and debootstrap are not required on the system (the
> docker file installs qemu-user-static). This way the pre script is not needed.
>
> The upside is debootstrap can make use of docker cache, so updating is easy,
> but we have to handle chroot in run script, before running the test
> command.

But really the native Debian host doesn't take the time to setup, it is
the debootstrap that chugs along. Until docker hub starts supporting multiple
architecture docker images (which I believe is coming) it doesn't make
much difference to the image build time.

> This seems cleaner in host side dependencies to me, what do you think?

I think we should support both. Keeping all the dependencies inside the
docker environment works really well for keeping everything host
distribution agnostic and is probably the way forward to support "cross"
build setups from the Makefiles.

One advantage of the pre script approach is we can have a generic recipe
for Debian which allows the user to spin up various architectures just
by tweaking the environment variables. And from my point of view one of
the drivers for my patches was being able to setup rootfs without
complicated bind mounts and chroots so I do like "docker run" dropping
directly into a translated shell.

Is there a similar boot strap we can use to setup cross fedora images?

Maybe we should merge the .py changes and your dockerfiles in
test/docker/dockerfiles and my general purpose (but requiring host
requirements) into contrib/docker?


>
> Fam
>
> Fam Zheng (4):
>   docker: More sensible run script
>   docker: Fix exit code if $CMD failed
>   docker: Support "QEMU_CHROOT" in dockerfiles
>   docker: Add debootstrap-arm image
>
>  tests/docker/Makefile.include                   |  5 ++--
>  tests/docker/dockerfiles/debootstrap-arm.docker | 35 +++++++++++++++++++++++++
>  tests/docker/run                                | 28 +++++++++++++++++---
>  3 files changed, 62 insertions(+), 6 deletions(-)
>  create mode 100644 tests/docker/dockerfiles/debootstrap-arm.docker


--
Alex Bennée

Re: [Qemu-devel] [PATCH RFC v4 0/4] docker: Support building qemu-user powered docker test images

[no-reply]

---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org