* [Qemu-devel] [PATCH v2 01/15] tests: Add utilities for docker testing
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-29 16:46 ` Alex Bennée
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 02/15] Makefile: Rules " Fam Zheng
` (14 subsequent siblings)
15 siblings, 1 reply; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
docker_run: A wrapper for "docker run" (or "sudo -n docker run" if
necessary), which takes care of killing and removing the running
container at SIGINT.
docker_clean: A tool to tear down all the containers including inactive
ones that are started by docker_run.
docker_build: A tool to compare an image from given dockerfile and
rebuild it if they're different.
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/docker.py | 113 ++++++++++++++++++++++++++++++++++++++++++++++
tests/docker/docker_build | 42 +++++++++++++++++
tests/docker/docker_clean | 22 +++++++++
tests/docker/docker_run | 29 ++++++++++++
4 files changed, 206 insertions(+)
create mode 100755 tests/docker/docker.py
create mode 100755 tests/docker/docker_build
create mode 100755 tests/docker/docker_clean
create mode 100755 tests/docker/docker_run
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
new file mode 100755
index 0000000..d175a86
--- /dev/null
+++ b/tests/docker/docker.py
@@ -0,0 +1,113 @@
+#!/usr/bin/env python2 -B
+#
+# Docker controlling module
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+import os
+import subprocess
+import json
+import hashlib
+import atexit
+import uuid
+
+class ContainerTerminated(Exception):
+ """ Raised if the container has already existed """
+ pass
+
+class Docker(object):
+ """ Running Docker commands """
+ def __init__(self):
+ self._command = self._guess_command()
+ self._instances = []
+ atexit.register(self._kill_instances)
+
+ def _do(self, cmd, quiet=True, **kwargs):
+ if quiet:
+ kwargs["stdout"] = subprocess.PIPE
+ return subprocess.call(self._command + cmd, **kwargs)
+
+ def _do_kill_instances(self, only_known, only_active=True):
+ cmd = ["ps", "-q"]
+ if not only_active:
+ cmd.append("-a")
+ for i in self._output(cmd).split():
+ resp = self._output(["inspect", i])
+ labels = json.loads(resp)[0]["Config"]["Labels"]
+ active = json.loads(resp)[0]["State"]["Running"]
+ if not labels:
+ continue
+ instance_uuid = labels.get("com.qemu.instance.uuid", None)
+ if not instance_uuid:
+ continue
+ if only_known and instance_uuid not in self._instances:
+ continue
+ print "Terminating", i
+ if active:
+ self._do(["kill", i])
+ self._do(["rm", i])
+
+ def clean(self):
+ self._do_kill_instances(False, False)
+ return 0
+
+ def _kill_instances(self):
+ return self._do_kill_instances(True)
+
+ def _output(self, cmd, **kwargs):
+ return subprocess.check_output(self._command + cmd,
+ stderr=subprocess.STDOUT,
+ **kwargs)
+
+ def _guess_command(self):
+ commands = [["docker"], ["sudo", "-n", "docker"]]
+ for cmd in commands:
+ if subprocess.call(cmd + ["images"],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE) == 0:
+ return cmd
+ commands_txt = "\n".join([" " + " ".join(x) for x in commands])
+ raise Exception("Cannot find working docker command. Tried:\n%s" % commands_txt)
+
+ def get_image_dockerfile_checksum(self, tag):
+ resp = self._output(["inspect", tag])
+ labels = json.loads(resp)[0]["Config"].get("Labels", {})
+ return labels.get("com.qemu.dockerfile-checksum", "")
+
+ def checksum(self, text):
+ return hashlib.sha1(text).hexdigest()
+
+ def build_image(self, tag, dockerfile, df, quiet=True):
+ tmp = dockerfile + "\n" + \
+ "LABEL com.qemu.dockerfile-checksum=%s" % self.checksum(dockerfile)
+ tmp_df = df + ".tmp"
+ tmp_file = open(tmp_df, "wb")
+ tmp_file.write(tmp)
+ tmp_file.close()
+ self._do(["build", "-t", tag, "-f", tmp_df, os.path.dirname(df)],
+ quiet=quiet)
+ os.unlink(tmp_df)
+
+ def image_matches_dockerfile(self, tag, dockerfile):
+ try:
+ checksum = self.get_image_dockerfile_checksum(tag)
+ except:
+ return False
+ return checksum == self.checksum(dockerfile)
+
+ def run(self, cmd, keep, quiet):
+ label = uuid.uuid1().hex
+ if not keep:
+ self._instances.append(label)
+ ret = self._do(["run", "--label", "com.qemu.instance.uuid=" + label] + cmd, quiet=quiet)
+ if not keep:
+ self._instances.remove(label)
+ return ret
+
diff --git a/tests/docker/docker_build b/tests/docker/docker_build
new file mode 100755
index 0000000..6948e2c
--- /dev/null
+++ b/tests/docker/docker_build
@@ -0,0 +1,42 @@
+#!/usr/bin/env python2
+#
+# Compare to Dockerfile and rebuild a docker image if necessary.
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+import sys
+import docker
+import argparse
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("tag",
+ help="Image Tag")
+ parser.add_argument("dockerfile",
+ help="Dockerfile name")
+ parser.add_argument("--verbose", "-v", action="store_true",
+ help="Print verbose information")
+ args = parser.parse_args()
+
+ dockerfile = open(args.dockerfile, "rb").read()
+ tag = args.tag
+
+ dkr = docker.Docker()
+ if dkr.image_matches_dockerfile(tag, dockerfile):
+ if args.verbose:
+ print "Image is up to date."
+ return 0
+
+ quiet = not args.verbose
+ dkr.build_image(tag, dockerfile, args.dockerfile, quiet=quiet)
+ return 0
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tests/docker/docker_clean b/tests/docker/docker_clean
new file mode 100755
index 0000000..88cdba6
--- /dev/null
+++ b/tests/docker/docker_clean
@@ -0,0 +1,22 @@
+#!/usr/bin/env python2
+#
+# Clean up uselsee containers.
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+import sys
+import docker
+
+def main():
+ docker.Docker().clean()
+ return 0
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tests/docker/docker_run b/tests/docker/docker_run
new file mode 100755
index 0000000..4c46d90
--- /dev/null
+++ b/tests/docker/docker_run
@@ -0,0 +1,29 @@
+#!/usr/bin/env python2
+#
+# Wrapper for "docker run" with automatical clean up if the execution is
+# iterrupted.
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+import sys
+import argparse
+import docker
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--keep", action="store_true",
+ help="Don't remove image when the command completes")
+ parser.add_argument("--quiet", action="store_true",
+ help="Run quietly unless an error occured")
+ args, argv = parser.parse_known_args()
+ return docker.Docker().run(argv, args.keep, quiet=args.quiet)
+
+if __name__ == "__main__":
+ sys.exit(main())
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* Re: [Qemu-devel] [PATCH v2 01/15] tests: Add utilities for docker testing
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 01/15] tests: Add utilities for docker testing Fam Zheng
@ 2016-02-29 16:46 ` Alex Bennée
2016-03-01 1:12 ` Fam Zheng
0 siblings, 1 reply; 32+ messages in thread
From: Alex Bennée @ 2016-02-29 16:46 UTC (permalink / raw)
To: Fam Zheng
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
Fam Zheng <famz@redhat.com> writes:
> docker_run: A wrapper for "docker run" (or "sudo -n docker run" if
> necessary), which takes care of killing and removing the running
> container at SIGINT.
>
> docker_clean: A tool to tear down all the containers including inactive
> ones that are started by docker_run.
>
> docker_build: A tool to compare an image from given dockerfile and
> rebuild it if they're different.
>
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
> tests/docker/docker.py | 113 ++++++++++++++++++++++++++++++++++++++++++++++
> tests/docker/docker_build | 42 +++++++++++++++++
> tests/docker/docker_clean | 22 +++++++++
> tests/docker/docker_run | 29 ++++++++++++
> 4 files changed, 206 insertions(+)
> create mode 100755 tests/docker/docker.py
> create mode 100755 tests/docker/docker_build
> create mode 100755 tests/docker/docker_clean
> create mode 100755 tests/docker/docker_run
>
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> new file mode 100755
> index 0000000..d175a86
> --- /dev/null
> +++ b/tests/docker/docker.py
> @@ -0,0 +1,113 @@
> +#!/usr/bin/env python2 -B
> +#
> +# Docker controlling module
> +#
> +# Copyright (c) 2016 Red Hat Inc.
> +#
> +# Authors:
> +# Fam Zheng <famz@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2
> +# or (at your option) any later version. See the COPYING file in
> +# the top-level directory.
> +
> +import os
> +import subprocess
> +import json
> +import hashlib
> +import atexit
> +import uuid
> +
> +class ContainerTerminated(Exception):
> + """ Raised if the container has already existed """
> + pass
> +
> +class Docker(object):
> + """ Running Docker commands """
> + def __init__(self):
> + self._command = self._guess_command()
> + self._instances = []
> + atexit.register(self._kill_instances)
> +
> + def _do(self, cmd, quiet=True, **kwargs):
> + if quiet:
> + kwargs["stdout"] = subprocess.PIPE
> + return subprocess.call(self._command + cmd, **kwargs)
> +
> + def _do_kill_instances(self, only_known, only_active=True):
> + cmd = ["ps", "-q"]
Hmm ps -q barfs on my command line:
16:04 alex@zen/x86_64 [qemu.git/mttcg/multi_tcg_v8_ajb-r2] >ps -q
error: unsupported SysV option
Is there not a more portable way of doing this, even if it is a standard
library?
> + if not only_active:
> + cmd.append("-a")
> + for i in self._output(cmd).split():
> + resp = self._output(["inspect", i])
> + labels = json.loads(resp)[0]["Config"]["Labels"]
> + active = json.loads(resp)[0]["State"]["Running"]
> + if not labels:
> + continue
> + instance_uuid = labels.get("com.qemu.instance.uuid", None)
> + if not instance_uuid:
> + continue
> + if only_known and instance_uuid not in self._instances:
> + continue
> + print "Terminating", i
> + if active:
> + self._do(["kill", i])
> + self._do(["rm", i])
> +
> + def clean(self):
> + self._do_kill_instances(False, False)
> + return 0
> +
> + def _kill_instances(self):
> + return self._do_kill_instances(True)
> +
> + def _output(self, cmd, **kwargs):
> + return subprocess.check_output(self._command + cmd,
> + stderr=subprocess.STDOUT,
> + **kwargs)
> +
> + def _guess_command(self):
> + commands = [["docker"], ["sudo", "-n", "docker"]]
> + for cmd in commands:
> + if subprocess.call(cmd + ["images"],
> + stdout=subprocess.PIPE,
> + stderr=subprocess.PIPE) == 0:
> + return cmd
> + commands_txt = "\n".join([" " + " ".join(x) for x in commands])
> + raise Exception("Cannot find working docker command. Tried:\n%s" % commands_txt)
> +
> + def get_image_dockerfile_checksum(self, tag):
> + resp = self._output(["inspect", tag])
> + labels = json.loads(resp)[0]["Config"].get("Labels", {})
> + return labels.get("com.qemu.dockerfile-checksum", "")
> +
> + def checksum(self, text):
> + return hashlib.sha1(text).hexdigest()
> +
> + def build_image(self, tag, dockerfile, df, quiet=True):
> + tmp = dockerfile + "\n" + \
> + "LABEL com.qemu.dockerfile-checksum=%s" % self.checksum(dockerfile)
> + tmp_df = df + ".tmp"
> + tmp_file = open(tmp_df, "wb")
> + tmp_file.write(tmp)
> + tmp_file.close()
> + self._do(["build", "-t", tag, "-f", tmp_df, os.path.dirname(df)],
> + quiet=quiet)
> + os.unlink(tmp_df)
> +
> + def image_matches_dockerfile(self, tag, dockerfile):
> + try:
> + checksum = self.get_image_dockerfile_checksum(tag)
> + except:
> + return False
> + return checksum == self.checksum(dockerfile)
> +
> + def run(self, cmd, keep, quiet):
> + label = uuid.uuid1().hex
> + if not keep:
> + self._instances.append(label)
> + ret = self._do(["run", "--label", "com.qemu.instance.uuid=" + label] + cmd, quiet=quiet)
> + if not keep:
> + self._instances.remove(label)
> + return ret
I think it might be useful to catch some arguments here for testing
things. It is likely to be the first script someone runs while poking
around so some help text would be useful even if it just points at the
other commands.
In fact I'm not sure why all the various commands aren't in one script
for now given this does most of the heavy lifting.
> +
> diff --git a/tests/docker/docker_build b/tests/docker/docker_build
> new file mode 100755
> index 0000000..6948e2c
> --- /dev/null
> +++ b/tests/docker/docker_build
> @@ -0,0 +1,42 @@
> +#!/usr/bin/env python2
> +#
> +# Compare to Dockerfile and rebuild a docker image if necessary.
> +#
> +# Copyright (c) 2016 Red Hat Inc.
> +#
> +# Authors:
> +# Fam Zheng <famz@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2
> +# or (at your option) any later version. See the COPYING file in
> +# the top-level directory.
> +
> +import sys
> +import docker
> +import argparse
> +
> +def main():
> + parser = argparse.ArgumentParser()
> + parser.add_argument("tag",
> + help="Image Tag")
> + parser.add_argument("dockerfile",
> + help="Dockerfile name")
> + parser.add_argument("--verbose", "-v", action="store_true",
> + help="Print verbose information")
> + args = parser.parse_args()
> +
> + dockerfile = open(args.dockerfile, "rb").read()
> + tag = args.tag
> +
> + dkr = docker.Docker()
> + if dkr.image_matches_dockerfile(tag, dockerfile):
> + if args.verbose:
> + print "Image is up to date."
> + return 0
> +
> + quiet = not args.verbose
> + dkr.build_image(tag, dockerfile, args.dockerfile, quiet=quiet)
> + return 0
> +
> +if __name__ == "__main__":
> + sys.exit(main())
> diff --git a/tests/docker/docker_clean b/tests/docker/docker_clean
> new file mode 100755
> index 0000000..88cdba6
> --- /dev/null
> +++ b/tests/docker/docker_clean
> @@ -0,0 +1,22 @@
> +#!/usr/bin/env python2
> +#
> +# Clean up uselsee containers.
> +#
> +# Copyright (c) 2016 Red Hat Inc.
> +#
> +# Authors:
> +# Fam Zheng <famz@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2
> +# or (at your option) any later version. See the COPYING file in
> +# the top-level directory.
> +
> +import sys
> +import docker
> +
> +def main():
> + docker.Docker().clean()
> + return 0
> +
> +if __name__ == "__main__":
> + sys.exit(main())
Of all the scripts run if you call with --help this just does something
straight away. It should at least attempt a usage() text to prevent
accidents.
> diff --git a/tests/docker/docker_run b/tests/docker/docker_run
> new file mode 100755
> index 0000000..4c46d90
> --- /dev/null
> +++ b/tests/docker/docker_run
> @@ -0,0 +1,29 @@
> +#!/usr/bin/env python2
> +#
> +# Wrapper for "docker run" with automatical clean up if the execution is
> +# iterrupted.
> +#
> +# Copyright (c) 2016 Red Hat Inc.
> +#
> +# Authors:
> +# Fam Zheng <famz@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2
> +# or (at your option) any later version. See the COPYING file in
> +# the top-level directory.
> +
> +import sys
> +import argparse
> +import docker
> +
> +def main():
> + parser = argparse.ArgumentParser()
> + parser.add_argument("--keep", action="store_true",
> + help="Don't remove image when the command completes")
> + parser.add_argument("--quiet", action="store_true",
> + help="Run quietly unless an error occured")
> + args, argv = parser.parse_known_args()
> + return docker.Docker().run(argv, args.keep, quiet=args.quiet)
> +
> +if __name__ == "__main__":
> + sys.exit(main())
--
Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread* Re: [Qemu-devel] [PATCH v2 01/15] tests: Add utilities for docker testing
2016-02-29 16:46 ` Alex Bennée
@ 2016-03-01 1:12 ` Fam Zheng
0 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-03-01 1:12 UTC (permalink / raw)
To: Alex Bennée
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
On Mon, 02/29 16:46, Alex Bennée wrote:
>
> Fam Zheng <famz@redhat.com> writes:
>
> > docker_run: A wrapper for "docker run" (or "sudo -n docker run" if
> > necessary), which takes care of killing and removing the running
> > container at SIGINT.
> >
> > docker_clean: A tool to tear down all the containers including inactive
> > ones that are started by docker_run.
> >
> > docker_build: A tool to compare an image from given dockerfile and
> > rebuild it if they're different.
> >
> > Signed-off-by: Fam Zheng <famz@redhat.com>
> > ---
> > tests/docker/docker.py | 113 ++++++++++++++++++++++++++++++++++++++++++++++
> > tests/docker/docker_build | 42 +++++++++++++++++
> > tests/docker/docker_clean | 22 +++++++++
> > tests/docker/docker_run | 29 ++++++++++++
> > 4 files changed, 206 insertions(+)
> > create mode 100755 tests/docker/docker.py
> > create mode 100755 tests/docker/docker_build
> > create mode 100755 tests/docker/docker_clean
> > create mode 100755 tests/docker/docker_run
> >
> > diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> > new file mode 100755
> > index 0000000..d175a86
> > --- /dev/null
> > +++ b/tests/docker/docker.py
> > @@ -0,0 +1,113 @@
> > +#!/usr/bin/env python2 -B
> > +#
> > +# Docker controlling module
> > +#
> > +# Copyright (c) 2016 Red Hat Inc.
> > +#
> > +# Authors:
> > +# Fam Zheng <famz@redhat.com>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2
> > +# or (at your option) any later version. See the COPYING file in
> > +# the top-level directory.
> > +
> > +import os
> > +import subprocess
> > +import json
> > +import hashlib
> > +import atexit
> > +import uuid
> > +
> > +class ContainerTerminated(Exception):
> > + """ Raised if the container has already existed """
> > + pass
> > +
> > +class Docker(object):
> > + """ Running Docker commands """
> > + def __init__(self):
> > + self._command = self._guess_command()
> > + self._instances = []
> > + atexit.register(self._kill_instances)
> > +
> > + def _do(self, cmd, quiet=True, **kwargs):
> > + if quiet:
> > + kwargs["stdout"] = subprocess.PIPE
> > + return subprocess.call(self._command + cmd, **kwargs)
> > +
> > + def _do_kill_instances(self, only_known, only_active=True):
> > + cmd = ["ps", "-q"]
>
> Hmm ps -q barfs on my command line:
>
> 16:04 alex@zen/x86_64 [qemu.git/mttcg/multi_tcg_v8_ajb-r2] >ps -q
> error: unsupported SysV option
>
> Is there not a more portable way of doing this, even if it is a standard
> library?
Down the road this is "sudo docker ps" command. :)
>
> > + if not only_active:
> > + cmd.append("-a")
> > + for i in self._output(cmd).split():
> > + resp = self._output(["inspect", i])
> > + labels = json.loads(resp)[0]["Config"]["Labels"]
> > + active = json.loads(resp)[0]["State"]["Running"]
> > + if not labels:
> > + continue
> > + instance_uuid = labels.get("com.qemu.instance.uuid", None)
> > + if not instance_uuid:
> > + continue
> > + if only_known and instance_uuid not in self._instances:
> > + continue
> > + print "Terminating", i
> > + if active:
> > + self._do(["kill", i])
> > + self._do(["rm", i])
> > +
> > + def clean(self):
> > + self._do_kill_instances(False, False)
> > + return 0
> > +
> > + def _kill_instances(self):
> > + return self._do_kill_instances(True)
> > +
> > + def _output(self, cmd, **kwargs):
> > + return subprocess.check_output(self._command + cmd,
> > + stderr=subprocess.STDOUT,
> > + **kwargs)
> > +
> > + def _guess_command(self):
> > + commands = [["docker"], ["sudo", "-n", "docker"]]
> > + for cmd in commands:
> > + if subprocess.call(cmd + ["images"],
> > + stdout=subprocess.PIPE,
> > + stderr=subprocess.PIPE) == 0:
> > + return cmd
> > + commands_txt = "\n".join([" " + " ".join(x) for x in commands])
> > + raise Exception("Cannot find working docker command. Tried:\n%s" % commands_txt)
> > +
> > + def get_image_dockerfile_checksum(self, tag):
> > + resp = self._output(["inspect", tag])
> > + labels = json.loads(resp)[0]["Config"].get("Labels", {})
> > + return labels.get("com.qemu.dockerfile-checksum", "")
> > +
> > + def checksum(self, text):
> > + return hashlib.sha1(text).hexdigest()
> > +
> > + def build_image(self, tag, dockerfile, df, quiet=True):
> > + tmp = dockerfile + "\n" + \
> > + "LABEL com.qemu.dockerfile-checksum=%s" % self.checksum(dockerfile)
> > + tmp_df = df + ".tmp"
> > + tmp_file = open(tmp_df, "wb")
> > + tmp_file.write(tmp)
> > + tmp_file.close()
> > + self._do(["build", "-t", tag, "-f", tmp_df, os.path.dirname(df)],
> > + quiet=quiet)
> > + os.unlink(tmp_df)
> > +
> > + def image_matches_dockerfile(self, tag, dockerfile):
> > + try:
> > + checksum = self.get_image_dockerfile_checksum(tag)
> > + except:
> > + return False
> > + return checksum == self.checksum(dockerfile)
> > +
> > + def run(self, cmd, keep, quiet):
> > + label = uuid.uuid1().hex
> > + if not keep:
> > + self._instances.append(label)
> > + ret = self._do(["run", "--label", "com.qemu.instance.uuid=" + label] + cmd, quiet=quiet)
> > + if not keep:
> > + self._instances.remove(label)
> > + return ret
>
> I think it might be useful to catch some arguments here for testing
> things. It is likely to be the first script someone runs while poking
> around so some help text would be useful even if it just points at the
> other commands.
Sure, I can do that.
>
> In fact I'm not sure why all the various commands aren't in one script
> for now given this does most of the heavy lifting.
OK, I can merge them into one script.
>
> > +
> > diff --git a/tests/docker/docker_build b/tests/docker/docker_build
> > new file mode 100755
> > index 0000000..6948e2c
> > --- /dev/null
> > +++ b/tests/docker/docker_build
> > @@ -0,0 +1,42 @@
> > +#!/usr/bin/env python2
> > +#
> > +# Compare to Dockerfile and rebuild a docker image if necessary.
> > +#
> > +# Copyright (c) 2016 Red Hat Inc.
> > +#
> > +# Authors:
> > +# Fam Zheng <famz@redhat.com>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2
> > +# or (at your option) any later version. See the COPYING file in
> > +# the top-level directory.
> > +
> > +import sys
> > +import docker
> > +import argparse
> > +
> > +def main():
> > + parser = argparse.ArgumentParser()
> > + parser.add_argument("tag",
> > + help="Image Tag")
> > + parser.add_argument("dockerfile",
> > + help="Dockerfile name")
> > + parser.add_argument("--verbose", "-v", action="store_true",
> > + help="Print verbose information")
> > + args = parser.parse_args()
> > +
> > + dockerfile = open(args.dockerfile, "rb").read()
> > + tag = args.tag
> > +
> > + dkr = docker.Docker()
> > + if dkr.image_matches_dockerfile(tag, dockerfile):
> > + if args.verbose:
> > + print "Image is up to date."
> > + return 0
> > +
> > + quiet = not args.verbose
> > + dkr.build_image(tag, dockerfile, args.dockerfile, quiet=quiet)
> > + return 0
> > +
> > +if __name__ == "__main__":
> > + sys.exit(main())
> > diff --git a/tests/docker/docker_clean b/tests/docker/docker_clean
> > new file mode 100755
> > index 0000000..88cdba6
> > --- /dev/null
> > +++ b/tests/docker/docker_clean
> > @@ -0,0 +1,22 @@
> > +#!/usr/bin/env python2
> > +#
> > +# Clean up uselsee containers.
> > +#
> > +# Copyright (c) 2016 Red Hat Inc.
> > +#
> > +# Authors:
> > +# Fam Zheng <famz@redhat.com>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2
> > +# or (at your option) any later version. See the COPYING file in
> > +# the top-level directory.
> > +
> > +import sys
> > +import docker
> > +
> > +def main():
> > + docker.Docker().clean()
> > + return 0
> > +
> > +if __name__ == "__main__":
> > + sys.exit(main())
>
> Of all the scripts run if you call with --help this just does something
> straight away. It should at least attempt a usage() text to prevent
> accidents.
Yes. Will address.
Fam
>
> > diff --git a/tests/docker/docker_run b/tests/docker/docker_run
> > new file mode 100755
> > index 0000000..4c46d90
> > --- /dev/null
> > +++ b/tests/docker/docker_run
> > @@ -0,0 +1,29 @@
> > +#!/usr/bin/env python2
> > +#
> > +# Wrapper for "docker run" with automatical clean up if the execution is
> > +# iterrupted.
> > +#
> > +# Copyright (c) 2016 Red Hat Inc.
> > +#
> > +# Authors:
> > +# Fam Zheng <famz@redhat.com>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2
> > +# or (at your option) any later version. See the COPYING file in
> > +# the top-level directory.
> > +
> > +import sys
> > +import argparse
> > +import docker
> > +
> > +def main():
> > + parser = argparse.ArgumentParser()
> > + parser.add_argument("--keep", action="store_true",
> > + help="Don't remove image when the command completes")
> > + parser.add_argument("--quiet", action="store_true",
> > + help="Run quietly unless an error occured")
> > + args, argv = parser.parse_known_args()
> > + return docker.Docker().run(argv, args.keep, quiet=args.quiet)
> > +
> > +if __name__ == "__main__":
> > + sys.exit(main())
>
>
> --
> Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread
* [Qemu-devel] [PATCH v2 02/15] Makefile: Rules for docker testing
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 01/15] tests: Add utilities for docker testing Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-29 17:08 ` Alex Bennée
2016-03-01 18:12 ` Alex Bennée
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 03/15] docker: Add images Fam Zheng
` (13 subsequent siblings)
15 siblings, 2 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
This adds a group of make targets to run docker tests, all are available
in source tree without running ./configure.
The usage is shown by "make docker".
Besides the fixed ones, dynamic targets for building each image and
running each test in each image are generated automatically by make,
scanning $(SRC_PATH)/tests/docker/ files with specific patterns.
Alternative to manually list particular targets (docker-run-FOO@BAR)
set, you can control which tests/images to run by filtering variables,
TESTS= and IMAGES=, which are expressed in Makefile pattern syntax,
"foo% %bar ...". For example:
$ make docker-run IMAGES="ubuntu fedora"
Unfortunately, it's impossible to propagate "-j $JOBS" into make in
containers, however since each combination is made a first class target
is the top Makefile, "make -j$N docker-run" still parallels the tests
coarsely.
Signed-off-by: Fam Zheng <famz@redhat.com>
---
Makefile | 4 +-
tests/docker/Makefile.include | 93 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 96 insertions(+), 1 deletion(-)
create mode 100644 tests/docker/Makefile.include
diff --git a/Makefile b/Makefile
index f9fae3a..f104a08 100644
--- a/Makefile
+++ b/Makefile
@@ -6,7 +6,7 @@ BUILD_DIR=$(CURDIR)
# Before including a proper config-host.mak, assume we are in the source tree
SRC_PATH=.
-UNCHECKED_GOALS := %clean TAGS cscope ctags
+UNCHECKED_GOALS := %clean TAGS cscope ctags docker docker-%
# All following code might depend on configuration variables
ifneq ($(wildcard config-host.mak),)
@@ -651,3 +651,5 @@ endif
# Include automatically generated dependency files
# Dependencies in Makefile.objs files come from our recursive subdir rules
-include $(wildcard *.d tests/*.d)
+
+include $(SRC_PATH)/tests/docker/Makefile.include
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
new file mode 100644
index 0000000..d8c55c3
--- /dev/null
+++ b/tests/docker/Makefile.include
@@ -0,0 +1,93 @@
+# Makefile for Docker tests
+
+$(if $(quiet-command),,$(eval include $(SRC_PATH)/rules.mak))
+
+.PHONY: docker docker-test docker-clean docker-image
+
+DOCKER_SUFFIX := .docker
+
+DOCKER_IMAGES := $(notdir $(basename \
+ $(wildcard $(SRC_PATH)/tests/docker/dockerfiles/*)))
+
+DOCKER_TESTS := $(notdir $(shell \
+ find $(SRC_PATH)/tests/docker/ -name 'test-*' -type f -executable))
+
+DOCKER_TOOLS := travis
+
+TESTS ?= %
+IMAGES ?= %
+
+$(foreach i,$(DOCKER_IMAGES), \
+ $(eval docker-image: docker-image-$i) \
+ $(eval docker-image-$i: IMAGE = $i) \
+ $(eval .PHONY: docker-image-$i docker-@$i) \
+ $(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
+ $(eval .PHONY: docker-$t@$i) \
+ $(eval docker-$t@$i: docker-image-$i) \
+ $(eval docker-$t@$i: docker-run-$t@$i) \
+ $(eval docker-@$i: docker-$t@$i) \
+ $(eval docker-test: docker-run-$t@$i) \
+ ) \
+)
+
+docker:
+ @echo 'Building QEMU and running tests inside Docker containers'
+ @echo
+ @echo 'Available targets:'
+ @echo
+ @echo ' docker: Print this help.'
+ @echo ' docker-test: Run all image/test combinations.'
+ @echo ' docker-clean: Kill and remove residual docker testing containers.'
+ @echo ' docker-TEST@IMAGE: Run "TEST" in container "IMAGE".'
+ @echo ' Note: "TEST" is one of the listed test name,'
+ @echo ' or a script name under $$QEMU_SRC/tests/docker/;'
+ @echo ' "IMAGE" is one of the listed container name."'
+ @echo ' docker-@IMAGE: Run all tests in container "IMAGE".'
+ @echo ' docker-image: Build all images.'
+ @echo ' docker-image-IMAGE: Build image "IMG".'
+ @echo
+ @echo 'Available container images:'
+ @echo ' $(DOCKER_IMAGES)'
+ @echo
+ @echo 'Available tests:'
+ @echo ' $(DOCKER_TESTS)'
+ @echo
+ @echo 'Available tools:'
+ @echo ' $(DOCKER_TOOLS)'
+ @echo
+ @echo 'Special variables:'
+ @echo ' IMAGES="a b c ..": Filters which images to build or run.'
+ @echo ' TESTS="x y z .." Filters which tests to run (for docker-test).'
+ @echo ' J=[0..9]* Overrides the -jN parameter for make commands'
+ @echo ' (default is 1)'
+
+docker-image-%:
+ @if test -z "$(IMAGE)"; then echo "Invalid target"; exit 1; fi
+ $(if $(filter $(IMAGES),$(IMAGE)), $(call quiet-command,\
+ $(SRC_PATH)/tests/docker/docker_build qemu:$(IMAGE) \
+ $(SRC_PATH)/tests/docker/dockerfiles/$(IMAGE).docker \
+ $(if $V,-v,), " BUILD $(IMAGE)"))
+
+
+docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')
+docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')
+docker-run-%:
+ @if test -z "$(IMAGE)" || test -z "$(CMD)"; \
+ then echo "Invalid target"; exit 1; \
+ fi
+ $(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \
+ $(call quiet-command,\
+ $(SRC_PATH)/tests/docker/docker_run $(if $(KEEP),,--rm) \
+ --privileged -t --net=none \
+ -v $$(realpath $(SRC_PATH)):/var/tmp/qemu \
+ -e QEMU_SRC=/var/tmp/qemu \
+ -e V=$V -e J=$J \
+ -v /var/tmp/qemu-docker-ccache:/var/tmp/ccache \
+ -e CCACHE_DIR=/var/tmp/ccache \
+ qemu:$(IMAGE) \
+ /var/tmp/qemu/tests/docker/run \
+ /var/tmp/qemu/tests/docker/$(CMD); \
+ , " RUN $(CMD) in $(IMAGE)")))
+
+docker-clean:
+ $(call quiet-command, $(SRC_PATH)/tests/docker/docker_clean)
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* Re: [Qemu-devel] [PATCH v2 02/15] Makefile: Rules for docker testing
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 02/15] Makefile: Rules " Fam Zheng
@ 2016-02-29 17:08 ` Alex Bennée
2016-03-01 1:04 ` Fam Zheng
2016-03-01 18:12 ` Alex Bennée
1 sibling, 1 reply; 32+ messages in thread
From: Alex Bennée @ 2016-02-29 17:08 UTC (permalink / raw)
To: Fam Zheng
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
Fam Zheng <famz@redhat.com> writes:
> This adds a group of make targets to run docker tests, all are available
> in source tree without running ./configure.
>
> The usage is shown by "make docker".
>
> Besides the fixed ones, dynamic targets for building each image and
> running each test in each image are generated automatically by make,
> scanning $(SRC_PATH)/tests/docker/ files with specific patterns.
>
> Alternative to manually list particular targets (docker-run-FOO@BAR)
> set, you can control which tests/images to run by filtering variables,
> TESTS= and IMAGES=, which are expressed in Makefile pattern syntax,
> "foo% %bar ...". For example:
>
> $ make docker-run IMAGES="ubuntu fedora"
This doesn't seem to work for me:
16:49 alex@zen/x86_64 [qemu.git/review/docker-tests-v2] >make
docker-run IMAGES="ubuntu" V=1
/home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: overriding
commands for target `clean-timestamp'
/home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: ignoring old
commands for target `clean-timestamp'
make: *** No rule to make target `docker-run'. Stop.
`
>
> Unfortunately, it's impossible to propagate "-j $JOBS" into make in
> containers, however since each combination is made a first class target
> is the top Makefile, "make -j$N docker-run" still parallels the tests
> coarsely.
>
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
> Makefile | 4 +-
> tests/docker/Makefile.include | 93 +++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 96 insertions(+), 1 deletion(-)
> create mode 100644 tests/docker/Makefile.include
>
> diff --git a/Makefile b/Makefile
> index f9fae3a..f104a08 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -6,7 +6,7 @@ BUILD_DIR=$(CURDIR)
> # Before including a proper config-host.mak, assume we are in the source tree
> SRC_PATH=.
>
> -UNCHECKED_GOALS := %clean TAGS cscope ctags
> +UNCHECKED_GOALS := %clean TAGS cscope ctags docker docker-%
>
> # All following code might depend on configuration variables
> ifneq ($(wildcard config-host.mak),)
> @@ -651,3 +651,5 @@ endif
> # Include automatically generated dependency files
> # Dependencies in Makefile.objs files come from our recursive subdir rules
> -include $(wildcard *.d tests/*.d)
> +
> +include $(SRC_PATH)/tests/docker/Makefile.include
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> new file mode 100644
> index 0000000..d8c55c3
> --- /dev/null
> +++ b/tests/docker/Makefile.include
> @@ -0,0 +1,93 @@
> +# Makefile for Docker tests
> +
> +$(if $(quiet-command),,$(eval include $(SRC_PATH)/rules.mak))
> +
> +.PHONY: docker docker-test docker-clean docker-image
> +
> +DOCKER_SUFFIX := .docker
> +
> +DOCKER_IMAGES := $(notdir $(basename \
> + $(wildcard $(SRC_PATH)/tests/docker/dockerfiles/*)))
> +
> +DOCKER_TESTS := $(notdir $(shell \
> + find $(SRC_PATH)/tests/docker/ -name 'test-*' -type f -executable))
> +
> +DOCKER_TOOLS := travis
> +
> +TESTS ?= %
> +IMAGES ?= %
> +
> +$(foreach i,$(DOCKER_IMAGES), \
> + $(eval docker-image: docker-image-$i) \
> + $(eval docker-image-$i: IMAGE = $i) \
> + $(eval .PHONY: docker-image-$i docker-@$i) \
> + $(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
> + $(eval .PHONY: docker-$t@$i) \
> + $(eval docker-$t@$i: docker-image-$i) \
> + $(eval docker-$t@$i: docker-run-$t@$i) \
> + $(eval docker-@$i: docker-$t@$i) \
> + $(eval docker-test: docker-run-$t@$i) \
> + ) \
> +)
> +
> +docker:
> + @echo 'Building QEMU and running tests inside Docker containers'
> + @echo
> + @echo 'Available targets:'
> + @echo
> + @echo ' docker: Print this help.'
> + @echo ' docker-test: Run all image/test combinations.'
> + @echo ' docker-clean: Kill and remove residual docker testing containers.'
> + @echo ' docker-TEST@IMAGE: Run "TEST" in container "IMAGE".'
> + @echo ' Note: "TEST" is one of the listed test name,'
> + @echo ' or a script name under $$QEMU_SRC/tests/docker/;'
> + @echo ' "IMAGE" is one of the listed container name."'
> + @echo ' docker-@IMAGE: Run all tests in container "IMAGE".'
> + @echo ' docker-image: Build all images.'
> + @echo ' docker-image-IMAGE: Build image "IMG".'
> + @echo
> + @echo 'Available container images:'
> + @echo ' $(DOCKER_IMAGES)'
> + @echo
> + @echo 'Available tests:'
> + @echo ' $(DOCKER_TESTS)'
> + @echo
> + @echo 'Available tools:'
> + @echo ' $(DOCKER_TOOLS)'
> + @echo
> + @echo 'Special variables:'
> + @echo ' IMAGES="a b c ..": Filters which images to build or run.'
> + @echo ' TESTS="x y z .." Filters which tests to run (for docker-test).'
> + @echo ' J=[0..9]* Overrides the -jN parameter for make commands'
> + @echo ' (default is 1)'
OK none of these test targets seem to be working for me:
COPY SRC /tmp/qemu-src.JCUng
RUN test-clang in centos6
Prerequisite 'clang' not present, skip
RUN test-mingw in centos6
Prerequisite 'mingw' not present, skip
RUN test-quick in centos6
make: *** [docker-run-test-quick@centos6] Error 2
Running with V=1 doesn't elucidate me much further:
/home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: overriding commands for target `clean-timestamp'
/home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: ignoring old commands for target `clean-timestamp'
mkdir -p /tmp/qemu-src.HtY32 && (if git diff-index --quiet HEAD -- &>/dev/null; then git archive -1 HEAD --format=tar.gz; else git archive -1 $(git stash create) --format=tar.gz; fi) | tar -xz -C /tmp/qemu-src.HtY32
/home/alex/lsrc/qemu/qemu.git/tests/docker/docker_run --rm --privileged -t --net=none -v $(realpath /home/alex/lsrc/qemu/qemu.git):/var/tmp/qemu -e QEMU_SRC=/var/tmp/qemu -e V=1 -e J= -e PAUSE= -v /var/tmp/qemu-docker-ccache:/var/tmp/ccache -e CCACHE_DIR=/var/tmp/ccache qemu:centos6 /var/tmp/qemu/tests/docker/run /var/tmp/qemu/tests/docker/test-clang;
Prerequisite 'clang' not present, skip
/home/alex/lsrc/qemu/qemu.git/tests/docker/docker_run --rm --privileged -t --net=none -v $(realpath /home/alex/lsrc/qemu/qemu.git):/var/tmp/qemu -e QEMU_SRC=/var/tmp/qemu -e V=1 -e J= -e PAUSE= -v /var/tmp/qemu-docker-ccache:/var/tmp/ccache -e CCACHE_DIR=/var/tmp/ccache qemu:centos6 /var/tmp/qemu/tests/docker/run /var/tmp/qemu/tests/docker/test-mingw;
Prerequisite 'mingw' not present, skip
/home/alex/lsrc/qemu/qemu.git/tests/docker/docker_run --rm --privileged -t --net=none -v $(realpath /home/alex/lsrc/qemu/qemu.git):/var/tmp/qemu -e QEMU_SRC=/var/tmp/qemu -e V=1 -e J= -e PAUSE= -v /var/tmp/qemu-docker-ccache:/var/tmp/ccache -e CCACHE_DIR=/var/tmp/ccache qemu:centos6 /var/tmp/qemu/tests/docker/run /var/tmp/qemu/tests/docker/test-quick;
make: *** [docker-run-test-quick@centos6] Error 2
If I drop to the shell using:
make docker-test-quick@centos6 V=1 PAUSE=1
And run:
/var/tmp/qemu/tests/docker/test-quick
I get a cryptic silent fail. I think we need more verbosity (select-able
when required) to see whats going on in the depths.
> +
> +docker-image-%:
> + @if test -z "$(IMAGE)"; then echo "Invalid target"; exit 1; fi
> + $(if $(filter $(IMAGES),$(IMAGE)), $(call quiet-command,\
> + $(SRC_PATH)/tests/docker/docker_build qemu:$(IMAGE) \
> + $(SRC_PATH)/tests/docker/dockerfiles/$(IMAGE).docker \
> + $(if $V,-v,), " BUILD $(IMAGE)"))
> +
> +
> +docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')
> +docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')
> +docker-run-%:
> + @if test -z "$(IMAGE)" || test -z "$(CMD)"; \
> + then echo "Invalid target"; exit 1; \
> + fi
> + $(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \
> + $(call quiet-command,\
> + $(SRC_PATH)/tests/docker/docker_run $(if $(KEEP),,--rm) \
> + --privileged -t --net=none \
> + -v $$(realpath $(SRC_PATH)):/var/tmp/qemu \
> + -e QEMU_SRC=/var/tmp/qemu \
> + -e V=$V -e J=$J \
> + -v /var/tmp/qemu-docker-ccache:/var/tmp/ccache \
> + -e CCACHE_DIR=/var/tmp/ccache \
> + qemu:$(IMAGE) \
> + /var/tmp/qemu/tests/docker/run \
> + /var/tmp/qemu/tests/docker/$(CMD); \
> + , " RUN $(CMD) in $(IMAGE)")))
> +
> +docker-clean:
> + $(call quiet-command, $(SRC_PATH)/tests/docker/docker_clean)
--
Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Qemu-devel] [PATCH v2 02/15] Makefile: Rules for docker testing
2016-02-29 17:08 ` Alex Bennée
@ 2016-03-01 1:04 ` Fam Zheng
2016-03-01 9:56 ` Alex Bennée
0 siblings, 1 reply; 32+ messages in thread
From: Fam Zheng @ 2016-03-01 1:04 UTC (permalink / raw)
To: Alex Bennée
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
On Mon, 02/29 17:08, Alex Bennée wrote:
>
> Fam Zheng <famz@redhat.com> writes:
>
> > This adds a group of make targets to run docker tests, all are available
> > in source tree without running ./configure.
> >
> > The usage is shown by "make docker".
> >
> > Besides the fixed ones, dynamic targets for building each image and
> > running each test in each image are generated automatically by make,
> > scanning $(SRC_PATH)/tests/docker/ files with specific patterns.
> >
> > Alternative to manually list particular targets (docker-run-FOO@BAR)
> > set, you can control which tests/images to run by filtering variables,
> > TESTS= and IMAGES=, which are expressed in Makefile pattern syntax,
> > "foo% %bar ...". For example:
> >
> > $ make docker-run IMAGES="ubuntu fedora"
>
> This doesn't seem to work for me:
>
> 16:49 alex@zen/x86_64 [qemu.git/review/docker-tests-v2] >make
> docker-run IMAGES="ubuntu" V=1
> /home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: overriding
> commands for target `clean-timestamp'
> /home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: ignoring old
> commands for target `clean-timestamp'
> make: *** No rule to make target `docker-run'. Stop.
The commit message is stale, as the help text, the target is now "docker-test"
which is more intuitive.
Fam
> `
>
> >
> > Unfortunately, it's impossible to propagate "-j $JOBS" into make in
> > containers, however since each combination is made a first class target
> > is the top Makefile, "make -j$N docker-run" still parallels the tests
> > coarsely.
> >
> > Signed-off-by: Fam Zheng <famz@redhat.com>
> > ---
> > Makefile | 4 +-
> > tests/docker/Makefile.include | 93 +++++++++++++++++++++++++++++++++++++++++++
> > 2 files changed, 96 insertions(+), 1 deletion(-)
> > create mode 100644 tests/docker/Makefile.include
> >
> > diff --git a/Makefile b/Makefile
> > index f9fae3a..f104a08 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -6,7 +6,7 @@ BUILD_DIR=$(CURDIR)
> > # Before including a proper config-host.mak, assume we are in the source tree
> > SRC_PATH=.
> >
> > -UNCHECKED_GOALS := %clean TAGS cscope ctags
> > +UNCHECKED_GOALS := %clean TAGS cscope ctags docker docker-%
> >
> > # All following code might depend on configuration variables
> > ifneq ($(wildcard config-host.mak),)
> > @@ -651,3 +651,5 @@ endif
> > # Include automatically generated dependency files
> > # Dependencies in Makefile.objs files come from our recursive subdir rules
> > -include $(wildcard *.d tests/*.d)
> > +
> > +include $(SRC_PATH)/tests/docker/Makefile.include
> > diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> > new file mode 100644
> > index 0000000..d8c55c3
> > --- /dev/null
> > +++ b/tests/docker/Makefile.include
> > @@ -0,0 +1,93 @@
> > +# Makefile for Docker tests
> > +
> > +$(if $(quiet-command),,$(eval include $(SRC_PATH)/rules.mak))
> > +
> > +.PHONY: docker docker-test docker-clean docker-image
> > +
> > +DOCKER_SUFFIX := .docker
> > +
> > +DOCKER_IMAGES := $(notdir $(basename \
> > + $(wildcard $(SRC_PATH)/tests/docker/dockerfiles/*)))
> > +
> > +DOCKER_TESTS := $(notdir $(shell \
> > + find $(SRC_PATH)/tests/docker/ -name 'test-*' -type f -executable))
> > +
> > +DOCKER_TOOLS := travis
> > +
> > +TESTS ?= %
> > +IMAGES ?= %
> > +
> > +$(foreach i,$(DOCKER_IMAGES), \
> > + $(eval docker-image: docker-image-$i) \
> > + $(eval docker-image-$i: IMAGE = $i) \
> > + $(eval .PHONY: docker-image-$i docker-@$i) \
> > + $(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
> > + $(eval .PHONY: docker-$t@$i) \
> > + $(eval docker-$t@$i: docker-image-$i) \
> > + $(eval docker-$t@$i: docker-run-$t@$i) \
> > + $(eval docker-@$i: docker-$t@$i) \
> > + $(eval docker-test: docker-run-$t@$i) \
> > + ) \
> > +)
> > +
> > +docker:
> > + @echo 'Building QEMU and running tests inside Docker containers'
> > + @echo
> > + @echo 'Available targets:'
> > + @echo
> > + @echo ' docker: Print this help.'
> > + @echo ' docker-test: Run all image/test combinations.'
> > + @echo ' docker-clean: Kill and remove residual docker testing containers.'
> > + @echo ' docker-TEST@IMAGE: Run "TEST" in container "IMAGE".'
> > + @echo ' Note: "TEST" is one of the listed test name,'
> > + @echo ' or a script name under $$QEMU_SRC/tests/docker/;'
> > + @echo ' "IMAGE" is one of the listed container name."'
> > + @echo ' docker-@IMAGE: Run all tests in container "IMAGE".'
> > + @echo ' docker-image: Build all images.'
> > + @echo ' docker-image-IMAGE: Build image "IMG".'
> > + @echo
> > + @echo 'Available container images:'
> > + @echo ' $(DOCKER_IMAGES)'
> > + @echo
> > + @echo 'Available tests:'
> > + @echo ' $(DOCKER_TESTS)'
> > + @echo
> > + @echo 'Available tools:'
> > + @echo ' $(DOCKER_TOOLS)'
> > + @echo
> > + @echo 'Special variables:'
> > + @echo ' IMAGES="a b c ..": Filters which images to build or run.'
> > + @echo ' TESTS="x y z .." Filters which tests to run (for docker-test).'
> > + @echo ' J=[0..9]* Overrides the -jN parameter for make commands'
> > + @echo ' (default is 1)'
>
> OK none of these test targets seem to be working for me:
>
> COPY SRC /tmp/qemu-src.JCUng
> RUN test-clang in centos6
> Prerequisite 'clang' not present, skip
> RUN test-mingw in centos6
> Prerequisite 'mingw' not present, skip
> RUN test-quick in centos6
> make: *** [docker-run-test-quick@centos6] Error 2
>
> Running with V=1 doesn't elucidate me much further:
>
> /home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: overriding commands for target `clean-timestamp'
> /home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: ignoring old commands for target `clean-timestamp'
> mkdir -p /tmp/qemu-src.HtY32 && (if git diff-index --quiet HEAD -- &>/dev/null; then git archive -1 HEAD --format=tar.gz; else git archive -1 $(git stash create) --format=tar.gz; fi) | tar -xz -C /tmp/qemu-src.HtY32
> /home/alex/lsrc/qemu/qemu.git/tests/docker/docker_run --rm --privileged -t --net=none -v $(realpath /home/alex/lsrc/qemu/qemu.git):/var/tmp/qemu -e QEMU_SRC=/var/tmp/qemu -e V=1 -e J= -e PAUSE= -v /var/tmp/qemu-docker-ccache:/var/tmp/ccache -e CCACHE_DIR=/var/tmp/ccache qemu:centos6 /var/tmp/qemu/tests/docker/run /var/tmp/qemu/tests/docker/test-clang;
> Prerequisite 'clang' not present, skip
> /home/alex/lsrc/qemu/qemu.git/tests/docker/docker_run --rm --privileged -t --net=none -v $(realpath /home/alex/lsrc/qemu/qemu.git):/var/tmp/qemu -e QEMU_SRC=/var/tmp/qemu -e V=1 -e J= -e PAUSE= -v /var/tmp/qemu-docker-ccache:/var/tmp/ccache -e CCACHE_DIR=/var/tmp/ccache qemu:centos6 /var/tmp/qemu/tests/docker/run /var/tmp/qemu/tests/docker/test-mingw;
> Prerequisite 'mingw' not present, skip
> /home/alex/lsrc/qemu/qemu.git/tests/docker/docker_run --rm --privileged -t --net=none -v $(realpath /home/alex/lsrc/qemu/qemu.git):/var/tmp/qemu -e QEMU_SRC=/var/tmp/qemu -e V=1 -e J= -e PAUSE= -v /var/tmp/qemu-docker-ccache:/var/tmp/ccache -e CCACHE_DIR=/var/tmp/ccache qemu:centos6 /var/tmp/qemu/tests/docker/run /var/tmp/qemu/tests/docker/test-quick;
> make: *** [docker-run-test-quick@centos6] Error 2
>
> If I drop to the shell using:
>
> make docker-test-quick@centos6 V=1 PAUSE=1
>
> And run:
>
> /var/tmp/qemu/tests/docker/test-quick
>
> I get a cryptic silent fail. I think we need more verbosity (select-able
> when required) to see whats going on in the depths.
>
> > +
> > +docker-image-%:
> > + @if test -z "$(IMAGE)"; then echo "Invalid target"; exit 1; fi
> > + $(if $(filter $(IMAGES),$(IMAGE)), $(call quiet-command,\
> > + $(SRC_PATH)/tests/docker/docker_build qemu:$(IMAGE) \
> > + $(SRC_PATH)/tests/docker/dockerfiles/$(IMAGE).docker \
> > + $(if $V,-v,), " BUILD $(IMAGE)"))
> > +
> > +
> > +docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')
> > +docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')
> > +docker-run-%:
> > + @if test -z "$(IMAGE)" || test -z "$(CMD)"; \
> > + then echo "Invalid target"; exit 1; \
> > + fi
> > + $(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \
> > + $(call quiet-command,\
> > + $(SRC_PATH)/tests/docker/docker_run $(if $(KEEP),,--rm) \
> > + --privileged -t --net=none \
> > + -v $$(realpath $(SRC_PATH)):/var/tmp/qemu \
> > + -e QEMU_SRC=/var/tmp/qemu \
> > + -e V=$V -e J=$J \
> > + -v /var/tmp/qemu-docker-ccache:/var/tmp/ccache \
> > + -e CCACHE_DIR=/var/tmp/ccache \
> > + qemu:$(IMAGE) \
> > + /var/tmp/qemu/tests/docker/run \
> > + /var/tmp/qemu/tests/docker/$(CMD); \
> > + , " RUN $(CMD) in $(IMAGE)")))
> > +
> > +docker-clean:
> > + $(call quiet-command, $(SRC_PATH)/tests/docker/docker_clean)
>
>
> --
> Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Qemu-devel] [PATCH v2 02/15] Makefile: Rules for docker testing
2016-03-01 1:04 ` Fam Zheng
@ 2016-03-01 9:56 ` Alex Bennée
0 siblings, 0 replies; 32+ messages in thread
From: Alex Bennée @ 2016-03-01 9:56 UTC (permalink / raw)
To: Fam Zheng
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
Fam Zheng <famz@redhat.com> writes:
> On Mon, 02/29 17:08, Alex Bennée wrote:
>>
>> Fam Zheng <famz@redhat.com> writes:
>>
>> > This adds a group of make targets to run docker tests, all are available
>> > in source tree without running ./configure.
>> >
>> > The usage is shown by "make docker".
>> >
>> > Besides the fixed ones, dynamic targets for building each image and
>> > running each test in each image are generated automatically by make,
>> > scanning $(SRC_PATH)/tests/docker/ files with specific patterns.
>> >
>> > Alternative to manually list particular targets (docker-run-FOO@BAR)
>> > set, you can control which tests/images to run by filtering variables,
>> > TESTS= and IMAGES=, which are expressed in Makefile pattern syntax,
>> > "foo% %bar ...". For example:
>> >
>> > $ make docker-run IMAGES="ubuntu fedora"
>>
>> This doesn't seem to work for me:
>>
>> 16:49 alex@zen/x86_64 [qemu.git/review/docker-tests-v2] >make
>> docker-run IMAGES="ubuntu" V=1
>> /home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: overriding
>> commands for target `clean-timestamp'
>> /home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: ignoring old
>> commands for target `clean-timestamp'
>> make: *** No rule to make target `docker-run'. Stop.
>
> The commit message is stale, as the help text, the target is now "docker-test"
> which is more intuitive.
>
> Fam
>
>> `
>>
>> >
>> > Unfortunately, it's impossible to propagate "-j $JOBS" into make in
>> > containers, however since each combination is made a first class target
>> > is the top Makefile, "make -j$N docker-run" still parallels the tests
>> > coarsely.
>> >
>> > Signed-off-by: Fam Zheng <famz@redhat.com>
>> > ---
>> > Makefile | 4 +-
>> > tests/docker/Makefile.include | 93 +++++++++++++++++++++++++++++++++++++++++++
>> > 2 files changed, 96 insertions(+), 1 deletion(-)
>> > create mode 100644 tests/docker/Makefile.include
>> >
>> > diff --git a/Makefile b/Makefile
>> > index f9fae3a..f104a08 100644
>> > --- a/Makefile
>> > +++ b/Makefile
>> > @@ -6,7 +6,7 @@ BUILD_DIR=$(CURDIR)
>> > # Before including a proper config-host.mak, assume we are in the source tree
>> > SRC_PATH=.
>> >
>> > -UNCHECKED_GOALS := %clean TAGS cscope ctags
>> > +UNCHECKED_GOALS := %clean TAGS cscope ctags docker docker-%
>> >
>> > # All following code might depend on configuration variables
>> > ifneq ($(wildcard config-host.mak),)
>> > @@ -651,3 +651,5 @@ endif
>> > # Include automatically generated dependency files
>> > # Dependencies in Makefile.objs files come from our recursive subdir rules
>> > -include $(wildcard *.d tests/*.d)
>> > +
>> > +include $(SRC_PATH)/tests/docker/Makefile.include
>> > diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> > new file mode 100644
>> > index 0000000..d8c55c3
>> > --- /dev/null
>> > +++ b/tests/docker/Makefile.include
>> > @@ -0,0 +1,93 @@
>> > +# Makefile for Docker tests
>> > +
>> > +$(if $(quiet-command),,$(eval include $(SRC_PATH)/rules.mak))
>> > +
>> > +.PHONY: docker docker-test docker-clean docker-image
>> > +
>> > +DOCKER_SUFFIX := .docker
>> > +
>> > +DOCKER_IMAGES := $(notdir $(basename \
>> > + $(wildcard $(SRC_PATH)/tests/docker/dockerfiles/*)))
>> > +
>> > +DOCKER_TESTS := $(notdir $(shell \
>> > + find $(SRC_PATH)/tests/docker/ -name 'test-*' -type f -executable))
>> > +
>> > +DOCKER_TOOLS := travis
>> > +
>> > +TESTS ?= %
>> > +IMAGES ?= %
>> > +
>> > +$(foreach i,$(DOCKER_IMAGES), \
>> > + $(eval docker-image: docker-image-$i) \
>> > + $(eval docker-image-$i: IMAGE = $i) \
>> > + $(eval .PHONY: docker-image-$i docker-@$i) \
>> > + $(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
>> > + $(eval .PHONY: docker-$t@$i) \
>> > + $(eval docker-$t@$i: docker-image-$i) \
>> > + $(eval docker-$t@$i: docker-run-$t@$i) \
>> > + $(eval docker-@$i: docker-$t@$i) \
>> > + $(eval docker-test: docker-run-$t@$i) \
>> > + ) \
>> > +)
Also I think you need rules for updating the images here. I just updated
ubuntu.docker to add libfdt-dev but on running on the command line:
09:53 alex@zen/x86_64 [qemu.git/review/docker-tests-v2] >make docker-image-ubuntu V=1
/home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: overriding commands for target `clean-timestamp'
/home/alex/lsrc/qemu/qemu.git/rules.mak:178: warning: ignoring old commands for target `clean-timestamp'
make: Nothing to be done for `docker-image-ubuntu'.
--
Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Qemu-devel] [PATCH v2 02/15] Makefile: Rules for docker testing
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 02/15] Makefile: Rules " Fam Zheng
2016-02-29 17:08 ` Alex Bennée
@ 2016-03-01 18:12 ` Alex Bennée
1 sibling, 0 replies; 32+ messages in thread
From: Alex Bennée @ 2016-03-01 18:12 UTC (permalink / raw)
To: Fam Zheng
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
[-- Attachment #1: Type: text/plain, Size: 506 bytes --]
Fam Zheng <famz@redhat.com> writes:
> This adds a group of make targets to run docker tests, all are available
> in source tree without running ./configure.
>
> The usage is shown by "make docker".
>
<snip>
OK I've made some tweaks which I think improve the generation and allow
for re-creation of containers when the rules change. I still need an
easy way to see the failed build when it does fail. I think this
requires "docker logs" magic.
Anyway the current state of my Makefile.include attached:
[-- Attachment #2: Current hacked up makfile.include --]
[-- Type: application/octet-stream, Size: 4017 bytes --]
# Makefile for Docker tests
$(if $(quiet-command),,$(eval include $(SRC_PATH)/rules.mak))
.PHONY: docker docker-test docker-clean docker-image docker-qemu-src
DOCKER_SUFFIX := .docker
DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
DOCKER_IMAGES := $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*)))
DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
$(warning DOCKER_FILES_DIR=${DOCKER_FILES_DIR})
$(warning DOCKER_IMAGES=${DOCKER_IMAGES})
$(warning DOCKER_TARGETS=${DOCKER_TARGETS})
DOCKER_TESTS := $(notdir $(shell \
find $(SRC_PATH)/tests/docker/ -name 'test-*' -type f -executable))
DOCKER_TOOLS := travis
TESTS ?= %
IMAGES ?= %
SRC_COPY := $(shell mktemp -u /tmp/qemu-src.XXXXX)
.PHONY: $(SRC_COPY)
$(SRC_COPY):
@mkdir -p $@
$(call quiet-command, \
mkdir -p $@ && (if git diff-index --quiet HEAD -- &>/dev/null; then \
git archive -1 HEAD --format=tar.gz; \
else \
git archive -1 $$(git stash create) --format=tar.gz; \
fi) | tar -xz -C $@, \
" COPY SRC $@")
docker-qemu-src: $(SRC_COPY)
.PHONY: docker-image
docker-image: ${DOCKER_TARGETS}
# General rule for building docker images
docker-image-%: $(DOCKER_FILES_DIR)/%.docker
$(call quiet-command,\
$(SRC_PATH)/tests/docker/docker_build qemu:$* $< \
$(if $V,-v,), " BUILD $(IMAGE)")
# Expand all the pre-requistes for each docker image and test combination
$(foreach i,$(DOCKER_IMAGES), \
$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
$(eval .PHONY: docker-$t@$i) \
$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
$(eval docker-@$i: docker-$t@$i) \
$(eval docker-test: docker-run-$t@$i) \
) \
)
docker:
@echo 'Building QEMU and running tests inside Docker containers'
@echo
@echo 'Available targets:'
@echo
@echo ' docker: Print this help.'
@echo ' docker-test: Run all image/test combinations.'
@echo ' docker-clean: Kill and remove residual docker testing containers.'
@echo ' docker-TEST@IMAGE: Run "TEST" in container "IMAGE".'
@echo ' Note: "TEST" is one of the listed test name,'
@echo ' or a script name under $$QEMU_SRC/tests/docker/;'
@echo ' "IMAGE" is one of the listed container name."'
@echo ' docker-@IMAGE: Run all tests in container "IMAGE".'
@echo ' docker-image: Build all images.'
@echo ' docker-image-IMAGE: Build image "IMG".'
@echo
@echo 'Available container images:'
@echo ' $(DOCKER_IMAGES)'
@echo
@echo 'Available tests:'
@echo ' $(DOCKER_TESTS)'
@echo
@echo 'Available tools:'
@echo ' $(DOCKER_TOOLS)'
@echo
@echo 'Special variables:'
@echo ' IMAGES="a b c ..": Filters which images to build or run.'
@echo ' TESTS="x y z .." Filters which tests to run (for docker-test).'
@echo ' J=[0..9]* Overrides the -jN parameter for make commands'
@echo ' (default is 1)'
@echo ' PAUSE=1 Stop and drop to shell in the created container'
@echo ' before running the command.'
docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')
docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')
docker-run-%: docker-qemu-src
@if test -z "$(IMAGE)" || test -z "$(CMD)"; \
then echo "Invalid target"; exit 1; \
fi
$(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \
$(call quiet-command,\
$(SRC_PATH)/tests/docker/docker_run $(if $V,,--rm) \
--privileged -t --net=none \
$(if $(PAUSE),-i) \
-v $$(realpath $(SRC_PATH)):/var/tmp/qemu \
-e QEMU_SRC=/var/tmp/qemu \
-e V=$V -e J=$J -e PAUSE=$(PAUSE)\
-v /var/tmp/qemu-docker-ccache:/var/tmp/ccache \
-e CCACHE_DIR=/var/tmp/ccache \
qemu:$(IMAGE) \
/var/tmp/qemu/tests/docker/run \
/var/tmp/qemu/tests/docker/$(CMD); \
, " RUN $(CMD) in $(IMAGE)")))
docker-clean:
$(call quiet-command, $(SRC_PATH)/tests/docker/docker_clean)
[-- Attachment #3: Type: text/plain, Size: 18 bytes --]
--
Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread
* [Qemu-devel] [PATCH v2 03/15] docker: Add images
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 01/15] tests: Add utilities for docker testing Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 02/15] Makefile: Rules " Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-29 17:18 ` Alex Bennée
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 04/15] docker: Add test runner Fam Zheng
` (12 subsequent siblings)
15 siblings, 1 reply; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/dockerfiles/centos6.docker | 5 +++++
tests/docker/dockerfiles/fedora.docker | 7 +++++++
tests/docker/dockerfiles/ubuntu.docker | 9 +++++++++
3 files changed, 21 insertions(+)
create mode 100644 tests/docker/dockerfiles/centos6.docker
create mode 100644 tests/docker/dockerfiles/fedora.docker
create mode 100644 tests/docker/dockerfiles/ubuntu.docker
diff --git a/tests/docker/dockerfiles/centos6.docker b/tests/docker/dockerfiles/centos6.docker
new file mode 100644
index 0000000..7daa2fb
--- /dev/null
+++ b/tests/docker/dockerfiles/centos6.docker
@@ -0,0 +1,5 @@
+FROM centos:6
+RUN yum install -y \
+ git make gcc g++ \
+ zlib-devel glib2-devel SDL-devel pixman-devel
+
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
new file mode 100644
index 0000000..81091b7
--- /dev/null
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -0,0 +1,7 @@
+FROM fedora:23
+RUN dnf install -y \
+ ccache git \
+ glib2-devel pixman-devel zlib-devel SDL-devel \
+ gcc gcc-c++ clang make perl which bc findutils \
+ mingw{32,64}-{pixman,glib2,gmp,SDL,pkg-config,gtk2,gtk3,gnutls,nettle,libtasn1,libjpeg-turbo,libpng,curl,libssh2,bzip2}
+ENV FEATURES mingw clang
diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
new file mode 100644
index 0000000..f1dc518
--- /dev/null
+++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -0,0 +1,9 @@
+FROM ubuntu:14.04
+RUN apt-get update
+RUN apt-get -y install \
+ libusb-1.0-0-dev libiscsi-dev librados-dev libncurses5-dev \
+ libseccomp-dev libgnutls-dev libssh2-1-dev libspice-server-dev \
+ libspice-protocol-dev libnss3-dev \
+ libgtk-3-dev libvte-2.90-dev libsdl1.2-dev libpng12-dev libpixman-1-dev \
+ git make ccache python-yaml gcc clang
+ENV FEATURES clang ccache pyyaml
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* Re: [Qemu-devel] [PATCH v2 03/15] docker: Add images
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 03/15] docker: Add images Fam Zheng
@ 2016-02-29 17:18 ` Alex Bennée
2016-03-01 1:19 ` Fam Zheng
0 siblings, 1 reply; 32+ messages in thread
From: Alex Bennée @ 2016-02-29 17:18 UTC (permalink / raw)
To: Fam Zheng
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
Fam Zheng <famz@redhat.com> writes:
> Signed-off-by: Fam Zheng <famz@redhat.com>
I think we need to include the dtc libs in here unless we need to ship
all submodules in the tree as well.
Is there a way to get a plain install shell? PAUSE=1 on the test shells
doesn't seem to allow me to install anything.
> ---
> tests/docker/dockerfiles/centos6.docker | 5 +++++
> tests/docker/dockerfiles/fedora.docker | 7 +++++++
> tests/docker/dockerfiles/ubuntu.docker | 9 +++++++++
> 3 files changed, 21 insertions(+)
> create mode 100644 tests/docker/dockerfiles/centos6.docker
> create mode 100644 tests/docker/dockerfiles/fedora.docker
> create mode 100644 tests/docker/dockerfiles/ubuntu.docker
>
> diff --git a/tests/docker/dockerfiles/centos6.docker b/tests/docker/dockerfiles/centos6.docker
> new file mode 100644
> index 0000000..7daa2fb
> --- /dev/null
> +++ b/tests/docker/dockerfiles/centos6.docker
> @@ -0,0 +1,5 @@
> +FROM centos:6
> +RUN yum install -y \
> + git make gcc g++ \
> + zlib-devel glib2-devel SDL-devel pixman-devel
> +
> diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
> new file mode 100644
> index 0000000..81091b7
> --- /dev/null
> +++ b/tests/docker/dockerfiles/fedora.docker
> @@ -0,0 +1,7 @@
> +FROM fedora:23
> +RUN dnf install -y \
> + ccache git \
> + glib2-devel pixman-devel zlib-devel SDL-devel \
> + gcc gcc-c++ clang make perl which bc findutils \
> + mingw{32,64}-{pixman,glib2,gmp,SDL,pkg-config,gtk2,gtk3,gnutls,nettle,libtasn1,libjpeg-turbo,libpng,curl,libssh2,bzip2}
> +ENV FEATURES mingw clang
> diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
> new file mode 100644
> index 0000000..f1dc518
> --- /dev/null
> +++ b/tests/docker/dockerfiles/ubuntu.docker
> @@ -0,0 +1,9 @@
> +FROM ubuntu:14.04
> +RUN apt-get update
> +RUN apt-get -y install \
> + libusb-1.0-0-dev libiscsi-dev librados-dev libncurses5-dev \
> + libseccomp-dev libgnutls-dev libssh2-1-dev libspice-server-dev \
> + libspice-protocol-dev libnss3-dev \
> + libgtk-3-dev libvte-2.90-dev libsdl1.2-dev libpng12-dev libpixman-1-dev \
> + git make ccache python-yaml gcc clang
> +ENV FEATURES clang ccache pyyaml
--
Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread* Re: [Qemu-devel] [PATCH v2 03/15] docker: Add images
2016-02-29 17:18 ` Alex Bennée
@ 2016-03-01 1:19 ` Fam Zheng
2016-03-01 9:52 ` Alex Bennée
0 siblings, 1 reply; 32+ messages in thread
From: Fam Zheng @ 2016-03-01 1:19 UTC (permalink / raw)
To: Alex Bennée
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
On Mon, 02/29 17:18, Alex Bennée wrote:
>
> Fam Zheng <famz@redhat.com> writes:
>
> > Signed-off-by: Fam Zheng <famz@redhat.com>
>
> I think we need to include the dtc libs in here unless we need to ship
> all submodules in the tree as well.
OK, will take a look.
>
> Is there a way to get a plain install shell? PAUSE=1 on the test shells
> doesn't seem to allow me to install anything.
What is the error when you install things? I think it is an unmodified shell
and "yum" or "apt-get" should just work. It's just that when the container
exits the changes will be discarded.
Fam
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Qemu-devel] [PATCH v2 03/15] docker: Add images
2016-03-01 1:19 ` Fam Zheng
@ 2016-03-01 9:52 ` Alex Bennée
2016-03-02 2:23 ` Fam Zheng
0 siblings, 1 reply; 32+ messages in thread
From: Alex Bennée @ 2016-03-01 9:52 UTC (permalink / raw)
To: Fam Zheng
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
Fam Zheng <famz@redhat.com> writes:
> On Mon, 02/29 17:18, Alex Bennée wrote:
>>
>> Fam Zheng <famz@redhat.com> writes:
>>
>> > Signed-off-by: Fam Zheng <famz@redhat.com>
>>
>> I think we need to include the dtc libs in here unless we need to ship
>> all submodules in the tree as well.
>
> OK, will take a look.
>
>>
>> Is there a way to get a plain install shell? PAUSE=1 on the test shells
>> doesn't seem to allow me to install anything.
>
> What is the error when you install things? I think it is an unmodified shell
> and "yum" or "apt-get" should just work. It's just that when the container
> exits the changes will be discarded.
So running:
make docker-test-quick@ubuntu PAUSE=1
And I got a shell I wanted to try and get working:
root@72d09615d5b2:/var/tmp/qemu.tmp# ./configure
ERROR: DTC (libfdt) version >= 1.4.0 not present. Your options:
(1) Preferred: Install the DTC (libfdt) devel package
(2) Fetch the DTC submodule, using:
git submodule update --init dtc
root@72d09615d5b2:/var/tmp/qemu.tmp# apt-cache search libfdt
libfdt-dev - Flat Device Trees manipulation library - development
files
libfdt1 - Flat Device Trees manipulation library
root@72d09615d5b2:/var/tmp/qemu.tmp# apt-get install libfdt-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libfdt1
The following NEW packages will be installed:
libfdt-dev libfdt1
0 upgraded, 2 newly installed, 0 to remove and 10 not upgraded.
Need to get 37.8 kB of archives.
After this operation, 216 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Err http://archive.ubuntu.com/ubuntu/ trusty/main libfdt1 amd64
1.4.0+dfsg-1
Could not resolve 'archive.ubuntu.com'
Err http://archive.ubuntu.com/ubuntu/ trusty/main libfdt-dev amd64
1.4.0+dfsg-1
Could not resolve 'archive.ubuntu.com'
E: Failed to fetch
http://archive.ubuntu.com/ubuntu/pool/main/d/device-tree-compiler/libfdt1_1.4.0+dfsg-1_amd64.deb
Could not resolve 'archive.ubuntu.com'
E: Failed to fetch
http://archive.ubuntu.com/ubuntu/pool/main/d/device-tree-compiler/libfdt-dev_1.4.0+dfsg-1_amd64.deb
Could not resolve 'archive.ubuntu.com'
E: Unable to fetch some archives, maybe run apt-get update or try with
--fix-missing?
root@72d09615d5b2:/var/tmp/qemu.tmp# apt-get update
Err http://archive.ubuntu.com trusty InRelease
Err http://archive.ubuntu.com trusty-updates InRelease
Err http://archive.ubuntu.com trusty-security InRelease
Err http://archive.ubuntu.com trusty Release.gpg
Could not resolve 'archive.ubuntu.com'
Err http://archive.ubuntu.com trusty-updates Release.gpg
Could not resolve 'archive.ubuntu.com'
Err http://archive.ubuntu.com trusty-security Release.gpg
Could not resolve 'archive.ubuntu.com'
Reading package lists... Done
W: Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/trusty/InRelease
W: Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InRelease
W: Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease
W: Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/trusty/Release.gpg Could not
resolve 'archive.ubuntu.com'
W: Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/trusty-updates/Release.gpg
Could not resolve 'archive.ubuntu.com'
W: Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/trusty-security/Release.gpg
Could not resolve 'archive.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old
ones used instead.
So it looks like networking isn't working in the test images. For
debugging problems I think we need to be able to install stuff on a
running image so we can then fix the base recipes.
>
> Fam
--
Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Qemu-devel] [PATCH v2 03/15] docker: Add images
2016-03-01 9:52 ` Alex Bennée
@ 2016-03-02 2:23 ` Fam Zheng
2016-03-02 7:26 ` Alex Bennée
0 siblings, 1 reply; 32+ messages in thread
From: Fam Zheng @ 2016-03-02 2:23 UTC (permalink / raw)
To: Alex Bennée
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
On Tue, 03/01 09:52, Alex Bennée wrote:
>
> Fam Zheng <famz@redhat.com> writes:
>
> > On Mon, 02/29 17:18, Alex Bennée wrote:
> >>
> >> Fam Zheng <famz@redhat.com> writes:
> >>
> >> > Signed-off-by: Fam Zheng <famz@redhat.com>
> >>
> >> I think we need to include the dtc libs in here unless we need to ship
> >> all submodules in the tree as well.
> >
> > OK, will take a look.
> >
> >>
> >> Is there a way to get a plain install shell? PAUSE=1 on the test shells
> >> doesn't seem to allow me to install anything.
> >
> > What is the error when you install things? I think it is an unmodified shell
> > and "yum" or "apt-get" should just work. It's just that when the container
> > exits the changes will be discarded.
>
> So running:
>
> make docker-test-quick@ubuntu PAUSE=1
>
> And I got a shell I wanted to try and get working:
>
> root@72d09615d5b2:/var/tmp/qemu.tmp# ./configure
>
> ERROR: DTC (libfdt) version >= 1.4.0 not present. Your options:
> (1) Preferred: Install the DTC (libfdt) devel package
> (2) Fetch the DTC submodule, using:
> git submodule update --init dtc
>
> root@72d09615d5b2:/var/tmp/qemu.tmp# apt-cache search libfdt
> libfdt-dev - Flat Device Trees manipulation library - development
> files
> libfdt1 - Flat Device Trees manipulation library
> root@72d09615d5b2:/var/tmp/qemu.tmp# apt-get install libfdt-dev
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> The following extra packages will be installed:
> libfdt1
> The following NEW packages will be installed:
> libfdt-dev libfdt1
> 0 upgraded, 2 newly installed, 0 to remove and 10 not upgraded.
> Need to get 37.8 kB of archives.
> After this operation, 216 kB of additional disk space will be used.
> Do you want to continue? [Y/n]
> Err http://archive.ubuntu.com/ubuntu/ trusty/main libfdt1 amd64
> 1.4.0+dfsg-1
> Could not resolve 'archive.ubuntu.com'
> Err http://archive.ubuntu.com/ubuntu/ trusty/main libfdt-dev amd64
> 1.4.0+dfsg-1
> Could not resolve 'archive.ubuntu.com'
> E: Failed to fetch
> http://archive.ubuntu.com/ubuntu/pool/main/d/device-tree-compiler/libfdt1_1.4.0+dfsg-1_amd64.deb
> Could not resolve 'archive.ubuntu.com'
>
> E: Failed to fetch
> http://archive.ubuntu.com/ubuntu/pool/main/d/device-tree-compiler/libfdt-dev_1.4.0+dfsg-1_amd64.deb
> Could not resolve 'archive.ubuntu.com'
>
> E: Unable to fetch some archives, maybe run apt-get update or try with
> --fix-missing?
> root@72d09615d5b2:/var/tmp/qemu.tmp# apt-get update
> Err http://archive.ubuntu.com trusty InRelease
>
> Err http://archive.ubuntu.com trusty-updates InRelease
>
> Err http://archive.ubuntu.com trusty-security InRelease
>
> Err http://archive.ubuntu.com trusty Release.gpg
> Could not resolve 'archive.ubuntu.com'
> Err http://archive.ubuntu.com trusty-updates Release.gpg
> Could not resolve 'archive.ubuntu.com'
> Err http://archive.ubuntu.com trusty-security Release.gpg
> Could not resolve 'archive.ubuntu.com'
> Reading package lists... Done
> W: Failed to fetch
> http://archive.ubuntu.com/ubuntu/dists/trusty/InRelease
>
> W: Failed to fetch
> http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InRelease
>
> W: Failed to fetch
> http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease
>
> W: Failed to fetch
> http://archive.ubuntu.com/ubuntu/dists/trusty/Release.gpg Could not
> resolve 'archive.ubuntu.com'
>
> W: Failed to fetch
> http://archive.ubuntu.com/ubuntu/dists/trusty-updates/Release.gpg
> Could not resolve 'archive.ubuntu.com'
>
> W: Failed to fetch
> http://archive.ubuntu.com/ubuntu/dists/trusty-security/Release.gpg
> Could not resolve 'archive.ubuntu.com'
>
> W: Some index files failed to download. They have been ignored, or old
> ones used instead.
>
>
> So it looks like networking isn't working in the test images. For
> debugging problems I think we need to be able to install stuff on a
> running image so we can then fix the base recipes.
Oh yes, I added "-net=none" to the "docker run" command for better isolation.
So in this case we should enable it. Let's rename PAUSE=1 to DEBUG=1 and
enable network.
Fam
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Qemu-devel] [PATCH v2 03/15] docker: Add images
2016-03-02 2:23 ` Fam Zheng
@ 2016-03-02 7:26 ` Alex Bennée
0 siblings, 0 replies; 32+ messages in thread
From: Alex Bennée @ 2016-03-02 7:26 UTC (permalink / raw)
To: Fam Zheng
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
Fam Zheng <famz@redhat.com> writes:
> On Tue, 03/01 09:52, Alex Bennée wrote:
>>
>> Fam Zheng <famz@redhat.com> writes:
>>
>> > On Mon, 02/29 17:18, Alex Bennée wrote:
>> >>
>> >> Fam Zheng <famz@redhat.com> writes:
>> >>
>> >> > Signed-off-by: Fam Zheng <famz@redhat.com>
>> >>
>> >> I think we need to include the dtc libs in here unless we need to ship
>> >> all submodules in the tree as well.
>> >
>> > OK, will take a look.
>> >
>> >>
>> >> Is there a way to get a plain install shell? PAUSE=1 on the test shells
>> >> doesn't seem to allow me to install anything.
>> >
>> > What is the error when you install things? I think it is an unmodified shell
>> > and "yum" or "apt-get" should just work. It's just that when the container
>> > exits the changes will be discarded.
>>
>> So running:
>>
>> make docker-test-quick@ubuntu PAUSE=1
>>
>> And I got a shell I wanted to try and get working:
>>
>> root@72d09615d5b2:/var/tmp/qemu.tmp# ./configure
>>
>> ERROR: DTC (libfdt) version >= 1.4.0 not present. Your options:
>> (1) Preferred: Install the DTC (libfdt) devel package
>> (2) Fetch the DTC submodule, using:
>> git submodule update --init dtc
>>
>> root@72d09615d5b2:/var/tmp/qemu.tmp# apt-cache search libfdt
>> libfdt-dev - Flat Device Trees manipulation library - development
>> files
>> libfdt1 - Flat Device Trees manipulation library
>> root@72d09615d5b2:/var/tmp/qemu.tmp# apt-get install libfdt-dev
>> Reading package lists... Done
>> Building dependency tree
>> Reading state information... Done
>> The following extra packages will be installed:
>> libfdt1
>> The following NEW packages will be installed:
>> libfdt-dev libfdt1
>> 0 upgraded, 2 newly installed, 0 to remove and 10 not upgraded.
>> Need to get 37.8 kB of archives.
>> After this operation, 216 kB of additional disk space will be used.
>> Do you want to continue? [Y/n]
>> Err http://archive.ubuntu.com/ubuntu/ trusty/main libfdt1 amd64
>> 1.4.0+dfsg-1
>> Could not resolve 'archive.ubuntu.com'
>> Err http://archive.ubuntu.com/ubuntu/ trusty/main libfdt-dev amd64
>> 1.4.0+dfsg-1
>> Could not resolve 'archive.ubuntu.com'
>> E: Failed to fetch
>> http://archive.ubuntu.com/ubuntu/pool/main/d/device-tree-compiler/libfdt1_1.4.0+dfsg-1_amd64.deb
>> Could not resolve 'archive.ubuntu.com'
>>
>> E: Failed to fetch
>> http://archive.ubuntu.com/ubuntu/pool/main/d/device-tree-compiler/libfdt-dev_1.4.0+dfsg-1_amd64.deb
>> Could not resolve 'archive.ubuntu.com'
>>
>> E: Unable to fetch some archives, maybe run apt-get update or try with
>> --fix-missing?
>> root@72d09615d5b2:/var/tmp/qemu.tmp# apt-get update
>> Err http://archive.ubuntu.com trusty InRelease
>>
>> Err http://archive.ubuntu.com trusty-updates InRelease
>>
>> Err http://archive.ubuntu.com trusty-security InRelease
>>
>> Err http://archive.ubuntu.com trusty Release.gpg
>> Could not resolve 'archive.ubuntu.com'
>> Err http://archive.ubuntu.com trusty-updates Release.gpg
>> Could not resolve 'archive.ubuntu.com'
>> Err http://archive.ubuntu.com trusty-security Release.gpg
>> Could not resolve 'archive.ubuntu.com'
>> Reading package lists... Done
>> W: Failed to fetch
>> http://archive.ubuntu.com/ubuntu/dists/trusty/InRelease
>>
>> W: Failed to fetch
>> http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InRelease
>>
>> W: Failed to fetch
>> http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease
>>
>> W: Failed to fetch
>> http://archive.ubuntu.com/ubuntu/dists/trusty/Release.gpg Could not
>> resolve 'archive.ubuntu.com'
>>
>> W: Failed to fetch
>> http://archive.ubuntu.com/ubuntu/dists/trusty-updates/Release.gpg
>> Could not resolve 'archive.ubuntu.com'
>>
>> W: Failed to fetch
>> http://archive.ubuntu.com/ubuntu/dists/trusty-security/Release.gpg
>> Could not resolve 'archive.ubuntu.com'
>>
>> W: Some index files failed to download. They have been ignored, or old
>> ones used instead.
>>
>>
>> So it looks like networking isn't working in the test images. For
>> debugging problems I think we need to be able to install stuff on a
>> running image so we can then fix the base recipes.
>
> Oh yes, I added "-net=none" to the "docker run" command for better isolation.
> So in this case we should enable it. Let's rename PAUSE=1 to DEBUG=1 and
> enable network.
Sounds good.
>
> Fam
--
Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread
* [Qemu-devel] [PATCH v2 04/15] docker: Add test runner
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (2 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 03/15] docker: Add images Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 05/15] docker: Add common.rc Fam Zheng
` (11 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
It's better to have a launcher for all tests, to make it easier to
initialize and manage the environment.
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/run | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
create mode 100755 tests/docker/run
diff --git a/tests/docker/run b/tests/docker/run
new file mode 100755
index 0000000..55b2737
--- /dev/null
+++ b/tests/docker/run
@@ -0,0 +1,26 @@
+#!/bin/bash -e
+#
+# Docker test runner
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+# Prepare the environment
+. /etc/profile || true
+export PATH=/usr/lib/ccache:$PATH
+
+tmp="$QEMU_SRC"
+QEMU_SRC=/var/tmp/qemu.tmp
+cp -r "$tmp" "$QEMU_SRC"
+if test -n "$J"; then
+ export MAKEFLAGS="$MAKEFLAGS -j$J"
+fi
+
+cd $QEMU_SRC/tests/docker
+"$@"
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* [Qemu-devel] [PATCH v2 05/15] docker: Add common.rc
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (3 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 04/15] docker: Add test runner Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 06/15] docker: Add quick test Fam Zheng
` (10 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
"requires" checks the "FEATURE" environment for specified prerequisits,
and skip the execution of test if not found.
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/common.rc | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
create mode 100755 tests/docker/common.rc
diff --git a/tests/docker/common.rc b/tests/docker/common.rc
new file mode 100755
index 0000000..34b009e
--- /dev/null
+++ b/tests/docker/common.rc
@@ -0,0 +1,29 @@
+#!/bin/sh
+#
+# Common routines for docker test scripts.
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+requires()
+{
+ for c in $@; do
+ if ! echo "$FEATURES" | grep -wq -e "$c"; then
+ echo "Prerequisite '$c' not present, skip"
+ exit 0
+ fi
+ done
+}
+
+build_qemu()
+{
+ make -C $QEMU_SRC distclean &>/dev/null
+ $QEMU_SRC/configure "$@"
+ make $MAKEFLAGS
+}
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* [Qemu-devel] [PATCH v2 06/15] docker: Add quick test
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (4 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 05/15] docker: Add common.rc Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 07/15] docker: Add full test Fam Zheng
` (9 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/test-quick | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
create mode 100755 tests/docker/test-quick
diff --git a/tests/docker/test-quick b/tests/docker/test-quick
new file mode 100755
index 0000000..ce3cc99
--- /dev/null
+++ b/tests/docker/test-quick
@@ -0,0 +1,23 @@
+#!/bin/bash -e
+#
+# Quick compiling test that everyone already does. But why not automate it?
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+. common.rc
+
+mkdir build
+mkdir install
+cd build
+build_qemu \
+ --prefix="${pwd}/install" \
+ --target-list="$(echo {x86_64,i386,arm,aarch64}-{softmmu,linux-user})"
+make check $MAKEFLAGS
+make install
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* [Qemu-devel] [PATCH v2 07/15] docker: Add full test
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (5 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 06/15] docker: Add quick test Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 08/15] docker: Add clang test Fam Zheng
` (8 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
This builds all available targets.
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/test-full | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
create mode 100755 tests/docker/test-full
diff --git a/tests/docker/test-full b/tests/docker/test-full
new file mode 100755
index 0000000..5708d0b
--- /dev/null
+++ b/tests/docker/test-full
@@ -0,0 +1,21 @@
+#!/bin/bash -e
+#
+# Compile all the targets.
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+. common.rc
+
+mkdir build
+mkdir install
+cd build
+build_qemu --prefix="${pwd}/install"
+make check $MAKEFLAGS
+make install
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* [Qemu-devel] [PATCH v2 08/15] docker: Add clang test
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (6 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 07/15] docker: Add full test Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 09/15] docker: Add mingw test Fam Zheng
` (7 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
The configure options are suggested by John Snow <jsnow@redhat.com>.
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/test-clang | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
create mode 100755 tests/docker/test-clang
diff --git a/tests/docker/test-clang b/tests/docker/test-clang
new file mode 100755
index 0000000..212f481
--- /dev/null
+++ b/tests/docker/test-clang
@@ -0,0 +1,28 @@
+#!/bin/bash -e
+#
+# Compile and check with clang.
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+. common.rc
+
+requires clang
+
+mkdir build
+cd build
+# -fsanitize=undefined is broken on Fedora 23, skip it for now
+# See also: https://bugzilla.redhat.com/show_bug.cgi?id=1263834
+#OPTS="$OPTS --extra-cflags=-fsanitize=undefined \
+ #--extra-cflags=-fno-sanitize=float-divide-by-zero"
+build_qemu \
+ --target-list=x86_64-softmmu,aarch64-softmmu \
+ --enable-debug --cxx=clang++ --cc=clang --host-cc=clang \
+ $OPTS
+make $MAKEFLAGS check
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* [Qemu-devel] [PATCH v2 09/15] docker: Add mingw test
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (7 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 08/15] docker: Add clang test Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 10/15] docker: Add travis tool Fam Zheng
` (6 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/test-mingw | 36 ++++++++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
create mode 100755 tests/docker/test-mingw
diff --git a/tests/docker/test-mingw b/tests/docker/test-mingw
new file mode 100755
index 0000000..25afa64
--- /dev/null
+++ b/tests/docker/test-mingw
@@ -0,0 +1,36 @@
+#!/bin/bash -e
+#
+# Cross compile QEMU with mingw toolchain on Linux.
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+. common.rc
+
+requires mingw
+
+mkdir build
+cd build
+for prefix in x86_64-w64-mingw32- i686-w64-mingw32-; do
+ build_qemu --cross-prefix=$prefix \
+ --target-list=x86_64-softmmu,aarch64-softmmu \
+ --enable-trace-backends=simple \
+ --enable-debug \
+ --enable-gnutls \
+ --enable-nettle \
+ --enable-curl \
+ --enable-vnc \
+ --enable-bzip2 \
+ --enable-guest-agent \
+ --with-sdlabi=1.2 \
+ --with-gtkabi=2.0
+ make clean
+
+done
+
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* [Qemu-devel] [PATCH v2 10/15] docker: Add travis tool
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (8 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 09/15] docker: Add mingw test Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 11/15] docs: Add text for tests/docker in build-system.txt Fam Zheng
` (5 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
The script is not named test-travis.sh so it won't run with "make
docker-run", because it can take too long.
Run it with "make docker-run-travis.sh@ubuntu".
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/travis | 23 +++++++++++++++++++++++
tests/docker/travis.py | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 71 insertions(+)
create mode 100755 tests/docker/travis
create mode 100755 tests/docker/travis.py
diff --git a/tests/docker/travis b/tests/docker/travis
new file mode 100755
index 0000000..dfea61f
--- /dev/null
+++ b/tests/docker/travis
@@ -0,0 +1,23 @@
+#!/bin/bash -e
+#
+# Mimic a travis testing matrix
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+. common.rc
+
+requires pyyaml
+
+cd "$QEMU_SRC"
+
+cmdfile=/tmp/travis_cmd_list.sh
+
+./tests/docker/travis.py .travis.yml > $cmdfile
+. $cmdfile
diff --git a/tests/docker/travis.py b/tests/docker/travis.py
new file mode 100755
index 0000000..8dcc964
--- /dev/null
+++ b/tests/docker/travis.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env python
+#
+# Travis YAML config parser
+#
+# Copyright (c) 2016 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+import sys
+import yaml
+import itertools
+
+def load_yaml(fname):
+ return yaml.load(open(fname, "r").read())
+
+def conf_iter(conf):
+ def env_to_list(env):
+ return env if isinstance(env, list) else [env]
+ global_env = conf["env"]["global"]
+ for entry in conf["matrix"]["include"]:
+ yield {"env": global_env + env_to_list(entry["env"]),
+ "compiler": entry["compiler"]}
+ for entry in itertools.product(conf["compiler"],
+ conf["env"]["matrix"]):
+ yield {"env": global_env + env_to_list(entry[1]),
+ "compiler": entry[0]}
+
+def main():
+ if len(sys.argv) < 2:
+ sys.stderr.write("Usage: %s <travis-file>\n" % sys.argv[0])
+ return 1
+ conf = load_yaml(sys.argv[1])
+ for config in conf_iter(conf):
+ print "("
+ print "\n".join(config["env"])
+ print "alias cc=" + config["compiler"]
+ print "\n".join(conf["before_script"])
+ print "\n".join(conf["script"])
+ print ")"
+ return 0
+
+if __name__ == "__main__":
+ sys.exit(main())
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* [Qemu-devel] [PATCH v2 11/15] docs: Add text for tests/docker in build-system.txt
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (9 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 10/15] docker: Add travis tool Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 12/15] .gitignore: Ignore temporary dockerfile Fam Zheng
` (4 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
Signed-off-by: Fam Zheng <famz@redhat.com>
---
docs/build-system.txt | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/docs/build-system.txt b/docs/build-system.txt
index 5ddddea..2af1e66 100644
--- a/docs/build-system.txt
+++ b/docs/build-system.txt
@@ -438,6 +438,11 @@ top level Makefile, so anything defined in this file will influence the
entire build system. Care needs to be taken when writing rules for tests
to ensure they only apply to the unit test execution / build.
+- tests/docker/Makefile.include
+
+Rules for Docker tests. Like tests/Makefile, this file is included
+directly by the top level Makefile, anything defined in this file will
+influence the entire build system.
- po/Makefile
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* [Qemu-devel] [PATCH v2 12/15] .gitignore: Ignore temporary dockerfile
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (10 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 11/15] docs: Add text for tests/docker in build-system.txt Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 13/15] MAINTAINERS: Add tests/docker Fam Zheng
` (3 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
Docker build requires a "context" directory and we use the
$QEMU_SRC/tests/docker/ directory, and the temoprary dockerfile has to be in
the context.
docker_build normally cleans up this file but let's add an entry here just in
case it fails to.
Signed-off-by: Fam Zheng <famz@redhat.com>
---
.gitignore | 1 +
1 file changed, 1 insertion(+)
diff --git a/.gitignore b/.gitignore
index 88a80ff..a335b7b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -109,3 +109,4 @@ cscope.*
tags
TAGS
*~
+/tests/docker/*.docker.tmp
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* [Qemu-devel] [PATCH v2 13/15] MAINTAINERS: Add tests/docker
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (11 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 12/15] .gitignore: Ignore temporary dockerfile Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 14/15] docker: Support interactive shell for debugging Fam Zheng
` (2 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
MAINTAINERS | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 02710f8..15b8b93 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1584,3 +1584,10 @@ Build system architecture
M: Daniel P. Berrange <berrange@redhat.com>
S: Odd Fixes
F: docs/build-system.txt
+
+Docker testing
+--------------
+Docker based testing framework and cases
+M: Fam Zheng <famz@redhat.com>
+S: Maintained
+F: tests/docker/
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* [Qemu-devel] [PATCH v2 14/15] docker: Support interactive shell for debugging
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (12 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 13/15] MAINTAINERS: Add tests/docker Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 15/15] tests/docker/Makefile.include: snapshot the src for docker Fam Zheng
2016-02-24 7:17 ` [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
15 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
Specify "PAUSE=1" when invoking make docker-%, and a shell prompt will
show up before the test runs.
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/Makefile.include | 5 ++++-
tests/docker/run | 8 ++++++++
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index d8c55c3..aaf7d23 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -60,6 +60,8 @@ docker:
@echo ' TESTS="x y z .." Filters which tests to run (for docker-test).'
@echo ' J=[0..9]* Overrides the -jN parameter for make commands'
@echo ' (default is 1)'
+ @echo ' PAUSE=1 Stop and drop to shell in the created container'
+ @echo ' before running the command.'
docker-image-%:
@if test -z "$(IMAGE)"; then echo "Invalid target"; exit 1; fi
@@ -79,9 +81,10 @@ docker-run-%:
$(call quiet-command,\
$(SRC_PATH)/tests/docker/docker_run $(if $(KEEP),,--rm) \
--privileged -t --net=none \
+ $(if $(PAUSE),-i) \
-v $$(realpath $(SRC_PATH)):/var/tmp/qemu \
-e QEMU_SRC=/var/tmp/qemu \
- -e V=$V -e J=$J \
+ -e V=$V -e J=$J -e PAUSE=$(PAUSE)\
-v /var/tmp/qemu-docker-ccache:/var/tmp/ccache \
-e CCACHE_DIR=/var/tmp/ccache \
qemu:$(IMAGE) \
diff --git a/tests/docker/run b/tests/docker/run
index 55b2737..cc8d246 100755
--- a/tests/docker/run
+++ b/tests/docker/run
@@ -23,4 +23,12 @@ if test -n "$J"; then
fi
cd $QEMU_SRC/tests/docker
+
+if test -n "$PAUSE"; then
+ echo "* Prepared to run command:"
+ echo " $@"
+ echo "* Hit Ctrl-D to continue, or type 'exit 1' to quit"
+ echo
+ $SHELL
+fi
"$@"
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* [Qemu-devel] [PATCH v2 15/15] tests/docker/Makefile.include: snapshot the src for docker
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (13 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 14/15] docker: Support interactive shell for debugging Fam Zheng
@ 2016-02-16 12:39 ` Fam Zheng
2016-03-01 20:16 ` Alex Bennée
2016-02-24 7:17 ` [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
15 siblings, 1 reply; 32+ messages in thread
From: Fam Zheng @ 2016-02-16 12:39 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, peter.maydell, jsnow, stefanha, sw, Paolo Bonzini,
Alex Bennée, david
From: Alex Bennée <alex.bennee@linaro.org>
Instead of providing a live version of the source tree to the docker
container we snapshot it with git-archive. This ensure the tree is in a
pristine state for whatever operations the container is going to run on
them.
Uncommitted changes known to files known by the git index will be
included in the snapshot if there are any.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
[Generate an unique directory under /tmp/ - Fam Zheng]
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/Makefile.include | 22 ++++++++++++++++++----
1 file changed, 18 insertions(+), 4 deletions(-)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index aaf7d23..7cc2130 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -2,7 +2,7 @@
$(if $(quiet-command),,$(eval include $(SRC_PATH)/rules.mak))
-.PHONY: docker docker-test docker-clean docker-image
+.PHONY: docker docker-test docker-clean docker-image docker-qemu-src
DOCKER_SUFFIX := .docker
@@ -16,6 +16,21 @@ DOCKER_TOOLS := travis
TESTS ?= %
IMAGES ?= %
+SRC_COPY := $(shell mktemp -u /tmp/qemu-src.XXXXX)
+
+.PHONY: $(SRC_COPY)
+
+$(SRC_COPY):
+ @mkdir -p $@
+ $(call quiet-command, \
+ mkdir -p $@ && (if git diff-index --quiet HEAD -- &>/dev/null; then \
+ git archive -1 HEAD --format=tar.gz; \
+ else \
+ git archive -1 $$(git stash create) --format=tar.gz; \
+ fi) | tar -xz -C $@, \
+ " COPY SRC $@")
+
+docker-qemu-src: $(SRC_COPY)
$(foreach i,$(DOCKER_IMAGES), \
$(eval docker-image: docker-image-$i) \
@@ -23,8 +38,7 @@ $(foreach i,$(DOCKER_IMAGES), \
$(eval .PHONY: docker-image-$i docker-@$i) \
$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
$(eval .PHONY: docker-$t@$i) \
- $(eval docker-$t@$i: docker-image-$i) \
- $(eval docker-$t@$i: docker-run-$t@$i) \
+ $(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
$(eval docker-@$i: docker-$t@$i) \
$(eval docker-test: docker-run-$t@$i) \
) \
@@ -73,7 +87,7 @@ docker-image-%:
docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')
docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')
-docker-run-%:
+docker-run-%: docker-qemu-src
@if test -z "$(IMAGE)" || test -z "$(CMD)"; \
then echo "Invalid target"; exit 1; \
fi
--
2.4.3
^ permalink raw reply related [flat|nested] 32+ messages in thread* Re: [Qemu-devel] [PATCH v2 15/15] tests/docker/Makefile.include: snapshot the src for docker
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 15/15] tests/docker/Makefile.include: snapshot the src for docker Fam Zheng
@ 2016-03-01 20:16 ` Alex Bennée
2016-03-02 3:38 ` Fam Zheng
0 siblings, 1 reply; 32+ messages in thread
From: Alex Bennée @ 2016-03-01 20:16 UTC (permalink / raw)
To: Fam Zheng
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
Fam Zheng <famz@redhat.com> writes:
> From: Alex Bennée <alex.bennee@linaro.org>
>
> Instead of providing a live version of the source tree to the docker
> container we snapshot it with git-archive. This ensure the tree is in a
> pristine state for whatever operations the container is going to run on
> them.
>
> Uncommitted changes known to files known by the git index will be
> included in the snapshot if there are any.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> [Generate an unique directory under /tmp/ - Fam Zheng]
But the docker-run-% rule never uses the source copy we just made.
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
> tests/docker/Makefile.include | 22 ++++++++++++++++++----
> 1 file changed, 18 insertions(+), 4 deletions(-)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index aaf7d23..7cc2130 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -2,7 +2,7 @@
>
> $(if $(quiet-command),,$(eval include $(SRC_PATH)/rules.mak))
>
> -.PHONY: docker docker-test docker-clean docker-image
> +.PHONY: docker docker-test docker-clean docker-image docker-qemu-src
>
> DOCKER_SUFFIX := .docker
>
> @@ -16,6 +16,21 @@ DOCKER_TOOLS := travis
>
> TESTS ?= %
> IMAGES ?= %
> +SRC_COPY := $(shell mktemp -u /tmp/qemu-src.XXXXX)
> +
> +.PHONY: $(SRC_COPY)
> +
> +$(SRC_COPY):
> + @mkdir -p $@
> + $(call quiet-command, \
> + mkdir -p $@ && (if git diff-index --quiet HEAD -- &>/dev/null; then \
> + git archive -1 HEAD --format=tar.gz; \
> + else \
> + git archive -1 $$(git stash create) --format=tar.gz; \
> + fi) | tar -xz -C $@, \
> + " COPY SRC $@")
> +
> +docker-qemu-src: $(SRC_COPY)
>
> $(foreach i,$(DOCKER_IMAGES), \
> $(eval docker-image: docker-image-$i) \
> @@ -23,8 +38,7 @@ $(foreach i,$(DOCKER_IMAGES), \
> $(eval .PHONY: docker-image-$i docker-@$i) \
> $(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
> $(eval .PHONY: docker-$t@$i) \
> - $(eval docker-$t@$i: docker-image-$i) \
> - $(eval docker-$t@$i: docker-run-$t@$i) \
> + $(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
> $(eval docker-@$i: docker-$t@$i) \
> $(eval docker-test: docker-run-$t@$i) \
> ) \
> @@ -73,7 +87,7 @@ docker-image-%:
>
> docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')
> docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')
> -docker-run-%:
> +docker-run-%: docker-qemu-src
> @if test -z "$(IMAGE)" || test -z "$(CMD)"; \
> then echo "Invalid target"; exit 1; \
> fi
--
Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Qemu-devel] [PATCH v2 15/15] tests/docker/Makefile.include: snapshot the src for docker
2016-03-01 20:16 ` Alex Bennée
@ 2016-03-02 3:38 ` Fam Zheng
0 siblings, 0 replies; 32+ messages in thread
From: Fam Zheng @ 2016-03-02 3:38 UTC (permalink / raw)
To: Alex Bennée
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
On Tue, 03/01 20:16, Alex Bennée wrote:
>
> Fam Zheng <famz@redhat.com> writes:
>
> > From: Alex Bennée <alex.bennee@linaro.org>
> >
> > Instead of providing a live version of the source tree to the docker
> > container we snapshot it with git-archive. This ensure the tree is in a
> > pristine state for whatever operations the container is going to run on
> > them.
> >
> > Uncommitted changes known to files known by the git index will be
> > included in the snapshot if there are any.
> >
> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > [Generate an unique directory under /tmp/ - Fam Zheng]
>
> But the docker-run-% rule never uses the source copy we just made.
I failed in Makefile magic: pattern rules and dynamically generated rules
mustn't be used mixed, otherwise the former will be ignored. Will fix.
Fam
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests
2016-02-16 12:39 [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
` (14 preceding siblings ...)
2016-02-16 12:39 ` [Qemu-devel] [PATCH v2 15/15] tests/docker/Makefile.include: snapshot the src for docker Fam Zheng
@ 2016-02-24 7:17 ` Fam Zheng
2016-02-29 14:51 ` Alex Bennée
15 siblings, 1 reply; 32+ messages in thread
From: Fam Zheng @ 2016-02-24 7:17 UTC (permalink / raw)
To: qemu-devel, Alex Bennée
Cc: kwolf, peter.maydell, sw, stefanha, Paolo Bonzini, jsnow, david
On Tue, 02/16 20:39, Fam Zheng wrote:
> v2: - Fix a few coding style warnings of shellcheck and pylint.
> [Marc-André Lureau]
> - Rename make targets:
> docker-run => docker-test
> docker-build => docker-image
> docker-run-$TEST.sh@$IMAGE => docker-$TEST@$IMAGE
> - In help, document the magic makefile vars which are
> "IMAGES=", "TESTS=", "J=" and "PAUSE="
> - Rename test-basic => test-quick
> - Move docker files to tests/docker/dockerfiles/
> - Add "test-full" that builds all targets
> - Workaround broken clang for fedora 23
> - Add commit log for .gitignore changes [Alex]
> - Add "rev-by" in patch 13. [Alex]
> - Add patch 14 to make debugging easier [Alex]
> - Adopt Alex's "git archive" idea
Alex, any comments on this?
Fam
^ permalink raw reply [flat|nested] 32+ messages in thread* Re: [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests
2016-02-24 7:17 ` [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests Fam Zheng
@ 2016-02-29 14:51 ` Alex Bennée
2016-02-29 17:22 ` Alex Bennée
0 siblings, 1 reply; 32+ messages in thread
From: Alex Bennée @ 2016-02-29 14:51 UTC (permalink / raw)
To: Fam Zheng
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
Fam Zheng <famz@redhat.com> writes:
> On Tue, 02/16 20:39, Fam Zheng wrote:
>> v2: - Fix a few coding style warnings of shellcheck and pylint.
>> [Marc-André Lureau]
>> - Rename make targets:
>> docker-run => docker-test
>> docker-build => docker-image
>> docker-run-$TEST.sh@$IMAGE => docker-$TEST@$IMAGE
>> - In help, document the magic makefile vars which are
>> "IMAGES=", "TESTS=", "J=" and "PAUSE="
>> - Rename test-basic => test-quick
>> - Move docker files to tests/docker/dockerfiles/
>> - Add "test-full" that builds all targets
>> - Workaround broken clang for fedora 23
>> - Add commit log for .gitignore changes [Alex]
>> - Add "rev-by" in patch 13. [Alex]
>> - Add patch 14 to make debugging easier [Alex]
>> - Adopt Alex's "git archive" idea
>
> Alex, any comments on this?
Yeah sorry for the delay, I'm just starting on this now.
--
Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Qemu-devel] [PATCH v2 00/15] tests: Introducing docker tests
2016-02-29 14:51 ` Alex Bennée
@ 2016-02-29 17:22 ` Alex Bennée
0 siblings, 0 replies; 32+ messages in thread
From: Alex Bennée @ 2016-02-29 17:22 UTC (permalink / raw)
To: Fam Zheng
Cc: kwolf, peter.maydell, sw, qemu-devel, stefanha, Paolo Bonzini,
jsnow, david
Alex Bennée <alex.bennee@linaro.org> writes:
> Fam Zheng <famz@redhat.com> writes:
>
>> On Tue, 02/16 20:39, Fam Zheng wrote:
>>> v2: - Fix a few coding style warnings of shellcheck and pylint.
>>> [Marc-André Lureau]
>>> - Rename make targets:
>>> docker-run => docker-test
>>> docker-build => docker-image
>>> docker-run-$TEST.sh@$IMAGE => docker-$TEST@$IMAGE
>>> - In help, document the magic makefile vars which are
>>> "IMAGES=", "TESTS=", "J=" and "PAUSE="
>>> - Rename test-basic => test-quick
>>> - Move docker files to tests/docker/dockerfiles/
>>> - Add "test-full" that builds all targets
>>> - Workaround broken clang for fedora 23
>>> - Add commit log for .gitignore changes [Alex]
>>> - Add "rev-by" in patch 13. [Alex]
>>> - Add patch 14 to make debugging easier [Alex]
>>> - Adopt Alex's "git archive" idea
>>
>> Alex, any comments on this?
>
> Yeah sorry for the delay, I'm just starting on this now.
OK I'm not sure I can go on without an easier way off seeing where it is
failing when it fails. Can we pass optional verbosity down to the final
build and tests?
--
Alex Bennée
^ permalink raw reply [flat|nested] 32+ messages in thread