[PATCH 0/4] tests/docker: improve detection of docker/podman

[PATCH 0/4] tests/docker: improve detection of docker/podman

[Daniel P. Berrangé]

This improves integration such that tests/docker/Makefile.include will
correctly honour the exact command detected by docker.py's probing
logic. Currently the probe command gets stripped down to just a bare
'podman' or 'docker' command name. This means while commands run via
'docker.py' would use 'sudo -n docker', commands run directly from
Makefile.include would only use 'docker'.

This series fixes that so that 'docker.py probe' correctly reports the
full argv, and configure honours that argv untouched.

With that fixed, we can then also add support for 'podman --remote'
and 'podman-remote', which allow use of podman when already inside
podman which is the scenario for my development environment that is
using 'toolbox'.

Finally this also improves CI by ensuring that stdout from docker is
not thrown away, so we can have a chance of diagnosing build failures
from CI.

Daniel P. Berrangé (4):
  tests/docker: improve handling of docker probes
  tests/docker: allow display of docker output
  gitlab: ensure docker output is always displayed in CI
  tests/docker: add support for podman remote access

 .gitlab-ci.d/base.yml         |  3 +++
 configure                     | 19 +++++++------------
 tests/docker/Makefile.include | 23 ++++++++++++++---------
 tests/docker/docker.py        | 17 +++++++----------
 4 files changed, 31 insertions(+), 31 deletions(-)

-- 
2.52.0

[PATCH 1/4] tests/docker: improve handling of docker probes

[Daniel P. Berrangé]

The docker.py script has logic to guess the container command and
detects one of

  * docker
  * sudo -n docker
  * podman

but the "docker.py probe" command then throws away the detected argv
and prints a slightly different argv based soley on the detected
argv[0]. The result is that 'probe' will print

  * docker
  * sudo docker
  * podman

which means that if sudo was detected & the result of 'probe' were
used direclty, it would end up prompting for password interaction
every time.

The 'configure' script, however, runs 'probe' and then throws away
the printed argv again, reporting only 'podman' or 'docker', which
is used to set the $(RUNC) variable for tests/docker/Makefile.include
which is in turn used to pass --engine to docker.py. So the docker.py
command will re-detect the need for 'sudo -n' and use it correctly

The problem with this is that some commands in Makefile.include do
not call docker.py at all, they invoke $(RUNC) directly. Since
configure threw away the 'sudo' command prefix Makefile.in won't
be adding either 'sudo' or 'sudo -n', it'll just run plani 'docker'
which is wrong.

This commit sanitizes things so that the 'docker.py probe' prints
out the exact detected ARGV, and configure fully preserves this
ARGV when setting $(RUNC). Since "$(RUNC)" is no longer just a bare
engine name, however, we must now also set the $(CONTAINER_ENGINE)
variable for Makefile.include so it can pass something sane to
the --engine arg for docker.py

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 configure                     | 19 +++++++------------
 tests/docker/Makefile.include |  5 +++--
 tests/docker/docker.py        |  7 +------
 3 files changed, 11 insertions(+), 20 deletions(-)

diff --git a/configure b/configure
index 55e0bd3425..1adbb357df 100755
--- a/configure
+++ b/configure
@@ -1316,17 +1316,11 @@ fi
 ##########################################
 # functions to probe cross compilers
 
-container="no"
-runc=""
+runc="no"
 if test $use_containers = "yes" && (has "docker" || has "podman"); then
-    case $($python "$source_path"/tests/docker/docker.py --engine "$container_engine" probe) in
-        *docker) container=docker ;;
-        podman) container=podman ;;
-        no) container=no ;;
-    esac
-    if test "$container" != "no"; then
-        docker_py="$python $source_path/tests/docker/docker.py --engine $container"
-        runc=$container
+    runc=$($python "$source_path"/tests/docker/docker.py --engine "$container_engine" probe)
+    if test "$runc" != "no"; then
+        docker_py="$python $source_path/tests/docker/docker.py --engine $container_engine"
     fi
 fi
 
@@ -1446,7 +1440,7 @@ probe_target_compiler() {
   esac
 
   for host in $container_hosts; do
-    test "$container" != no || continue
+    test "$runc" != no || continue
     test "$host" = "$cpu" || continue
     case $target_arch in
       # debian-all-test-cross architectures
@@ -1775,8 +1769,9 @@ echo all: >> $config_host_mak
 echo "SRC_PATH=$source_path" >> $config_host_mak
 echo "TARGET_DIRS=$target_list" >> $config_host_mak
 echo "GDB=$gdb_bin" >> $config_host_mak
-if test "$container" != no; then
+if test "$runc" != no; then
     echo "RUNC=$runc" >> $config_host_mak
+    echo "CONTAINER_ENGINE=$container_engine" >> $config_host_mak
 fi
 echo "SUBDIRS=$subdirs" >> $config_host_mak
 if test "$rust" != disabled; then
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 38467cca61..7d4582b6a8 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -16,8 +16,9 @@ DOCKER_DEFAULT_REGISTRY := registry.gitlab.com/qemu-project/qemu
 endif
 DOCKER_REGISTRY := $(if $(REGISTRY),$(REGISTRY),$(DOCKER_DEFAULT_REGISTRY))
 
-RUNC ?= $(if $(shell command -v docker), docker, podman)
-DOCKER_SCRIPT=$(SRC_PATH)/tests/docker/docker.py --engine $(RUNC)
+CONTAINER_ENGINE = auto
+DOCKER_SCRIPT=$(SRC_PATH)/tests/docker/docker.py --engine $(CONTAINER_ENGINE)
+RUNC ?= $(shell $(DOCKER_SCRIPT) probe)
 
 CUR_TIME := $(shell date +%Y-%m-%d-%H.%M.%S.$$$$)
 DOCKER_SRC_COPY := $(BUILD_DIR)/docker-src.$(CUR_TIME)
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 3b8a26704d..ff68c7bf6f 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -618,12 +618,7 @@ class ProbeCommand(SubCommand):
     def run(self, args, argv):
         try:
             docker = Docker()
-            if docker._command[0] == "docker":
-                print("docker")
-            elif docker._command[0] == "sudo":
-                print("sudo docker")
-            elif docker._command[0] == "podman":
-                print("podman")
+            print(" ".join(docker._command))
         except Exception:
             print("no")
 
-- 
2.52.0

[PATCH 2/4] tests/docker: allow display of docker output

[Daniel P. Berrangé]

The --quiet command is used with docker unless V=1 is passed to make,
and as a result stdout from docker is never visible by default, making
it hard to diagnose failures building / running containers.

Meanwhile passing V=1 is undesirable as that makes the entire build
system verbose.

Introduce a $(DOCKER_V) make variable which is initialized from $(V)

It is thus possible to display docker output without also enabling
make verbose output.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/docker/Makefile.include | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 7d4582b6a8..d58a280333 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -40,14 +40,14 @@ docker-qemu-src: $(DOCKER_SRC_COPY)
 docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 	  $(call quiet-command,			\
 		DOCKER_BUILDKIT=1 $(RUNC) build		\
-		$(if $V,,--quiet)			\
+		$(if $(DOCKER_V),,--quiet)		\
 		$(if $(NOCACHE),--no-cache,		\
 			$(if $(DOCKER_REGISTRY),--cache-from $(DOCKER_REGISTRY)/qemu/$*)) \
 		--build-arg BUILDKIT_INLINE_CACHE=1 	\
 		$(if $(NOUSER),,			\
 			--build-arg USER=$(USER)	\
 			--build-arg UID=$(UID))		\
-		-t qemu/$* - < $< $(if $V,,> /dev/null),\
+		-t qemu/$* - < $< $(if $(DOCKER_V),,> /dev/null),\
 		"BUILD", $*)
 
 # General rule for inspecting registry images.
@@ -73,7 +73,7 @@ docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
 			DEB_TYPE=$(DEB_TYPE) 					\
 			$(if $(DEB_URL),DEB_URL=$(DEB_URL),)			\
 			$(DOCKER_SCRIPT) build -t qemu/debian-$* -f $< 		\
-			$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) 		\
+			$(if $(DOCKER_V),,--quiet) $(if $(NOCACHE),--no-cache) 	\
 			$(if $(NOUSER),,--add-current-user) 			\
 			$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))	\
 			$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)), \
@@ -105,16 +105,17 @@ debian-toolchain-run = \
 	$(if $(NOCACHE)$(NOFETCH),					\
 		$(call quiet-command,					\
 			$(DOCKER_SCRIPT) build -t qemu/$1 -f $< 	\
-			$(if $V,,--quiet) $(if $(NOCACHE),--no-cache)	\
+			$(if $(DOCKER_V),,--quiet) 			\
+			$(if $(NOCACHE),--no-cache)			\
 			--registry $(DOCKER_REGISTRY) --extra-files	\
 			$(DOCKER_FILES_DIR)/$1.d/build-toolchain.sh,	\
 			"BUILD", $1),				        \
 		$(call quiet-command,					\
-			$(DOCKER_SCRIPT) fetch $(if $V,,--quiet)	\
+			$(DOCKER_SCRIPT) fetch $(if $(DOCKER_V),,--quiet) \
 				qemu/$1 $(DOCKER_REGISTRY),		\
 			"FETCH", $1)					\
 		$(call quiet-command,					\
-			$(DOCKER_SCRIPT) update $(if $V,,--quiet) 	\
+			$(DOCKER_SCRIPT) update $(if $(DOCKER_V),,--quiet) \
 				qemu/$1 				\
 				$(if $(NOUSER),,--add-current-user) 	\
 			"PREPARE", $1))
@@ -231,7 +232,10 @@ docker-run: docker-qemu-src
 			-e TARGET_LIST=$(subst $(SPACE),$(COMMA),$(TARGET_LIST))	\
 			-e EXTRA_CONFIGURE_OPTS="$(EXTRA_CONFIGURE_OPTS)" \
 			-e TEST_COMMAND="$(TEST_COMMAND)" 		\
-			-e V=$V -e J=$J -e DEBUG=$(DEBUG)		\
+			-e V=$V						\
+			-e DOCKER_V=$(DOCKER_V)				\
+			-e J=$J						\
+			-e DEBUG=$(DEBUG)				\
 			-e SHOW_ENV=$(SHOW_ENV) 			\
 			$(if $(NOUSER),,				\
 				-v $(DOCKER_QEMU_CACHE_DIR):$(DOCKER_QEMU_CACHE_DIR) 	\
-- 
2.52.0

[PATCH 3/4] gitlab: ensure docker output is always displayed in CI

[Daniel P. Berrangé]

Set the new $(DOCKER_V) variable from the previous commit, so that any
CI jobs invoking docker will show the full stdout content. This improves
the ability to diagnose any build failures in CI that involve docker.

For example, when a 'docker build' command fails, it lets us see which
command in the Dockerfile failed and why.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 921c562000..7640a1d52c 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -28,6 +28,9 @@ variables:
     # we don't need. The --filter options avoid blobs and tree references we aren't going to use
     # and we also avoid fetching tags.
     GIT_FETCH_EXTRA_FLAGS: --filter=blob:none --filter=tree:0 --no-tags --prune --quiet
+    # Ensure docker.py / tests/docker/Makefile.include always displays stdout
+    # from any docker commands to aid debugging of failures
+    DOCKER_V: 1
 
   interruptible: true
 
-- 
2.52.0

[PATCH 4/4] tests/docker: add support for podman remote access

[Daniel P. Berrangé]

When a developer's environment is already within a podman container it
is not possible to use 'podman' again to create containers. It will
usually result in wierd errors such as:

  Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover

Podman offers the ability to talk to a daemon outside the container,
however, which could be leveraged by QEMU.

This can be used by invoking "podman --remote", or equivalently the
separate "podman-remote" binary:

  https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md

The current 'podman version' check is insufficient to detect the
inability to launch containers, so it is replaced with the stronger
'podman info' check.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/docker/docker.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index ff68c7bf6f..9e18b984f4 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -76,14 +76,16 @@ def _guess_engine_command():
     commands = []
 
     if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
-        commands += [["podman"]]
+        commands += [["podman"], ["podman-remote"], ["podman", "--remote"]]
     if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
         commands += [["docker"], ["sudo", "-n", "docker"]]
     for cmd in commands:
         try:
-            # docker version will return the client details in stdout
-            # but still report a status of 1 if it can't contact the daemon
-            if subprocess.call(cmd + ["version"],
+            # 'version' is not sufficient to prove a working binary
+            # for podman. 'info' is a stronger check that is more
+            # likely to correlate with ability to create containers,
+            # and required to detect the need for podman remote
+            if subprocess.call(cmd + ["info"],
                                stdout=DEVNULL, stderr=DEVNULL) == 0:
                 return cmd
         except OSError:
-- 
2.52.0

Re: [PATCH 0/4] tests/docker: improve detection of docker/podman

[Daniel P. Berrangé]

Ping: any thoughts on this series of CI improvements ?

On Wed, Jan 07, 2026 at 01:01:41PM +0000, Daniel P. Berrangé wrote:
> This improves integration such that tests/docker/Makefile.include will
> correctly honour the exact command detected by docker.py's probing
> logic. Currently the probe command gets stripped down to just a bare
> 'podman' or 'docker' command name. This means while commands run via
> 'docker.py' would use 'sudo -n docker', commands run directly from
> Makefile.include would only use 'docker'.
> 
> This series fixes that so that 'docker.py probe' correctly reports the
> full argv, and configure honours that argv untouched.
> 
> With that fixed, we can then also add support for 'podman --remote'
> and 'podman-remote', which allow use of podman when already inside
> podman which is the scenario for my development environment that is
> using 'toolbox'.
> 
> Finally this also improves CI by ensuring that stdout from docker is
> not thrown away, so we can have a chance of diagnosing build failures
> from CI.
> 
> Daniel P. Berrangé (4):
>   tests/docker: improve handling of docker probes
>   tests/docker: allow display of docker output
>   gitlab: ensure docker output is always displayed in CI
>   tests/docker: add support for podman remote access
> 
>  .gitlab-ci.d/base.yml         |  3 +++
>  configure                     | 19 +++++++------------
>  tests/docker/Makefile.include | 23 ++++++++++++++---------
>  tests/docker/docker.py        | 17 +++++++----------
>  4 files changed, 31 insertions(+), 31 deletions(-)
> 
> -- 
> 2.52.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 1/4] tests/docker: improve handling of docker probes

[Thomas Huth]

On 07/01/2026 14.01, Daniel P. Berrangé wrote:
> The docker.py script has logic to guess the container command and
> detects one of
> 
>    * docker
>    * sudo -n docker
>    * podman
> 
> but the "docker.py probe" command then throws away the detected argv
> and prints a slightly different argv based soley on the detected

s/soley/solely/ ?

> argv[0]. The result is that 'probe' will print
> 
>    * docker
>    * sudo docker
>    * podman
> 
> which means that if sudo was detected & the result of 'probe' were
> used direclty, it would end up prompting for password interaction

s/direclty/directly/

> every time.
> 
> The 'configure' script, however, runs 'probe' and then throws away
> the printed argv again, reporting only 'podman' or 'docker', which
> is used to set the $(RUNC) variable for tests/docker/Makefile.include
> which is in turn used to pass --engine to docker.py. So the docker.py
> command will re-detect the need for 'sudo -n' and use it correctly
> 
> The problem with this is that some commands in Makefile.include do
> not call docker.py at all, they invoke $(RUNC) directly. Since
> configure threw away the 'sudo' command prefix Makefile.in won't
> be adding either 'sudo' or 'sudo -n', it'll just run plani 'docker'
>

s/plani/plain/

Apart from the nits:
Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 2/4] tests/docker: allow display of docker output

[Thomas Huth]

On 07/01/2026 14.01, Daniel P. Berrangé wrote:
> The --quiet command is used with docker unless V=1 is passed to make,
> and as a result stdout from docker is never visible by default, making
> it hard to diagnose failures building / running containers.
> 
> Meanwhile passing V=1 is undesirable as that makes the entire build
> system verbose.
> 
> Introduce a $(DOCKER_V) make variable which is initialized from $(V)
> 
> It is thus possible to display docker output without also enabling
> make verbose output.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   tests/docker/Makefile.include | 18 +++++++++++-------
>   1 file changed, 11 insertions(+), 7 deletions(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 7d4582b6a8..d58a280333 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -40,14 +40,14 @@ docker-qemu-src: $(DOCKER_SRC_COPY)
>   docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>   	  $(call quiet-command,			\
>   		DOCKER_BUILDKIT=1 $(RUNC) build		\
> -		$(if $V,,--quiet)			\
> +		$(if $(DOCKER_V),,--quiet)		\

Hmm, do we maybe want a

DOCKER_V ?= $(V)

earlier in this file, so that DOCKER_V gets enabled when running in normal 
verbose mode, too?

  Thomas

Re: [PATCH 3/4] gitlab: ensure docker output is always displayed in CI

[Thomas Huth]

On 07/01/2026 14.01, Daniel P. Berrangé wrote:
> Set the new $(DOCKER_V) variable from the previous commit, so that any
> CI jobs invoking docker will show the full stdout content. This improves
> the ability to diagnose any build failures in CI that involve docker.
> 
> For example, when a 'docker build' command fails, it lets us see which
> command in the Dockerfile failed and why.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/base.yml | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index 921c562000..7640a1d52c 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -28,6 +28,9 @@ variables:
>       # we don't need. The --filter options avoid blobs and tree references we aren't going to use
>       # and we also avoid fetching tags.
>       GIT_FETCH_EXTRA_FLAGS: --filter=blob:none --filter=tree:0 --no-tags --prune --quiet
> +    # Ensure docker.py / tests/docker/Makefile.include always displays stdout
> +    # from any docker commands to aid debugging of failures
> +    DOCKER_V: 1

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 2/4] tests/docker: allow display of docker output

[Daniel P. Berrangé]

On Fri, Feb 06, 2026 at 02:48:34PM +0100, Thomas Huth wrote:
> On 07/01/2026 14.01, Daniel P. Berrangé wrote:
> > The --quiet command is used with docker unless V=1 is passed to make,
> > and as a result stdout from docker is never visible by default, making
> > it hard to diagnose failures building / running containers.
> > 
> > Meanwhile passing V=1 is undesirable as that makes the entire build
> > system verbose.
> > 
> > Introduce a $(DOCKER_V) make variable which is initialized from $(V)
> > 
> > It is thus possible to display docker output without also enabling
> > make verbose output.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   tests/docker/Makefile.include | 18 +++++++++++-------
> >   1 file changed, 11 insertions(+), 7 deletions(-)
> > 
> > diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> > index 7d4582b6a8..d58a280333 100644
> > --- a/tests/docker/Makefile.include
> > +++ b/tests/docker/Makefile.include
> > @@ -40,14 +40,14 @@ docker-qemu-src: $(DOCKER_SRC_COPY)
> >   docker-image-%: $(DOCKER_FILES_DIR)/%.docker
> >   	  $(call quiet-command,			\
> >   		DOCKER_BUILDKIT=1 $(RUNC) build		\
> > -		$(if $V,,--quiet)			\
> > +		$(if $(DOCKER_V),,--quiet)		\
> 
> Hmm, do we maybe want a
> 
> DOCKER_V ?= $(V)
> 
> earlier in this file, so that DOCKER_V gets enabled when running in normal
> verbose mode, too?

I was on the fence about it. V=1 is traditionally focused on details of the
commands being executed, rather than their output, but I don't mind much
either way. 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|