[PATCH v2 0/4] tests/docker: improve detection of docker/podman

[PATCH v2 0/4] tests/docker: improve detection of docker/podman

[Daniel P. Berrangé]

This improves integration such that tests/docker/Makefile.include will
correctly honour the exact command detected by docker.py's probing
logic. Currently the probe command gets stripped down to just a bare
'podman' or 'docker' command name. This means while commands run via
'docker.py' would use 'sudo -n docker', commands run directly from
Makefile.include would only use 'docker'.

This series fixes that so that 'docker.py probe' correctly reports the
full argv, and configure honours that argv untouched.

With that fixed, we can then also add support for 'podman --remote'
and 'podman-remote', which allow use of podman when already inside
podman which is the scenario for my development environment that is
using 'toolbox'.

Finally this also improves CI by ensuring that stdout from docker is
not thrown away, so we can have a chance of diagnosing build failures
from CI.

Changed in v2:

 - Fix misc typos
 - Default $(DOCKER_V) value from $(V)

Daniel P. Berrangé (4):
  tests/docker: improve handling of docker probes
  tests/docker: add support for podman remote access
  tests/docker: allow display of docker output
  gitlab: ensure docker output is always displayed in CI

 .gitlab-ci.d/base.yml         |  3 +++
 configure                     | 19 +++++++------------
 tests/docker/Makefile.include | 25 ++++++++++++++++---------
 tests/docker/docker.py        | 17 +++++++----------
 4 files changed, 33 insertions(+), 31 deletions(-)

-- 
2.53.0

[PATCH v2 1/4] tests/docker: improve handling of docker probes

[Daniel P. Berrangé]

The docker.py script has logic to guess the container command and
detects one of

  * docker
  * sudo -n docker
  * podman

but the "docker.py probe" command then throws away the detected argv
and prints a slightly different argv based solely on the detected
argv[0]. The result is that 'probe' will print

  * docker
  * sudo docker
  * podman

which means that if sudo was detected & the result of 'probe' were
used directly, it would end up prompting for password interaction
every time.

The 'configure' script, however, runs 'probe' and then throws away
the printed argv again, reporting only 'podman' or 'docker', which
is used to set the $(RUNC) variable for tests/docker/Makefile.include
which is in turn used to pass --engine to docker.py. So the docker.py
command will re-detect the need for 'sudo -n' and use it correctly

The problem with this is that some commands in Makefile.include do
not call docker.py at all, they invoke $(RUNC) directly. Since
configure threw away the 'sudo' command prefix Makefile.in won't
be adding either 'sudo' or 'sudo -n', it'll just run plain 'docker'
which is wrong.

This commit sanitizes things so that the 'docker.py probe' prints
out the exact detected ARGV, and configure fully preserves this
ARGV when setting $(RUNC). Since "$(RUNC)" is no longer just a bare
engine name, however, we must now also set the $(CONTAINER_ENGINE)
variable for Makefile.include so it can pass something sane to
the --engine arg for docker.py

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 configure                     | 19 +++++++------------
 tests/docker/Makefile.include |  5 +++--
 tests/docker/docker.py        |  7 +------
 3 files changed, 11 insertions(+), 20 deletions(-)

diff --git a/configure b/configure
index 4b61fd3bbf..ecb446d5f6 100755
--- a/configure
+++ b/configure
@@ -1311,17 +1311,11 @@ fi
 ##########################################
 # functions to probe cross compilers
 
-container="no"
-runc=""
+runc="no"
 if test $use_containers = "yes" && (has "docker" || has "podman"); then
-    case $($python "$source_path"/tests/docker/docker.py --engine "$container_engine" probe) in
-        *docker) container=docker ;;
-        podman) container=podman ;;
-        no) container=no ;;
-    esac
-    if test "$container" != "no"; then
-        docker_py="$python $source_path/tests/docker/docker.py --engine $container"
-        runc=$container
+    runc=$($python "$source_path"/tests/docker/docker.py --engine "$container_engine" probe)
+    if test "$runc" != "no"; then
+        docker_py="$python $source_path/tests/docker/docker.py --engine $container_engine"
     fi
 fi
 
@@ -1441,7 +1435,7 @@ probe_target_compiler() {
   esac
 
   for host in $container_hosts; do
-    test "$container" != no || continue
+    test "$runc" != no || continue
     test "$host" = "$cpu" || continue
     case $target_arch in
       # debian-all-test-cross architectures
@@ -1771,8 +1765,9 @@ echo all: >> $config_host_mak
 echo "SRC_PATH=$source_path" >> $config_host_mak
 echo "TARGET_DIRS=$target_list" >> $config_host_mak
 echo "GDB=$gdb_bin" >> $config_host_mak
-if test "$container" != no; then
+if test "$runc" != no; then
     echo "RUNC=$runc" >> $config_host_mak
+    echo "CONTAINER_ENGINE=$container_engine" >> $config_host_mak
 fi
 echo "SUBDIRS=$subdirs" >> $config_host_mak
 if test "$rust" != disabled; then
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 38467cca61..7d4582b6a8 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -16,8 +16,9 @@ DOCKER_DEFAULT_REGISTRY := registry.gitlab.com/qemu-project/qemu
 endif
 DOCKER_REGISTRY := $(if $(REGISTRY),$(REGISTRY),$(DOCKER_DEFAULT_REGISTRY))
 
-RUNC ?= $(if $(shell command -v docker), docker, podman)
-DOCKER_SCRIPT=$(SRC_PATH)/tests/docker/docker.py --engine $(RUNC)
+CONTAINER_ENGINE = auto
+DOCKER_SCRIPT=$(SRC_PATH)/tests/docker/docker.py --engine $(CONTAINER_ENGINE)
+RUNC ?= $(shell $(DOCKER_SCRIPT) probe)
 
 CUR_TIME := $(shell date +%Y-%m-%d-%H.%M.%S.$$$$)
 DOCKER_SRC_COPY := $(BUILD_DIR)/docker-src.$(CUR_TIME)
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 3b8a26704d..ff68c7bf6f 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -618,12 +618,7 @@ class ProbeCommand(SubCommand):
     def run(self, args, argv):
         try:
             docker = Docker()
-            if docker._command[0] == "docker":
-                print("docker")
-            elif docker._command[0] == "sudo":
-                print("sudo docker")
-            elif docker._command[0] == "podman":
-                print("podman")
+            print(" ".join(docker._command))
         except Exception:
             print("no")
 
-- 
2.53.0

[PATCH v2 2/4] tests/docker: add support for podman remote access

[Daniel P. Berrangé]

When a developer's environment is already within a podman container it
is not possible to use 'podman' again to create containers. It will
usually result in wierd errors such as:

  Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover

Podman offers the ability to talk to a daemon outside the container,
however, which could be leveraged by QEMU.

This can be used by invoking "podman --remote", or equivalently the
separate "podman-remote" binary:

  https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md

The current 'podman version' check is insufficient to detect the
inability to launch containers, so it is replaced with the stronger
'podman info' check.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/docker/docker.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index ff68c7bf6f..9e18b984f4 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -76,14 +76,16 @@ def _guess_engine_command():
     commands = []
 
     if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
-        commands += [["podman"]]
+        commands += [["podman"], ["podman-remote"], ["podman", "--remote"]]
     if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
         commands += [["docker"], ["sudo", "-n", "docker"]]
     for cmd in commands:
         try:
-            # docker version will return the client details in stdout
-            # but still report a status of 1 if it can't contact the daemon
-            if subprocess.call(cmd + ["version"],
+            # 'version' is not sufficient to prove a working binary
+            # for podman. 'info' is a stronger check that is more
+            # likely to correlate with ability to create containers,
+            # and required to detect the need for podman remote
+            if subprocess.call(cmd + ["info"],
                                stdout=DEVNULL, stderr=DEVNULL) == 0:
                 return cmd
         except OSError:
-- 
2.53.0

[PATCH v2 3/4] tests/docker: allow display of docker output

[Daniel P. Berrangé]

The --quiet command is used with docker unless V=1 is passed to make,
and as a result stdout from docker is never visible by default, making
it hard to diagnose failures building / running containers.

Meanwhile passing V=1 is undesirable as that makes the entire build
system verbose.

Introduce a $(DOCKER_V) make variable which is initialized from $(V)

It is thus possible to display docker output without also enabling
make verbose output.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/docker/Makefile.include | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 7d4582b6a8..df14538c0f 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -23,6 +23,8 @@ RUNC ?= $(shell $(DOCKER_SCRIPT) probe)
 CUR_TIME := $(shell date +%Y-%m-%d-%H.%M.%S.$$$$)
 DOCKER_SRC_COPY := $(BUILD_DIR)/docker-src.$(CUR_TIME)
 
+DOCKER_V ?= $(V)
+
 .DELETE_ON_ERROR: $(DOCKER_SRC_COPY)
 $(DOCKER_SRC_COPY):
 	@mkdir $@
@@ -40,14 +42,14 @@ docker-qemu-src: $(DOCKER_SRC_COPY)
 docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 	  $(call quiet-command,			\
 		DOCKER_BUILDKIT=1 $(RUNC) build		\
-		$(if $V,,--quiet)			\
+		$(if $(DOCKER_V),,--quiet)		\
 		$(if $(NOCACHE),--no-cache,		\
 			$(if $(DOCKER_REGISTRY),--cache-from $(DOCKER_REGISTRY)/qemu/$*)) \
 		--build-arg BUILDKIT_INLINE_CACHE=1 	\
 		$(if $(NOUSER),,			\
 			--build-arg USER=$(USER)	\
 			--build-arg UID=$(UID))		\
-		-t qemu/$* - < $< $(if $V,,> /dev/null),\
+		-t qemu/$* - < $< $(if $(DOCKER_V),,> /dev/null),\
 		"BUILD", $*)
 
 # General rule for inspecting registry images.
@@ -73,7 +75,7 @@ docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
 			DEB_TYPE=$(DEB_TYPE) 					\
 			$(if $(DEB_URL),DEB_URL=$(DEB_URL),)			\
 			$(DOCKER_SCRIPT) build -t qemu/debian-$* -f $< 		\
-			$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) 		\
+			$(if $(DOCKER_V),,--quiet) $(if $(NOCACHE),--no-cache) 	\
 			$(if $(NOUSER),,--add-current-user) 			\
 			$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))	\
 			$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)), \
@@ -105,16 +107,17 @@ debian-toolchain-run = \
 	$(if $(NOCACHE)$(NOFETCH),					\
 		$(call quiet-command,					\
 			$(DOCKER_SCRIPT) build -t qemu/$1 -f $< 	\
-			$(if $V,,--quiet) $(if $(NOCACHE),--no-cache)	\
+			$(if $(DOCKER_V),,--quiet) 			\
+			$(if $(NOCACHE),--no-cache)			\
 			--registry $(DOCKER_REGISTRY) --extra-files	\
 			$(DOCKER_FILES_DIR)/$1.d/build-toolchain.sh,	\
 			"BUILD", $1),				        \
 		$(call quiet-command,					\
-			$(DOCKER_SCRIPT) fetch $(if $V,,--quiet)	\
+			$(DOCKER_SCRIPT) fetch $(if $(DOCKER_V),,--quiet) \
 				qemu/$1 $(DOCKER_REGISTRY),		\
 			"FETCH", $1)					\
 		$(call quiet-command,					\
-			$(DOCKER_SCRIPT) update $(if $V,,--quiet) 	\
+			$(DOCKER_SCRIPT) update $(if $(DOCKER_V),,--quiet) \
 				qemu/$1 				\
 				$(if $(NOUSER),,--add-current-user) 	\
 			"PREPARE", $1))
@@ -231,7 +234,10 @@ docker-run: docker-qemu-src
 			-e TARGET_LIST=$(subst $(SPACE),$(COMMA),$(TARGET_LIST))	\
 			-e EXTRA_CONFIGURE_OPTS="$(EXTRA_CONFIGURE_OPTS)" \
 			-e TEST_COMMAND="$(TEST_COMMAND)" 		\
-			-e V=$V -e J=$J -e DEBUG=$(DEBUG)		\
+			-e V=$V						\
+			-e DOCKER_V=$(DOCKER_V)				\
+			-e J=$J						\
+			-e DEBUG=$(DEBUG)				\
 			-e SHOW_ENV=$(SHOW_ENV) 			\
 			$(if $(NOUSER),,				\
 				-v $(DOCKER_QEMU_CACHE_DIR):$(DOCKER_QEMU_CACHE_DIR) 	\
-- 
2.53.0

[PATCH v2 4/4] gitlab: ensure docker output is always displayed in CI

[Daniel P. Berrangé]

Set the new $(DOCKER_V) variable from the previous commit, so that any
CI jobs invoking docker will show the full stdout content. This improves
the ability to diagnose any build failures in CI that involve docker.

For example, when a 'docker build' command fails, it lets us see which
command in the Dockerfile failed and why.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 921c562000..7640a1d52c 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -28,6 +28,9 @@ variables:
     # we don't need. The --filter options avoid blobs and tree references we aren't going to use
     # and we also avoid fetching tags.
     GIT_FETCH_EXTRA_FLAGS: --filter=blob:none --filter=tree:0 --no-tags --prune --quiet
+    # Ensure docker.py / tests/docker/Makefile.include always displays stdout
+    # from any docker commands to aid debugging of failures
+    DOCKER_V: 1
 
   interruptible: true
 
-- 
2.53.0

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Thomas Huth]

On 10/02/2026 17.35, Daniel P. Berrangé wrote:
> When a developer's environment is already within a podman container it
> is not possible to use 'podman' again to create containers. It will
> usually result in wierd errors such as:
> 
>    Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> 
> Podman offers the ability to talk to a daemon outside the container,
> however, which could be leveraged by QEMU.
> 
> This can be used by invoking "podman --remote", or equivalently the
> separate "podman-remote" binary:
> 
>    https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md
> 
> The current 'podman version' check is insufficient to detect the
> inability to launch containers, so it is replaced with the stronger
> 'podman info' check.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   tests/docker/docker.py | 10 ++++++----
>   1 file changed, 6 insertions(+), 4 deletions(-)

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v2 3/4] tests/docker: allow display of docker output

[Thomas Huth]

On 10/02/2026 17.35, Daniel P. Berrangé wrote:
> The --quiet command is used with docker unless V=1 is passed to make,
> and as a result stdout from docker is never visible by default, making
> it hard to diagnose failures building / running containers.
> 
> Meanwhile passing V=1 is undesirable as that makes the entire build
> system verbose.
> 
> Introduce a $(DOCKER_V) make variable which is initialized from $(V)
> 
> It is thus possible to display docker output without also enabling
> make verbose output.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   tests/docker/Makefile.include | 20 +++++++++++++-------
>   1 file changed, 13 insertions(+), 7 deletions(-)

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Pierrick Bouvier]

On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
> When a developer's environment is already within a podman container it
> is not possible to use 'podman' again to create containers. It will
> usually result in wierd errors such as:
> 
>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> 
> Podman offers the ability to talk to a daemon outside the container,
> however, which could be leveraged by QEMU.
> 
> This can be used by invoking "podman --remote", or equivalently the
> separate "podman-remote" binary:
> 
>   https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md
> 
> The current 'podman version' check is insufficient to detect the
> inability to launch containers, so it is replaced with the stronger
> 'podman info' check.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  tests/docker/docker.py | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index ff68c7bf6f..9e18b984f4 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -76,14 +76,16 @@ def _guess_engine_command():
>      commands = []
>  
>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
> -        commands += [["podman"]]
> +        commands += [["podman"], ["podman-remote"], ["podman", "--remote"]]
>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
>          commands += [["docker"], ["sudo", "-n", "docker"]]
>      for cmd in commands:
>          try:
> -            # docker version will return the client details in stdout
> -            # but still report a status of 1 if it can't contact the daemon
> -            if subprocess.call(cmd + ["version"],
> +            # 'version' is not sufficient to prove a working binary
> +            # for podman. 'info' is a stronger check that is more
> +            # likely to correlate with ability to create containers,
> +            # and required to detect the need for podman remote
> +            if subprocess.call(cmd + ["info"],
>                                 stdout=DEVNULL, stderr=DEVNULL) == 0:
>                  return cmd
>          except OSError:

Taking a look at why tcg tests are slow to compile, I reached this
commit. It seems like that calling 'podman info' is 5 to 10 times slower
than calling 'podman version'.
Thus, a container run now takes +1s when it was 0.2 before.

On my setup, podman (remote) version fails correctly if it can't
establish a connection.

The commit description briefly says
```
The current 'podman version' check is insufficient to detect the
inability to launch containers, so it is replaced with the stronger
'podman info' check.
```
but it's not what I observed.

Do you have more information about what is the exact reason we can't use
'podman version' for podman-remote?

Regards,
Pierrick

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Daniel P. Berrangé]

On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
> > When a developer's environment is already within a podman container it
> > is not possible to use 'podman' again to create containers. It will
> > usually result in wierd errors such as:
> > 
> >   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> > 
> > Podman offers the ability to talk to a daemon outside the container,
> > however, which could be leveraged by QEMU.
> > 
> > This can be used by invoking "podman --remote", or equivalently the
> > separate "podman-remote" binary:
> > 
> >   https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md
> > 
> > The current 'podman version' check is insufficient to detect the
> > inability to launch containers, so it is replaced with the stronger
> > 'podman info' check.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  tests/docker/docker.py | 10 ++++++----
> >  1 file changed, 6 insertions(+), 4 deletions(-)
> > 
> > diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> > index ff68c7bf6f..9e18b984f4 100755
> > --- a/tests/docker/docker.py
> > +++ b/tests/docker/docker.py
> > @@ -76,14 +76,16 @@ def _guess_engine_command():
> >      commands = []
> >  
> >      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
> > -        commands += [["podman"]]
> > +        commands += [["podman"], ["podman-remote"], ["podman", "--remote"]]
> >      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
> >          commands += [["docker"], ["sudo", "-n", "docker"]]
> >      for cmd in commands:
> >          try:
> > -            # docker version will return the client details in stdout
> > -            # but still report a status of 1 if it can't contact the daemon
> > -            if subprocess.call(cmd + ["version"],
> > +            # 'version' is not sufficient to prove a working binary
> > +            # for podman. 'info' is a stronger check that is more
> > +            # likely to correlate with ability to create containers,
> > +            # and required to detect the need for podman remote
> > +            if subprocess.call(cmd + ["info"],
> >                                 stdout=DEVNULL, stderr=DEVNULL) == 0:
> >                  return cmd
> >          except OSError:
> 
> Taking a look at why tcg tests are slow to compile, I reached this
> commit. It seems like that calling 'podman info' is 5 to 10 times slower
> than calling 'podman version'.
> Thus, a container run now takes +1s when it was 0.2 before.
> 
> On my setup, podman (remote) version fails correctly if it can't
> establish a connection.
> 
> The commit description briefly says
> ```
> The current 'podman version' check is insufficient to detect the
> inability to launch containers, so it is replaced with the stronger
> 'podman info' check.
> ```
> but it's not what I observed.
> 
> Do you have more information about what is the exact reason we can't use
> 'podman version' for podman-remote?

My developer env involves toolbox, which is a wrapper around podman:

"podman version" is not an operational check - it is effectively
nothing more than a variant of command line "help". Just shows
the binary can print to stdout.

By contrast "docker version" is useful as IIUC it shows you can
connect to docker daemon, but podman is daemonless so the 'version'
chck doesn't offer any useful proof.

"podman info" is an operational check that correlates with the
ability to actually create containers.

⚙️  [host:fedora-44 ~]$ toolbox enter almalinux-toolbox-9 
⚙️  [podman:almalinux-9.8 ~]$ sudo dnf -y install podman
⚙️  [podman:almalinux-9.8 ~] podman version
Client:       Podman Engine
Version:      5.8.2
API Version:  5.8.2
Go Version:   go1.26.3 (Red Hat 1.26.3-1.el9_8)
Built:        Tue Jun 16 23:52:20 2026
OS/Arch:      linux/amd64

⚙️  [podman:almalinux-9.8 ~]$ podman info
Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover

⚙️  [podman:almalinux-9.8 ~] $ podman --remote info
OS: linux/amd64
provider: qemu
version: 5.8.2
..snip...


With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Pierrick Bouvier]

On 7/9/2026 8:40 AM, Daniel P. Berrangé wrote:
> On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
>> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
>>> When a developer's environment is already within a podman container it
>>> is not possible to use 'podman' again to create containers. It will
>>> usually result in wierd errors such as:
>>>
>>>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
>>>
>>> Podman offers the ability to talk to a daemon outside the container,
>>> however, which could be leveraged by QEMU.
>>>
>>> This can be used by invoking "podman --remote", or equivalently the
>>> separate "podman-remote" binary:
>>>
>>>   https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md
>>>
>>> The current 'podman version' check is insufficient to detect the
>>> inability to launch containers, so it is replaced with the stronger
>>> 'podman info' check.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>  tests/docker/docker.py | 10 ++++++----
>>>  1 file changed, 6 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>>> index ff68c7bf6f..9e18b984f4 100755
>>> --- a/tests/docker/docker.py
>>> +++ b/tests/docker/docker.py
>>> @@ -76,14 +76,16 @@ def _guess_engine_command():
>>>      commands = []
>>>  
>>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
>>> -        commands += [["podman"]]
>>> +        commands += [["podman"], ["podman-remote"], ["podman", "--remote"]]
>>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
>>>          commands += [["docker"], ["sudo", "-n", "docker"]]
>>>      for cmd in commands:
>>>          try:
>>> -            # docker version will return the client details in stdout
>>> -            # but still report a status of 1 if it can't contact the daemon
>>> -            if subprocess.call(cmd + ["version"],
>>> +            # 'version' is not sufficient to prove a working binary
>>> +            # for podman. 'info' is a stronger check that is more
>>> +            # likely to correlate with ability to create containers,
>>> +            # and required to detect the need for podman remote
>>> +            if subprocess.call(cmd + ["info"],
>>>                                 stdout=DEVNULL, stderr=DEVNULL) == 0:
>>>                  return cmd
>>>          except OSError:
>>
>> Taking a look at why tcg tests are slow to compile, I reached this
>> commit. It seems like that calling 'podman info' is 5 to 10 times slower
>> than calling 'podman version'.
>> Thus, a container run now takes +1s when it was 0.2 before.
>>
>> On my setup, podman (remote) version fails correctly if it can't
>> establish a connection.
>>
>> The commit description briefly says
>> ```
>> The current 'podman version' check is insufficient to detect the
>> inability to launch containers, so it is replaced with the stronger
>> 'podman info' check.
>> ```
>> but it's not what I observed.
>>
>> Do you have more information about what is the exact reason we can't use
>> 'podman version' for podman-remote?
> 
> My developer env involves toolbox, which is a wrapper around podman:
> 
> "podman version" is not an operational check - it is effectively
> nothing more than a variant of command line "help". Just shows
> the binary can print to stdout.
> 
> By contrast "docker version" is useful as IIUC it shows you can
> connect to docker daemon, but podman is daemonless so the 'version'
> chck doesn't offer any useful proof.
> 
> "podman info" is an operational check that correlates with the
> ability to actually create containers.
> 
> ⚙️  [host:fedora-44 ~]$ toolbox enter almalinux-toolbox-9 
> ⚙️  [podman:almalinux-9.8 ~]$ sudo dnf -y install podman
> ⚙️  [podman:almalinux-9.8 ~] podman version
> Client:       Podman Engine
> Version:      5.8.2
> API Version:  5.8.2
> Go Version:   go1.26.3 (Red Hat 1.26.3-1.el9_8)
> Built:        Tue Jun 16 23:52:20 2026
> OS/Arch:      linux/amd64
> 
> ⚙️  [podman:almalinux-9.8 ~]$ podman info
> Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> 
> ⚙️  [podman:almalinux-9.8 ~] $ podman --remote info
> OS: linux/amd64
> provider: qemu
> version: 5.8.2
> ..snip...
>

In this case, isn't the problem is that the environment should not have
podman installed if it can't actually create containers?

podman info is not enough neither considering this:
$ podman run -it --rm docker.io/fedora
$ dnf install -y podman
$ podman version && podman info && echo "it works"
$ podman run -it --rm docker.io/fedora
...
Error: unable to copy from source docker://fedora:latest: copying system
image from manifest list: writing blob: adding layer with blob
"sha256:594bd6d79d070efa3007bb9bc092a4889026b51ebcaa7d396d46276a0f315e5e"/""/"sha256:44157061143cb91d5818b1f3d2dc2109476e3efcf3eb9e495c9eb5a7db49c170":
unpacking failed (error: exit status 1; output: potentially insufficient
UIDs or GIDs available in user namespace (requested 0:12 for
/var/spool/mail): Check /etc/subuid and /etc/subgid if configured
locally and run "podman system migrate": lchown /var/spool/mail: invalid
argument)

Considering this, I think it would be better to have a "fast" default,
that only checks version, and leave the rest to the user to make sure
their env is correctly configured (or that they remove podman if that's
not the case).

What do you think?

> 
> With regards,
> Daniel

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Daniel P. Berrangé]

On Thu, Jul 09, 2026 at 08:55:08AM -0700, Pierrick Bouvier wrote:
> On 7/9/2026 8:40 AM, Daniel P. Berrangé wrote:
> > On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
> >> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
> >>> When a developer's environment is already within a podman container it
> >>> is not possible to use 'podman' again to create containers. It will
> >>> usually result in wierd errors such as:
> >>>
> >>>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> >>>
> >>> Podman offers the ability to talk to a daemon outside the container,
> >>> however, which could be leveraged by QEMU.
> >>>
> >>> This can be used by invoking "podman --remote", or equivalently the
> >>> separate "podman-remote" binary:
> >>>
> >>>   https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md
> >>>
> >>> The current 'podman version' check is insufficient to detect the
> >>> inability to launch containers, so it is replaced with the stronger
> >>> 'podman info' check.
> >>>
> >>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >>> ---
> >>>  tests/docker/docker.py | 10 ++++++----
> >>>  1 file changed, 6 insertions(+), 4 deletions(-)
> >>>
> >>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> >>> index ff68c7bf6f..9e18b984f4 100755
> >>> --- a/tests/docker/docker.py
> >>> +++ b/tests/docker/docker.py
> >>> @@ -76,14 +76,16 @@ def _guess_engine_command():
> >>>      commands = []
> >>>  
> >>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
> >>> -        commands += [["podman"]]
> >>> +        commands += [["podman"], ["podman-remote"], ["podman", "--remote"]]
> >>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
> >>>          commands += [["docker"], ["sudo", "-n", "docker"]]
> >>>      for cmd in commands:
> >>>          try:
> >>> -            # docker version will return the client details in stdout
> >>> -            # but still report a status of 1 if it can't contact the daemon
> >>> -            if subprocess.call(cmd + ["version"],
> >>> +            # 'version' is not sufficient to prove a working binary
> >>> +            # for podman. 'info' is a stronger check that is more
> >>> +            # likely to correlate with ability to create containers,
> >>> +            # and required to detect the need for podman remote
> >>> +            if subprocess.call(cmd + ["info"],
> >>>                                 stdout=DEVNULL, stderr=DEVNULL) == 0:
> >>>                  return cmd
> >>>          except OSError:
> >>
> >> Taking a look at why tcg tests are slow to compile, I reached this
> >> commit. It seems like that calling 'podman info' is 5 to 10 times slower
> >> than calling 'podman version'.
> >> Thus, a container run now takes +1s when it was 0.2 before.
> >>
> >> On my setup, podman (remote) version fails correctly if it can't
> >> establish a connection.
> >>
> >> The commit description briefly says
> >> ```
> >> The current 'podman version' check is insufficient to detect the
> >> inability to launch containers, so it is replaced with the stronger
> >> 'podman info' check.
> >> ```
> >> but it's not what I observed.
> >>
> >> Do you have more information about what is the exact reason we can't use
> >> 'podman version' for podman-remote?
> > 
> > My developer env involves toolbox, which is a wrapper around podman:
> > 
> > "podman version" is not an operational check - it is effectively
> > nothing more than a variant of command line "help". Just shows
> > the binary can print to stdout.
> > 
> > By contrast "docker version" is useful as IIUC it shows you can
> > connect to docker daemon, but podman is daemonless so the 'version'
> > chck doesn't offer any useful proof.
> > 
> > "podman info" is an operational check that correlates with the
> > ability to actually create containers.
> > 
> > ⚙️  [host:fedora-44 ~]$ toolbox enter almalinux-toolbox-9 
> > ⚙️  [podman:almalinux-9.8 ~]$ sudo dnf -y install podman
> > ⚙️  [podman:almalinux-9.8 ~] podman version
> > Client:       Podman Engine
> > Version:      5.8.2
> > API Version:  5.8.2
> > Go Version:   go1.26.3 (Red Hat 1.26.3-1.el9_8)
> > Built:        Tue Jun 16 23:52:20 2026
> > OS/Arch:      linux/amd64
> > 
> > ⚙️  [podman:almalinux-9.8 ~]$ podman info
> > Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> > 
> > ⚙️  [podman:almalinux-9.8 ~] $ podman --remote info
> > OS: linux/amd64
> > provider: qemu
> > version: 5.8.2
> > ..snip...
> >
> 
> In this case, isn't the problem is that the environment should not have
> podman installed if it can't actually create containers?

podman *can* create containers by using the '--remote' flag, which
is what we made work in QEMU.

> Considering this, I think it would be better to have a "fast" default,
> that only checks version, and leave the rest to the user to make sure
> their env is correctly configured (or that they remove podman if that's
> not the case).

My env is correctly configured so that podman --remote works. Doing
onmly a version check will make QEMU not work once more.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Pierrick Bouvier]

On 7/9/2026 9:01 AM, Daniel P. Berrangé wrote:
> On Thu, Jul 09, 2026 at 08:55:08AM -0700, Pierrick Bouvier wrote:
>> On 7/9/2026 8:40 AM, Daniel P. Berrangé wrote:
>>> On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
>>>> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
>>>>> When a developer's environment is already within a podman container it
>>>>> is not possible to use 'podman' again to create containers. It will
>>>>> usually result in wierd errors such as:
>>>>>
>>>>>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
>>>>>
>>>>> Podman offers the ability to talk to a daemon outside the container,
>>>>> however, which could be leveraged by QEMU.
>>>>>
>>>>> This can be used by invoking "podman --remote", or equivalently the
>>>>> separate "podman-remote" binary:
>>>>>
>>>>>   https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md
>>>>>
>>>>> The current 'podman version' check is insufficient to detect the
>>>>> inability to launch containers, so it is replaced with the stronger
>>>>> 'podman info' check.
>>>>>
>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>> ---
>>>>>  tests/docker/docker.py | 10 ++++++----
>>>>>  1 file changed, 6 insertions(+), 4 deletions(-)
>>>>>
>>>>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>>>>> index ff68c7bf6f..9e18b984f4 100755
>>>>> --- a/tests/docker/docker.py
>>>>> +++ b/tests/docker/docker.py
>>>>> @@ -76,14 +76,16 @@ def _guess_engine_command():
>>>>>      commands = []
>>>>>  
>>>>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
>>>>> -        commands += [["podman"]]
>>>>> +        commands += [["podman"], ["podman-remote"], ["podman", "--remote"]]
>>>>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
>>>>>          commands += [["docker"], ["sudo", "-n", "docker"]]
>>>>>      for cmd in commands:
>>>>>          try:
>>>>> -            # docker version will return the client details in stdout
>>>>> -            # but still report a status of 1 if it can't contact the daemon
>>>>> -            if subprocess.call(cmd + ["version"],
>>>>> +            # 'version' is not sufficient to prove a working binary
>>>>> +            # for podman. 'info' is a stronger check that is more
>>>>> +            # likely to correlate with ability to create containers,
>>>>> +            # and required to detect the need for podman remote
>>>>> +            if subprocess.call(cmd + ["info"],
>>>>>                                 stdout=DEVNULL, stderr=DEVNULL) == 0:
>>>>>                  return cmd
>>>>>          except OSError:
>>>>
>>>> Taking a look at why tcg tests are slow to compile, I reached this
>>>> commit. It seems like that calling 'podman info' is 5 to 10 times slower
>>>> than calling 'podman version'.
>>>> Thus, a container run now takes +1s when it was 0.2 before.
>>>>
>>>> On my setup, podman (remote) version fails correctly if it can't
>>>> establish a connection.
>>>>
>>>> The commit description briefly says
>>>> ```
>>>> The current 'podman version' check is insufficient to detect the
>>>> inability to launch containers, so it is replaced with the stronger
>>>> 'podman info' check.
>>>> ```
>>>> but it's not what I observed.
>>>>
>>>> Do you have more information about what is the exact reason we can't use
>>>> 'podman version' for podman-remote?
>>>
>>> My developer env involves toolbox, which is a wrapper around podman:
>>>
>>> "podman version" is not an operational check - it is effectively
>>> nothing more than a variant of command line "help". Just shows
>>> the binary can print to stdout.
>>>
>>> By contrast "docker version" is useful as IIUC it shows you can
>>> connect to docker daemon, but podman is daemonless so the 'version'
>>> chck doesn't offer any useful proof.
>>>
>>> "podman info" is an operational check that correlates with the
>>> ability to actually create containers.
>>>
>>> ⚙️  [host:fedora-44 ~]$ toolbox enter almalinux-toolbox-9 
>>> ⚙️  [podman:almalinux-9.8 ~]$ sudo dnf -y install podman
>>> ⚙️  [podman:almalinux-9.8 ~] podman version
>>> Client:       Podman Engine
>>> Version:      5.8.2
>>> API Version:  5.8.2
>>> Go Version:   go1.26.3 (Red Hat 1.26.3-1.el9_8)
>>> Built:        Tue Jun 16 23:52:20 2026
>>> OS/Arch:      linux/amd64
>>>
>>> ⚙️  [podman:almalinux-9.8 ~]$ podman info
>>> Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
>>>
>>> ⚙️  [podman:almalinux-9.8 ~] $ podman --remote info
>>> OS: linux/amd64
>>> provider: qemu
>>> version: 5.8.2
>>> ..snip...
>>>
>>
>> In this case, isn't the problem is that the environment should not have
>> podman installed if it can't actually create containers?
> 
> podman *can* create containers by using the '--remote' flag, which
> is what we made work in QEMU.
> 
>> Considering this, I think it would be better to have a "fast" default,
>> that only checks version, and leave the rest to the user to make sure
>> their env is correctly configured (or that they remove podman if that's
>> not the case).
> 
> My env is correctly configured so that podman --remote works. Doing
> onmly a version check will make QEMU not work once more.
>

In this case, maybe we can test first podman --remote, before podman.
If it can't connect to podman socket, version returns an error as expected.

This way, it *works* on your personal setup, and it's fast for everyone
else.

Sounds like a good compromise?

> With regards,
> Daniel

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Daniel P. Berrangé]

On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
> > When a developer's environment is already within a podman container it
> > is not possible to use 'podman' again to create containers. It will
> > usually result in wierd errors such as:
> > 
> >   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> > 
> > Podman offers the ability to talk to a daemon outside the container,
> > however, which could be leveraged by QEMU.
> > 
> > This can be used by invoking "podman --remote", or equivalently the
> > separate "podman-remote" binary:
> > 
> >   https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md
> > 
> > The current 'podman version' check is insufficient to detect the
> > inability to launch containers, so it is replaced with the stronger
> > 'podman info' check.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  tests/docker/docker.py | 10 ++++++----
> >  1 file changed, 6 insertions(+), 4 deletions(-)
> > 
> > diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> > index ff68c7bf6f..9e18b984f4 100755
> > --- a/tests/docker/docker.py
> > +++ b/tests/docker/docker.py
> > @@ -76,14 +76,16 @@ def _guess_engine_command():
> >      commands = []
> >  
> >      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
> > -        commands += [["podman"]]
> > +        commands += [["podman"], ["podman-remote"], ["podman", "--remote"]]
> >      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
> >          commands += [["docker"], ["sudo", "-n", "docker"]]
> >      for cmd in commands:
> >          try:
> > -            # docker version will return the client details in stdout
> > -            # but still report a status of 1 if it can't contact the daemon
> > -            if subprocess.call(cmd + ["version"],
> > +            # 'version' is not sufficient to prove a working binary
> > +            # for podman. 'info' is a stronger check that is more
> > +            # likely to correlate with ability to create containers,
> > +            # and required to detect the need for podman remote
> > +            if subprocess.call(cmd + ["info"],
> >                                 stdout=DEVNULL, stderr=DEVNULL) == 0:
> >                  return cmd
> >          except OSError:
> 
> Taking a look at why tcg tests are slow to compile, I reached this
> commit. It seems like that calling 'podman info' is 5 to 10 times slower
> than calling 'podman version'.
> Thus, a container run now takes +1s when it was 0.2 before.

Hmmm, but we call 'docker.py probe' in configure and cache the result,
so that probe should only be done once.  0.2 vs 1s is just noise in
the context of configure.

That you're seeing any negative effects suggests that we've got
something not using the cached probe result, which we should fix.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Daniel P. Berrangé]

On Thu, Jul 09, 2026 at 09:04:36AM -0700, Pierrick Bouvier wrote:
> On 7/9/2026 9:01 AM, Daniel P. Berrangé wrote:
> > On Thu, Jul 09, 2026 at 08:55:08AM -0700, Pierrick Bouvier wrote:
> >> On 7/9/2026 8:40 AM, Daniel P. Berrangé wrote:
> >>> On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
> >>>> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
> >>>>> When a developer's environment is already within a podman container it
> >>>>> is not possible to use 'podman' again to create containers. It will
> >>>>> usually result in wierd errors such as:
> >>>>>
> >>>>>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> >>>>>
> >>>>> Podman offers the ability to talk to a daemon outside the container,
> >>>>> however, which could be leveraged by QEMU.
> >>>>>
> >>>>> This can be used by invoking "podman --remote", or equivalently the
> >>>>> separate "podman-remote" binary:
> >>>>>
> >>>>>   https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md
> >>>>>
> >>>>> The current 'podman version' check is insufficient to detect the
> >>>>> inability to launch containers, so it is replaced with the stronger
> >>>>> 'podman info' check.
> >>>>>
> >>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >>>>> ---
> >>>>>  tests/docker/docker.py | 10 ++++++----
> >>>>>  1 file changed, 6 insertions(+), 4 deletions(-)
> >>>>>
> >>>>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> >>>>> index ff68c7bf6f..9e18b984f4 100755
> >>>>> --- a/tests/docker/docker.py
> >>>>> +++ b/tests/docker/docker.py
> >>>>> @@ -76,14 +76,16 @@ def _guess_engine_command():
> >>>>>      commands = []
> >>>>>  
> >>>>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
> >>>>> -        commands += [["podman"]]
> >>>>> +        commands += [["podman"], ["podman-remote"], ["podman", "--remote"]]
> >>>>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
> >>>>>          commands += [["docker"], ["sudo", "-n", "docker"]]
> >>>>>      for cmd in commands:
> >>>>>          try:
> >>>>> -            # docker version will return the client details in stdout
> >>>>> -            # but still report a status of 1 if it can't contact the daemon
> >>>>> -            if subprocess.call(cmd + ["version"],
> >>>>> +            # 'version' is not sufficient to prove a working binary
> >>>>> +            # for podman. 'info' is a stronger check that is more
> >>>>> +            # likely to correlate with ability to create containers,
> >>>>> +            # and required to detect the need for podman remote
> >>>>> +            if subprocess.call(cmd + ["info"],
> >>>>>                                 stdout=DEVNULL, stderr=DEVNULL) == 0:
> >>>>>                  return cmd
> >>>>>          except OSError:
> >>>>
> >>>> Taking a look at why tcg tests are slow to compile, I reached this
> >>>> commit. It seems like that calling 'podman info' is 5 to 10 times slower
> >>>> than calling 'podman version'.
> >>>> Thus, a container run now takes +1s when it was 0.2 before.
> >>>>
> >>>> On my setup, podman (remote) version fails correctly if it can't
> >>>> establish a connection.
> >>>>
> >>>> The commit description briefly says
> >>>> ```
> >>>> The current 'podman version' check is insufficient to detect the
> >>>> inability to launch containers, so it is replaced with the stronger
> >>>> 'podman info' check.
> >>>> ```
> >>>> but it's not what I observed.
> >>>>
> >>>> Do you have more information about what is the exact reason we can't use
> >>>> 'podman version' for podman-remote?
> >>>
> >>> My developer env involves toolbox, which is a wrapper around podman:
> >>>
> >>> "podman version" is not an operational check - it is effectively
> >>> nothing more than a variant of command line "help". Just shows
> >>> the binary can print to stdout.
> >>>
> >>> By contrast "docker version" is useful as IIUC it shows you can
> >>> connect to docker daemon, but podman is daemonless so the 'version'
> >>> chck doesn't offer any useful proof.
> >>>
> >>> "podman info" is an operational check that correlates with the
> >>> ability to actually create containers.
> >>>
> >>> ⚙️  [host:fedora-44 ~]$ toolbox enter almalinux-toolbox-9 
> >>> ⚙️  [podman:almalinux-9.8 ~]$ sudo dnf -y install podman
> >>> ⚙️  [podman:almalinux-9.8 ~] podman version
> >>> Client:       Podman Engine
> >>> Version:      5.8.2
> >>> API Version:  5.8.2
> >>> Go Version:   go1.26.3 (Red Hat 1.26.3-1.el9_8)
> >>> Built:        Tue Jun 16 23:52:20 2026
> >>> OS/Arch:      linux/amd64
> >>>
> >>> ⚙️  [podman:almalinux-9.8 ~]$ podman info
> >>> Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> >>>
> >>> ⚙️  [podman:almalinux-9.8 ~] $ podman --remote info
> >>> OS: linux/amd64
> >>> provider: qemu
> >>> version: 5.8.2
> >>> ..snip...
> >>>
> >>
> >> In this case, isn't the problem is that the environment should not have
> >> podman installed if it can't actually create containers?
> > 
> > podman *can* create containers by using the '--remote' flag, which
> > is what we made work in QEMU.
> > 
> >> Considering this, I think it would be better to have a "fast" default,
> >> that only checks version, and leave the rest to the user to make sure
> >> their env is correctly configured (or that they remove podman if that's
> >> not the case).
> > 
> > My env is correctly configured so that podman --remote works. Doing
> > onmly a version check will make QEMU not work once more.
> >
> 
> In this case, maybe we can test first podman --remote, before podman.
> If it can't connect to podman socket, version returns an error as expected.
> 
> This way, it *works* on your personal setup, and it's fast for everyone
> else.
> 
> Sounds like a good compromise?

Checking plain docker first was intentional, because it ensure you
avoid the indirection of the podman daemon if you don't actually
need it. This makes it all work optimally both inside and outside
the toolbox containers.


With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Pierrick Bouvier]

On 7/9/2026 9:13 AM, Daniel P. Berrangé wrote:
> On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
>> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
>>> When a developer's environment is already within a podman container it
>>> is not possible to use 'podman' again to create containers. It will
>>> usually result in wierd errors such as:
>>>
>>>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
>>>
>>> Podman offers the ability to talk to a daemon outside the container,
>>> however, which could be leveraged by QEMU.
>>>
>>> This can be used by invoking "podman --remote", or equivalently the
>>> separate "podman-remote" binary:
>>>
>>>   https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md
>>>
>>> The current 'podman version' check is insufficient to detect the
>>> inability to launch containers, so it is replaced with the stronger
>>> 'podman info' check.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>  tests/docker/docker.py | 10 ++++++----
>>>  1 file changed, 6 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>>> index ff68c7bf6f..9e18b984f4 100755
>>> --- a/tests/docker/docker.py
>>> +++ b/tests/docker/docker.py
>>> @@ -76,14 +76,16 @@ def _guess_engine_command():
>>>      commands = []
>>>  
>>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
>>> -        commands += [["podman"]]
>>> +        commands += [["podman"], ["podman-remote"], ["podman", "--remote"]]
>>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
>>>          commands += [["docker"], ["sudo", "-n", "docker"]]
>>>      for cmd in commands:
>>>          try:
>>> -            # docker version will return the client details in stdout
>>> -            # but still report a status of 1 if it can't contact the daemon
>>> -            if subprocess.call(cmd + ["version"],
>>> +            # 'version' is not sufficient to prove a working binary
>>> +            # for podman. 'info' is a stronger check that is more
>>> +            # likely to correlate with ability to create containers,
>>> +            # and required to detect the need for podman remote
>>> +            if subprocess.call(cmd + ["info"],
>>>                                 stdout=DEVNULL, stderr=DEVNULL) == 0:
>>>                  return cmd
>>>          except OSError:
>>
>> Taking a look at why tcg tests are slow to compile, I reached this
>> commit. It seems like that calling 'podman info' is 5 to 10 times slower
>> than calling 'podman version'.
>> Thus, a container run now takes +1s when it was 0.2 before.
> 
> Hmmm, but we call 'docker.py probe' in configure and cache the result,
> so that probe should only be done once.  0.2 vs 1s is just noise in
> the context of configure.
> 
> That you're seeing any negative effects suggests that we've got
> something not using the cached probe result, which we should fix.
>

Not really, docker.py --engine X still calls X info.

https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L675
https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L78

Maybe the fix is to remove the check if user specified an explicit
engine. But in this case, there is a problem with podman-remote not
being detected anymore.

We can fix this by introducing engine=podman-remote, and skip the check
if engine is explicitly set.
What do you think?

I'm open to any other approach as long as we can have something fast
once we initially probed what is available.

> With regards,
> Daniel

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Daniel P. Berrangé]

On Thu, Jul 09, 2026 at 09:24:12AM -0700, Pierrick Bouvier wrote:
> On 7/9/2026 9:13 AM, Daniel P. Berrangé wrote:
> > On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
> >> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
> >>> When a developer's environment is already within a podman container it
> >>> is not possible to use 'podman' again to create containers. It will
> >>> usually result in wierd errors such as:
> >>>
> >>>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> >>>
> >>> Podman offers the ability to talk to a daemon outside the container,
> >>> however, which could be leveraged by QEMU.
> >>>
> >>> This can be used by invoking "podman --remote", or equivalently the
> >>> separate "podman-remote" binary:
> >>>
> >>>   https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md
> >>>
> >>> The current 'podman version' check is insufficient to detect the
> >>> inability to launch containers, so it is replaced with the stronger
> >>> 'podman info' check.
> >>>
> >>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >>> ---
> >>>  tests/docker/docker.py | 10 ++++++----
> >>>  1 file changed, 6 insertions(+), 4 deletions(-)
> >>>
> >>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> >>> index ff68c7bf6f..9e18b984f4 100755
> >>> --- a/tests/docker/docker.py
> >>> +++ b/tests/docker/docker.py
> >>> @@ -76,14 +76,16 @@ def _guess_engine_command():
> >>>      commands = []
> >>>  
> >>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
> >>> -        commands += [["podman"]]
> >>> +        commands += [["podman"], ["podman-remote"], ["podman", "--remote"]]
> >>>      if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
> >>>          commands += [["docker"], ["sudo", "-n", "docker"]]
> >>>      for cmd in commands:
> >>>          try:
> >>> -            # docker version will return the client details in stdout
> >>> -            # but still report a status of 1 if it can't contact the daemon
> >>> -            if subprocess.call(cmd + ["version"],
> >>> +            # 'version' is not sufficient to prove a working binary
> >>> +            # for podman. 'info' is a stronger check that is more
> >>> +            # likely to correlate with ability to create containers,
> >>> +            # and required to detect the need for podman remote
> >>> +            if subprocess.call(cmd + ["info"],
> >>>                                 stdout=DEVNULL, stderr=DEVNULL) == 0:
> >>>                  return cmd
> >>>          except OSError:
> >>
> >> Taking a look at why tcg tests are slow to compile, I reached this
> >> commit. It seems like that calling 'podman info' is 5 to 10 times slower
> >> than calling 'podman version'.
> >> Thus, a container run now takes +1s when it was 0.2 before.
> > 
> > Hmmm, but we call 'docker.py probe' in configure and cache the result,
> > so that probe should only be done once.  0.2 vs 1s is just noise in
> > the context of configure.
> > 
> > That you're seeing any negative effects suggests that we've got
> > something not using the cached probe result, which we should fix.
> >
> 
> Not really, docker.py --engine X still calls X info.
> 
> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L675
> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L78
> 
> Maybe the fix is to remove the check if user specified an explicit
> engine. But in this case, there is a problem with podman-remote not
> being detected anymore.
> 
> We can fix this by introducing engine=podman-remote, and skip the check
> if engine is explicitly set.
> What do you think?

Urgh, so i see now we have two code paths / make variables

  RUNC=podman --remote
  CONTAINER_ENGINE=auto

Sometimes we'll invoke docker.py --engine $(CONTAINER_ENGINE)
and sometimes we'll directly invoke $(RUNC).

The result of "probe" cannot be fed back in to "--engine" which
is a bit of a mess. I think the "engine" concept should not be
exposed on the cli, and instead have a "--runc" arg which takes
the full arg(s), since we need probe to report the runc args
for other reasons.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH v2 3/4] tests/docker: allow display of docker output

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> The --quiet command is used with docker unless V=1 is passed to make,
> and as a result stdout from docker is never visible by default, making
> it hard to diagnose failures building / running containers.
>
> Meanwhile passing V=1 is undesirable as that makes the entire build
> system verbose.
>
> Introduce a $(DOCKER_V) make variable which is initialized from $(V)
>
> It is thus possible to display docker output without also enabling
> make verbose output.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

(added Edgar for Microblaze)

> On Thu, Jul 09, 2026 at 09:24:12AM -0700, Pierrick Bouvier wrote:
>> On 7/9/2026 9:13 AM, Daniel P. Berrangé wrote:
>> > On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
>> >> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
>> >>> When a developer's environment is already within a podman container it
>> >>> is not possible to use 'podman' again to create containers. It will
>> >>> usually result in wierd errors such as:
>> >>>
>> >>>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
>> >>>
>> >>> Podman offers the ability to talk to a daemon outside the container,
>> >>> however, which could be leveraged by QEMU.
>> >>>
<snip>
>> >> Taking a look at why tcg tests are slow to compile, I reached this
>> >> commit. It seems like that calling 'podman info' is 5 to 10 times slower
>> >> than calling 'podman version'.
>> >> Thus, a container run now takes +1s when it was 0.2 before.
>> > 
>> > Hmmm, but we call 'docker.py probe' in configure and cache the result,
>> > so that probe should only be done once.  0.2 vs 1s is just noise in
>> > the context of configure.
>> > 
>> > That you're seeing any negative effects suggests that we've got
>> > something not using the cached probe result, which we should fix.
>> >
>> 
>> Not really, docker.py --engine X still calls X info.
>> 
>> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L675
>> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L78
>> 
>> Maybe the fix is to remove the check if user specified an explicit
>> engine. But in this case, there is a problem with podman-remote not
>> being detected anymore.
>> 
>> We can fix this by introducing engine=podman-remote, and skip the check
>> if engine is explicitly set.
>> What do you think?
>
> Urgh, so i see now we have two code paths / make variables
>
>   RUNC=podman --remote
>   CONTAINER_ENGINE=auto
>
> Sometimes we'll invoke docker.py --engine $(CONTAINER_ENGINE)
> and sometimes we'll directly invoke $(RUNC).

Yeah that could do with cleaning up. I suspect the only places we
actually still need the script is where we are doing binfmt_misc enabled
containers unless there is a way to emulate the copying of qemu into the
container with a plain call? Maybe all the systems now have the
persistent flags which would skip this.

The toolchain case could be skipped if we could find hosted compilers
for microblaze. However I do see we have a patch in there.

>
> The result of "probe" cannot be fed back in to "--engine" which
> is a bit of a mess. I think the "engine" concept should not be
> exposed on the cli, and instead have a "--runc" arg which takes
> the full arg(s), since we need probe to report the runc args
> for other reasons.
>
> With regards,
> Daniel

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Daniel P. Berrangé]

On Fri, Jul 10, 2026 at 12:47:28PM +0100, Alex Bennée wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> (added Edgar for Microblaze)
> 
> > On Thu, Jul 09, 2026 at 09:24:12AM -0700, Pierrick Bouvier wrote:
> >> On 7/9/2026 9:13 AM, Daniel P. Berrangé wrote:
> >> > On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
> >> >> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
> >> >>> When a developer's environment is already within a podman container it
> >> >>> is not possible to use 'podman' again to create containers. It will
> >> >>> usually result in wierd errors such as:
> >> >>>
> >> >>>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> >> >>>
> >> >>> Podman offers the ability to talk to a daemon outside the container,
> >> >>> however, which could be leveraged by QEMU.
> >> >>>
> <snip>
> >> >> Taking a look at why tcg tests are slow to compile, I reached this
> >> >> commit. It seems like that calling 'podman info' is 5 to 10 times slower
> >> >> than calling 'podman version'.
> >> >> Thus, a container run now takes +1s when it was 0.2 before.
> >> > 
> >> > Hmmm, but we call 'docker.py probe' in configure and cache the result,
> >> > so that probe should only be done once.  0.2 vs 1s is just noise in
> >> > the context of configure.
> >> > 
> >> > That you're seeing any negative effects suggests that we've got
> >> > something not using the cached probe result, which we should fix.
> >> >
> >> 
> >> Not really, docker.py --engine X still calls X info.
> >> 
> >> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L675
> >> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L78
> >> 
> >> Maybe the fix is to remove the check if user specified an explicit
> >> engine. But in this case, there is a problem with podman-remote not
> >> being detected anymore.
> >> 
> >> We can fix this by introducing engine=podman-remote, and skip the check
> >> if engine is explicitly set.
> >> What do you think?
> >
> > Urgh, so i see now we have two code paths / make variables
> >
> >   RUNC=podman --remote
> >   CONTAINER_ENGINE=auto
> >
> > Sometimes we'll invoke docker.py --engine $(CONTAINER_ENGINE)
> > and sometimes we'll directly invoke $(RUNC).
> 
> Yeah that could do with cleaning up. I suspect the only places we
> actually still need the script is where we are doing binfmt_misc enabled
> containers unless there is a way to emulate the copying of qemu into the
> container with a plain call? Maybe all the systems now have the
> persistent flags which would skip this.

Yeah, I'd be inclined to say this is a job for the distros to have
qemu-user integration, and simplify our life. THe docker.py script
always confuses me.  We could choose podman vs podman --remote vs
docker simply in the meson.build file, as we do for other dev env
choices.

> The toolchain case could be skipped if we could find hosted compilers
> for microblaze. However I do see we have a patch in there.
> 
> >
> > The result of "probe" cannot be fed back in to "--engine" which
> > is a bit of a mess. I think the "engine" concept should not be
> > exposed on the cli, and instead have a "--runc" arg which takes
> > the full arg(s), since we need probe to report the runc args
> > for other reasons.
> >
> > With regards,
> > Daniel
> 
> -- 
> Alex Bennée
> Virtualisation Tech Lead @ Linaro
> 

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Pierrick Bouvier]

On 7/10/2026 4:52 AM, Daniel P. Berrangé wrote:
> On Fri, Jul 10, 2026 at 12:47:28PM +0100, Alex Bennée wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>
>> (added Edgar for Microblaze)
>>
>>> On Thu, Jul 09, 2026 at 09:24:12AM -0700, Pierrick Bouvier wrote:
>>>> On 7/9/2026 9:13 AM, Daniel P. Berrangé wrote:
>>>>> On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
>>>>>> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
>>>>>>> When a developer's environment is already within a podman container it
>>>>>>> is not possible to use 'podman' again to create containers. It will
>>>>>>> usually result in wierd errors such as:
>>>>>>>
>>>>>>>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
>>>>>>>
>>>>>>> Podman offers the ability to talk to a daemon outside the container,
>>>>>>> however, which could be leveraged by QEMU.
>>>>>>>
>> <snip>
>>>>>> Taking a look at why tcg tests are slow to compile, I reached this
>>>>>> commit. It seems like that calling 'podman info' is 5 to 10 times slower
>>>>>> than calling 'podman version'.
>>>>>> Thus, a container run now takes +1s when it was 0.2 before.
>>>>>
>>>>> Hmmm, but we call 'docker.py probe' in configure and cache the result,
>>>>> so that probe should only be done once.  0.2 vs 1s is just noise in
>>>>> the context of configure.
>>>>>
>>>>> That you're seeing any negative effects suggests that we've got
>>>>> something not using the cached probe result, which we should fix.
>>>>>
>>>>
>>>> Not really, docker.py --engine X still calls X info.
>>>>
>>>> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L675
>>>> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L78
>>>>
>>>> Maybe the fix is to remove the check if user specified an explicit
>>>> engine. But in this case, there is a problem with podman-remote not
>>>> being detected anymore.
>>>>
>>>> We can fix this by introducing engine=podman-remote, and skip the check
>>>> if engine is explicitly set.
>>>> What do you think?
>>>
>>> Urgh, so i see now we have two code paths / make variables
>>>
>>>   RUNC=podman --remote
>>>   CONTAINER_ENGINE=auto
>>>
>>> Sometimes we'll invoke docker.py --engine $(CONTAINER_ENGINE)
>>> and sometimes we'll directly invoke $(RUNC).
>>
>> Yeah that could do with cleaning up. I suspect the only places we
>> actually still need the script is where we are doing binfmt_misc enabled
>> containers unless there is a way to emulate the copying of qemu into the
>> container with a plain call? Maybe all the systems now have the
>> persistent flags which would skip this.
> 
> Yeah, I'd be inclined to say this is a job for the distros to have
> qemu-user integration, and simplify our life. THe docker.py script
> always confuses me.  We could choose podman vs podman --remote vs
> docker simply in the meson.build file, as we do for other dev env
> choices.
> 
>> The toolchain case could be skipped if we could find hosted compilers
>> for microblaze. However I do see we have a patch in there.
>>
>>>
>>> The result of "probe" cannot be fed back in to "--engine" which
>>> is a bit of a mess. I think the "engine" concept should not be
>>> exposed on the cli, and instead have a "--runc" arg which takes
>>> the full arg(s), since we need probe to report the runc args
>>> for other reasons.
>>>
>>> With regards,
>>> Daniel
>>
>> -- 
>> Alex Bennée
>> Virtualisation Tech Lead @ Linaro
>>
> 
> With regards,
> Daniel

Is it me or did we incredibly derailed from original issue which was:
"It was fast to query which container engine is available, and now it's
5-10 times slower to accommodate toolbox workflow where podman is
available but not really available?"

I agree it's sad to have several ways to set container engine, but it's
absolutely not related to the current issue.

Daniel, since you introduced the slow check path with podman info for
your personal need, I would appreciate if it's something you could fix also.

Regards,
Pierrick

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Daniel P. Berrangé]

On Fri, Jul 10, 2026 at 11:20:37AM -0700, Pierrick Bouvier wrote:
> On 7/10/2026 4:52 AM, Daniel P. Berrangé wrote:
> > On Fri, Jul 10, 2026 at 12:47:28PM +0100, Alex Bennée wrote:
> >> Daniel P. Berrangé <berrange@redhat.com> writes:
> >>
> >> (added Edgar for Microblaze)
> >>
> >>> On Thu, Jul 09, 2026 at 09:24:12AM -0700, Pierrick Bouvier wrote:
> >>>> On 7/9/2026 9:13 AM, Daniel P. Berrangé wrote:
> >>>>> On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
> >>>>>> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
> >>>>>>> When a developer's environment is already within a podman container it
> >>>>>>> is not possible to use 'podman' again to create containers. It will
> >>>>>>> usually result in wierd errors such as:
> >>>>>>>
> >>>>>>>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
> >>>>>>>
> >>>>>>> Podman offers the ability to talk to a daemon outside the container,
> >>>>>>> however, which could be leveraged by QEMU.
> >>>>>>>
> >> <snip>
> >>>>>> Taking a look at why tcg tests are slow to compile, I reached this
> >>>>>> commit. It seems like that calling 'podman info' is 5 to 10 times slower
> >>>>>> than calling 'podman version'.
> >>>>>> Thus, a container run now takes +1s when it was 0.2 before.
> >>>>>
> >>>>> Hmmm, but we call 'docker.py probe' in configure and cache the result,
> >>>>> so that probe should only be done once.  0.2 vs 1s is just noise in
> >>>>> the context of configure.
> >>>>>
> >>>>> That you're seeing any negative effects suggests that we've got
> >>>>> something not using the cached probe result, which we should fix.
> >>>>>
> >>>>
> >>>> Not really, docker.py --engine X still calls X info.
> >>>>
> >>>> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L675
> >>>> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L78
> >>>>
> >>>> Maybe the fix is to remove the check if user specified an explicit
> >>>> engine. But in this case, there is a problem with podman-remote not
> >>>> being detected anymore.
> >>>>
> >>>> We can fix this by introducing engine=podman-remote, and skip the check
> >>>> if engine is explicitly set.
> >>>> What do you think?
> >>>
> >>> Urgh, so i see now we have two code paths / make variables
> >>>
> >>>   RUNC=podman --remote
> >>>   CONTAINER_ENGINE=auto
> >>>
> >>> Sometimes we'll invoke docker.py --engine $(CONTAINER_ENGINE)
> >>> and sometimes we'll directly invoke $(RUNC).
> >>
> >> Yeah that could do with cleaning up. I suspect the only places we
> >> actually still need the script is where we are doing binfmt_misc enabled
> >> containers unless there is a way to emulate the copying of qemu into the
> >> container with a plain call? Maybe all the systems now have the
> >> persistent flags which would skip this.
> > 
> > Yeah, I'd be inclined to say this is a job for the distros to have
> > qemu-user integration, and simplify our life. THe docker.py script
> > always confuses me.  We could choose podman vs podman --remote vs
> > docker simply in the meson.build file, as we do for other dev env
> > choices.
> > 
> >> The toolchain case could be skipped if we could find hosted compilers
> >> for microblaze. However I do see we have a patch in there.
> >>
> >>>
> >>> The result of "probe" cannot be fed back in to "--engine" which
> >>> is a bit of a mess. I think the "engine" concept should not be
> >>> exposed on the cli, and instead have a "--runc" arg which takes
> >>> the full arg(s), since we need probe to report the runc args
> >>> for other reasons.
> >>>
> >>> With regards,
> >>> Daniel
> >>
> >> -- 
> >> Alex Bennée
> >> Virtualisation Tech Lead @ Linaro
> >>
> > 
> > With regards,
> > Daniel
> 
> Is it me or did we incredibly derailed from original issue which was:
> "It was fast to query which container engine is available, and now it's
> 5-10 times slower to accommodate toolbox workflow where podman is
> available but not really available?"
> I agree it's sad to have several ways to set container engine, but it's
> absolutely not related to the current issue.

It is relevant because that difference in speed is not something that
should be noticable unless something is repeatedly probing over & over
again which should not be done.

> Daniel, since you introduced the slow check path with podman info for
> your personal need, I would appreciate if it's something you could fix also.

I'll looking into the missing caching.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH v2 2/4] tests/docker: add support for podman remote access

[Pierrick Bouvier]

On 7/13/2026 1:32 AM, Daniel P. Berrangé wrote:
> On Fri, Jul 10, 2026 at 11:20:37AM -0700, Pierrick Bouvier wrote:
>> On 7/10/2026 4:52 AM, Daniel P. Berrangé wrote:
>>> On Fri, Jul 10, 2026 at 12:47:28PM +0100, Alex Bennée wrote:
>>>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>>>
>>>> (added Edgar for Microblaze)
>>>>
>>>>> On Thu, Jul 09, 2026 at 09:24:12AM -0700, Pierrick Bouvier wrote:
>>>>>> On 7/9/2026 9:13 AM, Daniel P. Berrangé wrote:
>>>>>>> On Thu, Jul 09, 2026 at 07:48:14AM -0700, Pierrick Bouvier wrote:
>>>>>>>> On 2/10/2026 8:35 AM, Daniel P. Berrangé wrote:
>>>>>>>>> When a developer's environment is already within a podman container it
>>>>>>>>> is not possible to use 'podman' again to create containers. It will
>>>>>>>>> usually result in wierd errors such as:
>>>>>>>>>
>>>>>>>>>   Error: fatal error, invalid internal status, unable to create a new pause process: cannot re-exec process to join the existing user namespace. Try running "podman system migrate" and if that doesn't work reboot to recover
>>>>>>>>>
>>>>>>>>> Podman offers the ability to talk to a daemon outside the container,
>>>>>>>>> however, which could be leveraged by QEMU.
>>>>>>>>>
>>>> <snip>
>>>>>>>> Taking a look at why tcg tests are slow to compile, I reached this
>>>>>>>> commit. It seems like that calling 'podman info' is 5 to 10 times slower
>>>>>>>> than calling 'podman version'.
>>>>>>>> Thus, a container run now takes +1s when it was 0.2 before.
>>>>>>>
>>>>>>> Hmmm, but we call 'docker.py probe' in configure and cache the result,
>>>>>>> so that probe should only be done once.  0.2 vs 1s is just noise in
>>>>>>> the context of configure.
>>>>>>>
>>>>>>> That you're seeing any negative effects suggests that we've got
>>>>>>> something not using the cached probe result, which we should fix.
>>>>>>>
>>>>>>
>>>>>> Not really, docker.py --engine X still calls X info.
>>>>>>
>>>>>> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L675
>>>>>> https://gitlab.com/qemu-project/qemu/-/blob/master/tests/docker/docker.py#L78
>>>>>>
>>>>>> Maybe the fix is to remove the check if user specified an explicit
>>>>>> engine. But in this case, there is a problem with podman-remote not
>>>>>> being detected anymore.
>>>>>>
>>>>>> We can fix this by introducing engine=podman-remote, and skip the check
>>>>>> if engine is explicitly set.
>>>>>> What do you think?
>>>>>
>>>>> Urgh, so i see now we have two code paths / make variables
>>>>>
>>>>>   RUNC=podman --remote
>>>>>   CONTAINER_ENGINE=auto
>>>>>
>>>>> Sometimes we'll invoke docker.py --engine $(CONTAINER_ENGINE)
>>>>> and sometimes we'll directly invoke $(RUNC).
>>>>
>>>> Yeah that could do with cleaning up. I suspect the only places we
>>>> actually still need the script is where we are doing binfmt_misc enabled
>>>> containers unless there is a way to emulate the copying of qemu into the
>>>> container with a plain call? Maybe all the systems now have the
>>>> persistent flags which would skip this.
>>>
>>> Yeah, I'd be inclined to say this is a job for the distros to have
>>> qemu-user integration, and simplify our life. THe docker.py script
>>> always confuses me.  We could choose podman vs podman --remote vs
>>> docker simply in the meson.build file, as we do for other dev env
>>> choices.
>>>
>>>> The toolchain case could be skipped if we could find hosted compilers
>>>> for microblaze. However I do see we have a patch in there.
>>>>
>>>>>
>>>>> The result of "probe" cannot be fed back in to "--engine" which
>>>>> is a bit of a mess. I think the "engine" concept should not be
>>>>> exposed on the cli, and instead have a "--runc" arg which takes
>>>>> the full arg(s), since we need probe to report the runc args
>>>>> for other reasons.
>>>>>
>>>>> With regards,
>>>>> Daniel
>>>>
>>>> -- 
>>>> Alex Bennée
>>>> Virtualisation Tech Lead @ Linaro
>>>>
>>>
>>> With regards,
>>> Daniel
>>
>> Is it me or did we incredibly derailed from original issue which was:
>> "It was fast to query which container engine is available, and now it's
>> 5-10 times slower to accommodate toolbox workflow where podman is
>> available but not really available?"
>> I agree it's sad to have several ways to set container engine, but it's
>> absolutely not related to the current issue.
> 
> It is relevant because that difference in speed is not something that
> should be noticable unless something is repeatedly probing over & over
> again which should not be done.
> 
>> Daniel, since you introduced the slow check path with podman info for
>> your personal need, I would appreciate if it's something you could fix also.
> 
> I'll looking into the missing caching.
>

Thanks!

> With regards,
> Daniel