[PATCH v5 0/1] python/machine: Fix AF_UNIX path too long

[PATCH v5 0/1] python/machine: Fix AF_UNIX path too long

[Peter Delevoryas]

v1: https://lore.kernel.org/qemu-devel/20220705214659.73369-1-peter@pjd.dev/
v2: https://lore.kernel.org/qemu-devel/20220716173434.17183-1-peter@pjd.dev/
v3:
    - Changed QEMUMachine._name to f"{id(self):x}". Suggestion was to do
      f"{id(self):02x}", but the id's look like they are probably just the
      object address (8-byte pointer), so the "02" had no effect.
    - Changed QMP socket name suffix from "-monitor.sock" to ".qmp".
    - Changed console socket name suffix from "-console.sock" to ".con".
v4:
	- Just resending v3 after rebasing from a long time ago
v5:
    - I forgot to disable the VPN on my laptop before sending the v4 series,
      and I always have very mysterious errors and/or delays when I send email
      through the VPN because our security team doesn't allow split tunneling.
    - I also forgot to include the reviewed-by tags. Daniel reviewed v3,
      Phil reviewed v4.

I tried to run `make check-avocado` before sending again, but it looks like
there is some other issue. Probably related to the python version I have (I
have like 5 different Python versions installed on my work laptop).

Thanks,
Peter

$ make check-avocado
changing dir to build for /Library/Developer/CommandLineTools/usr/bin/make "check-avocado"...
  GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
  VENV    /Users/pdel/qemu/build/tests/venv
  VENVPIP install -e /Users/pdel/qemu/python/
  VENVPIP install -r /Users/pdel/qemu/tests/requirements.txt
  MKDIR   /Users/pdel/qemu/build/tests/results
  AVOCADO Downloading avocado tests VM image for aarch64
The image was downloaded:
Provider Version Architecture File
fedora   31      aarch64      /Users/pdel/avocado/data/cache/by_location/4f156e531446a679cbfe13caef8b7c9f9f79aafa/Fedora-C
loud-Base-31-1.9.aarch64.qcow2
  AVOCADO tests/avocado
Fetching asset from tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf
Fetching asset from tests/avocado/boot_xen.py:BootXen.test_arm64_xen_411_and_dom0
Fetching asset from tests/avocado/boot_xen.py:BootXen.test_arm64_xen_414_and_dom0
Fetching asset from tests/avocado/boot_xen.py:BootXen.test_arm64_xen_415_and_dom0
Fetching asset from tests/avocado/machine_aarch64_virt.py:Aarch64VirtMachine.test_alpine_virt_tcg_gic_max
Fetching asset from tests/avocado/machine_aarch64_virt.py:Aarch64VirtMachine.test_aarch64_virt
Fetching asset from tests/avocado/replay_kernel.py:ReplayKernelNormal.test_aarch64_virt
Fetching asset from tests/avocado/reverse_debugging.py:ReverseDebugging_AArch64.test_aarch64_virt
JOB ID     : 18a949ed9150e22d6ecea69b99ede1ded17233f4
JOB LOG    : /Users/pdel/qemu/build/tests/results/job-2023-01-10T00.03-18a949e/job.log

Avocado crashed: TypeError: cannot pickle '_thread.RLock' object

Peter Delevoryas (1):
  python/machine: Fix AF_UNIX path too long on macOS

 python/qemu/machine/machine.py         | 6 +++---
 tests/avocado/avocado_qemu/__init__.py | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

-- 
2.39.0

[PATCH v5 1/1] python/machine: Fix AF_UNIX path too long on macOS

[Peter Delevoryas]

On macOS, private $TMPDIR's are the default. These $TMPDIR's are
generated from a user's unix UID and UUID [1], which can create a
relatively long path:

    /var/folders/d7/rz20f6hd709c1ty8f6_6y_z40000gn/T/

QEMU's avocado tests create a temporary directory prefixed by
"avo_qemu_sock_", and create QMP sockets within _that_ as well.
The QMP socket is unnecessarily long, because a temporary directory
is created for every QEMUMachine object.

    /avo_qemu_sock_uh3w_dgc/qemu-37331-10bacf110-monitor.sock

The path limit for unix sockets on macOS is 104: [2]

    /*
     * [XSI] Definitions for UNIX IPC domain.
     */
    struct  sockaddr_un {
        unsigned char   sun_len;        /* sockaddr len including null */
        sa_family_t     sun_family;     /* [XSI] AF_UNIX */
        char            sun_path[104];  /* [XSI] path name (gag) */
    };

This results in avocado tests failing on macOS because the QMP unix
socket can't be created, because the path is too long:

    ERROR| Failed to establish connection: OSError: AF_UNIX path too long

This change resolves by reducing the size of the socket directory prefix
and the suffix on the QMP and console socket names.

The result is paths like this:

    pdel@pdel-mbp:/var/folders/d7/rz20f6hd709c1ty8f6_6y_z40000gn/T
    $ tree qemu*
    qemu_df4evjeq
    qemu_jbxel3gy
    qemu_ml9s_gg7
    qemu_oc7h7f3u
    qemu_oqb1yf97
    ├── 10a004050.con
    └── 10a004050.qmp

[1] https://apple.stackexchange.com/questions/353832/why-is-mac-osx-temp-directory-in-weird-path
[2] /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/sys/un.h

Signed-off-by: Peter Delevoryas <peter@pjd.dev>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 python/qemu/machine/machine.py         | 6 +++---
 tests/avocado/avocado_qemu/__init__.py | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/python/qemu/machine/machine.py b/python/qemu/machine/machine.py
index 748a0d807c9d..d70977378305 100644
--- a/python/qemu/machine/machine.py
+++ b/python/qemu/machine/machine.py
@@ -157,7 +157,7 @@ def __init__(self,
         self._wrapper = wrapper
         self._qmp_timer = qmp_timer
 
-        self._name = name or f"qemu-{os.getpid()}-{id(self):02x}"
+        self._name = name or f"{id(self):x}"
         self._temp_dir: Optional[str] = None
         self._base_temp_dir = base_temp_dir
         self._sock_dir = sock_dir
@@ -167,7 +167,7 @@ def __init__(self,
             self._monitor_address = monitor_address
         else:
             self._monitor_address = os.path.join(
-                self.sock_dir, f"{self._name}-monitor.sock"
+                self.sock_dir, f"{self._name}.qmp"
             )
 
         self._console_log_path = console_log
@@ -192,7 +192,7 @@ def __init__(self,
         self._console_set = False
         self._console_device_type: Optional[str] = None
         self._console_address = os.path.join(
-            self.sock_dir, f"{self._name}-console.sock"
+            self.sock_dir, f"{self._name}.con"
         )
         self._console_socket: Optional[socket.socket] = None
         self._remove_files: List[str] = []
diff --git a/tests/avocado/avocado_qemu/__init__.py b/tests/avocado/avocado_qemu/__init__.py
index 910f3ba1eab8..25a546842fab 100644
--- a/tests/avocado/avocado_qemu/__init__.py
+++ b/tests/avocado/avocado_qemu/__init__.py
@@ -306,7 +306,7 @@ def require_netdev(self, netdevname):
             self.cancel('no support for user networking')
 
     def _new_vm(self, name, *args):
-        self._sd = tempfile.TemporaryDirectory(prefix="avo_qemu_sock_")
+        self._sd = tempfile.TemporaryDirectory(prefix="qemu_")
         vm = QEMUMachine(self.qemu_bin, base_temp_dir=self.workdir,
                          sock_dir=self._sd.name, log_dir=self.logdir)
         self.log.debug('QEMUMachine "%s" created', name)
-- 
2.39.0

Re: [PATCH v5 0/1] python/machine: Fix AF_UNIX path too long

[Philippe Mathieu-Daudé]

On 10/1/23 09:29, Peter Delevoryas wrote:

> $ make check-avocado

> Avocado crashed: TypeError: cannot pickle '_thread.RLock' object

Yeah... you have to pick Cleber's patches from
https://gitlab.com/cleber.gnu/qemu/-/commits/WIP/tests_fixes
and run as:

$ TMPDIR=/tmp avocado run --test-runner=nrunner ...

This is tracked on the avocado project:
https://github.com/avocado-framework/avocado/issues/5138#issuecomment-1314551148

Re: [PATCH v5 0/1] python/machine: Fix AF_UNIX path too long

[Peter Delevoryas]

On Tue, Jan 10, 2023 at 09:38:38AM +0100, Philippe Mathieu-Daudé wrote:
> On 10/1/23 09:29, Peter Delevoryas wrote:
> 
> > $ make check-avocado
> 
> > Avocado crashed: TypeError: cannot pickle '_thread.RLock' object
> 
> Yeah... you have to pick Cleber's patches from
> https://gitlab.com/cleber.gnu/qemu/-/commits/WIP/tests_fixes
> and run as:
> 
> $ TMPDIR=/tmp avocado run --test-runner=nrunner ...

I guess I still get this error from `make check-avocado` on Aarch64 tests:

 (17/35) tests/avocado/replay_linux.py:ReplayLinuxAarch64.test_virt_gicv2: ERROR: Failed to fetch Fedora-Cloud-Base-31-1.9
.aarch64.qcow2 (Can't pickle local object 'url_download.<locals>.download'). (0.89 s)

But `./build/tests/venv/bin/avocado run tests/avocado` worked ok.

> 
> This is tracked on the avocado project:
> https://github.com/avocado-framework/avocado/issues/5138#issuecomment-1314551148

Thanks!
- Peter

>