[PATCH v4] scripts/symlink-install-tree.py: Fix MESONINTROSPECT parsing

[PATCH v4] scripts/symlink-install-tree.py: Fix MESONINTROSPECT parsing

[Peter Maydell]

From: Akihiko Odaki <akihiko.odaki@daynix.com>

The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
must be parsed with shlex.split().  Otherwise the script will fail if
the build directory has a character like "~" in it.

Note: this fix cannot be backported directly to any stable branch
that doesn't require Meson version 1.4.0 or better; otherwise it will
work OK on Linux but will break on Windows hosts.

(Unfortunately, Meson prior to version 1.4.0 was inconsistent between
host OSes about how it quoted arguments, and used a different quoting
process on Windows hosts.  Our current git trunk already requires
1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
the stable branches are still on older Meson.)

Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
Reported-by: Michael Tokarev <mjt@tls.msk.ru>
Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
[PMM: Updated commit message to give all the detail about the
Meson version compability requirements.]
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
This is essentially back to version 1 of Akihiko's patch, now we
have a new enough Meson; I just updated the commit message.
 https://patchew.org/QEMU/20230812061540.5398-1-akihiko.odaki@daynix.com/
(I have dropped the various reviewed-by and tested-by headers because
I figured the passage of time was enough to make them moot.)

 scripts/symlink-install-tree.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/symlink-install-tree.py b/scripts/symlink-install-tree.py
index 8ed97e3c943..b72563895c5 100644
--- a/scripts/symlink-install-tree.py
+++ b/scripts/symlink-install-tree.py
@@ -4,6 +4,7 @@
 import errno
 import json
 import os
+import shlex
 import subprocess
 import sys
 
@@ -14,7 +15,7 @@ def destdir_join(d1: str, d2: str) -> str:
     return str(PurePath(d1, *PurePath(d2).parts[1:]))
 
 introspect = os.environ.get('MESONINTROSPECT')
-out = subprocess.run([*introspect.split(' '), '--installed'],
+out = subprocess.run([*shlex.split(introspect), '--installed'],
                      stdout=subprocess.PIPE, check=True).stdout
 for source, dest in json.loads(out).items():
     bundle_dest = destdir_join('qemu-bundle', dest)
-- 
2.34.1

Re: [PATCH v4] scripts/symlink-install-tree.py: Fix MESONINTROSPECT parsing

[Pierrick Bouvier]

On 10/18/24 06:08, Peter Maydell wrote:
> From: Akihiko Odaki <akihiko.odaki@daynix.com>
> 
> The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
> must be parsed with shlex.split().  Otherwise the script will fail if
> the build directory has a character like "~" in it.
> 
> Note: this fix cannot be backported directly to any stable branch
> that doesn't require Meson version 1.4.0 or better; otherwise it will
> work OK on Linux but will break on Windows hosts.
> 
> (Unfortunately, Meson prior to version 1.4.0 was inconsistent between
> host OSes about how it quoted arguments, and used a different quoting
> process on Windows hosts.  Our current git trunk already requires
> 1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
> the stable branches are still on older Meson.)
> 
> Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
> Reported-by: Michael Tokarev <mjt@tls.msk.ru>
> Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
> [PMM: Updated commit message to give all the detail about the
> Meson version compability requirements.]
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---
> This is essentially back to version 1 of Akihiko's patch, now we
> have a new enough Meson; I just updated the commit message.
>   https://patchew.org/QEMU/20230812061540.5398-1-akihiko.odaki@daynix.com/
> (I have dropped the various reviewed-by and tested-by headers because
> I figured the passage of time was enough to make them moot.)
> 
>   scripts/symlink-install-tree.py | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/symlink-install-tree.py b/scripts/symlink-install-tree.py
> index 8ed97e3c943..b72563895c5 100644
> --- a/scripts/symlink-install-tree.py
> +++ b/scripts/symlink-install-tree.py
> @@ -4,6 +4,7 @@
>   import errno
>   import json
>   import os
> +import shlex
>   import subprocess
>   import sys
>   
> @@ -14,7 +15,7 @@ def destdir_join(d1: str, d2: str) -> str:
>       return str(PurePath(d1, *PurePath(d2).parts[1:]))
>   
>   introspect = os.environ.get('MESONINTROSPECT')
> -out = subprocess.run([*introspect.split(' '), '--installed'],
> +out = subprocess.run([*shlex.split(introspect), '--installed'],
>                        stdout=subprocess.PIPE, check=True).stdout
>   for source, dest in json.loads(out).items():
>       bundle_dest = destdir_join('qemu-bundle', dest)

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Tested-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

I confirm this fixes the error when the path to qemu src tree contains '~'.

Re: [PATCH v4] scripts/symlink-install-tree.py: Fix MESONINTROSPECT parsing

[Christophe Lyon]

[-- Attachment #1: Type: text/plain, Size: 2983 bytes --]

Hi,



Le lun. 21 oct. 2024, 19:27, Pierrick Bouvier <pierrick.bouvier@linaro.org>
a écrit :

> On 10/18/24 06:08, Peter Maydell wrote:
> > From: Akihiko Odaki <akihiko.odaki@daynix.com>
> >
> > The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
> > must be parsed with shlex.split().  Otherwise the script will fail if
> > the build directory has a character like "~" in it.
> >
> > Note: this fix cannot be backported directly to any stable branch
> > that doesn't require Meson version 1.4.0 or better; otherwise it will
> > work OK on Linux but will break on Windows hosts.
> >
> > (Unfortunately, Meson prior to version 1.4.0 was inconsistent between
> > host OSes about how it quoted arguments, and used a different quoting
> > process on Windows hosts.  Our current git trunk already requires
> > 1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
> > the stable branches are still on older Meson.)
> >
> > Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
> > Reported-by: Michael Tokarev <mjt@tls.msk.ru>
> > Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
> > [PMM: Updated commit message to give all the detail about the
> > Meson version compability requirements.]
> > Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> > ---
> > This is essentially back to version 1 of Akihiko's patch, now we
> > have a new enough Meson; I just updated the commit message.
> >
> https://patchew.org/QEMU/20230812061540.5398-1-akihiko.odaki@daynix.com/
> > (I have dropped the various reviewed-by and tested-by headers because
> > I figured the passage of time was enough to make them moot.)
> >
> >   scripts/symlink-install-tree.py | 3 ++-
> >   1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/scripts/symlink-install-tree.py
> b/scripts/symlink-install-tree.py
> > index 8ed97e3c943..b72563895c5 100644
> > --- a/scripts/symlink-install-tree.py
> > +++ b/scripts/symlink-install-tree.py
> > @@ -4,6 +4,7 @@
> >   import errno
> >   import json
> >   import os
> > +import shlex
> >   import subprocess
> >   import sys
> >
> > @@ -14,7 +15,7 @@ def destdir_join(d1: str, d2: str) -> str:
> >       return str(PurePath(d1, *PurePath(d2).parts[1:]))
> >
> >   introspect = os.environ.get('MESONINTROSPECT')
> > -out = subprocess.run([*introspect.split(' '), '--installed'],
> > +out = subprocess.run([*shlex.split(introspect), '--installed'],
> >                        stdout=subprocess.PIPE, check=True).stdout
> >   for source, dest in json.loads(out).items():
> >       bundle_dest = destdir_join('qemu-bundle', dest)
>
> Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> Tested-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>
> I confirm this fixes the error when the path to qemu src tree contains '~'.
>

I confirm this fixes the problem we detected in our CI (where some build
dirnames have '~'

Thanks

>

[-- Attachment #2: Type: text/html, Size: 4603 bytes --]

Re: [PATCH v4] scripts/symlink-install-tree.py: Fix MESONINTROSPECT parsing

[Pierrick Bouvier]

On 10/18/24 06:08, Peter Maydell wrote:
> From: Akihiko Odaki <akihiko.odaki@daynix.com>
> 
> The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
> must be parsed with shlex.split().  Otherwise the script will fail if
> the build directory has a character like "~" in it.
> 
> Note: this fix cannot be backported directly to any stable branch
> that doesn't require Meson version 1.4.0 or better; otherwise it will
> work OK on Linux but will break on Windows hosts.
> 
> (Unfortunately, Meson prior to version 1.4.0 was inconsistent between
> host OSes about how it quoted arguments, and used a different quoting
> process on Windows hosts.  Our current git trunk already requires
> 1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
> the stable branches are still on older Meson.)
> 
> Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
> Reported-by: Michael Tokarev <mjt@tls.msk.ru>
> Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
> [PMM: Updated commit message to give all the detail about the
> Meson version compability requirements.]
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---
> This is essentially back to version 1 of Akihiko's patch, now we
> have a new enough Meson; I just updated the commit message.
>   https://patchew.org/QEMU/20230812061540.5398-1-akihiko.odaki@daynix.com/
> (I have dropped the various reviewed-by and tested-by headers because
> I figured the passage of time was enough to make them moot.)
> 
>   scripts/symlink-install-tree.py | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/symlink-install-tree.py b/scripts/symlink-install-tree.py
> index 8ed97e3c943..b72563895c5 100644
> --- a/scripts/symlink-install-tree.py
> +++ b/scripts/symlink-install-tree.py
> @@ -4,6 +4,7 @@
>   import errno
>   import json
>   import os
> +import shlex
>   import subprocess
>   import sys
>   
> @@ -14,7 +15,7 @@ def destdir_join(d1: str, d2: str) -> str:
>       return str(PurePath(d1, *PurePath(d2).parts[1:]))
>   
>   introspect = os.environ.get('MESONINTROSPECT')
> -out = subprocess.run([*introspect.split(' '), '--installed'],
> +out = subprocess.run([*shlex.split(introspect), '--installed'],
>                        stdout=subprocess.PIPE, check=True).stdout
>   for source, dest in json.loads(out).items():
>       bundle_dest = destdir_join('qemu-bundle', dest)

Hi,

would that be possible to pull this patch please?
It's currently blocking the devs who reported it initially.

Thanks,
Pierrick

Re: [PATCH v4] scripts/symlink-install-tree.py: Fix MESONINTROSPECT parsing

[Peter Maydell]

On Mon, 28 Oct 2024 at 19:43, Pierrick Bouvier
<pierrick.bouvier@linaro.org> wrote:
>
> On 10/18/24 06:08, Peter Maydell wrote:
> > From: Akihiko Odaki <akihiko.odaki@daynix.com>
> >
> > The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
> > must be parsed with shlex.split().  Otherwise the script will fail if
> > the build directory has a character like "~" in it.
> >
> > Note: this fix cannot be backported directly to any stable branch
> > that doesn't require Meson version 1.4.0 or better; otherwise it will
> > work OK on Linux but will break on Windows hosts.
> >
> > (Unfortunately, Meson prior to version 1.4.0 was inconsistent between
> > host OSes about how it quoted arguments, and used a different quoting
> > process on Windows hosts.  Our current git trunk already requires
> > 1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
> > the stable branches are still on older Meson.)
> >
> > Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
> > Reported-by: Michael Tokarev <mjt@tls.msk.ru>
> > Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
> > [PMM: Updated commit message to give all the detail about the
> > Meson version compability requirements.]
> > Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> > ---
> > This is essentially back to version 1 of Akihiko's patch, now we
> > have a new enough Meson; I just updated the commit message.
> >   https://patchew.org/QEMU/20230812061540.5398-1-akihiko.odaki@daynix.com/
> > (I have dropped the various reviewed-by and tested-by headers because
> > I figured the passage of time was enough to make them moot.)
> >
> >   scripts/symlink-install-tree.py | 3 ++-
> >   1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/scripts/symlink-install-tree.py b/scripts/symlink-install-tree.py
> > index 8ed97e3c943..b72563895c5 100644
> > --- a/scripts/symlink-install-tree.py
> > +++ b/scripts/symlink-install-tree.py
> > @@ -4,6 +4,7 @@
> >   import errno
> >   import json
> >   import os
> > +import shlex
> >   import subprocess
> >   import sys
> >
> > @@ -14,7 +15,7 @@ def destdir_join(d1: str, d2: str) -> str:
> >       return str(PurePath(d1, *PurePath(d2).parts[1:]))
> >
> >   introspect = os.environ.get('MESONINTROSPECT')
> > -out = subprocess.run([*introspect.split(' '), '--installed'],
> > +out = subprocess.run([*shlex.split(introspect), '--installed'],
> >                        stdout=subprocess.PIPE, check=True).stdout
> >   for source, dest in json.loads(out).items():
> >       bundle_dest = destdir_join('qemu-bundle', dest)
>
> Hi,
>
> would that be possible to pull this patch please?
> It's currently blocking the devs who reported it initially.

Yes, I'll put it in my upcoming pullreq.

thanks
-- PMM

Re: [PATCH v4] scripts/symlink-install-tree.py: Fix MESONINTROSPECT parsing

[Pierrick Bouvier]

On 10/29/24 03:04, Peter Maydell wrote:
> On Mon, 28 Oct 2024 at 19:43, Pierrick Bouvier
> <pierrick.bouvier@linaro.org> wrote:
>>
>> On 10/18/24 06:08, Peter Maydell wrote:
>>> From: Akihiko Odaki <akihiko.odaki@daynix.com>
>>>
>>> The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
>>> must be parsed with shlex.split().  Otherwise the script will fail if
>>> the build directory has a character like "~" in it.
>>>
>>> Note: this fix cannot be backported directly to any stable branch
>>> that doesn't require Meson version 1.4.0 or better; otherwise it will
>>> work OK on Linux but will break on Windows hosts.
>>>
>>> (Unfortunately, Meson prior to version 1.4.0 was inconsistent between
>>> host OSes about how it quoted arguments, and used a different quoting
>>> process on Windows hosts.  Our current git trunk already requires
>>> 1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
>>> the stable branches are still on older Meson.)
>>>
>>> Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
>>> Reported-by: Michael Tokarev <mjt@tls.msk.ru>
>>> Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
>>> [PMM: Updated commit message to give all the detail about the
>>> Meson version compability requirements.]
>>> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
>>> ---
>>> This is essentially back to version 1 of Akihiko's patch, now we
>>> have a new enough Meson; I just updated the commit message.
>>>    https://patchew.org/QEMU/20230812061540.5398-1-akihiko.odaki@daynix.com/
>>> (I have dropped the various reviewed-by and tested-by headers because
>>> I figured the passage of time was enough to make them moot.)
>>>
>>>    scripts/symlink-install-tree.py | 3 ++-
>>>    1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/scripts/symlink-install-tree.py b/scripts/symlink-install-tree.py
>>> index 8ed97e3c943..b72563895c5 100644
>>> --- a/scripts/symlink-install-tree.py
>>> +++ b/scripts/symlink-install-tree.py
>>> @@ -4,6 +4,7 @@
>>>    import errno
>>>    import json
>>>    import os
>>> +import shlex
>>>    import subprocess
>>>    import sys
>>>
>>> @@ -14,7 +15,7 @@ def destdir_join(d1: str, d2: str) -> str:
>>>        return str(PurePath(d1, *PurePath(d2).parts[1:]))
>>>
>>>    introspect = os.environ.get('MESONINTROSPECT')
>>> -out = subprocess.run([*introspect.split(' '), '--installed'],
>>> +out = subprocess.run([*shlex.split(introspect), '--installed'],
>>>                         stdout=subprocess.PIPE, check=True).stdout
>>>    for source, dest in json.loads(out).items():
>>>        bundle_dest = destdir_join('qemu-bundle', dest)
>>
>> Hi,
>>
>> would that be possible to pull this patch please?
>> It's currently blocking the devs who reported it initially.
> 
> Yes, I'll put it in my upcoming pullreq.
> 
> thanks
> -- PMM

Thanks a lot Peter :).