[RFC PATCH] tests/vm/netbsd: Remove missing py311-expat package

[RFC PATCH] tests/vm/netbsd: Remove missing py311-expat package

[Philippe Mathieu-Daudé]

Per commits a9dbde71da ("mkvenv: add better error message for
broken or missing ensurepip") and 1dee66c693 ("tests/vm: add
py310-expat to NetBSD"), we need py-expat to use ensurepip.

However the py311-expat package isn't available anymore:

  ### Installing packages ...
  processing remote summary (http://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/9.3/All)...
  database for http://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/9.3/All is up-to-date
  py311-expat is not available in the repository
  ...
  calculating dependencies.../py311-expat is not available in the repository
  pkg_install error log can be found in /var/db/pkgin/pkg_install-err.log

Dropping it from the default packages list allows creating the
NetBSD VM and build / test QEMU, without error from ensurepip.

This reverts commit 1dee66c693 ("tests/vm: add py310-expat to NetBSD").

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2109
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
RFC because ensurepip seems important due to cited commits
    but I'm not sure where it is used, and this fixes my CI build.
---
 tests/vm/netbsd | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tests/vm/netbsd b/tests/vm/netbsd
index 649fcad353..a3f6dd6b3c 100755
--- a/tests/vm/netbsd
+++ b/tests/vm/netbsd
@@ -31,7 +31,6 @@ class NetBSDVM(basevm.BaseVM):
         "pkgconf",
         "xz",
         "python311",
-        "py311-expat",
         "ninja-build",
 
         # gnu tools
-- 
2.41.0

Re: [RFC PATCH] tests/vm/netbsd: Remove missing py311-expat package

[Thomas Huth]

On 17/01/2024 12.24, Philippe Mathieu-Daudé wrote:
> Per commits a9dbde71da ("mkvenv: add better error message for
> broken or missing ensurepip") and 1dee66c693 ("tests/vm: add
> py310-expat to NetBSD"), we need py-expat to use ensurepip.
> 
> However the py311-expat package isn't available anymore:
> 
>    ### Installing packages ...
>    processing remote summary (http://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/9.3/All)...
>    database for http://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/9.3/All is up-to-date
>    py311-expat is not available in the repository
>    ...
>    calculating dependencies.../py311-expat is not available in the repository
>    pkg_install error log can be found in /var/db/pkgin/pkg_install-err.log
> 
> Dropping it from the default packages list allows creating the
> NetBSD VM and build / test QEMU, without error from ensurepip.
> 
> This reverts commit 1dee66c693 ("tests/vm: add py310-expat to NetBSD").
> 
> Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2109
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
> RFC because ensurepip seems important due to cited commits
>      but I'm not sure where it is used, and this fixes my CI build.

Thanks for tackling it, I also ran into this problem already and just did 
not have enough spare time yet to investigate.

I searched a little bit, and found this text here:
https://mail-index.netbsd.org/netbsd-announce/2024/01/01/msg000360.html

"
- Several packages have been folded into base packages.  While the
    result is simpler, those updating may need to force-remove the
    secondary packages, depending on the update method.  When doing
    make replace, one has to pkg_delete -f the secondary packages.
    pkgin handles at least the python packages correctly, removing the
    split package when updating python.  Specific packages and the
    former packages now included:

      * cairo: cairo-gobject
      * python: py-cElementTree py-curses py-cursespanel py-expat
        py-readline py-sqlite3
"

So it seems like the py-expat package has simply been merged into a base 
package now. Thus I think your patch is fine. It also fixes the problem for 
me, so:

Tested-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFC PATCH] tests/vm/netbsd: Remove missing py311-expat package

[Philippe Mathieu-Daudé]

On 17/1/24 13:10, Thomas Huth wrote:
> On 17/01/2024 12.24, Philippe Mathieu-Daudé wrote:
>> Per commits a9dbde71da ("mkvenv: add better error message for
>> broken or missing ensurepip") and 1dee66c693 ("tests/vm: add
>> py310-expat to NetBSD"), we need py-expat to use ensurepip.
>>
>> However the py311-expat package isn't available anymore:
>>
>>    ### Installing packages ...
>>    processing remote summary 
>> (http://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/9.3/All)...
>>    database for 
>> http://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/9.3/All is 
>> up-to-date
>>    py311-expat is not available in the repository
>>    ...
>>    calculating dependencies.../py311-expat is not available in the 
>> repository
>>    pkg_install error log can be found in 
>> /var/db/pkgin/pkg_install-err.log
>>
>> Dropping it from the default packages list allows creating the
>> NetBSD VM and build / test QEMU, without error from ensurepip.
>>
>> This reverts commit 1dee66c693 ("tests/vm: add py310-expat to NetBSD").
>>
>> Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2109
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>> ---
>> RFC because ensurepip seems important due to cited commits
>>      but I'm not sure where it is used, and this fixes my CI build.
> 
> Thanks for tackling it, I also ran into this problem already and just 
> did not have enough spare time yet to investigate.
> 
> I searched a little bit, and found this text here:
> https://mail-index.netbsd.org/netbsd-announce/2024/01/01/msg000360.html
> 
> "
> - Several packages have been folded into base packages.  While the
>     result is simpler, those updating may need to force-remove the
>     secondary packages, depending on the update method.  When doing
>     make replace, one has to pkg_delete -f the secondary packages.
>     pkgin handles at least the python packages correctly, removing the
>     split package when updating python.  Specific packages and the
>     former packages now included:
> 
>       * cairo: cairo-gobject
>       * python: py-cElementTree py-curses py-cursespanel py-expat
>         py-readline py-sqlite3
> "

Thanks for digging, I'll update the description.

> So it seems like the py-expat package has simply been merged into a base 
> package now. Thus I think your patch is fine. It also fixes the problem 
> for me, so:
> 
> Tested-by: Thomas Huth <thuth@redhat.com>
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 

Re: [RFC PATCH] tests/vm/netbsd: Remove missing py311-expat package

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@linaro.org> writes:

> Per commits a9dbde71da ("mkvenv: add better error message for
> broken or missing ensurepip") and 1dee66c693 ("tests/vm: add
> py310-expat to NetBSD"), we need py-expat to use ensurepip.

Queued to testing/next, thanks.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [RFC PATCH] tests/vm/netbsd: Remove missing py311-expat package

[Philippe Mathieu-Daudé]

On 17/1/24 16:09, Alex Bennée wrote:
> Philippe Mathieu-Daudé <philmd@linaro.org> writes:
> 
>> Per commits a9dbde71da ("mkvenv: add better error message for
>> broken or missing ensurepip") and 1dee66c693 ("tests/vm: add
>> py310-expat to NetBSD"), we need py-expat to use ensurepip.
> 
> Queued to testing/next, thanks.

Please take v2 instead:
https://lore.kernel.org/qemu-devel/20240117140746.23511-1-philmd@linaro.org/

Re: [RFC PATCH] tests/vm/netbsd: Remove missing py311-expat package

[John Snow]

On Wed, Jan 17, 2024 at 9:03 AM Philippe Mathieu-Daudé
<philmd@linaro.org> wrote:
>
> On 17/1/24 13:10, Thomas Huth wrote:
> > On 17/01/2024 12.24, Philippe Mathieu-Daudé wrote:
> >> Per commits a9dbde71da ("mkvenv: add better error message for
> >> broken or missing ensurepip") and 1dee66c693 ("tests/vm: add
> >> py310-expat to NetBSD"), we need py-expat to use ensurepip.
> >>
> >> However the py311-expat package isn't available anymore:
> >>
> >>    ### Installing packages ...
> >>    processing remote summary
> >> (http://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/9.3/All)...
> >>    database for
> >> http://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/9.3/All is
> >> up-to-date
> >>    py311-expat is not available in the repository
> >>    ...
> >>    calculating dependencies.../py311-expat is not available in the
> >> repository
> >>    pkg_install error log can be found in
> >> /var/db/pkgin/pkg_install-err.log
> >>
> >> Dropping it from the default packages list allows creating the
> >> NetBSD VM and build / test QEMU, without error from ensurepip.
> >>
> >> This reverts commit 1dee66c693 ("tests/vm: add py310-expat to NetBSD").
> >>
> >> Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2109
> >> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> >> ---
> >> RFC because ensurepip seems important due to cited commits
> >>      but I'm not sure where it is used, and this fixes my CI build.
> >
> > Thanks for tackling it, I also ran into this problem already and just
> > did not have enough spare time yet to investigate.
> >
> > I searched a little bit, and found this text here:
> > https://mail-index.netbsd.org/netbsd-announce/2024/01/01/msg000360.html
> >
> > "
> > - Several packages have been folded into base packages.  While the
> >     result is simpler, those updating may need to force-remove the
> >     secondary packages, depending on the update method.  When doing
> >     make replace, one has to pkg_delete -f the secondary packages.
> >     pkgin handles at least the python packages correctly, removing the
> >     split package when updating python.  Specific packages and the
> >     former packages now included:
> >
> >       * cairo: cairo-gobject
> >       * python: py-cElementTree py-curses py-cursespanel py-expat
> >         py-readline py-sqlite3
> > "
>
> Thanks for digging, I'll update the description.
>
> > So it seems like the py-expat package has simply been merged into a base
> > package now. Thus I think your patch is fine. It also fixes the problem
> > for me, so:
> >
> > Tested-by: Thomas Huth <thuth@redhat.com>
> > Reviewed-by: Thomas Huth <thuth@redhat.com>
> >
>

Quite late, but thank you both. If py-expat was folded in, then this
is definitely safe. mkvenv.py could possibly use a small update to its
error message hint:

            "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)"

I don't know if this is safe to remove in its entirety just yet,
because I don't know right away if any NetBSD distributions that we
consider "supported" still use a de-bundled py-expat. (Maybe this
error is still helpful for certain users in some cases we technically
support?)

If that's not the case, then we can simplify mkvenv and remove this
error handling case entirely.

--js