Re: [Buildroot] [PATCH] package/micropython: make use of recent micropython-lib

Re: [Buildroot] [PATCH] package/micropython: make use of recent micropython-lib

[Chris Packham]

[-- Attachment #1.1: Type: text/plain, Size: 10460 bytes --]

Hi Abilio,

On Mon, 29 Jan 2024, 7:30 am Abilio Marques, <abiliojr@gmail.com> wrote:

> Until now, micropython-lib was a package that installed v1.9.3,
> which is more than 6 years old. This was acceptable since micropython
> never made any other official release of the library until v1.20.
>
> Meanwhile, the libraries underwent a reorganization, and they are now
> available in a directory structure that cannot be copied directly into
> the target. This is my theory on why v1.9.3 is still present in the
> current day buildroot (which comes with micropython v1.22).
>
> This commit introduces an auxiliary script to collect those libraries
> and reorder them into a structure that can then be copied into
> /usr/lib/micropython. The script utilizes a module from the tools
> directory of the micropython repo.
>
> As part of the changes made by the micropython project, the libraries
> are now released together with the interpreter. They are cloned as a
> submodule into the lib/micropython-lib directory.
>
> These are the 2 main reasons why the old buildroot micropython-lib
> package is being removed.
>
> With this commit, micropython-lib is installed (optionally) as part
> of micropython. The original config variable name was retained for
> backwards compatibility.
>
> This commit also ensures that the libraries in micropython-lib will
> be updated together with newer versions of micropython.
>
> Signed-off-by: Abilio Marques <abiliojr@gmail.com>
>

Thanks for looking at this. Indeed I knew about the reorganization of
micropython-lib but hadn't had time to really look in to adapting buildroot
to it.

One thought I had was to build micropython as a host package and use that
to install the libs into the target rootfs.

I haven't had a chance to actually look at your patch in detail. I am a
little worried that we'll be dependent on some internal detail of how
things are currently laid out in micropython-lib. But this is certainly
better than anything I've been able to come up with to date.

---
>  DEVELOPERS                                    |  1 -
>  package/Config.in                             |  1 -
>  package/micropython-lib/Config.in             |  8 --
>  package/micropython-lib/micropython-lib.hash  |  3 -
>  package/micropython-lib/micropython-lib.mk    | 18 ----
>  package/micropython/Config.in                 | 10 +++
>  .../micropython/collect_micropython_lib.py    | 90 +++++++++++++++++++
>  package/micropython/micropython.mk            | 12 +++
>  8 files changed, 112 insertions(+), 31 deletions(-)
>  delete mode 100644 package/micropython-lib/Config.in
>  delete mode 100644 package/micropython-lib/micropython-lib.hash
>  delete mode 100644 package/micropython-lib/micropython-lib.mk
>  create mode 100755 package/micropython/collect_micropython_lib.py
>
> diff --git a/DEVELOPERS b/DEVELOPERS
> index 9528837dd0..cc9bc5b5cc 100644
> --- a/DEVELOPERS
> +++ b/DEVELOPERS
> @@ -572,7 +572,6 @@ F:  package/coremark/
>  F:     package/coremark-pro/
>  F:     package/gstreamer1/gst1-shark/
>  F:     package/micropython/
> -F:     package/micropython-lib/
>  F:     package/syslog-ng/
>
>  N:     Christian Kellermann <christian.kellermann@solectrix.de>
> diff --git a/package/Config.in b/package/Config.in
> index 5b8b15fa54..426bd7d090 100644
> --- a/package/Config.in
> +++ b/package/Config.in
> @@ -798,7 +798,6 @@ menu "Lua libraries/modules"
>  endmenu
>  endif
>         source "package/micropython/Config.in"
> -       source "package/micropython-lib/Config.in"
>         source "package/moarvm/Config.in"
>         source "package/mono/Config.in"
>  if BR2_PACKAGE_MONO
> diff --git a/package/micropython-lib/Config.in
> b/package/micropython-lib/Config.in
> deleted file mode 100644
> index 76557b220b..0000000000
> --- a/package/micropython-lib/Config.in
> +++ /dev/null
> @@ -1,8 +0,0 @@
> -config BR2_PACKAGE_MICROPYTHON_LIB
> -       bool "micropython-lib"
> -       depends on BR2_PACKAGE_MICROPYTHON
> -       select BR2_PACKAGE_PCRE # runtime
> -       help
> -         Core Python libraries ported to MicroPython.
> -
> -         http://micropython.org
> diff --git a/package/micropython-lib/micropython-lib.hash
> b/package/micropython-lib/micropython-lib.hash
> deleted file mode 100644
> index cbdda23844..0000000000
> --- a/package/micropython-lib/micropython-lib.hash
> +++ /dev/null
> @@ -1,3 +0,0 @@
> -# Locally computed
> -sha256  66e15380eb109613263beb6825b8eecb9191088270c1a59e8c7d922dd57183c7
> micropython-lib-1.9.3.tar.gz
> -sha256  baed4196a4310c576c2010f0a49f987a49e63856df7cd45af11cb3571df4bf74
> LICENSE
> diff --git a/package/micropython-lib/micropython-lib.mk
> b/package/micropython-lib/micropython-lib.mk
> deleted file mode 100644
> index 78ac0d3b35..0000000000
> --- a/package/micropython-lib/micropython-lib.mk
> +++ /dev/null
> @@ -1,18 +0,0 @@
>
> -################################################################################
> -#
> -# micropython-lib
> -#
>
> -################################################################################
> -
> -MICROPYTHON_LIB_VERSION = 1.9.3
> -MICROPYTHON_LIB_SITE = $(call
> github,micropython,micropython-lib,v$(MICROPYTHON_LIB_VERSION))
> -MICROPYTHON_LIB_LICENSE = Python-2.0 (some modules), MIT (everything else)
> -MICROPYTHON_LIB_LICENSE_FILES = LICENSE
> -
> -define MICROPYTHON_LIB_INSTALL_TARGET_CMDS
> -       $(TARGET_MAKE_ENV) $(MAKE) -C $(@D) \
> -               PREFIX=$(TARGET_DIR)/usr/lib/micropython \
> -               install
> -endef
> -
> -$(eval $(generic-package))
> diff --git a/package/micropython/Config.in b/package/micropython/Config.in
> index 30161c8b70..2c2ae7a4df 100644
> --- a/package/micropython/Config.in
> +++ b/package/micropython/Config.in
> @@ -9,5 +9,15 @@ config BR2_PACKAGE_MICROPYTHON
>
>           http://micropython.org
>
> +config BR2_PACKAGE_MICROPYTHON_LIB
> +       bool "micropython-lib"
> +       default y
> +       depends on BR2_PACKAGE_MICROPYTHON
> +       select BR2_PACKAGE_PCRE # runtime
> +       help
> +         Core Python libraries ported to MicroPython.
> +
> +         http://micropython.org
> +
>  comment "micropython needs a toolchain w/ threads, dynamic library"
>         depends on !BR2_TOOLCHAIN_HAS_THREADS || BR2_STATIC_LIBS
> diff --git a/package/micropython/collect_micropython_lib.py
> b/package/micropython/collect_micropython_lib.py
> new file mode 100755
> index 0000000000..778a1c4c7d
> --- /dev/null
> +++ b/package/micropython/collect_micropython_lib.py
> @@ -0,0 +1,90 @@
> +#!/usr/bin/env python3
> +
> +"""
> +Sometime after v1.9.3, micropython-lib's directory structure was cleaned
> up and
> +enhanced with manifest files.
> +
> +Though cleaner, it means it cannot be directly copied into the target.
> +This script is during build process to perform that conversion.
> +
> +It makes use of manifestfile.py, which is normally located in
> micropython's
> +tool directory.
> +
> +It also depends on the micropython-lib that is cloned in
> lib/micropython-lib
> +during build.
> +"""
> +
> +import argparse
> +import importlib
> +import os
> +import shutil
> +import sys
> +
> +
> +def get_library_name_type(manifest_path: str) -> tuple[str, bool]:
> +    split = manifest_path.split("/")
> +    return (split[-2], "unix-ffi" in split)  # -1: "manifest.py", -2:
> library name
> +
> +
> +def get_all_libraries(mpy_lib_dir: str):
> +    # reuse manifestfile module capabilities to scan the micropython-lib
> directory
> +
> +    collected_list = manifestfile.ManifestFile(
> +        manifestfile.MODE_FREEZE, {"MPY_LIB_DIR": mpy_lib_dir}
> +    )
> +    collected_list.freeze(mpy_lib_dir)
> +
> +    for file in collected_list.files():
> +        if file.target_path.endswith("manifest.py"):
> +            yield get_library_name_type(file.full_path)
> +
> +
> +def copy_file(src: str, target: str, destination: str):
> +    s = target.split("/")
> +    s.pop()
> +    d = f"{destination}/{'/'.join(s)}"
> +    if not os.path.exists(d):
> +        os.makedirs(d)
> +    shutil.copy(src, f"{destination}/{target}")
> +
> +
> +def copy_libraries(manifest, destination: str):
> +    for f in manifest.files():
> +        copy_file(f.full_path, f.target_path, destination)
> +
> +
> +def process_cmdline_args():
> +    parser = argparse.ArgumentParser(
> +        description="Prepare micropython-lib to be installed"
> +    )
> +    parser.add_argument("micropython", help="Path to micropython source
> directory")
> +    parser.add_argument("destination", help="Destination directory")
> +
> +    args = parser.parse_args()
> +    return os.path.abspath(args.micropython),
> os.path.abspath(args.destination)
> +
> +
> +def load_manifestfile_module(micropython_dir: str):
> +    global manifestfile
> +    sys.path.append(f"{micropython_dir}/tools")
> +    manifestfile = importlib.import_module("manifestfile")
> +
> +
> +def main():
> +    micropython_dir, destination_dir = process_cmdline_args()
> +    mpy_lib_dir = f"{micropython_dir}/lib/micropython-lib"
> +
> +    # load manifest file after knowing where it is
> +    load_manifestfile_module(micropython_dir)
> +
> +    manifest = manifestfile.ManifestFile(
> +        manifestfile.MODE_FREEZE, {"MPY_LIB_DIR": mpy_lib_dir}
> +    )
> +
> +    for library, is_ffi in get_all_libraries(mpy_lib_dir):
> +        manifest.require(library, unix_ffi=is_ffi)
> +
> +    copy_libraries(manifest, destination_dir)
> +
> +
> +main()
> diff --git a/package/micropython/micropython.mk b/package/micropython/
> micropython.mk
> index 37c148da94..a9757e266c 100644
> --- a/package/micropython/micropython.mk
> +++ b/package/micropython/micropython.mk
> @@ -47,6 +47,8 @@ define MICROPYTHON_BUILD_CMDS
>         $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross
>         $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \
>                 $(MICROPYTHON_MAKE_OPTS)
> +       $(EXTRA_ENV) package/micropython/collect_micropython_lib.py \
> +               $(@D) $(@D)/.built_pylib
>  endef
>
>  define MICROPYTHON_INSTALL_TARGET_CMDS
> @@ -57,4 +59,14 @@ define MICROPYTHON_INSTALL_TARGET_CMDS
>                 install
>  endef
>
> +define MICROPYTHON_INSTALL_LIBS
> +       $(INSTALL) -d -m 0755 $(TARGET_DIR)/usr/lib/micropython
> +       cp -a $(@D)/.built_pylib/* $(TARGET_DIR)/usr/lib/micropython
> +endef
> +
> +ifeq ($(BR2_PACKAGE_MICROPYTHON_LIB),y)
> +MICROPYTHON_POST_INSTALL_TARGET_HOOKS += MICROPYTHON_INSTALL_LIBS
> +endif
> +
> +
>  $(eval $(generic-package))
> --
> 2.43.0
>
>

[-- Attachment #1.2: Type: text/html, Size: 13736 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/micropython: make use of recent micropython-lib

[Abilio Marques]

[-- Attachment #1.1: Type: text/plain, Size: 23370 bytes --]

Hello Chris,

Because this is my first buildroot submission, I tried to keep it simple.
It makes use of existing micropython tools (used to process manifests) to
discover the list of packages available in micropython-lib. My hope is that
by relying on them, any future changes in directory structure will be
covered by their own "manifestfile.py" tool.

Also, it still copies the files as plain source code ".py" files. I did it
this way to avoid abrupt changes that could annoy people using the package.

I have an idea that mixes part of my code (the one that discovers the
packages) and freezes them (using the same manifests tool), but I thought
it would be a bridge too far for a successful first submission. Also, I
want to find time to make it configurable using menuconfig options (each
one has it's drawbacks):
- frozen as mpy files
- frozen inside the micropython binary
- keep current text files

If there is interest, I can prepare it in a couple weeks from now. Ideas
are welcomed.

Kind regards,
Abilio



On Sun, Jan 28, 2024 at 10:58 AM Chris Packham <judge.packham@gmail.com>
wrote:

> Hi Abilio,
>
> On Mon, 29 Jan 2024, 7:30 am Abilio Marques, <abiliojr@gmail.com> wrote:
>
>> Until now, micropython-lib was a package that installed v1.9.3,
>> which is more than 6 years old. This was acceptable since micropython
>> never made any other official release of the library until v1.20.
>>
>> Meanwhile, the libraries underwent a reorganization, and they are now
>> available in a directory structure that cannot be copied directly into
>> the target. This is my theory on why v1.9.3 is still present in the
>> current day buildroot (which comes with micropython v1.22).
>>
>> This commit introduces an auxiliary script to collect those libraries
>> and reorder them into a structure that can then be copied into
>> /usr/lib/micropython. The script utilizes a module from the tools
>> directory of the micropython repo.
>>
>> As part of the changes made by the micropython project, the libraries
>> are now released together with the interpreter. They are cloned as a
>> submodule into the lib/micropython-lib directory.
>>
>> These are the 2 main reasons why the old buildroot micropython-lib
>> package is being removed.
>>
>> With this commit, micropython-lib is installed (optionally) as part
>> of micropython. The original config variable name was retained for
>> backwards compatibility.
>>
>> This commit also ensures that the libraries in micropython-lib will
>> be updated together with newer versions of micropython.
>>
>> Signed-off-by: Abilio Marques <abiliojr@gmail.com>
>>
>
> Thanks for looking at this. Indeed I knew about the reorganization of
> micropython-lib but hadn't had time to really look in to adapting buildroot
> to it.
>
> One thought I had was to build micropython as a host package and use that
> to install the libs into the target rootfs.
>
> I haven't had a chance to actually look at your patch in detail. I am a
> little worried that we'll be dependent on some internal detail of how
> things are currently laid out in micropython-lib. But this is certainly
> better than anything I've been able to come up with to date.
>
> ---
>>  DEVELOPERS                                    |  1 -
>>  package/Config.in                             |  1 -
>>  package/micropython-lib/Config.in             |  8 --
>>  package/micropython-lib/micropython-lib.hash  |  3 -
>>  package/micropython-lib/micropython-lib.mk    | 18 ----
>>  package/micropython/Config.in                 | 10 +++
>>  .../micropython/collect_micropython_lib.py    | 90 +++++++++++++++++++
>>  package/micropython/micropython.mk            | 12 +++
>>  8 files changed, 112 insertions(+), 31 deletions(-)
>>  delete mode 100644 package/micropython-lib/Config.in
>>  delete mode 100644 package/micropython-lib/micropython-lib.hash
>>  delete mode 100644 package/micropython-lib/micropython-lib.mk
>>  create mode 100755 package/micropython/collect_micropython_lib.py
>>
>> diff --git a/DEVELOPERS b/DEVELOPERS
>> index 9528837dd0..cc9bc5b5cc 100644
>> --- a/DEVELOPERS
>> +++ b/DEVELOPERS
>> @@ -572,7 +572,6 @@ F:  package/coremark/
>>  F:     package/coremark-pro/
>>  F:     package/gstreamer1/gst1-shark/
>>  F:     package/micropython/
>> -F:     package/micropython-lib/
>>  F:     package/syslog-ng/
>>
>>  N:     Christian Kellermann <christian.kellermann@solectrix.de>
>> diff --git a/package/Config.in b/package/Config.in
>> index 5b8b15fa54..426bd7d090 100644
>> --- a/package/Config.in
>> +++ b/package/Config.in
>> @@ -798,7 +798,6 @@ menu "Lua libraries/modules"
>>  endmenu
>>  endif
>>         source "package/micropython/Config.in"
>> -       source "package/micropython-lib/Config.in"
>>         source "package/moarvm/Config.in"
>>         source "package/mono/Config.in"
>>  if BR2_PACKAGE_MONO
>> diff --git a/package/micropython-lib/Config.in
>> b/package/micropython-lib/Config.in
>> deleted file mode 100644
>> index 76557b220b..0000000000
>> --- a/package/micropython-lib/Config.in
>> +++ /dev/null
>> @@ -1,8 +0,0 @@
>> -config BR2_PACKAGE_MICROPYTHON_LIB
>> -       bool "micropython-lib"
>> -       depends on BR2_PACKAGE_MICROPYTHON
>> -       select BR2_PACKAGE_PCRE # runtime
>> -       help
>> -         Core Python libraries ported to MicroPython.
>> -
>> -         http://micropython.org
>> diff --git a/package/micropython-lib/micropython-lib.hash
>> b/package/micropython-lib/micropython-lib.hash
>> deleted file mode 100644
>> index cbdda23844..0000000000
>> --- a/package/micropython-lib/micropython-lib.hash
>> +++ /dev/null
>> @@ -1,3 +0,0 @@
>> -# Locally computed
>> -sha256
>> 66e15380eb109613263beb6825b8eecb9191088270c1a59e8c7d922dd57183c7
>> micropython-lib-1.9.3.tar.gz
>> -sha256
>> baed4196a4310c576c2010f0a49f987a49e63856df7cd45af11cb3571df4bf74  LICENSE
>> diff --git a/package/micropython-lib/micropython-lib.mk
>> b/package/micropython-lib/micropython-lib.mk
>> deleted file mode 100644
>> index 78ac0d3b35..0000000000
>> --- a/package/micropython-lib/micropython-lib.mk
>> +++ /dev/null
>> @@ -1,18 +0,0 @@
>>
>> -################################################################################
>> -#
>> -# micropython-lib
>> -#
>>
>> -################################################################################
>> -
>> -MICROPYTHON_LIB_VERSION = 1.9.3
>> -MICROPYTHON_LIB_SITE = $(call
>> github,micropython,micropython-lib,v$(MICROPYTHON_LIB_VERSION))
>> -MICROPYTHON_LIB_LICENSE = Python-2.0 (some modules), MIT (everything
>> else)
>> -MICROPYTHON_LIB_LICENSE_FILES = LICENSE
>> -
>> -define MICROPYTHON_LIB_INSTALL_TARGET_CMDS
>> -       $(TARGET_MAKE_ENV) $(MAKE) -C $(@D) \
>> -               PREFIX=$(TARGET_DIR)/usr/lib/micropython \
>> -               install
>> -endef
>> -
>> -$(eval $(generic-package))
>> diff --git a/package/micropython/Config.in b/package/micropython/Config.in
>> index 30161c8b70..2c2ae7a4df 100644
>> --- a/package/micropython/Config.in
>> +++ b/package/micropython/Config.in
>> @@ -9,5 +9,15 @@ config BR2_PACKAGE_MICROPYTHON
>>
>>           http://micropython.org
>>
>> +config BR2_PACKAGE_MICROPYTHON_LIB
>> +       bool "micropython-lib"
>> +       default y
>> +       depends on BR2_PACKAGE_MICROPYTHON
>> +       select BR2_PACKAGE_PCRE # runtime
>> +       help
>> +         Core Python libraries ported to MicroPython.
>> +
>> +         http://micropython.org
>> +
>>  comment "micropython needs a toolchain w/ threads, dynamic library"
>>         depends on !BR2_TOOLCHAIN_HAS_THREADS || BR2_STATIC_LIBS
>> diff --git a/package/micropython/collect_micropython_lib.py
>> b/package/micropython/collect_micropython_lib.py
>> new file mode 100755
>> index 0000000000..778a1c4c7d
>> --- /dev/null
>> +++ b/package/micropython/collect_micropython_lib.py
>> @@ -0,0 +1,90 @@
>> +#!/usr/bin/env python3
>> +
>> +"""
>> +Sometime after v1.9.3, micropython-lib's directory structure was cleaned
>> up and
>> +enhanced with manifest files.
>> +
>> +Though cleaner, it means it cannot be directly copied into the target.
>> +This script is during build process to perform that conversion.
>> +
>> +It makes use of manifestfile.py, which is normally located in
>> micropython's
>> +tool directory.
>> +
>> +It also depends on the micropython-lib that is cloned in
>> lib/micropython-lib
>> +during build.
>> +"""
>> +
>> +import argparse
>> +import importlib
>> +import os
>> +import shutil
>> +import sys
>> +
>> +
>> +def get_library_name_type(manifest_path: str) -> tuple[str, bool]:
>> +    split = manifest_path.split("/")
>> +    return (split[-2], "unix-ffi" in split)  # -1: "manifest.py", -2:
>> library name
>> +
>> +
>> +def get_all_libraries(mpy_lib_dir: str):
>> +    # reuse manifestfile module capabilities to scan the micropython-lib
>> directory
>> +
>> +    collected_list = manifestfile.ManifestFile(
>> +        manifestfile.MODE_FREEZE, {"MPY_LIB_DIR": mpy_lib_dir}
>> +    )
>> +    collected_list.freeze(mpy_lib_dir)
>> +
>> +    for file in collected_list.files():
>> +        if file.target_path.endswith("manifest.py"):
>> +            yield get_library_name_type(file.full_path)
>> +
>> +
>> +def copy_file(src: str, target: str, destination: str):
>> +    s = target.split("/")
>> +    s.pop()
>> +    d = f"{destination}/{'/'.join(s)}"
>> +    if not os.path.exists(d):
>> +        os.makedirs(d)
>> +    shutil.copy(src, f"{destination}/{target}")
>> +
>> +
>> +def copy_libraries(manifest, destination: str):
>> +    for f in manifest.files():
>> +        copy_file(f.full_path, f.target_path, destination)
>> +
>> +
>> +def process_cmdline_args():
>> +    parser = argparse.ArgumentParser(
>> +        description="Prepare micropython-lib to be installed"
>> +    )
>> +    parser.add_argument("micropython", help="Path to micropython source
>> directory")
>> +    parser.add_argument("destination", help="Destination directory")
>> +
>> +    args = parser.parse_args()
>> +    return os.path.abspath(args.micropython),
>> os.path.abspath(args.destination)
>> +
>> +
>> +def load_manifestfile_module(micropython_dir: str):
>> +    global manifestfile
>> +    sys.path.append(f"{micropython_dir}/tools")
>> +    manifestfile = importlib.import_module("manifestfile")
>> +
>> +
>> +def main():
>> +    micropython_dir, destination_dir = process_cmdline_args()
>> +    mpy_lib_dir = f"{micropython_dir}/lib/micropython-lib"
>> +
>> +    # load manifest file after knowing where it is
>> +    load_manifestfile_module(micropython_dir)
>> +
>> +    manifest = manifestfile.ManifestFile(
>> +        manifestfile.MODE_FREEZE, {"MPY_LIB_DIR": mpy_lib_dir}
>> +    )
>> +
>> +    for library, is_ffi in get_all_libraries(mpy_lib_dir):
>> +        manifest.require(library, unix_ffi=is_ffi)
>> +
>> +    copy_libraries(manifest, destination_dir)
>> +
>> +
>> +main()
>> diff --git a/package/micropython/micropython.mk b/package/micropython/
>> micropython.mk
>> index 37c148da94..a9757e266c 100644
>> --- a/package/micropython/micropython.mk
>> +++ b/package/micropython/micropython.mk
>> @@ -47,6 +47,8 @@ define MICROPYTHON_BUILD_CMDS
>>         $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross
>>         $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \
>>                 $(MICROPYTHON_MAKE_OPTS)
>> +       $(EXTRA_ENV) package/micropython/collect_micropython_lib.py \
>> +               $(@D) $(@D)/.built_pylib
>>  endef
>>
>>  define MICROPYTHON_INSTALL_TARGET_CMDS
>> @@ -57,4 +59,14 @@ define MICROPYTHON_INSTALL_TARGET_CMDS
>>                 install
>>  endef
>>
>> +define MICROPYTHON_INSTALL_LIBS
>> +       $(INSTALL) -d -m 0755 $(TARGET_DIR)/usr/lib/micropython
>> +       cp -a $(@D)/.built_pylib/* $(TARGET_DIR)/usr/lib/micropython
>> +endef
>> +
>> +ifeq ($(BR2_PACKAGE_MICROPYTHON_LIB),y)
>> +MICROPYTHON_POST_INSTALL_TARGET_HOOKS += MICROPYTHON_INSTALL_LIBS
>> +endif
>> +
>> +
>>  $(eval $(generic-package))
>> --
>> 2.43.0
>>
>>
On Sun, Jan 28, 2024 at 10:58 AM Chris Packham <judge.packham@gmail.com>
wrote:

> Hi Abilio,
>
> On Mon, 29 Jan 2024, 7:30 am Abilio Marques, <abiliojr@gmail.com> wrote:
>
>> Until now, micropython-lib was a package that installed v1.9.3,
>> which is more than 6 years old. This was acceptable since micropython
>> never made any other official release of the library until v1.20.
>>
>> Meanwhile, the libraries underwent a reorganization, and they are now
>> available in a directory structure that cannot be copied directly into
>> the target. This is my theory on why v1.9.3 is still present in the
>> current day buildroot (which comes with micropython v1.22).
>>
>> This commit introduces an auxiliary script to collect those libraries
>> and reorder them into a structure that can then be copied into
>> /usr/lib/micropython. The script utilizes a module from the tools
>> directory of the micropython repo.
>>
>> As part of the changes made by the micropython project, the libraries
>> are now released together with the interpreter. They are cloned as a
>> submodule into the lib/micropython-lib directory.
>>
>> These are the 2 main reasons why the old buildroot micropython-lib
>> package is being removed.
>>
>> With this commit, micropython-lib is installed (optionally) as part
>> of micropython. The original config variable name was retained for
>> backwards compatibility.
>>
>> This commit also ensures that the libraries in micropython-lib will
>> be updated together with newer versions of micropython.
>>
>> Signed-off-by: Abilio Marques <abiliojr@gmail.com>
>>
>
> Thanks for looking at this. Indeed I knew about the reorganization of
> micropython-lib but hadn't had time to really look in to adapting buildroot
> to it.
>
> One thought I had was to build micropython as a host package and use that
> to install the libs into the target rootfs.
>
> I haven't had a chance to actually look at your patch in detail. I am a
> little worried that we'll be dependent on some internal detail of how
> things are currently laid out in micropython-lib. But this is certainly
> better than anything I've been able to come up with to date.
>
> ---
>>  DEVELOPERS                                    |  1 -
>>  package/Config.in                             |  1 -
>>  package/micropython-lib/Config.in             |  8 --
>>  package/micropython-lib/micropython-lib.hash  |  3 -
>>  package/micropython-lib/micropython-lib.mk    | 18 ----
>>  package/micropython/Config.in                 | 10 +++
>>  .../micropython/collect_micropython_lib.py    | 90 +++++++++++++++++++
>>  package/micropython/micropython.mk            | 12 +++
>>  8 files changed, 112 insertions(+), 31 deletions(-)
>>  delete mode 100644 package/micropython-lib/Config.in
>>  delete mode 100644 package/micropython-lib/micropython-lib.hash
>>  delete mode 100644 package/micropython-lib/micropython-lib.mk
>>  create mode 100755 package/micropython/collect_micropython_lib.py
>>
>> diff --git a/DEVELOPERS b/DEVELOPERS
>> index 9528837dd0..cc9bc5b5cc 100644
>> --- a/DEVELOPERS
>> +++ b/DEVELOPERS
>> @@ -572,7 +572,6 @@ F:  package/coremark/
>>  F:     package/coremark-pro/
>>  F:     package/gstreamer1/gst1-shark/
>>  F:     package/micropython/
>> -F:     package/micropython-lib/
>>  F:     package/syslog-ng/
>>
>>  N:     Christian Kellermann <christian.kellermann@solectrix.de>
>> diff --git a/package/Config.in b/package/Config.in
>> index 5b8b15fa54..426bd7d090 100644
>> --- a/package/Config.in
>> +++ b/package/Config.in
>> @@ -798,7 +798,6 @@ menu "Lua libraries/modules"
>>  endmenu
>>  endif
>>         source "package/micropython/Config.in"
>> -       source "package/micropython-lib/Config.in"
>>         source "package/moarvm/Config.in"
>>         source "package/mono/Config.in"
>>  if BR2_PACKAGE_MONO
>> diff --git a/package/micropython-lib/Config.in
>> b/package/micropython-lib/Config.in
>> deleted file mode 100644
>> index 76557b220b..0000000000
>> --- a/package/micropython-lib/Config.in
>> +++ /dev/null
>> @@ -1,8 +0,0 @@
>> -config BR2_PACKAGE_MICROPYTHON_LIB
>> -       bool "micropython-lib"
>> -       depends on BR2_PACKAGE_MICROPYTHON
>> -       select BR2_PACKAGE_PCRE # runtime
>> -       help
>> -         Core Python libraries ported to MicroPython.
>> -
>> -         http://micropython.org
>> diff --git a/package/micropython-lib/micropython-lib.hash
>> b/package/micropython-lib/micropython-lib.hash
>> deleted file mode 100644
>> index cbdda23844..0000000000
>> --- a/package/micropython-lib/micropython-lib.hash
>> +++ /dev/null
>> @@ -1,3 +0,0 @@
>> -# Locally computed
>> -sha256
>> 66e15380eb109613263beb6825b8eecb9191088270c1a59e8c7d922dd57183c7
>> micropython-lib-1.9.3.tar.gz
>> -sha256
>> baed4196a4310c576c2010f0a49f987a49e63856df7cd45af11cb3571df4bf74  LICENSE
>> diff --git a/package/micropython-lib/micropython-lib.mk
>> b/package/micropython-lib/micropython-lib.mk
>> deleted file mode 100644
>> index 78ac0d3b35..0000000000
>> --- a/package/micropython-lib/micropython-lib.mk
>> +++ /dev/null
>> @@ -1,18 +0,0 @@
>>
>> -################################################################################
>> -#
>> -# micropython-lib
>> -#
>>
>> -################################################################################
>> -
>> -MICROPYTHON_LIB_VERSION = 1.9.3
>> -MICROPYTHON_LIB_SITE = $(call
>> github,micropython,micropython-lib,v$(MICROPYTHON_LIB_VERSION))
>> -MICROPYTHON_LIB_LICENSE = Python-2.0 (some modules), MIT (everything
>> else)
>> -MICROPYTHON_LIB_LICENSE_FILES = LICENSE
>> -
>> -define MICROPYTHON_LIB_INSTALL_TARGET_CMDS
>> -       $(TARGET_MAKE_ENV) $(MAKE) -C $(@D) \
>> -               PREFIX=$(TARGET_DIR)/usr/lib/micropython \
>> -               install
>> -endef
>> -
>> -$(eval $(generic-package))
>> diff --git a/package/micropython/Config.in b/package/micropython/Config.in
>> index 30161c8b70..2c2ae7a4df 100644
>> --- a/package/micropython/Config.in
>> +++ b/package/micropython/Config.in
>> @@ -9,5 +9,15 @@ config BR2_PACKAGE_MICROPYTHON
>>
>>           http://micropython.org
>>
>> +config BR2_PACKAGE_MICROPYTHON_LIB
>> +       bool "micropython-lib"
>> +       default y
>> +       depends on BR2_PACKAGE_MICROPYTHON
>> +       select BR2_PACKAGE_PCRE # runtime
>> +       help
>> +         Core Python libraries ported to MicroPython.
>> +
>> +         http://micropython.org
>> +
>>  comment "micropython needs a toolchain w/ threads, dynamic library"
>>         depends on !BR2_TOOLCHAIN_HAS_THREADS || BR2_STATIC_LIBS
>> diff --git a/package/micropython/collect_micropython_lib.py
>> b/package/micropython/collect_micropython_lib.py
>> new file mode 100755
>> index 0000000000..778a1c4c7d
>> --- /dev/null
>> +++ b/package/micropython/collect_micropython_lib.py
>> @@ -0,0 +1,90 @@
>> +#!/usr/bin/env python3
>> +
>> +"""
>> +Sometime after v1.9.3, micropython-lib's directory structure was cleaned
>> up and
>> +enhanced with manifest files.
>> +
>> +Though cleaner, it means it cannot be directly copied into the target.
>> +This script is during build process to perform that conversion.
>> +
>> +It makes use of manifestfile.py, which is normally located in
>> micropython's
>> +tool directory.
>> +
>> +It also depends on the micropython-lib that is cloned in
>> lib/micropython-lib
>> +during build.
>> +"""
>> +
>> +import argparse
>> +import importlib
>> +import os
>> +import shutil
>> +import sys
>> +
>> +
>> +def get_library_name_type(manifest_path: str) -> tuple[str, bool]:
>> +    split = manifest_path.split("/")
>> +    return (split[-2], "unix-ffi" in split)  # -1: "manifest.py", -2:
>> library name
>> +
>> +
>> +def get_all_libraries(mpy_lib_dir: str):
>> +    # reuse manifestfile module capabilities to scan the micropython-lib
>> directory
>> +
>> +    collected_list = manifestfile.ManifestFile(
>> +        manifestfile.MODE_FREEZE, {"MPY_LIB_DIR": mpy_lib_dir}
>> +    )
>> +    collected_list.freeze(mpy_lib_dir)
>> +
>> +    for file in collected_list.files():
>> +        if file.target_path.endswith("manifest.py"):
>> +            yield get_library_name_type(file.full_path)
>> +
>> +
>> +def copy_file(src: str, target: str, destination: str):
>> +    s = target.split("/")
>> +    s.pop()
>> +    d = f"{destination}/{'/'.join(s)}"
>> +    if not os.path.exists(d):
>> +        os.makedirs(d)
>> +    shutil.copy(src, f"{destination}/{target}")
>> +
>> +
>> +def copy_libraries(manifest, destination: str):
>> +    for f in manifest.files():
>> +        copy_file(f.full_path, f.target_path, destination)
>> +
>> +
>> +def process_cmdline_args():
>> +    parser = argparse.ArgumentParser(
>> +        description="Prepare micropython-lib to be installed"
>> +    )
>> +    parser.add_argument("micropython", help="Path to micropython source
>> directory")
>> +    parser.add_argument("destination", help="Destination directory")
>> +
>> +    args = parser.parse_args()
>> +    return os.path.abspath(args.micropython),
>> os.path.abspath(args.destination)
>> +
>> +
>> +def load_manifestfile_module(micropython_dir: str):
>> +    global manifestfile
>> +    sys.path.append(f"{micropython_dir}/tools")
>> +    manifestfile = importlib.import_module("manifestfile")
>> +
>> +
>> +def main():
>> +    micropython_dir, destination_dir = process_cmdline_args()
>> +    mpy_lib_dir = f"{micropython_dir}/lib/micropython-lib"
>> +
>> +    # load manifest file after knowing where it is
>> +    load_manifestfile_module(micropython_dir)
>> +
>> +    manifest = manifestfile.ManifestFile(
>> +        manifestfile.MODE_FREEZE, {"MPY_LIB_DIR": mpy_lib_dir}
>> +    )
>> +
>> +    for library, is_ffi in get_all_libraries(mpy_lib_dir):
>> +        manifest.require(library, unix_ffi=is_ffi)
>> +
>> +    copy_libraries(manifest, destination_dir)
>> +
>> +
>> +main()
>> diff --git a/package/micropython/micropython.mk b/package/micropython/
>> micropython.mk
>> index 37c148da94..a9757e266c 100644
>> --- a/package/micropython/micropython.mk
>> +++ b/package/micropython/micropython.mk
>> @@ -47,6 +47,8 @@ define MICROPYTHON_BUILD_CMDS
>>         $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross
>>         $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \
>>                 $(MICROPYTHON_MAKE_OPTS)
>> +       $(EXTRA_ENV) package/micropython/collect_micropython_lib.py \
>> +               $(@D) $(@D)/.built_pylib
>>  endef
>>
>>  define MICROPYTHON_INSTALL_TARGET_CMDS
>> @@ -57,4 +59,14 @@ define MICROPYTHON_INSTALL_TARGET_CMDS
>>                 install
>>  endef
>>
>> +define MICROPYTHON_INSTALL_LIBS
>> +       $(INSTALL) -d -m 0755 $(TARGET_DIR)/usr/lib/micropython
>> +       cp -a $(@D)/.built_pylib/* $(TARGET_DIR)/usr/lib/micropython
>> +endef
>> +
>> +ifeq ($(BR2_PACKAGE_MICROPYTHON_LIB),y)
>> +MICROPYTHON_POST_INSTALL_TARGET_HOOKS += MICROPYTHON_INSTALL_LIBS
>> +endif
>> +
>> +
>>  $(eval $(generic-package))
>> --
>> 2.43.0
>>
>>

[-- Attachment #1.2: Type: text/html, Size: 30693 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/micropython: make use of recent micropython-lib

[Yann E. MORIN]

Abilio, All,

On 2024-01-28 11:28 -0800, Abilio Marques spake thusly:
> Because this is my first buildroot submission, I tried to keep it simple. It
> makes use of existing micropython tools (used to process manifests) to discover
> the list of packages available in micropython-lib. My hope is that by relying
> on them, any future changes in directory structure will be covered by their own
> "manifestfile.py" tool.

This is a good rationale, and maybe you can add that to the commit log.

> Also, it still copies the files as plain source code ".py" files. I did it this
> way to avoid abrupt changes that could annoy people using the package.

Yes, agreed.

> I have an idea that mixes part of my code (the one that discovers the packages)
> and freezes them (using the same manifests tool), but I thought it would be a
> bridge too far for a successful first submission.

Agreed: I like that your commit does not change the existing behaviour.
This is good.

> Also, I want to find time to
> make it configurable using menuconfig options (each one has it's drawbacks):
> - frozen as mpy files
> - frozen inside the micropython binary
> - keep current text files

We have a similar choice for the full-blown CPython, where we can keep
only .py files, keep only .pyc files, or keep both. So si=uch a choice
for micropython seems totally acceptable. Just o make your future
changes better, it would be two separate changes;
 1. introduce frozen vs. text choice
 2. add frozen-in-binary to the choice

> If there is interest, I can prepare it in a couple weeks from now. Ideas are
> welcomed.

The interest is on your side: if you do not have that interest, do not
do it just for the show; a new feature needs to have someone that
actually needs it and tested it (to some extent).

And speaking of tests, we now have a micropython test:
    support/testing/tests/package/test_micropython.py

So it would be nice that, when/if you add the frozen support, you extend
that test to also test frozen libs.

Thanks!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/micropython: make use of recent micropython-lib

[Yann E. MORIN]

Chris, All,

On 2024-01-29 07:58 +1300, Chris Packham spake thusly:
> On Mon, 29 Jan 2024, 7:30 am Abilio Marques, <[1]abiliojr@gmail.com> wrote:
[--SNIP--]
>     This commit also ensures that the libraries in micropython-lib will
>     be updated together with newer versions of micropython.
[--SNIP--]
> One thought I had was to build micropython as a host package and use that to
> install the libs into the target rootfs.

Yes, this is also my opinion: use host-micropython, if possible.

I see that micropython already depends on host-python3, though. Would
host-micropython be sufficient in lieu of host-python3, to build target
micropython?

And would host-python3 be needed to build host-micropython? Or can we
get along with just the system python3? If we can use the system pytho,
then we can make the dependency on BR2_PYTHON3_HOST_DEPENDENCY rather
than hard-code host-python3.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/micropython: make use of recent micropython-lib

[Abilio Marques]

[-- Attachment #1.1: Type: text/plain, Size: 3672 bytes --]

Hi Yann, Chris,

The micropython project does not attempt to perform any kind of
bootstrapping during its build process. It relies on cpython for building,
testing, and also for host tools (mostly used with microcontrollers). So
for building host-micropython, we would still need cpython. For example,
the tools used for processing manifest files (tools/makemanifest.py,
tools/manifestfile.py) are definitely not micropython compatible, at least
not without the libraries I'm actually trying to collect ;). Another hint,
the scripts are crowned with a python3 shebang.

If we want to do this using micropython code, it also requires
re-implementing a lot of the functionality I'm currently borrowing from
manifestfile.py . In my mind, that also means a lot more chances of
requiring maintenance it in case the upstream changes the way they do
things.

As far as I know, to generate the mpy files, only 'mpy-cross' is needed.
During the target build process, this is built first (but runs on host). It
will be used to freeze modules within the micropython binary (currently
asyncio is one example). Internally, it makes use of the same
"manifestfile.py" I'm borrowing from. Creating a manifest is the route I
plan to use to add support for freezing packages within the binary. The
problem is, you still need to collect the list of all the packages
available in the micropython-lib repo.

To avoid the import trick, I could also:
- Add the path to micropython/tools into the PYTHONPATH at the time of
calling the script.
- Copy my script into the tools directory and run it from there during
build.

Base on all of the above, I fail to see positive benefits of writing this
script so it runs in micropython. Could you please indicate what advantages
do you see?

Thanks for all your comments, they make for great improvements, so I will
update the patch. This is my first time using git send-mail, and I wouldn't
know how to make the updated version part of this chain. Do you have any
suggestions?

Kind regards,
Abilio Marques


On Sun, Jan 28, 2024 at 12:09 PM Yann E. MORIN <yann.morin.1998@free.fr>
wrote:

> Chris, All,
>
> On 2024-01-29 07:58 +1300, Chris Packham spake thusly:
> > On Mon, 29 Jan 2024, 7:30 am Abilio Marques, <[1]abiliojr@gmail.com>
> wrote:
> [--SNIP--]
> >     This commit also ensures that the libraries in micropython-lib will
> >     be updated together with newer versions of micropython.
> [--SNIP--]
> > One thought I had was to build micropython as a host package and use
> that to
> > install the libs into the target rootfs.
>
> Yes, this is also my opinion: use host-micropython, if possible.
>
> I see that micropython already depends on host-python3, though. Would
> host-micropython be sufficient in lieu of host-python3, to build target
> micropython?
>
> And would host-python3 be needed to build host-micropython? Or can we
> get along with just the system python3? If we can use the system pytho,
> then we can make the dependency on BR2_PYTHON3_HOST_DEPENDENCY rather
> than hard-code host-python3.
>
> Regards,
> Yann E. MORIN.
>
> --
>
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics'
> conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___
>      |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is
> no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v
>  conspiracy.  |
>
> '------------------------------^-------^------------------^--------------------'
>

[-- Attachment #1.2: Type: text/html, Size: 5603 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/micropython: make use of recent micropython-lib

[Yann E. MORIN]

Abilio, All,

On 2024-01-28 12:56 -0800, Abilio Marques spake thusly:
> The micropython project does not attempt to perform any kind of bootstrapping
> during its build process. It relies on cpython for building, testing, and also
> for host tools (mostly used with microcontrollers). So for building
> host-micropython, we would still need cpython. For example, the tools used for
> processing manifest files (tools/makemanifest.py, tools/manifestfile.py) are
> definitely not micropython compatible, at least not without the libraries I'm
> actually trying to collect ;). Another hint, the scripts are crowned with a
> python3 shebang.

So we can't use host-micropython to build micropython; that's OK. Let's
forget about that idea, then.

> If we want to do this using micropython code, it also requires re-implementing
> a lot of the functionality I'm currently borrowing from manifestfile.py . In my
> mind, that also means a lot more chances of requiring maintenance it in case
> the upstream changes the way they do things.

Yes, that was just a long shot. Let's drop the idea of using micropython
to build itself, then. Thanks for the feedback.

> As far as I know, to generate the mpy files, only 'mpy-cross' is needed. During
> the target build process, this is built first (but runs on host). It will be
> used to freeze modules within the micropython binary (currently asyncio is one
> example). Internally, it makes use of the same "manifestfile.py" I'm borrowing
> from. Creating a manifest is the route I plan to use to add support for
> freezing packages within the binary. The problem is, you still need to collect
> the list of all the packages available in the micropython-lib repo.
> 
> To avoid the import trick, I could also:
> - Add the path to micropython/tools into the PYTHONPATH at the time of calling
> the script.
> - Copy my script into the tools directory and run it from there during build.
> 
> Base on all of the above, I fail to see positive benefits of writing this
> script so it runs in micropython. Could you please indicate what advantages do
> you see?

If there was host-micropython installed, then I would expect you could
write your script something along the lines of;

    #!/usr/bin/env micropython

    import argparse
    import manifestfile  # Import automatically rather than manually
    [...]

    def main():
        # Here you don't need to inmport it, as it's imported
        # by the python interpreter above.
        manifest = manifestfile.ManifestFile(yadyada)

        [...]

This is also a classic eat-your-own-dog-food situation, where using
one's own tool to manage one's tool, is better than using a third-party
tool. ;-)

Of course, I'd accept the fact that this is not possible if micropython
is not up to the task.

> Thanks for all your comments, they make for great improvements, so I will
> update the patch. This is my first time using git send-mail, and I wouldn't
> know how to make the updated version part of this chain. Do you have any
> suggestions?

I personally prefer a new iteration not be threaded to the previous one,
but be sure to label it 'v2'. Otherwise, you should be able to use
--in-reply-to=MESSAGE_ID.

Regards,
Yann E. MORIN.

> Kind regards,
> Abilio Marques
> 
> 
> On Sun, Jan 28, 2024 at 12:09 PM Yann E. MORIN <[1]yann.morin.1998@free.fr>
> wrote:
> 
>     Chris, All,
> 
>     On 2024-01-29 07:58 +1300, Chris Packham spake thusly:
>     > On Mon, 29 Jan 2024, 7:30 am Abilio Marques, <[1][2]abiliojr@gmail.com>
>     wrote:
>     [--SNIP--]
>     >     This commit also ensures that the libraries in micropython-lib will
>     >     be updated together with newer versions of micropython.
>     [--SNIP--]
>     > One thought I had was to build micropython as a host package and use that
>     to
>     > install the libs into the target rootfs.
> 
>     Yes, this is also my opinion: use host-micropython, if possible.
> 
>     I see that micropython already depends on host-python3, though. Would
>     host-micropython be sufficient in lieu of host-python3, to build target
>     micropython?
> 
>     And would host-python3 be needed to build host-micropython? Or can we
>     get along with just the system python3? If we can use the system pytho,
>     then we can make the dependency on BR2_PYTHON3_HOST_DEPENDENCY rather
>     than hard-code host-python3.
> 
>     Regards,
>     Yann E. MORIN.
> 
>     --
>     .-----------------.--------------------.------------------.--------------------.
>     |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics'
>     conspiracy: |
>     | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___           
>        |
>     | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is
>     no  |
>     | [3]http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v 
>      conspiracy.  |
>     '------------------------------^-------^------------------^
>     --------------------'
> 
> 
> References:
> 
> [1] mailto:yann.morin.1998@free.fr
> [2] mailto:abiliojr@gmail.com
> [3] http://ymorin.is-a-geek.org/

> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot


-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH] package/micropython: make use of recent micropython-lib

[Abilio Marques]

Until now, micropython-lib was a package that installed v1.9.3,
which is more than 6 years old. This was acceptable since micropython
never made any other official release of the library until v1.20.

Meanwhile, the libraries underwent a reorganization, and they are now
available in a directory structure that cannot be copied directly into
the target. This is my theory on why v1.9.3 is still present in the
current day buildroot (which comes with micropython v1.22).

This commit introduces an auxiliary script to collect those libraries
and reorder them into a structure that can then be copied into
/usr/lib/micropython. The script utilizes a module from the tools
directory of the micropython repo.

As part of the changes made by the micropython project, the libraries
are now released together with the interpreter. They are cloned as a
submodule into the lib/micropython-lib directory.

These are the 2 main reasons why the old buildroot micropython-lib
package is being removed.

With this commit, micropython-lib is installed (optionally) as part
of micropython. The original config variable name was retained for
backwards compatibility.

This commit also ensures that the libraries in micropython-lib will
be updated together with newer versions of micropython.

Signed-off-by: Abilio Marques <abiliojr@gmail.com>
---
 DEVELOPERS                                    |  1 -
 package/Config.in                             |  1 -
 package/micropython-lib/Config.in             |  8 --
 package/micropython-lib/micropython-lib.hash  |  3 -
 package/micropython-lib/micropython-lib.mk    | 18 ----
 package/micropython/Config.in                 | 10 +++
 .../micropython/collect_micropython_lib.py    | 90 +++++++++++++++++++
 package/micropython/micropython.mk            | 12 +++
 8 files changed, 112 insertions(+), 31 deletions(-)
 delete mode 100644 package/micropython-lib/Config.in
 delete mode 100644 package/micropython-lib/micropython-lib.hash
 delete mode 100644 package/micropython-lib/micropython-lib.mk
 create mode 100755 package/micropython/collect_micropython_lib.py

diff --git a/DEVELOPERS b/DEVELOPERS
index 9528837dd0..cc9bc5b5cc 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -572,7 +572,6 @@ F:	package/coremark/
 F:	package/coremark-pro/
 F:	package/gstreamer1/gst1-shark/
 F:	package/micropython/
-F:	package/micropython-lib/
 F:	package/syslog-ng/
 
 N:	Christian Kellermann <christian.kellermann@solectrix.de>
diff --git a/package/Config.in b/package/Config.in
index 5b8b15fa54..426bd7d090 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -798,7 +798,6 @@ menu "Lua libraries/modules"
 endmenu
 endif
 	source "package/micropython/Config.in"
-	source "package/micropython-lib/Config.in"
 	source "package/moarvm/Config.in"
 	source "package/mono/Config.in"
 if BR2_PACKAGE_MONO
diff --git a/package/micropython-lib/Config.in b/package/micropython-lib/Config.in
deleted file mode 100644
index 76557b220b..0000000000
--- a/package/micropython-lib/Config.in
+++ /dev/null
@@ -1,8 +0,0 @@
-config BR2_PACKAGE_MICROPYTHON_LIB
-	bool "micropython-lib"
-	depends on BR2_PACKAGE_MICROPYTHON
-	select BR2_PACKAGE_PCRE # runtime
-	help
-	  Core Python libraries ported to MicroPython.
-
-	  http://micropython.org
diff --git a/package/micropython-lib/micropython-lib.hash b/package/micropython-lib/micropython-lib.hash
deleted file mode 100644
index cbdda23844..0000000000
--- a/package/micropython-lib/micropython-lib.hash
+++ /dev/null
@@ -1,3 +0,0 @@
-# Locally computed
-sha256  66e15380eb109613263beb6825b8eecb9191088270c1a59e8c7d922dd57183c7  micropython-lib-1.9.3.tar.gz
-sha256  baed4196a4310c576c2010f0a49f987a49e63856df7cd45af11cb3571df4bf74  LICENSE
diff --git a/package/micropython-lib/micropython-lib.mk b/package/micropython-lib/micropython-lib.mk
deleted file mode 100644
index 78ac0d3b35..0000000000
--- a/package/micropython-lib/micropython-lib.mk
+++ /dev/null
@@ -1,18 +0,0 @@
-################################################################################
-#
-# micropython-lib
-#
-################################################################################
-
-MICROPYTHON_LIB_VERSION = 1.9.3
-MICROPYTHON_LIB_SITE = $(call github,micropython,micropython-lib,v$(MICROPYTHON_LIB_VERSION))
-MICROPYTHON_LIB_LICENSE = Python-2.0 (some modules), MIT (everything else)
-MICROPYTHON_LIB_LICENSE_FILES = LICENSE
-
-define MICROPYTHON_LIB_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) \
-		PREFIX=$(TARGET_DIR)/usr/lib/micropython \
-		install
-endef
-
-$(eval $(generic-package))
diff --git a/package/micropython/Config.in b/package/micropython/Config.in
index 30161c8b70..2c2ae7a4df 100644
--- a/package/micropython/Config.in
+++ b/package/micropython/Config.in
@@ -9,5 +9,15 @@ config BR2_PACKAGE_MICROPYTHON
 
 	  http://micropython.org
 
+config BR2_PACKAGE_MICROPYTHON_LIB
+	bool "micropython-lib"
+	default y
+	depends on BR2_PACKAGE_MICROPYTHON
+	select BR2_PACKAGE_PCRE # runtime
+	help
+	  Core Python libraries ported to MicroPython.
+
+	  http://micropython.org
+
 comment "micropython needs a toolchain w/ threads, dynamic library"
 	depends on !BR2_TOOLCHAIN_HAS_THREADS || BR2_STATIC_LIBS
diff --git a/package/micropython/collect_micropython_lib.py b/package/micropython/collect_micropython_lib.py
new file mode 100755
index 0000000000..778a1c4c7d
--- /dev/null
+++ b/package/micropython/collect_micropython_lib.py
@@ -0,0 +1,90 @@
+#!/usr/bin/env python3
+
+"""
+Sometime after v1.9.3, micropython-lib's directory structure was cleaned up and
+enhanced with manifest files.
+
+Though cleaner, it means it cannot be directly copied into the target.
+This script is during build process to perform that conversion.
+
+It makes use of manifestfile.py, which is normally located in micropython's
+tool directory.
+
+It also depends on the micropython-lib that is cloned in lib/micropython-lib
+during build.
+"""
+
+import argparse
+import importlib
+import os
+import shutil
+import sys
+
+
+def get_library_name_type(manifest_path: str) -> tuple[str, bool]:
+    split = manifest_path.split("/")
+    return (split[-2], "unix-ffi" in split)  # -1: "manifest.py", -2: library name
+
+
+def get_all_libraries(mpy_lib_dir: str):
+    # reuse manifestfile module capabilities to scan the micropython-lib directory
+
+    collected_list = manifestfile.ManifestFile(
+        manifestfile.MODE_FREEZE, {"MPY_LIB_DIR": mpy_lib_dir}
+    )
+    collected_list.freeze(mpy_lib_dir)
+
+    for file in collected_list.files():
+        if file.target_path.endswith("manifest.py"):
+            yield get_library_name_type(file.full_path)
+
+
+def copy_file(src: str, target: str, destination: str):
+    s = target.split("/")
+    s.pop()
+    d = f"{destination}/{'/'.join(s)}"
+    if not os.path.exists(d):
+        os.makedirs(d)
+    shutil.copy(src, f"{destination}/{target}")
+
+
+def copy_libraries(manifest, destination: str):
+    for f in manifest.files():
+        copy_file(f.full_path, f.target_path, destination)
+
+
+def process_cmdline_args():
+    parser = argparse.ArgumentParser(
+        description="Prepare micropython-lib to be installed"
+    )
+    parser.add_argument("micropython", help="Path to micropython source directory")
+    parser.add_argument("destination", help="Destination directory")
+
+    args = parser.parse_args()
+    return os.path.abspath(args.micropython), os.path.abspath(args.destination)
+
+
+def load_manifestfile_module(micropython_dir: str):
+    global manifestfile
+    sys.path.append(f"{micropython_dir}/tools")
+    manifestfile = importlib.import_module("manifestfile")
+
+
+def main():
+    micropython_dir, destination_dir = process_cmdline_args()
+    mpy_lib_dir = f"{micropython_dir}/lib/micropython-lib"
+
+    # load manifest file after knowing where it is
+    load_manifestfile_module(micropython_dir)
+
+    manifest = manifestfile.ManifestFile(
+        manifestfile.MODE_FREEZE, {"MPY_LIB_DIR": mpy_lib_dir}
+    )
+
+    for library, is_ffi in get_all_libraries(mpy_lib_dir):
+        manifest.require(library, unix_ffi=is_ffi)
+
+    copy_libraries(manifest, destination_dir)
+
+
+main()
diff --git a/package/micropython/micropython.mk b/package/micropython/micropython.mk
index 37c148da94..a9757e266c 100644
--- a/package/micropython/micropython.mk
+++ b/package/micropython/micropython.mk
@@ -47,6 +47,8 @@ define MICROPYTHON_BUILD_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \
 		$(MICROPYTHON_MAKE_OPTS)
+	$(EXTRA_ENV) package/micropython/collect_micropython_lib.py \
+		$(@D) $(@D)/.built_pylib
 endef
 
 define MICROPYTHON_INSTALL_TARGET_CMDS
@@ -57,4 +59,14 @@ define MICROPYTHON_INSTALL_TARGET_CMDS
 		install
 endef
 
+define MICROPYTHON_INSTALL_LIBS
+	$(INSTALL) -d -m 0755 $(TARGET_DIR)/usr/lib/micropython
+	cp -a $(@D)/.built_pylib/* $(TARGET_DIR)/usr/lib/micropython
+endef
+
+ifeq ($(BR2_PACKAGE_MICROPYTHON_LIB),y)
+MICROPYTHON_POST_INSTALL_TARGET_HOOKS += MICROPYTHON_INSTALL_LIBS
+endif
+
+
 $(eval $(generic-package))
-- 
2.43.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/micropython: make use of recent micropython-lib

[Yann E. MORIN]

Abilio, All,

On 2024-01-28 10:37 -0800, Abilio Marques spake thusly:
> Until now, micropython-lib was a package that installed v1.9.3,
> which is more than 6 years old. This was acceptable since micropython
> never made any other official release of the library until v1.20.
> 
> Meanwhile, the libraries underwent a reorganization, and they are now
> available in a directory structure that cannot be copied directly into
> the target. This is my theory on why v1.9.3 is still present in the
> current day buildroot (which comes with micropython v1.22).
> 
> This commit introduces an auxiliary script to collect those libraries
> and reorder them into a structure that can then be copied into
> /usr/lib/micropython. The script utilizes a module from the tools
> directory of the micropython repo.
> 
> As part of the changes made by the micropython project, the libraries
> are now released together with the interpreter. They are cloned as a
> submodule into the lib/micropython-lib directory.

Thank you for the detailed explanations! 👍

> These are the 2 main reasons why the old buildroot micropython-lib
> package is being removed.
> 
> With this commit, micropython-lib is installed (optionally) as part
> of micropython. The original config variable name was retained for
> backwards compatibility.

Minor mitpick about the phrasing:

    ... The original config variable name was retained as it fits with
    the micropython package 'namespace", and thus this is backward
    compatible and no legacy handling is needed.

> This commit also ensures that the libraries in micropython-lib will
> be updated together with newer versions of micropython.
> 
> Signed-off-by: Abilio Marques <abiliojr@gmail.com>

I think the idea of this commit is very interesting, and the commit is
already quite good; still, I have a few comments and questions, see
below.

[--SNIP--]
> diff --git a/package/micropython/Config.in b/package/micropython/Config.in
> index 30161c8b70..2c2ae7a4df 100644
> --- a/package/micropython/Config.in
> +++ b/package/micropython/Config.in
> @@ -9,5 +9,15 @@ config BR2_PACKAGE_MICROPYTHON
>  
>  	  http://micropython.org
>  
> +config BR2_PACKAGE_MICROPYTHON_LIB
> +	bool "micropython-lib"
> +	default y
> +	depends on BR2_PACKAGE_MICROPYTHON
> +	select BR2_PACKAGE_PCRE # runtime
> +	help
> +	  Core Python libraries ported to MicroPython.
> +
> +	  http://micropython.org

Minor nitpick: no need to duplicate the homepage here.

[--SNIP--]
> diff --git a/package/micropython/collect_micropython_lib.py b/package/micropython/collect_micropython_lib.py
> new file mode 100755
> index 0000000000..778a1c4c7d
> --- /dev/null
> +++ b/package/micropython/collect_micropython_lib.py
> @@ -0,0 +1,90 @@
> +#!/usr/bin/env python3

As Chris hinted,maybe this should be a micropython script, so that we do
not need to play tricks with the module loading.

Note that I have not entirely reviewed this script for now; this is a
quick-ish review, there might be some mor ecomments on it later...

[--SNIP--]
> +def get_library_name_type(manifest_path: str) -> tuple[str, bool]:
> +    split = manifest_path.split("/")
> +    return (split[-2], "unix-ffi" in split)  # -1: "manifest.py", -2: library name
> +
> +
> +def get_all_libraries(mpy_lib_dir: str):
> +    # reuse manifestfile module capabilities to scan the micropython-lib directory
> +
> +    collected_list = manifestfile.ManifestFile(
> +        manifestfile.MODE_FREEZE, {"MPY_LIB_DIR": mpy_lib_dir}
> +    )
> +    collected_list.freeze(mpy_lib_dir)
> +
> +    for file in collected_list.files():
> +        if file.target_path.endswith("manifest.py"):
> +            yield get_library_name_type(file.full_path)
> +
> +
> +def copy_file(src: str, target: str, destination: str):
> +    s = target.split("/")
> +    s.pop()
> +    d = f"{destination}/{'/'.join(s)}"
> +    if not os.path.exists(d):
> +        os.makedirs(d)

No need to test before creating; just use exist_ok=False;

    os.makedirs(d, exist_ok=False)

> +    shutil.copy(src, f"{destination}/{target}")
> +
> +
> +def copy_libraries(manifest, destination: str):
> +    for f in manifest.files():
> +        copy_file(f.full_path, f.target_path, destination)
> +
> +
> +def process_cmdline_args():
> +    parser = argparse.ArgumentParser(
> +        description="Prepare micropython-lib to be installed"
> +    )
> +    parser.add_argument("micropython", help="Path to micropython source directory")
> +    parser.add_argument("destination", help="Destination directory")
> +
> +    args = parser.parse_args()
> +    return os.path.abspath(args.micropython), os.path.abspath(args.destination)
> +
> +
> +def load_manifestfile_module(micropython_dir: str):
> +    global manifestfile
> +    sys.path.append(f"{micropython_dir}/tools")
> +    manifestfile = importlib.import_module("manifestfile")

I don't think there is a need to set a global variable. Just return the
module you found. Except that, if this script becomes a micropython
script, then the loading could be done with a mere 'import' anyway.

> +def main():
> +    micropython_dir, destination_dir = process_cmdline_args()
> +    mpy_lib_dir = f"{micropython_dir}/lib/micropython-lib"
> +
> +    # load manifest file after knowing where it is
> +    load_manifestfile_module(micropython_dir)
> +
> +    manifest = manifestfile.ManifestFile(
> +        manifestfile.MODE_FREEZE, {"MPY_LIB_DIR": mpy_lib_dir}
> +    )
> +
> +    for library, is_ffi in get_all_libraries(mpy_lib_dir):
> +        manifest.require(library, unix_ffi=is_ffi)
> +
> +    copy_libraries(manifest, destination_dir)
> +
> +
> +main()
> diff --git a/package/micropython/micropython.mk b/package/micropython/micropython.mk
> index 37c148da94..a9757e266c 100644
> --- a/package/micropython/micropython.mk
> +++ b/package/micropython/micropython.mk
> @@ -47,6 +47,8 @@ define MICROPYTHON_BUILD_CMDS
>  	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross
>  	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \
>  		$(MICROPYTHON_MAKE_OPTS)
> +	$(EXTRA_ENV) package/micropython/collect_micropython_lib.py \
> +		$(@D) $(@D)/.built_pylib

No need to collect the libraries if BR2_PACKAGE_MICROPYTHON_LIB is not
set (see below).

>  endef
>  
>  define MICROPYTHON_INSTALL_TARGET_CMDS
> @@ -57,4 +59,14 @@ define MICROPYTHON_INSTALL_TARGET_CMDS
>  		install
>  endef
>  
> +define MICROPYTHON_INSTALL_LIBS
> +	$(INSTALL) -d -m 0755 $(TARGET_DIR)/usr/lib/micropython
> +	cp -a $(@D)/.built_pylib/* $(TARGET_DIR)/usr/lib/micropython
> +endef
> +
> +ifeq ($(BR2_PACKAGE_MICROPYTHON_LIB),y)
> +MICROPYTHON_POST_INSTALL_TARGET_HOOKS += MICROPYTHON_INSTALL_LIBS
> +endif

We define the hook and assign it in the if-block:

    ifeq ($(BR2_PACKAGE_MICROPYTHON_LIB),y)
    define MICROPYTHON_COLLECT_LIBS
        ...
    endef
    define MICROPYTHON_INSTALL_LIBS
        ...
    endef
    MICROPYTHON_POST_BUILD_HOOKS += MICROPYTHON_COLLECT_LIBS
    MICROPYTHON_POST_INSTALL_TARGET_HOOKS += MICROPYTHON_INSTALL_LIBS
    endif

> +
> +
>  $(eval $(generic-package))

So, as Chris suggested, maybe we want to build this as a host package,
so that we can use micropython at build time, for example to generate
the mpy files, or to use it as interpreter to the lib-install script.

Thank you!

Regards,
Yann E. MORIN.

> -- 
> 2.43.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot