[Buildroot] [PATCH] package/micropython: add support for manifest.py in the configuration

[Buildroot] [PATCH] package/micropython: add support for manifest.py in the configuration

[Abilio Marques]

Micropython can embed packages an modules as frozen bytecode. What code
gets built this way can be defined by means of a "manifest.py" file.

This commit exposes the variable FROZEN_MANIFEST to Buildroot users
through a new variable called BR2_PACKAGE_MICROPYTHON_MANIFEST.

Please check Micropython's documentation for more information:
https://docs.micropython.org/en/latest/reference/manifest.html

Signed-off-by: Abilio Marques <abiliojr@gmail.com>
---
 package/micropython/Config.in      | 8 ++++++++
 package/micropython/micropython.mk | 5 +++++
 2 files changed, 13 insertions(+)

diff --git a/package/micropython/Config.in b/package/micropython/Config.in
index 26a00baab0..52717d815a 100644
--- a/package/micropython/Config.in
+++ b/package/micropython/Config.in
@@ -17,6 +17,14 @@ config BR2_PACKAGE_MICROPYTHON_LIB
 	help
 	  Core Python libraries ported to MicroPython.
 
+config BR2_PACKAGE_MICROPYTHON_MANIFEST
+	string "Path to a manifest.py file"
+	help
+	  MicroPython allows Python code to be “frozen” as bytecode
+	  into its binary, as an alternative to loading code from
+	  the filesystem. See MicroPython's documentation for more
+	  information.
+
 endif # BR2_PACKAGE_MICROPYTHON
 
 comment "micropython needs a toolchain w/ threads, dynamic library"
diff --git a/package/micropython/micropython.mk b/package/micropython/micropython.mk
index 125a0edcfb..5a2c136547 100644
--- a/package/micropython/micropython.mk
+++ b/package/micropython/micropython.mk
@@ -43,6 +43,11 @@ else
 MICROPYTHON_MAKE_OPTS += MICROPY_PY_FFI=0
 endif
 
+ifneq ($(BR2_PACKAGE_MICROPYTHON_MANIFEST),"")
+MICROPYTHON_MAKE_OPTS += \
+	FROZEN_MANIFEST=$(BR2_PACKAGE_MICROPYTHON_MANIFEST)
+endif
+
 define MICROPYTHON_BUILD_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \
-- 
2.43.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/micropython: add support for manifest.py in the configuration

[Yann E. MORIN]

Abilio, All,

On 2024-02-03 22:26 -0800, Abilio Marques spake thusly:
> Micropython can embed packages an modules as frozen bytecode. What code
> gets built this way can be defined by means of a "manifest.py" file.
> 
> This commit exposes the variable FROZEN_MANIFEST to Buildroot users
> through a new variable called BR2_PACKAGE_MICROPYTHON_MANIFEST.
> 
> Please check Micropython's documentation for more information:
> https://docs.micropython.org/en/latest/reference/manifest.html

Minor nit: this blurb should have been in the help text of the new
option, and I could have done that when applying, but I have a much
more involved question, see below.

> Signed-off-by: Abilio Marques <abiliojr@gmail.com>
> ---
>  package/micropython/Config.in      | 8 ++++++++
>  package/micropython/micropython.mk | 5 +++++
>  2 files changed, 13 insertions(+)
> 
> diff --git a/package/micropython/Config.in b/package/micropython/Config.in
> index 26a00baab0..52717d815a 100644
> --- a/package/micropython/Config.in
> +++ b/package/micropython/Config.in
> @@ -17,6 +17,14 @@ config BR2_PACKAGE_MICROPYTHON_LIB
>  	help
>  	  Core Python libraries ported to MicroPython.
>  
> +config BR2_PACKAGE_MICROPYTHON_MANIFEST
> +	string "Path to a manifest.py file"
> +	help
> +	  MicroPython allows Python code to be “frozen” as bytecode
> +	  into its binary, as an alternative to loading code from
> +	  the filesystem. See MicroPython's documentation for more
> +	  information.
> +
>  endif # BR2_PACKAGE_MICROPYTHON
>  
>  comment "micropython needs a toolchain w/ threads, dynamic library"
> diff --git a/package/micropython/micropython.mk b/package/micropython/micropython.mk
> index 125a0edcfb..5a2c136547 100644
> --- a/package/micropython/micropython.mk
> +++ b/package/micropython/micropython.mk
> @@ -43,6 +43,11 @@ else
>  MICROPYTHON_MAKE_OPTS += MICROPY_PY_FFI=0
>  endif
>  
> +ifneq ($(BR2_PACKAGE_MICROPYTHON_MANIFEST),"")
> +MICROPYTHON_MAKE_OPTS += \
> +	FROZEN_MANIFEST=$(BR2_PACKAGE_MICROPYTHON_MANIFEST)
> +endif

So, as I understand this, micropython will grab the files (from modules,
files, etc...) listed from the manifest and bundle them in the
micropython executable. It seems the paths in the manifest can be either
absolute, or relative; in the latter case, it not documented what they
would be relative to; additionally there are placeholders (see below)
that can be used but are absolute paths.

So this has a few implications:

  - the files must be available before micropython is built, so some
    dependency order is needed to ensure that: if files are provided in
    a package, then we need a dependency and it is going to be a little
    bit difficult to do and we need a way to address that;

  - the paths can't be hard-coded to absolute paths in the manifest file
    itself, because we don't know beforehand where the buildroot build
    directory will be, so we need a way to shoehorn $(TOPDIR) or some
    such variable in the manifest;

  - absolute paths must be fixed to accommodate the per-package option,
    because those paths change for eaach package.

From the micropython doc, that you pointed above, there are a few
placeholders that will be replaced in the manifest, but I don't think
we can reuse those:

    Any paths used in manifest files can include the following variables.
    These all resolve to absolute paths.
      * $(MPY_DIR) – path to the micropython repo.
      * $(MPY_LIB_DIR) – path to the micropython-lib submodule. Prefer to use require().
      * $(PORT_DIR) – path to the current port (e.g. ports/stm32)
      * $(BOARD_DIR) – path to the current board (e.g.  ports/stm32/boards/PYBV11)

So we need a bit more information how this feature is to be used, and an
example (or a few) would probably help a lot in this respect. We also
have a runtime test for micropython, in:
    support/testing/tests/package/test_micropython.py

It would be nice to extend that runtime test as well.

Regards,
Yann E. MORIN.

>  define MICROPYTHON_BUILD_CMDS
>  	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross
>  	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \
> -- 
> 2.43.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/micropython: add support for manifest.py in the configuration

[Abilio Marques]

[-- Attachment #1.1: Type: text/plain, Size: 6451 bytes --]

Hello Yann,

One of the uses of manifest.py is to specify which modules of
micropython-lib should be frozen within the binary.  For those modules you
don't need to include the path. e.g.,

package('os')

That's the application I'm going for. I know it seems limited but it's
really useful, and probably welcomed by other people using Micropython in
the next release of Buildroot.

I have ideas on how to solve the path and dependency order problem while
still using the "official" manifest.py concept. The biggest challenge is
that currently there are no other 3rd party Micropython modules available
for Buildroot, so that makes it all very theoretical. All solutions require
a bigger amount of work than the one needed for this patch. Also, I would
like to discuss them before actually presenting a patch that allows 3rd
party modules to be frozen.

I always try to go for an incremental approach, where I get the bigger bang
for the buck. I believe that allowing people to freeze the official
Micropython modules is already a big step forward. But at the same time,
I'm new to the Buildroot project, so please advice on the approach.


Kind regards,
Abilio


On Mon, Feb 5, 2024 at 5:24 AM Yann E. MORIN <yann.morin.1998@free.fr>
wrote:

> Abilio, All,
>
> On 2024-02-03 22:26 -0800, Abilio Marques spake thusly:
> > Micropython can embed packages an modules as frozen bytecode. What code
> > gets built this way can be defined by means of a "manifest.py" file.
> >
> > This commit exposes the variable FROZEN_MANIFEST to Buildroot users
> > through a new variable called BR2_PACKAGE_MICROPYTHON_MANIFEST.
> >
> > Please check Micropython's documentation for more information:
> > https://docs.micropython.org/en/latest/reference/manifest.html
>
> Minor nit: this blurb should have been in the help text of the new
> option, and I could have done that when applying, but I have a much
> more involved question, see below.
>
> > Signed-off-by: Abilio Marques <abiliojr@gmail.com>
> > ---
> >  package/micropython/Config.in      | 8 ++++++++
> >  package/micropython/micropython.mk | 5 +++++
> >  2 files changed, 13 insertions(+)
> >
> > diff --git a/package/micropython/Config.in
> b/package/micropython/Config.in
> > index 26a00baab0..52717d815a 100644
> > --- a/package/micropython/Config.in
> > +++ b/package/micropython/Config.in
> > @@ -17,6 +17,14 @@ config BR2_PACKAGE_MICROPYTHON_LIB
> >       help
> >         Core Python libraries ported to MicroPython.
> >
> > +config BR2_PACKAGE_MICROPYTHON_MANIFEST
> > +     string "Path to a manifest.py file"
> > +     help
> > +       MicroPython allows Python code to be “frozen” as bytecode
> > +       into its binary, as an alternative to loading code from
> > +       the filesystem. See MicroPython's documentation for more
> > +       information.
> > +
> >  endif # BR2_PACKAGE_MICROPYTHON
> >
> >  comment "micropython needs a toolchain w/ threads, dynamic library"
> > diff --git a/package/micropython/micropython.mk b/package/micropython/
> micropython.mk
> > index 125a0edcfb..5a2c136547 100644
> > --- a/package/micropython/micropython.mk
> > +++ b/package/micropython/micropython.mk
> > @@ -43,6 +43,11 @@ else
> >  MICROPYTHON_MAKE_OPTS += MICROPY_PY_FFI=0
> >  endif
> >
> > +ifneq ($(BR2_PACKAGE_MICROPYTHON_MANIFEST),"")
> > +MICROPYTHON_MAKE_OPTS += \
> > +     FROZEN_MANIFEST=$(BR2_PACKAGE_MICROPYTHON_MANIFEST)
> > +endif
>
> So, as I understand this, micropython will grab the files (from modules,
> files, etc...) listed from the manifest and bundle them in the
> micropython executable. It seems the paths in the manifest can be either
> absolute, or relative; in the latter case, it not documented what they
> would be relative to; additionally there are placeholders (see below)
> that can be used but are absolute paths.
>
> So this has a few implications:
>
>   - the files must be available before micropython is built, so some
>     dependency order is needed to ensure that: if files are provided in
>     a package, then we need a dependency and it is going to be a little
>     bit difficult to do and we need a way to address that;
>
>   - the paths can't be hard-coded to absolute paths in the manifest file
>     itself, because we don't know beforehand where the buildroot build
>     directory will be, so we need a way to shoehorn $(TOPDIR) or some
>     such variable in the manifest;
>
>   - absolute paths must be fixed to accommodate the per-package option,
>     because those paths change for eaach package.
>
> From the micropython doc, that you pointed above, there are a few
> placeholders that will be replaced in the manifest, but I don't think
> we can reuse those:
>
>     Any paths used in manifest files can include the following variables.
>     These all resolve to absolute paths.
>       * $(MPY_DIR) – path to the micropython repo.
>       * $(MPY_LIB_DIR) – path to the micropython-lib submodule. Prefer to
> use require().
>       * $(PORT_DIR) – path to the current port (e.g. ports/stm32)
>       * $(BOARD_DIR) – path to the current board (e.g.
> ports/stm32/boards/PYBV11)
>
> So we need a bit more information how this feature is to be used, and an
> example (or a few) would probably help a lot in this respect. We also
> have a runtime test for micropython, in:
>     support/testing/tests/package/test_micropython.py
>
> It would be nice to extend that runtime test as well.
>
> Regards,
> Yann E. MORIN.
>
> >  define MICROPYTHON_BUILD_CMDS
> >       $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross
> >       $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \
> > --
> > 2.43.0
> >
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
>
> --
>
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics'
> conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___
>      |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is
> no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v
>  conspiracy.  |
>
> '------------------------------^-------^------------------^--------------------'
>

[-- Attachment #1.2: Type: text/html, Size: 9311 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/micropython: add support for manifest.py in the configuration

[Yann E. MORIN]

Abilio, All,

[Please, don't top-post, but reply in-line]

On 2024-02-11 09:15 -0800, Abilio Marques spake thusly:
> One of the uses of manifest.py is to specify which modules of
> micropython-lib should be frozen within the binary.  For those modules you
> don't need to include the path. e.g.,
> package('os')
> 
> That's the application I'm going for. I know it seems limited but it's
> really useful,

OK, so maybe this can be explained in the help text of the new option,
like:

    Note: in Buildroot, only modules provided with micropython-lib
    can be frozen with a manifest; freezing arbitrary files is not
    supported.

along with a little blurb in the commit log, statng something like;

    We only support freezing of modules from micropython-lib. Freezing
    arbitrary modules would require some handling of absoulte paths, and
    a guarantee that the files be available before micropython is built,
    which is considered a bit too complex; an interested party can
    provide such support in the future.

And thus, would it be possible to sanity-check the manifest to ensure
that it indeed only references modules from icropython-lib, so that
people do not get the impression it works when in fact it does not?

Like, grep that only require(...) are used?

(Note: you mentioned package(...) but that is not limited to files from
micropython-lib; require(...) however is.)

> and probably welcomed by other people using Micropython in
> the next release of Buildroot.

One question I was wondering about: if a module is frozen in the binary,
then it is no longer needed in the filesystem, right? If so, should we
have a way to remove them?

> I have ideas on how to solve the path and dependency order problem while
> still using the "official" manifest.py concept. The biggest challenge is
> that currently there are no other 3rd party Micropython modules available
> for Buildroot, so that makes it all very theoretical. All solutions require
> a bigger amount of work than the one needed for this patch. Also, I would
> like to discuss them before actually presenting a patch that allows 3rd
> party modules to be frozen.

Indeed, without a few actual examples, it's going to be difficult to see
a common pattern and abstract that away. Are there any pulicly
available?

> I always try to go for an incremental approach, where I get the bigger bang
> for the buck. I believe that allowing people to freeze the official
> Micropython modules is already a big step forward. But at the same time,
> I'm new to the Buildroot project, so please advice on the approach.

The incremental path is totally OK; I even prefer it. Of course, any
limitation (such as those we are discussing) should be explained in the
commit log.

Given all the feedback in this thread, can you respin a v2 taking the
comments into account?

As for your ideas for lifting those limitations, you can just explain
them in a reply in this thread. Usually, a patchset doing the job is
also a good first step to start the dicsussion.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/micropython: add support for manifest.py in the configuration

[Abilio Marques]

On Sun, Feb 11, 2024 at 9:54 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Abilio, All,
>
> [Please, don't top-post, but reply in-line]
>
> On 2024-02-11 09:15 -0800, Abilio Marques spake thusly:
> > One of the uses of manifest.py is to specify which modules of
> > micropython-lib should be frozen within the binary.  For those modules you
> > don't need to include the path. e.g.,
> > package('os')
> >
> > That's the application I'm going for. I know it seems limited but it's
> > really useful,
>
> OK, so maybe this can be explained in the help text of the new option,
> like:
>
>     Note: in Buildroot, only modules provided with micropython-lib
>     can be frozen with a manifest; freezing arbitrary files is not
>     supported.
>
> along with a little blurb in the commit log, statng something like;
>
>     We only support freezing of modules from micropython-lib. Freezing
>     arbitrary modules would require some handling of absoulte paths, and
>     a guarantee that the files be available before micropython is built,
>     which is considered a bit too complex; an interested party can
>     provide such support in the future.
>
> And thus, would it be possible to sanity-check the manifest to ensure
> that it indeed only references modules from icropython-lib, so that
> people do not get the impression it works when in fact it does not?
>
> Like, grep that only require(...) are used?

Not sure how to do it correctly. The manifest is a regular python file, and I'm
not sure how to do it in a way that does the correct the thing. e.g.,
the user might
want to add comment lines or an include to a second part of the manifest.

>
> (Note: you mentioned package(...) but that is not limited to files from
> micropython-lib; require(...) however is.)
>
> > and probably welcomed by other people using Micropython in
> > the next release of Buildroot.
>
> One question I was wondering about: if a module is frozen in the binary,
> then it is no longer needed in the filesystem, right? If so, should we
> have a way to remove them?

Good idea. Do you mind if I add this in a separate patch?

>
> > I have ideas on how to solve the path and dependency order problem while
> > still using the "official" manifest.py concept. The biggest challenge is
> > that currently there are no other 3rd party Micropython modules available
> > for Buildroot, so that makes it all very theoretical. All solutions require
> > a bigger amount of work than the one needed for this patch. Also, I would
> > like to discuss them before actually presenting a patch that allows 3rd
> > party modules to be frozen.
>
> Indeed, without a few actual examples, it's going to be difficult to see
> a common pattern and abstract that away. Are there any pulicly
> available?
>
> > I always try to go for an incremental approach, where I get the bigger bang
> > for the buck. I believe that allowing people to freeze the official
> > Micropython modules is already a big step forward. But at the same time,
> > I'm new to the Buildroot project, so please advice on the approach.
>
> The incremental path is totally OK; I even prefer it. Of course, any
> limitation (such as those we are discussing) should be explained in the
> commit log.
>
> Given all the feedback in this thread, can you respin a v2 taking the
> comments into account?

I did, but just found out that I submitted a patch with a whitespace
problem, in this line
    "can be frozen with a manifest; freezing arbitrary files is"
Could you please correct it?

>
> As for your ideas for lifting those limitations, you can just explain
> them in a reply in this thread. Usually, a patchset doing the job is
> also a good first step to start the dicsussion.

Will try a few things and submit one.

>
> Regards,
> Yann E. MORIN.
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot