[PATCH] overview-manual: concepts: add details on package splitting

[PATCH] overview-manual: concepts: add details on package splitting

[Antonin Godard]

The package splitting section of the overview manual currently lacks any
explanation of how package splitting is implemented and redirects to
the package class, which is not really understandable for newcomers to
the project.

This patch adds a short explanation of what is done:

* How the PACKAGES variable is defined.
* How the FILES variable is defined.
* How the two work together.
* How to add a custom package.

This should give enough details to a new user on what package splitting
achieves and how to add a custom package.

Adresses [YOCTO #13225]

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
---
 documentation/overview-manual/concepts.rst | 49 +++++++++++++++++++++++++++---
 1 file changed, 44 insertions(+), 5 deletions(-)

diff --git a/documentation/overview-manual/concepts.rst b/documentation/overview-manual/concepts.rst
index 62f2327a7e27a777f738523e7fafe62422186abb..d3e90b80d4fe1954c7aacba8f0fe243cba04c1b5 100644
--- a/documentation/overview-manual/concepts.rst
+++ b/documentation/overview-manual/concepts.rst
@@ -912,11 +912,50 @@ the analysis and package splitting process use several areas:
    execute on a system and it generates code for yet another machine
    (e.g. :ref:`ref-classes-cross-canadian` recipes).
 
-The :term:`FILES` variable defines the
-files that go into each package in
-:term:`PACKAGES`. If you want
-details on how this is accomplished, you can look at
-:yocto_git:`package.bbclass </poky/tree/meta/classes-global/package.bbclass>`.
+Packages for a recipe are listed in the :term:`PACKAGES` variable. The
+:yocto_git:`meta/conf/bitbake.conf </poky/tree/meta/conf/bitbake.conf>`
+configuration file defines the following default list of packages::
+
+  PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
+
+Each of these packages contain a default list of files defined with the
+:term:`FILES` variable. For example, the package ``${PN}-dev`` represents the
+development variant of the main package ``${PN}`` and its default list of file
+contains development-oriented files only::
+
+  FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
+                  ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
+                  ${datadir}/aclocal ${base_libdir}/*.o \
+                  ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
+                  ${libdir}/cmake ${datadir}/cmake"
+
+The paths in this list must be *absolute*, and must use the path prefixes
+defined by the :yocto_git:`bitbake.conf </poky/tree/meta/conf/bitbake.conf>`
+configuration file, such as ``${sysconfdir}``, ``${bindir}``, etc. They can
+optionally use wildcards (``*`` ) to match multiple files installed in a
+directory.
+
+.. note::
+
+   The list of files for a package is defined using the override syntax by
+   separating :term:`FILES` and the package name by a semi-colon (``:``).
+
+The order of the packages in the :term:`PACKAGES` is important: during package
+splitting, going from the first element in :term:`PACKAGES` to the last, each
+file matching a pattern specified in the corresponding :term:`FILES` definition
+will be included in the package and *excluded* for the remaining packages listed
+in :term:`PACKAGES`. As a consequence, custom packages should be added using the
+prepend operator (``=+``) to be prioritized.
+
+For example, to add a custom package variant of the ``${PN}`` recipe named
+``${PN}-extra`` (name is arbitrary), one should write::
+
+  PACKAGES =+ "${PN}-extra"
+
+Alternatively, a custom package can be added to the
+:term:`PACKAGE_BEFORE_PN` variable::
+
+  PACKAGE_BEFORE_PN += "${PN}-extra"
 
 Depending on the type of packages being created (RPM, DEB, or IPK), the
 :ref:`do_package_write_* <ref-tasks-package_write_deb>`

---
base-commit: 3820a7bec0bb23bcd26bd99be2275d6d214a3a04
change-id: 20241015-bug-13225-package-split-57c8dc0f96b5

Best regards,
-- 
Antonin Godard <antonin.godard@bootlin.com>

Re: [docs] [PATCH] overview-manual: concepts: add details on package splitting

[Quentin Schulz]

Hi Antonin,

On 10/16/24 3:19 PM, Antonin Godard via lists.yoctoproject.org wrote:
> The package splitting section of the overview manual currently lacks any
> explanation of how package splitting is implemented and redirects to
> the package class, which is not really understandable for newcomers to
> the project.
> 
> This patch adds a short explanation of what is done:
> 
> * How the PACKAGES variable is defined.
> * How the FILES variable is defined.
> * How the two work together.
> * How to add a custom package.
> 
> This should give enough details to a new user on what package splitting
> achieves and how to add a custom package.
> 
> Adresses [YOCTO #13225]
> 
> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
> ---
>   documentation/overview-manual/concepts.rst | 49 +++++++++++++++++++++++++++---
>   1 file changed, 44 insertions(+), 5 deletions(-)
> 
> diff --git a/documentation/overview-manual/concepts.rst b/documentation/overview-manual/concepts.rst
> index 62f2327a7e27a777f738523e7fafe62422186abb..d3e90b80d4fe1954c7aacba8f0fe243cba04c1b5 100644
> --- a/documentation/overview-manual/concepts.rst
> +++ b/documentation/overview-manual/concepts.rst
> @@ -912,11 +912,50 @@ the analysis and package splitting process use several areas:
>      execute on a system and it generates code for yet another machine
>      (e.g. :ref:`ref-classes-cross-canadian` recipes).
>   
> -The :term:`FILES` variable defines the
> -files that go into each package in
> -:term:`PACKAGES`. If you want
> -details on how this is accomplished, you can look at
> -:yocto_git:`package.bbclass </poky/tree/meta/classes-global/package.bbclass>`.
> +Packages for a recipe are listed in the :term:`PACKAGES` variable. The
> +:yocto_git:`meta/conf/bitbake.conf </poky/tree/meta/conf/bitbake.conf>`
> +configuration file defines the following default list of packages::
> +
> +  PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
> +
> +Each of these packages contain a default list of files defined with the
> +:term:`FILES` variable. For example, the package ``${PN}-dev`` represents the
> +development variant of the main package ``${PN}`` and its default list of file

I misread the first time I read that because I thought "and its default 
list of files contains development-oriented files" applied to "${PN}" 
and not "${PN}-dev". I could suggest simply adding a coma:

"""
the package ``${PN}-dev`` represents the development variant of the main 
package ``${PN}``, and its default list
"""

Just a suggestion though, could be that it's just me reading it this way :)

s/file/files/ in any case.

Additionally, I think PN-dev does not represent the development variant 
of the main package, it represents the files that are useful for 
developing an application depending on PN main package. Using "variant" 
seems to indicate that PN-dev could be self-sufficient and potentially 
have the same binaries/shlibs than in PN but compiled differently.

> +contains development-oriented files only::
> +
> +  FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
> +                  ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
> +                  ${datadir}/aclocal ${base_libdir}/*.o \
> +                  ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
> +                  ${libdir}/cmake ${datadir}/cmake"
> +
> +The paths in this list must be *absolute*, and must use the path prefixes

There is a very important piece of information that needs to be added 
here and a source of many newcomers' mistake: these are absolute paths 
from the PoV of the root of the filesystem on the target, NOT from bitbake.

Indeed, in do_install (and do_package, etc.) we need to prefix our paths 
with $D (and $PKGD/$PKGDEST) when installing/packaging files, however, 
FILES doesn't not need those, even more it should NOT have those.

> +defined by the :yocto_git:`bitbake.conf </poky/tree/meta/conf/bitbake.conf>`

use oe_git here instead since bitbake is maintained by OE folks.

> +configuration file, such as ``${sysconfdir}``, ``${bindir}``, etc. They can
> +optionally use wildcards (``*`` ) to match multiple files installed in a
> +directory.
> +
> +.. note::
> +
> +   The list of files for a package is defined using the override syntax by
> +   separating :term:`FILES` and the package name by a semi-colon (``:``).
> +
> +The order of the packages in the :term:`PACKAGES` is important: during package
> +splitting, going from the first element in :term:`PACKAGES` to the last, each

I would say leftmost to rightmost as we have some other variables that 
go rightmost to leftmost (hello OVERRIDES :) ).

> +file matching a pattern specified in the corresponding :term:`FILES` definition
> +will be included in the package and *excluded* for the remaining packages listed
> +in :term:`PACKAGES`. As a consequence, custom packages should be added using the

I could suggest:
"""
A given file can only ever be in one package. Therefore, the first (from 
leftmost to rightmost in :term:`PACKAGES`) package for which the given 
file matches a path from its :term:`FILES` will contain the given file.
"""

Still a bit too complex of a sentence but couldn't come up with anything 
better for now.

> +prepend operator (``=+``) to be prioritized.
> +

I am not sure it is wise to say "prepend" operator as we have three ways 
of prepending:
- =+
- =.
- :prepend

But since you explicit which one you're talking about, I guess it's 
fine? The bitbake docs only says "=+" operator apparently: 
https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#appending-and-prepending-with-spaces

Also nitpicking, but it's not necessary that a package be added with =+, 
e.g. if it should contain files that aren't matched by any other 
path/pattern in other FILES. But doesn't hurt to say to always do it.

> +For example, to add a custom package variant of the ``${PN}`` recipe named
> +``${PN}-extra`` (name is arbitrary), one should write::
> +
> +  PACKAGES =+ "${PN}-extra"
> +
> +Alternatively, a custom package can be added to the
> +:term:`PACKAGE_BEFORE_PN` variable::
> +
> +  PACKAGE_BEFORE_PN += "${PN}-extra"
>   

I would more recommend people to use PACKAGE_BEFORE_PN than the PACKAGES 
=+ scheme. The reason is quite simple, most people are used to += and 
using that instead of =+ could easily be missed by reviewers.

The former is only required if you need to have files typically in any 
of ${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale, 
in another package.

That's my opinion though and both are correct (and both should be 
mentioned!), so that would be fine to keep as is.

Cheers,
Quentin

Re: [docs] [PATCH] overview-manual: concepts: add details on package splitting

[Antonin Godard]

Hi Quentin,

Thanks for the great comments :)

On Wed Oct 16, 2024 at 4:10 PM CEST, Quentin Schulz wrote:
> Hi Antonin,
>
> On 10/16/24 3:19 PM, Antonin Godard via lists.yoctoproject.org wrote:
> > The package splitting section of the overview manual currently lacks any
> > explanation of how package splitting is implemented and redirects to
> > the package class, which is not really understandable for newcomers to
> > the project.
> >
> > This patch adds a short explanation of what is done:
> >
> > * How the PACKAGES variable is defined.
> > * How the FILES variable is defined.
> > * How the two work together.
> > * How to add a custom package.
> >
> > This should give enough details to a new user on what package splitting
> > achieves and how to add a custom package.
> >
> > Adresses [YOCTO #13225]
> >
> > Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
> > ---
> >   documentation/overview-manual/concepts.rst | 49 +++++++++++++++++++++++++++---
> >   1 file changed, 44 insertions(+), 5 deletions(-)
> >
> > diff --git a/documentation/overview-manual/concepts.rst b/documentation/overview-manual/concepts.rst
> > index 62f2327a7e27a777f738523e7fafe62422186abb..d3e90b80d4fe1954c7aacba8f0fe243cba04c1b5 100644
> > --- a/documentation/overview-manual/concepts.rst
> > +++ b/documentation/overview-manual/concepts.rst
> > @@ -912,11 +912,50 @@ the analysis and package splitting process use several areas:
> >      execute on a system and it generates code for yet another machine
> >      (e.g. :ref:`ref-classes-cross-canadian` recipes).
> >
> > -The :term:`FILES` variable defines the
> > -files that go into each package in
> > -:term:`PACKAGES`. If you want
> > -details on how this is accomplished, you can look at
> > -:yocto_git:`package.bbclass </poky/tree/meta/classes-global/package.bbclass>`.
> > +Packages for a recipe are listed in the :term:`PACKAGES` variable. The
> > +:yocto_git:`meta/conf/bitbake.conf </poky/tree/meta/conf/bitbake.conf>`
> > +configuration file defines the following default list of packages::
> > +
> > +  PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
> > +
> > +Each of these packages contain a default list of files defined with the
> > +:term:`FILES` variable. For example, the package ``${PN}-dev`` represents the
> > +development variant of the main package ``${PN}`` and its default list of file
>
> I misread the first time I read that because I thought "and its default
> list of files contains development-oriented files" applied to "${PN}"
> and not "${PN}-dev". I could suggest simply adding a coma:
>
> """
> the package ``${PN}-dev`` represents the development variant of the main
> package ``${PN}``, and its default list
> """
>
> Just a suggestion though, could be that it's just me reading it this way :)

Right, I feel like the comma makes sense too.

> s/file/files/ in any case.

+1

> Additionally, I think PN-dev does not represent the development variant
> of the main package, it represents the files that are useful for
> developing an application depending on PN main package. Using "variant"
> seems to indicate that PN-dev could be self-sufficient and potentially
> have the same binaries/shlibs than in PN but compiled differently.

That's correct, 'variant' is misleading here. In the end, what about:

"""
For example, the package ``${PN}-dev`` represents files useful to the
development of applications depending on ``${PN}``. The default list of files
for ``${PN}-dev``, also defined in :oe_git:`meta/conf/bitbake.conf
</openembedded-core/tree/meta/conf/bitbake.conf>`, is defined as follows::
"""

> > +contains development-oriented files only::
> > +
> > +  FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
> > +                  ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
> > +                  ${datadir}/aclocal ${base_libdir}/*.o \
> > +                  ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
> > +                  ${libdir}/cmake ${datadir}/cmake"
> > +
> > +The paths in this list must be *absolute*, and must use the path prefixes
>
> There is a very important piece of information that needs to be added
> here and a source of many newcomers' mistake: these are absolute paths
> from the PoV of the root of the filesystem on the target, NOT from bitbake.
>
> Indeed, in do_install (and do_package, etc.) we need to prefix our paths
> with $D (and $PKGD/$PKGDEST) when installing/packaging files, however,
> FILES doesn't not need those, even more it should NOT have those.

You're right, will change to:

"""
The paths must use the path prefixes defined by the
:oe_git:`meta/conf/bitbake.conf </openembedded-core/tree/meta/conf/bitbake.conf>`
configuration file, such as ``${sysconfdir}``, ``${bindir}``, etc.

The paths in this list must be *absolute* paths from the point of view of the
root filesystem on the target. For example, assuming the following path is
added to the :term:`FILES` variable::

  ${sysconfdir}/foo.conf

In a later phase of the build system, the created package will be used to
install the file ``/etc/foo.conf`` on the target, since ``${sysconfdir}``
equals to ``/etc`` (unless explicitely overwritten).
"""

> > +defined by the :yocto_git:`bitbake.conf </poky/tree/meta/conf/bitbake.conf>`
>
> use oe_git here instead since bitbake is maintained by OE folks.

+1

> > +configuration file, such as ``${sysconfdir}``, ``${bindir}``, etc. They can
> > +optionally use wildcards (``*`` ) to match multiple files installed in a
> > +directory.
> > +
> > +.. note::
> > +
> > +   The list of files for a package is defined using the override syntax by
> > +   separating :term:`FILES` and the package name by a semi-colon (``:``).
> > +
> > +The order of the packages in the :term:`PACKAGES` is important: during package
> > +splitting, going from the first element in :term:`PACKAGES` to the last, each
>
> I would say leftmost to rightmost as we have some other variables that
> go rightmost to leftmost (hello OVERRIDES :) ).

+1

> > +file matching a pattern specified in the corresponding :term:`FILES` definition
> > +will be included in the package and *excluded* for the remaining packages listed
> > +in :term:`PACKAGES`. As a consequence, custom packages should be added using the
>
> I could suggest:
> """
> A given file can only ever be in one package. Therefore, the first (from
> leftmost to rightmost in :term:`PACKAGES`) package for which the given
> file matches a path from its :term:`FILES` will contain the given file.
> """
>
> Still a bit too complex of a sentence but couldn't come up with anything
> better for now.

How about:

"""
A given file can only ever be in one package. By iterating from the
leftmost to rightmost package in :term:`PACKAGES`, each file matching one of the
pattern defined in the corresponding :term:`FILES` definition is included in the
package. At the end of package splitting, all the remaining files are stored in
the base ``${PN}`` package.

The result of package splitting is stored in the :term:`PKGDEST` directory, and
contains one directory per package.
"""

> > +prepend operator (``=+``) to be prioritized.
> > +
>
> I am not sure it is wise to say "prepend" operator as we have three ways
> of prepending:
> - =+
> - =.
> - :prepend
>
> But since you explicit which one you're talking about, I guess it's
> fine? The bitbake docs only says "=+" operator apparently:
> https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#appending-and-prepending-with-spaces

I think using the "prepend operator" + showing it in parenthesis makes sense
here, as newcomers may not be aware that it is used for prepending (there is no
explanation on operators in this document).

> Also nitpicking, but it's not necessary that a package be added with =+,
> e.g. if it should contain files that aren't matched by any other
> path/pattern in other FILES. But doesn't hurt to say to always do it.

Although you're correct, I would say let's keep it simple here since it's
already confusing enough for newcomers :)

> > +For example, to add a custom package variant of the ``${PN}`` recipe named
> > +``${PN}-extra`` (name is arbitrary), one should write::
> > +
> > +  PACKAGES =+ "${PN}-extra"
> > +
> > +Alternatively, a custom package can be added to the
> > +:term:`PACKAGE_BEFORE_PN` variable::
> > +
> > +  PACKAGE_BEFORE_PN += "${PN}-extra"
> >
>
> I would more recommend people to use PACKAGE_BEFORE_PN than the PACKAGES
> =+ scheme. The reason is quite simple, most people are used to += and
> using that instead of =+ could easily be missed by reviewers.
>
> The former is only required if you need to have files typically in any
> of ${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale,
> in another package.
>
> That's my opinion though and both are correct (and both should be
> mentioned!), so that would be fine to keep as is.

Absolutely, I will swap PACKAGE_BEFORE_PN and the =+ technique, so the most
important is introduced first.

Cheers,
Antonin

--
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH] overview-manual: concepts: add details on package splitting

[Quentin Schulz]

Hi Antonin,

On 10/17/24 10:44 AM, Antonin Godard wrote:
> Hi Quentin,
> 
> Thanks for the great comments :)
> 
> On Wed Oct 16, 2024 at 4:10 PM CEST, Quentin Schulz wrote:
>> Hi Antonin,
>>
>> On 10/16/24 3:19 PM, Antonin Godard via lists.yoctoproject.org wrote:
[...]
>> Additionally, I think PN-dev does not represent the development variant
>> of the main package, it represents the files that are useful for
>> developing an application depending on PN main package. Using "variant"
>> seems to indicate that PN-dev could be self-sufficient and potentially
>> have the same binaries/shlibs than in PN but compiled differently.
> 
> That's correct, 'variant' is misleading here. In the end, what about:
> 
> """
> For example, the package ``${PN}-dev`` represents files useful to the
> development of applications depending on ``${PN}``. The default list of files
> for ``${PN}-dev``, also defined in :oe_git:`meta/conf/bitbake.conf
> </openembedded-core/tree/meta/conf/bitbake.conf>`, is defined as follows::
> """
> 

Looks fine to me. A small nitpick here is that it's not necessarily ONLY 
for ${PN}, e.g. if we have more than one package containing 
binaries/shlibs. But at the same time, if a recipe has multiple of such 
packages, it could make sense to have their counterpart in -dev, -src, 
-dbg as well. Anyway, just a random thought but I think this is good 
enough and not really misleading :)

>>> +contains development-oriented files only::
>>> +
>>> +  FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
>>> +                  ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
>>> +                  ${datadir}/aclocal ${base_libdir}/*.o \
>>> +                  ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
>>> +                  ${libdir}/cmake ${datadir}/cmake"
>>> +
>>> +The paths in this list must be *absolute*, and must use the path prefixes
>>
>> There is a very important piece of information that needs to be added
>> here and a source of many newcomers' mistake: these are absolute paths
>> from the PoV of the root of the filesystem on the target, NOT from bitbake.
>>
>> Indeed, in do_install (and do_package, etc.) we need to prefix our paths
>> with $D (and $PKGD/$PKGDEST) when installing/packaging files, however,
>> FILES doesn't not need those, even more it should NOT have those.
> 
> You're right, will change to:
> 
> """
> The paths must use the path prefixes defined by the
> :oe_git:`meta/conf/bitbake.conf </openembedded-core/tree/meta/conf/bitbake.conf>`
> configuration file, such as ``${sysconfdir}``, ``${bindir}``, etc.
> 
> The paths in this list must be *absolute* paths from the point of view of the
> root filesystem on the target. For example, assuming the following path is
> added to the :term:`FILES` variable::
> 
>    ${sysconfdir}/foo.conf
> 
> In a later phase of the build system, the created package will be used to
> install the file ``/etc/foo.conf`` on the target, since ``${sysconfdir}``
> equals to ``/etc`` (unless explicitely overwritten).
> """
> 

Note sure the last paragraph makes a lot of sense in that context, I 
would simply remove it. I realize that I may have mislead you in my 
suggestion, it was intended to be a justification as to why I think this 
mistakes happen often, because we often have an additional variable in 
front of the path as one would see it on the target.

I would add a note to make it extra clear that ${D} should NOT be added 
to it. It's implied here by "omission" but I've seen this happen too 
many times that I would really suggest adding a note for that common 
mistake :)

[...]

>>> +file matching a pattern specified in the corresponding :term:`FILES` definition
>>> +will be included in the package and *excluded* for the remaining packages listed
>>> +in :term:`PACKAGES`. As a consequence, custom packages should be added using the
>>
>> I could suggest:
>> """
>> A given file can only ever be in one package. Therefore, the first (from
>> leftmost to rightmost in :term:`PACKAGES`) package for which the given
>> file matches a path from its :term:`FILES` will contain the given file.
>> """
>>
>> Still a bit too complex of a sentence but couldn't come up with anything
>> better for now.
> 
> How about:
> 
> """
> A given file can only ever be in one package. By iterating from the
> leftmost to rightmost package in :term:`PACKAGES`, each file matching one of the
> pattern defined in the corresponding :term:`FILES` definition is included in the

s/pattern/patterns/

> package. At the end of package splitting, all the remaining files are stored in
> the base ``${PN}`` package.
> 

Is that true? I'm pretty sure you get a warning/build failure if a file 
is not in any package. It just happens that most common paths are in 
FILES:${PN} but I don't think there's anything making bitbake have PN a 
catch-all package?

This would also mean it'd make no sense to have a package listed on the 
right side of PN in PACKAGES, but that's a valid use-case for anything 
not matching any prior FILES:. But maybe I'm misremembering :)

> The result of package splitting is stored in the :term:`PKGDEST` directory, and
> contains one directory per package.

Not sure if that isn't too advanced already for something in the 
overview-manual.

I guess we could tell the user how to find out in which package a file is?

oe-pkgdata-util find-path '/etc/foo.conf'

should do that if I remember correctly.

> """
> 
>>> +prepend operator (``=+``) to be prioritized.
>>> +
>>
>> I am not sure it is wise to say "prepend" operator as we have three ways
>> of prepending:
>> - =+
>> - =.
>> - :prepend
>>
>> But since you explicit which one you're talking about, I guess it's
>> fine? The bitbake docs only says "=+" operator apparently:
>> https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#appending-and-prepending-with-spaces
> 
> I think using the "prepend operator" + showing it in parenthesis makes sense
> here, as newcomers may not be aware that it is used for prepending (there is no
> explanation on operators in this document).
> 

Fair enough.

Cheers,
Quentin

Re: [docs] [PATCH] overview-manual: concepts: add details on package splitting

[Antonin Godard]

Hi Quentin,

On Fri Oct 18, 2024 at 2:37 PM CEST, Quentin Schulz wrote:
>>>> +contains development-oriented files only::
>>>> +
>>>> +  FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
>>>> +                  ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
>>>> +                  ${datadir}/aclocal ${base_libdir}/*.o \
>>>> +                  ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
>>>> +                  ${libdir}/cmake ${datadir}/cmake"
>>>> +
>>>> +The paths in this list must be *absolute*, and must use the path prefixes
>>>
>>> There is a very important piece of information that needs to be added
>>> here and a source of many newcomers' mistake: these are absolute paths
>>> from the PoV of the root of the filesystem on the target, NOT from bitbake.
>>>
>>> Indeed, in do_install (and do_package, etc.) we need to prefix our paths
>>> with $D (and $PKGD/$PKGDEST) when installing/packaging files, however,
>>> FILES doesn't not need those, even more it should NOT have those.
>>
>> You're right, will change to:
>>
>> """
>> The paths must use the path prefixes defined by the
>> :oe_git:`meta/conf/bitbake.conf </openembedded-core/tree/meta/conf/bitbake.conf>`
>> configuration file, such as ``${sysconfdir}``, ``${bindir}``, etc.
>>
>> The paths in this list must be *absolute* paths from the point of view of the
>> root filesystem on the target. For example, assuming the following path is
>> added to the :term:`FILES` variable::
>>
>>    ${sysconfdir}/foo.conf
>>
>> In a later phase of the build system, the created package will be used to
>> install the file ``/etc/foo.conf`` on the target, since ``${sysconfdir}``
>> equals to ``/etc`` (unless explicitely overwritten).
>> """
>>
>
> Note sure the last paragraph makes a lot of sense in that context, I
> would simply remove it. I realize that I may have mislead you in my
> suggestion, it was intended to be a justification as to why I think this
> mistakes happen often, because we often have an additional variable in
> front of the path as one would see it on the target.
>
> I would add a note to make it extra clear that ${D} should NOT be added
> to it. It's implied here by "omission" but I've seen this happen too
> many times that I would really suggest adding a note for that common
> mistake :)

Something like this?

"""
The paths in this list must be *absolute* paths from the point of view of the
root filesystem on the target, and should make reference to the variable
:term:`D` or any :term:`WORKDIR` related variable. An example would be::

  ${sysconfdir}/foo.conf
"""

> [...]
>
>>>> +file matching a pattern specified in the corresponding :term:`FILES` definition
>>>> +will be included in the package and *excluded* for the remaining packages listed
>>>> +in :term:`PACKAGES`. As a consequence, custom packages should be added using the
>>>
>>> I could suggest:
>>> """
>>> A given file can only ever be in one package. Therefore, the first (from
>>> leftmost to rightmost in :term:`PACKAGES`) package for which the given
>>> file matches a path from its :term:`FILES` will contain the given file.
>>> """
>>>
>>> Still a bit too complex of a sentence but couldn't come up with anything
>>> better for now.
>>
>> How about:
>>
>> """
>> A given file can only ever be in one package. By iterating from the
>> leftmost to rightmost package in :term:`PACKAGES`, each file matching one of the
>> pattern defined in the corresponding :term:`FILES` definition is included in the
>
> s/pattern/patterns/
>
>> package. At the end of package splitting, all the remaining files are stored in
>> the base ``${PN}`` package.
>>
>
> Is that true? I'm pretty sure you get a warning/build failure if a file
> is not in any package. It just happens that most common paths are in
> FILES:${PN} but I don't think there's anything making bitbake have PN a
> catch-all package?
>
> This would also mean it'd make no sense to have a package listed on the
> right side of PN in PACKAGES, but that's a valid use-case for anything
> not matching any prior FILES:. But maybe I'm misremembering :)

You're right. Actually, FILES:$PN matches a lot of files by default, but not all
of them, so my sentence is a bit of a shortcut. I'll simply remove it.

>> The result of package splitting is stored in the :term:`PKGDEST` directory, and
>> contains one directory per package.
>
> Not sure if that isn't too advanced already for something in the
> overview-manual.
>
> I guess we could tell the user how to find out in which package a file is?
>
> oe-pkgdata-util find-path '/etc/foo.conf'
>
> should do that if I remember correctly.

Good suggestion, I'll add that, and link to more in on this utility from the dev
manual.

Cheers,
Antonin

--
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH] overview-manual: concepts: add details on package splitting

[Quentin Schulz]

Hi Antonin,

On 10/22/24 1:54 PM, Antonin Godard wrote:
> Hi Quentin,
> 
> On Fri Oct 18, 2024 at 2:37 PM CEST, Quentin Schulz wrote:
>>>>> +contains development-oriented files only::
>>>>> +
>>>>> +  FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
>>>>> +                  ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
>>>>> +                  ${datadir}/aclocal ${base_libdir}/*.o \
>>>>> +                  ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
>>>>> +                  ${libdir}/cmake ${datadir}/cmake"
>>>>> +
>>>>> +The paths in this list must be *absolute*, and must use the path prefixes
>>>>
>>>> There is a very important piece of information that needs to be added
>>>> here and a source of many newcomers' mistake: these are absolute paths
>>>> from the PoV of the root of the filesystem on the target, NOT from bitbake.
>>>>
>>>> Indeed, in do_install (and do_package, etc.) we need to prefix our paths
>>>> with $D (and $PKGD/$PKGDEST) when installing/packaging files, however,
>>>> FILES doesn't not need those, even more it should NOT have those.
>>>
>>> You're right, will change to:
>>>
>>> """
>>> The paths must use the path prefixes defined by the
>>> :oe_git:`meta/conf/bitbake.conf </openembedded-core/tree/meta/conf/bitbake.conf>`
>>> configuration file, such as ``${sysconfdir}``, ``${bindir}``, etc.
>>>
>>> The paths in this list must be *absolute* paths from the point of view of the
>>> root filesystem on the target. For example, assuming the following path is
>>> added to the :term:`FILES` variable::
>>>
>>>     ${sysconfdir}/foo.conf
>>>
>>> In a later phase of the build system, the created package will be used to
>>> install the file ``/etc/foo.conf`` on the target, since ``${sysconfdir}``
>>> equals to ``/etc`` (unless explicitely overwritten).
>>> """
>>>
>>
>> Note sure the last paragraph makes a lot of sense in that context, I
>> would simply remove it. I realize that I may have mislead you in my
>> suggestion, it was intended to be a justification as to why I think this
>> mistakes happen often, because we often have an additional variable in
>> front of the path as one would see it on the target.
>>
>> I would add a note to make it extra clear that ${D} should NOT be added
>> to it. It's implied here by "omission" but I've seen this happen too
>> many times that I would really suggest adding a note for that common
>> mistake :)
> 
> Something like this?
> 
> """
> The paths in this list must be *absolute* paths from the point of view of the
> root filesystem on the target, and should make reference to the variable
> :term:`D` or any :term:`WORKDIR` related variable. An example would be::
> 
>    ${sysconfdir}/foo.conf
> """
> 

should NOT, but otherwise yes.

I would also states that ${sysconfdir}/foo.conf is a **correct** 
example, because we finish the last sentence with something to NOT do 
and then provide an example, so it can be a bit confusing.

Cheers,
Quentin

Re: [docs] [PATCH] overview-manual: concepts: add details on package splitting

[Antonin Godard]

Hi Quentin,

On Tue Oct 22, 2024 at 2:02 PM CEST, Quentin Schulz wrote:
> Hi Antonin,
>
> On 10/22/24 1:54 PM, Antonin Godard wrote:
>> Hi Quentin,
>>
>> On Fri Oct 18, 2024 at 2:37 PM CEST, Quentin Schulz wrote:
>>>>>> +contains development-oriented files only::
>>>>>> +
>>>>>> +  FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
>>>>>> +                  ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
>>>>>> +                  ${datadir}/aclocal ${base_libdir}/*.o \
>>>>>> +                  ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
>>>>>> +                  ${libdir}/cmake ${datadir}/cmake"
>>>>>> +
>>>>>> +The paths in this list must be *absolute*, and must use the path prefixes
>>>>>
>>>>> There is a very important piece of information that needs to be added
>>>>> here and a source of many newcomers' mistake: these are absolute paths
>>>>> from the PoV of the root of the filesystem on the target, NOT from bitbake.
>>>>>
>>>>> Indeed, in do_install (and do_package, etc.) we need to prefix our paths
>>>>> with $D (and $PKGD/$PKGDEST) when installing/packaging files, however,
>>>>> FILES doesn't not need those, even more it should NOT have those.
>>>>
>>>> You're right, will change to:
>>>>
>>>> """
>>>> The paths must use the path prefixes defined by the
>>>> :oe_git:`meta/conf/bitbake.conf </openembedded-core/tree/meta/conf/bitbake.conf>`
>>>> configuration file, such as ``${sysconfdir}``, ``${bindir}``, etc.
>>>>
>>>> The paths in this list must be *absolute* paths from the point of view of the
>>>> root filesystem on the target. For example, assuming the following path is
>>>> added to the :term:`FILES` variable::
>>>>
>>>>     ${sysconfdir}/foo.conf
>>>>
>>>> In a later phase of the build system, the created package will be used to
>>>> install the file ``/etc/foo.conf`` on the target, since ``${sysconfdir}``
>>>> equals to ``/etc`` (unless explicitely overwritten).
>>>> """
>>>>
>>>
>>> Note sure the last paragraph makes a lot of sense in that context, I
>>> would simply remove it. I realize that I may have mislead you in my
>>> suggestion, it was intended to be a justification as to why I think this
>>> mistakes happen often, because we often have an additional variable in
>>> front of the path as one would see it on the target.
>>>
>>> I would add a note to make it extra clear that ${D} should NOT be added
>>> to it. It's implied here by "omission" but I've seen this happen too
>>> many times that I would really suggest adding a note for that common
>>> mistake :)
>>
>> Something like this?
>>
>> """
>> The paths in this list must be *absolute* paths from the point of view of the
>> root filesystem on the target, and should make reference to the variable
>> :term:`D` or any :term:`WORKDIR` related variable. An example would be::
>>
>>    ${sysconfdir}/foo.conf
>> """
>>
>
> should NOT, but otherwise yes.

Ouch, thanks for noticing this!

> I would also states that ${sysconfdir}/foo.conf is a **correct**
> example, because we finish the last sentence with something to NOT do
> and then provide an example, so it can be a bit confusing.

+1, will add that.

Sending a v2 in a couple of minutes…

Thanks!
Antonin

--
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com