Re: [docs] [PATCH] overview-manual: concepts: add details on package splitting

["Antonin Godard" <antonin.godard@bootlin.com>]

Hi Quentin,

On Fri Oct 18, 2024 at 2:37 PM CEST, Quentin Schulz wrote:
>>>> +contains development-oriented files only::
>>>> +
>>>> +  FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
>>>> +                  ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
>>>> +                  ${datadir}/aclocal ${base_libdir}/*.o \
>>>> +                  ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
>>>> +                  ${libdir}/cmake ${datadir}/cmake"
>>>> +
>>>> +The paths in this list must be *absolute*, and must use the path prefixes
>>>
>>> There is a very important piece of information that needs to be added
>>> here and a source of many newcomers' mistake: these are absolute paths
>>> from the PoV of the root of the filesystem on the target, NOT from bitbake.
>>>
>>> Indeed, in do_install (and do_package, etc.) we need to prefix our paths
>>> with $D (and $PKGD/$PKGDEST) when installing/packaging files, however,
>>> FILES doesn't not need those, even more it should NOT have those.
>>
>> You're right, will change to:
>>
>> """
>> The paths must use the path prefixes defined by the
>> :oe_git:`meta/conf/bitbake.conf </openembedded-core/tree/meta/conf/bitbake.conf>`
>> configuration file, such as ``${sysconfdir}``, ``${bindir}``, etc.
>>
>> The paths in this list must be *absolute* paths from the point of view of the
>> root filesystem on the target. For example, assuming the following path is
>> added to the :term:`FILES` variable::
>>
>>    ${sysconfdir}/foo.conf
>>
>> In a later phase of the build system, the created package will be used to
>> install the file ``/etc/foo.conf`` on the target, since ``${sysconfdir}``
>> equals to ``/etc`` (unless explicitely overwritten).
>> """
>>
>
> Note sure the last paragraph makes a lot of sense in that context, I
> would simply remove it. I realize that I may have mislead you in my
> suggestion, it was intended to be a justification as to why I think this
> mistakes happen often, because we often have an additional variable in
> front of the path as one would see it on the target.
>
> I would add a note to make it extra clear that ${D} should NOT be added
> to it. It's implied here by "omission" but I've seen this happen too
> many times that I would really suggest adding a note for that common
> mistake :)

Something like this?

"""
The paths in this list must be *absolute* paths from the point of view of the
root filesystem on the target, and should make reference to the variable
:term:`D` or any :term:`WORKDIR` related variable. An example would be::

  ${sysconfdir}/foo.conf
"""

> [...]
>
>>>> +file matching a pattern specified in the corresponding :term:`FILES` definition
>>>> +will be included in the package and *excluded* for the remaining packages listed
>>>> +in :term:`PACKAGES`. As a consequence, custom packages should be added using the
>>>
>>> I could suggest:
>>> """
>>> A given file can only ever be in one package. Therefore, the first (from
>>> leftmost to rightmost in :term:`PACKAGES`) package for which the given
>>> file matches a path from its :term:`FILES` will contain the given file.
>>> """
>>>
>>> Still a bit too complex of a sentence but couldn't come up with anything
>>> better for now.
>>
>> How about:
>>
>> """
>> A given file can only ever be in one package. By iterating from the
>> leftmost to rightmost package in :term:`PACKAGES`, each file matching one of the
>> pattern defined in the corresponding :term:`FILES` definition is included in the
>
> s/pattern/patterns/
>
>> package. At the end of package splitting, all the remaining files are stored in
>> the base ``${PN}`` package.
>>
>
> Is that true? I'm pretty sure you get a warning/build failure if a file
> is not in any package. It just happens that most common paths are in
> FILES:${PN} but I don't think there's anything making bitbake have PN a
> catch-all package?
>
> This would also mean it'd make no sense to have a package listed on the
> right side of PN in PACKAGES, but that's a valid use-case for anything
> not matching any prior FILES:. But maybe I'm misremembering :)

You're right. Actually, FILES:$PN matches a lot of files by default, but not all
of them, so my sentence is a bit of a shortcut. I'll simply remove it.

>> The result of package splitting is stored in the :term:`PKGDEST` directory, and
>> contains one directory per package.
>
> Not sure if that isn't too advanced already for something in the
> overview-manual.
>
> I guess we could tell the user how to find out in which package a file is?
>
> oe-pkgdata-util find-path '/etc/foo.conf'
>
> should do that if I remember correctly.

Good suggestion, I'll add that, and link to more in on this utility from the dev
manual.

Cheers,
Antonin

--
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com