Re: [docs] [PATCH 1/2] poky.yaml: introduce DISTRO_LATEST_TAG

["Antonin Godard" <antonin.godard@bootlin.com>]

Hi Quentin,

On Wed Apr 16, 2025 at 11:58 AM CEST, Quentin Schulz wrote:
> Hi Antonin,
>
> On 4/16/25 9:46 AM, Antonin Godard wrote:
>> Hi Quentin,
>> 
>> On Mon Apr 14, 2025 at 2:18 PM CEST, Quentin Schulz wrote:
>>> Hi Antonin,
>>>
>>> On 4/9/25 11:55 AM, Antonin Godard via lists.yoctoproject.org wrote:
>>>> Introduce the DISTRO_LATEST_TAG macro, which should always point to the
>>>> latest existing tag in the documentation, unlike DISTRO which may point
>>>> to A.B.999 to represent the tip of a branch.
>>>>
>>>> This variable is needed to fix dead links in the documentation that
>>>> currently use the DISTRO macro.
>>>>
>>>> Also, make DISTRO_REL_TAG use the DISTRO macro directly, to avoid
>>>> repetition, and add a DISTRO_REL_LATEST_TAG macro that has the same role
>>>> as DISTRO_LATEST_TAG but with "yocto-" prepended to it.
>>>>
>>>> In set_versions.py, run the "git describe --abbrev=0 --tags
>>>> --match='yocto-*'" command to get the latest existing tag on the
>>>> currently checked out commit. Fallback to ourversion in case we didn't
>>>> find any.
>>>>
>>>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>>>> ---
>>>>    documentation/poky.yaml.in    | 11 ++++++++++-
>>>>    documentation/set_versions.py | 16 +++++++++++++++-
>>>>    2 files changed, 25 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
>>>> index 836f11454..26c21e346 100644
>>>> --- a/documentation/poky.yaml.in
>>>> +++ b/documentation/poky.yaml.in
>>>> @@ -2,13 +2,22 @@
>>>>    # Macros used in the documentation
>>>>    #
>>>>    
>>>> +# The DISTRO variable represents the current docs version. It should be used
>>>> +# when referring to the current docs version. See also DISTRO_LATEST_TAG.
>>>>    DISTRO : "5.1"
>>>> +# The DISTRO_LATEST_TAG represents the latest tag on the current branch. It
>>>> +# should be used in HTTP link referring to the current docs version. In these
>>>> +# cases, the DISTRO may point to A.B.999 which does not exist (just used to
>>>> +# represent the latest HEAD revision on the branch). DISTRO_LATEST_TAG should
>>>> +# always point to an existing tag.
>>>
>>> I don't remember why/where we need A.B.999? I assume most shouldn't
>>> point at .999, ever?
>> 
>> The default landing page of docs.yoctoproject.org is a 999 version (tip of the
>> stable release branch).
>> 
>
> That essentially should be for the version number in text and in the JS 
> dropdown? We should really check our use of &DISTRO; in the docs.
>
> For example:
>
> documentation/contributor-guide/report-defect.rst, not sure we don't 
> want to use &DISTRO_LATEST_TAG; (or another variable) instead, as it'll 
> always be wrong on the default landing page of the docs (always x.x.999)?
>
> documentation/dev-manual/multiconfig.rst, I assume we also do not want 
> to have x.x.999 there? I'm also not entirely sure why we explicitly list 
> the version or if it's still appropriate?
>
> documentation/dev-manual/start.rst:774 we should probably get rid of 
> &DISTRO; (but we can keep &DISTRO_NAME; as that matches the code block 
> under?)
>
> documentation/dev-manual/start.rst:843 is plain wrong as the tag does 
> not exist so we should be using DISTRO_LATEST_TAG. Same for
> documentation/dev-manual/start.rst:854, same for 
> documentation/overview-manual/development-environment.rst:459.
>
> documentation/ref-manual/release-process.rst:15 should probably use 
> DISTRO_NAME or DISTRO_LATEST_TAG? (or probably another variable which is 
> only X.Y and not X.Y.Z if there's one, since X.Y.Z is a minor release).
>
> documentation/ref-manual/release-process.rst:59 probably should be using 
> DISTRO_LATEST_TAG too?
>
> documentation/ref-manual/system-requirements.rst:58, could be using 
> DISTRO_LATEST_TAG but probably not a big deal if we keep DISTRO instead?
>
> documentation/ref-manual/system-requirements.rst:349, I woudl assume is 
> completely wrong and should be DISTRO_LATEST_TAG instead? Same for 
> documentation/ref-manual/system-requirements.rst:350?
>
> documentation/ref-manual/system-requirements.rst:403,407,411,474,478,482 
> need to use DISTRO_LATEST_TAG to match the name of the buildtools.
>
> documentation/sdk-manual/appendix-obtain.rst:66,72,176 needs to match 
> the toolchain you can download, so needs to be DISTRO_LATEST_TAG.
>
> documentation/sdk-manual/appendix-obtain.rst:275 probably wants 
> DISTRO_LATEST_TAG too.
>
> documentation/sdk-manual/extensible.rst:120,124,126 probably wants 
> DISTRO_LATEST_TAG too.
>
> documentation/sdk-manual/using.rst:74,78,80,101,102,104,105,110,130,141, 
> probably wants DISTRO_LATEST_TAG too.
>
> documentation/sdk-manual/working-projects.rst:85,88,219,223,225,280 
> probably wants DISTRO_LATEST_TAG too.
>
> documentation/toaster-manual/reference.rst:161,295 DISTRO_LATEST_TAG

Thanks for going through the different usages of DISTRO. You're right, with
DISTRO_LATEST_TAG now I also think it would make more sense to use it rather
than DISTRO. I will try to work on that a bit later (or feel free to send a
patch :), the initial focus for this series was to fix dead links .

> Nothing uses YOCTO_RELEASE_DL_URL, maybe we can remove it entirely?

Indeed we can remove this variable.

>>> Also, it would be nice to provide an example as to what is supposed to
>>> be stored in each variable based on a specific example.
>>>
>>> e.g.
>>>
>>> """
>>> # If building from master, e.g. commit
>>> 3b50193fa0c9acf4a601aeae6e1c78d0e4a05aef
>>> # DISTRO will be 5.2.999
>>> # DISTRO_LATEST_TAG will be 5.1
>>> #
>>> # If building from a release branch, e.g. styhead (e.g. commit
>>> 85e738e4c0e62f69699fff4bb0482ee3e3121496)
>>> # DISTRO will be 5.1.999
>>> # DISTRO_LATEST_TAG will be 5.1.4
>>> #
>>> # If building from a tag, e.g. yocto-5.1.4
>>> # DISTRO will be 5.1.4
>>> # DISTRO_LATEST_TAG will be 5.1.4
>>> """
>>>
>>> Note that I haven't thoroughly checked the above is correct.
>> 
>> That's it. Although, as you see from set_versions.py, these variables are not to
>> be set manually in poky.yaml.in, and are automatically set by the script.
>> I could add a comment on that in poky.yaml.in.
>> 
>
> It's actually been bothering me a lot that we have some parts of 
> poky.yaml.in that are taken verbatim and some modified before making it 
> to poky.yaml.
>
> I seem to recall we wanted the variables set by set_versions.py to still 
> be able to build the docs from tarballs and not force the use of git 
> repos? But it seems like we force to fetch git tags at the very 
> beginning of the Python script.

  $ make html
  ./set_versions.py
  Please run 'git fetch --tags' before building the documentation
  make: *** [Makefile:72: html] Error 1

It is indeed a bug!
Created one here: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15834

> So I'm wondering if we should only have variables that aren't replaced 
> in poky.yaml.in and eventually have the ones set_versions.py adds in 
> poky.yaml (instead of replacing for example) documented in comments to 
> explain what they are and in which context(s) to use them?

That would be a bit cleaner yes.

Maybe standards.md would be a good place? i.e.:
"Use &DISTRO; in the docs when wanting to refer to the latest possible version"
"Use &DISTRO_LATEST_TAG; in the docs when wanting to refer to the latest
existing tag"
etc.

>>> If that is correct, do we really want to return 5.1 for LATEST_TAG if
>>> building from master?
>> 
>> DISTRO_LATEST_TAG is used in URLs and in most cases point to
>> downloads.yoctoproject.org. Now I'm not sure for master we want to point to a
>> dead link, or the latest tag across all possible branches, or just the previous
>> tag on the master branch. For simplicity I'd favor the last option, especially
>> because master is a dev branch so we don't expect any sort of stable information
>> on there.
>> 
>
> And the dev/ branch isn't the one accessed by default when going to 
> docs.yoctoproject.org/ so I think we can afford not linking to the 
> latest tag in the latest release branch and instead the latest tag in 
> the master branch (i.e. the last release's first tag).
>
> [...]
>
> Cheers,
> Quentin

Thanks,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com