Version bump before release

Version bump before release

[Michael Halstead]

[-- Attachment #1: Type: text/plain, Size: 1179 bytes --]

Hello,
In the past we've correctly bumped the versions in the yocto-docs release
branches before the release build and I only need to update the master
branch to update the switcher on the website. This doesn't happen reliably
though and we've been having to patch the build system to make the docs
build correctly.

Ideally all that would be needed at release time is a patch like
https://git.yoctoproject.org/yocto-docs/commit/?id=78545b8f42657c21bc3cac96429d3950b07a3aca
to update the website.

Lately I've also had to make patches like these as well during release:

https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=ca8e95276d96b94d97e269b0f4f9ad306f8417d7
https://git.yoctoproject.org/yocto-docs/commit/?h=honister&id=7980cfe942a1c8ceb3fd6b003ec6346d9bc11015
(honister branch)

What process can we put in place to make sure these version bumps happen on
the named branch before the release candidate builds?

Richard suggested a test that would make RC builds fail if the docs version
were outdated. Any idea how to write that test? What other ideas come
to mind?

Thank you,
-- 
Michael Halstead
Linux Foundation / Yocto Project
Systems Operations Engineer

[-- Attachment #2: Type: text/html, Size: 1881 bytes --]

Re: [docs] Version bump before release

[Steve Sakoman]

On Thu, Mar 17, 2022 at 2:04 PM Michael Halstead
<mhalstead@linuxfoundation.org> wrote:
>
> Hello,
> In the past we've correctly bumped the versions in the yocto-docs release branches before the release build and I only need to update the master branch to update the switcher on the website. This doesn't happen reliably though and we've been having to patch the build system to make the docs build correctly.

I've been confused about this too.  I always submit a docs patch prior
to the release, and until recently they were take pre-release.
However lately this has changed and I too am not sure why!

Steve

>
> Ideally all that would be needed at release time is a patch like https://git.yoctoproject.org/yocto-docs/commit/?id=78545b8f42657c21bc3cac96429d3950b07a3aca to update the website.
>
> Lately I've also had to make patches like these as well during release:
>
> https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=ca8e95276d96b94d97e269b0f4f9ad306f8417d7
> https://git.yoctoproject.org/yocto-docs/commit/?h=honister&id=7980cfe942a1c8ceb3fd6b003ec6346d9bc11015  (honister branch)
>
> What process can we put in place to make sure these version bumps happen on the named branch before the release candidate builds?
>
> Richard suggested a test that would make RC builds fail if the docs version were outdated. Any idea how to write that test? What other ideas come to mind?
>
> Thank you,
> --
> Michael Halstead
> Linux Foundation / Yocto Project
> Systems Operations Engineer
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#2619): https://lists.yoctoproject.org/g/docs/message/2619
> Mute This Topic: https://lists.yoctoproject.org/mt/89858420/3620601
> Group Owner: docs+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [steve@sakoman.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

RE: [docs] Version bump before release

[Lee, Chee Yang]

> -----Original Message-----
> From: docs@lists.yoctoproject.org <docs@lists.yoctoproject.org> On Behalf
> Of Steve Sakoman
> Sent: Friday, March 18, 2022 8:33 AM
> To: Michael Halstead <mhalstead@linuxfoundation.org>
> Cc: YP docs mailing list <docs@lists.yoctoproject.org>
> Subject: Re: [docs] Version bump before release
> 
> On Thu, Mar 17, 2022 at 2:04 PM Michael Halstead
> <mhalstead@linuxfoundation.org> wrote:
> >
> > Hello,
> > In the past we've correctly bumped the versions in the yocto-docs release
> branches before the release build and I only need to update the master
> branch to update the switcher on the website. This doesn't happen reliably
> though and we've been having to patch the build system to make the docs
> build correctly.
> 
> I've been confused about this too.  I always submit a docs patch prior to the
> release, and until recently they were take pre-release.
> However lately this has changed and I too am not sure why!
> 

Found this discussion about the version bump timing, Is it the process changed after this ?
https://lists.yoctoproject.org/g/docs/topic/87282773#2200

Yocto-docs are tagged before release announcement, I think would be nice to have the version bump before release so the tag can align with the version bump commit.


Chee Yang

> Steve
> 
> >
> > Ideally all that would be needed at release time is a patch like
> https://git.yoctoproject.org/yocto-
> docs/commit/?id=78545b8f42657c21bc3cac96429d3950b07a3aca to update
> the website.
> >
> > Lately I've also had to make patches like these as well during release:
> >
> > https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=ca8e9
> > 5276d96b94d97e269b0f4f9ad306f8417d7
> > https://git.yoctoproject.org/yocto-docs/commit/?h=honister&id=7980cfe9
> > 42a1c8ceb3fd6b003ec6346d9bc11015  (honister branch)
> >
> > What process can we put in place to make sure these version bumps
> happen on the named branch before the release candidate builds?
> >
> > Richard suggested a test that would make RC builds fail if the docs version
> were outdated. Any idea how to write that test? What other ideas come to
> mind?
> >
> > Thank you,
> > --
> > Michael Halstead
> > Linux Foundation / Yocto Project
> > Systems Operations Engineer
> >
> >
> >

Re: [docs] Version bump before release

[Quentin Schulz]

Hi all,

On March 18, 2022 1:04:32 AM GMT+01:00, Michael Halstead <mhalstead@linuxfoundation.org> wrote:
>Hello,
>In the past we've correctly bumped the versions in the yocto-docs release
>branches before the release build and I only need to update the master
>branch to update the switcher on the website. This doesn't happen reliably
>though and we've been having to patch the build system to make the docs
>build correctly.
>
>Ideally all that would be needed at release time is a patch like
>https://git.yoctoproject.org/yocto-docs/commit/?id=78545b8f42657c21bc3cac96429d3950b07a3aca
>to update the website.
>

For the new releases (not the dot releases), it is essential this patch is merged with correct infos before tagging, specifically the poky.yaml file.

For dot releases, what matters is the bump of the dot release in poky.yaml, the rest of the changes aren't that important because it's impossible to always keep it up-to-date.

releases.rst and switchers.js are taken from master for all builds anyways by yocto-autobuilder-help so we don't technically have to worry about those making it before tagging day. Those need to make it to the master branch as soon as possible though (basically after tagging).

Two changes to conf.py need to be done too for releases. current_version needs to be bumped and bitbake_version to be set.

>Lately I've also had to make patches like these as well during release:
>
>https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=ca8e95276d96b94d97e269b0f4f9ad306f8417d7
>https://git.yoctoproject.org/yocto-docs/commit/?h=honister&id=7980cfe942a1c8ceb3fd6b003ec6346d9bc11015
>(honister branch)
>

What is REALLY important is to get the current_version and bitbake_version correct in conf.py. poky.yaml changes might be required too but I haven't checked yet.

>What process can we put in place to make sure these version bumps happen on
>the named branch before the release candidate builds?
>
>Richard suggested a test that would make RC builds fail if the docs version
>were outdated. Any idea how to write that test?

Check the content of conf.py that current_version is NOT dev and that bitbake_version is set. Not sure how to detect a pre-release not having the current_version bumped though.

If current_version is set, extract it and make sure it's used in poky.yaml.

Cheers,
Quentin

> What other ideas come
>to mind?
>
>Thank you,

Re: [docs] Version bump before release

[Michael Opdenacker]

Greetings,

On 3/18/22 07:47, Quentin Schulz wrote:
> Hi all,
>
> On March 18, 2022 1:04:32 AM GMT+01:00, Michael Halstead <mhalstead@linuxfoundation.org> wrote:
>> Hello,
>> In the past we've correctly bumped the versions in the yocto-docs release
>> branches before the release build and I only need to update the master
>> branch to update the switcher on the website. This doesn't happen reliably
>> though and we've been having to patch the build system to make the docs
>> build correctly.
>>
>> Ideally all that would be needed at release time is a patch like
>> https://git.yoctoproject.org/yocto-docs/commit/?id=78545b8f42657c21bc3cac96429d3950b07a3aca
>> to update the website.
>>
> For the new releases (not the dot releases), it is essential this patch is merged with correct infos before tagging, specifically the poky.yaml file.
>
> For dot releases, what matters is the bump of the dot release in poky.yaml, the rest of the changes aren't that important because it's impossible to always keep it up-to-date.
>
> releases.rst and switchers.js are taken from master for all builds anyways by yocto-autobuilder-help so we don't technically have to worry about those making it before tagging day. Those need to make it to the master branch as soon as possible though (basically after tagging).
>
> Two changes to conf.py need to be done too for releases. current_version needs to be bumped and bitbake_version to be set.


Some of the mess is probably my fault. I didn't want to update *master*
to refer to releases that didn't exist yet, because this causes warnings
on the documentation website. However, I realize there should be no
issue updating the release branches (such as
https://docs.yoctoproject.org/dunfell/, which I believe are not
advertised on https://docs.yoctoproject.org/), so I was wrong to apply
the changes only after the release was out.

My apologies!

Let me try to understand if I understand the desired workflow correctly:

  * Prior (weeks before) to a future new release, create a new release
    branch on "yocto-docs", e.g. "kirkstone"
  * Update documentation/conf.py (version specific) and
    documentation/poky.yaml in the release branch (e.g. "dunfell")
    according to the upcoming release (e.g. "3.1.15")
  * Before making the release, check in the content of conf.py that
    current_version is NOT dev and that bitbake_version is set. If
    current_version is set, extract it and make sure it's used in
    poky.yaml too. Also check that the YOCTO_DOC_VERSION_MINUS_ONE
    setting in poky.yaml corresponds to the number of the latest n-1
    release.
  * Make the release. I guess, when we have a new release, this means
    that a new branch for BitBake was created.
  * Make sure
    https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/run-docs-build
    supports the new BitBake branch.
  * Add new tags to the branch (e.g. "yocto-3.1.15" + "dunfell-3.1.15").
    Isn't the "yocto-" tag sufficient by the way?
  * Update the following files in master:
      o documentation/conf.py
      o documentation/poky.yaml
      o documentation/releases.rst
      o documentation/sphinx-static/switchers.js

Would this be a correct workflow? I propose to keep this in a wiki page
by the way, so that updates and notes are easy to manage.

Thanks,
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] Version bump before release

[Quentin Schulz]

On 3/18/22 11:05, Michael Opdenacker via lists.yoctoproject.org wrote:
> Greetings,
> 
> On 3/18/22 07:47, Quentin Schulz wrote:
>> Hi all,
>>
>> On March 18, 2022 1:04:32 AM GMT+01:00, Michael Halstead <mhalstead@linuxfoundation.org> wrote:
>>> Hello,
>>> In the past we've correctly bumped the versions in the yocto-docs release
>>> branches before the release build and I only need to update the master
>>> branch to update the switcher on the website. This doesn't happen reliably
>>> though and we've been having to patch the build system to make the docs
>>> build correctly.
>>>
>>> Ideally all that would be needed at release time is a patch like
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_yocto-2Ddocs_commit_-3Fid-3D78545b8f42657c21bc3cac96429d3950b07a3aca&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=cTlw8ZyDgLCzGzd6TrnLJ9k1dlLUKczlSySrRTH_ur9xCYJt7xfrVe11YI4lEcXy&s=DXdHKanD7Qz-e9W7u3ZpHzJwt1R4FvqpmXp2_rbthiw&e=
>>> to update the website.
>>> >> For the new releases (not the dot releases), it is essential this 
patch is merged with correct infos before tagging, specifically the 
poky.yaml file.
>>
>> For dot releases, what matters is the bump of the dot release in poky.yaml, the rest of the changes aren't that important because it's impossible to always keep it up-to-date.
>>
>> releases.rst and switchers.js are taken from master for all builds anyways by yocto-autobuilder-help so we don't technically have to worry about those making it before tagging day. Those need to make it to the master branch as soon as possible though (basically after tagging).
>>
>> Two changes to conf.py need to be done too for releases. current_version needs to be bumped and bitbake_version to be set.
> 
> 
> Some of the mess is probably my fault. I didn't want to update *master*
> to refer to releases that didn't exist yet, because this causes warnings
> on the documentation website. However, I realize there should be no
> issue updating the release branches (such as
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.yoctoproject.org_dunfell_&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=cTlw8ZyDgLCzGzd6TrnLJ9k1dlLUKczlSySrRTH_ur9xCYJt7xfrVe11YI4lEcXy&s=fkjdSlAymDOkLJTh9p3Ga6YaRm4Rqd6-QxTDE61UbUs&e= , which I believe are not
> advertised on https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.yoctoproject.org_&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=cTlw8ZyDgLCzGzd6TrnLJ9k1dlLUKczlSySrRTH_ur9xCYJt7xfrVe11YI4lEcXy&s=ehS2rLKxADXnphei-i1w9MfnDpE6s_eabXSGdHf3IRU&e= ), so I was wrong to apply
> the changes only after the release was out.
> 

Branches are forgiving because they are moving and we always take the 
latest commit in those branches. So nothing to say about those because 
it IS possible to fix them when we encounter an issue or if there was an 
oversight.

We cannot (policy-wise) move a tag so we need some process to make sure 
it's properly done to avoid having too many patches in the 
yocto-autobuilder-helper (especially since most of the patches are 
identical).

> My apologies!
> 

Can't blame the people if there's no documented process :)

> Let me try to understand if I understand the desired workflow correctly:
> 
>    * Prior (weeks before) to a future new release, create a new release
>      branch on "yocto-docs", e.g. "kirkstone"
I think weeks before is too early to branch off. I think this should be 
done on release day. The issue of branching off too soon is that you 
need to apply patches to both master and kirkstone branches.

>    * Update documentation/conf.py (version specific) and
>      documentation/poky.yaml in the release branch (e.g. "dunfell")
>      according to the upcoming release (e.g. "3.1.15")
>    * Before making the release, check in the content of conf.py that
>      current_version is NOT dev and that bitbake_version is set. If
>      current_version is set, extract it and make sure it's used in
>      poky.yaml too. Also check that the YOCTO_DOC_VERSION_MINUS_ONE
>      setting in poky.yaml corresponds to the number of the latest n-1
>      release.
>    * Make the release. I guess, when we have a new release, this means
>      that a new branch for BitBake was created.

Bitbake should have a new release branch before yocto-docs is 
tagged/updated, yes. So that objects.inv from bitbake release docs can 
be used.

>    * Make sure
>      https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_yocto-2Dautobuilder-2Dhelper_tree_scripts_run-2Ddocs-2Dbuild&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=cTlw8ZyDgLCzGzd6TrnLJ9k1dlLUKczlSySrRTH_ur9xCYJt7xfrVe11YI4lEcXy&s=BDfclI2ORIfQDqFY9Cdc_eYIcWJ9qzUukRH1i7ZT6o4&e=
>      supports the new BitBake branch.

Yes, though IIRC I've something in some git stash to avoid having to 
maintain this list. Please ping me until I've sent a patch related to this.

>    * Add new tags to the branch (e.g. "yocto-3.1.15" + "dunfell-3.1.15").
>      Isn't the "yocto-" tag sufficient by the way? >    * Update the following files in master:
>        o documentation/conf.py

No. conf.py in master stays with current_version="dev" and 
bitbake_version="" otherwise it's incorrectly advertised and using an 
outdated version of bitbake docs.

>        o documentation/poky.yaml
>        o documentation/releases.rst
>        o documentation/sphinx-static/switchers.js
> 
> Would this be a correct workflow? I propose to keep this in a wiki page
> by the way, so that updates and notes are easy to manage.
> 

I would actually update poky.yaml in master branch before branching off 
kirkstone because poky.yaml from master is technically using latest 
release information (not necessarily dot release though), otherwise you 
would have the patch in both master and kirkstone. Not a big difference 
though. Just need to agree on the process.

Yes to a Wiki page. We need to document the issues we are having, what 
are the requirements and the suggested solutions. There's nothing 
straightforward to implement I'm afraid, but we need to improve this to 
make it easier on maintainers and lower the risk of human 
error/oversight happening somewhere.

Cheers,
Quentin

Re: [docs] Version bump before release

[Richard Purdie]

I know I've been a bit against having magic versions as the tag information may
not always be present in a git checkout of the docs. That said, I was curious to
see how close we could get the information with and without tag information.
I've included a script below which tries to use any branch/tag information
available to guess the correct release series and version to use.

We can make sure the tag information is there for docs builds on our
infrastructure so that bit is easy. The question is therefore whether this does
generate good enough information for people with local checkouts?

The objective here would be to automate generation of the version information in
the docs so we don't have to manually update it other than adding new release
series mappings to master.

See what you think of the script/idea.

I've just realised we could detect the missing tags and tell the user to fetch
them before building too. That would solve most of my worries I think?

Cheers,

Richard



#!/usr/bin/env python3

import subprocess
import collections
import sys

devbranch = "langdale"

release_series = collections.OrderedDict()
release_series["langdale"] = "4.1"
release_series["kirkstone"] = "4.0"
release_series["honister"] = "3.4"
release_series["hardknott"] = "3.3"
release_series["gatesgarth"] = "3.2"
release_series["dunfell"] = "3.1"

ourversion = None
ourseries = None
ourbranch = None

# Try and figure out what we are
tags = subprocess.run(["git", "tag", "--points-at", "HEAD"], capture_output=True, text=True).stdout
for t in tags.split():
    if t.startswith("yocto-"):
        ourversion = t[6:]

if ourversion:
    components = ourversion.split(".")
    baseversion = components[0] + "." + components[1]
    for i in release_series:
        if release_series[i] == baseversion:
            ourseries = i
            ourbranch = i
else:
    branch = subprocess.run(["git", "branch", "--show-current"], capture_output=True, text=True).stdout.strip()
    ourbranch = branch
    if branch == "master":
        ourseries = devbranch
    elif branch in release_series:
        ourseries = branch
    else:
        sys.exit("Unknown series for branch %s" % branch)

    previoustags = subprocess.run(["git", "tag", "--merged", "HEAD"], capture_output=True, text=True).stdout
    previoustags = [t[6:] for t in previoustags.split() if t.startswith("yocto-" + release_series[ourseries])]
    futuretags = subprocess.run(["git", "tag", "--merged", ourbranch], capture_output=True, text=True).stdout
    futuretags = [t[6:] for t in futuretags.split() if t.startswith("yocto-" + release_series[ourseries])]

    if len(previoustags) != len(futuretags):
        ourversion = previoustags[-1] + ".999"
    else:
        ourversion = release_series[ourseries] + ".999"

print("Version calculated to be %s" % ourversion)
print("Release series calculated to be %s" % ourseries)