[PATCH 1/3] migration-guides: placeholder files for 5.1

[PATCH 1/3] migration-guides: placeholder files for 5.1

[michael.opdenacker]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
---
 documentation/migration-guides/index.rst      |  1 +
 .../migration-guides/migration-5.1.rst        | 83 +++++++++++++++++++
 .../migration-guides/release-5.1.rst          |  9 ++
 .../migration-guides/release-notes-5.1.rst    | 67 +++++++++++++++
 4 files changed, 160 insertions(+)
 create mode 100644 documentation/migration-guides/migration-5.1.rst
 create mode 100644 documentation/migration-guides/release-5.1.rst
 create mode 100644 documentation/migration-guides/release-notes-5.1.rst

diff --git a/documentation/migration-guides/index.rst b/documentation/migration-guides/index.rst
index d8edd05b89..e9d7f72809 100644
--- a/documentation/migration-guides/index.rst
+++ b/documentation/migration-guides/index.rst
@@ -12,6 +12,7 @@ to move to one release of the Yocto Project from the previous one.
 .. toctree::
 
    migration-general
+   release-5.1
    release-5.0
    release-4.3
    release-4.2
diff --git a/documentation/migration-guides/migration-5.1.rst b/documentation/migration-guides/migration-5.1.rst
new file mode 100644
index 0000000000..b4db04bcee
--- /dev/null
+++ b/documentation/migration-guides/migration-5.1.rst
@@ -0,0 +1,83 @@
+.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
+
+Release 5.1 LTS (styhead)
+===========================
+
+Migration notes for 5.1 (styhead)
+------------------------------------
+
+This section provides migration information for moving to the Yocto
+Project 5.1 Release (codename "styhead") from the prior release.
+
+.. _migration-5.1-supported-kernel-versions:
+
+Supported kernel versions
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :term:`OLDEST_KERNEL` setting is still "5.15" in this release, meaning that
+out the box, older kernels are not supported. See :ref:`4.3 migration notes
+<migration-4.3-supported-kernel-versions>` for details.
+
+.. _migration-5.1-supported-distributions:
+
+Supported distributions
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Compared to the previous releases, running BitBake is supported on new
+GNU/Linux distributions:
+
+On the other hand, some earlier distributions are no longer supported:
+
+See :ref:`all supported distributions <system-requirements-supported-distros>`.
+
+.. _migration-5.1-go-changes:
+
+Go language changes
+~~~~~~~~~~~~~~~~~~~
+
+.. _migration-5.1-systemd-changes:
+
+systemd changes
+~~~~~~~~~~~~~~~
+
+.. _migration-5.1-recipe-changes:
+
+Recipe changes
+~~~~~~~~~~~~~~
+
+.. _migration-5.1-deprecated-variables:
+
+Deprecated variables
+~~~~~~~~~~~~~~~~~~~~
+
+.. _migration-5.1-removed-variables:
+
+Removed variables
+~~~~~~~~~~~~~~~~~
+
+The following variables have been removed:
+
+.. _migration-5.1-removed-recipes:
+
+Removed recipes
+~~~~~~~~~~~~~~~
+
+The following recipes have been removed in this release:
+
+.. _migration-5.1-removed-classes:
+
+Removed classes
+~~~~~~~~~~~~~~~
+
+No classes have been removed in this release.
+
+.. _migration-5.1-qemu-changes:
+
+QEMU changes
+~~~~~~~~~~~~
+
+.. _migration-5.1-misc-changes:
+
+Miscellaneous changes
+~~~~~~~~~~~~~~~~~~~~~
+
diff --git a/documentation/migration-guides/release-5.1.rst b/documentation/migration-guides/release-5.1.rst
new file mode 100644
index 0000000000..768edfa2b0
--- /dev/null
+++ b/documentation/migration-guides/release-5.1.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
+
+Release 5.1 (styhead)
+=======================
+
+.. toctree::
+
+   migration-5.1
+   release-notes-5.1
diff --git a/documentation/migration-guides/release-notes-5.1.rst b/documentation/migration-guides/release-notes-5.1.rst
new file mode 100644
index 0000000000..32cb2de2b4
--- /dev/null
+++ b/documentation/migration-guides/release-notes-5.1.rst
@@ -0,0 +1,67 @@
+.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
+
+Release notes for 5.1 (styhead)
+---------------------------------
+
+New Features / Enhancements in 5.1
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  Linux kernel 6.X, gcc 14.X, glibc 2.X, LLVM 18.X, and over XXX other recipe upgrades
+
+-  New variables:
+
+-  Architecture-specific enhancements:
+
+-  Kernel-related enhancements:
+
+-  New core recipes:
+
+-  QEMU / ``runqemu`` enhancements:
+
+-  Rust improvements:
+
+-  wic Image Creator enhancements:
+
+-  SDK-related improvements:
+
+-  Testing:
+
+-  Utility script changes:
+
+-  BitBake improvements:
+
+-  devtool improvements:
+
+-  recipetool improvements:
+
+-  Packaging changes:
+
+-  Security improvements:
+
+-  Toaster Web UI improvements:
+
+-  Prominent documentation updates:
+
+-  Miscellaneous changes:
+
+Known Issues in 5.1
+~~~~~~~~~~~~~~~~~~~
+
+Recipe License changes in 5.1
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following corrections have been made to the :term:`LICENSE` values set by recipes:
+
+Security Fixes in 5.1
+~~~~~~~~~~~~~~~~~~~~~
+
+Recipe Upgrades in 5.1
+~~~~~~~~~~~~~~~~~~~~~~
+
+Contributors to 5.1
+~~~~~~~~~~~~~~~~~~~
+
+Thanks to the following people who contributed to this release:
+
+Repositories / Downloads for Yocto-5.1
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- 
2.34.1

[PATCH 2/3] ref-manual: introduce UNPACKDIR variable

[michael.opdenacker]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

Note that this doesn't touch the "Source Fetching" section
in overview-manual/concepts.rst yet, as the unpack implementation
may not be finalized yet.

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
---
 documentation/ref-manual/tasks.rst     | 8 ++++----
 documentation/ref-manual/variables.rst | 5 +++++
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst
index 2e4b23408d..70c0dea3bb 100644
--- a/documentation/ref-manual/tasks.rst
+++ b/documentation/ref-manual/tasks.rst
@@ -412,12 +412,12 @@ them. You can learn more by looking at the
 -------------
 
 Unpacks the source code into a working directory pointed to by
-``${``\ :term:`WORKDIR`\ ``}``. The :term:`S`
-variable also plays a role in where unpacked source files ultimately
-reside. For more information on how source files are unpacked, see the
+``${``\ :term:`UNPACKDIR`\ ``}``. Another, legacy way to specify
+this directory is through the :term:`S` and :term:`WORKDIR` variables.
+For more information on how source files are unpacked, see the
 ":ref:`overview-manual/concepts:source fetching`"
 section in the Yocto Project Overview and Concepts Manual and also see
-the :term:`WORKDIR` and :term:`S` variable descriptions.
+the :term:`UNPACKDIR`, :term:`WORKDIR` and :term:`S` variable descriptions.
 
 Manually Called Tasks
 =====================
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 546c674699..07b5b6f95c 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -9698,6 +9698,11 @@ system and gives an overview of their function and contents.
       :ref:`ref-classes-insane` class and is only enabled if the
       recipe inherits the :ref:`ref-classes-autotools` class.
 
+   :term:`UNPACKDIR`
+      This variable, used by the :ref:`ref-classes-base` class,
+      specifies where fetches sources should be unpacked by the
+      :ref:`ref-tasks-unpack` task.
+
    :term:`UPDATERCPN`
       For recipes inheriting the
       :ref:`ref-classes-update-rc.d` class, :term:`UPDATERCPN`
-- 
2.34.1

Re: [docs] [PATCH 2/3] ref-manual: introduce UNPACKDIR variable

[Andrew Murray]

Hi Michael,

On Thu, 23 May 2024 at 14:56, Michael Opdenacker via
lists.yoctoproject.org
<michael.opdenacker=bootlin.com@lists.yoctoproject.org> wrote:
>
> From: Michael Opdenacker <michael.opdenacker@bootlin.com>
>
> Note that this doesn't touch the "Source Fetching" section
> in overview-manual/concepts.rst yet, as the unpack implementation
> may not be finalized yet.
>
> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> ---
>  documentation/ref-manual/tasks.rst     | 8 ++++----
>  documentation/ref-manual/variables.rst | 5 +++++
>  2 files changed, 9 insertions(+), 4 deletions(-)
>
> diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst
> index 2e4b23408d..70c0dea3bb 100644
> --- a/documentation/ref-manual/tasks.rst
> +++ b/documentation/ref-manual/tasks.rst
> @@ -412,12 +412,12 @@ them. You can learn more by looking at the
>  -------------
>
>  Unpacks the source code into a working directory pointed to by
> -``${``\ :term:`WORKDIR`\ ``}``. The :term:`S`
> -variable also plays a role in where unpacked source files ultimately
> -reside. For more information on how source files are unpacked, see the
> +``${``\ :term:`UNPACKDIR`\ ``}``. Another, legacy way to specify
> +this directory is through the :term:`S` and :term:`WORKDIR` variables.

I find this sentence a little confusing to read.

"Another, legacy" suggests you've already given a method that was
legacy. But here you're really saying "Another way, which is legacy".
Perhaps this could be phrased differently, e.g. "A legacy way to
specify this directory is through..."

Also it doesn't really tell you how to use S and WORKDIR. I wonder if
you need to mention these at all, given they'll be described elsewhere
in the document. Why point a user to something legacy?

Thanks,

Andrew Murray

Re: [docs] [PATCH 2/3] ref-manual: introduce UNPACKDIR variable

[Michael Opdenacker]

Hi Andy

On 5/23/24 at 16:15, Andrew Murray wrote:
> Hi Michael,
>
> On Thu, 23 May 2024 at 14:56, Michael Opdenacker via
> lists.yoctoproject.org
> <michael.opdenacker=bootlin.com@lists.yoctoproject.org> wrote:
>> From: Michael Opdenacker <michael.opdenacker@bootlin.com>
>>
>> Note that this doesn't touch the "Source Fetching" section
>> in overview-manual/concepts.rst yet, as the unpack implementation
>> may not be finalized yet.
>>
>> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
>> ---
>>   documentation/ref-manual/tasks.rst     | 8 ++++----
>>   documentation/ref-manual/variables.rst | 5 +++++
>>   2 files changed, 9 insertions(+), 4 deletions(-)
>>
>> diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst
>> index 2e4b23408d..70c0dea3bb 100644
>> --- a/documentation/ref-manual/tasks.rst
>> +++ b/documentation/ref-manual/tasks.rst
>> @@ -412,12 +412,12 @@ them. You can learn more by looking at the
>>   -------------
>>
>>   Unpacks the source code into a working directory pointed to by
>> -``${``\ :term:`WORKDIR`\ ``}``. The :term:`S`
>> -variable also plays a role in where unpacked source files ultimately
>> -reside. For more information on how source files are unpacked, see the
>> +``${``\ :term:`UNPACKDIR`\ ``}``. Another, legacy way to specify
>> +this directory is through the :term:`S` and :term:`WORKDIR` variables.
> I find this sentence a little confusing to read.
>
> "Another, legacy" suggests you've already given a method that was
> legacy. But here you're really saying "Another way, which is legacy".
> Perhaps this could be phrased differently, e.g. "A legacy way to
> specify this directory is through..."
>
> Also it doesn't really tell you how to use S and WORKDIR. I wonder if
> you need to mention these at all, given they'll be described elsewhere
> in the document. Why point a user to something legacy?

This definitely makes sense, thanks!

I simplified my paragraph as follows:

Unpacks the source code into a working directory pointed to by
``${``\ :term:`UNPACKDIR`\ ``}``. A legacy way to specify
this directory is through the :term:`S` and :term:`WORKDIR` variables.
For more information on how source files are unpacked, see the
":ref:`overview-manual/concepts:source fetching`"
section in the Yocto Project Overview and Concepts Manual.

Thanks again
Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH 2/3] ref-manual: introduce UNPACKDIR variable

[Andrew Murray]

On Fri, 24 May 2024 at 09:26, Michael Opdenacker
<michael.opdenacker@bootlin.com> wrote:
>
> Hi Andy
>
> On 5/23/24 at 16:15, Andrew Murray wrote:
> > Hi Michael,
> >
> > On Thu, 23 May 2024 at 14:56, Michael Opdenacker via
> > lists.yoctoproject.org
> > <michael.opdenacker=bootlin.com@lists.yoctoproject.org> wrote:
> >> From: Michael Opdenacker <michael.opdenacker@bootlin.com>
> >>
> >> Note that this doesn't touch the "Source Fetching" section
> >> in overview-manual/concepts.rst yet, as the unpack implementation
> >> may not be finalized yet.
> >>
> >> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> >> ---
> >>   documentation/ref-manual/tasks.rst     | 8 ++++----
> >>   documentation/ref-manual/variables.rst | 5 +++++
> >>   2 files changed, 9 insertions(+), 4 deletions(-)
> >>
> >> diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst
> >> index 2e4b23408d..70c0dea3bb 100644
> >> --- a/documentation/ref-manual/tasks.rst
> >> +++ b/documentation/ref-manual/tasks.rst
> >> @@ -412,12 +412,12 @@ them. You can learn more by looking at the
> >>   -------------
> >>
> >>   Unpacks the source code into a working directory pointed to by
> >> -``${``\ :term:`WORKDIR`\ ``}``. The :term:`S`
> >> -variable also plays a role in where unpacked source files ultimately
> >> -reside. For more information on how source files are unpacked, see the
> >> +``${``\ :term:`UNPACKDIR`\ ``}``. Another, legacy way to specify
> >> +this directory is through the :term:`S` and :term:`WORKDIR` variables.
> > I find this sentence a little confusing to read.
> >
> > "Another, legacy" suggests you've already given a method that was
> > legacy. But here you're really saying "Another way, which is legacy".
> > Perhaps this could be phrased differently, e.g. "A legacy way to
> > specify this directory is through..."
> >
> > Also it doesn't really tell you how to use S and WORKDIR. I wonder if
> > you need to mention these at all, given they'll be described elsewhere
> > in the document. Why point a user to something legacy?
>
> This definitely makes sense, thanks!
>
> I simplified my paragraph as follows:
>
> Unpacks the source code into a working directory pointed to by
> ``${``\ :term:`UNPACKDIR`\ ``}``. A legacy way to specify
> this directory is through the :term:`S` and :term:`WORKDIR` variables.
> For more information on how source files are unpacked, see the
> ":ref:`overview-manual/concepts:source fetching`"
> section in the Yocto Project Overview and Concepts Manual.

That looks good. Feel free to add my:

Reviewed-by: Andrew Murray <amurray@thegoodpenguin.co.uk>

Re: [docs] [PATCH 2/3] ref-manual: introduce UNPACKDIR variable

[Michael Opdenacker]

On 5/24/24 at 12:01, Andrew Murray wrote:
> That looks good. Feel free to add my:
>
> Reviewed-by: Andrew Murray <amurray@thegoodpenguin.co.uk>


Done. Many thanks for the review!
Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[PATCH 3/3] migration-5.1.rst: preliminary description of UNPACKDIR changes

[michael.opdenacker]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 .../migration-guides/migration-5.1.rst        | 55 +++++++++++++++++++
 1 file changed, 55 insertions(+)

diff --git a/documentation/migration-guides/migration-5.1.rst b/documentation/migration-guides/migration-5.1.rst
index b4db04bcee..84ef35e28a 100644
--- a/documentation/migration-guides/migration-5.1.rst
+++ b/documentation/migration-guides/migration-5.1.rst
@@ -11,6 +11,61 @@ Project 5.1 Release (codename "styhead") from the prior release.
 
 .. _migration-5.1-supported-kernel-versions:
 
+:term:`WORKDIR` changes
+~~~~~~~~~~~~~~~~~~~~~~~
+
+S = ${WORKDIR} no longer supported
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If a recipe has :term:`S` set to be :term:`WORKDIR`, this is no longer
+supported, and an error will be issued. The recipe should be changed to::
+
+    S = "${WORKDIR}/sources"
+    UNPACKDIR = "${S}"
+
+Any :term:`WORKDIR` references where files from :term:`SRC_URI` are referenced
+should be changed to :term:`S`. These are commonly in :ref:`ref-tasks-compile`,
+:ref:`ref-tasks-compile`, :ref:`ref-tasks-install` and :term:`LIC_FILES_CHKSUM`.
+
+:term:`WORKDIR` references in recipes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:term:`WORKDIR` references in other recipes need auditing. If they reference
+files from :term:`SRC_URI`, they likely need changing to :term:`UNPACKDIR`.
+These are commonly in :ref:`ref-tasks-compile` and :ref:`ref-tasks-install`
+for things like service or configuration files. One unusual case is
+``${WORKDIR}/${BP}`` which should probably be set to ``${S}``.
+
+References to ``../`` in :term:`LIC_FILES_CHKSUM` or elsewhere may need changing
+to :term:`UNPACKDIR`. References to :term:`WORKDIR` in ``sed`` commands are
+usually left as they are.
+
+General notes
+^^^^^^^^^^^^^
+
+Files from :ref:`ref-tasks-unpack` now unpack to ``WORKDIR/sources-unpack/``
+rather than ``WORKDIR/``.
+
+If :term:`S` is set to a subdirectory under :term:`WORKDIR` and that
+subdirectory exists in ``sources-unpack`` after :ref:`ref-tasks-unpack` runs,
+it is moved to :term:`WORKDIR`. This means that ``S = "${WORKDIR}/${BP}"``,
+``S = "${WORKDIR}/git"`` and also deeper paths continue to work as expected
+without changes. We cannot use symlinks to do this as it breaks autotools
+based recipes. Keeping all sources under ``sources-unpack`` wasn't considered
+as it meant more invasive recipes changes. The key objective was separating the
+:ref:`ref-tasks-unpack` task output from :term:`WORKDIR`.
+
+Previously, :term:`S` was always created but after the recent changes it is no
+longer the case. This means the check in ``do_unpack_qa`` triggers where
+:term:`S` is not created by a recipe while it didn't happen before. This can
+require to add an :term:`S` definition to a recipe that only uses
+``file://`` :term:`SRC_URI` entries. Building C files from :term:`UNPACKDIR`
+without setting :term:`S` to point at it does not work as the debug prefix
+mapping doesn't handle that.
+
+``devtool``  and ``recipetool`` have been updated to handle this and their
+support for ``S = WORKDIR`` and ``oe-local-files`` has been removed.
+
 Supported kernel versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- 
2.34.1