[OE-core][PATCH v5] kernel.bbclass: make do_symlink_kernelsrc reentrant

[OE-core][PATCH v5] kernel.bbclass: make do_symlink_kernelsrc reentrant

[Etienne Cordonnier]

[-- Attachment #1: Type: text/plain, Size: 1776 bytes --]

From: Etienne Cordonnier <ecordonnier@snap.com>

The function do_symlink_kernsrc is not reentrant in the case where S is
defined
to a non-default value. This causes build-failures e.g. when building
linux-yocto, then updating
poky to a commit which modifies kernel.bbclass, and then building
linux-yocto again.

Bugzilla: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15325

Tested with a recipe "my-custom-linux" which unpacks sources to a custom
${S} directory
and ran symlink_kernsrc several times:
$ bitbake -f -c symlink_kernsrc my-custom-linux

Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
 meta/classes-recipe/kernel.bbclass | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/classes-recipe/kernel.bbclass
b/meta/classes-recipe/kernel.bbclass
index 9ff37f5c38..45b63f1fa1 100644
--- a/meta/classes-recipe/kernel.bbclass
+++ b/meta/classes-recipe/kernel.bbclass
@@ -189,11 +189,17 @@ python do_symlink_kernsrc () {
             # drop trailing slash, so that os.symlink(kernsrc, s) doesn't
use s as
             # directory name and fail
             s = s[:-1]
-        if d.getVar("EXTERNALSRC"):
+        if d.getVar("EXTERNALSRC") and not os.path.islink(s):
             # With EXTERNALSRC S will not be wiped so we can symlink to it
             os.symlink(s, kernsrc)
         else:
             import shutil
+            # perform idempotent/reentrant copy
+            s_copy = s + ".orig"
+            if not os.path.isdir(s_copy):
+                shutil.copytree(s, s_copy, ignore_dangling_symlinks=True)
+            bb.utils.remove(s, recurse=True)
+            shutil.copytree(s_copy, s, ignore_dangling_symlinks=True)
             shutil.move(s, kernsrc)
             os.symlink(kernsrc, s)
 }
-- 
2.34.1

[-- Attachment #2: Type: text/html, Size: 2245 bytes --]

Re: [OE-core][PATCH v5] kernel.bbclass: make do_symlink_kernelsrc reentrant

[Etienne Cordonnier]

[-- Attachment #1: Type: text/plain, Size: 3735 bytes --]

patch v5: added ignore_dangling_symlinks to avoid error when linux sources
contain dangling symlinks

On Thu, Dec 21, 2023 at 10:49 PM Etienne Cordonnier via
lists.openembedded.org <ecordonnier=snap.com@lists.openembedded.org> wrote:

> From: Etienne Cordonnier <ecordonnier@snap.com>
>
> The function do_symlink_kernsrc is not reentrant in the case where S is
> defined
> to a non-default value. This causes build-failures e.g. when building
> linux-yocto, then updating
> poky to a commit which modifies kernel.bbclass, and then building
> linux-yocto again.
>
> Bugzilla: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15325
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.yoctoproject.org_show-5Fbug.cgi-3Fid-3D15325&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=eo9dzHi2uXPV1dA2f4GoA6Y7QoDiG2TTF1R2YQ__SwnF1U1d0O2JxX3PprS_T9BN&s=e9FG4cG4GbLjCy8loEN6qVisSUax4Ngvd_Ok8TPonvY&e=>
>
> Tested with a recipe "my-custom-linux" which unpacks sources to a custom
> ${S} directory
> and ran symlink_kernsrc several times:
> $ bitbake -f -c symlink_kernsrc my-custom-linux
>
> Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
> ---
>  meta/classes-recipe/kernel.bbclass | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/kernel.bbclass
> b/meta/classes-recipe/kernel.bbclass
> index 9ff37f5c38..45b63f1fa1 100644
> --- a/meta/classes-recipe/kernel.bbclass
> +++ b/meta/classes-recipe/kernel.bbclass
> @@ -189,11 +189,17 @@ python do_symlink_kernsrc () {
>              # drop trailing slash, so that os.symlink(kernsrc, s) doesn't
> use s as
>              # directory name and fail
>              s = s[:-1]
> -        if d.getVar("EXTERNALSRC"):
> +        if d.getVar("EXTERNALSRC") and not os.path.islink(s):
>              # With EXTERNALSRC S will not be wiped so we can symlink to it
>              os.symlink(s, kernsrc)
>          else:
>              import shutil
> +            # perform idempotent/reentrant copy
> +            s_copy = s + ".orig"
> +            if not os.path.isdir(s_copy):
> +                shutil.copytree(s, s_copy, ignore_dangling_symlinks=True)
> +            bb.utils.remove(s, recurse=True)
> +            shutil.copytree(s_copy, s, ignore_dangling_symlinks=True)
>              shutil.move(s, kernsrc)
>              os.symlink(kernsrc, s)
>  }
> --
> 2.34.1
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#192850):
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_192850&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=eo9dzHi2uXPV1dA2f4GoA6Y7QoDiG2TTF1R2YQ__SwnF1U1d0O2JxX3PprS_T9BN&s=nSj1eesCBnUi3mKI3TDrQr4rYtnlCUqYquM2YdP3kwc&e=
> Mute This Topic:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_103308574_7048771&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=eo9dzHi2uXPV1dA2f4GoA6Y7QoDiG2TTF1R2YQ__SwnF1U1d0O2JxX3PprS_T9BN&s=xqoor0ld3IbCh7N9Dbyl9PAlnSZbl2j66iPA0PYKAl0&e=
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=eo9dzHi2uXPV1dA2f4GoA6Y7QoDiG2TTF1R2YQ__SwnF1U1d0O2JxX3PprS_T9BN&s=cvRKeVUNWw1O0f_mV9dU_qkoh5ifOuu8pqC7Wg81hSg&e=
> [ecordonnier@snap.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 5809 bytes --]

[OE-core][PATCH v5] kernel.bbclass: make do_symlink_kernelsrc reentrant

[ecordonnier]

From: Etienne Cordonnier <ecordonnier@snap.com>

The function do_symlink_kernsrc is not reentrant in the case where S is defined
to a non-default value. This causes build-failures e.g. when building linux-yocto, then updating
poky to a commit which modifies kernel.bbclass, and then building linux-yocto again.

Bugzilla: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15325

Tested with a recipe "my-custom-linux" which unpacks sources to a custom ${S} directory
and ran symlink_kernsrc several times:
$ bitbake -f -c symlink_kernsrc my-custom-linux

Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
 meta/classes-recipe/kernel.bbclass | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
index 9ff37f5c38..45b63f1fa1 100644
--- a/meta/classes-recipe/kernel.bbclass
+++ b/meta/classes-recipe/kernel.bbclass
@@ -189,11 +189,17 @@ python do_symlink_kernsrc () {
             # drop trailing slash, so that os.symlink(kernsrc, s) doesn't use s as
             # directory name and fail
             s = s[:-1]
-        if d.getVar("EXTERNALSRC"):
+        if d.getVar("EXTERNALSRC") and not os.path.islink(s):
             # With EXTERNALSRC S will not be wiped so we can symlink to it
             os.symlink(s, kernsrc)
         else:
             import shutil
+            # perform idempotent/reentrant copy
+            s_copy = s + ".orig"
+            if not os.path.isdir(s_copy):
+                shutil.copytree(s, s_copy, ignore_dangling_symlinks=True)
+            bb.utils.remove(s, recurse=True)
+            shutil.copytree(s_copy, s, ignore_dangling_symlinks=True)
             shutil.move(s, kernsrc)
             os.symlink(kernsrc, s)
 }
-- 
2.36.1.vfs.0.0

Re: [OE-core][PATCH v5] kernel.bbclass: make do_symlink_kernelsrc reentrant

[Etienne Cordonnier]

[-- Attachment #1: Type: text/plain, Size: 4320 bytes --]

Hi,
what is the status of this patch? Was it rejected?

Étienne

On Thu, Dec 21, 2023 at 10:50 PM Etienne Cordonnier via
lists.openembedded.org <ecordonnier=snap.com@lists.openembedded.org> wrote:

> patch v5: added ignore_dangling_symlinks to avoid error when linux sources
> contain dangling symlinks
>
> On Thu, Dec 21, 2023 at 10:49 PM Etienne Cordonnier via
> lists.openembedded.org
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openembedded.org&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=UCoMTsJKrnqkV-eQGGRGBeC-wIGTP7H0LRPvcTWkn9673LluHqkCpFInRctDEEct&s=Af3lZ-qHdVwBo6b8tlJ45PXYNer8GU6q0uxJ740caCA&e=>
> <ecordonnier=snap.com@lists.openembedded.org> wrote:
>
>> From: Etienne Cordonnier <ecordonnier@snap.com>
>>
>> The function do_symlink_kernsrc is not reentrant in the case where S is
>> defined
>> to a non-default value. This causes build-failures e.g. when building
>> linux-yocto, then updating
>> poky to a commit which modifies kernel.bbclass, and then building
>> linux-yocto again.
>>
>> Bugzilla: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15325
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.yoctoproject.org_show-5Fbug.cgi-3Fid-3D15325&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=eo9dzHi2uXPV1dA2f4GoA6Y7QoDiG2TTF1R2YQ__SwnF1U1d0O2JxX3PprS_T9BN&s=e9FG4cG4GbLjCy8loEN6qVisSUax4Ngvd_Ok8TPonvY&e=>
>>
>> Tested with a recipe "my-custom-linux" which unpacks sources to a custom
>> ${S} directory
>> and ran symlink_kernsrc several times:
>> $ bitbake -f -c symlink_kernsrc my-custom-linux
>>
>> Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
>> ---
>>  meta/classes-recipe/kernel.bbclass | 8 +++++++-
>>  1 file changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes-recipe/kernel.bbclass
>> b/meta/classes-recipe/kernel.bbclass
>> index 9ff37f5c38..45b63f1fa1 100644
>> --- a/meta/classes-recipe/kernel.bbclass
>> +++ b/meta/classes-recipe/kernel.bbclass
>> @@ -189,11 +189,17 @@ python do_symlink_kernsrc () {
>>              # drop trailing slash, so that os.symlink(kernsrc, s)
>> doesn't use s as
>>              # directory name and fail
>>              s = s[:-1]
>> -        if d.getVar("EXTERNALSRC"):
>> +        if d.getVar("EXTERNALSRC") and not os.path.islink(s):
>>              # With EXTERNALSRC S will not be wiped so we can symlink to
>> it
>>              os.symlink(s, kernsrc)
>>          else:
>>              import shutil
>> +            # perform idempotent/reentrant copy
>> +            s_copy = s + ".orig"
>> +            if not os.path.isdir(s_copy):
>> +                shutil.copytree(s, s_copy, ignore_dangling_symlinks=True)
>> +            bb.utils.remove(s, recurse=True)
>> +            shutil.copytree(s_copy, s, ignore_dangling_symlinks=True)
>>              shutil.move(s, kernsrc)
>>              os.symlink(kernsrc, s)
>>  }
>> --
>> 2.34.1
>>
>>
>>
>>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#192851):
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_192851&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=UCoMTsJKrnqkV-eQGGRGBeC-wIGTP7H0LRPvcTWkn9673LluHqkCpFInRctDEEct&s=B4VcMp4VsG3L9jLpib5LmpAeOIt_ttUjQEayF-XUy_4&e=
> Mute This Topic:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_103308574_7048771&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=UCoMTsJKrnqkV-eQGGRGBeC-wIGTP7H0LRPvcTWkn9673LluHqkCpFInRctDEEct&s=5tvkZnNNtvxAsWwN88Hi9dl8qljZl0T9lLH0iQgy0Z0&e=
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=UCoMTsJKrnqkV-eQGGRGBeC-wIGTP7H0LRPvcTWkn9673LluHqkCpFInRctDEEct&s=Ka7j1CI_uOKYxtXQbSnfvzJwM4X1moECnj3FEMOL_UI&e=
> [ecordonnier@snap.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 6690 bytes --]

Re: [OE-core][PATCH v5] kernel.bbclass: make do_symlink_kernelsrc reentrant

[Alexandre Belloni]

On 30/01/2024 16:39:27+0100, Etienne Cordonnier via lists.openembedded.org wrote:
> Hi,
> what is the status of this patch? Was it rejected?
> 

I'll test it again because I don't remember but I possibly add a
build/test failure.

> Étienne
> 
> On Thu, Dec 21, 2023 at 10:50 PM Etienne Cordonnier via
> lists.openembedded.org <ecordonnier=snap.com@lists.openembedded.org> wrote:
> 
> > patch v5: added ignore_dangling_symlinks to avoid error when linux sources
> > contain dangling symlinks
> >
> > On Thu, Dec 21, 2023 at 10:49 PM Etienne Cordonnier via
> > lists.openembedded.org
> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openembedded.org&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=UCoMTsJKrnqkV-eQGGRGBeC-wIGTP7H0LRPvcTWkn9673LluHqkCpFInRctDEEct&s=Af3lZ-qHdVwBo6b8tlJ45PXYNer8GU6q0uxJ740caCA&e=>
> > <ecordonnier=snap.com@lists.openembedded.org> wrote:
> >
> >> From: Etienne Cordonnier <ecordonnier@snap.com>
> >>
> >> The function do_symlink_kernsrc is not reentrant in the case where S is
> >> defined
> >> to a non-default value. This causes build-failures e.g. when building
> >> linux-yocto, then updating
> >> poky to a commit which modifies kernel.bbclass, and then building
> >> linux-yocto again.
> >>
> >> Bugzilla: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15325
> >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.yoctoproject.org_show-5Fbug.cgi-3Fid-3D15325&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=eo9dzHi2uXPV1dA2f4GoA6Y7QoDiG2TTF1R2YQ__SwnF1U1d0O2JxX3PprS_T9BN&s=e9FG4cG4GbLjCy8loEN6qVisSUax4Ngvd_Ok8TPonvY&e=>
> >>
> >> Tested with a recipe "my-custom-linux" which unpacks sources to a custom
> >> ${S} directory
> >> and ran symlink_kernsrc several times:
> >> $ bitbake -f -c symlink_kernsrc my-custom-linux
> >>
> >> Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
> >> ---
> >>  meta/classes-recipe/kernel.bbclass | 8 +++++++-
> >>  1 file changed, 7 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/meta/classes-recipe/kernel.bbclass
> >> b/meta/classes-recipe/kernel.bbclass
> >> index 9ff37f5c38..45b63f1fa1 100644
> >> --- a/meta/classes-recipe/kernel.bbclass
> >> +++ b/meta/classes-recipe/kernel.bbclass
> >> @@ -189,11 +189,17 @@ python do_symlink_kernsrc () {
> >>              # drop trailing slash, so that os.symlink(kernsrc, s)
> >> doesn't use s as
> >>              # directory name and fail
> >>              s = s[:-1]
> >> -        if d.getVar("EXTERNALSRC"):
> >> +        if d.getVar("EXTERNALSRC") and not os.path.islink(s):
> >>              # With EXTERNALSRC S will not be wiped so we can symlink to
> >> it
> >>              os.symlink(s, kernsrc)
> >>          else:
> >>              import shutil
> >> +            # perform idempotent/reentrant copy
> >> +            s_copy = s + ".orig"
> >> +            if not os.path.isdir(s_copy):
> >> +                shutil.copytree(s, s_copy, ignore_dangling_symlinks=True)
> >> +            bb.utils.remove(s, recurse=True)
> >> +            shutil.copytree(s_copy, s, ignore_dangling_symlinks=True)
> >>              shutil.move(s, kernsrc)
> >>              os.symlink(kernsrc, s)
> >>  }
> >> --
> >> 2.34.1
> >>
> >>
> >>
> >>
> > 
> >
> >

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#194496): https://lists.openembedded.org/g/openembedded-core/message/194496
> Mute This Topic: https://lists.openembedded.org/mt/104055728/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core][PATCH v5] kernel.bbclass: make do_symlink_kernelsrc reentrant

[Etienne Cordonnier]

[-- Attachment #1: Type: text/plain, Size: 6048 bytes --]

Thanks, note that based on the discussion at
https://lists.openembedded.org/g/openembedded-core/message/192676?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Acreated%2C0%2Creentrant%2C20%2C2%2C0%2C103208527
I indeed expect a build failure on an incremental build where S is
overridden, and the existing build directory has an "incorrect" structure
due to a previous build. In this case a one-time clean of the kernel is
required.

On Wed, Jan 31, 2024 at 5:53 PM Alexandre Belloni <
alexandre.belloni@bootlin.com> wrote:

> On 30/01/2024 16:39:27+0100, Etienne Cordonnier via lists.openembedded.org
> wrote:
> > Hi,
> > what is the status of this patch? Was it rejected?
> >
>
> I'll test it again because I don't remember but I possibly add a
> build/test failure.
>
> > Étienne
> >
> > On Thu, Dec 21, 2023 at 10:50 PM Etienne Cordonnier via
> > lists.openembedded.org <ecordonnier=snap.com@lists.openembedded.org>
> wrote:
> >
> > > patch v5: added ignore_dangling_symlinks to avoid error when linux
> sources
> > > contain dangling symlinks
> > >
> > > On Thu, Dec 21, 2023 at 10:49 PM Etienne Cordonnier via
> > > lists.openembedded.org
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openembedded.org&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=UCoMTsJKrnqkV-eQGGRGBeC-wIGTP7H0LRPvcTWkn9673LluHqkCpFInRctDEEct&s=Af3lZ-qHdVwBo6b8tlJ45PXYNer8GU6q0uxJ740caCA&e=
> >
> > > <ecordonnier=snap.com@lists.openembedded.org> wrote:
> > >
> > >> From: Etienne Cordonnier <ecordonnier@snap.com>
> > >>
> > >> The function do_symlink_kernsrc is not reentrant in the case where S
> is
> > >> defined
> > >> to a non-default value. This causes build-failures e.g. when building
> > >> linux-yocto, then updating
> > >> poky to a commit which modifies kernel.bbclass, and then building
> > >> linux-yocto again.
> > >>
> > >> Bugzilla:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.yoctoproject.org_show-5Fbug.cgi-3Fid-3D15325&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=iKgaTl7foinidQ3odGNYB-M4jWUTar8G6CFMlnChcA57K40tJPv4CtbC0vRLjJpq&s=01DEoIZCmVtfygqQ3U_vRyh1Y1G_g-TrnrOOEXd0ysU&e=
> > >> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.yoctoproject.org_show-5Fbug.cgi-3Fid-3D15325&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=eo9dzHi2uXPV1dA2f4GoA6Y7QoDiG2TTF1R2YQ__SwnF1U1d0O2JxX3PprS_T9BN&s=e9FG4cG4GbLjCy8loEN6qVisSUax4Ngvd_Ok8TPonvY&e=
> >
> > >>
> > >> Tested with a recipe "my-custom-linux" which unpacks sources to a
> custom
> > >> ${S} directory
> > >> and ran symlink_kernsrc several times:
> > >> $ bitbake -f -c symlink_kernsrc my-custom-linux
> > >>
> > >> Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
> > >> ---
> > >>  meta/classes-recipe/kernel.bbclass | 8 +++++++-
> > >>  1 file changed, 7 insertions(+), 1 deletion(-)
> > >>
> > >> diff --git a/meta/classes-recipe/kernel.bbclass
> > >> b/meta/classes-recipe/kernel.bbclass
> > >> index 9ff37f5c38..45b63f1fa1 100644
> > >> --- a/meta/classes-recipe/kernel.bbclass
> > >> +++ b/meta/classes-recipe/kernel.bbclass
> > >> @@ -189,11 +189,17 @@ python do_symlink_kernsrc () {
> > >>              # drop trailing slash, so that os.symlink(kernsrc, s)
> > >> doesn't use s as
> > >>              # directory name and fail
> > >>              s = s[:-1]
> > >> -        if d.getVar("EXTERNALSRC"):
> > >> +        if d.getVar("EXTERNALSRC") and not os.path.islink(s):
> > >>              # With EXTERNALSRC S will not be wiped so we can symlink
> to
> > >> it
> > >>              os.symlink(s, kernsrc)
> > >>          else:
> > >>              import shutil
> > >> +            # perform idempotent/reentrant copy
> > >> +            s_copy = s + ".orig"
> > >> +            if not os.path.isdir(s_copy):
> > >> +                shutil.copytree(s, s_copy,
> ignore_dangling_symlinks=True)
> > >> +            bb.utils.remove(s, recurse=True)
> > >> +            shutil.copytree(s_copy, s, ignore_dangling_symlinks=True)
> > >>              shutil.move(s, kernsrc)
> > >>              os.symlink(kernsrc, s)
> > >>  }
> > >> --
> > >> 2.34.1
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> > >
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#194496):
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_194496&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=iKgaTl7foinidQ3odGNYB-M4jWUTar8G6CFMlnChcA57K40tJPv4CtbC0vRLjJpq&s=G7I8YhqpwqafPUBBHcpFpmThNk5vTFfDQAA3EdTet9U&e=
> > Mute This Topic:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_104055728_3617179&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=iKgaTl7foinidQ3odGNYB-M4jWUTar8G6CFMlnChcA57K40tJPv4CtbC0vRLjJpq&s=7KEXcFZJh3ku2w0FRlLPGNz6p-IBgg7CaGGFAfBHrnA&e=
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=iKgaTl7foinidQ3odGNYB-M4jWUTar8G6CFMlnChcA57K40tJPv4CtbC0vRLjJpq&s=Y9laBkE0P4fGE0KBQVjm585hG7rjyw1Zh2YxwQ5crX0&e=
> [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bootlin.com&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=iKgaTl7foinidQ3odGNYB-M4jWUTar8G6CFMlnChcA57K40tJPv4CtbC0vRLjJpq&s=3OVhEgy2jWuMpzbOdK2jsZDChCURAw-6geaZ2FOllTs&e=
>

[-- Attachment #2: Type: text/html, Size: 10940 bytes --]

Re: [OE-core][PATCH v5] kernel.bbclass: make do_symlink_kernelsrc reentrant

[Alexandre Belloni]

On 31/01/2024 18:17:55+0100, Etienne Cordonnier via lists.openembedded.org wrote:
> Thanks, note that based on the discussion at
> https://lists.openembedded.org/g/openembedded-core/message/192676?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Acreated%2C0%2Creentrant%2C20%2C2%2C0%2C103208527
> I indeed expect a build failure on an incremental build where S is
> overridden, and the existing build directory has an "incorrect" structure
> due to a previous build. In this case a one-time clean of the kernel is
> required.

Actually, I've been carrying the patch for a while and it is waiting to
be reviewed/applied.

> 
> On Wed, Jan 31, 2024 at 5:53 PM Alexandre Belloni <
> alexandre.belloni@bootlin.com> wrote:
> 
> > On 30/01/2024 16:39:27+0100, Etienne Cordonnier via lists.openembedded.org
> > wrote:
> > > Hi,
> > > what is the status of this patch? Was it rejected?
> > >
> >
> > I'll test it again because I don't remember but I possibly add a
> > build/test failure.
> >
> > > Étienne
> > >
> > > On Thu, Dec 21, 2023 at 10:50 PM Etienne Cordonnier via
> > > lists.openembedded.org <ecordonnier=snap.com@lists.openembedded.org>
> > wrote:
> > >
> > > > patch v5: added ignore_dangling_symlinks to avoid error when linux
> > sources
> > > > contain dangling symlinks
> > > >
> > > > On Thu, Dec 21, 2023 at 10:49 PM Etienne Cordonnier via
> > > > lists.openembedded.org
> > > > <
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openembedded.org&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=UCoMTsJKrnqkV-eQGGRGBeC-wIGTP7H0LRPvcTWkn9673LluHqkCpFInRctDEEct&s=Af3lZ-qHdVwBo6b8tlJ45PXYNer8GU6q0uxJ740caCA&e=
> > >
> > > > <ecordonnier=snap.com@lists.openembedded.org> wrote:
> > > >
> > > >> From: Etienne Cordonnier <ecordonnier@snap.com>
> > > >>
> > > >> The function do_symlink_kernsrc is not reentrant in the case where S
> > is
> > > >> defined
> > > >> to a non-default value. This causes build-failures e.g. when building
> > > >> linux-yocto, then updating
> > > >> poky to a commit which modifies kernel.bbclass, and then building
> > > >> linux-yocto again.
> > > >>
> > > >> Bugzilla:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.yoctoproject.org_show-5Fbug.cgi-3Fid-3D15325&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=iKgaTl7foinidQ3odGNYB-M4jWUTar8G6CFMlnChcA57K40tJPv4CtbC0vRLjJpq&s=01DEoIZCmVtfygqQ3U_vRyh1Y1G_g-TrnrOOEXd0ysU&e=
> > > >> <
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.yoctoproject.org_show-5Fbug.cgi-3Fid-3D15325&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=eo9dzHi2uXPV1dA2f4GoA6Y7QoDiG2TTF1R2YQ__SwnF1U1d0O2JxX3PprS_T9BN&s=e9FG4cG4GbLjCy8loEN6qVisSUax4Ngvd_Ok8TPonvY&e=
> > >
> > > >>
> > > >> Tested with a recipe "my-custom-linux" which unpacks sources to a
> > custom
> > > >> ${S} directory
> > > >> and ran symlink_kernsrc several times:
> > > >> $ bitbake -f -c symlink_kernsrc my-custom-linux
> > > >>
> > > >> Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
> > > >> ---
> > > >>  meta/classes-recipe/kernel.bbclass | 8 +++++++-
> > > >>  1 file changed, 7 insertions(+), 1 deletion(-)
> > > >>
> > > >> diff --git a/meta/classes-recipe/kernel.bbclass
> > > >> b/meta/classes-recipe/kernel.bbclass
> > > >> index 9ff37f5c38..45b63f1fa1 100644
> > > >> --- a/meta/classes-recipe/kernel.bbclass
> > > >> +++ b/meta/classes-recipe/kernel.bbclass
> > > >> @@ -189,11 +189,17 @@ python do_symlink_kernsrc () {
> > > >>              # drop trailing slash, so that os.symlink(kernsrc, s)
> > > >> doesn't use s as
> > > >>              # directory name and fail
> > > >>              s = s[:-1]
> > > >> -        if d.getVar("EXTERNALSRC"):
> > > >> +        if d.getVar("EXTERNALSRC") and not os.path.islink(s):
> > > >>              # With EXTERNALSRC S will not be wiped so we can symlink
> > to
> > > >> it
> > > >>              os.symlink(s, kernsrc)
> > > >>          else:
> > > >>              import shutil
> > > >> +            # perform idempotent/reentrant copy
> > > >> +            s_copy = s + ".orig"
> > > >> +            if not os.path.isdir(s_copy):
> > > >> +                shutil.copytree(s, s_copy,
> > ignore_dangling_symlinks=True)
> > > >> +            bb.utils.remove(s, recurse=True)
> > > >> +            shutil.copytree(s_copy, s, ignore_dangling_symlinks=True)
> > > >>              shutil.move(s, kernsrc)
> > > >>              os.symlink(kernsrc, s)
> > > >>  }
> > > >> --
> > > >> 2.34.1
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > > >
> >
> > >
> > > 
> > >
> >
> >
> > --
> > Alexandre Belloni, co-owner and COO, Bootlin
> > Embedded Linux and Kernel engineering
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__bootlin.com&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=iKgaTl7foinidQ3odGNYB-M4jWUTar8G6CFMlnChcA57K40tJPv4CtbC0vRLjJpq&s=3OVhEgy2jWuMpzbOdK2jsZDChCURAw-6geaZ2FOllTs&e=
> >

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#194531): https://lists.openembedded.org/g/openembedded-core/message/194531
> Mute This Topic: https://lists.openembedded.org/mt/104055728/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core][PATCH v5] kernel.bbclass: make do_symlink_kernelsrc reentrant

[Richard Purdie]

On Thu, 2023-12-21 at 22:49 +0100, Etienne Cordonnier via
lists.openembedded.org wrote:
> From: Etienne Cordonnier <ecordonnier@snap.com>
> 
> The function do_symlink_kernsrc is not reentrant in the case where S is defined
> to a non-default value. This causes build-failures e.g. when building linux-yocto, then updating
> poky to a commit which modifies kernel.bbclass, and then building linux-yocto again.
> 
> Bugzilla: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15325
> 
> Tested with a recipe "my-custom-linux" which unpacks sources to a custom ${S} directory
> and ran symlink_kernsrc several times:
> $ bitbake -f -c symlink_kernsrc my-custom-linux

Sorry for the delay in getting back to review this patch. I'm extremely
worried about adding complexity into this function, particularly as
that complexity is now adding in significant overhead.

Firstly, can I ask why you're using a non-default directory for ${S}?
Is this as a way of doing a kind of external source usage?

Having spent time looking at what this code is doing, in normal usage,
S gets set to STAGING_KERNEL_DIR, the source is unpacked there and none
of this code triggers.

For EXTERNALSRC, a symlink is put in place. The code should remove a
symlink if present and create it with the current setup. In that sense,
this patch isn't doing the right thing.

I'm very tempted to say the "using a non default S value" just error
out.

Whilst I understand how we got to this level of complexity, I think it
will just end up causing us pain in the long run and I'd rather just
not support it.

Cheers,

Richard

Re: [OE-core][PATCH v5] kernel.bbclass: make do_symlink_kernelsrc reentrant

[Etienne Cordonnier]

[-- Attachment #1: Type: text/plain, Size: 3038 bytes --]

Hi Richard,

I also apologize for replying late. I was busy with other things and
haven't found time to rework this patch.

> Firstly, can I ask why you're using a non-default directory for ${S}?
> Is this as a way of doing a kind of external source usage?

My use-case is that the BSP I am using includes the source of the kernel as
source alongside the BSP yocto layer (not in an extra git repository), and
then points S to the directory containing the sources of the kernel without
using the externalsrc class (as far as I know the externalsrc class isn't
meant for this use-case, since there are several recipes provided as source
in this way, and externalsrc only has one path pointing to the external
source).

I agree with you that the complexity of this function is very high, so not
supporting a non-default S would be a better solution in this case. I'll
try to rework the patch and test it with externalsrc as well.

Étienne

On Fri, Feb 9, 2024 at 6:36 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Thu, 2023-12-21 at 22:49 +0100, Etienne Cordonnier via
> lists.openembedded.org wrote:
> > From: Etienne Cordonnier <ecordonnier@snap.com>
> >
> > The function do_symlink_kernsrc is not reentrant in the case where S is
> defined
> > to a non-default value. This causes build-failures e.g. when building
> linux-yocto, then updating
> > poky to a commit which modifies kernel.bbclass, and then building
> linux-yocto again.
> >
> > Bugzilla:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.yoctoproject.org_show-5Fbug.cgi-3Fid-3D15325&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=e0iYk69Iih3UnIjABITtc6yS38bhv_6P4NMuSpVmmoSHk1-sSuvH3y702O0nTeZj&s=OtiPnSuoZjUXFgz9pxbGOOyjQmKPEq-OhEbhtsAmvZE&e=
> >
> > Tested with a recipe "my-custom-linux" which unpacks sources to a custom
> ${S} directory
> > and ran symlink_kernsrc several times:
> > $ bitbake -f -c symlink_kernsrc my-custom-linux
>
> Sorry for the delay in getting back to review this patch. I'm extremely
> worried about adding complexity into this function, particularly as
> that complexity is now adding in significant overhead.
>
> Firstly, can I ask why you're using a non-default directory for ${S}?
> Is this as a way of doing a kind of external source usage?
>
> Having spent time looking at what this code is doing, in normal usage,
> S gets set to STAGING_KERNEL_DIR, the source is unpacked there and none
> of this code triggers.
>
> For EXTERNALSRC, a symlink is put in place. The code should remove a
> symlink if present and create it with the current setup. In that sense,
> this patch isn't doing the right thing.
>
> I'm very tempted to say the "using a non default S value" just error
> out.
>
> Whilst I understand how we got to this level of complexity, I think it
> will just end up causing us pain in the long run and I'd rather just
> not support it.
>
> Cheers,
>
> Richard
>

[-- Attachment #2: Type: text/html, Size: 4085 bytes --]

Re: [OE-core][PATCH v5] kernel.bbclass: make do_symlink_kernelsrc reentrant

[Alexandre Belloni]

On 27/05/2024 11:45:58+0200, Etienne Cordonnier via lists.openembedded.org wrote:
> Hi Richard,
> 
> I also apologize for replying late. I was busy with other things and
> haven't found time to rework this patch.
> 
> > Firstly, can I ask why you're using a non-default directory for ${S}?
> > Is this as a way of doing a kind of external source usage?
> 
> My use-case is that the BSP I am using includes the source of the kernel as
> source alongside the BSP yocto layer (not in an extra git repository), and
> then points S to the directory containing the sources of the kernel without
> using the externalsrc class (as far as I know the externalsrc class isn't
> meant for this use-case, since there are several recipes provided as source
> in this way, and externalsrc only has one path pointing to the external
> source).
> 

You should rather educate your vendor to provide a proper git repository
for the kernel.

> I agree with you that the complexity of this function is very high, so not
> supporting a non-default S would be a better solution in this case. I'll
> try to rework the patch and test it with externalsrc as well.
> 
> Étienne
> 
> On Fri, Feb 9, 2024 at 6:36 PM Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> 
> > On Thu, 2023-12-21 at 22:49 +0100, Etienne Cordonnier via
> > lists.openembedded.org wrote:
> > > From: Etienne Cordonnier <ecordonnier@snap.com>
> > >
> > > The function do_symlink_kernsrc is not reentrant in the case where S is
> > defined
> > > to a non-default value. This causes build-failures e.g. when building
> > linux-yocto, then updating
> > > poky to a commit which modifies kernel.bbclass, and then building
> > linux-yocto again.
> > >
> > > Bugzilla:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.yoctoproject.org_show-5Fbug.cgi-3Fid-3D15325&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=e0iYk69Iih3UnIjABITtc6yS38bhv_6P4NMuSpVmmoSHk1-sSuvH3y702O0nTeZj&s=OtiPnSuoZjUXFgz9pxbGOOyjQmKPEq-OhEbhtsAmvZE&e=
> > >
> > > Tested with a recipe "my-custom-linux" which unpacks sources to a custom
> > ${S} directory
> > > and ran symlink_kernsrc several times:
> > > $ bitbake -f -c symlink_kernsrc my-custom-linux
> >
> > Sorry for the delay in getting back to review this patch. I'm extremely
> > worried about adding complexity into this function, particularly as
> > that complexity is now adding in significant overhead.
> >
> > Firstly, can I ask why you're using a non-default directory for ${S}?
> > Is this as a way of doing a kind of external source usage?
> >
> > Having spent time looking at what this code is doing, in normal usage,
> > S gets set to STAGING_KERNEL_DIR, the source is unpacked there and none
> > of this code triggers.
> >
> > For EXTERNALSRC, a symlink is put in place. The code should remove a
> > symlink if present and create it with the current setup. In that sense,
> > this patch isn't doing the right thing.
> >
> > I'm very tempted to say the "using a non default S value" just error
> > out.
> >
> > Whilst I understand how we got to this level of complexity, I think it
> > will just end up causing us pain in the long run and I'd rather just
> > not support it.
> >
> > Cheers,
> >
> > Richard
> >

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#199898): https://lists.openembedded.org/g/openembedded-core/message/199898
> Mute This Topic: https://lists.openembedded.org/mt/103308574/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com