[PATCH] base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS

[PATCH] base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS

[changqing.li]

From: Changqing Li <changqing.li@windriver.com>

When there are tarballs in local mirror file://, do_fetch will
create symlink under DL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
after user update path from localmirrorA to localmirrorB, because
do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz
then, do_upack will failed since the tarball cannot be found.

Signed-off-by: Changqing Li <changqing.li@windriver.com>
---
 meta/classes-global/base.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 0999b42daa..5094a1540e 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
 do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
 do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
 do_fetch[network] = "1"
+do_fetch[vardeps] += "PREMIRRORS MIRRORS"
 python base_do_fetch() {
 
     src_uri = (d.getVar('SRC_URI') or "").split()
-- 
2.25.1

Re: [OE-core] [PATCH] base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS

[Martin Jansa]

How does this work when sharing sstate between builders with different
PREMIRRORS/MIRRORS? Doesn't this cause all signatures to be different
as do_unpack does depend on do_fetch?

On Mon, Apr 15, 2024 at 10:30 AM Changqing Li via
lists.openembedded.org
<changqing.li=eng.windriver.com@lists.openembedded.org> wrote:
>
> From: Changqing Li <changqing.li@windriver.com>
>
> When there are tarballs in local mirror file://, do_fetch will
> create symlink under DL_DIR, eg:
> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
> after user update path from localmirrorA to localmirrorB, because
> do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
> be rerun, so the symlink will not update to:
> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz
> then, do_upack will failed since the tarball cannot be found.
>
> Signed-off-by: Changqing Li <changqing.li@windriver.com>
> ---
>  meta/classes-global/base.bbclass | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
> index 0999b42daa..5094a1540e 100644
> --- a/meta/classes-global/base.bbclass
> +++ b/meta/classes-global/base.bbclass
> @@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
>  do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
>  do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
>  do_fetch[network] = "1"
> +do_fetch[vardeps] += "PREMIRRORS MIRRORS"
>  python base_do_fetch() {
>
>      src_uri = (d.getVar('SRC_URI') or "").split()
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#198222): https://lists.openembedded.org/g/openembedded-core/message/198222
> Mute This Topic: https://lists.openembedded.org/mt/105532159/3617156
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [martin.jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH] base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS

[Richard Purdie]

On Mon, 2024-04-15 at 16:30 +0800, Changqing Li via lists.openembedded.org wrote:
> From: Changqing Li <changqing.li@windriver.com>
> 
> When there are tarballs in local mirror file://, do_fetch will
> create symlink under DL_DIR, eg:
> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
> after user update path from localmirrorA to localmirrorB, because
> do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
> be rerun, so the symlink will not update to:
> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz
> then, do_upack will failed since the tarball cannot be found.
> 
> Signed-off-by: Changqing Li <changqing.li@windriver.com>
> ---
>  meta/classes-global/base.bbclass | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
> index 0999b42daa..5094a1540e 100644
> --- a/meta/classes-global/base.bbclass
> +++ b/meta/classes-global/base.bbclass
> @@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
>  do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
>  do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
>  do_fetch[network] = "1"
> +do_fetch[vardeps] += "PREMIRRORS MIRRORS"
>  python base_do_fetch() {
>  
>      src_uri = (d.getVar('SRC_URI') or "").split()
> 

Changing PREMIRRORS or MIRRORS should not change the output of the
task, ever, so do_fetch shouldn't depend upon them.

Cheers,

Richard

Re: [OE-core] [PATCH] base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS

[Changqing Li]

On 4/15/24 18:05, Richard Purdie wrote:
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> On Mon, 2024-04-15 at 16:30 +0800, Changqing Li via lists.openembedded.org wrote:
>> From: Changqing Li <changqing.li@windriver.com>
>>
>> When there are tarballs in local mirror file://, do_fetch will
>> create symlink under DL_DIR, eg:
>> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
>> after user update path from localmirrorA to localmirrorB, because
>> do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
>> be rerun, so the symlink will not update to:
>> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz
>> then, do_upack will failed since the tarball cannot be found.
>>
>> Signed-off-by: Changqing Li <changqing.li@windriver.com>
>> ---
>>   meta/classes-global/base.bbclass | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
>> index 0999b42daa..5094a1540e 100644
>> --- a/meta/classes-global/base.bbclass
>> +++ b/meta/classes-global/base.bbclass
>> @@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
>>   do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
>>   do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
>>   do_fetch[network] = "1"
>> +do_fetch[vardeps] += "PREMIRRORS MIRRORS"
>>   python base_do_fetch() {
>>
>>       src_uri = (d.getVar('SRC_URI') or "").split()
>>
> Changing PREMIRRORS or MIRRORS should not change the output of the
> task, ever, so do_fetch shouldn't depend upon them.

The failure case I met is like this:

I have a project with PREMIRRORS setting, and the tarballs are under it.

PREMIRRORS:append = "file:///project/path/to/localmirrorA"

after first bitbake pseudo-native, there will a symlink under DL_DIR:

pseudo-prebuilt-2.33.tar.xz -> 
/project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz

Then, Since all the tarballs are moved to /project/path/to/localmirrorB

I changed PREMIRRORS:append = "file:///project/path/to/localmirrorB"

Then bitbake pseudo-native again,   do_fetch  will not rerun since no 
signature change.

do_unpack rerun since one dependency  change, it will try to unpack 
${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,

but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz 
still point to localmirrorA, but localmirrorA not exist

since localmirrorB replace it.  Then will cause do_unpck failure:

NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to 
/build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz: 
/build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directory

So this is why I want to add this depends,  and want to make do_fetch 
rerun and resymlink to /project/path/to/localmirrorB

Regards

Changqing

>
> Cheers,
>
> Richard
>

Re: [OE-core] [PATCH] base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS

[Alexander Kanavin]

I'm not sure why files in DL_DIR would symlink to the local
(pre)mirrors instead of making an independent copy? Mirrors can indeed
move or disappear altogether.

Can you look into where the symlinking happens, and whether that can
be modified or configured differently (maybe also the git history for
that code)?

Alex

On Tue, 16 Apr 2024 at 03:48, Changqing Li via lists.openembedded.org
<changqing.li=eng.windriver.com@lists.openembedded.org> wrote:
>
>
> On 4/15/24 18:05, Richard Purdie wrote:
> > CAUTION: This email comes from a non Wind River email account!
> > Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> > On Mon, 2024-04-15 at 16:30 +0800, Changqing Li via lists.openembedded.org wrote:
> >> From: Changqing Li <changqing.li@windriver.com>
> >>
> >> When there are tarballs in local mirror file://, do_fetch will
> >> create symlink under DL_DIR, eg:
> >> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
> >> after user update path from localmirrorA to localmirrorB, because
> >> do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
> >> be rerun, so the symlink will not update to:
> >> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz
> >> then, do_upack will failed since the tarball cannot be found.
> >>
> >> Signed-off-by: Changqing Li <changqing.li@windriver.com>
> >> ---
> >>   meta/classes-global/base.bbclass | 1 +
> >>   1 file changed, 1 insertion(+)
> >>
> >> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
> >> index 0999b42daa..5094a1540e 100644
> >> --- a/meta/classes-global/base.bbclass
> >> +++ b/meta/classes-global/base.bbclass
> >> @@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
> >>   do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
> >>   do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
> >>   do_fetch[network] = "1"
> >> +do_fetch[vardeps] += "PREMIRRORS MIRRORS"
> >>   python base_do_fetch() {
> >>
> >>       src_uri = (d.getVar('SRC_URI') or "").split()
> >>
> > Changing PREMIRRORS or MIRRORS should not change the output of the
> > task, ever, so do_fetch shouldn't depend upon them.
>
> The failure case I met is like this:
>
> I have a project with PREMIRRORS setting, and the tarballs are under it.
>
> PREMIRRORS:append = "file:///project/path/to/localmirrorA"
>
> after first bitbake pseudo-native, there will a symlink under DL_DIR:
>
> pseudo-prebuilt-2.33.tar.xz ->
> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
>
> Then, Since all the tarballs are moved to /project/path/to/localmirrorB
>
> I changed PREMIRRORS:append = "file:///project/path/to/localmirrorB"
>
> Then bitbake pseudo-native again,   do_fetch  will not rerun since no
> signature change.
>
> do_unpack rerun since one dependency  change, it will try to unpack
> ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,
>
> but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz
> still point to localmirrorA, but localmirrorA not exist
>
> since localmirrorB replace it.  Then will cause do_unpck failure:
>
> NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to
> /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz:
> /build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directory
>
> So this is why I want to add this depends,  and want to make do_fetch
> rerun and resymlink to /project/path/to/localmirrorB
>
> Regards
>
> Changqing
>
> >
> > Cheers,
> >
> > Richard
> >
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#198246): https://lists.openembedded.org/g/openembedded-core/message/198246
> Mute This Topic: https://lists.openembedded.org/mt/105532159/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH] base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS

[Changqing Li]

[-- Attachment #1: Type: text/plain, Size: 4835 bytes --]


On 4/16/24 15:30, Alexander Kanavin wrote:
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> I'm not sure why files in DL_DIR would symlink to the local
> (pre)mirrors instead of making an independent copy? Mirrors can indeed
> move or disappear altogether.
>
> Can you look into where the symlinking happens, and whether that can
> be modified or configured differently (maybe also the git history for
> that code)?

Current code is here: 
https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/__init__.py#n1096

It is created during handle the (pre)mirrors in do_fetch.  And the 
history long time ago:

https://git.openembedded.org/bitbake/commit/?id=83f9445081595630488a5d3e170d282a13836c98

About if we can change to independent copy, I need to do more research, 
welcome to provide me some

background and notes.

Regards

Changqing

>
> Alex
>
> On Tue, 16 Apr 2024 at 03:48, Changqing Li via lists.openembedded.org
> <changqing.li=eng.windriver.com@lists.openembedded.org>  wrote:
>>
>> On 4/15/24 18:05, Richard Purdie wrote:
>>> CAUTION: This email comes from a non Wind River email account!
>>> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>> On Mon, 2024-04-15 at 16:30 +0800, Changqing Li via lists.openembedded.org wrote:
>>>> From: Changqing Li<changqing.li@windriver.com>
>>>>
>>>> When there are tarballs in local mirrorfile://, do_fetch will
>>>> create symlink under DL_DIR, eg:
>>>> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
>>>> after user update path from localmirrorA to localmirrorB, because
>>>> do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
>>>> be rerun, so the symlink will not update to:
>>>> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz
>>>> then, do_upack will failed since the tarball cannot be found.
>>>>
>>>> Signed-off-by: Changqing Li<changqing.li@windriver.com>
>>>> ---
>>>>    meta/classes-global/base.bbclass | 1 +
>>>>    1 file changed, 1 insertion(+)
>>>>
>>>> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
>>>> index 0999b42daa..5094a1540e 100644
>>>> --- a/meta/classes-global/base.bbclass
>>>> +++ b/meta/classes-global/base.bbclass
>>>> @@ -139,6 +139,7 @@ do_fetch[file-checksums] ="${@bb.fetch.get_checksum_file_list(d)}"
>>>>    do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
>>>>    do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
>>>>    do_fetch[network] = "1"
>>>> +do_fetch[vardeps] += "PREMIRRORS MIRRORS"
>>>>    python base_do_fetch() {
>>>>
>>>>        src_uri = (d.getVar('SRC_URI') or "").split()
>>>>
>>> Changing PREMIRRORS or MIRRORS should not change the output of the
>>> task, ever, so do_fetch shouldn't depend upon them.
>> The failure case I met is like this:
>>
>> I have a project with PREMIRRORS setting, and the tarballs are under it.
>>
>> PREMIRRORS:append ="file:///project/path/to/localmirrorA"
>>
>> after first bitbake pseudo-native, there will a symlink under DL_DIR:
>>
>> pseudo-prebuilt-2.33.tar.xz ->
>> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
>>
>> Then, Since all the tarballs are moved to /project/path/to/localmirrorB
>>
>> I changed PREMIRRORS:append ="file:///project/path/to/localmirrorB"
>>
>> Then bitbake pseudo-native again,   do_fetch  will not rerun since no
>> signature change.
>>
>> do_unpack rerun since one dependency  change, it will try to unpack
>> ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,
>>
>> but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz
>> still point to localmirrorA, but localmirrorA not exist
>>
>> since localmirrorB replace it.  Then will cause do_unpck failure:
>>
>> NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to
>> /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz:
>> /build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directory
>>
>> So this is why I want to add this depends,  and want to make do_fetch
>> rerun and resymlink to /project/path/to/localmirrorB
>>
>> Regards
>>
>> Changqing
>>
>>> Cheers,
>>>
>>> Richard
>>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#198246):https://lists.openembedded.org/g/openembedded-core/message/198246
>> Mute This Topic:https://lists.openembedded.org/mt/105532159/1686489
>> Group Owner:openembedded-core+owner@lists.openembedded.org
>> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub  [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

[-- Attachment #2: Type: text/html, Size: 7314 bytes --]

Re: [OE-core] [PATCH] base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS

[Richard Purdie]

On Tue, 2024-04-16 at 09:48 +0800, Changqing Li wrote:
> On 4/15/24 18:05, Richard Purdie wrote:
> > CAUTION: This email comes from a non Wind River email account!
> > Do not click links or open attachments unless you recognize the
> > sender and know the content is safe.
> > 
> > On Mon, 2024-04-15 at 16:30 +0800, Changqing Li via
> > lists.openembedded.org wrote:
> > > From: Changqing Li <changqing.li@windriver.com>
> > > 
> > > When there are tarballs in local mirror file://, do_fetch will
> > > create symlink under DL_DIR, eg:
> > > pseudo-prebuilt-2.33.tar.xz ->
> > > /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
> > > after user update path from localmirrorA to localmirrorB, because
> > > do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
> > > be rerun, so the symlink will not update to:
> > > pseudo-prebuilt-2.33.tar.xz ->
> > > /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz
> > > then, do_upack will failed since the tarball cannot be found.
> > > 
> > > Signed-off-by: Changqing Li <changqing.li@windriver.com>
> > > ---
> > >   meta/classes-global/base.bbclass | 1 +
> > >   1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/meta/classes-global/base.bbclass b/meta/classes-
> > > global/base.bbclass
> > > index 0999b42daa..5094a1540e 100644
> > > --- a/meta/classes-global/base.bbclass
> > > +++ b/meta/classes-global/base.bbclass
> > > @@ -139,6 +139,7 @@ do_fetch[file-checksums] =
> > > "${@bb.fetch.get_checksum_file_list(d)}"
> > >   do_fetch[file-checksums] += "
> > > ${@get_lic_checksum_file_list(d)}"
> > >   do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
> > >   do_fetch[network] = "1"
> > > +do_fetch[vardeps] += "PREMIRRORS MIRRORS"
> > >   python base_do_fetch() {
> > > 
> > >       src_uri = (d.getVar('SRC_URI') or "").split()
> > > 
> > Changing PREMIRRORS or MIRRORS should not change the output of the
> > task, ever, so do_fetch shouldn't depend upon them.
> 
> The failure case I met is like this:
> 
> I have a project with PREMIRRORS setting, and the tarballs are under
> it.
> 
> PREMIRRORS:append = "file:///project/path/to/localmirrorA"
> 
> after first bitbake pseudo-native, there will a symlink under DL_DIR:
> 
> pseudo-prebuilt-2.33.tar.xz -> 
> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
> 
> Then, Since all the tarballs are moved to
> /project/path/to/localmirrorB
> 
> I changed PREMIRRORS:append = "file:///project/path/to/localmirrorB"
> 
> Then bitbake pseudo-native again,   do_fetch  will not rerun since no
> signature change.
> 
> do_unpack rerun since one dependency  change, it will try to unpack 
> ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,
> 
> but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz 
> still point to localmirrorA, but localmirrorA not exist
> 
> since localmirrorB replace it.  Then will cause do_unpck failure:
> 
> NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to 
> /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz: 
> /build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or
> directory
> 
> So this is why I want to add this depends,  and want to make do_fetch
> rerun and resymlink to /project/path/to/localmirrorB

The question is if the fetch reruns, does it correct the path?

do_fetch should definitely not depend on PREMIRRORS or MIRRORS as those
can be set by local users for local setups. The idea is that result
would be the same regardless, else the fetch checksums (or git
revision) would not match. I will therefore not take any patch changing
that.

Also, despite what Alex said, the fetcher does only make symlinks for
local files, it does not copy every file into DL_DIR. It does this for
efficiency, both time and disk usage.

If you start moving files around locally, you're expected to fix up the
links too. I believe that if you manually run the fetch task, it will
update correctly so the only issue is it not automatically triggering
but I don't believe it should trigger.

So whilst I can see how you have an unfortunate build failure, it was
of your own making by moving mirror directories around.

Cheers,

Richard

Re: [OE-core] [PATCH] base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS

[Changqing Li]

[-- Attachment #1: Type: text/plain, Size: 2232 bytes --]


On 4/15/24 16:30, Changqing Li wrote:
> From: Changqing Li<changqing.li@windriver.com>
>
> When there are tarballs in local mirrorfile://, do_fetch will
> create symlink under DL_DIR, eg:
> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
> after user update path from localmirrorA to localmirrorB, because
> do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
> be rerun, so the symlink will not update to:
> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz
> then, do_upack will failed since the tarball cannot be found.
>
> Signed-off-by: Changqing Li<changqing.li@windriver.com>
> ---
>   meta/classes-global/base.bbclass | 1 +
>   1 file changed, 1 insertion(+)
>
> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
> index 0999b42daa..5094a1540e 100644
> --- a/meta/classes-global/base.bbclass
> +++ b/meta/classes-global/base.bbclass
> @@ -139,6 +139,7 @@ do_fetch[file-checksums] ="${@bb.fetch.get_checksum_file_list(d)}"
>   do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
>   do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
>   do_fetch[network] = "1"
> +do_fetch[vardeps] += "PREMIRRORS MIRRORS"
>   python base_do_fetch() {
>   
>       src_uri = (d.getVar('SRC_URI') or "").split()

This fix has a shortcoming,  it will make do_fetch rerun in some 
unnecessary cases, eg:

PREMIRRORS changed from

PREMIRRORS = "git://.*/.* https://downloads.yoctoproject.org/mirror/sources"

to
PREMIRRORS = "git://.*/.* 
https://downloads.yoctoproject.org/mirror/sources \

ftp://.*/.* https://downloads.yoctoproject.org/mirror/sources ".


Any better suggestion? Thanks.

Regards

Sandy


>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#198222):https://lists.openembedded.org/g/openembedded-core/message/198222
> Mute This Topic:https://lists.openembedded.org/mt/105532159/3616873
> Group Owner:openembedded-core+owner@lists.openembedded.org
> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub  [changqing.li@eng.windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

[-- Attachment #2: Type: text/html, Size: 5459 bytes --]