* [REGRESSION] linux (git/curl-native) and autorev
@ 2024-03-09 17:57 max.oss.09
2024-03-09 21:03 ` Bruce Ashfield
0 siblings, 1 reply; 10+ messages in thread
From: max.oss.09 @ 2024-03-09 17:57 UTC (permalink / raw)
To: openembedded-core; +Cc: bruce.ashfield, Max Krummenacher
From: Max Krummenacher <max.krummenacher@toradex.com>
Hello
If one builds a kernel using AUTOREV invoking bitbake only works once.
Any subsequent bitbake invocation fails parsing the meta data.
Reproducable with:
- latest poky, b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf
`SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"`
- bitbake virtual/kernel; bitbake virtual/kernel
On the second invocation parsing fails when the fetcher tries to
evaluate the latest SRCREV:
| ERROR: ExpansionError during parsing meta/recipes-kernel/linux/linux-yocto_6.6.bb
| Traceback (most recent call last):
| File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
| d.setVar("__BBAUTOREV_ACTED_UPON", True)
| > srcrev = ud.method.latest_revision(ud, d, name)
|
| File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
| except KeyError:
| > revs[key] = rev = self._latest_revision(ud, d, name)
| return rev
| File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
|
| > output = self._lsremote(ud, d, "")
| # Tags of the form ^{} may not work, need to fallback to other form
| File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
| bb.fetch2.check_network_access(d, cmd, repourl)
| > output = runfetchcmd(cmd, d, True)
| if not output:
| File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):
|
| > raise FetchError(error_message)
|
| bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git failed with exit code 128, output:
| fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting certificate file: build/tmp/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
|
| The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]
Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-native
`rm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/git`
Bisecting poky leads to commit
f7fa98cca8 ("kern-tools: depend on git-replacement-native")
Reverting it on top of b5624ee564 makes the parsing pass.
I assume that `git-replacement-native` does not work with https, the
fetch error also goes away if changing in SRC_URI from https to git.
Any comments?
Cheers
Max
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [REGRESSION] linux (git/curl-native) and autorev
2024-03-09 17:57 [REGRESSION] linux (git/curl-native) and autorev max.oss.09
@ 2024-03-09 21:03 ` Bruce Ashfield
2024-03-10 12:31 ` [OE-core] " Max
0 siblings, 1 reply; 10+ messages in thread
From: Bruce Ashfield @ 2024-03-09 21:03 UTC (permalink / raw)
To: max.oss.09, Richard Purdie; +Cc: openembedded-core, Max Krummenacher
On Sat, Mar 9, 2024 at 12:58 PM <max.oss.09@gmail.com> wrote:
>
> From: Max Krummenacher <max.krummenacher@toradex.com>
>
> Hello
>
> If one builds a kernel using AUTOREV invoking bitbake only works once.
> Any subsequent bitbake invocation fails parsing the meta data.
>
> Reproducable with:
> - latest poky, b5624ee564
> - Kernel with SRCREV = "AUTOREV", e.g. in local.conf
> `SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"`
> - bitbake virtual/kernel; bitbake virtual/kernel
>
> On the second invocation parsing fails when the fetcher tries to
> evaluate the latest SRCREV:
>
> | ERROR: ExpansionError during parsing meta/recipes-kernel/linux/linux-yocto_6.6.bb
> | Traceback (most recent call last):
> | File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
> | d.setVar("__BBAUTOREV_ACTED_UPON", True)
> | > srcrev = ud.method.latest_revision(ud, d, name)
> |
> | File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
> | except KeyError:
> | > revs[key] = rev = self._latest_revision(ud, d, name)
> | return rev
> | File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
> |
> | > output = self._lsremote(ud, d, "")
> | # Tags of the form ^{} may not work, need to fallback to other form
> | File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
> | bb.fetch2.check_network_access(d, cmd, repourl)
> | > output = runfetchcmd(cmd, d, True)
> | if not output:
> | File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):
> |
> | > raise FetchError(error_message)
> |
> | bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git failed with exit code 128, output:
> | fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting certificate file: build/tmp/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
> |
> | The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]
>
> Note:
> One gets out of that parser error by deleting the git binary in the
> kernel's work recipe-sysroot-native
> `rm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/git`
>
> Bisecting poky leads to commit
> f7fa98cca8 ("kern-tools: depend on git-replacement-native")
> Reverting it on top of b5624ee564 makes the parsing pass.
>
> I assume that `git-replacement-native` does not work with https, the
> fetch error also goes away if changing in SRC_URI from https to git.
>
> Any comments?
I didn't even know that curl was coming into play :)
Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
should resolve the issue.
I also tried exporting the SSL_CERT_DIR in the git-native wrapper, but
that didn't help
since of course it is curl that is looking for the certs.
I also thought we could export the SSL_CERT_DIR from within the
kern-tools, but really,
it does seem like curl should know that this is a dependency when used
as a native
tool (it already has ca-certficates as a RRECOMMENDS).
I haven't done a patch for it, because the intricacies of the native
classes and where
we should actually add that depends is someone lost on me.
Richard will likely know the right place to fix it .. and then I'm
happy to spin a patch.
Bruce
>
> Cheers
> Max
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OE-core] [REGRESSION] linux (git/curl-native) and autorev
2024-03-09 21:03 ` Bruce Ashfield
@ 2024-03-10 12:31 ` Max
2024-03-10 13:20 ` Richard Purdie
[not found] ` <17BB69D476D2F4EF.5850@lists.openembedded.org>
0 siblings, 2 replies; 10+ messages in thread
From: Max @ 2024-03-10 12:31 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: openembedded-core, Max Krummenacher
Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
> On Sat, Mar 9, 2024 at 12:58 PM <max.oss.09@gmail.com> wrote:
> >
> > From: Max Krummenacher <max.krummenacher@toradex.com>
> >
> > Hello
> >
> > If one builds a kernel using AUTOREV invoking bitbake only works once.
> > Any subsequent bitbake invocation fails parsing the meta data.
> >
> > Reproducable with:
> > - latest poky, b5624ee564
> > - Kernel with SRCREV = "AUTOREV", e.g. in local.conf
> > `SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"`
> > - bitbake virtual/kernel; bitbake virtual/kernel
> >
> > On the second invocation parsing fails when the fetcher tries to
> > evaluate the latest SRCREV:
> >
> > > ERROR: ExpansionError during parsing meta/recipes-kernel/linux/linux-yocto_6.6.bb
> > > Traceback (most recent call last):
> > > File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
> > > d.setVar("__BBAUTOREV_ACTED_UPON", True)
> > > > srcrev = ud.method.latest_revision(ud, d, name)
> > >
> > > File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
> > > except KeyError:
> > > > revs[key] = rev = self._latest_revision(ud, d, name)
> > > return rev
> > > File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
> > >
> > > > output = self._lsremote(ud, d, "")
> > > # Tags of the form ^{} may not work, need to fallback to other form
> > > File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
> > > bb.fetch2.check_network_access(d, cmd, repourl)
> > > > output = runfetchcmd(cmd, d, True)
> > > if not output:
> > > File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):
> > >
> > > > raise FetchError(error_message)
> > >
> > > bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git failed with exit code 128, output:
> > > fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting certificate file: build/tmp/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
> > >
> > > The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]
> >
> > Note:
> > One gets out of that parser error by deleting the git binary in the
> > kernel's work recipe-sysroot-native
> > `rm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/git`
> >
> > Bisecting poky leads to commit
> > f7fa98cca8 ("kern-tools: depend on git-replacement-native")
> > Reverting it on top of b5624ee564 makes the parsing pass.
> >
> > I assume that `git-replacement-native` does not work with https, the
> > fetch error also goes away if changing in SRC_URI from https to git.
> >
> > Any comments?
>
> I didn't even know that curl was coming into play :)
>
> Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
> should resolve the issue.
Looks like curl-native resp. libcurl hardcodes the lookup to its own
work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate or
rm_work is in INHERIT.
Max
>
> I also tried exporting the SSL_CERT_DIR in the git-native wrapper, but
> that didn't help
> since of course it is curl that is looking for the certs.
>
> I also thought we could export the SSL_CERT_DIR from within the
> kern-tools, but really,
> it does seem like curl should know that this is a dependency when used
> as a native
> tool (it already has ca-certficates as a RRECOMMENDS).
>
> I haven't done a patch for it, because the intricacies of the native
> classes and where
> we should actually add that depends is someone lost on me.
>
> Richard will likely know the right place to fix it .. and then I'm
> happy to spin a patch.
>
> Bruce
>
> >
> > Cheers
> > Max
>
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#196873): https://lists.openembedded.org/g/openembedded-core/message/196873
> Mute This Topic: https://lists.openembedded.org/mt/104831542/3617484
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [max.oss.09@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OE-core] [REGRESSION] linux (git/curl-native) and autorev
2024-03-10 12:31 ` [OE-core] " Max
@ 2024-03-10 13:20 ` Richard Purdie
[not found] ` <17BB69D476D2F4EF.5850@lists.openembedded.org>
1 sibling, 0 replies; 10+ messages in thread
From: Richard Purdie @ 2024-03-10 13:20 UTC (permalink / raw)
To: Max, Bruce Ashfield; +Cc: openembedded-core, Max Krummenacher
On Sun, 2024-03-10 at 13:31 +0100, Max wrote:
> Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
> > On Sat, Mar 9, 2024 at 12:58 PM <max.oss.09@gmail.com> wrote:
> > >
> > > From: Max Krummenacher <max.krummenacher@toradex.com>
> > >
> > > Hello
> > >
> > > If one builds a kernel using AUTOREV invoking bitbake only works
> > > once.
> > > Any subsequent bitbake invocation fails parsing the meta data.
> > >
> > > Reproducable with:
> > > - latest poky, b5624ee564
> > > - Kernel with SRCREV = "AUTOREV", e.g. in local.conf
> > > `SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"`
> > > - bitbake virtual/kernel; bitbake virtual/kernel
> > >
> > > On the second invocation parsing fails when the fetcher tries to
> > > evaluate the latest SRCREV:
> > >
> > > > ERROR: ExpansionError during parsing meta/recipes-
> > > > kernel/linux/linux-yocto_6.6.bb
> > > > Traceback (most recent call last):
> > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in
> > > > srcrev_internal_helper(ud=<bb.fetch2.FetchData object at
> > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > 0x7f8e26195890>, name='machine'):
> > > > d.setVar("__BBAUTOREV_ACTED_UPON", True)
> > > > > srcrev = ud.method.latest_revision(ud, d, name)
> > > >
> > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in
> > > > Git.latest_revision(ud=<bb.fetch2.FetchData object at
> > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > 0x7f8e26195890>, name='machine'):
> > > > except KeyError:
> > > > > revs[key] = rev = self._latest_revision(ud, d,
> > > > name)
> > > > return rev
> > > > File "bitbake/lib/bb/fetch2/git.py", line 850, in
> > > > Git._latest_revision(ud=<bb.fetch2.FetchData object at
> > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > 0x7f8e26195890>, name='machine'):
> > > >
> > > > > output = self._lsremote(ud, d, "")
> > > > # Tags of the form ^{} may not work, need to
> > > > fallback to other form
> > > > File "bitbake/lib/bb/fetch2/git.py", line 833, in
> > > > Git._lsremote(ud=<bb.fetch2.FetchData object at
> > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > 0x7f8e26195890>, search=''):
> > > > bb.fetch2.check_network_access(d, cmd,
> > > > repourl)
> > > > > output = runfetchcmd(cmd, d, True)
> > > > if not output:
> > > > File "bitbake/lib/bb/fetch2/__init__.py", line 957, in
> > > > runfetchcmd(cmd='export PSEUDO_DISABLED=1; export
> > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export
> > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-
> > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-
> > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-
> > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > yocto/6.6.20+git/recipe-sysroot-
> > > > native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > yocto/6.6.20+git/recipe-sysroot-
> > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > yocto/6.6.20+git/recipe-sysroot-
> > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
> > > > -c safe.bareRepository=all ls-remote
> > > > https://git.yoctoproject.org/linux-yocto.git ',
> > > > d=<bb.data_smart.DataSmart object at 0x7f8e26195890>,
> > > > quiet=True, cleanup=[], log=None, workdir=None):
> > > >
> > > > > raise FetchError(error_message)
> > > >
> > > > bb.data_smart.ExpansionError: Failure expanding variable
> > > > fetcher_hashes_dummyfunc[vardepvalue], expression was
> > > > ${@bb.fetch.get_hashvalue(d)} which triggered exception
> > > > FetchError: Fetcher failure: Fetch command export
> > > > PSEUDO_DISABLED=1; export
> > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export
> > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/q
> > > > emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-
> > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-
> > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > yocto/6.6.20+git/recipe-sysroot-
> > > > native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > yocto/6.6.20+git/recipe-sysroot-
> > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > yocto/6.6.20+git/recipe-sysroot-
> > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
> > > > -c safe.bareRepository=all ls-remote
> > > > https://git.yoctoproject.org/linux-yocto.git failed with exit
> > > > code 128, output:
> > > > fatal: unable to access
> > > > 'https://git.yoctoproject.org/linux-yocto.git/': error setting
> > > > certificate file: build/tmp/work/x86_64-linux/curl-
> > > > native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-
> > > > certificates.crt
> > > >
> > > > The variable dependency chain for the failure is:
> > > > fetcher_hashes_dummyfunc[vardepvalue]
> > >
> > > Note:
> > > One gets out of that parser error by deleting the git binary in
> > > the
> > > kernel's work recipe-sysroot-native
> > > `rm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-
> > > sysroot-native/usr/bin/git`
> > >
> > > Bisecting poky leads to commit
> > > f7fa98cca8 ("kern-tools: depend on git-replacement-native")
> > > Reverting it on top of b5624ee564 makes the parsing pass.
> > >
> > > I assume that `git-replacement-native` does not work with https,
> > > the
> > > fetch error also goes away if changing in SRC_URI from https to
> > > git.
> > >
> > > Any comments?
> >
> > I didn't even know that curl was coming into play :)
> >
> > Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
> > should resolve the issue.
>
> Looks like curl-native resp. libcurl hardcodes the lookup to its own
> work directory, i.e.:
> x86_64-linux/curl-native/8.6.0/recipe-sysroot-
> native/etc/ssl/certs/ca-certificates.crt
>
> So even if DEPENDS/RDEPENDS will install ca-certificates in the
> kernel's
> recipe-sysroot-native the parsing will fail if the curl-native
> directory
> is not/no longer populated, e.g. because curl-native came from sstate
> or
> rm_work is in INHERIT.
This all gets a bit messy.
We've relied upon scripts that use openssl to set variables like:
export SSL_CERT_DIR="XXXX/etc/ssl/certs/
so in theory we might be able to set an environment variable in a
wrapper around the git commands.
It may be better if we teach curl a relative path to the certs...
I suspect this isn't going to be an easy/neat fix unfortunately.
Cheers,
Richard
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OE-core] [REGRESSION] linux (git/curl-native) and autorev
[not found] ` <17BB69D476D2F4EF.5850@lists.openembedded.org>
@ 2024-03-10 15:52 ` Richard Purdie
2024-03-10 16:04 ` Bruce Ashfield
0 siblings, 1 reply; 10+ messages in thread
From: Richard Purdie @ 2024-03-10 15:52 UTC (permalink / raw)
To: Max, Bruce Ashfield; +Cc: openembedded-core, Max Krummenacher
On Sun, 2024-03-10 at 06:20 -0700, Richard Purdie via
lists.openembedded.org wrote:
> On Sun, 2024-03-10 at 13:31 +0100, Max wrote:
> > Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
> > > On Sat, Mar 9, 2024 at 12:58 PM <max.oss.09@gmail.com> wrote:
> > > >
> > > > From: Max Krummenacher <max.krummenacher@toradex.com>
> > > >
> > > > Hello
> > > >
> > > > If one builds a kernel using AUTOREV invoking bitbake only
> > > > works
> > > > once.
> > > > Any subsequent bitbake invocation fails parsing the meta data.
> > > >
> > > > Reproducable with:
> > > > - latest poky, b5624ee564
> > > > - Kernel with SRCREV = "AUTOREV", e.g. in local.conf
> > > > `SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"`
> > > > - bitbake virtual/kernel; bitbake virtual/kernel
> > > >
> > > > On the second invocation parsing fails when the fetcher tries
> > > > to
> > > > evaluate the latest SRCREV:
> > > >
> > > > > ERROR: ExpansionError during parsing meta/recipes-
> > > > > kernel/linux/linux-yocto_6.6.bb
> > > > > Traceback (most recent call last):
> > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in
> > > > > srcrev_internal_helper(ud=<bb.fetch2.FetchData object at
> > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > 0x7f8e26195890>, name='machine'):
> > > > > d.setVar("__BBAUTOREV_ACTED_UPON", True)
> > > > > > srcrev = ud.method.latest_revision(ud, d, name)
> > > > >
> > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in
> > > > > Git.latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > 0x7f8e26195890>, name='machine'):
> > > > > except KeyError:
> > > > > > revs[key] = rev = self._latest_revision(ud,
> > > > > d,
> > > > > name)
> > > > > return rev
> > > > > File "bitbake/lib/bb/fetch2/git.py", line 850, in
> > > > > Git._latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > 0x7f8e26195890>, name='machine'):
> > > > >
> > > > > > output = self._lsremote(ud, d, "")
> > > > > # Tags of the form ^{} may not work, need to
> > > > > fallback to other form
> > > > > File "bitbake/lib/bb/fetch2/git.py", line 833, in
> > > > > Git._lsremote(ud=<bb.fetch2.FetchData object at
> > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > 0x7f8e26195890>, search=''):
> > > > > bb.fetch2.check_network_access(d, cmd,
> > > > > repourl)
> > > > > > output = runfetchcmd(cmd, d, True)
> > > > > if not output:
> > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 957, in
> > > > > runfetchcmd(cmd='export PSEUDO_DISABLED=1; export
> > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > export
> > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-
> > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-
> > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-
> > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > core.pager=cat
> > > > > -c safe.bareRepository=all ls-remote
> > > > > https://git.yoctoproject.org/linux-yocto.git ',
> > > > > d=<bb.data_smart.DataSmart object at 0x7f8e26195890>,
> > > > > quiet=True, cleanup=[], log=None, workdir=None):
> > > > >
> > > > > > raise FetchError(error_message)
> > > > >
> > > > > bb.data_smart.ExpansionError: Failure expanding variable
> > > > > fetcher_hashes_dummyfunc[vardepvalue], expression was
> > > > > ${@bb.fetch.get_hashvalue(d)} which triggered exception
> > > > > FetchError: Fetcher failure: Fetch command export
> > > > > PSEUDO_DISABLED=1; export
> > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > export
> > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work
> > > > > /q
> > > > > emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-
> > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-
> > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > core.pager=cat
> > > > > -c safe.bareRepository=all ls-remote
> > > > > https://git.yoctoproject.org/linux-yocto.git failed with
> > > > > exit
> > > > > code 128, output:
> > > > > fatal: unable to access
> > > > > 'https://git.yoctoproject.org/linux-yocto.git/': error
> > > > > setting
> > > > > certificate file: build/tmp/work/x86_64-linux/curl-
> > > > > native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-
> > > > > certificates.crt
> > > > >
> > > > > The variable dependency chain for the failure is:
> > > > > fetcher_hashes_dummyfunc[vardepvalue]
> > > >
> > > > Note:
> > > > One gets out of that parser error by deleting the git binary in
> > > > the
> > > > kernel's work recipe-sysroot-native
> > > > `rm tmp/work/qemux86_64-poky-linux/linux-
> > > > yocto/6.6.20+git/recipe-
> > > > sysroot-native/usr/bin/git`
> > > >
> > > > Bisecting poky leads to commit
> > > > f7fa98cca8 ("kern-tools: depend on git-replacement-native")
> > > > Reverting it on top of b5624ee564 makes the parsing pass.
> > > >
> > > > I assume that `git-replacement-native` does not work with
> > > > https,
> > > > the
> > > > fetch error also goes away if changing in SRC_URI from https to
> > > > git.
> > > >
> > > > Any comments?
> > >
> > > I didn't even know that curl was coming into play :)
> > >
> > > Adding DEPENDS:class-native += "ca-certificates" to the curl
> > > recipe
> > > should resolve the issue.
> >
> > Looks like curl-native resp. libcurl hardcodes the lookup to its
> > own
> > work directory, i.e.:
> > x86_64-linux/curl-native/8.6.0/recipe-sysroot-
> > native/etc/ssl/certs/ca-certificates.crt
> >
> > So even if DEPENDS/RDEPENDS will install ca-certificates in the
> > kernel's
> > recipe-sysroot-native the parsing will fail if the curl-native
> > directory
> > is not/no longer populated, e.g. because curl-native came from
> > sstate
> > or
> > rm_work is in INHERIT.
>
> This all gets a bit messy.
>
> We've relied upon scripts that use openssl to set variables like:
>
> export SSL_CERT_DIR="XXXX/etc/ssl/certs/
>
> so in theory we might be able to set an environment variable in a
> wrapper around the git commands.
>
> It may be better if we teach curl a relative path to the certs...
>
> I suspect this isn't going to be an easy/neat fix unfortunately.
I'm trying not to get sucked into "work" today however I realised that
relative paths won't work without some implementation of "$ORIGIN"
support into these paths. Given it ultimately ends up in openssl, it
would probably be best there.
I'd not be against writing a $ORIGIN support patch and seeing what
upstream think about it. It would still mean finding a way to find the
path to the library file somehow.
For purposes of the release, setting the right envvars in the git
wrapper is probably the way forward for now, much as I dislike the
requirement to do that.
This does feel like a problem it would be good to solve for wider Linux
in general for relocatable (and reproducible) binaries.
Cheers,
Richard
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OE-core] [REGRESSION] linux (git/curl-native) and autorev
2024-03-10 15:52 ` Richard Purdie
@ 2024-03-10 16:04 ` Bruce Ashfield
2024-03-10 23:39 ` Richard Purdie
0 siblings, 1 reply; 10+ messages in thread
From: Bruce Ashfield @ 2024-03-10 16:04 UTC (permalink / raw)
To: Richard Purdie; +Cc: Max, openembedded-core, Max Krummenacher
On Sun, Mar 10, 2024 at 11:52 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Sun, 2024-03-10 at 06:20 -0700, Richard Purdie via
> lists.openembedded.org wrote:
> > On Sun, 2024-03-10 at 13:31 +0100, Max wrote:
> > > Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
> > > > On Sat, Mar 9, 2024 at 12:58 PM <max.oss.09@gmail.com> wrote:
> > > > >
> > > > > From: Max Krummenacher <max.krummenacher@toradex.com>
> > > > >
> > > > > Hello
> > > > >
> > > > > If one builds a kernel using AUTOREV invoking bitbake only
> > > > > works
> > > > > once.
> > > > > Any subsequent bitbake invocation fails parsing the meta data.
> > > > >
> > > > > Reproducable with:
> > > > > - latest poky, b5624ee564
> > > > > - Kernel with SRCREV = "AUTOREV", e.g. in local.conf
> > > > > `SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"`
> > > > > - bitbake virtual/kernel; bitbake virtual/kernel
> > > > >
> > > > > On the second invocation parsing fails when the fetcher tries
> > > > > to
> > > > > evaluate the latest SRCREV:
> > > > >
> > > > > > ERROR: ExpansionError during parsing meta/recipes-
> > > > > > kernel/linux/linux-yocto_6.6.bb
> > > > > > Traceback (most recent call last):
> > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in
> > > > > > srcrev_internal_helper(ud=<bb.fetch2.FetchData object at
> > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > d.setVar("__BBAUTOREV_ACTED_UPON", True)
> > > > > > > srcrev = ud.method.latest_revision(ud, d, name)
> > > > > >
> > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in
> > > > > > Git.latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > except KeyError:
> > > > > > > revs[key] = rev = self._latest_revision(ud,
> > > > > > d,
> > > > > > name)
> > > > > > return rev
> > > > > > File "bitbake/lib/bb/fetch2/git.py", line 850, in
> > > > > > Git._latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > 0x7f8e26195890>, name='machine'):
> > > > > >
> > > > > > > output = self._lsremote(ud, d, "")
> > > > > > # Tags of the form ^{} may not work, need to
> > > > > > fallback to other form
> > > > > > File "bitbake/lib/bb/fetch2/git.py", line 833, in
> > > > > > Git._lsremote(ud=<bb.fetch2.FetchData object at
> > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > 0x7f8e26195890>, search=''):
> > > > > > bb.fetch2.check_network_access(d, cmd,
> > > > > > repourl)
> > > > > > > output = runfetchcmd(cmd, d, True)
> > > > > > if not output:
> > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 957, in
> > > > > > runfetchcmd(cmd='export PSEUDO_DISABLED=1; export
> > > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > > export
> > > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > > linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-
> > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-
> > > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-
> > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > > core.pager=cat
> > > > > > -c safe.bareRepository=all ls-remote
> > > > > > https://git.yoctoproject.org/linux-yocto.git ',
> > > > > > d=<bb.data_smart.DataSmart object at 0x7f8e26195890>,
> > > > > > quiet=True, cleanup=[], log=None, workdir=None):
> > > > > >
> > > > > > > raise FetchError(error_message)
> > > > > >
> > > > > > bb.data_smart.ExpansionError: Failure expanding variable
> > > > > > fetcher_hashes_dummyfunc[vardepvalue], expression was
> > > > > > ${@bb.fetch.get_hashvalue(d)} which triggered exception
> > > > > > FetchError: Fetcher failure: Fetch command export
> > > > > > PSEUDO_DISABLED=1; export
> > > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > > export
> > > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > > linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work
> > > > > > /q
> > > > > > emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-
> > > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-
> > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > > core.pager=cat
> > > > > > -c safe.bareRepository=all ls-remote
> > > > > > https://git.yoctoproject.org/linux-yocto.git failed with
> > > > > > exit
> > > > > > code 128, output:
> > > > > > fatal: unable to access
> > > > > > 'https://git.yoctoproject.org/linux-yocto.git/': error
> > > > > > setting
> > > > > > certificate file: build/tmp/work/x86_64-linux/curl-
> > > > > > native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-
> > > > > > certificates.crt
> > > > > >
> > > > > > The variable dependency chain for the failure is:
> > > > > > fetcher_hashes_dummyfunc[vardepvalue]
> > > > >
> > > > > Note:
> > > > > One gets out of that parser error by deleting the git binary in
> > > > > the
> > > > > kernel's work recipe-sysroot-native
> > > > > `rm tmp/work/qemux86_64-poky-linux/linux-
> > > > > yocto/6.6.20+git/recipe-
> > > > > sysroot-native/usr/bin/git`
> > > > >
> > > > > Bisecting poky leads to commit
> > > > > f7fa98cca8 ("kern-tools: depend on git-replacement-native")
> > > > > Reverting it on top of b5624ee564 makes the parsing pass.
> > > > >
> > > > > I assume that `git-replacement-native` does not work with
> > > > > https,
> > > > > the
> > > > > fetch error also goes away if changing in SRC_URI from https to
> > > > > git.
> > > > >
> > > > > Any comments?
> > > >
> > > > I didn't even know that curl was coming into play :)
> > > >
> > > > Adding DEPENDS:class-native += "ca-certificates" to the curl
> > > > recipe
> > > > should resolve the issue.
> > >
> > > Looks like curl-native resp. libcurl hardcodes the lookup to its
> > > own
> > > work directory, i.e.:
> > > x86_64-linux/curl-native/8.6.0/recipe-sysroot-
> > > native/etc/ssl/certs/ca-certificates.crt
> > >
> > > So even if DEPENDS/RDEPENDS will install ca-certificates in the
> > > kernel's
> > > recipe-sysroot-native the parsing will fail if the curl-native
> > > directory
> > > is not/no longer populated, e.g. because curl-native came from
> > > sstate
> > > or
> > > rm_work is in INHERIT.
> >
> > This all gets a bit messy.
> >
> > We've relied upon scripts that use openssl to set variables like:
> >
> > export SSL_CERT_DIR="XXXX/etc/ssl/certs/
> >
> > so in theory we might be able to set an environment variable in a
> > wrapper around the git commands.
> >
> > It may be better if we teach curl a relative path to the certs...
> >
> > I suspect this isn't going to be an easy/neat fix unfortunately.
>
> I'm trying not to get sucked into "work" today however I realised that
> relative paths won't work without some implementation of "$ORIGIN"
> support into these paths. Given it ultimately ends up in openssl, it
> would probably be best there.
>
> I'd not be against writing a $ORIGIN support patch and seeing what
> upstream think about it. It would still mean finding a way to find the
> path to the library file somehow.
>
> For purposes of the release, setting the right envvars in the git
> wrapper is probably the way forward for now, much as I dislike the
> requirement to do that.
I tried this, but curl didn't seem to use it to locate the cert file.
i.e., I tried this:
---------------
diff --git a/meta/recipes-devtools/git/git_2.44.0.bb
b/meta/recipes-devtools/git/git_2.44.0.bb
index e6d1470873..f6b06ec601 100644
--- a/meta/recipes-devtools/git/git_2.44.0.bb
+++ b/meta/recipes-devtools/git/git_2.44.0.bb
@@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl"
PACKAGECONFIG[cvsserver] = ""
PACKAGECONFIG[svn] = ""
PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native"
-PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl"
+PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-certificates"
PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"
EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl-native/perl \
@@ -103,13 +103,15 @@ do_install:append:class-target () {
do_install:append:class-native() {
create_wrapper ${D}${bindir}/git \
GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
- GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}
+ GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR} \
+ SSL_CERT_DIR='`dirname $''realpath`'/../../etc/ssl/certs/
}
do_install:append:class-nativesdk() {
create_wrapper ${D}${bindir}/git \
GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}
+ SSL_CERT_DIR='`dirname $''realpath`'/../../etc/ssl/certs/
perl_native_fixup
}
---------
Probably because the exec of curl from git isn't getting the
environment ? Either
that, or I did it wrong. I didn't try it as an explicit export, and
that is probably it,
I can try that later tonight.
Only when I added the depends on ca-certifications to curl itself was it able to
fetch the autorevs.
Bruce
>
> This does feel like a problem it would be good to solve for wider Linux
> in general for relocatable (and reproducible) binaries.
>
> Cheers,
>
> Richard
>
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [OE-core] [REGRESSION] linux (git/curl-native) and autorev
2024-03-10 16:04 ` Bruce Ashfield
@ 2024-03-10 23:39 ` Richard Purdie
2024-03-11 12:54 ` Max Krummenacher
0 siblings, 1 reply; 10+ messages in thread
From: Richard Purdie @ 2024-03-10 23:39 UTC (permalink / raw)
To: Bruce Ashfield; +Cc: Max, openembedded-core, Max Krummenacher
On Sun, 2024-03-10 at 09:05 -0700, Bruce Ashfield wrote:
> On Sun, Mar 10, 2024 at 11:52 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Sun, 2024-03-10 at 06:20 -0700, Richard Purdie via
> > lists.openembedded.org wrote:
> > > On Sun, 2024-03-10 at 13:31 +0100, Max wrote:
> > > > Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce
> > > > Ashfield:
> > > > > On Sat, Mar 9, 2024 at 12:58 PM <max.oss.09@gmail.com> wrote:
> > > > > >
> > > > > > From: Max Krummenacher <max.krummenacher@toradex.com>
> > > > > >
> > > > > > Hello
> > > > > >
> > > > > > If one builds a kernel using AUTOREV invoking bitbake only
> > > > > > works
> > > > > > once.
> > > > > > Any subsequent bitbake invocation fails parsing the meta
> > > > > > data.
> > > > > >
> > > > > > Reproducable with:
> > > > > > - latest poky, b5624ee564
> > > > > > - Kernel with SRCREV = "AUTOREV", e.g. in local.conf
> > > > > > `SRCREV_machine:pn-linux-yocto:forcevariable =
> > > > > > "${AUTOREV}"`
> > > > > > - bitbake virtual/kernel; bitbake virtual/kernel
> > > > > >
> > > > > > On the second invocation parsing fails when the fetcher
> > > > > > tries
> > > > > > to
> > > > > > evaluate the latest SRCREV:
> > > > > >
> > > > > > > ERROR: ExpansionError during parsing meta/recipes-
> > > > > > > kernel/linux/linux-yocto_6.6.bb
> > > > > > > Traceback (most recent call last):
> > > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in
> > > > > > > srcrev_internal_helper(ud=<bb.fetch2.FetchData object at
> > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > > d.setVar("__BBAUTOREV_ACTED_UPON", True)
> > > > > > > > srcrev = ud.method.latest_revision(ud, d,
> > > > > > > name)
> > > > > > >
> > > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in
> > > > > > > Git.latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > > except KeyError:
> > > > > > > > revs[key] = rev =
> > > > > > > self._latest_revision(ud,
> > > > > > > d,
> > > > > > > name)
> > > > > > > return rev
> > > > > > > File "bitbake/lib/bb/fetch2/git.py", line 850, in
> > > > > > > Git._latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > >
> > > > > > > > output = self._lsremote(ud, d, "")
> > > > > > > # Tags of the form ^{} may not work, need to
> > > > > > > fallback to other form
> > > > > > > File "bitbake/lib/bb/fetch2/git.py", line 833, in
> > > > > > > Git._lsremote(ud=<bb.fetch2.FetchData object at
> > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > 0x7f8e26195890>, search=''):
> > > > > > > bb.fetch2.check_network_access(d,
> > > > > > > cmd,
> > > > > > > repourl)
> > > > > > > > output = runfetchcmd(cmd, d, True)
> > > > > > > if not output:
> > > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 957, in
> > > > > > > runfetchcmd(cmd='export PSEUDO_DISABLED=1; export
> > > > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > > > export
> > > > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > > > linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-
> > > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > > native/usr/bin/x86_64-poky-
> > > > > > > linux:build/tmp/work/qemux86_64-
> > > > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-
> > > > > > > poky-
> > > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-
> > > > > > > linux/linux-
> > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-
> > > > > > > linux/linux-
> > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > > > core.pager=cat
> > > > > > > -c safe.bareRepository=all ls-remote
> > > > > > > https://git.yoctoproject.org/linux-yocto.git ',
> > > > > > > d=<bb.data_smart.DataSmart object at 0x7f8e26195890>,
> > > > > > > quiet=True, cleanup=[], log=None, workdir=None):
> > > > > > >
> > > > > > > > raise FetchError(error_message)
> > > > > > >
> > > > > > > bb.data_smart.ExpansionError: Failure expanding variable
> > > > > > > fetcher_hashes_dummyfunc[vardepvalue], expression was
> > > > > > > ${@bb.fetch.get_hashvalue(d)} which triggered exception
> > > > > > > FetchError: Fetcher failure: Fetch command export
> > > > > > > PSEUDO_DISABLED=1; export
> > > > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > > > export
> > > > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > > > linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/
> > > > > > > work
> > > > > > > /q
> > > > > > > emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > > sysroot-
> > > > > > > native/usr/bin/x86_64-poky-
> > > > > > > linux:build/tmp/work/qemux86_64-
> > > > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-
> > > > > > > poky-
> > > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-
> > > > > > > linux/linux-
> > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-
> > > > > > > linux/linux-
> > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > > > core.pager=cat
> > > > > > > -c safe.bareRepository=all ls-remote
> > > > > > > https://git.yoctoproject.org/linux-yocto.git failed with
> > > > > > > exit
> > > > > > > code 128, output:
> > > > > > > fatal: unable to access
> > > > > > > 'https://git.yoctoproject.org/linux-yocto.git/': error
> > > > > > > setting
> > > > > > > certificate file: build/tmp/work/x86_64-linux/curl-
> > > > > > > native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-
> > > > > > > certificates.crt
> > > > > > >
> > > > > > > The variable dependency chain for the failure is:
> > > > > > > fetcher_hashes_dummyfunc[vardepvalue]
> > > > > >
> > > > > > Note:
> > > > > > One gets out of that parser error by deleting the git
> > > > > > binary in
> > > > > > the
> > > > > > kernel's work recipe-sysroot-native
> > > > > > `rm tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-
> > > > > > sysroot-native/usr/bin/git`
> > > > > >
> > > > > > Bisecting poky leads to commit
> > > > > > f7fa98cca8 ("kern-tools: depend on git-replacement-native")
> > > > > > Reverting it on top of b5624ee564 makes the parsing pass.
> > > > > >
> > > > > > I assume that `git-replacement-native` does not work with
> > > > > > https,
> > > > > > the
> > > > > > fetch error also goes away if changing in SRC_URI from
> > > > > > https to
> > > > > > git.
> > > > > >
> > > > > > Any comments?
> > > > >
> > > > > I didn't even know that curl was coming into play :)
> > > > >
> > > > > Adding DEPENDS:class-native += "ca-certificates" to the curl
> > > > > recipe
> > > > > should resolve the issue.
> > > >
> > > > Looks like curl-native resp. libcurl hardcodes the lookup to
> > > > its
> > > > own
> > > > work directory, i.e.:
> > > > x86_64-linux/curl-native/8.6.0/recipe-sysroot-
> > > > native/etc/ssl/certs/ca-certificates.crt
> > > >
> > > > So even if DEPENDS/RDEPENDS will install ca-certificates in the
> > > > kernel's
> > > > recipe-sysroot-native the parsing will fail if the curl-native
> > > > directory
> > > > is not/no longer populated, e.g. because curl-native came from
> > > > sstate
> > > > or
> > > > rm_work is in INHERIT.
> > >
> > > This all gets a bit messy.
> > >
> > > We've relied upon scripts that use openssl to set variables like:
> > >
> > > export SSL_CERT_DIR="XXXX/etc/ssl/certs/
> > >
> > > so in theory we might be able to set an environment variable in a
> > > wrapper around the git commands.
> > >
> > > It may be better if we teach curl a relative path to the certs...
> > >
> > > I suspect this isn't going to be an easy/neat fix unfortunately.
> >
> > I'm trying not to get sucked into "work" today however I realised
> > that
> > relative paths won't work without some implementation of "$ORIGIN"
> > support into these paths. Given it ultimately ends up in openssl,
> > it
> > would probably be best there.
> >
> > I'd not be against writing a $ORIGIN support patch and seeing what
> > upstream think about it. It would still mean finding a way to find
> > the
> > path to the library file somehow.
> >
> > For purposes of the release, setting the right envvars in the git
> > wrapper is probably the way forward for now, much as I dislike the
> > requirement to do that.
>
> I tried this, but curl didn't seem to use it to locate the cert file.
>
> i.e., I tried this:
>
> ---------------
>
> diff --git a/meta/recipes-devtools/git/git_2.44.0.bb
> b/meta/recipes-devtools/git/git_2.44.0.bb
> index e6d1470873..f6b06ec601 100644
> --- a/meta/recipes-devtools/git/git_2.44.0.bb
> +++ b/meta/recipes-devtools/git/git_2.44.0.bb
> @@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl"
> PACKAGECONFIG[cvsserver] = ""
> PACKAGECONFIG[svn] = ""
> PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native"
> -PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl"
> +PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-
> certificates"
> PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"
>
> EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl-
> native/perl \
> @@ -103,13 +103,15 @@ do_install:append:class-target () {
> do_install:append:class-native() {
> create_wrapper ${D}${bindir}/git \
> GIT_EXEC_PATH='`dirname
> $''realpath`'/${REL_GIT_EXEC_PATH} \
> - GIT_TEMPLATE_DIR='`dirname
> $''realpath`'/${REL_GIT_TEMPLATE_DIR}
> + GIT_TEMPLATE_DIR='`dirname
> $''realpath`'/${REL_GIT_TEMPLATE_DIR} \
> + SSL_CERT_DIR='`dirname
> $''realpath`'/../../etc/ssl/certs/
> }
>
> do_install:append:class-nativesdk() {
> create_wrapper ${D}${bindir}/git \
> GIT_EXEC_PATH='`dirname
> $''realpath`'/${REL_GIT_EXEC_PATH} \
> GIT_TEMPLATE_DIR='`dirname
> $''realpath`'/${REL_GIT_TEMPLATE_DIR}
> + SSL_CERT_DIR='`dirname
> $''realpath`'/../../etc/ssl/certs/
> perl_native_fixup
> }
>
> ---------
>
> Probably because the exec of curl from git isn't getting the
> environment ? Either
> that, or I did it wrong. I didn't try it as an explicit export, and
> that is probably it,
> I can try that later tonight.
>
> Only when I added the depends on ca-certifications to curl itself was
> it able to
> fetch the autorevs.
We probably need both the variable (to avoid the workdir removal issue)
and the dependency.
Cheers,
Richard
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OE-core] [REGRESSION] linux (git/curl-native) and autorev
2024-03-10 23:39 ` Richard Purdie
@ 2024-03-11 12:54 ` Max Krummenacher
2024-03-11 13:57 ` Bruce Ashfield
0 siblings, 1 reply; 10+ messages in thread
From: Max Krummenacher @ 2024-03-11 12:54 UTC (permalink / raw)
To: Richard Purdie; +Cc: Bruce Ashfield, openembedded-core, Max Krummenacher
On Sun, Mar 10, 2024 at 11:39:00PM +0000, Richard Purdie wrote:
> On Sun, 2024-03-10 at 09:05 -0700, Bruce Ashfield wrote:
> > On Sun, Mar 10, 2024 at 11:52 AM Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > >
> > > On Sun, 2024-03-10 at 06:20 -0700, Richard Purdie via
> > > lists.openembedded.org wrote:
> > > > On Sun, 2024-03-10 at 13:31 +0100, Max wrote:
> > > > > Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce
> > > > > Ashfield:
> > > > > > On Sat, Mar 9, 2024 at 12:58 PM <max.oss.09@gmail.com> wrote:
> > > > > > >
> > > > > > > From: Max Krummenacher <max.krummenacher@toradex.com>
> > > > > > >
> > > > > > > Hello
> > > > > > >
> > > > > > > If one builds a kernel using AUTOREV invoking bitbake only
> > > > > > > works
> > > > > > > once.
> > > > > > > Any subsequent bitbake invocation fails parsing the meta
> > > > > > > data.
> > > > > > >
> > > > > > > Reproducable with:
> > > > > > > - latest poky, b5624ee564
> > > > > > > - Kernel with SRCREV = "AUTOREV", e.g. in local.conf
> > > > > > > `SRCREV_machine:pn-linux-yocto:forcevariable =
> > > > > > > "${AUTOREV}"`
> > > > > > > - bitbake virtual/kernel; bitbake virtual/kernel
> > > > > > >
> > > > > > > On the second invocation parsing fails when the fetcher
> > > > > > > tries
> > > > > > > to
> > > > > > > evaluate the latest SRCREV:
> > > > > > >
> > > > > > > > ERROR: ExpansionError during parsing meta/recipes-
> > > > > > > > kernel/linux/linux-yocto_6.6.bb
> > > > > > > > Traceback (most recent call last):
> > > > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in
> > > > > > > > srcrev_internal_helper(ud=<bb.fetch2.FetchData object at
> > > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > > > d.setVar("__BBAUTOREV_ACTED_UPON", True)
> > > > > > > > > srcrev = ud.method.latest_revision(ud, d,
> > > > > > > > name)
> > > > > > > >
> > > > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in
> > > > > > > > Git.latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > > > except KeyError:
> > > > > > > > > revs[key] = rev =
> > > > > > > > self._latest_revision(ud,
> > > > > > > > d,
> > > > > > > > name)
> > > > > > > > return rev
> > > > > > > > File "bitbake/lib/bb/fetch2/git.py", line 850, in
> > > > > > > > Git._latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > > >
> > > > > > > > > output = self._lsremote(ud, d, "")
> > > > > > > > # Tags of the form ^{} may not work, need to
> > > > > > > > fallback to other form
> > > > > > > > File "bitbake/lib/bb/fetch2/git.py", line 833, in
> > > > > > > > Git._lsremote(ud=<bb.fetch2.FetchData object at
> > > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > > 0x7f8e26195890>, search=''):
> > > > > > > > bb.fetch2.check_network_access(d,
> > > > > > > > cmd,
> > > > > > > > repourl)
> > > > > > > > > output = runfetchcmd(cmd, d, True)
> > > > > > > > if not output:
> > > > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 957, in
> > > > > > > > runfetchcmd(cmd='export PSEUDO_DISABLED=1; export
> > > > > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > > > > export
> > > > > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > > > > linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-
> > > > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > native/usr/bin/x86_64-poky-
> > > > > > > > linux:build/tmp/work/qemux86_64-
> > > > > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-
> > > > > > > > poky-
> > > > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-
> > > > > > > > linux/linux-
> > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-
> > > > > > > > linux/linux-
> > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > > > > core.pager=cat
> > > > > > > > -c safe.bareRepository=all ls-remote
> > > > > > > > https://git.yoctoproject.org/linux-yocto.git ',
> > > > > > > > d=<bb.data_smart.DataSmart object at 0x7f8e26195890>,
> > > > > > > > quiet=True, cleanup=[], log=None, workdir=None):
> > > > > > > >
> > > > > > > > > raise FetchError(error_message)
> > > > > > > >
> > > > > > > > bb.data_smart.ExpansionError: Failure expanding variable
> > > > > > > > fetcher_hashes_dummyfunc[vardepvalue], expression was
> > > > > > > > ${@bb.fetch.get_hashvalue(d)} which triggered exception
> > > > > > > > FetchError: Fetcher failure: Fetch command export
> > > > > > > > PSEUDO_DISABLED=1; export
> > > > > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > > > > export
> > > > > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > > > > linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/
> > > > > > > > work
> > > > > > > > /q
> > > > > > > > emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > > > sysroot-
> > > > > > > > native/usr/bin/x86_64-poky-
> > > > > > > > linux:build/tmp/work/qemux86_64-
> > > > > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-
> > > > > > > > poky-
> > > > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-
> > > > > > > > linux/linux-
> > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-
> > > > > > > > linux/linux-
> > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > > > > core.pager=cat
> > > > > > > > -c safe.bareRepository=all ls-remote
> > > > > > > > https://git.yoctoproject.org/linux-yocto.git failed with
> > > > > > > > exit
> > > > > > > > code 128, output:
> > > > > > > > fatal: unable to access
> > > > > > > > 'https://git.yoctoproject.org/linux-yocto.git/': error
> > > > > > > > setting
> > > > > > > > certificate file: build/tmp/work/x86_64-linux/curl-
> > > > > > > > native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-
> > > > > > > > certificates.crt
> > > > > > > >
> > > > > > > > The variable dependency chain for the failure is:
> > > > > > > > fetcher_hashes_dummyfunc[vardepvalue]
> > > > > > >
> > > > > > > Note:
> > > > > > > One gets out of that parser error by deleting the git
> > > > > > > binary in
> > > > > > > the
> > > > > > > kernel's work recipe-sysroot-native
> > > > > > > `rm tmp/work/qemux86_64-poky-linux/linux-
> > > > > > > yocto/6.6.20+git/recipe-
> > > > > > > sysroot-native/usr/bin/git`
> > > > > > >
> > > > > > > Bisecting poky leads to commit
> > > > > > > f7fa98cca8 ("kern-tools: depend on git-replacement-native")
> > > > > > > Reverting it on top of b5624ee564 makes the parsing pass.
> > > > > > >
> > > > > > > I assume that `git-replacement-native` does not work with
> > > > > > > https,
> > > > > > > the
> > > > > > > fetch error also goes away if changing in SRC_URI from
> > > > > > > https to
> > > > > > > git.
> > > > > > >
> > > > > > > Any comments?
> > > > > >
> > > > > > I didn't even know that curl was coming into play :)
> > > > > >
> > > > > > Adding DEPENDS:class-native += "ca-certificates" to the curl
> > > > > > recipe
> > > > > > should resolve the issue.
> > > > >
> > > > > Looks like curl-native resp. libcurl hardcodes the lookup to
> > > > > its
> > > > > own
> > > > > work directory, i.e.:
> > > > > x86_64-linux/curl-native/8.6.0/recipe-sysroot-
> > > > > native/etc/ssl/certs/ca-certificates.crt
> > > > >
> > > > > So even if DEPENDS/RDEPENDS will install ca-certificates in the
> > > > > kernel's
> > > > > recipe-sysroot-native the parsing will fail if the curl-native
> > > > > directory
> > > > > is not/no longer populated, e.g. because curl-native came from
> > > > > sstate
> > > > > or
> > > > > rm_work is in INHERIT.
> > > >
> > > > This all gets a bit messy.
> > > >
> > > > We've relied upon scripts that use openssl to set variables like:
> > > >
> > > > export SSL_CERT_DIR="XXXX/etc/ssl/certs/
> > > >
> > > > so in theory we might be able to set an environment variable in a
> > > > wrapper around the git commands.
> > > >
> > > > It may be better if we teach curl a relative path to the certs...
> > > >
> > > > I suspect this isn't going to be an easy/neat fix unfortunately.
> > >
> > > I'm trying not to get sucked into "work" today however I realised
> > > that
> > > relative paths won't work without some implementation of "$ORIGIN"
> > > support into these paths. Given it ultimately ends up in openssl,
> > > it
> > > would probably be best there.
> > >
> > > I'd not be against writing a $ORIGIN support patch and seeing what
> > > upstream think about it. It would still mean finding a way to find
> > > the
> > > path to the library file somehow.
> > >
> > > For purposes of the release, setting the right envvars in the git
> > > wrapper is probably the way forward for now, much as I dislike the
> > > requirement to do that.
> >
> > I tried this, but curl didn't seem to use it to locate the cert file.
> >
> > i.e., I tried this:
> >
> > ---------------
> >
> > diff --git a/meta/recipes-devtools/git/git_2.44.0.bb
> > b/meta/recipes-devtools/git/git_2.44.0.bb
> > index e6d1470873..f6b06ec601 100644
> > --- a/meta/recipes-devtools/git/git_2.44.0.bb
> > +++ b/meta/recipes-devtools/git/git_2.44.0.bb
> > @@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl"
> > PACKAGECONFIG[cvsserver] = ""
> > PACKAGECONFIG[svn] = ""
> > PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native"
> > -PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl"
> > +PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-
> > certificates"
> > PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"
> >
> > EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl-
> > native/perl \
> > @@ -103,13 +103,15 @@ do_install:append:class-target () {
> > do_install:append:class-native() {
> > create_wrapper ${D}${bindir}/git \
> > GIT_EXEC_PATH='`dirname
> > $''realpath`'/${REL_GIT_EXEC_PATH} \
> > - GIT_TEMPLATE_DIR='`dirname
> > $''realpath`'/${REL_GIT_TEMPLATE_DIR}
> > + GIT_TEMPLATE_DIR='`dirname
> > $''realpath`'/${REL_GIT_TEMPLATE_DIR} \
> > + SSL_CERT_DIR='`dirname
> > $''realpath`'/../../etc/ssl/certs/
> > }
> >
> > do_install:append:class-nativesdk() {
> > create_wrapper ${D}${bindir}/git \
> > GIT_EXEC_PATH='`dirname
> > $''realpath`'/${REL_GIT_EXEC_PATH} \
> > GIT_TEMPLATE_DIR='`dirname
> > $''realpath`'/${REL_GIT_TEMPLATE_DIR}
> > + SSL_CERT_DIR='`dirname
> > $''realpath`'/../../etc/ssl/certs/
> > perl_native_fixup
> > }
> >
> > ---------
> >
> > Probably because the exec of curl from git isn't getting the
> > environment ? Either
> > that, or I did it wrong. I didn't try it as an explicit export, and
> > that is probably it,
> > I can try that later tonight.
> >
> > Only when I added the depends on ca-certifications to curl itself was
> > it able to
> > fetch the autorevs.
>
> We probably need both the variable (to avoid the workdir removal issue)
> and the dependency.
>
> Cheers,
>
> Richard
>
Looks like git has its own cainfo path env variable, SSL_CERT_DIR
seems to not work. [1]
The following changes did make AUTOREV work for me:
--- a/meta/recipes-devtools/git/git_2.44.0.bb
+++ b/meta/recipes-devtools/git/git_2.44.0.bb
@@ -4,6 +4,7 @@ DESCRIPTION = "Git is a free and open source distributed version control system
SECTION = "console/utils"
LICENSE = "GPL-2.0-only & GPL-2.0-or-later & BSD-3-Clause & MIT & BSL-1.0 & LGPL-2.1-or-later"
DEPENDS = "openssl zlib"
+DEPENDS:class-native += "ca-certificates"
PROVIDES:append:class-native = " git-replacement-native"
@@ -95,6 +96,7 @@ perl_native_fixup () {
REL_GIT_EXEC_PATH = "${@os.path.relpath(libexecdir, bindir)}/git-core"
REL_GIT_TEMPLATE_DIR = "${@os.path.relpath(datadir, bindir)}/git-core/templates"
+REL_GIT_SSL_CAINFO = "${@os.path.relpath(sysconfdir, bindir)}/ssl/certs/ca-certificates.crt"
do_install:append:class-target () {
perl_native_fixup
@@ -103,6 +105,7 @@ do_install:append:class-target () {
do_install:append:class-native() {
create_wrapper ${D}${bindir}/git \
GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
+ GIT_SSL_CAINFO='`dirname $''realpath`'/${REL_GIT_SSL_CAINFO} \
GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}
}
if that is acceptable I could send a proper patch for review.
Max
[1] https://stackoverflow.com/questions/11621768/how-can-i-make-git-accept-a-self-signed-certificate
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OE-core] [REGRESSION] linux (git/curl-native) and autorev
2024-03-11 12:54 ` Max Krummenacher
@ 2024-03-11 13:57 ` Bruce Ashfield
2024-03-11 14:36 ` [PATCH v1] git: git-replacement-native: depend on ca-certificate max.oss.09
0 siblings, 1 reply; 10+ messages in thread
From: Bruce Ashfield @ 2024-03-11 13:57 UTC (permalink / raw)
To: Max Krummenacher; +Cc: Richard Purdie, openembedded-core, Max Krummenacher
On Mon, Mar 11, 2024 at 8:54 AM Max Krummenacher <max.oss.09@gmail.com> wrote:
>
> On Sun, Mar 10, 2024 at 11:39:00PM +0000, Richard Purdie wrote:
> > On Sun, 2024-03-10 at 09:05 -0700, Bruce Ashfield wrote:
> > > On Sun, Mar 10, 2024 at 11:52 AM Richard Purdie
> > > <richard.purdie@linuxfoundation.org> wrote:
> > > >
> > > > On Sun, 2024-03-10 at 06:20 -0700, Richard Purdie via
> > > > lists.openembedded.org wrote:
> > > > > On Sun, 2024-03-10 at 13:31 +0100, Max wrote:
> > > > > > Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce
> > > > > > Ashfield:
> > > > > > > On Sat, Mar 9, 2024 at 12:58 PM <max.oss.09@gmail.com> wrote:
> > > > > > > >
> > > > > > > > From: Max Krummenacher <max.krummenacher@toradex.com>
> > > > > > > >
> > > > > > > > Hello
> > > > > > > >
> > > > > > > > If one builds a kernel using AUTOREV invoking bitbake only
> > > > > > > > works
> > > > > > > > once.
> > > > > > > > Any subsequent bitbake invocation fails parsing the meta
> > > > > > > > data.
> > > > > > > >
> > > > > > > > Reproducable with:
> > > > > > > > - latest poky, b5624ee564
> > > > > > > > - Kernel with SRCREV = "AUTOREV", e.g. in local.conf
> > > > > > > > `SRCREV_machine:pn-linux-yocto:forcevariable =
> > > > > > > > "${AUTOREV}"`
> > > > > > > > - bitbake virtual/kernel; bitbake virtual/kernel
> > > > > > > >
> > > > > > > > On the second invocation parsing fails when the fetcher
> > > > > > > > tries
> > > > > > > > to
> > > > > > > > evaluate the latest SRCREV:
> > > > > > > >
> > > > > > > > > ERROR: ExpansionError during parsing meta/recipes-
> > > > > > > > > kernel/linux/linux-yocto_6.6.bb
> > > > > > > > > Traceback (most recent call last):
> > > > > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in
> > > > > > > > > srcrev_internal_helper(ud=<bb.fetch2.FetchData object at
> > > > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > > > > d.setVar("__BBAUTOREV_ACTED_UPON", True)
> > > > > > > > > > srcrev = ud.method.latest_revision(ud, d,
> > > > > > > > > name)
> > > > > > > > >
> > > > > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in
> > > > > > > > > Git.latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > > > > except KeyError:
> > > > > > > > > > revs[key] = rev =
> > > > > > > > > self._latest_revision(ud,
> > > > > > > > > d,
> > > > > > > > > name)
> > > > > > > > > return rev
> > > > > > > > > File "bitbake/lib/bb/fetch2/git.py", line 850, in
> > > > > > > > > Git._latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > > > >
> > > > > > > > > > output = self._lsremote(ud, d, "")
> > > > > > > > > # Tags of the form ^{} may not work, need to
> > > > > > > > > fallback to other form
> > > > > > > > > File "bitbake/lib/bb/fetch2/git.py", line 833, in
> > > > > > > > > Git._lsremote(ud=<bb.fetch2.FetchData object at
> > > > > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > > > > 0x7f8e26195890>, search=''):
> > > > > > > > > bb.fetch2.check_network_access(d,
> > > > > > > > > cmd,
> > > > > > > > > repourl)
> > > > > > > > > > output = runfetchcmd(cmd, d, True)
> > > > > > > > > if not output:
> > > > > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 957, in
> > > > > > > > > runfetchcmd(cmd='export PSEUDO_DISABLED=1; export
> > > > > > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > > > > > export
> > > > > > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > > > > > linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-
> > > > > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > > native/usr/bin/x86_64-poky-
> > > > > > > > > linux:build/tmp/work/qemux86_64-
> > > > > > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-
> > > > > > > > > poky-
> > > > > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-
> > > > > > > > > linux/linux-
> > > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-
> > > > > > > > > linux/linux-
> > > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > > > > > core.pager=cat
> > > > > > > > > -c safe.bareRepository=all ls-remote
> > > > > > > > > https://git.yoctoproject.org/linux-yocto.git ',
> > > > > > > > > d=<bb.data_smart.DataSmart object at 0x7f8e26195890>,
> > > > > > > > > quiet=True, cleanup=[], log=None, workdir=None):
> > > > > > > > >
> > > > > > > > > > raise FetchError(error_message)
> > > > > > > > >
> > > > > > > > > bb.data_smart.ExpansionError: Failure expanding variable
> > > > > > > > > fetcher_hashes_dummyfunc[vardepvalue], expression was
> > > > > > > > > ${@bb.fetch.get_hashvalue(d)} which triggered exception
> > > > > > > > > FetchError: Fetcher failure: Fetch command export
> > > > > > > > > PSEUDO_DISABLED=1; export
> > > > > > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > > > > > export
> > > > > > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > > > > > linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/
> > > > > > > > > work
> > > > > > > > > /q
> > > > > > > > > emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > > > > sysroot-
> > > > > > > > > native/usr/bin/x86_64-poky-
> > > > > > > > > linux:build/tmp/work/qemux86_64-
> > > > > > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-
> > > > > > > > > poky-
> > > > > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-
> > > > > > > > > linux/linux-
> > > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-
> > > > > > > > > linux/linux-
> > > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > > > > > core.pager=cat
> > > > > > > > > -c safe.bareRepository=all ls-remote
> > > > > > > > > https://git.yoctoproject.org/linux-yocto.git failed with
> > > > > > > > > exit
> > > > > > > > > code 128, output:
> > > > > > > > > fatal: unable to access
> > > > > > > > > 'https://git.yoctoproject.org/linux-yocto.git/': error
> > > > > > > > > setting
> > > > > > > > > certificate file: build/tmp/work/x86_64-linux/curl-
> > > > > > > > > native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-
> > > > > > > > > certificates.crt
> > > > > > > > >
> > > > > > > > > The variable dependency chain for the failure is:
> > > > > > > > > fetcher_hashes_dummyfunc[vardepvalue]
> > > > > > > >
> > > > > > > > Note:
> > > > > > > > One gets out of that parser error by deleting the git
> > > > > > > > binary in
> > > > > > > > the
> > > > > > > > kernel's work recipe-sysroot-native
> > > > > > > > `rm tmp/work/qemux86_64-poky-linux/linux-
> > > > > > > > yocto/6.6.20+git/recipe-
> > > > > > > > sysroot-native/usr/bin/git`
> > > > > > > >
> > > > > > > > Bisecting poky leads to commit
> > > > > > > > f7fa98cca8 ("kern-tools: depend on git-replacement-native")
> > > > > > > > Reverting it on top of b5624ee564 makes the parsing pass.
> > > > > > > >
> > > > > > > > I assume that `git-replacement-native` does not work with
> > > > > > > > https,
> > > > > > > > the
> > > > > > > > fetch error also goes away if changing in SRC_URI from
> > > > > > > > https to
> > > > > > > > git.
> > > > > > > >
> > > > > > > > Any comments?
> > > > > > >
> > > > > > > I didn't even know that curl was coming into play :)
> > > > > > >
> > > > > > > Adding DEPENDS:class-native += "ca-certificates" to the curl
> > > > > > > recipe
> > > > > > > should resolve the issue.
> > > > > >
> > > > > > Looks like curl-native resp. libcurl hardcodes the lookup to
> > > > > > its
> > > > > > own
> > > > > > work directory, i.e.:
> > > > > > x86_64-linux/curl-native/8.6.0/recipe-sysroot-
> > > > > > native/etc/ssl/certs/ca-certificates.crt
> > > > > >
> > > > > > So even if DEPENDS/RDEPENDS will install ca-certificates in the
> > > > > > kernel's
> > > > > > recipe-sysroot-native the parsing will fail if the curl-native
> > > > > > directory
> > > > > > is not/no longer populated, e.g. because curl-native came from
> > > > > > sstate
> > > > > > or
> > > > > > rm_work is in INHERIT.
> > > > >
> > > > > This all gets a bit messy.
> > > > >
> > > > > We've relied upon scripts that use openssl to set variables like:
> > > > >
> > > > > export SSL_CERT_DIR="XXXX/etc/ssl/certs/
> > > > >
> > > > > so in theory we might be able to set an environment variable in a
> > > > > wrapper around the git commands.
> > > > >
> > > > > It may be better if we teach curl a relative path to the certs...
> > > > >
> > > > > I suspect this isn't going to be an easy/neat fix unfortunately.
> > > >
> > > > I'm trying not to get sucked into "work" today however I realised
> > > > that
> > > > relative paths won't work without some implementation of "$ORIGIN"
> > > > support into these paths. Given it ultimately ends up in openssl,
> > > > it
> > > > would probably be best there.
> > > >
> > > > I'd not be against writing a $ORIGIN support patch and seeing what
> > > > upstream think about it. It would still mean finding a way to find
> > > > the
> > > > path to the library file somehow.
> > > >
> > > > For purposes of the release, setting the right envvars in the git
> > > > wrapper is probably the way forward for now, much as I dislike the
> > > > requirement to do that.
> > >
> > > I tried this, but curl didn't seem to use it to locate the cert file.
> > >
> > > i.e., I tried this:
> > >
> > > ---------------
> > >
> > > diff --git a/meta/recipes-devtools/git/git_2.44.0.bb
> > > b/meta/recipes-devtools/git/git_2.44.0.bb
> > > index e6d1470873..f6b06ec601 100644
> > > --- a/meta/recipes-devtools/git/git_2.44.0.bb
> > > +++ b/meta/recipes-devtools/git/git_2.44.0.bb
> > > @@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl"
> > > PACKAGECONFIG[cvsserver] = ""
> > > PACKAGECONFIG[svn] = ""
> > > PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native"
> > > -PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl"
> > > +PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-
> > > certificates"
> > > PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"
> > >
> > > EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl-
> > > native/perl \
> > > @@ -103,13 +103,15 @@ do_install:append:class-target () {
> > > do_install:append:class-native() {
> > > create_wrapper ${D}${bindir}/git \
> > > GIT_EXEC_PATH='`dirname
> > > $''realpath`'/${REL_GIT_EXEC_PATH} \
> > > - GIT_TEMPLATE_DIR='`dirname
> > > $''realpath`'/${REL_GIT_TEMPLATE_DIR}
> > > + GIT_TEMPLATE_DIR='`dirname
> > > $''realpath`'/${REL_GIT_TEMPLATE_DIR} \
> > > + SSL_CERT_DIR='`dirname
> > > $''realpath`'/../../etc/ssl/certs/
> > > }
> > >
> > > do_install:append:class-nativesdk() {
> > > create_wrapper ${D}${bindir}/git \
> > > GIT_EXEC_PATH='`dirname
> > > $''realpath`'/${REL_GIT_EXEC_PATH} \
> > > GIT_TEMPLATE_DIR='`dirname
> > > $''realpath`'/${REL_GIT_TEMPLATE_DIR}
> > > + SSL_CERT_DIR='`dirname
> > > $''realpath`'/../../etc/ssl/certs/
> > > perl_native_fixup
> > > }
> > >
> > > ---------
> > >
> > > Probably because the exec of curl from git isn't getting the
> > > environment ? Either
> > > that, or I did it wrong. I didn't try it as an explicit export, and
> > > that is probably it,
> > > I can try that later tonight.
> > >
> > > Only when I added the depends on ca-certifications to curl itself was
> > > it able to
> > > fetch the autorevs.
> >
> > We probably need both the variable (to avoid the workdir removal issue)
> > and the dependency.
> >
> > Cheers,
> >
> > Richard
> >
>
> Looks like git has its own cainfo path env variable, SSL_CERT_DIR
> seems to not work. [1]
>
Aha! That explains why it was ignoring my attempts :)
> The following changes did make AUTOREV work for me:
>
> --- a/meta/recipes-devtools/git/git_2.44.0.bb
> +++ b/meta/recipes-devtools/git/git_2.44.0.bb
> @@ -4,6 +4,7 @@ DESCRIPTION = "Git is a free and open source distributed version control system
> SECTION = "console/utils"
> LICENSE = "GPL-2.0-only & GPL-2.0-or-later & BSD-3-Clause & MIT & BSL-1.0 & LGPL-2.1-or-later"
> DEPENDS = "openssl zlib"
> +DEPENDS:class-native += "ca-certificates"
>
> PROVIDES:append:class-native = " git-replacement-native"
>
> @@ -95,6 +96,7 @@ perl_native_fixup () {
>
> REL_GIT_EXEC_PATH = "${@os.path.relpath(libexecdir, bindir)}/git-core"
> REL_GIT_TEMPLATE_DIR = "${@os.path.relpath(datadir, bindir)}/git-core/templates"
> +REL_GIT_SSL_CAINFO = "${@os.path.relpath(sysconfdir, bindir)}/ssl/certs/ca-certificates.crt"
>
> do_install:append:class-target () {
> perl_native_fixup
> @@ -103,6 +105,7 @@ do_install:append:class-target () {
> do_install:append:class-native() {
> create_wrapper ${D}${bindir}/git \
> GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
> + GIT_SSL_CAINFO='`dirname $''realpath`'/${REL_GIT_SSL_CAINFO} \
> GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}
> }
>
> if that is acceptable I could send a proper patch for review.
It looks fine to me. It follows the structure of the other variables.
I don't see any reason for a conditional or packageconfig dependency
on ca-certificates
(since https isn't exactly uncommon), and keeping everything contained
within the git
recipe is better this close to release.
That being said, I still wonder / think that curl should also just provide its
own dependency on ca-certificates, since it is the application that is actually
looking for them. I don't know if this way we'll run into some sort of issue
with them not being available after sstate, cleanup or something else.
Definitely send it for review, hopefully those that are well versed in those
questions I have will see it and comment :)
Bruce
>
> Max
>
> [1] https://stackoverflow.com/questions/11621768/how-can-i-make-git-accept-a-self-signed-certificate
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v1] git: git-replacement-native: depend on ca-certificate
2024-03-11 13:57 ` Bruce Ashfield
@ 2024-03-11 14:36 ` max.oss.09
0 siblings, 0 replies; 10+ messages in thread
From: max.oss.09 @ 2024-03-11 14:36 UTC (permalink / raw)
To: openembedded-core
Cc: bruce.ashfield, max.krummenacher, max.oss.09, richard.purdie
From: Max Krummenacher <max.krummenacher@toradex.com>
git is delegating webacces for URLs using TLS to libcurl.
However our native libcurl build does not find a ca-certificate.crt
unless its curl-native work dir still exists and thus git will
fail.
If a recipe uses AUTOREV with a git repo using https as its protocol
parsing of that recipe will fail fetching the latest HEAD.
Fix that by depending on ca-certificate and give its location
to libcurl via git's envrironment variable GIT_SSL_CAINFO.
Signed-off-by: Max Krummenacher <max.krummenacher@toradex.com>
---
meta/recipes-devtools/git/git_2.44.0.bb | 3 +++
1 file changed, 3 insertions(+)
See also the ML discussion:
https://lore.kernel.org/all/20240309175750.2621579-1-max.oss.09@gmail.com/
diff --git a/meta/recipes-devtools/git/git_2.44.0.bb b/meta/recipes-devtools/git/git_2.44.0.bb
index e6d1470873..90e555eba7 100644
--- a/meta/recipes-devtools/git/git_2.44.0.bb
+++ b/meta/recipes-devtools/git/git_2.44.0.bb
@@ -4,6 +4,7 @@ DESCRIPTION = "Git is a free and open source distributed version control system
SECTION = "console/utils"
LICENSE = "GPL-2.0-only & GPL-2.0-or-later & BSD-3-Clause & MIT & BSL-1.0 & LGPL-2.1-or-later"
DEPENDS = "openssl zlib"
+DEPENDS:class-native += "ca-certificates"
PROVIDES:append:class-native = " git-replacement-native"
@@ -95,6 +96,7 @@ perl_native_fixup () {
REL_GIT_EXEC_PATH = "${@os.path.relpath(libexecdir, bindir)}/git-core"
REL_GIT_TEMPLATE_DIR = "${@os.path.relpath(datadir, bindir)}/git-core/templates"
+REL_GIT_SSL_CAINFO = "${@os.path.relpath(sysconfdir, bindir)}/ssl/certs/ca-certificates.crt"
do_install:append:class-target () {
perl_native_fixup
@@ -103,6 +105,7 @@ do_install:append:class-target () {
do_install:append:class-native() {
create_wrapper ${D}${bindir}/git \
GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
+ GIT_SSL_CAINFO='`dirname $''realpath`'/${REL_GIT_SSL_CAINFO} \
GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}
}
--
2.42.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-03-11 14:37 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-03-09 17:57 [REGRESSION] linux (git/curl-native) and autorev max.oss.09
2024-03-09 21:03 ` Bruce Ashfield
2024-03-10 12:31 ` [OE-core] " Max
2024-03-10 13:20 ` Richard Purdie
[not found] ` <17BB69D476D2F4EF.5850@lists.openembedded.org>
2024-03-10 15:52 ` Richard Purdie
2024-03-10 16:04 ` Bruce Ashfield
2024-03-10 23:39 ` Richard Purdie
2024-03-11 12:54 ` Max Krummenacher
2024-03-11 13:57 ` Bruce Ashfield
2024-03-11 14:36 ` [PATCH v1] git: git-replacement-native: depend on ca-certificate max.oss.09
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox