* gitsm:// question
@ 2023-10-18 13:04 Böszörményi Zoltán
2023-10-18 13:16 ` [OE-core] " Martin Jansa
0 siblings, 1 reply; 8+ messages in thread
From: Böszörményi Zoltán @ 2023-10-18 13:04 UTC (permalink / raw)
To: openembedded-core
Hi,
I have a working python3-pytorch recipe which originally used this:
=======================================================
addtask do_git_submodules after do_unpack before do_patch
do_git_submodules[network] = "1"
do_git_submodules () {
cd ${S}
git submodule update --init --recursive
}
=======================================================
I would like to replace the above with a series of gitsm:// SRC_URI lines.
Here's a subset of submodules that shows the problem I am seeing:
SRC_URI = " \
git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 \
...
gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm
\
gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit
\
gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo
\
gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass
\
gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest
\
gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch
\
...
"
As you can see, there are recursively placed git submodules
and do_fetch fails with an error message about SRCREV_FORMAT
having to be set.
The above is a limited subset of the complete list and
some of the repositories (like googletest, gloo, glog and pybind11)
occurs multiple times as child submodules of upper ones,
with different SRCREV values in different leaf submodules.
I have not found a conclusive example to use SRCREV_FORMAT
with git:// + gitsm://
Can someone enlighten me how to use SRCREV_FORMAT properly
for this case? Should I stick to the proven working nonstandard way
and just run git submodule update --init --recursive?
As an easy way out, is there a flag to git:// to process its
submodules automatically?
Thanks in advance,
Zoltán Böszörményi
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [OE-core] gitsm:// question 2023-10-18 13:04 gitsm:// question Böszörményi Zoltán @ 2023-10-18 13:16 ` Martin Jansa 2023-10-18 13:25 ` Böszörményi Zoltán [not found] ` <178F367FEBFA030F.14388@lists.openembedded.org> 0 siblings, 2 replies; 8+ messages in thread From: Martin Jansa @ 2023-10-18 13:16 UTC (permalink / raw) To: Zoltan Boszormenyi; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 3078 bytes --] Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update. On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <zboszor@gmail.com> wrote: > Hi, > > I have a working python3-pytorch recipe which originally used this: > > ======================================================= > addtask do_git_submodules after do_unpack before do_patch > > do_git_submodules[network] = "1" > > do_git_submodules () { > cd ${S} > git submodule update --init --recursive > } > ======================================================= > > I would like to replace the above with a series of gitsm:// SRC_URI lines. > Here's a subset of submodules that shows the problem I am seeing: > > SRC_URI = " \ > git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 \ > ... > gitsm:// > github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm > \ > gitsm:// > github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit > \ > gitsm:// > github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo > \ > gitsm:// > github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass > \ > gitsm:// > github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest > \ > gitsm:// > github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch > \ > ... > " > > As you can see, there are recursively placed git submodules > and do_fetch fails with an error message about SRCREV_FORMAT > having to be set. > > The above is a limited subset of the complete list and > some of the repositories (like googletest, gloo, glog and pybind11) > occurs multiple times as child submodules of upper ones, > with different SRCREV values in different leaf submodules. > > I have not found a conclusive example to use SRCREV_FORMAT > with git:// + gitsm:// > > Can someone enlighten me how to use SRCREV_FORMAT properly > for this case? Should I stick to the proven working nonstandard way > and just run git submodule update --init --recursive? > > As an easy way out, is there a flag to git:// to process its > submodules automatically? > > Thanks in advance, > Zoltán Böszörményi > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#189388): > https://lists.openembedded.org/g/openembedded-core/message/189388 > Mute This Topic: https://lists.openembedded.org/mt/102038382/3617156 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > martin.jansa@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > > [-- Attachment #2: Type: text/html, Size: 5117 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] gitsm:// question 2023-10-18 13:16 ` [OE-core] " Martin Jansa @ 2023-10-18 13:25 ` Böszörményi Zoltán 2023-10-18 13:28 ` Julien Stephan [not found] ` <178F367FEBFA030F.14388@lists.openembedded.org> 1 sibling, 1 reply; 8+ messages in thread From: Böszörményi Zoltán @ 2023-10-18 13:25 UTC (permalink / raw) To: Martin Jansa; +Cc: openembedded-core Thanks, I will try it. 2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta: > Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably > don't care about representing the other repos in SRCPV if you're updating them only > together with main pytorch update. > > On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <zboszor@gmail.com> wrote: > > Hi, > > I have a working python3-pytorch recipe which originally used this: > > ======================================================= > addtask do_git_submodules after do_unpack before do_patch > > do_git_submodules[network] = "1" > > do_git_submodules () { > cd ${S} > git submodule update --init --recursive > } > ======================================================= > > I would like to replace the above with a series of gitsm:// SRC_URI lines. > Here's a subset of submodules that shows the problem I am seeing: > > SRC_URI = " \ > git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 > <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \ > ... > gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm > <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm> > > \ > gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit > <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit> > > \ > gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo > <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo> > > \ > gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass > <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass> > > \ > gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest > <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest> > > \ > gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch > <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch> > > \ > ... > " > > As you can see, there are recursively placed git submodules > and do_fetch fails with an error message about SRCREV_FORMAT > having to be set. > > The above is a limited subset of the complete list and > some of the repositories (like googletest, gloo, glog and pybind11) > occurs multiple times as child submodules of upper ones, > with different SRCREV values in different leaf submodules. > > I have not found a conclusive example to use SRCREV_FORMAT > with git:// + gitsm:// > > Can someone enlighten me how to use SRCREV_FORMAT properly > for this case? Should I stick to the proven working nonstandard way > and just run git submodule update --init --recursive? > > As an easy way out, is there a flag to git:// to process its > submodules automatically? > > Thanks in advance, > Zoltán Böszörményi > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#189388): > https://lists.openembedded.org/g/openembedded-core/message/189388 > Mute This Topic: https://lists.openembedded.org/mt/102038382/3617156 > Group Owner: openembedded-core+owner@lists.openembedded.org > <mailto:openembedded-core%2Bowner@lists.openembedded.org> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub > [martin.jansa@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] gitsm:// question 2023-10-18 13:25 ` Böszörményi Zoltán @ 2023-10-18 13:28 ` Julien Stephan 2023-10-18 14:12 ` Böszörményi Zoltán 0 siblings, 1 reply; 8+ messages in thread From: Julien Stephan @ 2023-10-18 13:28 UTC (permalink / raw) To: Zoltan Boszormenyi; +Cc: Martin Jansa, openembedded-core Le mer. 18 oct. 2023 à 15:25, Zoltan Boszormenyi <zboszor@gmail.com> a écrit : > > Thanks, I will try it. > > 2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta: > > Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably > > don't care about representing the other repos in SRCPV if you're updating them only > > together with main pytorch update. > > > > On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <zboszor@gmail.com> wrote: > > > > Hi, > > > > I have a working python3-pytorch recipe which originally used this: > > > > ======================================================= > > addtask do_git_submodules after do_unpack before do_patch > > > > do_git_submodules[network] = "1" > > > > do_git_submodules () { > > cd ${S} > > git submodule update --init --recursive > > } > > ======================================================= > > > > I would like to replace the above with a series of gitsm:// SRC_URI lines. > > Here's a subset of submodules that shows the problem I am seeing: > > > > SRC_URI = " \ > > git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 > > <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \ > > ... > > gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm > > <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm> > > > > \ > > gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit > > <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit> > > > > \ > > gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo > > <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo> > > > > \ > > gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass > > <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass> > > > > \ > > gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest > > <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest> > > > > \ > > gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch > > <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch> > > > > \ > > ... > > " Hi Zoltan This is not how you are supposed to use gitsm.. This is much simpler: SRC_URI = " gitsm://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 " will do all the magic for you :) You can look at the vulkan-sample recipes in poky for an example of how to use it: https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=master Cheers Julien > > > > As you can see, there are recursively placed git submodules > > and do_fetch fails with an error message about SRCREV_FORMAT > > having to be set. > > > > The above is a limited subset of the complete list and > > some of the repositories (like googletest, gloo, glog and pybind11) > > occurs multiple times as child submodules of upper ones, > > with different SRCREV values in different leaf submodules. > > > > I have not found a conclusive example to use SRCREV_FORMAT > > with git:// + gitsm:// > > > > Can someone enlighten me how to use SRCREV_FORMAT properly > > for this case? Should I stick to the proven working nonstandard way > > and just run git submodule update --init --recursive? > > > > As an easy way out, is there a flag to git:// to process its > > submodules automatically? > > > > Thanks in advance, > > Zoltán Böszörményi > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390 > Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [jstephan@baylibre.com] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] gitsm:// question 2023-10-18 13:28 ` Julien Stephan @ 2023-10-18 14:12 ` Böszörményi Zoltán 2023-10-18 14:24 ` Julien Stephan 0 siblings, 1 reply; 8+ messages in thread From: Böszörményi Zoltán @ 2023-10-18 14:12 UTC (permalink / raw) To: Julien Stephan; +Cc: Martin Jansa, openembedded-core 2023. 10. 18. 15:28 keltezéssel, Julien Stephan írta: > Le mer. 18 oct. 2023 à 15:25, Zoltan Boszormenyi <zboszor@gmail.com> a écrit : >> Thanks, I will try it. >> >> 2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta: >>> Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably >>> don't care about representing the other repos in SRCPV if you're updating them only >>> together with main pytorch update. >>> >>> On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <zboszor@gmail.com> wrote: >>> >>> Hi, >>> >>> I have a working python3-pytorch recipe which originally used this: >>> >>> ======================================================= >>> addtask do_git_submodules after do_unpack before do_patch >>> >>> do_git_submodules[network] = "1" >>> >>> do_git_submodules () { >>> cd ${S} >>> git submodule update --init --recursive >>> } >>> ======================================================= >>> >>> I would like to replace the above with a series of gitsm:// SRC_URI lines. >>> Here's a subset of submodules that shows the problem I am seeing: >>> >>> SRC_URI = " \ >>> git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 >>> <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \ >>> ... >>> gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm >>> <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm> >>> >>> \ >>> gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit >>> <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit> >>> >>> \ >>> gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo >>> <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo> >>> >>> \ >>> gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass >>> <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass> >>> >>> \ >>> gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest >>> <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest> >>> >>> \ >>> gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch >>> <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch> >>> >>> \ >>> ... >>> " > Hi Zoltan > This is not how you are supposed to use gitsm.. This is much simpler: > > SRC_URI = " gitsm://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 > " > > will do all the magic for you :) When I first created that recipe (Yocto 4.0, PyTorch 1.13.0), it didn't. The problems were: * PyTorch source releases are not on https://pypi.org/project/torch This would have been the most convenient. * Yocto complains about using the (unstable) release tarball URL from Github. * Using the git:// SRC_URI did not populate the submodules' directories. This was noticed because some submodules need patches to build under Yocto. Without checking out submodules manually, those patches fail. Just checked the last one on mickledore, still a problem. Have Yocto master changed this? > You can look at the vulkan-sample recipes in poky for an example of > how to use it: https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=master > > Cheers > Julien > > >>> As you can see, there are recursively placed git submodules >>> and do_fetch fails with an error message about SRCREV_FORMAT >>> having to be set. >>> >>> The above is a limited subset of the complete list and >>> some of the repositories (like googletest, gloo, glog and pybind11) >>> occurs multiple times as child submodules of upper ones, >>> with different SRCREV values in different leaf submodules. >>> >>> I have not found a conclusive example to use SRCREV_FORMAT >>> with git:// + gitsm:// >>> >>> Can someone enlighten me how to use SRCREV_FORMAT properly >>> for this case? Should I stick to the proven working nonstandard way >>> and just run git submodule update --init --recursive? >>> >>> As an easy way out, is there a flag to git:// to process its >>> submodules automatically? >>> >>> Thanks in advance, >>> Zoltán Böszörményi >>> >>> >>> >>> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390 >> Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723 >> Group Owner: openembedded-core+owner@lists.openembedded.org >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [jstephan@baylibre.com] >> -=-=-=-=-=-=-=-=-=-=-=- >> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] gitsm:// question 2023-10-18 14:12 ` Böszörményi Zoltán @ 2023-10-18 14:24 ` Julien Stephan 2023-10-18 14:28 ` Martin Jansa 0 siblings, 1 reply; 8+ messages in thread From: Julien Stephan @ 2023-10-18 14:24 UTC (permalink / raw) To: Böszörményi Zoltán; +Cc: Martin Jansa, openembedded-core Le mer. 18 oct. 2023 à 16:12, Böszörményi Zoltán <zboszor@gmail.com> a écrit : > > 2023. 10. 18. 15:28 keltezéssel, Julien Stephan írta: > > Le mer. 18 oct. 2023 à 15:25, Zoltan Boszormenyi <zboszor@gmail.com> a écrit : > >> Thanks, I will try it. > >> > >> 2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta: > >>> Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably > >>> don't care about representing the other repos in SRCPV if you're updating them only > >>> together with main pytorch update. > >>> > >>> On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <zboszor@gmail.com> wrote: > >>> > >>> Hi, > >>> > >>> I have a working python3-pytorch recipe which originally used this: > >>> > >>> ======================================================= > >>> addtask do_git_submodules after do_unpack before do_patch > >>> > >>> do_git_submodules[network] = "1" > >>> > >>> do_git_submodules () { > >>> cd ${S} > >>> git submodule update --init --recursive > >>> } > >>> ======================================================= > >>> > >>> I would like to replace the above with a series of gitsm:// SRC_URI lines. > >>> Here's a subset of submodules that shows the problem I am seeing: > >>> > >>> SRC_URI = " \ > >>> git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 > >>> <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \ > >>> ... > >>> gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm > >>> <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm> > >>> > >>> \ > >>> gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit > >>> <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit> > >>> > >>> \ > >>> gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo > >>> <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo> > >>> > >>> \ > >>> gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass > >>> <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass> > >>> > >>> \ > >>> gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest > >>> <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest> > >>> > >>> \ > >>> gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch > >>> <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch> > >>> > >>> \ > >>> ... > >>> " > > Hi Zoltan > > This is not how you are supposed to use gitsm.. This is much simpler: > > > > SRC_URI = " gitsm://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 > > " > > > > will do all the magic for you :) > > When I first created that recipe (Yocto 4.0, PyTorch 1.13.0), it didn't. > > The problems were: > * PyTorch source releases are not on https://pypi.org/project/torch > This would have been the most convenient. > * Yocto complains about using the (unstable) release tarball URL from Github. > * Using the git:// SRC_URI did not populate the submodules' directories. > This was noticed because some submodules need patches to build under Yocto. > Without checking out submodules manually, those patches fail. > > Just checked the last one on mickledore, still a problem. > > Have Yocto master changed this? > I used to have a kirkstone recipe using gitsm properly to fetch the submodules. Can you reproduce your issue on master? And maybe share it to help debugging? Cheers Julien > > You can look at the vulkan-sample recipes in poky for an example of > > how to use it: https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=master > > > > Cheers > > Julien > > > > > >>> As you can see, there are recursively placed git submodules > >>> and do_fetch fails with an error message about SRCREV_FORMAT > >>> having to be set. > >>> > >>> The above is a limited subset of the complete list and > >>> some of the repositories (like googletest, gloo, glog and pybind11) > >>> occurs multiple times as child submodules of upper ones, > >>> with different SRCREV values in different leaf submodules. > >>> > >>> I have not found a conclusive example to use SRCREV_FORMAT > >>> with git:// + gitsm:// > >>> > >>> Can someone enlighten me how to use SRCREV_FORMAT properly > >>> for this case? Should I stick to the proven working nonstandard way > >>> and just run git submodule update --init --recursive? > >>> > >>> As an easy way out, is there a flag to git:// to process its > >>> submodules automatically? > >>> > >>> Thanks in advance, > >>> Zoltán Böszörményi > >>> > >>> > >>> > >>> > >> > >> -=-=-=-=-=-=-=-=-=-=-=- > >> Links: You receive all messages sent to this group. > >> View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390 > >> Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723 > >> Group Owner: openembedded-core+owner@lists.openembedded.org > >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [jstephan@baylibre.com] > >> -=-=-=-=-=-=-=-=-=-=-=- > >> > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] gitsm:// question 2023-10-18 14:24 ` Julien Stephan @ 2023-10-18 14:28 ` Martin Jansa 0 siblings, 0 replies; 8+ messages in thread From: Martin Jansa @ 2023-10-18 14:28 UTC (permalink / raw) To: Julien Stephan; +Cc: Böszörményi Zoltán, openembedded-core [-- Attachment #1: Type: text/plain, Size: 7088 bytes --] > Using the git:// SRC_URI did not populate the submodules' directories. That's expected, you need to use gitsm:// to do that (at least the ones listed in https://github.com/pytorch/pytorch/blob/main/.gitmodules). I did not realize that the other gitsm:// you were added were actually submodules of pytorch itself, but Julien is right, it should work with one gitsm:// in SRC_URI (even with patches applied to various submodules in do_patch). On Wed, Oct 18, 2023 at 4:24 PM Julien Stephan <jstephan@baylibre.com> wrote: > Le mer. 18 oct. 2023 à 16:12, Böszörményi Zoltán <zboszor@gmail.com> a > écrit : > > > > 2023. 10. 18. 15:28 keltezéssel, Julien Stephan írta: > > > Le mer. 18 oct. 2023 à 15:25, Zoltan Boszormenyi <zboszor@gmail.com> > a écrit : > > >> Thanks, I will try it. > > >> > > >> 2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta: > > >>> Just add the name to pytorch repo url and use only that in > SRCREV_FORMAT, you probably > > >>> don't care about representing the other repos in SRCPV if you're > updating them only > > >>> together with main pytorch update. > > >>> > > >>> On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi < > zboszor@gmail.com> wrote: > > >>> > > >>> Hi, > > >>> > > >>> I have a working python3-pytorch recipe which originally used > this: > > >>> > > >>> ======================================================= > > >>> addtask do_git_submodules after do_unpack before do_patch > > >>> > > >>> do_git_submodules[network] = "1" > > >>> > > >>> do_git_submodules () { > > >>> cd ${S} > > >>> git submodule update --init --recursive > > >>> } > > >>> ======================================================= > > >>> > > >>> I would like to replace the above with a series of gitsm:// > SRC_URI lines. > > >>> Here's a subset of submodules that shows the problem I am > seeing: > > >>> > > >>> SRC_URI = " \ > > >>> git:// > github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 > > >>> < > http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \ > > >>> ... > > >>> gitsm:// > github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm > > >>> < > http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm > > > > >>> > > >>> \ > > >>> gitsm:// > github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit > > >>> < > http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit > > > > >>> > > >>> \ > > >>> gitsm:// > github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo > > >>> < > http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo > > > > >>> > > >>> \ > > >>> gitsm:// > github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass > > >>> < > http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass > > > > >>> > > >>> \ > > >>> gitsm:// > github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest > > >>> < > http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest > > > > >>> > > >>> \ > > >>> gitsm:// > github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch > > >>> < > http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch > > > > >>> > > >>> \ > > >>> ... > > >>> " > > > Hi Zoltan > > > This is not how you are supposed to use gitsm.. This is much simpler: > > > > > > SRC_URI = " gitsm:// > github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 > > > " > > > > > > will do all the magic for you :) > > > > When I first created that recipe (Yocto 4.0, PyTorch 1.13.0), it didn't. > > > > The problems were: > > * PyTorch source releases are not on https://pypi.org/project/torch > > This would have been the most convenient. > > * Yocto complains about using the (unstable) release tarball URL from > Github. > > * Using the git:// SRC_URI did not populate the submodules' directories. > > This was noticed because some submodules need patches to build under > Yocto. > > Without checking out submodules manually, those patches fail. > > > > Just checked the last one on mickledore, still a problem. > > > > Have Yocto master changed this? > > > > I used to have a kirkstone recipe using gitsm properly to fetch the > submodules. Can you reproduce your issue on master? And maybe share it > to help debugging? > > Cheers > Julien > > > You can look at the vulkan-sample recipes in poky for an example of > > > how to use it: > https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=master > > > > > > Cheers > > > Julien > > > > > > > > >>> As you can see, there are recursively placed git submodules > > >>> and do_fetch fails with an error message about SRCREV_FORMAT > > >>> having to be set. > > >>> > > >>> The above is a limited subset of the complete list and > > >>> some of the repositories (like googletest, gloo, glog and > pybind11) > > >>> occurs multiple times as child submodules of upper ones, > > >>> with different SRCREV values in different leaf submodules. > > >>> > > >>> I have not found a conclusive example to use SRCREV_FORMAT > > >>> with git:// + gitsm:// > > >>> > > >>> Can someone enlighten me how to use SRCREV_FORMAT properly > > >>> for this case? Should I stick to the proven working nonstandard > way > > >>> and just run git submodule update --init --recursive? > > >>> > > >>> As an easy way out, is there a flag to git:// to process its > > >>> submodules automatically? > > >>> > > >>> Thanks in advance, > > >>> Zoltán Böszörményi > > >>> > > >>> > > >>> > > >>> > > >> > > >> -=-=-=-=-=-=-=-=-=-=-=- > > >> Links: You receive all messages sent to this group. > > >> View/Reply Online (#189390): > https://lists.openembedded.org/g/openembedded-core/message/189390 > > >> Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723 > > >> Group Owner: openembedded-core+owner@lists.openembedded.org > > >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub > [jstephan@baylibre.com] > > >> -=-=-=-=-=-=-=-=-=-=-=- > > >> > > > [-- Attachment #2: Type: text/html, Size: 12363 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <178F367FEBFA030F.14388@lists.openembedded.org>]
* Re: [OE-core] gitsm:// question [not found] ` <178F367FEBFA030F.14388@lists.openembedded.org> @ 2023-10-18 16:18 ` Böszörményi Zoltán 0 siblings, 0 replies; 8+ messages in thread From: Böszörményi Zoltán @ 2023-10-18 16:18 UTC (permalink / raw) To: Martin Jansa; +Cc: openembedded-core Using a name for the main git:// SRC_URI entry and setting SRCREV_FORMAT for it works. The rest were partly my bugs: I had to fix two copy&paste errors in that massive list of gitsm:// SRC_URI entries. I also had to change a couple of them to use git:// due to the submodules' incorrect declaration in their parent repos. This is detected by Yocto and gave me the error below, indicating the repos in question. ERROR: python3-pytorch do_fetch: Fetcher failure: Submodule refers to the parent repository. This will cause deadlock situation in current version of Bitbake.Consider using git fetcher instead. 2023. 10. 18. 15:25 keltezéssel, Zoltan Boszormenyi via lists.openembedded.org írta: > Thanks, I will try it. > > 2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta: >> Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably >> don't care about representing the other repos in SRCPV if you're updating them only >> together with main pytorch update. >> >> On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <zboszor@gmail.com> wrote: >> >> Hi, >> >> I have a working python3-pytorch recipe which originally used this: >> >> ======================================================= >> addtask do_git_submodules after do_unpack before do_patch >> >> do_git_submodules[network] = "1" >> >> do_git_submodules () { >> cd ${S} >> git submodule update --init --recursive >> } >> ======================================================= >> >> I would like to replace the above with a series of gitsm:// SRC_URI lines. >> Here's a subset of submodules that shows the problem I am seeing: >> >> SRC_URI = " \ >> git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 >> <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \ >> ... >> gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm >> <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm> >> >> \ >> gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit >> <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit> >> >> \ >> gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo >> <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo> >> >> \ >> gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass >> <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass> >> >> \ >> gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest >> <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest> >> >> \ >> gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch >> <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch> >> >> \ >> ... >> " >> >> As you can see, there are recursively placed git submodules >> and do_fetch fails with an error message about SRCREV_FORMAT >> having to be set. >> >> The above is a limited subset of the complete list and >> some of the repositories (like googletest, gloo, glog and pybind11) >> occurs multiple times as child submodules of upper ones, >> with different SRCREV values in different leaf submodules. >> >> I have not found a conclusive example to use SRCREV_FORMAT >> with git:// + gitsm:// >> >> Can someone enlighten me how to use SRCREV_FORMAT properly >> for this case? Should I stick to the proven working nonstandard way >> and just run git submodule update --init --recursive? >> >> As an easy way out, is there a flag to git:// to process its >> submodules automatically? >> >> Thanks in advance, >> Zoltán Böszörményi >> >> >> > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390 > Mute This Topic: https://lists.openembedded.org/mt/102038382/3617728 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [zboszor@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2023-10-18 16:18 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-10-18 13:04 gitsm:// question Böszörményi Zoltán
2023-10-18 13:16 ` [OE-core] " Martin Jansa
2023-10-18 13:25 ` Böszörményi Zoltán
2023-10-18 13:28 ` Julien Stephan
2023-10-18 14:12 ` Böszörményi Zoltán
2023-10-18 14:24 ` Julien Stephan
2023-10-18 14:28 ` Martin Jansa
[not found] ` <178F367FEBFA030F.14388@lists.openembedded.org>
2023-10-18 16:18 ` Böszörményi Zoltán
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox