* [PATCH 0/1] fetch2/git: Fix File name too long error @ 2024-04-08 8:42 liezhi.yang 2024-04-08 8:42 ` [PATCH 1/1] " liezhi.yang 0 siblings, 1 reply; 5+ messages in thread From: liezhi.yang @ 2024-04-08 8:42 UTC (permalink / raw) To: bitbake-devel From: Robert Yang <liezhi.yang@windriver.com> The following changes since commit 110ee701b33600f14e40544bf9354774cd85c5ce: sstatesig: fix netrc.NetrcParseError exception (2024-04-04 21:50:24 +0100) are available in the Git repository at: https://github.com/robertlinux/yocto rbt/long https://github.com/robertlinux/yocto/tree/rbt/long Robert Yang (1): fetch2/git: Fix File name too long error bitbake/lib/bb/fetch2/git.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) -- 2.42.0 ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/1] fetch2/git: Fix File name too long error 2024-04-08 8:42 [PATCH 0/1] fetch2/git: Fix File name too long error liezhi.yang @ 2024-04-08 8:42 ` liezhi.yang 2024-04-10 17:01 ` [bitbake-devel] " Alexandre Belloni 0 siblings, 1 reply; 5+ messages in thread From: liezhi.yang @ 2024-04-08 8:42 UTC (permalink / raw) To: bitbake-devel From: Robert Yang <liezhi.yang@windriver.com> There are "File name too long" errors when local PREMIRROR/MIRROR is in a deep directory (len(path) > 255 (The NAME_MAX)). This is becuase ud.path.replace() changes the path to filename, and the errors when happens when len(path) > 255. Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and avoid potential file name conflicts, now the files are like: $ bitbake mtd-utils-native -cfetch $ ls downloads/git2 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git git.infradead.org.mtd-utils.git -> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git git.infradead.org.mtd-utils.git.done The 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is gitsrcname. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- bitbake/lib/bb/fetch2/git.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index c7ff769fdfe..7ed51af336d 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -83,6 +83,7 @@ import subprocess import tempfile import bb import bb.progress +import hashlib from contextlib import contextmanager from bb.fetch2 import FetchMethod from bb.fetch2 import runfetchcmd @@ -277,7 +278,10 @@ class Git(FetchMethod): ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name) - gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file": + gitsrcname = '%s.%s' % (hashlib.md5(ud.path.encode('utf-8')).hexdigest(), os.path.basename(ud.path)) + else: + gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'): gitsrcname = gitsrcname[1:] -- 2.42.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [bitbake-devel] [PATCH 1/1] fetch2/git: Fix File name too long error 2024-04-08 8:42 ` [PATCH 1/1] " liezhi.yang @ 2024-04-10 17:01 ` Alexandre Belloni 2024-06-22 4:22 ` Robert Yang [not found] ` <17DB38D5908AF9B5.21660@lists.openembedded.org> 0 siblings, 2 replies; 5+ messages in thread From: Alexandre Belloni @ 2024-04-10 17:01 UTC (permalink / raw) To: liezhi.yang; +Cc: bitbake-devel I believe this causes the following error: ====================================================================== FAIL: test_shallow_submodules (bb.tests.fetch.GitShallowTest.test_shallow_submodules) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1946, in test_shallow_submodules self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule')) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1728, in assertRevCount self.assertEqual(expected_count, actual_count, msg='Object count `%d` is not the expected `%d`' % (actual_count, expected_count)) AssertionError: 1 != 2 : Object count `2` is not the expected `1` https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6634/steps/11/logs/stdio On 08/04/2024 01:42:58-0700, Robert Yang via lists.openembedded.org wrote: > From: Robert Yang <liezhi.yang@windriver.com> > > There are "File name too long" errors when local PREMIRROR/MIRROR > is in a deep directory (len(path) > 255 (The NAME_MAX)). > > This is becuase ud.path.replace() changes the path to filename, and > the errors when happens when len(path) > 255. > > Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and > avoid potential file name conflicts, now the files are like: > > $ bitbake mtd-utils-native -cfetch > $ ls downloads/git2 > 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git > git.infradead.org.mtd-utils.git -> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git > git.infradead.org.mtd-utils.git.done > > The 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is gitsrcname. > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > --- > bitbake/lib/bb/fetch2/git.py | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py > index c7ff769fdfe..7ed51af336d 100644 > --- a/bitbake/lib/bb/fetch2/git.py > +++ b/bitbake/lib/bb/fetch2/git.py > @@ -83,6 +83,7 @@ import subprocess > import tempfile > import bb > import bb.progress > +import hashlib > from contextlib import contextmanager > from bb.fetch2 import FetchMethod > from bb.fetch2 import runfetchcmd > @@ -277,7 +278,10 @@ class Git(FetchMethod): > ud.unresolvedrev[name] = ud.revisions[name] > ud.revisions[name] = self.latest_revision(ud, d, name) > > - gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) > + if ud.proto == "file": > + gitsrcname = '%s.%s' % (hashlib.md5(ud.path.encode('utf-8')).hexdigest(), os.path.basename(ud.path)) > + else: > + gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) > if gitsrcname.startswith('.'): > gitsrcname = gitsrcname[1:] > > -- > 2.42.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#16064): https://lists.openembedded.org/g/bitbake-devel/message/16064 > Mute This Topic: https://lists.openembedded.org/mt/105397591/3617179 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- > -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [bitbake-devel] [PATCH 1/1] fetch2/git: Fix File name too long error 2024-04-10 17:01 ` [bitbake-devel] " Alexandre Belloni @ 2024-06-22 4:22 ` Robert Yang [not found] ` <17DB38D5908AF9B5.21660@lists.openembedded.org> 1 sibling, 0 replies; 5+ messages in thread From: Robert Yang @ 2024-06-22 4:22 UTC (permalink / raw) To: Alexandre Belloni; +Cc: bitbake-devel On 4/11/24 01:01, Alexandre Belloni wrote: > > I believe this causes the following error: Sorry, after more thoughts, there are two issues: * The generated mirror tarball can't be used since the local path is added to to the tarball name. * There might be file name too long error when the local path is too long. It doesn't make sense to add local path to the generated mirror tarball name, but causes the above two issue, use basename can fix both of them. I will send a V2 for it. // Robert > > ====================================================================== > FAIL: test_shallow_submodules (bb.tests.fetch.GitShallowTest.test_shallow_submodules) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1946, in test_shallow_submodules > self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule')) > File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1728, in assertRevCount > self.assertEqual(expected_count, actual_count, msg='Object count `%d` is not the expected `%d`' % (actual_count, expected_count)) > AssertionError: 1 != 2 : Object count `2` is not the expected `1` > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6634/steps/11/logs/stdio > > On 08/04/2024 01:42:58-0700, Robert Yang via lists.openembedded.org wrote: >> From: Robert Yang <liezhi.yang@windriver.com> >> >> There are "File name too long" errors when local PREMIRROR/MIRROR >> is in a deep directory (len(path) > 255 (The NAME_MAX)). >> >> This is becuase ud.path.replace() changes the path to filename, and >> the errors when happens when len(path) > 255. >> >> Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and >> avoid potential file name conflicts, now the files are like: >> >> $ bitbake mtd-utils-native -cfetch >> $ ls downloads/git2 >> 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git >> git.infradead.org.mtd-utils.git -> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git >> git.infradead.org.mtd-utils.git.done >> >> The 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is gitsrcname. >> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >> --- >> bitbake/lib/bb/fetch2/git.py | 6 +++++- >> 1 file changed, 5 insertions(+), 1 deletion(-) >> >> diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py >> index c7ff769fdfe..7ed51af336d 100644 >> --- a/bitbake/lib/bb/fetch2/git.py >> +++ b/bitbake/lib/bb/fetch2/git.py >> @@ -83,6 +83,7 @@ import subprocess >> import tempfile >> import bb >> import bb.progress >> +import hashlib >> from contextlib import contextmanager >> from bb.fetch2 import FetchMethod >> from bb.fetch2 import runfetchcmd >> @@ -277,7 +278,10 @@ class Git(FetchMethod): >> ud.unresolvedrev[name] = ud.revisions[name] >> ud.revisions[name] = self.latest_revision(ud, d, name) >> >> - gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) >> + if ud.proto == "file": >> + gitsrcname = '%s.%s' % (hashlib.md5(ud.path.encode('utf-8')).hexdigest(), os.path.basename(ud.path)) >> + else: >> + gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) >> if gitsrcname.startswith('.'): >> gitsrcname = gitsrcname[1:] >> >> -- >> 2.42.0 >> > >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#16064): https://lists.openembedded.org/g/bitbake-devel/message/16064 >> Mute This Topic: https://lists.openembedded.org/mt/105397591/3617179 >> Group Owner: bitbake-devel+owner@lists.openembedded.org >> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com] >> -=-=-=-=-=-=-=-=-=-=-=- >> > > ^ permalink raw reply [flat|nested] 5+ messages in thread
[parent not found: <17DB38D5908AF9B5.21660@lists.openembedded.org>]
* Re: [bitbake-devel] [PATCH 1/1] fetch2/git: Fix File name too long error [not found] ` <17DB38D5908AF9B5.21660@lists.openembedded.org> @ 2024-06-22 5:21 ` Robert Yang 0 siblings, 0 replies; 5+ messages in thread From: Robert Yang @ 2024-06-22 5:21 UTC (permalink / raw) To: Alexandre Belloni; +Cc: bitbake-devel On 6/22/24 12:22, Robert Yang via lists.openembedded.org wrote: > > > On 4/11/24 01:01, Alexandre Belloni wrote: >> >> I believe this causes the following error: > > Sorry, after more thoughts, there are two issues: > * The generated mirror tarball can't be used since the local path is added to > to the tarball name. > > * There might be file name too long error when the local path is too long. > > It doesn't make sense to add local path to the generated mirror tarball name, > but causes the above two issue, use basename can fix both of them. > > I will send a V2 for it. I've sent a new version rather than V2 since the subject is changed: [bitbake-devel] [PATCH 0/1] fetch2/git: Use basename for gitsrcname when ud.proto is file // Robert > > // Robert > >> >> ====================================================================== >> FAIL: test_shallow_submodules >> (bb.tests.fetch.GitShallowTest.test_shallow_submodules) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1946, in test_shallow_submodules >> self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule')) >> File >> "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1728, in assertRevCount >> self.assertEqual(expected_count, actual_count, msg='Object count `%d` is >> not the expected `%d`' % (actual_count, expected_count)) >> AssertionError: 1 != 2 : Object count `2` is not the expected `1` >> >> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6634/steps/11/logs/stdio >> >> On 08/04/2024 01:42:58-0700, Robert Yang via lists.openembedded.org wrote: >>> From: Robert Yang <liezhi.yang@windriver.com> >>> >>> There are "File name too long" errors when local PREMIRROR/MIRROR >>> is in a deep directory (len(path) > 255 (The NAME_MAX)). >>> >>> This is becuase ud.path.replace() changes the path to filename, and >>> the errors when happens when len(path) > 255. >>> >>> Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and >>> avoid potential file name conflicts, now the files are like: >>> >>> $ bitbake mtd-utils-native -cfetch >>> $ ls downloads/git2 >>> 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git >>> git.infradead.org.mtd-utils.git -> >>> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git >>> git.infradead.org.mtd-utils.git.done >>> >>> The 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is >>> gitsrcname. >>> >>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >>> --- >>> bitbake/lib/bb/fetch2/git.py | 6 +++++- >>> 1 file changed, 5 insertions(+), 1 deletion(-) >>> >>> diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py >>> index c7ff769fdfe..7ed51af336d 100644 >>> --- a/bitbake/lib/bb/fetch2/git.py >>> +++ b/bitbake/lib/bb/fetch2/git.py >>> @@ -83,6 +83,7 @@ import subprocess >>> import tempfile >>> import bb >>> import bb.progress >>> +import hashlib >>> from contextlib import contextmanager >>> from bb.fetch2 import FetchMethod >>> from bb.fetch2 import runfetchcmd >>> @@ -277,7 +278,10 @@ class Git(FetchMethod): >>> ud.unresolvedrev[name] = ud.revisions[name] >>> ud.revisions[name] = self.latest_revision(ud, d, name) >>> - gitsrcname = '%s%s' % (ud.host.replace(':', '.'), >>> ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', >>> '_').replace(')', '_')) >>> + if ud.proto == "file": >>> + gitsrcname = '%s.%s' % >>> (hashlib.md5(ud.path.encode('utf-8')).hexdigest(), os.path.basename(ud.path)) >>> + else: >>> + gitsrcname = '%s%s' % (ud.host.replace(':', '.'), >>> ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', >>> '_').replace(')', '_')) >>> if gitsrcname.startswith('.'): >>> gitsrcname = gitsrcname[1:] >>> -- >>> 2.42.0 >>> >> >>> >>> >>> >> >> > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#16370): https://lists.openembedded.org/g/bitbake-devel/message/16370 > Mute This Topic: https://lists.openembedded.org/mt/105397591/3616940 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [liezhi.yang@windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-06-22 5:21 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-04-08 8:42 [PATCH 0/1] fetch2/git: Fix File name too long error liezhi.yang
2024-04-08 8:42 ` [PATCH 1/1] " liezhi.yang
2024-04-10 17:01 ` [bitbake-devel] " Alexandre Belloni
2024-06-22 4:22 ` Robert Yang
[not found] ` <17DB38D5908AF9B5.21660@lists.openembedded.org>
2024-06-22 5:21 ` Robert Yang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.