* [RFC] git: add BB_FETCH_PREMIRROR_READONLY option @ 2013-04-30 10:52 Martin Jansa 2013-04-30 12:57 ` Paul Eggleton 0 siblings, 1 reply; 9+ messages in thread From: Martin Jansa @ 2013-04-30 10:52 UTC (permalink / raw) To: bitbake-devel * just RFC, because I haven't even tested this change, use it only to describe the issue and discuss it * remove symlink to ud.fullmirror when BB_FETCH_PREMIRROR_READONLY is set * with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet). ERROR: Fetcher failure: Fetch command failed with exit code 141, output: tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system tar (child): Error is not recoverable: exiting now * maybe we can change the default behavior and always remove symlink without introducing new option Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> --- lib/bb/fetch2/git.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index 052802e..14a8bc2 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -217,6 +217,10 @@ class Git(FetchMethod): def build_mirror_data(self, url, ud, d): # Generate a mirror tarball if needed if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)): + premirror_readonly = (d.getVar("BB_FETCH_PREMIRROR_READONLY", True) == "1") + if premirror_readonly and os.path.islink(ud.fullmirror): + os.unlink(origud.localpath) + os.chdir(ud.clonedir) logger.info("Creating tarball of git repository") runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d) -- 1.8.2.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [RFC] git: add BB_FETCH_PREMIRROR_READONLY option 2013-04-30 10:52 [RFC] git: add BB_FETCH_PREMIRROR_READONLY option Martin Jansa @ 2013-04-30 12:57 ` Paul Eggleton 2013-04-30 14:46 ` Martin Jansa ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Paul Eggleton @ 2013-04-30 12:57 UTC (permalink / raw) To: Martin Jansa; +Cc: bitbake-devel On Tuesday 30 April 2013 12:52:40 Martin Jansa wrote: > * just RFC, because I haven't even tested this change, > use it only to describe the issue and discuss it > > * remove symlink to ud.fullmirror when BB_FETCH_PREMIRROR_READONLY is set > > * with read-only PREMIRROR (e.g. mounted over NFS or CIFS > and referenced as file:///mnt/premirror) we cannot use > BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz > files later became just symlinks to read-only location in PREMIRROR > (it works fine on first build and for new components, because > at that time there isn't tarball on PREMIRROR yet). > > ERROR: Fetcher failure: Fetch command failed with exit code 141, output: > tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only > file system tar (child): Error is not recoverable: exiting now > > * maybe we can change the default behavior and always remove symlink > without introducing new option Personally I think it would be better to do the latter. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC] git: add BB_FETCH_PREMIRROR_READONLY option 2013-04-30 12:57 ` Paul Eggleton @ 2013-04-30 14:46 ` Martin Jansa 2013-05-03 15:58 ` [PATCHv2] git: remove symling before updating mirror tarball Martin Jansa 2013-05-07 16:39 ` [PATCHv3] " Martin Jansa 2 siblings, 0 replies; 9+ messages in thread From: Martin Jansa @ 2013-04-30 14:46 UTC (permalink / raw) To: Paul Eggleton; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 1660 bytes --] On Tue, Apr 30, 2013 at 01:57:42PM +0100, Paul Eggleton wrote: > On Tuesday 30 April 2013 12:52:40 Martin Jansa wrote: > > * just RFC, because I haven't even tested this change, > > use it only to describe the issue and discuss it > > > > * remove symlink to ud.fullmirror when BB_FETCH_PREMIRROR_READONLY is set > > > > * with read-only PREMIRROR (e.g. mounted over NFS or CIFS > > and referenced as file:///mnt/premirror) we cannot use > > BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz > > files later became just symlinks to read-only location in PREMIRROR > > (it works fine on first build and for new components, because > > at that time there isn't tarball on PREMIRROR yet). > > > > ERROR: Fetcher failure: Fetch command failed with exit code 141, output: > > tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only > > file system tar (child): Error is not recoverable: exiting now > > > > * maybe we can change the default behavior and always remove symlink > > without introducing new option > > Personally I think it would be better to do the latter. Only case when keeping old behavior is IMHO desirable is when someone is using file:// and wants tar.gz updated in original location (e.g. to rsync less from downloads directory after build). But it won't update all (e.g. new components which don't have symlinks yet) so rsync is still needed and usefulness of such requirement is quite low. Cheers, > > Cheers, > Paul > > -- > > Paul Eggleton > Intel Open Source Technology Centre -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCHv2] git: remove symling before updating mirror tarball 2013-04-30 12:57 ` Paul Eggleton 2013-04-30 14:46 ` Martin Jansa @ 2013-05-03 15:58 ` Martin Jansa 2013-05-07 16:39 ` [PATCHv3] " Martin Jansa 2 siblings, 0 replies; 9+ messages in thread From: Martin Jansa @ 2013-05-03 15:58 UTC (permalink / raw) To: bitbake-devel * with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet). ERROR: Fetcher failure: Fetch command failed with exit code 141, output: tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system tar (child): Error is not recoverable: exiting now Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> --- lib/bb/fetch2/git.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index 052802e..1ec8908 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -217,6 +217,10 @@ class Git(FetchMethod): def build_mirror_data(self, url, ud, d): # Generate a mirror tarball if needed if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)): + # it's possible that this symlink points to read-only filesystem with PREMIRROR + if os.path.islink(ud.fullmirror): + os.unlink(origud.localpath) + os.chdir(ud.clonedir) logger.info("Creating tarball of git repository") runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d) -- 1.8.2.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCHv3] git: remove symling before updating mirror tarball 2013-04-30 12:57 ` Paul Eggleton 2013-04-30 14:46 ` Martin Jansa 2013-05-03 15:58 ` [PATCHv2] git: remove symling before updating mirror tarball Martin Jansa @ 2013-05-07 16:39 ` Martin Jansa 2013-05-07 21:51 ` Martin Jansa 2 siblings, 1 reply; 9+ messages in thread From: Martin Jansa @ 2013-05-07 16:39 UTC (permalink / raw) To: bitbake-devel * with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet). ERROR: Fetcher failure: Fetch command failed with exit code 141, output: tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system tar (child): Error is not recoverable: exiting now Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> --- lib/bb/fetch2/git.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index 052802e..6175e4c 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -217,6 +217,10 @@ class Git(FetchMethod): def build_mirror_data(self, url, ud, d): # Generate a mirror tarball if needed if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)): + # it's possible that this symlink points to read-only filesystem with PREMIRROR + if os.path.islink(ud.fullmirror): + os.unlink(ud.fullmirror) + os.chdir(ud.clonedir) logger.info("Creating tarball of git repository") runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d) -- 1.8.2.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCHv3] git: remove symling before updating mirror tarball 2013-05-07 16:39 ` [PATCHv3] " Martin Jansa @ 2013-05-07 21:51 ` Martin Jansa 2013-05-07 22:10 ` Martin Jansa 2013-05-10 9:35 ` Martin Jansa 0 siblings, 2 replies; 9+ messages in thread From: Martin Jansa @ 2013-05-07 21:51 UTC (permalink / raw) To: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 2988 bytes --] On Tue, May 07, 2013 at 06:39:20PM +0200, Martin Jansa wrote: > * with read-only PREMIRROR (e.g. mounted over NFS or CIFS > and referenced as file:///mnt/premirror) we cannot use > BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz > files later became just symlinks to read-only location in PREMIRROR > (it works fine on first build and for new components, because > at that time there isn't tarball on PREMIRROR yet). > > ERROR: Fetcher failure: Fetch command failed with exit code 141, output: > tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system > tar (child): Error is not recoverable: exiting now This one is tested and I would like to get it also in 1.16 and 1.18 Steps to reproduce the issue (without redownloading anything big) # mv downloads downloads-ro # mkdir downloads # mkdir tmp-eglibc/downloads # sudo mount -o ro,bind downloads-ro tmp-eglibc/downloads # sudo mount -o remount,ro tmp-eglibc/downloads conf/local.conf: SOURCE_MIRROR_URL = "file:///OE/oe-core/tmp-eglibc/downloads" INHERIT += "own-mirrors" # bitbake -c fetch qtbase # ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz should be a symlink bump SRCREV in qtbase_git.bb to some revision not included in git2_qt.gitorious.org.qt.qtbase.git.tar.gz # bitbake -c fetch qtbase WARNING: Failed to fetch URL git://qt.gitorious.org/qt/qtbase.git;branch=stable, attempting MIRRORS if available DEBUG: Fetcher failure: Fetch command failed with exit code 141, output: tar (child): /OE/oe-core/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz: Cannot open: Read-only file system tar (child): Error is not recoverable: exiting now apply this patch # bitbake -c fetch qtbase works fine # ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz -rw-r--r-- 1 bitbake bitbake 176M May 7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz is now proper file instead of symlink > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> > --- > lib/bb/fetch2/git.py | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py > index 052802e..6175e4c 100644 > --- a/lib/bb/fetch2/git.py > +++ b/lib/bb/fetch2/git.py > @@ -217,6 +217,10 @@ class Git(FetchMethod): > def build_mirror_data(self, url, ud, d): > # Generate a mirror tarball if needed > if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)): > + # it's possible that this symlink points to read-only filesystem with PREMIRROR > + if os.path.islink(ud.fullmirror): > + os.unlink(ud.fullmirror) > + > os.chdir(ud.clonedir) > logger.info("Creating tarball of git repository") > runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d) > -- > 1.8.2.1 > -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCHv3] git: remove symling before updating mirror tarball 2013-05-07 21:51 ` Martin Jansa @ 2013-05-07 22:10 ` Martin Jansa 2013-05-10 9:35 ` Martin Jansa 1 sibling, 0 replies; 9+ messages in thread From: Martin Jansa @ 2013-05-07 22:10 UTC (permalink / raw) To: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 4745 bytes --] On Tue, May 07, 2013 at 11:51:08PM +0200, Martin Jansa wrote: > On Tue, May 07, 2013 at 06:39:20PM +0200, Martin Jansa wrote: > > * with read-only PREMIRROR (e.g. mounted over NFS or CIFS > > and referenced as file:///mnt/premirror) we cannot use > > BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz > > files later became just symlinks to read-only location in PREMIRROR > > (it works fine on first build and for new components, because > > at that time there isn't tarball on PREMIRROR yet). > > > > ERROR: Fetcher failure: Fetch command failed with exit code 141, output: > > tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system > > tar (child): Error is not recoverable: exiting now > > This one is tested and I would like to get it also in 1.16 and 1.18 > > Steps to reproduce the issue (without redownloading anything big) > > # mv downloads downloads-ro > # mkdir downloads > # mkdir tmp-eglibc/downloads > # sudo mount -o ro,bind downloads-ro tmp-eglibc/downloads > # sudo mount -o remount,ro tmp-eglibc/downloads > > conf/local.conf: > SOURCE_MIRROR_URL = "file:///OE/oe-core/tmp-eglibc/downloads" > INHERIT += "own-mirrors" > > # bitbake -c fetch qtbase > # ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz > should be a symlink > > bump SRCREV in qtbase_git.bb to some revision not included in > git2_qt.gitorious.org.qt.qtbase.git.tar.gz > > # bitbake -c fetch qtbase > WARNING: Failed to fetch URL git://qt.gitorious.org/qt/qtbase.git;branch=stable, attempting MIRRORS if available > DEBUG: Fetcher failure: Fetch command failed with exit code 141, output: > tar (child): /OE/oe-core/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz: Cannot open: Read-only file system > tar (child): Error is not recoverable: exiting now > > apply this patch > # bitbake -c fetch qtbase > works fine > # ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz > -rw-r--r-- 1 bitbake bitbake 176M May 7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz > is now proper file instead of symlink There is one more interesting behavior with such setup where file:// fetcher is used in PREMIRROR AFAIK it's completely harmless and also not influenced by this change, but .done stamps for files downloaded from premirror are stored in ${DL_DIR}<path-to-local-premirror>/file.done, e.g. downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done qtbase was recreated locally thanks to this patch, that's why it has 2 .done stamps, one in downloads, 2nd in downloads/OE/oe-core/tmp-eglibc/downloads/ qtdeclarative has only symlink and .done stamp in downloads/OE/oe-core/tmp-eglibc/downloads/ OE @ ~/oe-core $ ll downloads/*qt{base,declarative}.git* -rw-r--r-- 1 bitbake bitbake 184015350 May 7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz -rw-r--r-- 1 bitbake bitbake 0 May 7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done lrwxrwxrwx 1 bitbake bitbake 82 May 8 00:00 downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz -> /OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz OE @ ~/oe-core $ ll downloads/OE/oe-core/tmp-eglibc/downloads/*qt{base,declarative}.git* -rw-r--r-- 1 bitbake bitbake 0 May 7 23:41 downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done -rw-r--r-- 1 bitbake bitbake 0 May 8 00:00 downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz.done > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> > > --- > > lib/bb/fetch2/git.py | 4 ++++ > > 1 file changed, 4 insertions(+) > > > > diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py > > index 052802e..6175e4c 100644 > > --- a/lib/bb/fetch2/git.py > > +++ b/lib/bb/fetch2/git.py > > @@ -217,6 +217,10 @@ class Git(FetchMethod): > > def build_mirror_data(self, url, ud, d): > > # Generate a mirror tarball if needed > > if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)): > > + # it's possible that this symlink points to read-only filesystem with PREMIRROR > > + if os.path.islink(ud.fullmirror): > > + os.unlink(ud.fullmirror) > > + > > os.chdir(ud.clonedir) > > logger.info("Creating tarball of git repository") > > runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d) > > -- > > 1.8.2.1 > > > > -- > Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCHv3] git: remove symling before updating mirror tarball 2013-05-07 21:51 ` Martin Jansa 2013-05-07 22:10 ` Martin Jansa @ 2013-05-10 9:35 ` Martin Jansa 2013-05-10 12:32 ` Richard Purdie 1 sibling, 1 reply; 9+ messages in thread From: Martin Jansa @ 2013-05-10 9:35 UTC (permalink / raw) To: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 1120 bytes --] On Tue, May 07, 2013 at 11:51:08PM +0200, Martin Jansa wrote: > On Tue, May 07, 2013 at 06:39:20PM +0200, Martin Jansa wrote: > > * with read-only PREMIRROR (e.g. mounted over NFS or CIFS > > and referenced as file:///mnt/premirror) we cannot use > > BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz > > files later became just symlinks to read-only location in PREMIRROR > > (it works fine on first build and for new components, because > > at that time there isn't tarball on PREMIRROR yet). > > > > ERROR: Fetcher failure: Fetch command failed with exit code 141, output: > > tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system > > tar (child): Error is not recoverable: exiting now > > This one is tested and I would like to get it also in 1.16 and 1.18 cherry-picking 3627b02f77c78beedadadd77c619b9e5edaae076 works fine for both, do you prefer to send patches with [1.16]/[1.18] in subject or should I just wait a bit more with patch in master and then ping you later? -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCHv3] git: remove symling before updating mirror tarball 2013-05-10 9:35 ` Martin Jansa @ 2013-05-10 12:32 ` Richard Purdie 0 siblings, 0 replies; 9+ messages in thread From: Richard Purdie @ 2013-05-10 12:32 UTC (permalink / raw) To: Martin Jansa; +Cc: bitbake-devel On Fri, 2013-05-10 at 11:35 +0200, Martin Jansa wrote: > On Tue, May 07, 2013 at 11:51:08PM +0200, Martin Jansa wrote: > > On Tue, May 07, 2013 at 06:39:20PM +0200, Martin Jansa wrote: > > > * with read-only PREMIRROR (e.g. mounted over NFS or CIFS > > > and referenced as file:///mnt/premirror) we cannot use > > > BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz > > > files later became just symlinks to read-only location in PREMIRROR > > > (it works fine on first build and for new components, because > > > at that time there isn't tarball on PREMIRROR yet). > > > > > > ERROR: Fetcher failure: Fetch command failed with exit code 141, output: > > > tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system > > > tar (child): Error is not recoverable: exiting now > > > > This one is tested and I would like to get it also in 1.16 and 1.18 > > cherry-picking 3627b02f77c78beedadadd77c619b9e5edaae076 works fine for > both, do you prefer to send patches with [1.16]/[1.18] in subject or > should I just wait a bit more with patch in master and then ping you > later? Just give this a short while, then we can backport. I don't like instantly backporting to other releases... Cheers, Richard ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2013-05-10 12:50 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-04-30 10:52 [RFC] git: add BB_FETCH_PREMIRROR_READONLY option Martin Jansa 2013-04-30 12:57 ` Paul Eggleton 2013-04-30 14:46 ` Martin Jansa 2013-05-03 15:58 ` [PATCHv2] git: remove symling before updating mirror tarball Martin Jansa 2013-05-07 16:39 ` [PATCHv3] " Martin Jansa 2013-05-07 21:51 ` Martin Jansa 2013-05-07 22:10 ` Martin Jansa 2013-05-10 9:35 ` Martin Jansa 2013-05-10 12:32 ` Richard Purdie
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.