* Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files [not found] <1285368419-12062-1-git-send-email-raj.khem@gmail.com> @ 2010-10-24 6:58 ` Martin Jansa 2010-10-24 7:40 ` Khem Raj 0 siblings, 1 reply; 8+ messages in thread From: Martin Jansa @ 2010-10-24 6:58 UTC (permalink / raw) To: Khem Raj; +Cc: bitbake-dev, openembedded-devel This commit is usefull for sure, but breaks ie some EFL recipes. ie python-elementary depends on elementary >= 0.7.0.52890 but elementary.pc says Version: 0.7.0.0 elementary.pc.in says Version: @VERSION@ and finally VERSION is defined in configure.ac: m4_define([v_maj], [0]) m4_define([v_min], [7]) m4_define([v_mic], [0]) m4_define([v_rev], m4_esyscmd([(svnversion "${SVN_REPO_PATH:-.}" | grep -v export || echo 0) | awk -F : '{printf("%s\n", $1);}' | tr -d ' :MSP\n'])) m4_if(v_rev, [0], [m4_define([v_rev], m4_esyscmd([git log 2> /dev/null | (grep -m1 git-svn-id || echo 0) | sed -e 's/.*@\([0-9]*\).*/\1/' | tr -d '\n']))]) and this works OK on checkout with .svn dirs included, but results to 0 after this patch. Any hints how to resolve it? Regards, On Sat, Sep 25, 2010 at 12:46 AM, Khem Raj <raj.khem@gmail.com> wrote: > This option will exclude the SCM metadata from tar files. > > Tested with gcc where svn tar which used to be 156M for gcc 4.5 > is now 77M > > Signed-off-by: Khem Raj <raj.khem@gmail.com> > --- > lib/bb/fetch/bzr.py | 2 +- > lib/bb/fetch/cvs.py | 2 +- > lib/bb/fetch/git.py | 2 +- > lib/bb/fetch/hg.py | 2 +- > lib/bb/fetch/svn.py | 2 +- > 5 files changed, 5 insertions(+), 5 deletions(-) > > diff --git a/lib/bb/fetch/bzr.py b/lib/bb/fetch/bzr.py > index 8b0bd9f..3ca125f 100644 > --- a/lib/bb/fetch/bzr.py > +++ b/lib/bb/fetch/bzr.py > @@ -107,7 +107,7 @@ class Bzr(Fetch): > os.chdir(ud.pkgdir) > # tar them up to a defined filename > try: > - runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.basename(ud.pkgdir)), d) > + runfetchcmd("tar --exclude '.bzr' --exclude '.bzrtags' -czf %s %s" % (ud.localpath, os.path.basename(ud.pkgdir)), d) > except: > t, v, tb = sys.exc_info() > try: > diff --git a/lib/bb/fetch/cvs.py b/lib/bb/fetch/cvs.py > index 1064b09..42d71ba 100644 > --- a/lib/bb/fetch/cvs.py > +++ b/lib/bb/fetch/cvs.py > @@ -162,7 +162,7 @@ class Cvs(Fetch): > # tar them up to a defined filename > if 'fullpath' in ud.parm: > os.chdir(pkgdir) > - myret = os.system("tar -czf %s %s" % (ud.localpath, localdir)) > + myret = os.system("tar --exclude 'CVS' -czf %s %s" % (ud.localpath, localdir)) > else: > os.chdir(moddir) > os.chdir('..') > diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py > index 1b1bc95..4acaa87 100644 > --- a/lib/bb/fetch/git.py > +++ b/lib/bb/fetch/git.py > @@ -149,7 +149,7 @@ class Git(Fetch): > > os.chdir(codir) > logger.info("Creating tarball of git checkout") > - runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d) > + runfetchcmd("tar --exclude '.git' -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d) > > os.chdir(ud.clonedir) > bb.utils.prunedir(codir) > diff --git a/lib/bb/fetch/hg.py b/lib/bb/fetch/hg.py > index ab00d43..9c11deb 100644 > --- a/lib/bb/fetch/hg.py > +++ b/lib/bb/fetch/hg.py > @@ -145,7 +145,7 @@ class Hg(Fetch): > > os.chdir(ud.pkgdir) > try: > - runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d) > + runfetchcmd("tar --exclude '.hg' --exclude '.hgrags' -czf %s %s" % (ud.localpath, ud.module), d) > except: > t, v, tb = sys.exc_info() > try: > diff --git a/lib/bb/fetch/svn.py b/lib/bb/fetch/svn.py > index 34f8132..c236e41 100644 > --- a/lib/bb/fetch/svn.py > +++ b/lib/bb/fetch/svn.py > @@ -159,7 +159,7 @@ class Svn(Fetch): > os.chdir(ud.pkgdir) > # tar them up to a defined filename > try: > - runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d) > + runfetchcmd("tar --exclude '.svn' -czf %s %s" % (ud.localpath, ud.module), d) > except: > t, v, tb = sys.exc_info() > try: > -- > 1.7.1 > > _______________________________________________ > Bitbake-dev mailing list > Bitbake-dev@lists.berlios.de > https://lists.berlios.de/mailman/listinfo/bitbake-dev > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files 2010-10-24 6:58 ` [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files Martin Jansa @ 2010-10-24 7:40 ` Khem Raj 2010-10-24 8:08 ` Martin Jansa 0 siblings, 1 reply; 8+ messages in thread From: Khem Raj @ 2010-10-24 7:40 UTC (permalink / raw) To: Martin Jansa; +Cc: bitbake-dev, openembedded-devel On Sat, Oct 23, 2010 at 11:58 PM, Martin Jansa <martin.jansa@gmail.com> wrote: > This commit is usefull for sure, but breaks ie some EFL recipes. > > ie python-elementary depends on elementary >= 0.7.0.52890 > but elementary.pc says Version: 0.7.0.0 > > elementary.pc.in says Version: @VERSION@ > > and finally VERSION is defined in configure.ac: > m4_define([v_maj], [0]) > m4_define([v_min], [7]) > m4_define([v_mic], [0]) > m4_define([v_rev], m4_esyscmd([(svnversion "${SVN_REPO_PATH:-.}" | > grep -v export || echo 0) | awk -F : '{printf("%s\n", $1);}' | tr -d ' > :MSP\n'])) > m4_if(v_rev, [0], [m4_define([v_rev], m4_esyscmd([git log 2> /dev/null > | (grep -m1 git-svn-id || echo 0) | sed -e 's/.*@\([0-9]*\).*/\1/' | > tr -d '\n']))]) > hmm strange that a configure scripts depends on SCM files. To me it seems wrong. Anyway I think you can patch configure.ac to replace v_rev definition to be based on SRCREV may be some sort of sed operation as do_configure_prepend might be a good way > and this works OK on checkout with .svn dirs included, but results to > 0 after this patch. > > Any hints how to resolve it? > > Regards, > > On Sat, Sep 25, 2010 at 12:46 AM, Khem Raj <raj.khem@gmail.com> wrote: >> This option will exclude the SCM metadata from tar files. >> >> Tested with gcc where svn tar which used to be 156M for gcc 4.5 >> is now 77M >> >> Signed-off-by: Khem Raj <raj.khem@gmail.com> >> --- >> lib/bb/fetch/bzr.py | 2 +- >> lib/bb/fetch/cvs.py | 2 +- >> lib/bb/fetch/git.py | 2 +- >> lib/bb/fetch/hg.py | 2 +- >> lib/bb/fetch/svn.py | 2 +- >> 5 files changed, 5 insertions(+), 5 deletions(-) >> >> diff --git a/lib/bb/fetch/bzr.py b/lib/bb/fetch/bzr.py >> index 8b0bd9f..3ca125f 100644 >> --- a/lib/bb/fetch/bzr.py >> +++ b/lib/bb/fetch/bzr.py >> @@ -107,7 +107,7 @@ class Bzr(Fetch): >> os.chdir(ud.pkgdir) >> # tar them up to a defined filename >> try: >> - runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.basename(ud.pkgdir)), d) >> + runfetchcmd("tar --exclude '.bzr' --exclude '.bzrtags' -czf %s %s" % (ud.localpath, os.path.basename(ud.pkgdir)), d) >> except: >> t, v, tb = sys.exc_info() >> try: >> diff --git a/lib/bb/fetch/cvs.py b/lib/bb/fetch/cvs.py >> index 1064b09..42d71ba 100644 >> --- a/lib/bb/fetch/cvs.py >> +++ b/lib/bb/fetch/cvs.py >> @@ -162,7 +162,7 @@ class Cvs(Fetch): >> # tar them up to a defined filename >> if 'fullpath' in ud.parm: >> os.chdir(pkgdir) >> - myret = os.system("tar -czf %s %s" % (ud.localpath, localdir)) >> + myret = os.system("tar --exclude 'CVS' -czf %s %s" % (ud.localpath, localdir)) >> else: >> os.chdir(moddir) >> os.chdir('..') >> diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py >> index 1b1bc95..4acaa87 100644 >> --- a/lib/bb/fetch/git.py >> +++ b/lib/bb/fetch/git.py >> @@ -149,7 +149,7 @@ class Git(Fetch): >> >> os.chdir(codir) >> logger.info("Creating tarball of git checkout") >> - runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d) >> + runfetchcmd("tar --exclude '.git' -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d) >> >> os.chdir(ud.clonedir) >> bb.utils.prunedir(codir) >> diff --git a/lib/bb/fetch/hg.py b/lib/bb/fetch/hg.py >> index ab00d43..9c11deb 100644 >> --- a/lib/bb/fetch/hg.py >> +++ b/lib/bb/fetch/hg.py >> @@ -145,7 +145,7 @@ class Hg(Fetch): >> >> os.chdir(ud.pkgdir) >> try: >> - runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d) >> + runfetchcmd("tar --exclude '.hg' --exclude '.hgrags' -czf %s %s" % (ud.localpath, ud.module), d) >> except: >> t, v, tb = sys.exc_info() >> try: >> diff --git a/lib/bb/fetch/svn.py b/lib/bb/fetch/svn.py >> index 34f8132..c236e41 100644 >> --- a/lib/bb/fetch/svn.py >> +++ b/lib/bb/fetch/svn.py >> @@ -159,7 +159,7 @@ class Svn(Fetch): >> os.chdir(ud.pkgdir) >> # tar them up to a defined filename >> try: >> - runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d) >> + runfetchcmd("tar --exclude '.svn' -czf %s %s" % (ud.localpath, ud.module), d) >> except: >> t, v, tb = sys.exc_info() >> try: >> -- >> 1.7.1 >> >> _______________________________________________ >> Bitbake-dev mailing list >> Bitbake-dev@lists.berlios.de >> https://lists.berlios.de/mailman/listinfo/bitbake-dev >> > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files 2010-10-24 7:40 ` Khem Raj @ 2010-10-24 8:08 ` Martin Jansa 2010-10-24 18:35 ` Khem Raj 0 siblings, 1 reply; 8+ messages in thread From: Martin Jansa @ 2010-10-24 8:08 UTC (permalink / raw) To: Khem Raj; +Cc: bitbake-dev, openembedded-devel On Sun, Oct 24, 2010 at 9:40 AM, Khem Raj <raj.khem@gmail.com> wrote: > On Sat, Oct 23, 2010 at 11:58 PM, Martin Jansa <martin.jansa@gmail.com> wrote: >> This commit is usefull for sure, but breaks ie some EFL recipes. >> >> ie python-elementary depends on elementary >= 0.7.0.52890 >> but elementary.pc says Version: 0.7.0.0 >> >> elementary.pc.in says Version: @VERSION@ >> >> and finally VERSION is defined in configure.ac: >> m4_define([v_maj], [0]) >> m4_define([v_min], [7]) >> m4_define([v_mic], [0]) >> m4_define([v_rev], m4_esyscmd([(svnversion "${SVN_REPO_PATH:-.}" | >> grep -v export || echo 0) | awk -F : '{printf("%s\n", $1);}' | tr -d ' >> :MSP\n'])) >> m4_if(v_rev, [0], [m4_define([v_rev], m4_esyscmd([git log 2> /dev/null >> | (grep -m1 git-svn-id || echo 0) | sed -e 's/.*@\([0-9]*\).*/\1/' | >> tr -d '\n']))]) >> > > hmm strange that a configure scripts depends on SCM files. To me it > seems wrong. Anyway I think you can patch configure.ac > to replace v_rev definition to be based on SRCREV may be some sort of > sed operation as do_configure_prepend might be a good way this is used in all efl recipes and I've noticed it in elementary only because python-elementary depends on such new version. I've also found similar construct in other projects.. so maybe it's not so efl related (and then harder to found&fix in all recipes). Regards, ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files 2010-10-24 8:08 ` Martin Jansa @ 2010-10-24 18:35 ` Khem Raj 2010-10-24 19:56 ` Holger Freyther 0 siblings, 1 reply; 8+ messages in thread From: Khem Raj @ 2010-10-24 18:35 UTC (permalink / raw) To: Martin Jansa; +Cc: bitbake-dev, openembedded-devel On Sun, Oct 24, 2010 at 1:08 AM, Martin Jansa <martin.jansa@gmail.com> wrote: > On Sun, Oct 24, 2010 at 9:40 AM, Khem Raj <raj.khem@gmail.com> wrote: >> On Sat, Oct 23, 2010 at 11:58 PM, Martin Jansa <martin.jansa@gmail.com> wrote: >>> This commit is usefull for sure, but breaks ie some EFL recipes. >>> >>> ie python-elementary depends on elementary >= 0.7.0.52890 >>> but elementary.pc says Version: 0.7.0.0 >>> >>> elementary.pc.in says Version: @VERSION@ >>> >>> and finally VERSION is defined in configure.ac: >>> m4_define([v_maj], [0]) >>> m4_define([v_min], [7]) >>> m4_define([v_mic], [0]) >>> m4_define([v_rev], m4_esyscmd([(svnversion "${SVN_REPO_PATH:-.}" | >>> grep -v export || echo 0) | awk -F : '{printf("%s\n", $1);}' | tr -d ' >>> :MSP\n'])) >>> m4_if(v_rev, [0], [m4_define([v_rev], m4_esyscmd([git log 2> /dev/null >>> | (grep -m1 git-svn-id || echo 0) | sed -e 's/.*@\([0-9]*\).*/\1/' | >>> tr -d '\n']))]) >>> >> >> hmm strange that a configure scripts depends on SCM files. To me it >> seems wrong. Anyway I think you can patch configure.ac >> to replace v_rev definition to be based on SRCREV may be some sort of >> sed operation as do_configure_prepend might be a good way > > this is used in all efl recipes and I've noticed it in elementary only > because python-elementary depends on such new version. Do you have a list of recipes which use this ? I am sure with git it will be hard to use such a thing as the git revs are random and simlarily many other SCMs > > I've also found similar construct in other projects.. so maybe it's > not so efl related (and then harder to found&fix in all recipes). > I still think packages should not depend upon SCM revisions and we should fix them > Regards, > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files 2010-10-24 18:35 ` Khem Raj @ 2010-10-24 19:56 ` Holger Freyther 2010-10-24 20:34 ` Chris Larson 2010-10-25 0:08 ` Khem Raj 0 siblings, 2 replies; 8+ messages in thread From: Holger Freyther @ 2010-10-24 19:56 UTC (permalink / raw) To: bitbake-dev, openembedded-devel On 10/24/2010 08:35 PM, Khem Raj wrote: >> >> this is used in all efl recipes and I've noticed it in elementary only >> because python-elementary depends on such new version. > > Do you have a list of recipes which use this ? I am sure with git it > will be hard to use such a thing > as the git revs are random and simlarily many other SCMs Speaking as upstream co-author of OpenBSC. We are using something like this to get the revision from the last tag of the repository. If someone is doing a make dist(check) we will copy a special version file into the tar.gz and use that instead. From an upstream point of view I think it is acceptable to say that the ones that build stuff from a SCM (in contrast to a tarball) should have the files of the SCM around... PS: I liked the --exclude patch a lot too... ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files 2010-10-24 19:56 ` Holger Freyther @ 2010-10-24 20:34 ` Chris Larson 2010-10-25 18:05 ` Khem Raj 2010-10-25 0:08 ` Khem Raj 1 sibling, 1 reply; 8+ messages in thread From: Chris Larson @ 2010-10-24 20:34 UTC (permalink / raw) To: openembedded-devel; +Cc: bitbake-dev On Sun, Oct 24, 2010 at 12:56 PM, Holger Freyther <holger+oe@freyther.de<holger%2Boe@freyther.de> > wrote: > On 10/24/2010 08:35 PM, Khem Raj wrote: > > >> > >> this is used in all efl recipes and I've noticed it in elementary only > >> because python-elementary depends on such new version. > > > > Do you have a list of recipes which use this ? I am sure with git it > > will be hard to use such a thing > > as the git revs are random and simlarily many other SCMs > > > Speaking as upstream co-author of OpenBSC. We are using something like this > to > get the revision from the last tag of the repository. If someone is doing a > make dist(check) we will copy a special version file into the tar.gz and > use > that instead. > > From an upstream point of view I think it is acceptable to say that the > ones > that build stuff from a SCM (in contrast to a tarball) should have the > files > of the SCM around... > > PS: I liked the --exclude patch a lot too... I don't think its necessarily the fact that we're building from an SCM that's the differentiating factor here, it's the build from a branch as opposed to a tag. The tagged sources in an scm may be (though aren't always) exactly the same as the tarball contents, in which case the scm files shouldn't be necessary. Ideally, the buildsystems would work with or without them, but I agree, I guess we may have to revert the patch :( -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files 2010-10-24 20:34 ` Chris Larson @ 2010-10-25 18:05 ` Khem Raj 0 siblings, 0 replies; 8+ messages in thread From: Khem Raj @ 2010-10-25 18:05 UTC (permalink / raw) To: Chris Larson; +Cc: bitbake-dev, openembedded-devel On 10/24/10, Chris Larson <clarson@kergoth.com> wrote: > On Sun, Oct 24, 2010 at 12:56 PM, Holger Freyther > <holger+oe@freyther.de<holger%2Boe@freyther.de> >> wrote: > >> On 10/24/2010 08:35 PM, Khem Raj wrote: >> >> >> >> >> this is used in all efl recipes and I've noticed it in elementary only >> >> because python-elementary depends on such new version. >> > >> > Do you have a list of recipes which use this ? I am sure with git it >> > will be hard to use such a thing >> > as the git revs are random and simlarily many other SCMs >> >> >> Speaking as upstream co-author of OpenBSC. We are using something like >> this >> to >> get the revision from the last tag of the repository. If someone is doing >> a >> make dist(check) we will copy a special version file into the tar.gz and >> use >> that instead. >> >> From an upstream point of view I think it is acceptable to say that the >> ones >> that build stuff from a SCM (in contrast to a tarball) should have the >> files >> of the SCM around... >> >> PS: I liked the --exclude patch a lot too... > > > I don't think its necessarily the fact that we're building from an SCM > that's the differentiating factor here, it's the build from a branch as > opposed to a tag. The tagged sources in an scm may be (though aren't > always) exactly the same as the tarball contents, in which case the scm > files shouldn't be necessary. Ideally, the buildsystems would work with or > without them, but I agree, I guess we may have to revert the patch :( another option is a to add a qualifier to bitbake fetcher to tell it that we want to keep SCM metadata for a given fetch and keep the default to exclude it but enable it for such cases. > -- > Christopher Larson > clarson at kergoth dot com > Founder - BitBake, OpenEmbedded, OpenZaurus > Maintainer - Tslib > Senior Software Engineer, Mentor Graphics > -- -Khem ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files 2010-10-24 19:56 ` Holger Freyther 2010-10-24 20:34 ` Chris Larson @ 2010-10-25 0:08 ` Khem Raj 1 sibling, 0 replies; 8+ messages in thread From: Khem Raj @ 2010-10-25 0:08 UTC (permalink / raw) To: openembedded-devel; +Cc: bitbake-dev On Sun, Oct 24, 2010 at 12:56 PM, Holger Freyther <holger+oe@freyther.de> wrote: > On 10/24/2010 08:35 PM, Khem Raj wrote: > >>> >>> this is used in all efl recipes and I've noticed it in elementary only >>> because python-elementary depends on such new version. >> >> Do you have a list of recipes which use this ? I am sure with git it >> will be hard to use such a thing >> as the git revs are random and simlarily many other SCMs > > > Speaking as upstream co-author of OpenBSC. We are using something like this to > get the revision from the last tag of the repository. If someone is doing a > make dist(check) we will copy a special version file into the tar.gz and use > that instead. > > From an upstream point of view I think it is acceptable to say that the ones > that build stuff from a SCM (in contrast to a tarball) should have the files > of the SCM around... well what if some used svn export <rev> to generate a tarball so it will break for him too. so distributors should package the SCM metadata and the user needs to have SCM tools to get the configure through. I would have added a script to update a constant var in src which would denote this incremental version like what gcc does with dates. > > PS: I liked the --exclude patch a lot too... > > _______________________________________________ > Openembedded-devel mailing list > Openembedded-devel@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel > ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2010-10-25 18:06 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1285368419-12062-1-git-send-email-raj.khem@gmail.com>
2010-10-24 6:58 ` [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files Martin Jansa
2010-10-24 7:40 ` Khem Raj
2010-10-24 8:08 ` Martin Jansa
2010-10-24 18:35 ` Khem Raj
2010-10-24 19:56 ` Holger Freyther
2010-10-24 20:34 ` Chris Larson
2010-10-25 18:05 ` Khem Raj
2010-10-25 0:08 ` Khem Raj
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.