Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files

Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files

[Martin Jansa]

This commit is usefull for sure, but breaks ie some EFL recipes.

ie python-elementary depends on elementary >= 0.7.0.52890
but elementary.pc says Version: 0.7.0.0

elementary.pc.in says Version: @VERSION@

and finally VERSION is defined in configure.ac:
m4_define([v_maj], [0])
m4_define([v_min], [7])
m4_define([v_mic], [0])
m4_define([v_rev], m4_esyscmd([(svnversion "${SVN_REPO_PATH:-.}" |
grep -v export || echo 0) | awk -F : '{printf("%s\n", $1);}' | tr -d '
:MSP\n']))
m4_if(v_rev, [0], [m4_define([v_rev], m4_esyscmd([git log 2> /dev/null
| (grep -m1 git-svn-id || echo 0) | sed -e 's/.*@\([0-9]*\).*/\1/' |
tr -d '\n']))])

and this works OK on checkout with .svn dirs included, but results to
0 after this patch.

Any hints how to resolve it?

Regards,

On Sat, Sep 25, 2010 at 12:46 AM, Khem Raj <raj.khem@gmail.com> wrote:
> This option will exclude the SCM metadata from tar files.
>
> Tested with gcc where svn tar which used to be 156M for gcc 4.5
> is now 77M
>
> Signed-off-by: Khem Raj <raj.khem@gmail.com>
> ---
>  lib/bb/fetch/bzr.py |    2 +-
>  lib/bb/fetch/cvs.py |    2 +-
>  lib/bb/fetch/git.py |    2 +-
>  lib/bb/fetch/hg.py  |    2 +-
>  lib/bb/fetch/svn.py |    2 +-
>  5 files changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/lib/bb/fetch/bzr.py b/lib/bb/fetch/bzr.py
> index 8b0bd9f..3ca125f 100644
> --- a/lib/bb/fetch/bzr.py
> +++ b/lib/bb/fetch/bzr.py
> @@ -107,7 +107,7 @@ class Bzr(Fetch):
>         os.chdir(ud.pkgdir)
>         # tar them up to a defined filename
>         try:
> -            runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.basename(ud.pkgdir)), d)
> +            runfetchcmd("tar --exclude '.bzr' --exclude '.bzrtags' -czf %s %s" % (ud.localpath, os.path.basename(ud.pkgdir)), d)
>         except:
>             t, v, tb = sys.exc_info()
>             try:
> diff --git a/lib/bb/fetch/cvs.py b/lib/bb/fetch/cvs.py
> index 1064b09..42d71ba 100644
> --- a/lib/bb/fetch/cvs.py
> +++ b/lib/bb/fetch/cvs.py
> @@ -162,7 +162,7 @@ class Cvs(Fetch):
>         # tar them up to a defined filename
>         if 'fullpath' in ud.parm:
>             os.chdir(pkgdir)
> -            myret = os.system("tar -czf %s %s" % (ud.localpath, localdir))
> +            myret = os.system("tar --exclude 'CVS' -czf %s %s" % (ud.localpath, localdir))
>         else:
>             os.chdir(moddir)
>             os.chdir('..')
> diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py
> index 1b1bc95..4acaa87 100644
> --- a/lib/bb/fetch/git.py
> +++ b/lib/bb/fetch/git.py
> @@ -149,7 +149,7 @@ class Git(Fetch):
>
>         os.chdir(codir)
>         logger.info("Creating tarball of git checkout")
> -        runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d)
> +        runfetchcmd("tar --exclude '.git' -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d)
>
>         os.chdir(ud.clonedir)
>         bb.utils.prunedir(codir)
> diff --git a/lib/bb/fetch/hg.py b/lib/bb/fetch/hg.py
> index ab00d43..9c11deb 100644
> --- a/lib/bb/fetch/hg.py
> +++ b/lib/bb/fetch/hg.py
> @@ -145,7 +145,7 @@ class Hg(Fetch):
>
>         os.chdir(ud.pkgdir)
>         try:
> -            runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d)
> +            runfetchcmd("tar --exclude '.hg' --exclude '.hgrags' -czf %s %s" % (ud.localpath, ud.module), d)
>         except:
>             t, v, tb = sys.exc_info()
>             try:
> diff --git a/lib/bb/fetch/svn.py b/lib/bb/fetch/svn.py
> index 34f8132..c236e41 100644
> --- a/lib/bb/fetch/svn.py
> +++ b/lib/bb/fetch/svn.py
> @@ -159,7 +159,7 @@ class Svn(Fetch):
>         os.chdir(ud.pkgdir)
>         # tar them up to a defined filename
>         try:
> -            runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d)
> +            runfetchcmd("tar --exclude '.svn' -czf %s %s" % (ud.localpath, ud.module), d)
>         except:
>             t, v, tb = sys.exc_info()
>             try:
> --
> 1.7.1
>
> _______________________________________________
> Bitbake-dev mailing list
> Bitbake-dev@lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/bitbake-dev
>

Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files

[Khem Raj]

On Sat, Oct 23, 2010 at 11:58 PM, Martin Jansa <martin.jansa@gmail.com> wrote:
> This commit is usefull for sure, but breaks ie some EFL recipes.
>
> ie python-elementary depends on elementary >= 0.7.0.52890
> but elementary.pc says Version: 0.7.0.0
>
> elementary.pc.in says Version: @VERSION@
>
> and finally VERSION is defined in configure.ac:
> m4_define([v_maj], [0])
> m4_define([v_min], [7])
> m4_define([v_mic], [0])
> m4_define([v_rev], m4_esyscmd([(svnversion "${SVN_REPO_PATH:-.}" |
> grep -v export || echo 0) | awk -F : '{printf("%s\n", $1);}' | tr -d '
> :MSP\n']))
> m4_if(v_rev, [0], [m4_define([v_rev], m4_esyscmd([git log 2> /dev/null
> | (grep -m1 git-svn-id || echo 0) | sed -e 's/.*@\([0-9]*\).*/\1/' |
> tr -d '\n']))])
>

hmm strange that a configure scripts depends on SCM files. To me it
seems wrong. Anyway I think you can patch configure.ac
to replace v_rev definition to be based on SRCREV may be some sort of
sed operation as do_configure_prepend might be a good way



> and this works OK on checkout with .svn dirs included, but results to
> 0 after this patch.
>
> Any hints how to resolve it?
>
> Regards,
>
> On Sat, Sep 25, 2010 at 12:46 AM, Khem Raj <raj.khem@gmail.com> wrote:
>> This option will exclude the SCM metadata from tar files.
>>
>> Tested with gcc where svn tar which used to be 156M for gcc 4.5
>> is now 77M
>>
>> Signed-off-by: Khem Raj <raj.khem@gmail.com>
>> ---
>>  lib/bb/fetch/bzr.py |    2 +-
>>  lib/bb/fetch/cvs.py |    2 +-
>>  lib/bb/fetch/git.py |    2 +-
>>  lib/bb/fetch/hg.py  |    2 +-
>>  lib/bb/fetch/svn.py |    2 +-
>>  5 files changed, 5 insertions(+), 5 deletions(-)
>>
>> diff --git a/lib/bb/fetch/bzr.py b/lib/bb/fetch/bzr.py
>> index 8b0bd9f..3ca125f 100644
>> --- a/lib/bb/fetch/bzr.py
>> +++ b/lib/bb/fetch/bzr.py
>> @@ -107,7 +107,7 @@ class Bzr(Fetch):
>>         os.chdir(ud.pkgdir)
>>         # tar them up to a defined filename
>>         try:
>> -            runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.basename(ud.pkgdir)), d)
>> +            runfetchcmd("tar --exclude '.bzr' --exclude '.bzrtags' -czf %s %s" % (ud.localpath, os.path.basename(ud.pkgdir)), d)
>>         except:
>>             t, v, tb = sys.exc_info()
>>             try:
>> diff --git a/lib/bb/fetch/cvs.py b/lib/bb/fetch/cvs.py
>> index 1064b09..42d71ba 100644
>> --- a/lib/bb/fetch/cvs.py
>> +++ b/lib/bb/fetch/cvs.py
>> @@ -162,7 +162,7 @@ class Cvs(Fetch):
>>         # tar them up to a defined filename
>>         if 'fullpath' in ud.parm:
>>             os.chdir(pkgdir)
>> -            myret = os.system("tar -czf %s %s" % (ud.localpath, localdir))
>> +            myret = os.system("tar --exclude 'CVS' -czf %s %s" % (ud.localpath, localdir))
>>         else:
>>             os.chdir(moddir)
>>             os.chdir('..')
>> diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py
>> index 1b1bc95..4acaa87 100644
>> --- a/lib/bb/fetch/git.py
>> +++ b/lib/bb/fetch/git.py
>> @@ -149,7 +149,7 @@ class Git(Fetch):
>>
>>         os.chdir(codir)
>>         logger.info("Creating tarball of git checkout")
>> -        runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d)
>> +        runfetchcmd("tar --exclude '.git' -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d)
>>
>>         os.chdir(ud.clonedir)
>>         bb.utils.prunedir(codir)
>> diff --git a/lib/bb/fetch/hg.py b/lib/bb/fetch/hg.py
>> index ab00d43..9c11deb 100644
>> --- a/lib/bb/fetch/hg.py
>> +++ b/lib/bb/fetch/hg.py
>> @@ -145,7 +145,7 @@ class Hg(Fetch):
>>
>>         os.chdir(ud.pkgdir)
>>         try:
>> -            runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d)
>> +            runfetchcmd("tar --exclude '.hg' --exclude '.hgrags' -czf %s %s" % (ud.localpath, ud.module), d)
>>         except:
>>             t, v, tb = sys.exc_info()
>>             try:
>> diff --git a/lib/bb/fetch/svn.py b/lib/bb/fetch/svn.py
>> index 34f8132..c236e41 100644
>> --- a/lib/bb/fetch/svn.py
>> +++ b/lib/bb/fetch/svn.py
>> @@ -159,7 +159,7 @@ class Svn(Fetch):
>>         os.chdir(ud.pkgdir)
>>         # tar them up to a defined filename
>>         try:
>> -            runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d)
>> +            runfetchcmd("tar --exclude '.svn' -czf %s %s" % (ud.localpath, ud.module), d)
>>         except:
>>             t, v, tb = sys.exc_info()
>>             try:
>> --
>> 1.7.1
>>
>> _______________________________________________
>> Bitbake-dev mailing list
>> Bitbake-dev@lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/bitbake-dev
>>
>

Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files

[Martin Jansa]

On Sun, Oct 24, 2010 at 9:40 AM, Khem Raj <raj.khem@gmail.com> wrote:
> On Sat, Oct 23, 2010 at 11:58 PM, Martin Jansa <martin.jansa@gmail.com> wrote:
>> This commit is usefull for sure, but breaks ie some EFL recipes.
>>
>> ie python-elementary depends on elementary >= 0.7.0.52890
>> but elementary.pc says Version: 0.7.0.0
>>
>> elementary.pc.in says Version: @VERSION@
>>
>> and finally VERSION is defined in configure.ac:
>> m4_define([v_maj], [0])
>> m4_define([v_min], [7])
>> m4_define([v_mic], [0])
>> m4_define([v_rev], m4_esyscmd([(svnversion "${SVN_REPO_PATH:-.}" |
>> grep -v export || echo 0) | awk -F : '{printf("%s\n", $1);}' | tr -d '
>> :MSP\n']))
>> m4_if(v_rev, [0], [m4_define([v_rev], m4_esyscmd([git log 2> /dev/null
>> | (grep -m1 git-svn-id || echo 0) | sed -e 's/.*@\([0-9]*\).*/\1/' |
>> tr -d '\n']))])
>>
>
> hmm strange that a configure scripts depends on SCM files. To me it
> seems wrong. Anyway I think you can patch configure.ac
> to replace v_rev definition to be based on SRCREV may be some sort of
> sed operation as do_configure_prepend might be a good way

this is used in all efl recipes and I've noticed it in elementary only
because python-elementary depends on such new version.

I've also found similar construct in other projects.. so maybe it's
not so efl related (and then harder to found&fix in all recipes).

Regards,

Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files

[Khem Raj]

On Sun, Oct 24, 2010 at 1:08 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
> On Sun, Oct 24, 2010 at 9:40 AM, Khem Raj <raj.khem@gmail.com> wrote:
>> On Sat, Oct 23, 2010 at 11:58 PM, Martin Jansa <martin.jansa@gmail.com> wrote:
>>> This commit is usefull for sure, but breaks ie some EFL recipes.
>>>
>>> ie python-elementary depends on elementary >= 0.7.0.52890
>>> but elementary.pc says Version: 0.7.0.0
>>>
>>> elementary.pc.in says Version: @VERSION@
>>>
>>> and finally VERSION is defined in configure.ac:
>>> m4_define([v_maj], [0])
>>> m4_define([v_min], [7])
>>> m4_define([v_mic], [0])
>>> m4_define([v_rev], m4_esyscmd([(svnversion "${SVN_REPO_PATH:-.}" |
>>> grep -v export || echo 0) | awk -F : '{printf("%s\n", $1);}' | tr -d '
>>> :MSP\n']))
>>> m4_if(v_rev, [0], [m4_define([v_rev], m4_esyscmd([git log 2> /dev/null
>>> | (grep -m1 git-svn-id || echo 0) | sed -e 's/.*@\([0-9]*\).*/\1/' |
>>> tr -d '\n']))])
>>>
>>
>> hmm strange that a configure scripts depends on SCM files. To me it
>> seems wrong. Anyway I think you can patch configure.ac
>> to replace v_rev definition to be based on SRCREV may be some sort of
>> sed operation as do_configure_prepend might be a good way
>
> this is used in all efl recipes and I've noticed it in elementary only
> because python-elementary depends on such new version.

Do you have a list of recipes which use this ? I am sure with git it
will be hard to use such a thing
as the git revs are random and simlarily many other SCMs

>
> I've also found similar construct in other projects.. so maybe it's
> not so efl related (and then harder to found&fix in all recipes).
>

I still think packages should not depend upon SCM revisions and we
should fix them

> Regards,
>

Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files

[Holger Freyther]

On 10/24/2010 08:35 PM, Khem Raj wrote:

>>
>> this is used in all efl recipes and I've noticed it in elementary only
>> because python-elementary depends on such new version.
> 
> Do you have a list of recipes which use this ? I am sure with git it
> will be hard to use such a thing
> as the git revs are random and simlarily many other SCMs


Speaking as upstream co-author of OpenBSC. We are using something like this to
get the revision from the last tag of the repository. If someone is doing a
make dist(check) we will copy a special version file into the tar.gz and use
that instead.

From an upstream point of view I think it is acceptable to say that the ones
that build stuff from a SCM (in contrast to a tarball) should have the files
of the SCM around...

PS: I liked the --exclude patch a lot too...

Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files

[Chris Larson]

On Sun, Oct 24, 2010 at 12:56 PM, Holger Freyther
<holger+oe@freyther.de<holger%2Boe@freyther.de>
> wrote:

> On 10/24/2010 08:35 PM, Khem Raj wrote:
>
> >>
> >> this is used in all efl recipes and I've noticed it in elementary only
> >> because python-elementary depends on such new version.
> >
> > Do you have a list of recipes which use this ? I am sure with git it
> > will be hard to use such a thing
> > as the git revs are random and simlarily many other SCMs
>
>
> Speaking as upstream co-author of OpenBSC. We are using something like this
> to
> get the revision from the last tag of the repository. If someone is doing a
> make dist(check) we will copy a special version file into the tar.gz and
> use
> that instead.
>
> From an upstream point of view I think it is acceptable to say that the
> ones
> that build stuff from a SCM (in contrast to a tarball) should have the
> files
> of the SCM around...
>
> PS: I liked the --exclude patch a lot too...


I don't think its necessarily the fact that we're building from an SCM
that's the differentiating factor here, it's the build from a branch as
opposed to a tag.  The tagged sources in an scm may be (though aren't
always) exactly the same as the tarball contents, in which case the scm
files shouldn't be necessary.  Ideally, the buildsystems would work with or
without them, but I agree, I guess we may have to revert the patch :(
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files

[Khem Raj]

On Sun, Oct 24, 2010 at 12:56 PM, Holger Freyther <holger+oe@freyther.de> wrote:
> On 10/24/2010 08:35 PM, Khem Raj wrote:
>
>>>
>>> this is used in all efl recipes and I've noticed it in elementary only
>>> because python-elementary depends on such new version.
>>
>> Do you have a list of recipes which use this ? I am sure with git it
>> will be hard to use such a thing
>> as the git revs are random and simlarily many other SCMs
>
>
> Speaking as upstream co-author of OpenBSC. We are using something like this to
> get the revision from the last tag of the repository. If someone is doing a
> make dist(check) we will copy a special version file into the tar.gz and use
> that instead.
>
> From an upstream point of view I think it is acceptable to say that the ones
> that build stuff from a SCM (in contrast to a tarball) should have the files
> of the SCM around...


well what if some used svn export <rev> to generate a tarball so it
will break for him too.
so distributors should package the SCM metadata and the user needs to
have SCM tools to
get the configure through. I would have added a script to update a
constant var in src
which would denote this incremental version like what gcc does with dates.
>
> PS: I liked the --exclude patch a lot too...
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: [Bitbake-dev] [PATCH] fetchers: Use tar --exclude pattern to remove SCM files

[Khem Raj]

On 10/24/10, Chris Larson <clarson@kergoth.com> wrote:
> On Sun, Oct 24, 2010 at 12:56 PM, Holger Freyther
> <holger+oe@freyther.de<holger%2Boe@freyther.de>
>> wrote:
>
>> On 10/24/2010 08:35 PM, Khem Raj wrote:
>>
>> >>
>> >> this is used in all efl recipes and I've noticed it in elementary only
>> >> because python-elementary depends on such new version.
>> >
>> > Do you have a list of recipes which use this ? I am sure with git it
>> > will be hard to use such a thing
>> > as the git revs are random and simlarily many other SCMs
>>
>>
>> Speaking as upstream co-author of OpenBSC. We are using something like
>> this
>> to
>> get the revision from the last tag of the repository. If someone is doing
>> a
>> make dist(check) we will copy a special version file into the tar.gz and
>> use
>> that instead.
>>
>> From an upstream point of view I think it is acceptable to say that the
>> ones
>> that build stuff from a SCM (in contrast to a tarball) should have the
>> files
>> of the SCM around...
>>
>> PS: I liked the --exclude patch a lot too...
>
>
> I don't think its necessarily the fact that we're building from an SCM
> that's the differentiating factor here, it's the build from a branch as
> opposed to a tag.  The tagged sources in an scm may be (though aren't
> always) exactly the same as the tarball contents, in which case the scm
> files shouldn't be necessary.  Ideally, the buildsystems would work with or
> without them, but I agree, I guess we may have to revert the patch :(

another option is a to add a qualifier to bitbake fetcher to tell it that
we want to keep SCM metadata for a given fetch and keep the default
to exclude it but enable it for such cases.

> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
>


-- 
-Khem