* [PATCH 0/5] refactor the archive*.bbcalss
@ 2014-01-13 16:20 Robert Yang
2014-01-13 10:21 ` Martin Jansa
` (7 more replies)
0 siblings, 8 replies; 24+ messages in thread
From: Robert Yang @ 2014-01-13 16:20 UTC (permalink / raw)
To: openembedded-core
* The archive*.bbclass didn't work, and there were a few problems, for
example:
1) There were a few duplicated code
2) There was no src_dir.org (or orig), but the diff command still use
it, and it is not easy to fix this issue if we don't change a lot
of the code.
3) It didn't archive the source for the native or gcc
4) The work flow is not very well
5) The "subprocess.call('fakeroot cp xxxx'" should be removed
6) And others ...
* So that we have to refactor it, the benefits are:
1) Fix the problems and make it work well.
2) Reduce more than 400 lines in total.
3) Make it easy to use.
// Robert
The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674:
bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib rbt/archiver
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver
Robert Yang (5):
classes/archive*.bbclass: remove archive-*-source.bbclass
archiver.bbclass: refactor it
package_rpm.bbclass: archive the source to srpm package
archiver.bbclass: move a few code to copyleft_compliance.bbclass
local.conf.sample.extended: update for the archiver
meta-yocto/conf/local.conf.sample.extended | 72 +--
meta/classes/archive-configured-source.bbclass | 65 ---
meta/classes/archive-original-source.bbclass | 65 ---
meta/classes/archive-patched-source.bbclass | 65 ---
meta/classes/archiver.bbclass | 723 ++++++++-----------------
meta/classes/copyleft_compliance.bbclass | 55 +-
meta/classes/package_rpm.bbclass | 31 +-
7 files changed, 316 insertions(+), 760 deletions(-)
delete mode 100644 meta/classes/archive-configured-source.bbclass
delete mode 100644 meta/classes/archive-original-source.bbclass
delete mode 100644 meta/classes/archive-patched-source.bbclass
--
1.8.3.1
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-13 16:20 [PATCH 0/5] refactor the archive*.bbcalss Robert Yang @ 2014-01-13 10:21 ` Martin Jansa 2014-01-13 10:47 ` Robert Yang 2014-01-13 16:20 ` [PATCH 1/5] classes/archive*.bbclass: remove archive-*-source.bbclass Robert Yang ` (6 subsequent siblings) 7 siblings, 1 reply; 24+ messages in thread From: Martin Jansa @ 2014-01-13 10:21 UTC (permalink / raw) To: Robert Yang; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 2576 bytes --] On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: > * The archive*.bbclass didn't work, and there were a few problems, for > example: > 1) There were a few duplicated code > 2) There was no src_dir.org (or orig), but the diff command still use > it, and it is not easy to fix this issue if we don't change a lot > of the code. > 3) It didn't archive the source for the native or gcc > 4) The work flow is not very well > 5) The "subprocess.call('fakeroot cp xxxx'" should be removed > 6) And others ... > > * So that we have to refactor it, the benefits are: > 1) Fix the problems and make it work well. > 2) Reduce more than 400 lines in total. > 3) Make it easy to use. Have you seen http://lists.openembedded.org/pipermail/openembedded-core/2013-December/087729.html ? Could you review it/integrate changes from it? > > // Robert > > The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: > > bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) > > are available in the git repository at: > > git://git.pokylinux.org/poky-contrib rbt/archiver > http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver > > Robert Yang (5): > classes/archive*.bbclass: remove archive-*-source.bbclass > archiver.bbclass: refactor it > package_rpm.bbclass: archive the source to srpm package > archiver.bbclass: move a few code to copyleft_compliance.bbclass > local.conf.sample.extended: update for the archiver > > meta-yocto/conf/local.conf.sample.extended | 72 +-- > meta/classes/archive-configured-source.bbclass | 65 --- > meta/classes/archive-original-source.bbclass | 65 --- > meta/classes/archive-patched-source.bbclass | 65 --- > meta/classes/archiver.bbclass | 723 ++++++++----------------- > meta/classes/copyleft_compliance.bbclass | 55 +- > meta/classes/package_rpm.bbclass | 31 +- > 7 files changed, 316 insertions(+), 760 deletions(-) > delete mode 100644 meta/classes/archive-configured-source.bbclass > delete mode 100644 meta/classes/archive-original-source.bbclass > delete mode 100644 meta/classes/archive-patched-source.bbclass > > -- > 1.8.3.1 > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-13 10:21 ` Martin Jansa @ 2014-01-13 10:47 ` Robert Yang 2014-01-13 11:52 ` Henning Heinold 0 siblings, 1 reply; 24+ messages in thread From: Robert Yang @ 2014-01-13 10:47 UTC (permalink / raw) To: Martin Jansa; +Cc: openembedded-core On 01/13/2014 06:21 PM, Martin Jansa wrote: > On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: >> * The archive*.bbclass didn't work, and there were a few problems, for >> example: >> 1) There were a few duplicated code >> 2) There was no src_dir.org (or orig), but the diff command still use >> it, and it is not easy to fix this issue if we don't change a lot >> of the code. >> 3) It didn't archive the source for the native or gcc >> 4) The work flow is not very well >> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed >> 6) And others ... >> >> * So that we have to refactor it, the benefits are: >> 1) Fix the problems and make it work well. >> 2) Reduce more than 400 lines in total. >> 3) Make it easy to use. > > Have you seen > http://lists.openembedded.org/pipermail/openembedded-core/2013-December/087729.html > ? > > Could you review it/integrate changes from it? I've looked at it just now, I think the problems that you mentioned have been fixed during the refactor. It won't install/remove the files manually any more, they will be installed/removed by the sstate. And the usage become easier, just this would be OK by default: INHERIT += "archiver" And we can easily add other functions when needed. // Robert > >> >> // Robert >> >> The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: >> >> bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) >> >> are available in the git repository at: >> >> git://git.pokylinux.org/poky-contrib rbt/archiver >> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver >> >> Robert Yang (5): >> classes/archive*.bbclass: remove archive-*-source.bbclass >> archiver.bbclass: refactor it >> package_rpm.bbclass: archive the source to srpm package >> archiver.bbclass: move a few code to copyleft_compliance.bbclass >> local.conf.sample.extended: update for the archiver >> >> meta-yocto/conf/local.conf.sample.extended | 72 +-- >> meta/classes/archive-configured-source.bbclass | 65 --- >> meta/classes/archive-original-source.bbclass | 65 --- >> meta/classes/archive-patched-source.bbclass | 65 --- >> meta/classes/archiver.bbclass | 723 ++++++++----------------- >> meta/classes/copyleft_compliance.bbclass | 55 +- >> meta/classes/package_rpm.bbclass | 31 +- >> 7 files changed, 316 insertions(+), 760 deletions(-) >> delete mode 100644 meta/classes/archive-configured-source.bbclass >> delete mode 100644 meta/classes/archive-original-source.bbclass >> delete mode 100644 meta/classes/archive-patched-source.bbclass >> >> -- >> 1.8.3.1 >> >> _______________________________________________ >> Openembedded-core mailing list >> Openembedded-core@lists.openembedded.org >> http://lists.openembedded.org/mailman/listinfo/openembedded-core > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-13 10:47 ` Robert Yang @ 2014-01-13 11:52 ` Henning Heinold 2014-01-13 11:58 ` Martin Jansa 0 siblings, 1 reply; 24+ messages in thread From: Henning Heinold @ 2014-01-13 11:52 UTC (permalink / raw) To: openembedded-core On Mon, Jan 13, 2014 at 06:47:39PM +0800, Robert Yang wrote: > > On 01/13/2014 06:21 PM, Martin Jansa wrote: > >On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: > >>* The archive*.bbclass didn't work, and there were a few problems, for > >> example: > >> 1) There were a few duplicated code > >> 2) There was no src_dir.org (or orig), but the diff command still use > >> it, and it is not easy to fix this issue if we don't change a lot > >> of the code. > >> 3) It didn't archive the source for the native or gcc > >> 4) The work flow is not very well > >> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed > >> 6) And others ... > >> > >>* So that we have to refactor it, the benefits are: > >> 1) Fix the problems and make it work well. > >> 2) Reduce more than 400 lines in total. > >> 3) Make it easy to use. > > > >Have you seen > >http://lists.openembedded.org/pipermail/openembedded-core/2013-December/087729.html > >? > > > >Could you review it/integrate changes from it? > > I've looked at it just now, I think the problems that you mentioned have been > fixed during the refactor. It won't install/remove the files manually any more, > they will be installed/removed by the sstate. And the usage become easier, just > this would be OK by default: > > INHERIT += "archiver" > > And we can easily add other functions when needed. > > // Robert > Hi Robert, does the whole thread means it is broken in dora too? Are there plans to backport the patches? Bye Henning ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-13 11:52 ` Henning Heinold @ 2014-01-13 11:58 ` Martin Jansa 2014-01-14 2:13 ` Robert Yang 0 siblings, 1 reply; 24+ messages in thread From: Martin Jansa @ 2014-01-13 11:58 UTC (permalink / raw) To: Henning Heinold; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 1926 bytes --] On Mon, Jan 13, 2014 at 12:52:49PM +0100, Henning Heinold wrote: > On Mon, Jan 13, 2014 at 06:47:39PM +0800, Robert Yang wrote: > > > > On 01/13/2014 06:21 PM, Martin Jansa wrote: > > >On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: > > >>* The archive*.bbclass didn't work, and there were a few problems, for > > >> example: > > >> 1) There were a few duplicated code > > >> 2) There was no src_dir.org (or orig), but the diff command still use > > >> it, and it is not easy to fix this issue if we don't change a lot > > >> of the code. > > >> 3) It didn't archive the source for the native or gcc > > >> 4) The work flow is not very well > > >> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed > > >> 6) And others ... > > >> > > >>* So that we have to refactor it, the benefits are: > > >> 1) Fix the problems and make it work well. > > >> 2) Reduce more than 400 lines in total. > > >> 3) Make it easy to use. > > > > > >Have you seen > > >http://lists.openembedded.org/pipermail/openembedded-core/2013-December/087729.html > > >? > > > > > >Could you review it/integrate changes from it? > > > > I've looked at it just now, I think the problems that you mentioned have been > > fixed during the refactor. It won't install/remove the files manually any more, > > they will be installed/removed by the sstate. And the usage become easier, just > > this would be OK by default: > > > > INHERIT += "archiver" > > > > And we can easily add other functions when needed. > > > > // Robert > > > > Hi Robert, > > does the whole thread means it is broken in dora too? Are there plans to backport the patches? AFAIK it's broken everywhere, I would like to have it backported to dylan as well (after testing that it indeed fixes source files being removed) -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-13 11:58 ` Martin Jansa @ 2014-01-14 2:13 ` Robert Yang 0 siblings, 0 replies; 24+ messages in thread From: Robert Yang @ 2014-01-14 2:13 UTC (permalink / raw) To: Martin Jansa, Henning Heinold; +Cc: openembedded-core On 01/13/2014 07:58 PM, Martin Jansa wrote: > On Mon, Jan 13, 2014 at 12:52:49PM +0100, Henning Heinold wrote: >> On Mon, Jan 13, 2014 at 06:47:39PM +0800, Robert Yang wrote: >>> >>> On 01/13/2014 06:21 PM, Martin Jansa wrote: >>>> On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: >>>>> * The archive*.bbclass didn't work, and there were a few problems, for >>>>> example: >>>>> 1) There were a few duplicated code >>>>> 2) There was no src_dir.org (or orig), but the diff command still use >>>>> it, and it is not easy to fix this issue if we don't change a lot >>>>> of the code. >>>>> 3) It didn't archive the source for the native or gcc >>>>> 4) The work flow is not very well >>>>> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed >>>>> 6) And others ... >>>>> >>>>> * So that we have to refactor it, the benefits are: >>>>> 1) Fix the problems and make it work well. >>>>> 2) Reduce more than 400 lines in total. >>>>> 3) Make it easy to use. >>>> >>>> Have you seen >>>> http://lists.openembedded.org/pipermail/openembedded-core/2013-December/087729.html >>>> ? >>>> >>>> Could you review it/integrate changes from it? >>> >>> I've looked at it just now, I think the problems that you mentioned have been >>> fixed during the refactor. It won't install/remove the files manually any more, >>> they will be installed/removed by the sstate. And the usage become easier, just >>> this would be OK by default: >>> >>> INHERIT += "archiver" >>> >>> And we can easily add other functions when needed. >>> >>> // Robert >>> >> >> Hi Robert, >> >> does the whole thread means it is broken in dora too? Are there plans to backport the patches? > > AFAIK it's broken everywhere, I would like to have it backported to > dylan as well (after testing that it indeed fixes source files being > removed) > Yes, Martin is right, but I'm not sure whether we should backport them since we have change a lot of the code, and there should be bugs, usually, though, I've done as many testing as I can. BTW., I've updated the code a little in the PULL: Use "1" or "0" rather than "yes" or "no", which seems more popular in oe, for example: ARCHIVER_MODE[dumpdata] = "1" (it was "yes" before). // Robert > > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core > ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 1/5] classes/archive*.bbclass: remove archive-*-source.bbclass 2014-01-13 16:20 [PATCH 0/5] refactor the archive*.bbcalss Robert Yang 2014-01-13 10:21 ` Martin Jansa @ 2014-01-13 16:20 ` Robert Yang 2014-01-13 16:20 ` [PATCH 2/5] archiver.bbclass: refactor it Robert Yang ` (5 subsequent siblings) 7 siblings, 0 replies; 24+ messages in thread From: Robert Yang @ 2014-01-13 16:20 UTC (permalink / raw) To: openembedded-core Nearly all of the codes in the following 3 files are the same, we can move the code to archiver.bbclass and remove them: archive-configured-source.bbclass archive-original-source.bbclass archive-patched-source.bbclass [YOCTO #5113] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta/classes/archive-configured-source.bbclass | 65 -------------------------- meta/classes/archive-original-source.bbclass | 65 -------------------------- meta/classes/archive-patched-source.bbclass | 65 -------------------------- meta/classes/archiver.bbclass | 63 ++++++++++++++++++++++++- 4 files changed, 62 insertions(+), 196 deletions(-) delete mode 100644 meta/classes/archive-configured-source.bbclass delete mode 100644 meta/classes/archive-original-source.bbclass delete mode 100644 meta/classes/archive-patched-source.bbclass diff --git a/meta/classes/archive-configured-source.bbclass b/meta/classes/archive-configured-source.bbclass deleted file mode 100644 index 961a532..0000000 --- a/meta/classes/archive-configured-source.bbclass +++ /dev/null @@ -1,65 +0,0 @@ -# This file is for getting archiving packages with configured -# sources(archive ${S} after configure stage), logs(archive 'temp' after -# package_write_rpm), dump data and creating diff file(get all -# environment variables and functions in building and mapping all -# content in ${S} including patches to xxx.diff.gz. All archived -# packages will be deployed in ${DEPLOY_DIR}/sources - -inherit archiver - -# Get archiving package with configured sources including patches -addtask do_archive_configured_sources after do_configure - -# Get archiving package with temp(logs) and scripts(.bb and inc files) -addtask do_archive_scripts_logs - -# Get dump date and create diff file -addtask do_dumpdata_create_diff_gz - -python () { - pn = d.getVar('PN', True) - packaging = d.getVar('IMAGE_PKGTYPE', True) - - if tar_filter(d): - return - - d.appendVarFlag('do_dumpdata_create_diff_gz', 'depends', ' %s:do_package_write_' %pn + packaging) - build_deps = ' %s:do_dumpdata_create_diff_gz' %pn - - if d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) == 'logs_with_scripts': - d.appendVarFlag('do_archive_scripts_logs', 'depends', ' %s:do_package_write_' %pn + packaging) - build_deps += ' %s:do_archive_scripts_logs' %pn - - if not not_tarball(d): - d.appendVarFlag('do_compile', 'depends', ' %s:do_archive_configured_sources' %pn) - build_deps += ' %s:do_archive_configured_sources' %pn - - if bb.data.inherits_class('image', d): - d.appendVarFlag('do_rootfs', 'depends', build_deps) - else: - d.appendVarFlag('do_build', 'depends', build_deps) -} - -ARCHIVE_SSTATE_OUTDIR = "${DEPLOY_DIR}/sources/" -ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR = "${WORKDIR}/script-logs/" -ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR = "${WORKDIR}/diffgz-envdata/" - -SSTATETASKS += "do_archive_scripts_logs" -do_archive_scripts_logs[sstate-inputdirs] = "${ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR}" -do_archive_scripts_logs[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}" - -python do_archive_scripts_logs_setscene () { - sstate_setscene(d) -} - -addtask do_archive_scripts_logs_setscene - -SSTATETASKS += "do_dumpdata_create_diff_gz" -do_dumpdata_create_diff_gz[sstate-inputdirs] = "${ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR}" -do_dumpdata_create_diff_gz[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}" - -python do_dumpdata_create_diff_gz_setscene () { - sstate_setscene(d) -} - -addtask do_dumpdata_create_diff_gz_setscene diff --git a/meta/classes/archive-original-source.bbclass b/meta/classes/archive-original-source.bbclass deleted file mode 100644 index e271a08..0000000 --- a/meta/classes/archive-original-source.bbclass +++ /dev/null @@ -1,65 +0,0 @@ -# This file is for getting archiving packages with original -# sources(archive ${S} after unpack stage), patches, logs(archive 'temp' -# after package_write_rpm), dump data and creating diff file(get all -# environment variables and functions in building and mapping all -# content in ${S} including patches to xxx.diff.gz. All archived packages -# will be deployed in ${DEPLOY_DIR}/sources - -inherit archiver - -# Get original sources archiving package with patches -addtask do_archive_original_sources_patches after do_unpack - -# Get archiving package with temp(logs) and scripts(.bb and inc files) -addtask do_archive_scripts_logs - -# Get dump date and create diff file -addtask do_dumpdata_create_diff_gz - -python () { - pn = d.getVar('PN', True) - packaging = d.getVar('IMAGE_PKGTYPE', True) - - if tar_filter(d): - return - - d.appendVarFlag('do_dumpdata_create_diff_gz', 'depends', ' %s:do_package_write_' %pn + packaging) - build_deps = ' %s:do_dumpdata_create_diff_gz' %pn - - if d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) == 'logs_with_scripts': - d.appendVarFlag('do_archive_scripts_logs', 'depends', ' %s:do_package_write_' %pn + packaging) - build_deps += ' %s:do_archive_scripts_logs' %pn - - if not not_tarball(d): - d.appendVarFlag('do_patch', 'depends', ' %s:do_archive_original_sources_patches' %pn) - build_deps += ' %s:do_archive_original_sources_patches' %pn - - if bb.data.inherits_class('image', d): - d.appendVarFlag('do_rootfs', 'depends', build_deps) - else: - d.appendVarFlag('do_build', 'depends', build_deps) -} - -ARCHIVE_SSTATE_OUTDIR = "${DEPLOY_DIR}/sources/" -ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR = "${WORKDIR}/script-logs/" -ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR = "${WORKDIR}/diffgz-envdata/" - -SSTATETASKS += "do_archive_scripts_logs" -do_archive_scripts_logs[sstate-inputdirs] = "${ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR}" -do_archive_scripts_logs[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}" - -python do_archive_scripts_logs_setscene () { - sstate_setscene(d) -} - -addtask do_archive_scripts_logs_setscene - -SSTATETASKS += "do_dumpdata_create_diff_gz" -do_dumpdata_create_diff_gz[sstate-inputdirs] = "${ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR}" -do_dumpdata_create_diff_gz[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}" - -python do_dumpdata_create_diff_gz_setscene () { - sstate_setscene(d) -} - -addtask do_dumpdata_create_diff_gz_setscene diff --git a/meta/classes/archive-patched-source.bbclass b/meta/classes/archive-patched-source.bbclass deleted file mode 100644 index c5f5cca..0000000 --- a/meta/classes/archive-patched-source.bbclass +++ /dev/null @@ -1,65 +0,0 @@ -# This file is for getting archiving packages with patched -# sources(archive ${S} before do_patch stage), logs(archive 'temp' after -# package_write_rpm), dump data and creating diff file(get all -# environment variables and functions in building and mapping all -# content in ${S} including patches to xxx.diff.gz. All archived -# packages will be deployed in ${DEPLOY_DIR}/sources - -inherit archiver - -# Get archiving package with patched sources including patches -addtask do_archive_patched_sources after do_patch - -# Get archiving package with logs(temp) and scripts(.bb and .inc files) -addtask do_archive_scripts_logs - -# Get dump date and create diff file -addtask do_dumpdata_create_diff_gz - -python () { - pn = d.getVar('PN', True) - packaging = d.getVar('IMAGE_PKGTYPE', True) - - if tar_filter(d): - return - - d.appendVarFlag('do_dumpdata_create_diff_gz', 'depends', ' %s:do_package_write_' %pn + packaging) - build_deps = ' %s:do_dumpdata_create_diff_gz' %pn - - if d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) == 'logs_with_scripts': - d.appendVarFlag('do_archive_scripts_logs', 'depends', ' %s:do_package_write_' %pn + packaging) - build_deps += ' %s:do_archive_scripts_logs' %pn - - if not not_tarball(d): - d.appendVarFlag('do_configure', 'depends', ' %s:do_archive_patched_sources' %pn) - build_deps += ' %s:do_archive_patched_sources' %pn - - if bb.data.inherits_class('image', d): - d.appendVarFlag('do_rootfs', 'depends', build_deps) - else: - d.appendVarFlag('do_build', 'depends', build_deps) -} - -ARCHIVE_SSTATE_OUTDIR = "${DEPLOY_DIR}/sources/" -ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR = "${WORKDIR}/script-logs/" -ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR = "${WORKDIR}/diffgz-envdata/" - -SSTATETASKS += "do_archive_scripts_logs" -do_archive_scripts_logs[sstate-inputdirs] = "${ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR}" -do_archive_scripts_logs[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}" - -python do_archive_scripts_logs_setscene () { - sstate_setscene(d) -} - -addtask do_archive_scripts_logs_setscene - -SSTATETASKS += "do_dumpdata_create_diff_gz" -do_dumpdata_create_diff_gz[sstate-inputdirs] = "${ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR}" -do_dumpdata_create_diff_gz[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}" - -python do_dumpdata_create_diff_gz_setscene () { - sstate_setscene(d) -} - -addtask do_dumpdata_create_diff_gz_setscene diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass index 66efe7d..43373ae 100644 --- a/meta/classes/archiver.bbclass +++ b/meta/classes/archiver.bbclass @@ -36,6 +36,33 @@ COPYLEFT_AVAILABLE_RECIPE_TYPES = 'target native nativesdk cross crosssdk cross- COPYLEFT_AVAILABLE_RECIPE_TYPES[type] = 'list' COPYLEFT_AVAILABLE_RECIPE_TYPES[doc] = 'Space separated list of available recipe types' +python () { + pn = d.getVar('PN', True) + packaging = d.getVar('IMAGE_PKGTYPE', True) + + if tar_filter(d): + return + + if d.getVar('PACKAGES', True) != '': + d.appendVarFlag('do_dumpdata_create_diff_gz', 'depends', ' %s:do_package_write_%s' % (pn, packaging)) + + build_deps = ' %s:do_dumpdata_create_diff_gz' % pn + + if d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) == 'logs_with_scripts': + d.appendVarFlag('do_archive_scripts_logs', 'depends', ' %s:do_package_write_%s' % (pn, packaging)) + build_deps += ' %s:do_archive_scripts_logs' % pn + + if not not_tarball(d): + archiver_mode = d.getVar('ARCHIVER_MODE') + d.appendVarFlag('do_compile', 'depends', ' %s:do_archive_%s_sources' % (pn, archiver_mode)) + build_deps += ' %s:do_archive_%s_sources' % (pn, archiver_mode) + + if bb.data.inherits_class('image', d): + d.appendVarFlag('do_rootfs', 'depends', build_deps) + else: + d.appendVarFlag('do_build', 'depends', build_deps) +} + def copyleft_recipe_type(d): for recipe_type in oe.data.typed_value('COPYLEFT_AVAILABLE_RECIPE_TYPES', d): if oe.utils.inherits(d, recipe_type): @@ -516,7 +543,7 @@ def create_diff_gz(d): # This function will run when user want to get tarball for sources and # patches after do_unpack -python do_archive_original_sources_patches(){ +python do_archive_original_sources(){ archive_sources_patches(d, 'prepatch') } @@ -567,3 +594,37 @@ python do_delete_tarlist(){ } do_delete_tarlist[deptask] = "do_archive_scripts_logs" do_package_write_rpm[postfuncs] += "do_delete_tarlist " + +# Get archiving package with temp(logs) and scripts(.bb and .inc files) +addtask do_archive_scripts_logs + +# Get dump date and create diff file +addtask do_dumpdata_create_diff_gz + +ARCHIVE_SSTATE_OUTDIR = "${DEPLOY_DIR}/sources/" +ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR = "${WORKDIR}/script-logs/" +ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR = "${WORKDIR}/diffgz-envdata/" + +SSTATETASKS += "do_archive_scripts_logs" +do_archive_scripts_logs[sstate-inputdirs] = "${ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR}" +do_archive_scripts_logs[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}" + +python do_archive_scripts_logs_setscene () { + sstate_setscene(d) +} + +addtask do_archive_scripts_logs_setscene + +SSTATETASKS += "do_dumpdata_create_diff_gz" +do_dumpdata_create_diff_gz[sstate-inputdirs] = "${ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR}" +do_dumpdata_create_diff_gz[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}" + +python do_dumpdata_create_diff_gz_setscene () { + sstate_setscene(d) +} + +addtask do_dumpdata_create_diff_gz_setscene + +addtask do_archive_original_sources after do_unpack +addtask do_archive_patched_sources after do_patch +addtask do_archive_configured_sources after do_configure -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 2/5] archiver.bbclass: refactor it 2014-01-13 16:20 [PATCH 0/5] refactor the archive*.bbcalss Robert Yang 2014-01-13 10:21 ` Martin Jansa 2014-01-13 16:20 ` [PATCH 1/5] classes/archive*.bbclass: remove archive-*-source.bbclass Robert Yang @ 2014-01-13 16:20 ` Robert Yang 2014-01-13 16:20 ` [PATCH 3/5] package_rpm.bbclass: archive the source to srpm package Robert Yang ` (4 subsequent siblings) 7 siblings, 0 replies; 24+ messages in thread From: Robert Yang @ 2014-01-13 16:20 UTC (permalink / raw) To: openembedded-core The archiver didn't work, and there were a few problems, for example: 1) There was no src_dir.org (or orig), but the diff command still use it 2) There were a few duplicated code 3) It didn't archive the source for the native or gcc 4) The work flow is not very well 5) The "subprocess.call('fakeroot cp xxxx'" should be removed 6) And others ... So that we have to refactor it, the benefits are: 1) Fix the problems and make it work well. 2) Reduce more than 300 lines 3) Make it easy to use. Hre are the explanation about the bbclass: The bbclass is used for creating the tarball from: 1) prepatch source: ARCHIVER_MODE[src] = "prepatch" 2) patched source: ARCHIVER_MODE[src] = "patched" 3) configured source: ARCHIVER_MODE[src] = "configured" 4) The patches between do_unpack and do_patch: ARCHIVER_MODE[diff] = "yes" And set the files that will be excluded from diff: ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches" 5) The environment data, similar to 'bitbake -e recipe': ARCHIVER_MODE[dumpdata] = "yes" 6) The recipe (.bb and .inc): ARCHIVER_MODE[recipe] = "yes" 7) The log.* and run.* files under ${T} ARCHIVER_MODE[log] = "yes" All of the above can be packed into a .src.rpm package: (when PACKAGES != "") ARCHIVER_MODE[type] = "srpm", except the seventh, because the logs are not completed until the rpm package has been built: ARCHIVER_MODE[type] = "srpm" [YOCTO #4986] [YOCTO #5113] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta/classes/archiver.bbclass | 709 ++++++++++++------------------------------ 1 file changed, 202 insertions(+), 507 deletions(-) diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass index 43373ae..f2d7184 100644 --- a/meta/classes/archiver.bbclass +++ b/meta/classes/archiver.bbclass @@ -1,21 +1,39 @@ # ex:ts=4:sw=4:sts=4:et # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # -# This file is used for archiving sources, patches, and logs to a -# tarball. It also output building environment to xxx.dump.data and -# create xxx.diff.gz to record all content in ${S} to a diff file. +# This bbclass is used for creating the tarball from: +# 1) prepatch source: ARCHIVER_MODE[src] = "prepatch" +# 2) patched source: ARCHIVER_MODE[src] = "patched" +# 3) configured source: ARCHIVER_MODE[src] = "configured" +# 4) The patches between do_unpack and do_patch: +# ARCHIVER_MODE[diff] = "yes" +# And you can set the one that you'd like to exclude from the diff: +# AR_EXCLUDE ?= "${@d.getVarFlag('ARCHIVER_MODE', 'diff-exclude', True)} +# 5) The environment data, similar to 'bitbake -e recipe': +# ARCHIVER_MODE[dumpdata] = "yes" +# 6) The recipe (.bb and .inc): ARCHIVER_MODE[recipe] = "yes" +# 7) The log.* and run.* files under ${T} +# ARCHIVER_MODE[log] = "yes" # - -ARCHIVE_EXCLUDE_FROM ?= ".pc autom4te.cache" -ARCHIVE_TYPE ?= "tar srpm" -PATCHES_ARCHIVE_WITH_SERIES = 'yes' -SOURCE_ARCHIVE_LOG_WITH_SCRIPTS ?= '${@d.getVarFlag('ARCHIVER_MODE', 'log_type') \ - if d.getVarFlag('ARCHIVER_MODE', 'log_type') != 'none' else 'logs_with_scripts'}' -SOURCE_ARCHIVE_PACKAGE_TYPE ?= '${@d.getVarFlag('ARCHIVER_MODE', 'type') \ - if d.getVarFlag('ARCHIVER_MODE', 'log_type') != 'none' else 'tar'}' -FILTER ?= '${@d.getVarFlag('ARCHIVER_MODE', 'filter') \ - if d.getVarFlag('ARCHIVER_MODE', 'filter')!= 'none' else 'no'}' - +# All of the above can be packed into a .src.rpm package: (when PACKAGES != "") +# ARCHIVER_MODE[type] = "srpm", +# except the seventh, because the logs are not completed until the rpm +# package has been built: ARCHIVER_MODE[type] = "srpm" + +ARCHIVER_MODE[type] ?= "tar" +ARCHIVER_MODE[src] ?= "configured" +ARCHIVER_MODE[diff] ?= "yes" +ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches" +ARCHIVER_MODE[dumpdata] ?= "no" +ARCHIVER_MODE[log] ?= "no" +ARCHIVER_MODE[recipe] ?= "no" + +DEPLOY_DIR_SRC ?= "${DEPLOY_DIR}/sources" +AR_TOPDIR = "${WORKDIR}/deploy-sources" +ARCHIVER_OUTDIR = "${AR_TOPDIR}/${TARGET_SYS}/${PF}" + +# This is a convenience for the shell script to use it +AR_EXCLUDE ?= "${@d.getVarFlag('ARCHIVER_MODE', 'diff-exclude', True)}" COPYLEFT_LICENSE_INCLUDE ?= 'GPL* LGPL*' COPYLEFT_LICENSE_INCLUDE[type] = 'list' @@ -23,7 +41,7 @@ COPYLEFT_LICENSE_INCLUDE[doc] = 'Space separated list of globs which include lic COPYLEFT_LICENSE_EXCLUDE ?= 'CLOSED Proprietary' COPYLEFT_LICENSE_EXCLUDE[type] = 'list' -COPYLEFT_LICENSE_INCLUDE[doc] = 'Space separated list of globs which exclude licenses' +COPYLEFT_LICENSE_EXCLUDE[doc] = 'Space separated list of globs which exclude licenses' COPYLEFT_RECIPE_TYPE ?= '${@copyleft_recipe_type(d)}' COPYLEFT_RECIPE_TYPE[doc] = 'The "type" of the current recipe (e.g. target, native, cross)' @@ -40,22 +58,58 @@ python () { pn = d.getVar('PN', True) packaging = d.getVar('IMAGE_PKGTYPE', True) - if tar_filter(d): - return - - if d.getVar('PACKAGES', True) != '': - d.appendVarFlag('do_dumpdata_create_diff_gz', 'depends', ' %s:do_package_write_%s' % (pn, packaging)) - - build_deps = ' %s:do_dumpdata_create_diff_gz' % pn - - if d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) == 'logs_with_scripts': - d.appendVarFlag('do_archive_scripts_logs', 'depends', ' %s:do_package_write_%s' % (pn, packaging)) - build_deps += ' %s:do_archive_scripts_logs' % pn - - if not not_tarball(d): - archiver_mode = d.getVar('ARCHIVER_MODE') - d.appendVarFlag('do_compile', 'depends', ' %s:do_archive_%s_sources' % (pn, archiver_mode)) - build_deps += ' %s:do_archive_%s_sources' % (pn, archiver_mode) + ar_src = d.getVarFlag('ARCHIVER_MODE', 'src', True) + # We create the tarball for the prepatch source before do_patch + # rather then after do_unpack, this is good for: + # * The recipe like linux-yocto, the source is checked out by + # do_kernel_checkout but not do_unpack. + # * For the dependencies chain, if we use: + # do_unpack[postfuncs] += do_newtask + # it doesn't affact the dependencies chain, but the prefuncs will + # do. + # The similar to other do_ar_patched and do_ar_configured. + if ar_src == 'prepatch': + d.appendVarFlag('do_patch', 'prefuncs', ' do_ar_prepatch') + elif ar_src == 'patched': + d.appendVarFlag('do_configure', 'prefuncs', ' do_ar_patched') + elif ar_src == 'configured': + d.appendVarFlag('do_compile', 'prefuncs', ' do_ar_configured') + else: + bb.fatal("Invalid ARCHIVER_MODE[src]: %s" % ar_src) + + # Check the archive type + ar_type = d.getVarFlag('ARCHIVER_MODE', 'type', True) + if ar_type == 'srpm': + if d.getVar('PACKAGES', True) != '' and packaging == 'rpm': + d.appendVarFlag('do_deploy_archives', 'depends', ' %s:do_package_write_rpm' % pn) + elif ar_type != 'tar': + bb.fatal("Invalid ARCHIVER_MODE[type]: %s" % ar_type) + + + # Check whether create the .diff.gz + create_diff = d.getVarFlag('ARCHIVER_MODE', 'diff', True) + if create_diff == "yes": + # Save the source to source.orig before do_patch + d.appendVarFlag('do_patch', 'prefuncs', ' do_save_orig') + # Create the .diff.gz between the source and source.orig after + # do_patch + d.appendVarFlag('do_patch', 'postfuncs', ' create_diff_gz') + elif create_diff and create_diff != "no": + bb.fatal("Invalid ARCHIVER_MODE[diff]: %s" % create_diff) + + build_deps = ' %s:do_deploy_archives' % pn + + ar_log = d.getVarFlag('ARCHIVER_MODE', 'log', True) + # The do_ar_log should be run as late as possible + if ar_log == "yes": + if d.getVar('PACKAGES', True) != '': + d.appendVarFlag('do_ar_log', 'depends', ' %s:do_package_write_%s' % (pn, packaging)) + build_deps += ' %s:do_ar_log' % pn + else: + d.appendVarFlag('do_ar_log', 'depends', ' %s:do_populate_sysroot' % pn) + build_deps += ' %s:do_ar_log' % pn + elif ar_log and ar_log != 'no': + bb.fatal("Invalid ARCHIVER_MODE[log]: %s" % create_diff) if bb.data.inherits_class('image', d): d.appendVarFlag('do_rootfs', 'depends', build_deps) @@ -97,398 +151,100 @@ def copyleft_should_include(d): else: return False, 'recipe has excluded licenses: %s' % ', '.join(reason) -def tar_filter(d): - """ - Only archive the package belongs to COPYLEFT_LICENSE_INCLUDE - and ignore the one in COPYLEFT_LICENSE_EXCLUDE. Don't exclude any - packages when \"FILTER\" is \"no\" - """ - if d.getVar('FILTER', True) == "yes": - included, reason = copyleft_should_include(d) - return not included - else: - return False - -def get_bb_inc(d): - """ - create a directory "script-logs" including .bb and .inc file in ${WORKDIR} - """ - import re - import shutil - - bbinc = [] - pat=re.compile('require\s*([^\s]*\.*)(.*)') - work_dir = d.getVar('WORKDIR', True) - bbfile = d.getVar('FILE', True) - bbdir = os.path.dirname(bbfile) - target_sys = d.getVar('TARGET_SYS', True) - pf = d.getVar('PF', True) - licenses = get_licenses(d) - script_logs = os.path.join(work_dir, 'script-logs/'+ target_sys + '/' + licenses + '/' + pf + '/script-logs') - bb_inc = os.path.join(script_logs, 'bb_inc') - bb.utils.mkdirhier(bb_inc) - - def find_file(dir, file): - for root, dirs, files in os.walk(dir): - if file in files: - return os.path.join(root, file) - - def get_inc (file): - f = open(file, 'r') - for line in f.readlines(): - if 'require' not in line: - bbinc.append(file) - else: - try: - incfile = pat.match(line).group(1) - incfile = bb.data.expand(os.path.basename(incfile), d) - abs_incfile = find_file(bbdir, incfile) - if abs_incfile: - bbinc.append(abs_incfile) - get_inc(abs_incfile) - except AttributeError: - pass - get_inc(bbfile) - bbinc = list(set(bbinc)) - for bbincfile in bbinc: - shutil.copy(bbincfile, bb_inc) - - return script_logs - -def get_logs(d): - """ - create a directory "script-logs" in ${WORKDIR} - """ - work_dir = d.getVar('WORKDIR', True) - target_sys = d.getVar('TARGET_SYS', True) - pf = d.getVar('PF', True) - licenses = get_licenses(d) - script_logs = os.path.join(work_dir, 'script-logs/'+ target_sys + '/' + licenses + '/' + pf + '/script-logs') - - try: - bb.utils.mkdirhier(os.path.join(script_logs, 'temp')) - oe.path.copytree(os.path.join(work_dir, 'temp'), os.path.join(script_logs, 'temp')) - except (IOError, AttributeError): - pass - return script_logs - -def get_series(d): - """ - copy patches and series file to a pointed directory which will be - archived to tarball in ${WORKDIR} - """ - import shutil - - src_patches=[] - pf = d.getVar('PF', True) - work_dir = d.getVar('WORKDIR', True) - s = d.getVar('S', True) - dest = os.path.join(work_dir, pf + '-series') - shutil.rmtree(dest, ignore_errors=True) - bb.utils.mkdirhier(dest) - - src_uri = d.getVar('SRC_URI', True).split() - fetch = bb.fetch2.Fetch(src_uri, d) - locals = (fetch.localpath(url) for url in fetch.urls) - for local in locals: - src_patches.append(local) - if not cmp(work_dir, s): - tmp_list = src_patches - else: - tmp_list = src_patches[1:] - - for patch in tmp_list: - try: - shutil.copy(patch, dest) - except IOError: - if os.path.isdir(patch): - bb.utils.mkdirhier(os.path.join(dest, patch)) - oe.path.copytree(patch, os.path.join(dest, patch)) - return dest - -def get_applying_patches(d): +def create_tarball(d, srcdir, suffix): """ - only copy applying patches to a pointed directory which will be - archived to tarball + create the tarball from srcdir """ - import shutil - - pf = d.getVar('PF', True) - work_dir = d.getVar('WORKDIR', True) - dest = os.path.join(work_dir, pf + '-patches') - shutil.rmtree(dest, ignore_errors=True) - bb.utils.mkdirhier(dest) + import tarfile - patches = src_patches(d) - for patch in patches: - _, _, local, _, _, parm = bb.fetch.decodeurl(patch) - if local: - shutil.copy(local, dest) - return dest -def not_tarball(d): - """ - packages including key words 'work-shared', 'native', 'packagegroup-' will be passed - """ - workdir = d.getVar('WORKDIR', True) - s = d.getVar('S', True) - if 'work-shared' in s or 'packagegroup-' in workdir or 'native' in workdir: - return True - else: - return False + outdir = d.getVar('ARCHIVER_OUTDIR', True) + bb.utils.mkdirhier(outdir) + tarname = os.path.join(outdir, "%s-%s.tar.gz" % \ + (d.getVar("PF", True), suffix)) -def get_source_from_downloads(d, stage_name): - """ - copy tarball of $P to $WORKDIR when this tarball exists in $DL_DIR - """ - if stage_name in 'patched' 'configured': - return - pf = d.getVar('PF', True) - dl_dir = d.getVar('DL_DIR', True) - try: - source = os.path.join(dl_dir, os.path.basename(d.getVar('SRC_URI', True).split()[0])) - if os.path.exists(source) and not os.path.isdir(source): - return source - except (IndexError, OSError): - pass - return '' - -def do_tarball(workdir, srcdir, tarname): - """ - tar "srcdir" under "workdir" to "tarname" - """ - import tarfile - - sav_dir = os.getcwd() - os.chdir(workdir) - if (len(os.listdir(srcdir))) != 0: + srcdir = srcdir.rstrip('/') + dirname = os.path.dirname(srcdir) + basename = os.path.basename(srcdir) + olddir = os.getcwd() + os.chdir(dirname) + bb.note("Creating %s" % tarname) + if (len(os.listdir(basename))) != 0: tar = tarfile.open(tarname, "w:gz") - tar.add(srcdir) + tar.add(basename) tar.close() else: tarname = '' - os.chdir(sav_dir) - return tarname + os.chdir(olddir) -def archive_sources_from_directory(d, stage_name): +python do_ar_log() { """ - archive sources codes tree to tarball when tarball of $P doesn't - exist in $DL_DIR + archive log.* and run.* under the ${T} dir. """ - s = d.getVar('S', True) - work_dir=d.getVar('WORKDIR', True) - PF = d.getVar('PF', True) - tarname = PF + '-' + stage_name + ".tar.gz" - - if os.path.exists(s) and work_dir in s: - try: - source_dir = os.path.join(work_dir, [ i for i in s.replace(work_dir, '').split('/') if i][0]) - except IndexError: - if not cmp(s, work_dir): - return '' - else: - return '' - source = os.path.basename(source_dir) - return do_tarball(work_dir, source, tarname) - -def archive_sources(d, stage_name): - """ - copy tarball from $DL_DIR to $WORKDIR if have tarball, archive - source codes tree in $WORKDIR if $P is directory instead of tarball - """ - import shutil - - work_dir = d.getVar('WORKDIR', True) - file = get_source_from_downloads(d, stage_name) - if file: - shutil.copy(file, work_dir) - file = os.path.basename(file) - else: - file = archive_sources_from_directory(d, stage_name) - return file - -def archive_patches(d, patchdir, series): - """ - archive patches to tarball and also include series files if 'series' is True - """ - import shutil - - s = d.getVar('S', True) - work_dir = d.getVar('WORKDIR', True) - patch_dir = os.path.basename(patchdir) - tarname = patch_dir + ".tar.gz" - if series == 'all' and os.path.exists(os.path.join(s, 'patches/series')): - shutil.copy(os.path.join(s, 'patches/series'), patchdir) - tarname = do_tarball(work_dir, patch_dir, tarname) - shutil.rmtree(patchdir, ignore_errors=True) - return tarname - -def select_archive_patches(d, option): - """ - select to archive all patches including non-applying and series or - applying patches - """ - if option == "all": - patchdir = get_series(d) - elif option == "applying": - patchdir = get_applying_patches(d) - try: - os.rmdir(patchdir) - except OSError: - tarpatch = archive_patches(d, patchdir, option) - return tarpatch - return - -def archive_logs(d, logdir, bbinc=False): - """ - archive logs in temp to tarball and .bb and .inc files if bbinc is True - """ - import shutil - - pf = d.getVar('PF', True) - work_dir = d.getVar('WORKDIR', True) - log_dir = os.path.basename(logdir) - tarname = pf + '-' + log_dir + ".tar.gz" - archive_dir = os.path.join( logdir, '..' ) - tarname = do_tarball(archive_dir, log_dir, tarname) - if bbinc: - shutil.rmtree(logdir, ignore_errors=True) - return tarname - -def get_licenses(d): - """get licenses for running .bb file""" - import oe.license - - licenses_type = d.getVar('LICENSE', True) or "" - lics = oe.license.is_included(licenses_type)[1:][0] - lice = '' - for lic in lics: - licens = d.getVarFlag('SPDXLICENSEMAP', lic) - if licens != None: - lice += licens - else: - lice += lic - return lice - - -def move_tarball_deploy(d, tarball_list): - """move tarball in location to ${DEPLOY_DIR}/sources""" - import shutil - - if tarball_list is []: + ar_log = d.getVarFlag('ARCHIVER_MODE', 'log', True) + if ar_log == 'no': return - target_sys = d.getVar('TARGET_SYS', True) - pf = d.getVar('PF', True) - licenses = get_licenses(d) - work_dir = d.getVar('WORKDIR', True) - tar_sources = d.getVar('DEPLOY_DIR', True) + '/sources/' + target_sys + '/' + licenses + '/' + pf - if not os.path.exists(tar_sources): - bb.utils.mkdirhier(tar_sources) - for source in tarball_list: - if source: - if os.path.exists(os.path.join(tar_sources, source)): - os.remove(os.path.join(tar_sources, source)) - shutil.move(os.path.join(work_dir, source), tar_sources) - -def check_archiving_type(d): - """check the type for archiving package('tar' or 'srpm')""" - if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) not in d.getVar('ARCHIVE_TYPE', True).split(): - bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" is \'tar\' or \'srpm\', no other types") - -def store_package(d, package_name): - """ - store tarbablls name to file "tar-package" - """ - f = open(os.path.join(d.getVar('WORKDIR', True), 'tar-package'), 'a') - f.write(package_name + ' ') - f.close() - -def get_package(d): - """ - get tarballs name from "tar-package" - """ - work_dir = (d.getVar('WORKDIR', True)) - tarlist = os.path.join(work_dir, 'tar-package') - if os.path.exists(tarlist): - f = open(tarlist, 'r') - line = f.readline().rstrip('\n').split() - f.close() - return line - return [] + elif ar_log and ar_log != 'yes': + bb.fatal("Invalid ARCHIVER_MODE[log]: %s" % ar_log) + create_tarball(d, d.getVar('T', True), 'log') +} -def archive_sources_patches(d, stage_name): +python do_ar_recipe() { """ - archive sources and patches to tarball. stage_name will append - strings ${stage_name} to ${PR} as middle name. for example, - zlib-1.4.6-prepatch(stage_name).tar.gz + archive the recipe, including .bb and .inc. """ + import re import shutil - check_archiving_type(d) + ar_recipe = d.getVarFlag('ARCHIVER_MODE', 'recipe', True) + if ar_recipe == 'no': + return + elif ar_recipe and ar_recipe != 'yes': + bb.fatal("Invalid ARCHIVER_MODE[recipe]: %s" % ar_recipe) - source_tar_name = archive_sources(d, stage_name) - if stage_name == "prepatch": - if d.getVar('PATCHES_ARCHIVE_WITH_SERIES', True) == 'yes': - patch_tar_name = select_archive_patches(d, "all") - elif d.getVar('PATCHES_ARCHIVE_WITH_SERIES', True) == 'no': - patch_tar_name = select_archive_patches(d, "applying") - else: - bb.fatal("Please define 'PATCHES_ARCHIVE_WITH_SERIES' to 'yes' or 'no' ") - else: - patch_tar_name = '' + require_re = re.compile( r"require\s+(.+)" ) + include_re = re.compile( r"include\s+(.+)" ) + outdir = os.path.join(d.getVar('ARCHIVER_OUTDIR', True), d.getVar('PF', True)) + bbfile = d.getVar('FILE', True) + bb.utils.mkdirhier(outdir) + shutil.copy(bbfile, outdir) + + dirname = os.path.dirname(bbfile) + bbpath = "%s:%s" % (dirname, d.getVar('BBPATH', True)) + f = open(bbfile, 'r') + for line in f.readlines(): + incfile = None + if require_re.match(line): + incfile = require_re.match(line).group(1) + elif include_re.match(line): + incfile = include_re.match(line).group(1) + if incfile: + incfile = bb.utils.which(bbpath, incfile) + if incfile: + shutil.copy(incfile, outdir) + + create_tarball(d, outdir, 'recipe') + shutil.rmtree(outdir) +} - if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) != 'srpm': - move_tarball_deploy(d, [source_tar_name, patch_tar_name]) - else: - tarlist = os.path.join(d.getVar('WORKDIR', True), 'tar-package') - if os.path.exists(tarlist): - os.remove(tarlist) - for package in os.path.basename(source_tar_name), patch_tar_name: - if package: - store_package(d, str(package) + ' ') - -def archive_scripts_logs(d): +python do_dumpdata() { """ - archive scripts and logs. scripts include .bb and .inc files and - logs include stuff in "temp". + dump environment data to ${PF}-showdata.dump """ - import shutil - work_dir = d.getVar('WORKDIR', True) - temp_dir = os.path.join(work_dir, 'temp') - source_archive_log_with_scripts = d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) - if source_archive_log_with_scripts == 'logs_with_scripts': - logdir = get_logs(d) - logdir = get_bb_inc(d) - elif source_archive_log_with_scripts == 'logs': - logdir = get_logs(d) - else: + # Check whether dump data (bb.data.emit_env) + emit_env = d.getVarFlag('ARCHIVER_MODE', 'dumpdata', True) + if not emit_env or emit_env == "no": return + elif emit_env != "yes": + bb.fatal("Invalid ARCHIVER_MODE[dumpdata]: %s" % emit_env) - tarlog = archive_logs(d, logdir, True) + bb.utils.mkdirhier(d.getVar('ARCHIVER_OUTDIR', True)) - if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) == 'srpm': - store_package(d, tarlog) - -def dumpdata(d): - """ - dump environment to "${P}-${PR}.showdata.dump" including all - kinds of variables and functions when running a task - """ - - workdir = bb.data.getVar('WORKDIR', d, 1) - distro = bb.data.getVar('DISTRO', d, 1) - s = d.getVar('S', True) - pf = d.getVar('PF', True) - target_sys = d.getVar('TARGET_SYS', True) - licenses = get_licenses(d) - dumpdir = os.path.join(workdir, 'diffgz-envdata/'+ target_sys + '/' + licenses + '/' + pf ) - if not os.path.exists(dumpdir): - bb.utils.mkdirhier(dumpdir) - - dumpfile = os.path.join(dumpdir, bb.data.expand("${P}-${PR}.showdata.dump", d)) + dumpfile = "%s/%s-showdata.dump" % \ + (d.getVar('ARCHIVER_OUTDIR', True), d.getVar("PF", True)) bb.note("Dumping metadata into '%s'" % dumpfile) f = open(dumpfile, "w") @@ -497,134 +253,73 @@ def dumpdata(d): # emit the metadata which isn't valid shell for e in d.keys(): if bb.data.getVarFlag(e, 'python', d): - f.write("\npython %s () {\n%s}\n" % (e, bb.data.getVar(e, d, 1))) - f.close() - -def create_diff_gz(d): - """ - creating .diff.gz in ${DEPLOY_DIR_SRC}/${P}-${PR}.diff.g gz for - mapping all content in 's' including patches to xxx.diff.gz - """ - import shutil - import subprocess - - work_dir = d.getVar('WORKDIR', True) - exclude_from = d.getVar('ARCHIVE_EXCLUDE_FROM', True).split() - pf = d.getVar('PF', True) - licenses = get_licenses(d) - target_sys = d.getVar('TARGET_SYS', True) - diff_dir = os.path.join(work_dir, 'diffgz-envdata/'+ target_sys + '/' + licenses + '/' + pf ) - diff_file = os.path.join(diff_dir, bb.data.expand("${P}-${PR}.diff.gz",d)) - - f = open(os.path.join(work_dir,'temp/exclude-from-file'), 'a') - for i in exclude_from: - f.write(i) - f.write("\n") + f.write("\npython %s () {\n%s}\n" % (e, bb.data.getVar(e, d, True))) f.close() - - s=d.getVar('S', True) - distro = d.getVar('DISTRO',True) or "" - dest = s + '/' + distro + '/files' - if not os.path.exists(dest): - bb.utils.mkdirhier(dest) - for i in os.listdir(os.getcwd()): - if os.path.isfile(i): - try: - shutil.copy(i, dest) - except IOError: - subprocess.call('fakeroot cp -rf ' + i + " " + dest, shell=True) - - bb.note("Creating .diff.gz in ${DEPLOY_DIR_SRC}/${P}-${PR}.diff.gz") - cmd = "LC_ALL=C TZ=UTC0 diff --exclude-from=" + work_dir + "/temp/exclude-from-file -Naur " + s + '.org' + ' ' + s + " | gzip -c > " + diff_file - d.setVar('DIFF', cmd + "\n") - d.setVarFlag('DIFF', 'func', '1') - bb.build.exec_func('DIFF', d) - shutil.rmtree(s + '.org', ignore_errors=True) - -# This function will run when user want to get tarball for sources and -# patches after do_unpack -python do_archive_original_sources(){ - archive_sources_patches(d, 'prepatch') } -# This function will run when user want to get tarball for patched -# sources after do_patch -python do_archive_patched_sources(){ - archive_sources_patches(d, 'patched') +# Save ${S} to ${S}.orig +python do_save_orig() { + s = d.getVar('S', True).rstrip('/') + oe.path.copytree(s, "%s.orig" % s) } -# This function will run when user want to get tarball for configured -# sources after do_configure -python do_archive_configured_sources(){ - archive_sources_patches(d, 'configured') +# creating .diff.gz between source.orig and source +create_diff_gz() { + # creating .diff.gz between source.orig and source + + diff_file="${ARCHIVER_OUTDIR}/${PF}-diff.gz" + # FIXME: + # The diff --exclude can't exclude the file with path, so we copy + # the patched source, and remove the files that we'd like to + # exclude. + s=`echo ${S} | sed 's#/*$##'` + cp -a $s ${s}.patched + for i in ${AR_EXCLUDE}; do + rm -fr ${s}.orig/$i + rm -fr ${s}.patched/$i + done + + mkdir -p "${ARCHIVER_OUTDIR}" + dname=$(dirname ${s}) + basename=$(basename ${s}) + olddir=$(pwd) + cd $dname || exit 1 + LC_ALL=C TZ=UTC0 diff -Naur \ + ${basename}.orig ${basename}.patched | gzip -c > $diff_file + rm -fr ${s}.orig ${s}.patched + cd $olddir } -# This function will run when user want to get tarball for logs or both -# logs and scripts(.bb and .inc files) -python do_archive_scripts_logs(){ - archive_scripts_logs(d) +# Create the tarball after do_unpack +python do_ar_prepatch(){ + create_tarball(d, d.getVar('S', True), 'prepatch') } -# This function will run when user want to know what variable and -# functions in a running task are and also can get a diff file including -# all content a package should include. -python do_dumpdata_create_diff_gz(){ - dumpdata(d) - create_diff_gz(d) +# Create the tarball after do_patch +python do_ar_patched(){ + create_tarball(d, d.getVar('S', True), 'patched') } -# This functions prepare for archiving "linux-yocto" because this -# package create directory 's' before do_patch instead of after -# do_unpack. This is special control for archiving linux-yocto only. -python do_archive_linux_yocto(){ - s = d.getVar('S', True) - if 'linux-yocto' in s: - source_tar_name = archive_sources(d, '') - if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) != 'srpm': - move_tarball_deploy(d, [source_tar_name, '']) +# Create the tarball after do_configure +python do_ar_configured(){ + create_tarball(d, d.getVar('S', True), 'configured') } -do_kernel_checkout[postfuncs] += "do_archive_linux_yocto " - -# remove tarball for sources, patches and logs after creating srpm. -python do_delete_tarlist(){ - work_dir = d.getVar('WORKDIR', True) - tarlist = os.path.join(work_dir, 'tar-package') - if os.path.exists(tarlist): - os.remove(tarlist) -} -do_delete_tarlist[deptask] = "do_archive_scripts_logs" -do_package_write_rpm[postfuncs] += "do_delete_tarlist " - -# Get archiving package with temp(logs) and scripts(.bb and .inc files) -addtask do_archive_scripts_logs - -# Get dump date and create diff file -addtask do_dumpdata_create_diff_gz -ARCHIVE_SSTATE_OUTDIR = "${DEPLOY_DIR}/sources/" -ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR = "${WORKDIR}/script-logs/" -ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR = "${WORKDIR}/diffgz-envdata/" - -SSTATETASKS += "do_archive_scripts_logs" -do_archive_scripts_logs[sstate-inputdirs] = "${ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR}" -do_archive_scripts_logs[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}" - -python do_archive_scripts_logs_setscene () { - sstate_setscene(d) +do_deploy_archives () { + echo "Deploying source archive files ..." } -addtask do_archive_scripts_logs_setscene - -SSTATETASKS += "do_dumpdata_create_diff_gz" -do_dumpdata_create_diff_gz[sstate-inputdirs] = "${ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR}" -do_dumpdata_create_diff_gz[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}" - -python do_dumpdata_create_diff_gz_setscene () { +SSTATETASKS += "do_deploy_archives" +python do_deploy_archives_setscene () { sstate_setscene(d) } -addtask do_dumpdata_create_diff_gz_setscene +do_deploy_archives[sstate-inputdirs] = "${AR_TOPDIR}" +do_deploy_archives[sstate-outputdirs] = "${DEPLOY_DIR_SRC}" + +# This is used for deploying the archives +addtask do_deploy_archives after do_compile -addtask do_archive_original_sources after do_unpack -addtask do_archive_patched_sources after do_patch -addtask do_archive_configured_sources after do_configure +addtask do_dumpdata before do_deploy_archives +addtask do_ar_recipe before do_deploy_archives +addtask do_ar_log before do_deploy_archives -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 3/5] package_rpm.bbclass: archive the source to srpm package 2014-01-13 16:20 [PATCH 0/5] refactor the archive*.bbcalss Robert Yang ` (2 preceding siblings ...) 2014-01-13 16:20 ` [PATCH 2/5] archiver.bbclass: refactor it Robert Yang @ 2014-01-13 16:20 ` Robert Yang 2014-01-13 16:20 ` [PATCH 4/5] archiver.bbclass: move a few code to copyleft_compliance.bbclass Robert Yang ` (3 subsequent siblings) 7 siblings, 0 replies; 24+ messages in thread From: Robert Yang @ 2014-01-13 16:20 UTC (permalink / raw) To: openembedded-core The archiver.bbclass will put the sources to ARCHIVER_OUTDIR according to configuration, then the rpmbuild -bs will create the srpm. [YOCTO #4986] [YOCTO #5113] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta/classes/package_rpm.bbclass | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass index fcde73e..2da2b44 100644 --- a/meta/classes/package_rpm.bbclass +++ b/meta/classes/package_rpm.bbclass @@ -6,7 +6,6 @@ RPM="rpm" RPMBUILD="rpmbuild" PKGWRITEDIRRPM = "${WORKDIR}/deploy-rpms" -PKGWRITEDIRSRPM = "${DEPLOY_DIR}/sources/deploy-srpm" # Maintaining the perfile dependencies has singificant overhead when writing the # packages. When set, this value merges them for efficiency. @@ -559,23 +558,26 @@ python write_specfile () { # append information for logs and patches to %prep def add_prep(d,spec_files_bottom): - if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) == 'srpm': + if d.getVarFlag('ARCHIVER_MODE', 'type', True) == 'srpm': spec_files_bottom.append('%%prep -n %s' % d.getVar('PN', True) ) spec_files_bottom.append('%s' % "echo \"include logs and patches, Please check them in SOURCES\"") spec_files_bottom.append('') # append the name of tarball to key word 'SOURCE' in xxx.spec. def tail_source(d): - if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) == 'srpm': - source_list = get_package(d) + if d.getVarFlag('ARCHIVER_MODE', 'type', True) == 'srpm': + ar_dir = d.getVar('ARCHIVER_OUTDIR', True) + if not os.path.exists(ar_dir): + return + source_list = os.listdir(ar_dir) source_number = 0 - workdir = d.getVar('WORKDIR', True) for source in source_list: # The rpmbuild doesn't need the root permission, but it needs # to know the file's user and group name, the only user and # group in fakeroot is "root" when working in fakeroot. - os.chown("%s/%s" % (workdir, source), 0, 0) - spec_preamble_top.append('Source' + str(source_number) + ': %s' % source) + f = os.path.join(ar_dir, source) + os.chown(f, 0, 0) + spec_preamble_top.append('Source%s: %s' % (source_number, source)) source_number += 1 # We need a simple way to remove the MLPREFIX from the package name, # and dependency information... @@ -1070,15 +1072,6 @@ python write_specfile () { } python do_package_rpm () { - def creat_srpm_dir(d): - if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) == 'srpm': - clean_licenses = get_licenses(d) - pkgwritesrpmdir = bb.data.expand('${PKGWRITEDIRSRPM}/${PACKAGE_ARCH_EXTEND}', d) - pkgwritesrpmdir = pkgwritesrpmdir + '/' + clean_licenses - bb.utils.mkdirhier(pkgwritesrpmdir) - os.chmod(pkgwritesrpmdir, 0755) - return pkgwritesrpmdir - # We need a simple way to remove the MLPREFIX from the package name, # and dependency information... def strip_multilib(name, d): @@ -1146,9 +1139,9 @@ python do_package_rpm () { cmd = cmd + " --define 'debug_package %{nil}'" cmd = cmd + " --define '_rpmfc_magic_path " + magicfile + "'" cmd = cmd + " --define '_tmppath " + workdir + "'" - if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) == 'srpm': - cmd = cmd + " --define '_sourcedir " + workdir + "'" - cmdsrpm = cmd + " --define '_srcrpmdir " + creat_srpm_dir(d) + "'" + if d.getVarFlag('ARCHIVER_MODE', 'type', True) == 'srpm': + cmd = cmd + " --define '_sourcedir " + d.getVar('ARCHIVER_OUTDIR', True) + "'" + cmdsrpm = cmd + " --define '_srcrpmdir " + d.getVar('ARCHIVER_OUTDIR', True) + "'" cmdsrpm = cmdsrpm + " -bs " + outspecfile # Build the .src.rpm d.setVar('SBUILDSPEC', cmdsrpm + "\n") -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 4/5] archiver.bbclass: move a few code to copyleft_compliance.bbclass 2014-01-13 16:20 [PATCH 0/5] refactor the archive*.bbcalss Robert Yang ` (3 preceding siblings ...) 2014-01-13 16:20 ` [PATCH 3/5] package_rpm.bbclass: archive the source to srpm package Robert Yang @ 2014-01-13 16:20 ` Robert Yang 2014-01-13 16:20 ` [PATCH 5/5] local.conf.sample.extended: update for the archiver Robert Yang ` (2 subsequent siblings) 7 siblings, 0 replies; 24+ messages in thread From: Robert Yang @ 2014-01-13 16:20 UTC (permalink / raw) To: openembedded-core Move the code which is only used by copyleft_compliance.bbclass from archiver.bbclassc, and remove the "inherit archiver" from copyleft_compliance.bbclass. The archiver.bbclass is used for archiving various types of sources, but the copyleft_compliance.bbclass is used for analysing the license, they don't have much relationships. [YOCTO #4986] [YOCTO #5113] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta/classes/archiver.bbclass | 53 ------------------------------ meta/classes/copyleft_compliance.bbclass | 55 ++++++++++++++++++++++++++++++-- 2 files changed, 53 insertions(+), 55 deletions(-) diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass index f2d7184..c3e5bbf 100644 --- a/meta/classes/archiver.bbclass +++ b/meta/classes/archiver.bbclass @@ -35,25 +35,6 @@ ARCHIVER_OUTDIR = "${AR_TOPDIR}/${TARGET_SYS}/${PF}" # This is a convenience for the shell script to use it AR_EXCLUDE ?= "${@d.getVarFlag('ARCHIVER_MODE', 'diff-exclude', True)}" -COPYLEFT_LICENSE_INCLUDE ?= 'GPL* LGPL*' -COPYLEFT_LICENSE_INCLUDE[type] = 'list' -COPYLEFT_LICENSE_INCLUDE[doc] = 'Space separated list of globs which include licenses' - -COPYLEFT_LICENSE_EXCLUDE ?= 'CLOSED Proprietary' -COPYLEFT_LICENSE_EXCLUDE[type] = 'list' -COPYLEFT_LICENSE_EXCLUDE[doc] = 'Space separated list of globs which exclude licenses' - -COPYLEFT_RECIPE_TYPE ?= '${@copyleft_recipe_type(d)}' -COPYLEFT_RECIPE_TYPE[doc] = 'The "type" of the current recipe (e.g. target, native, cross)' - -COPYLEFT_RECIPE_TYPES ?= 'target' -COPYLEFT_RECIPE_TYPES[type] = 'list' -COPYLEFT_RECIPE_TYPES[doc] = 'Space separated list of recipe types to include' - -COPYLEFT_AVAILABLE_RECIPE_TYPES = 'target native nativesdk cross crosssdk cross-canadian' -COPYLEFT_AVAILABLE_RECIPE_TYPES[type] = 'list' -COPYLEFT_AVAILABLE_RECIPE_TYPES[doc] = 'Space separated list of available recipe types' - python () { pn = d.getVar('PN', True) packaging = d.getVar('IMAGE_PKGTYPE', True) @@ -117,40 +98,6 @@ python () { d.appendVarFlag('do_build', 'depends', build_deps) } -def copyleft_recipe_type(d): - for recipe_type in oe.data.typed_value('COPYLEFT_AVAILABLE_RECIPE_TYPES', d): - if oe.utils.inherits(d, recipe_type): - return recipe_type - return 'target' - -def copyleft_should_include(d): - """ - Determine if this recipe's sources should be deployed for compliance - """ - import ast - import oe.license - from fnmatch import fnmatchcase as fnmatch - - recipe_type = d.getVar('COPYLEFT_RECIPE_TYPE', True) - if recipe_type not in oe.data.typed_value('COPYLEFT_RECIPE_TYPES', d): - return False, 'recipe type "%s" is excluded' % recipe_type - - include = oe.data.typed_value('COPYLEFT_LICENSE_INCLUDE', d) - exclude = oe.data.typed_value('COPYLEFT_LICENSE_EXCLUDE', d) - - try: - is_included, reason = oe.license.is_included(d.getVar('LICENSE', True), include, exclude) - except oe.license.LicenseError as exc: - bb.fatal('%s: %s' % (d.getVar('PF', True), exc)) - else: - if is_included: - if reason: - return True, 'recipe has included licenses: %s' % ', '.join(reason) - else: - return False, 'recipe does not include a copyleft license' - else: - return False, 'recipe has excluded licenses: %s' % ', '.join(reason) - def create_tarball(d, srcdir, suffix): """ create the tarball from srcdir diff --git a/meta/classes/copyleft_compliance.bbclass b/meta/classes/copyleft_compliance.bbclass index 32aa757..b47c2c9 100644 --- a/meta/classes/copyleft_compliance.bbclass +++ b/meta/classes/copyleft_compliance.bbclass @@ -6,11 +6,62 @@ # # vi:sts=4:sw=4:et -# Need the copyleft_should_include -inherit archiver +COPYLEFT_LICENSE_INCLUDE ?= 'GPL* LGPL*' +COPYLEFT_LICENSE_INCLUDE[type] = 'list' +COPYLEFT_LICENSE_INCLUDE[doc] = 'Space separated list of globs which include licenses' + +COPYLEFT_LICENSE_EXCLUDE ?= 'CLOSED Proprietary' +COPYLEFT_LICENSE_EXCLUDE[type] = 'list' +COPYLEFT_LICENSE_EXCLUDE[doc] = 'Space separated list of globs which exclude licenses' + +COPYLEFT_RECIPE_TYPE ?= '${@copyleft_recipe_type(d)}' +COPYLEFT_RECIPE_TYPE[doc] = 'The "type" of the current recipe (e.g. target, native, cross)' + +COPYLEFT_RECIPE_TYPES ?= 'target' +COPYLEFT_RECIPE_TYPES[type] = 'list' +COPYLEFT_RECIPE_TYPES[doc] = 'Space separated list of recipe types to include' + +COPYLEFT_AVAILABLE_RECIPE_TYPES = 'target native nativesdk cross crosssdk cross-canadian' +COPYLEFT_AVAILABLE_RECIPE_TYPES[type] = 'list' +COPYLEFT_AVAILABLE_RECIPE_TYPES[doc] = 'Space separated list of available recipe types' COPYLEFT_SOURCES_DIR ?= '${DEPLOY_DIR}/copyleft_sources' +def copyleft_recipe_type(d): + for recipe_type in oe.data.typed_value('COPYLEFT_AVAILABLE_RECIPE_TYPES', d): + if oe.utils.inherits(d, recipe_type): + return recipe_type + return 'target' + +def copyleft_should_include(d): + """ + Determine if this recipe's sources should be deployed for compliance + """ + import ast + import oe.license + from fnmatch import fnmatchcase as fnmatch + + recipe_type = d.getVar('COPYLEFT_RECIPE_TYPE', True) + if recipe_type not in oe.data.typed_value('COPYLEFT_RECIPE_TYPES', d): + return False, 'recipe type "%s" is excluded' % recipe_type + + include = oe.data.typed_value('COPYLEFT_LICENSE_INCLUDE', d) + exclude = oe.data.typed_value('COPYLEFT_LICENSE_EXCLUDE', d) + + try: + is_included, reason = oe.license.is_included(d.getVar('LICENSE', True), include, exclude) + except oe.license.LicenseError as exc: + bb.fatal('%s: %s' % (d.getVar('PF', True), exc)) + else: + if is_included: + if reason: + return True, 'recipe has included licenses: %s' % ', '.join(reason) + else: + return False, 'recipe does not include a copyleft license' + else: + return False, 'recipe has excluded licenses: %s' % ', '.join(reason) + + python do_prepare_copyleft_sources () { """Populate a tree of the recipe sources and emit patch series files""" import os.path -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 5/5] local.conf.sample.extended: update for the archiver 2014-01-13 16:20 [PATCH 0/5] refactor the archive*.bbcalss Robert Yang ` (4 preceding siblings ...) 2014-01-13 16:20 ` [PATCH 4/5] archiver.bbclass: move a few code to copyleft_compliance.bbclass Robert Yang @ 2014-01-13 16:20 ` Robert Yang 2014-01-15 16:55 ` [PATCH 0/5] refactor the archive*.bbcalss Martin Jansa 2014-02-13 9:54 ` Robert Yang 7 siblings, 0 replies; 24+ messages in thread From: Robert Yang @ 2014-01-13 16:20 UTC (permalink / raw) To: openembedded-core Updated it since we have refactored the archiver.bbclass. [YOCTO #5113] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta-yocto/conf/local.conf.sample.extended | 72 ++++++++++++++++-------------- 1 file changed, 38 insertions(+), 34 deletions(-) diff --git a/meta-yocto/conf/local.conf.sample.extended b/meta-yocto/conf/local.conf.sample.extended index c7c4f40..c0a98ee 100644 --- a/meta-yocto/conf/local.conf.sample.extended +++ b/meta-yocto/conf/local.conf.sample.extended @@ -188,40 +188,44 @@ # when the disk space reduces 50M (or the amount of inode reduces 5k). #BB_DISKMON_WARNINTERVAL = "50M,5K" -# Archiving source code, configure what kind of sources will be archived -# and the output format. The output files will be put in -# ${DEPLOY_DIR}/sources/. -# -# You can add the following 3 lines to the conf file to get a quick -# usage: -#ARCHIVER_MODE ?= "original" -#ARCHIVER_CLASS = "${@'archive-${ARCHIVER_MODE}-source' if ARCHIVER_MODE != 'none' else ''}" -#INHERIT += "${ARCHIVER_CLASS}" -# -# Detailed configuration: -# What kind of sources will be archived, the ARCHIVER_MODE could be: -# original: the ${S} after do_unpack -# patched : the ${S} after do_patch -# configured: the ${S} after do_configure -#ARCHIVER_MODE ?= "original" -# -# The output format type, tar or srpm, the default is "tar". -#ARCHIVER_MODE[type] ?= "tar" -# -# Whether include the log file under ${T} and the recipe (.bb and .inc), -# the default is "logs_with_scripts". -#ARCHIVER_MODE[log_type] ?= "logs_with_scripts" -# -# license filter: -# yes: Only the COPYLEFT_LICENSE_INCLUDE recipe will be archived -# no: All kinds of license will be archived -# The default is "no" -#ARCHIVER_MODE[filter] ?= "no" -# -# The following lines should be added to your conf file verbatim -# (uncommented though of course): -#ARCHIVER_CLASS = "${@'archive-${ARCHIVER_MODE}-source' if ARCHIVER_MODE != 'none' else ''}" -#INHERIT += "${ARCHIVER_CLASS}" +# Archive the source and put them to ${DEPLOY_DIR}/sources/. +# +#INHERIT += "archiver" +# +# The tarball for the configured source, the diff between do_unpack and +# do_patch will be archived by default, and you can configure the +# archiver when needed: +# +#The output format: +# 1) tar: (default) +#ARCHIVER_MODE[type] = "tar" +# 2) srpm: (when PACKAGES != "") +#ARCHIVER_MODE[type] = "srpm", +# NOTE: The logs can't be packed into the srpm package because they are +# incompleted until the rpm package has been built. +# +# Archive the source: +# 1) before do_patch: +#ARCHIVER_MODE[src] = "prepatch" +# 2) after do_patch: +#ARCHIVER_MODE[src] = "patched" +# 3) after do_configure: (default) +#ARCHIVER_MODE[src] = "configured" +# +# 4) the patches between do_unpack and do_patch: +#ARCHIVER_MODE[diff] = "yes" +# set the files that you'd like to exclude from the diff: +#ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches" +# +# 5) the environment data, similar to 'bitbake -e recipe': +#ARCHIVER_MODE[dumpdata] = "yes" +# +# 6) the recipe (.bb and .inc): +#ARCHIVER_MODE[recipe] = "yes" +# +# 7) the log.* and run.* files under ${T} +#ARCHIVER_MODE[log] = "yes" +# # Remove the old image before the new one generated to save disk space #RM_OLD_IMAGE = "1" -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-13 16:20 [PATCH 0/5] refactor the archive*.bbcalss Robert Yang ` (5 preceding siblings ...) 2014-01-13 16:20 ` [PATCH 5/5] local.conf.sample.extended: update for the archiver Robert Yang @ 2014-01-15 16:55 ` Martin Jansa 2014-01-16 2:14 ` Robert Yang 2014-02-13 9:54 ` Robert Yang 7 siblings, 1 reply; 24+ messages in thread From: Martin Jansa @ 2014-01-15 16:55 UTC (permalink / raw) To: Robert Yang; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 2459 bytes --] On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: > * The archive*.bbclass didn't work, and there were a few problems, for > example: > 1) There were a few duplicated code > 2) There was no src_dir.org (or orig), but the diff command still use > it, and it is not easy to fix this issue if we don't change a lot > of the code. > 3) It didn't archive the source for the native or gcc > 4) The work flow is not very well > 5) The "subprocess.call('fakeroot cp xxxx'" should be removed > 6) And others ... > > * So that we have to refactor it, the benefits are: > 1) Fix the problems and make it work well. > 2) Reduce more than 400 lines in total. > 3) Make it easy to use. > > // Robert > > The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: > > bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) > > are available in the git repository at: > > git://git.pokylinux.org/poky-contrib rbt/archiver > http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver Can you send pull-request for oe-core? > Robert Yang (5): > classes/archive*.bbclass: remove archive-*-source.bbclass > archiver.bbclass: refactor it > package_rpm.bbclass: archive the source to srpm package > archiver.bbclass: move a few code to copyleft_compliance.bbclass > local.conf.sample.extended: update for the archiver > > meta-yocto/conf/local.conf.sample.extended | 72 +-- > meta/classes/archive-configured-source.bbclass | 65 --- > meta/classes/archive-original-source.bbclass | 65 --- > meta/classes/archive-patched-source.bbclass | 65 --- > meta/classes/archiver.bbclass | 723 ++++++++----------------- > meta/classes/copyleft_compliance.bbclass | 55 +- > meta/classes/package_rpm.bbclass | 31 +- > 7 files changed, 316 insertions(+), 760 deletions(-) > delete mode 100644 meta/classes/archive-configured-source.bbclass > delete mode 100644 meta/classes/archive-original-source.bbclass > delete mode 100644 meta/classes/archive-patched-source.bbclass > > -- > 1.8.3.1 > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-15 16:55 ` [PATCH 0/5] refactor the archive*.bbcalss Martin Jansa @ 2014-01-16 2:14 ` Robert Yang 2014-01-16 9:25 ` Martin Jansa 0 siblings, 1 reply; 24+ messages in thread From: Robert Yang @ 2014-01-16 2:14 UTC (permalink / raw) To: Martin Jansa; +Cc: openembedded-core On 01/16/2014 12:55 AM, Martin Jansa wrote: > On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: >> * The archive*.bbclass didn't work, and there were a few problems, for >> example: >> 1) There were a few duplicated code >> 2) There was no src_dir.org (or orig), but the diff command still use >> it, and it is not easy to fix this issue if we don't change a lot >> of the code. >> 3) It didn't archive the source for the native or gcc >> 4) The work flow is not very well >> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed >> 6) And others ... >> >> * So that we have to refactor it, the benefits are: >> 1) Fix the problems and make it work well. >> 2) Reduce more than 400 lines in total. >> 3) Make it easy to use. >> >> // Robert >> >> The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: >> >> bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) >> >> are available in the git repository at: >> >> git://git.pokylinux.org/poky-contrib rbt/archiver >> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver > > Can you send pull-request for oe-core? > Here it is: git://git.pokylinux.org/poky-contrib rbt/ar_oe http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rbt/ar_oe // Robert >> Robert Yang (5): >> classes/archive*.bbclass: remove archive-*-source.bbclass >> archiver.bbclass: refactor it >> package_rpm.bbclass: archive the source to srpm package >> archiver.bbclass: move a few code to copyleft_compliance.bbclass >> local.conf.sample.extended: update for the archiver >> >> meta-yocto/conf/local.conf.sample.extended | 72 +-- >> meta/classes/archive-configured-source.bbclass | 65 --- >> meta/classes/archive-original-source.bbclass | 65 --- >> meta/classes/archive-patched-source.bbclass | 65 --- >> meta/classes/archiver.bbclass | 723 ++++++++----------------- >> meta/classes/copyleft_compliance.bbclass | 55 +- >> meta/classes/package_rpm.bbclass | 31 +- >> 7 files changed, 316 insertions(+), 760 deletions(-) >> delete mode 100644 meta/classes/archive-configured-source.bbclass >> delete mode 100644 meta/classes/archive-original-source.bbclass >> delete mode 100644 meta/classes/archive-patched-source.bbclass >> >> -- >> 1.8.3.1 >> >> _______________________________________________ >> Openembedded-core mailing list >> Openembedded-core@lists.openembedded.org >> http://lists.openembedded.org/mailman/listinfo/openembedded-core > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-16 2:14 ` Robert Yang @ 2014-01-16 9:25 ` Martin Jansa 2014-01-16 9:33 ` Robert Yang 0 siblings, 1 reply; 24+ messages in thread From: Martin Jansa @ 2014-01-16 9:25 UTC (permalink / raw) To: Robert Yang; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 3056 bytes --] On Thu, Jan 16, 2014 at 10:14:30AM +0800, Robert Yang wrote: > > > On 01/16/2014 12:55 AM, Martin Jansa wrote: > > On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: > >> * The archive*.bbclass didn't work, and there were a few problems, for > >> example: > >> 1) There were a few duplicated code > >> 2) There was no src_dir.org (or orig), but the diff command still use > >> it, and it is not easy to fix this issue if we don't change a lot > >> of the code. > >> 3) It didn't archive the source for the native or gcc > >> 4) The work flow is not very well > >> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed > >> 6) And others ... > >> > >> * So that we have to refactor it, the benefits are: > >> 1) Fix the problems and make it work well. > >> 2) Reduce more than 400 lines in total. > >> 3) Make it easy to use. > >> > >> // Robert > >> > >> The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: > >> > >> bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) > >> > >> are available in the git repository at: > >> > >> git://git.pokylinux.org/poky-contrib rbt/archiver > >> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver > > > > Can you send pull-request for oe-core? > > > > Here it is: > git://git.pokylinux.org/poky-contrib rbt/ar_oe > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rbt/ar_oe This still looks like poky-contrib (so it will download whole poky-contrib when I add it as remote to oe-core repo). > // Robert > > >> Robert Yang (5): > >> classes/archive*.bbclass: remove archive-*-source.bbclass > >> archiver.bbclass: refactor it > >> package_rpm.bbclass: archive the source to srpm package > >> archiver.bbclass: move a few code to copyleft_compliance.bbclass > >> local.conf.sample.extended: update for the archiver > >> > >> meta-yocto/conf/local.conf.sample.extended | 72 +-- > >> meta/classes/archive-configured-source.bbclass | 65 --- > >> meta/classes/archive-original-source.bbclass | 65 --- > >> meta/classes/archive-patched-source.bbclass | 65 --- > >> meta/classes/archiver.bbclass | 723 ++++++++----------------- > >> meta/classes/copyleft_compliance.bbclass | 55 +- > >> meta/classes/package_rpm.bbclass | 31 +- > >> 7 files changed, 316 insertions(+), 760 deletions(-) > >> delete mode 100644 meta/classes/archive-configured-source.bbclass > >> delete mode 100644 meta/classes/archive-original-source.bbclass > >> delete mode 100644 meta/classes/archive-patched-source.bbclass > >> > >> -- > >> 1.8.3.1 > >> > >> _______________________________________________ > >> Openembedded-core mailing list > >> Openembedded-core@lists.openembedded.org > >> http://lists.openembedded.org/mailman/listinfo/openembedded-core > > -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-16 9:25 ` Martin Jansa @ 2014-01-16 9:33 ` Robert Yang 2014-01-18 12:26 ` Martin Jansa 0 siblings, 1 reply; 24+ messages in thread From: Robert Yang @ 2014-01-16 9:33 UTC (permalink / raw) To: Martin Jansa; +Cc: openembedded-core On 01/16/2014 05:25 PM, Martin Jansa wrote: > On Thu, Jan 16, 2014 at 10:14:30AM +0800, Robert Yang wrote: >> >> >> On 01/16/2014 12:55 AM, Martin Jansa wrote: >>> On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: >>>> * The archive*.bbclass didn't work, and there were a few problems, for >>>> example: >>>> 1) There were a few duplicated code >>>> 2) There was no src_dir.org (or orig), but the diff command still use >>>> it, and it is not easy to fix this issue if we don't change a lot >>>> of the code. >>>> 3) It didn't archive the source for the native or gcc >>>> 4) The work flow is not very well >>>> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed >>>> 6) And others ... >>>> >>>> * So that we have to refactor it, the benefits are: >>>> 1) Fix the problems and make it work well. >>>> 2) Reduce more than 400 lines in total. >>>> 3) Make it easy to use. >>>> >>>> // Robert >>>> >>>> The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: >>>> >>>> bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) >>>> >>>> are available in the git repository at: >>>> >>>> git://git.pokylinux.org/poky-contrib rbt/archiver >>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver >>> >>> Can you send pull-request for oe-core? >>> >> >> Here it is: >> git://git.pokylinux.org/poky-contrib rbt/ar_oe >> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rbt/ar_oe > > This still looks like poky-contrib (so it will download whole > poky-contrib when I add it as remote to oe-core repo). > Sorry, I can't create repo on oe-contrib, seems that I lost my key. How about: git fetch git://git.pokylinux.org/poky-contrib rbt/ar_oe // Robert >> // Robert >> >>>> Robert Yang (5): >>>> classes/archive*.bbclass: remove archive-*-source.bbclass >>>> archiver.bbclass: refactor it >>>> package_rpm.bbclass: archive the source to srpm package >>>> archiver.bbclass: move a few code to copyleft_compliance.bbclass >>>> local.conf.sample.extended: update for the archiver >>>> >>>> meta-yocto/conf/local.conf.sample.extended | 72 +-- >>>> meta/classes/archive-configured-source.bbclass | 65 --- >>>> meta/classes/archive-original-source.bbclass | 65 --- >>>> meta/classes/archive-patched-source.bbclass | 65 --- >>>> meta/classes/archiver.bbclass | 723 ++++++++----------------- >>>> meta/classes/copyleft_compliance.bbclass | 55 +- >>>> meta/classes/package_rpm.bbclass | 31 +- >>>> 7 files changed, 316 insertions(+), 760 deletions(-) >>>> delete mode 100644 meta/classes/archive-configured-source.bbclass >>>> delete mode 100644 meta/classes/archive-original-source.bbclass >>>> delete mode 100644 meta/classes/archive-patched-source.bbclass >>>> >>>> -- >>>> 1.8.3.1 >>>> >>>> _______________________________________________ >>>> Openembedded-core mailing list >>>> Openembedded-core@lists.openembedded.org >>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core >>> > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-16 9:33 ` Robert Yang @ 2014-01-18 12:26 ` Martin Jansa 2014-01-21 10:40 ` Robert Yang 2014-01-21 10:41 ` Robert Yang 0 siblings, 2 replies; 24+ messages in thread From: Martin Jansa @ 2014-01-18 12:26 UTC (permalink / raw) To: Robert Yang; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 4092 bytes --] On Thu, Jan 16, 2014 at 05:33:20PM +0800, Robert Yang wrote: > > > On 01/16/2014 05:25 PM, Martin Jansa wrote: > > On Thu, Jan 16, 2014 at 10:14:30AM +0800, Robert Yang wrote: > >> > >> > >> On 01/16/2014 12:55 AM, Martin Jansa wrote: > >>> On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: > >>>> * The archive*.bbclass didn't work, and there were a few problems, for > >>>> example: > >>>> 1) There were a few duplicated code > >>>> 2) There was no src_dir.org (or orig), but the diff command still use > >>>> it, and it is not easy to fix this issue if we don't change a lot > >>>> of the code. > >>>> 3) It didn't archive the source for the native or gcc > >>>> 4) The work flow is not very well > >>>> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed > >>>> 6) And others ... > >>>> > >>>> * So that we have to refactor it, the benefits are: > >>>> 1) Fix the problems and make it work well. > >>>> 2) Reduce more than 400 lines in total. > >>>> 3) Make it easy to use. > >>>> > >>>> // Robert > >>>> > >>>> The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: > >>>> > >>>> bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) > >>>> > >>>> are available in the git repository at: > >>>> > >>>> git://git.pokylinux.org/poky-contrib rbt/archiver > >>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver > >>> > >>> Can you send pull-request for oe-core? > >>> > >> > >> Here it is: > >> git://git.pokylinux.org/poky-contrib rbt/ar_oe > >> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rbt/ar_oe > > > > This still looks like poky-contrib (so it will download whole > > poky-contrib when I add it as remote to oe-core repo). > > > > Sorry, I can't create repo on oe-contrib, seems that I lost my key. That's sad, you should send new one to khem to replace it, before someone else finds your old key. > How about: > > git fetch git://git.pokylinux.org/poky-contrib rbt/ar_oe That still fetches whole poky repo I'm not interested in. So I've downloaded just archiver.bbclass manually and tested it on my use-case, I can confirm that it doesn't remove required file anymore, but it seems that there is something terribly broken when used in combination with rm_work or multimachine builds. I haven't debugged it yet, but 2nd build (in the same directory) with this new archiver included almost every recipe failed in do_patch task. > >> // Robert > >> > >>>> Robert Yang (5): > >>>> classes/archive*.bbclass: remove archive-*-source.bbclass > >>>> archiver.bbclass: refactor it > >>>> package_rpm.bbclass: archive the source to srpm package > >>>> archiver.bbclass: move a few code to copyleft_compliance.bbclass > >>>> local.conf.sample.extended: update for the archiver > >>>> > >>>> meta-yocto/conf/local.conf.sample.extended | 72 +-- > >>>> meta/classes/archive-configured-source.bbclass | 65 --- > >>>> meta/classes/archive-original-source.bbclass | 65 --- > >>>> meta/classes/archive-patched-source.bbclass | 65 --- > >>>> meta/classes/archiver.bbclass | 723 ++++++++----------------- > >>>> meta/classes/copyleft_compliance.bbclass | 55 +- > >>>> meta/classes/package_rpm.bbclass | 31 +- > >>>> 7 files changed, 316 insertions(+), 760 deletions(-) > >>>> delete mode 100644 meta/classes/archive-configured-source.bbclass > >>>> delete mode 100644 meta/classes/archive-original-source.bbclass > >>>> delete mode 100644 meta/classes/archive-patched-source.bbclass > >>>> > >>>> -- > >>>> 1.8.3.1 > >>>> > >>>> _______________________________________________ > >>>> Openembedded-core mailing list > >>>> Openembedded-core@lists.openembedded.org > >>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core > >>> > > -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-18 12:26 ` Martin Jansa @ 2014-01-21 10:40 ` Robert Yang 2014-01-21 10:41 ` Robert Yang 1 sibling, 0 replies; 24+ messages in thread From: Robert Yang @ 2014-01-21 10:40 UTC (permalink / raw) To: Martin Jansa; +Cc: openembedded-core Hi Mark, I've tested the archiver.bbclass with rm_work.bbclasss again, it works for me, it seems that you only downloaded the archiver.bbclass was not enough, would you please try this PULL? It is really on oe-core-contrib now:-) And if the error occurs, would you please show the configuration and error logs ? git://git.openembedded.org/openembedded-core-contrib rbt/archiver // Robert On 01/18/2014 08:26 PM, Martin Jansa wrote: > On Thu, Jan 16, 2014 at 05:33:20PM +0800, Robert Yang wrote: >> >> >> On 01/16/2014 05:25 PM, Martin Jansa wrote: >>> On Thu, Jan 16, 2014 at 10:14:30AM +0800, Robert Yang wrote: >>>> >>>> >>>> On 01/16/2014 12:55 AM, Martin Jansa wrote: >>>>> On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: >>>>>> * The archive*.bbclass didn't work, and there were a few problems, for >>>>>> example: >>>>>> 1) There were a few duplicated code >>>>>> 2) There was no src_dir.org (or orig), but the diff command still use >>>>>> it, and it is not easy to fix this issue if we don't change a lot >>>>>> of the code. >>>>>> 3) It didn't archive the source for the native or gcc >>>>>> 4) The work flow is not very well >>>>>> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed >>>>>> 6) And others ... >>>>>> >>>>>> * So that we have to refactor it, the benefits are: >>>>>> 1) Fix the problems and make it work well. >>>>>> 2) Reduce more than 400 lines in total. >>>>>> 3) Make it easy to use. >>>>>> >>>>>> // Robert >>>>>> >>>>>> The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: >>>>>> >>>>>> bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) >>>>>> >>>>>> are available in the git repository at: >>>>>> >>>>>> git://git.pokylinux.org/poky-contrib rbt/archiver >>>>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver >>>>> >>>>> Can you send pull-request for oe-core? >>>>> >>>> >>>> Here it is: >>>> git://git.pokylinux.org/poky-contrib rbt/ar_oe >>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rbt/ar_oe >>> >>> This still looks like poky-contrib (so it will download whole >>> poky-contrib when I add it as remote to oe-core repo). >>> >> >> Sorry, I can't create repo on oe-contrib, seems that I lost my key. > > That's sad, you should send new one to khem to replace it, before > someone else finds your old key. > >> How about: >> >> git fetch git://git.pokylinux.org/poky-contrib rbt/ar_oe > > That still fetches whole poky repo I'm not interested in. > > So I've downloaded just archiver.bbclass manually and tested it on my > use-case, I can confirm that it doesn't remove required file anymore, > but it seems that there is something terribly broken when used in > combination with rm_work or multimachine builds. > > I haven't debugged it yet, but 2nd build (in the same directory) with > this new archiver included almost every recipe failed in do_patch task. > >>>> // Robert >>>> >>>>>> Robert Yang (5): >>>>>> classes/archive*.bbclass: remove archive-*-source.bbclass >>>>>> archiver.bbclass: refactor it >>>>>> package_rpm.bbclass: archive the source to srpm package >>>>>> archiver.bbclass: move a few code to copyleft_compliance.bbclass >>>>>> local.conf.sample.extended: update for the archiver >>>>>> >>>>>> meta-yocto/conf/local.conf.sample.extended | 72 +-- >>>>>> meta/classes/archive-configured-source.bbclass | 65 --- >>>>>> meta/classes/archive-original-source.bbclass | 65 --- >>>>>> meta/classes/archive-patched-source.bbclass | 65 --- >>>>>> meta/classes/archiver.bbclass | 723 ++++++++----------------- >>>>>> meta/classes/copyleft_compliance.bbclass | 55 +- >>>>>> meta/classes/package_rpm.bbclass | 31 +- >>>>>> 7 files changed, 316 insertions(+), 760 deletions(-) >>>>>> delete mode 100644 meta/classes/archive-configured-source.bbclass >>>>>> delete mode 100644 meta/classes/archive-original-source.bbclass >>>>>> delete mode 100644 meta/classes/archive-patched-source.bbclass >>>>>> >>>>>> -- >>>>>> 1.8.3.1 >>>>>> >>>>>> _______________________________________________ >>>>>> Openembedded-core mailing list >>>>>> Openembedded-core@lists.openembedded.org >>>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core >>>>> >>> > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-18 12:26 ` Martin Jansa 2014-01-21 10:40 ` Robert Yang @ 2014-01-21 10:41 ` Robert Yang 2014-01-21 12:37 ` Martin Jansa 1 sibling, 1 reply; 24+ messages in thread From: Robert Yang @ 2014-01-21 10:41 UTC (permalink / raw) To: Martin Jansa; +Cc: openembedded-core Hi Martin, I've tested the archiver.bbclass with rm_work.bbclasss again, it works for me, it seems that you only downloaded the archiver.bbclass was not enough, would you please try this PULL? It is based on oe-core and on oe-core-contrib now:-) And if the error occurs, would you please show the configuration and error logs ? git://git.openembedded.org/openembedded-core-contrib rbt/archiver // Robert On 01/18/2014 08:26 PM, Martin Jansa wrote: > On Thu, Jan 16, 2014 at 05:33:20PM +0800, Robert Yang wrote: >> >> >> On 01/16/2014 05:25 PM, Martin Jansa wrote: >>> On Thu, Jan 16, 2014 at 10:14:30AM +0800, Robert Yang wrote: >>>> >>>> >>>> On 01/16/2014 12:55 AM, Martin Jansa wrote: >>>>> On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: >>>>>> * The archive*.bbclass didn't work, and there were a few problems, for >>>>>> example: >>>>>> 1) There were a few duplicated code >>>>>> 2) There was no src_dir.org (or orig), but the diff command still use >>>>>> it, and it is not easy to fix this issue if we don't change a lot >>>>>> of the code. >>>>>> 3) It didn't archive the source for the native or gcc >>>>>> 4) The work flow is not very well >>>>>> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed >>>>>> 6) And others ... >>>>>> >>>>>> * So that we have to refactor it, the benefits are: >>>>>> 1) Fix the problems and make it work well. >>>>>> 2) Reduce more than 400 lines in total. >>>>>> 3) Make it easy to use. >>>>>> >>>>>> // Robert >>>>>> >>>>>> The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: >>>>>> >>>>>> bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) >>>>>> >>>>>> are available in the git repository at: >>>>>> >>>>>> git://git.pokylinux.org/poky-contrib rbt/archiver >>>>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver >>>>> >>>>> Can you send pull-request for oe-core? >>>>> >>>> >>>> Here it is: >>>> git://git.pokylinux.org/poky-contrib rbt/ar_oe >>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rbt/ar_oe >>> >>> This still looks like poky-contrib (so it will download whole >>> poky-contrib when I add it as remote to oe-core repo). >>> >> >> Sorry, I can't create repo on oe-contrib, seems that I lost my key. > > That's sad, you should send new one to khem to replace it, before > someone else finds your old key. > >> How about: >> >> git fetch git://git.pokylinux.org/poky-contrib rbt/ar_oe > > That still fetches whole poky repo I'm not interested in. > > So I've downloaded just archiver.bbclass manually and tested it on my > use-case, I can confirm that it doesn't remove required file anymore, > but it seems that there is something terribly broken when used in > combination with rm_work or multimachine builds. > > I haven't debugged it yet, but 2nd build (in the same directory) with > this new archiver included almost every recipe failed in do_patch task. > >>>> // Robert >>>> >>>>>> Robert Yang (5): >>>>>> classes/archive*.bbclass: remove archive-*-source.bbclass >>>>>> archiver.bbclass: refactor it >>>>>> package_rpm.bbclass: archive the source to srpm package >>>>>> archiver.bbclass: move a few code to copyleft_compliance.bbclass >>>>>> local.conf.sample.extended: update for the archiver >>>>>> >>>>>> meta-yocto/conf/local.conf.sample.extended | 72 +-- >>>>>> meta/classes/archive-configured-source.bbclass | 65 --- >>>>>> meta/classes/archive-original-source.bbclass | 65 --- >>>>>> meta/classes/archive-patched-source.bbclass | 65 --- >>>>>> meta/classes/archiver.bbclass | 723 ++++++++----------------- >>>>>> meta/classes/copyleft_compliance.bbclass | 55 +- >>>>>> meta/classes/package_rpm.bbclass | 31 +- >>>>>> 7 files changed, 316 insertions(+), 760 deletions(-) >>>>>> delete mode 100644 meta/classes/archive-configured-source.bbclass >>>>>> delete mode 100644 meta/classes/archive-original-source.bbclass >>>>>> delete mode 100644 meta/classes/archive-patched-source.bbclass >>>>>> >>>>>> -- >>>>>> 1.8.3.1 >>>>>> >>>>>> _______________________________________________ >>>>>> Openembedded-core mailing list >>>>>> Openembedded-core@lists.openembedded.org >>>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core >>>>> >>> > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-21 10:41 ` Robert Yang @ 2014-01-21 12:37 ` Martin Jansa 2014-01-22 2:59 ` Robert Yang 0 siblings, 1 reply; 24+ messages in thread From: Martin Jansa @ 2014-01-21 12:37 UTC (permalink / raw) To: Robert Yang; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 7480 bytes --] On Tue, Jan 21, 2014 at 06:41:04PM +0800, Robert Yang wrote: > > Hi Martin, > > I've tested the archiver.bbclass with rm_work.bbclasss again, it works for me, > it seems that you only downloaded the archiver.bbclass was not enough, would > you please try this PULL? It is based on oe-core and on oe-core-contrib now:-) > And if the error occurs, would you please show the configuration and error > logs ? I've tried to enable "everything" to test it properly, so my config looks like this (BTW: it would be nice to show in local.conf.sample what are the default values): # Create source archive for complience with the licenses # Enable all to properly test newer archiver backported from oe-core/master INHERIT_DISTRO += "archiver" ARCHIVER_MODE[src] = "prepatch" ARCHIVER_MODE[diff] = "1" ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches" ARCHIVER_MODE[dumpdata] = "1" ARCHIVER_MODE[recipe] = "1" ARCHIVER_MODE[log] = "1" And error in do_patch: NOTE: recipe perf-1.0-r8: task do_patch: Started ERROR: Error executing a python function in /OE/openembedded-core/meta/recipes-kernel/perf/perf.bb: OSError: [Errno 2] No such file or directory: '/OE/tmp-eglibc/sysroots/qemux86/usr/src' ERROR: The stack trace of python calls that resulted in this exception/failure was: ERROR: File "do_ar_prepatch", line 6, in <module> ERROR: ERROR: File "do_ar_prepatch", line 3, in do_ar_prepatch ERROR: ERROR: File "archiver.bbclass", line 17, in create_tarball ERROR: ERROR: The code that was being executed was: ERROR: 0002:def do_ar_prepatch(d): ERROR: 0003: create_tarball(d, d.getVar('S', True), 'prepatch') ERROR: 0004: ERROR: 0005: ERROR: *** 0006:do_ar_prepatch(d) ERROR: 0007: ERROR: [From file: 'do_ar_prepatch', lineno: 6, function: <module>] ERROR: 0001: ERROR: 0002:def do_ar_prepatch(d): ERROR: *** 0003: create_tarball(d, d.getVar('S', True), 'prepatch') ERROR: 0004: ERROR: 0005: ERROR: 0006:do_ar_prepatch(d) ERROR: 0007: ERROR: [From file: 'do_ar_prepatch', lineno: 3, function: do_ar_prepatch] ERROR: 0013: srcdir = srcdir.rstrip('/') ERROR: 0014: dirname = os.path.dirname(srcdir) ERROR: 0015: basename = os.path.basename(srcdir) ERROR: 0016: olddir = os.getcwd() ERROR: *** 0017: os.chdir(dirname) ERROR: 0018: bb.note("Creating %s" % tarname) ERROR: 0019: if (len(os.listdir(basename))) != 0: ERROR: 0020: tar = tarfile.open(tarname, "w:gz") ERROR: 0021: tar.add(basename) ERROR: [From file: 'archiver.bbclass', lineno: 17, function: create_tarball] ERROR: Function failed: do_ar_prepatch ERROR: Logfile of failure stored in: /OE/tmp-eglibc/work/i586-oe-linux/perf/1.0-r8/temp/log.do_patch.28364 NOTE: recipe perf-1.0-r8: task do_patch: Failed ERROR: Task 20159 (/OE/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_patch) failed with exit code '1' > git://git.openembedded.org/openembedded-core-contrib rbt/archiver > > // Robert > > On 01/18/2014 08:26 PM, Martin Jansa wrote: > > On Thu, Jan 16, 2014 at 05:33:20PM +0800, Robert Yang wrote: > >> > >> > >> On 01/16/2014 05:25 PM, Martin Jansa wrote: > >>> On Thu, Jan 16, 2014 at 10:14:30AM +0800, Robert Yang wrote: > >>>> > >>>> > >>>> On 01/16/2014 12:55 AM, Martin Jansa wrote: > >>>>> On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: > >>>>>> * The archive*.bbclass didn't work, and there were a few problems, for > >>>>>> example: > >>>>>> 1) There were a few duplicated code > >>>>>> 2) There was no src_dir.org (or orig), but the diff command still use > >>>>>> it, and it is not easy to fix this issue if we don't change a lot > >>>>>> of the code. > >>>>>> 3) It didn't archive the source for the native or gcc > >>>>>> 4) The work flow is not very well > >>>>>> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed > >>>>>> 6) And others ... > >>>>>> > >>>>>> * So that we have to refactor it, the benefits are: > >>>>>> 1) Fix the problems and make it work well. > >>>>>> 2) Reduce more than 400 lines in total. > >>>>>> 3) Make it easy to use. > >>>>>> > >>>>>> // Robert > >>>>>> > >>>>>> The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: > >>>>>> > >>>>>> bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) > >>>>>> > >>>>>> are available in the git repository at: > >>>>>> > >>>>>> git://git.pokylinux.org/poky-contrib rbt/archiver > >>>>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver > >>>>> > >>>>> Can you send pull-request for oe-core? > >>>>> > >>>> > >>>> Here it is: > >>>> git://git.pokylinux.org/poky-contrib rbt/ar_oe > >>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rbt/ar_oe > >>> > >>> This still looks like poky-contrib (so it will download whole > >>> poky-contrib when I add it as remote to oe-core repo). > >>> > >> > >> Sorry, I can't create repo on oe-contrib, seems that I lost my key. > > > > That's sad, you should send new one to khem to replace it, before > > someone else finds your old key. > > > >> How about: > >> > >> git fetch git://git.pokylinux.org/poky-contrib rbt/ar_oe > > > > That still fetches whole poky repo I'm not interested in. > > > > So I've downloaded just archiver.bbclass manually and tested it on my > > use-case, I can confirm that it doesn't remove required file anymore, > > but it seems that there is something terribly broken when used in > > combination with rm_work or multimachine builds. > > > > I haven't debugged it yet, but 2nd build (in the same directory) with > > this new archiver included almost every recipe failed in do_patch task. > > > >>>> // Robert > >>>> > >>>>>> Robert Yang (5): > >>>>>> classes/archive*.bbclass: remove archive-*-source.bbclass > >>>>>> archiver.bbclass: refactor it > >>>>>> package_rpm.bbclass: archive the source to srpm package > >>>>>> archiver.bbclass: move a few code to copyleft_compliance.bbclass > >>>>>> local.conf.sample.extended: update for the archiver > >>>>>> > >>>>>> meta-yocto/conf/local.conf.sample.extended | 72 +-- > >>>>>> meta/classes/archive-configured-source.bbclass | 65 --- > >>>>>> meta/classes/archive-original-source.bbclass | 65 --- > >>>>>> meta/classes/archive-patched-source.bbclass | 65 --- > >>>>>> meta/classes/archiver.bbclass | 723 ++++++++----------------- > >>>>>> meta/classes/copyleft_compliance.bbclass | 55 +- > >>>>>> meta/classes/package_rpm.bbclass | 31 +- > >>>>>> 7 files changed, 316 insertions(+), 760 deletions(-) > >>>>>> delete mode 100644 meta/classes/archive-configured-source.bbclass > >>>>>> delete mode 100644 meta/classes/archive-original-source.bbclass > >>>>>> delete mode 100644 meta/classes/archive-patched-source.bbclass > >>>>>> > >>>>>> -- > >>>>>> 1.8.3.1 > >>>>>> > >>>>>> _______________________________________________ > >>>>>> Openembedded-core mailing list > >>>>>> Openembedded-core@lists.openembedded.org > >>>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core > >>>>> > >>> > > -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-21 12:37 ` Martin Jansa @ 2014-01-22 2:59 ` Robert Yang 2014-01-22 10:13 ` Robert Yang 0 siblings, 1 reply; 24+ messages in thread From: Robert Yang @ 2014-01-22 2:59 UTC (permalink / raw) To: Martin Jansa Cc: Purdie, Richard, Wold, Saul, Paul Eggleton, openembedded-core On 01/21/2014 08:37 PM, Martin Jansa wrote: > On Tue, Jan 21, 2014 at 06:41:04PM +0800, Robert Yang wrote: >> >> Hi Martin, >> >> I've tested the archiver.bbclass with rm_work.bbclasss again, it works for me, >> it seems that you only downloaded the archiver.bbclass was not enough, would >> you please try this PULL? It is based on oe-core and on oe-core-contrib now:-) >> And if the error occurs, would you please show the configuration and error >> logs ? > > I've tried to enable "everything" to test it properly, so my config > looks like this (BTW: it would be nice to show in local.conf.sample > what are the default values): > > # Create source archive for complience with the licenses > # Enable all to properly test newer archiver backported from oe-core/master > INHERIT_DISTRO += "archiver" I changed it to INHERIT += "archiver" since the INHERIT_DISTRO will override this: meta/conf/distro/defaultsetup.conf:INHERIT_DISTRO ?= "debian devshell sstate license" Everything works fine to me, I'm afraid that set INHERIT_DISTRO += "archiver" may override other INHERIT_DISTRO's settings. I also added INHERIT += "rm_work" And please see my comments below. > ARCHIVER_MODE[src] = "prepatch" > ARCHIVER_MODE[diff] = "1" > ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches" > ARCHIVER_MODE[dumpdata] = "1" > ARCHIVER_MODE[recipe] = "1" > ARCHIVER_MODE[log] = "1" > > And error in do_patch: > > NOTE: recipe perf-1.0-r8: task do_patch: Started > ERROR: Error executing a python function in /OE/openembedded-core/meta/recipes-kernel/perf/perf.bb: > OSError: [Errno 2] No such file or directory: '/OE/tmp-eglibc/sysroots/qemux86/usr/src' The perf's ${S} is sysroots/qemux86/usr/src/kernel, but the problem is that why the /OE/tmp-eglibc/sysroots/qemux86/usr/src doesn't exist ? The rm_work doesn't remove it, no matter how many times I tried, it still works. So, please: 1) Check whether your other INHERIT_DISTRO settings have been overrided or not, and try to use INHERIT rather than INHERIT_DISTRO ? 2) If it still fails, would you please provide more info ? For example, how to reproduce it, exactly ? 3) It would be great if we can talk oneline, how about gtalk ? I will try to reach you there. 4) It would be great if others can help me to test it. // Robert > > ERROR: The stack trace of python calls that resulted in this exception/failure was: > ERROR: File "do_ar_prepatch", line 6, in <module> > ERROR: > ERROR: File "do_ar_prepatch", line 3, in do_ar_prepatch > ERROR: > ERROR: File "archiver.bbclass", line 17, in create_tarball > ERROR: > ERROR: The code that was being executed was: > ERROR: 0002:def do_ar_prepatch(d): > ERROR: 0003: create_tarball(d, d.getVar('S', True), 'prepatch') > ERROR: 0004: > ERROR: 0005: > ERROR: *** 0006:do_ar_prepatch(d) > ERROR: 0007: > ERROR: [From file: 'do_ar_prepatch', lineno: 6, function: <module>] > ERROR: 0001: > ERROR: 0002:def do_ar_prepatch(d): > ERROR: *** 0003: create_tarball(d, d.getVar('S', True), 'prepatch') > ERROR: 0004: > ERROR: 0005: > ERROR: 0006:do_ar_prepatch(d) > ERROR: 0007: > ERROR: [From file: 'do_ar_prepatch', lineno: 3, function: do_ar_prepatch] > ERROR: 0013: srcdir = srcdir.rstrip('/') > ERROR: 0014: dirname = os.path.dirname(srcdir) > ERROR: 0015: basename = os.path.basename(srcdir) > ERROR: 0016: olddir = os.getcwd() > ERROR: *** 0017: os.chdir(dirname) > ERROR: 0018: bb.note("Creating %s" % tarname) > ERROR: 0019: if (len(os.listdir(basename))) != 0: > ERROR: 0020: tar = tarfile.open(tarname, "w:gz") > ERROR: 0021: tar.add(basename) > ERROR: [From file: 'archiver.bbclass', lineno: 17, function: create_tarball] > ERROR: Function failed: do_ar_prepatch > ERROR: Logfile of failure stored in: /OE/tmp-eglibc/work/i586-oe-linux/perf/1.0-r8/temp/log.do_patch.28364 > NOTE: recipe perf-1.0-r8: task do_patch: Failed > ERROR: Task 20159 (/OE/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_patch) failed with exit code '1' > >> git://git.openembedded.org/openembedded-core-contrib rbt/archiver >> >> // Robert >> >> On 01/18/2014 08:26 PM, Martin Jansa wrote: >>> On Thu, Jan 16, 2014 at 05:33:20PM +0800, Robert Yang wrote: >>>> >>>> >>>> On 01/16/2014 05:25 PM, Martin Jansa wrote: >>>>> On Thu, Jan 16, 2014 at 10:14:30AM +0800, Robert Yang wrote: >>>>>> >>>>>> >>>>>> On 01/16/2014 12:55 AM, Martin Jansa wrote: >>>>>>> On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: >>>>>>>> * The archive*.bbclass didn't work, and there were a few problems, for >>>>>>>> example: >>>>>>>> 1) There were a few duplicated code >>>>>>>> 2) There was no src_dir.org (or orig), but the diff command still use >>>>>>>> it, and it is not easy to fix this issue if we don't change a lot >>>>>>>> of the code. >>>>>>>> 3) It didn't archive the source for the native or gcc >>>>>>>> 4) The work flow is not very well >>>>>>>> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed >>>>>>>> 6) And others ... >>>>>>>> >>>>>>>> * So that we have to refactor it, the benefits are: >>>>>>>> 1) Fix the problems and make it work well. >>>>>>>> 2) Reduce more than 400 lines in total. >>>>>>>> 3) Make it easy to use. >>>>>>>> >>>>>>>> // Robert >>>>>>>> >>>>>>>> The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: >>>>>>>> >>>>>>>> bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) >>>>>>>> >>>>>>>> are available in the git repository at: >>>>>>>> >>>>>>>> git://git.pokylinux.org/poky-contrib rbt/archiver >>>>>>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver >>>>>>> >>>>>>> Can you send pull-request for oe-core? >>>>>>> >>>>>> >>>>>> Here it is: >>>>>> git://git.pokylinux.org/poky-contrib rbt/ar_oe >>>>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rbt/ar_oe >>>>> >>>>> This still looks like poky-contrib (so it will download whole >>>>> poky-contrib when I add it as remote to oe-core repo). >>>>> >>>> >>>> Sorry, I can't create repo on oe-contrib, seems that I lost my key. >>> >>> That's sad, you should send new one to khem to replace it, before >>> someone else finds your old key. >>> >>>> How about: >>>> >>>> git fetch git://git.pokylinux.org/poky-contrib rbt/ar_oe >>> >>> That still fetches whole poky repo I'm not interested in. >>> >>> So I've downloaded just archiver.bbclass manually and tested it on my >>> use-case, I can confirm that it doesn't remove required file anymore, >>> but it seems that there is something terribly broken when used in >>> combination with rm_work or multimachine builds. >>> >>> I haven't debugged it yet, but 2nd build (in the same directory) with >>> this new archiver included almost every recipe failed in do_patch task. >>> >>>>>> // Robert >>>>>> >>>>>>>> Robert Yang (5): >>>>>>>> classes/archive*.bbclass: remove archive-*-source.bbclass >>>>>>>> archiver.bbclass: refactor it >>>>>>>> package_rpm.bbclass: archive the source to srpm package >>>>>>>> archiver.bbclass: move a few code to copyleft_compliance.bbclass >>>>>>>> local.conf.sample.extended: update for the archiver >>>>>>>> >>>>>>>> meta-yocto/conf/local.conf.sample.extended | 72 +-- >>>>>>>> meta/classes/archive-configured-source.bbclass | 65 --- >>>>>>>> meta/classes/archive-original-source.bbclass | 65 --- >>>>>>>> meta/classes/archive-patched-source.bbclass | 65 --- >>>>>>>> meta/classes/archiver.bbclass | 723 ++++++++----------------- >>>>>>>> meta/classes/copyleft_compliance.bbclass | 55 +- >>>>>>>> meta/classes/package_rpm.bbclass | 31 +- >>>>>>>> 7 files changed, 316 insertions(+), 760 deletions(-) >>>>>>>> delete mode 100644 meta/classes/archive-configured-source.bbclass >>>>>>>> delete mode 100644 meta/classes/archive-original-source.bbclass >>>>>>>> delete mode 100644 meta/classes/archive-patched-source.bbclass >>>>>>>> >>>>>>>> -- >>>>>>>> 1.8.3.1 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Openembedded-core mailing list >>>>>>>> Openembedded-core@lists.openembedded.org >>>>>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core >>>>>>> >>>>> >>> > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-22 2:59 ` Robert Yang @ 2014-01-22 10:13 ` Robert Yang 2014-01-31 15:57 ` Martin Jansa 0 siblings, 1 reply; 24+ messages in thread From: Robert Yang @ 2014-01-22 10:13 UTC (permalink / raw) To: Purdie, Richard; +Cc: Paul Eggleton, openembedded-core, Wold, Saul On 01/22/2014 10:59 AM, Robert Yang wrote: > > > On 01/21/2014 08:37 PM, Martin Jansa wrote: >> On Tue, Jan 21, 2014 at 06:41:04PM +0800, Robert Yang wrote: >>> >>> Hi Martin, >>> I've talked with Martin, the difference is that he tried it on dylan, this is only for master, I'd like to backport if for dora, we may also backport it for dylan, but this may need more discussions. // Robert >>> I've tested the archiver.bbclass with rm_work.bbclasss again, it works for me, >>> it seems that you only downloaded the archiver.bbclass was not enough, would >>> you please try this PULL? It is based on oe-core and on oe-core-contrib now:-) >>> And if the error occurs, would you please show the configuration and error >>> logs ? >> >> I've tried to enable "everything" to test it properly, so my config >> looks like this (BTW: it would be nice to show in local.conf.sample >> what are the default values): >> >> # Create source archive for complience with the licenses >> # Enable all to properly test newer archiver backported from oe-core/master >> INHERIT_DISTRO += "archiver" > > I changed it to INHERIT += "archiver" since the INHERIT_DISTRO will override > this: > > meta/conf/distro/defaultsetup.conf:INHERIT_DISTRO ?= "debian devshell sstate > license" > > Everything works fine to me, I'm afraid that set INHERIT_DISTRO += "archiver" > may override other INHERIT_DISTRO's settings. > > I also added INHERIT += "rm_work" > > And please see my comments below. > >> ARCHIVER_MODE[src] = "prepatch" >> ARCHIVER_MODE[diff] = "1" >> ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches" >> ARCHIVER_MODE[dumpdata] = "1" >> ARCHIVER_MODE[recipe] = "1" >> ARCHIVER_MODE[log] = "1" >> >> And error in do_patch: >> >> NOTE: recipe perf-1.0-r8: task do_patch: Started >> ERROR: Error executing a python function in >> /OE/openembedded-core/meta/recipes-kernel/perf/perf.bb: >> OSError: [Errno 2] No such file or directory: >> '/OE/tmp-eglibc/sysroots/qemux86/usr/src' > > The perf's ${S} is sysroots/qemux86/usr/src/kernel, but the problem is that why > the /OE/tmp-eglibc/sysroots/qemux86/usr/src doesn't exist ? The rm_work doesn't > remove it, no matter how many times I tried, it still works. > > So, please: > > 1) Check whether your other INHERIT_DISTRO settings have been overrided or not, > and try to use INHERIT rather than INHERIT_DISTRO ? > > 2) If it still fails, would you please provide more info ? For example, how to > reproduce it, exactly ? > > 3) It would be great if we can talk oneline, how about gtalk ? I will try to > reach you there. > > 4) It would be great if others can help me to test it. > > // Robert > >> >> ERROR: The stack trace of python calls that resulted in this exception/failure >> was: >> ERROR: File "do_ar_prepatch", line 6, in <module> >> ERROR: >> ERROR: File "do_ar_prepatch", line 3, in do_ar_prepatch >> ERROR: >> ERROR: File "archiver.bbclass", line 17, in create_tarball >> ERROR: >> ERROR: The code that was being executed was: >> ERROR: 0002:def do_ar_prepatch(d): >> ERROR: 0003: create_tarball(d, d.getVar('S', True), 'prepatch') >> ERROR: 0004: >> ERROR: 0005: >> ERROR: *** 0006:do_ar_prepatch(d) >> ERROR: 0007: >> ERROR: [From file: 'do_ar_prepatch', lineno: 6, function: <module>] >> ERROR: 0001: >> ERROR: 0002:def do_ar_prepatch(d): >> ERROR: *** 0003: create_tarball(d, d.getVar('S', True), 'prepatch') >> ERROR: 0004: >> ERROR: 0005: >> ERROR: 0006:do_ar_prepatch(d) >> ERROR: 0007: >> ERROR: [From file: 'do_ar_prepatch', lineno: 3, function: do_ar_prepatch] >> ERROR: 0013: srcdir = srcdir.rstrip('/') >> ERROR: 0014: dirname = os.path.dirname(srcdir) >> ERROR: 0015: basename = os.path.basename(srcdir) >> ERROR: 0016: olddir = os.getcwd() >> ERROR: *** 0017: os.chdir(dirname) >> ERROR: 0018: bb.note("Creating %s" % tarname) >> ERROR: 0019: if (len(os.listdir(basename))) != 0: >> ERROR: 0020: tar = tarfile.open(tarname, "w:gz") >> ERROR: 0021: tar.add(basename) >> ERROR: [From file: 'archiver.bbclass', lineno: 17, function: create_tarball] >> ERROR: Function failed: do_ar_prepatch >> ERROR: Logfile of failure stored in: >> /OE/tmp-eglibc/work/i586-oe-linux/perf/1.0-r8/temp/log.do_patch.28364 >> NOTE: recipe perf-1.0-r8: task do_patch: Failed >> ERROR: Task 20159 (/OE/openembedded-core/meta/recipes-kernel/perf/perf.bb, >> do_patch) failed with exit code '1' >> >>> git://git.openembedded.org/openembedded-core-contrib rbt/archiver >>> >>> // Robert >>> >>> On 01/18/2014 08:26 PM, Martin Jansa wrote: >>>> On Thu, Jan 16, 2014 at 05:33:20PM +0800, Robert Yang wrote: >>>>> >>>>> >>>>> On 01/16/2014 05:25 PM, Martin Jansa wrote: >>>>>> On Thu, Jan 16, 2014 at 10:14:30AM +0800, Robert Yang wrote: >>>>>>> >>>>>>> >>>>>>> On 01/16/2014 12:55 AM, Martin Jansa wrote: >>>>>>>> On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: >>>>>>>>> * The archive*.bbclass didn't work, and there were a few problems, for >>>>>>>>> example: >>>>>>>>> 1) There were a few duplicated code >>>>>>>>> 2) There was no src_dir.org (or orig), but the diff command still >>>>>>>>> use >>>>>>>>> it, and it is not easy to fix this issue if we don't change a lot >>>>>>>>> of the code. >>>>>>>>> 3) It didn't archive the source for the native or gcc >>>>>>>>> 4) The work flow is not very well >>>>>>>>> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed >>>>>>>>> 6) And others ... >>>>>>>>> >>>>>>>>> * So that we have to refactor it, the benefits are: >>>>>>>>> 1) Fix the problems and make it work well. >>>>>>>>> 2) Reduce more than 400 lines in total. >>>>>>>>> 3) Make it easy to use. >>>>>>>>> >>>>>>>>> // Robert >>>>>>>>> >>>>>>>>> The following changes since commit >>>>>>>>> 1b636173ca88e5ccca1992f9a12367a1189fa674: >>>>>>>>> >>>>>>>>> bitbake: toaster: Toaster GUI, generic search, filter and order >>>>>>>>> (2014-01-10 15:20:26 +0000) >>>>>>>>> >>>>>>>>> are available in the git repository at: >>>>>>>>> >>>>>>>>> git://git.pokylinux.org/poky-contrib rbt/archiver >>>>>>>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver >>>>>>>> >>>>>>>> Can you send pull-request for oe-core? >>>>>>>> >>>>>>> >>>>>>> Here it is: >>>>>>> git://git.pokylinux.org/poky-contrib rbt/ar_oe >>>>>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rbt/ar_oe >>>>>> >>>>>> This still looks like poky-contrib (so it will download whole >>>>>> poky-contrib when I add it as remote to oe-core repo). >>>>>> >>>>> >>>>> Sorry, I can't create repo on oe-contrib, seems that I lost my key. >>>> >>>> That's sad, you should send new one to khem to replace it, before >>>> someone else finds your old key. >>>> >>>>> How about: >>>>> >>>>> git fetch git://git.pokylinux.org/poky-contrib rbt/ar_oe >>>> >>>> That still fetches whole poky repo I'm not interested in. >>>> >>>> So I've downloaded just archiver.bbclass manually and tested it on my >>>> use-case, I can confirm that it doesn't remove required file anymore, >>>> but it seems that there is something terribly broken when used in >>>> combination with rm_work or multimachine builds. >>>> >>>> I haven't debugged it yet, but 2nd build (in the same directory) with >>>> this new archiver included almost every recipe failed in do_patch task. >>>> >>>>>>> // Robert >>>>>>> >>>>>>>>> Robert Yang (5): >>>>>>>>> classes/archive*.bbclass: remove archive-*-source.bbclass >>>>>>>>> archiver.bbclass: refactor it >>>>>>>>> package_rpm.bbclass: archive the source to srpm package >>>>>>>>> archiver.bbclass: move a few code to copyleft_compliance.bbclass >>>>>>>>> local.conf.sample.extended: update for the archiver >>>>>>>>> >>>>>>>>> meta-yocto/conf/local.conf.sample.extended | 72 +-- >>>>>>>>> meta/classes/archive-configured-source.bbclass | 65 --- >>>>>>>>> meta/classes/archive-original-source.bbclass | 65 --- >>>>>>>>> meta/classes/archive-patched-source.bbclass | 65 --- >>>>>>>>> meta/classes/archiver.bbclass | 723 >>>>>>>>> ++++++++----------------- >>>>>>>>> meta/classes/copyleft_compliance.bbclass | 55 +- >>>>>>>>> meta/classes/package_rpm.bbclass | 31 +- >>>>>>>>> 7 files changed, 316 insertions(+), 760 deletions(-) >>>>>>>>> delete mode 100644 meta/classes/archive-configured-source.bbclass >>>>>>>>> delete mode 100644 meta/classes/archive-original-source.bbclass >>>>>>>>> delete mode 100644 meta/classes/archive-patched-source.bbclass >>>>>>>>> >>>>>>>>> -- >>>>>>>>> 1.8.3.1 >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Openembedded-core mailing list >>>>>>>>> Openembedded-core@lists.openembedded.org >>>>>>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core >>>>>>>> >>>>>> >>>> >> > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core > > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-22 10:13 ` Robert Yang @ 2014-01-31 15:57 ` Martin Jansa 0 siblings, 0 replies; 24+ messages in thread From: Martin Jansa @ 2014-01-31 15:57 UTC (permalink / raw) To: Robert Yang; +Cc: Purdie, Richard, openembedded-core, Paul Eggleton, Wold, Saul [-- Attachment #1: Type: text/plain, Size: 17858 bytes --] On Wed, Jan 22, 2014 at 06:13:37PM +0800, Robert Yang wrote: > > On 01/22/2014 10:59 AM, Robert Yang wrote: > > > > > > On 01/21/2014 08:37 PM, Martin Jansa wrote: > >> On Tue, Jan 21, 2014 at 06:41:04PM +0800, Robert Yang wrote: > >>> > >>> Hi Martin, > >>> > > I've talked with Martin, the difference is that he tried it on dylan, > this is only for master, I'd like to backport if for dora, we may also > backport it for dylan, but this may need more discussions. Have you tried multi-machine build as suggested? In master I see the same behavior, these tasks are reexecuted after MACHINE switch causing not only rebuild of everything (probably caused by combination with rm_work), but also warnings like this: NOTE: recipe gcc-cross-4.8.2-r0: task do_deploy_archives: Started WARNING: The recipe gcc-cross is trying to install files into a shared area when those files already exist. Those files and their manifest location are: /OE/build/oe-core/tmp-eglibc/deploy/sources/i586-oe-linux/gcc-cross-4.8.2-r0/gcc-cross-4.8.2-r0-log.tar.gz Matched in manifest-x86_64_qemux86-gcc-cross.deploy_archives /OE/build/oe-core/tmp-eglibc/deploy/sources/i586-oe-linux/gcc-cross-4.8.2-r0/gcc-cross-4.8.2-r0-showdata.dump Matched in manifest-x86_64_qemux86-gcc-cross.deploy_archives /OE/build/oe-core/tmp-eglibc/deploy/sources/i586-oe-linux/gcc-cross-4.8.2-r0/gcc-cross-4.8.2-r0-recipe.tar.gz Matched in manifest-x86_64_qemux86-gcc-cross.deploy_archives Please verify which package should provide the above files. NOTE: recipe gcc-cross-4.8.2-r0: task do_deploy_archives: Succeeded and in some cases enabling archiver (or changing its configuration) causes not only archiver-related tasks to be executed (do_dumpdata, do_ar_recipe, do_ar_log, do_deploy_archives) but triggers also another do_patch task in already patched ${S} (probably because of changed prefunc signature), causing do_patch errors. (I haven't found simple reproducer, maybe its related to cache as well, because repeating the failing setup in the same tmpdir doesn't work - but also doesn't work when I delete cache and tmp-eglibc/cache between steps). Interestingly bitbake -S doesn't show different signatures when compared with: oe-core/scripts/sstate-diff-machines.sh --tmpdir=tmp-eglibc --targets="libffi" --machines="qemux86 qemux86copy" you can cherry-pick qemux86copy test MACHINE from: http://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/tune2-test I'm testing it on libffi recipe, because it has extra patch in SRC_URI and exists also in -native version which is even faster to build. OE qemux86@ ~/build/oe-core $ bitbake libffi | tee log.qemux86.prepatch NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 47743, PID: 8217 Parsing recipes...WARNING: Unable to get checksum for cogl-1.0 SRC_URI entry kms-include-stddef.h-before-drm.h.patch: file could not be found done. Parsing of 847 .bb files complete (0 cached, 847 parsed). 1203 targets, 35 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.21.1" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "Gentoo-2.2" TARGET_SYS = "i586-oe-linux" MACHINE = "qemux86" DISTRO = "nodistro" DISTRO_VERSION = "nodistro.0" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta = "jansa/test:9d7f81bc7257b9e5f902daa11ce63c84046256d2" NOTE: Preparing runqueue NOTE: Executing SetScene Tasks NOTE: Running setscene task 121 of 128 (/OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_populate_lic_setscene) NOTE: Running setscene task 122 of 128 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/libtool/libtool-cross_2.4.2.bb, do_populate_lic_setscene) NOTE: recipe libffi-3.0.13-r0: task do_populate_lic_setscene: Started NOTE: recipe libtool-cross-2.4.2-r6.1: task do_populate_lic_setscene: Started .... NOTE: Running task 725 of 728 (ID: 17, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_deploy_archives) NOTE: recipe libffi-3.0.13-r0: task do_deploy_archives: Started NOTE: recipe libffi-3.0.13-r0: task do_deploy_archives: Succeeded NOTE: Running noexec task 726 of 728 (ID: 11, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_build) NOTE: Running task 727 of 728 (ID: 12, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_rm_work) NOTE: recipe libffi-3.0.13-r0: task do_rm_work: Started NOTE: recipe libffi-3.0.13-r0: task do_rm_work: Succeeded NOTE: Running task 728 of 728 (ID: 13, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_rm_work_all) NOTE: recipe libffi-3.0.13-r0: task do_rm_work_all: Started NOTE: recipe libffi-3.0.13-r0: task do_rm_work_all: Succeeded NOTE: Tasks Summary: Attempted 728 tasks of which 74 didn't need to be rerun and all succeeded. NOTE: Writing buildhistory Summary: There was 1 WARNING message shown. OE qemux86@ ~/build/oe-core $ export MACHINE=qemux86copy OE qemux86copy@ ~/build/oe-core $ bitbake libffi | tee log.qemux86.prepatch2 NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 55093, PID: 7136 Parsing recipes...WARNING: Unable to get checksum for cogl-1.0 SRC_URI entry kms-include-stddef.h-before-drm.h.patch: file could not be found done. Parsing of 847 .bb files complete (0 cached, 847 parsed). 1203 targets, 35 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.21.1" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "Gentoo-2.2" TARGET_SYS = "i586-oe-linux" MACHINE = "qemux86copy" DISTRO = "nodistro" DISTRO_VERSION = "nodistro.0" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta = "jansa/test:9d7f81bc7257b9e5f902daa11ce63c84046256d2" NOTE: Preparing runqueue NOTE: Executing SetScene Tasks NOTE: Running setscene task 99 of 128 (/OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_populate_sysroot_setscene) NOTE: Running setscene task 100 of 128 (/OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_package_write_ipk_setscene) NOTE: recipe libffi-3.0.13-r0: task do_populate_sysroot_setscene: Started NOTE: recipe libffi-3.0.13-r0: task do_package_write_ipk_setscene: Started NOTE: recipe libffi-3.0.13-r0: task do_populate_sysroot_setscene: Succeeded NOTE: Running setscene task 101 of 128 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/libtool/libtool-cross_2.4.2.bb, do_package_write_ipk_setscene) ... NOTE: Running task 725 of 728 (ID: 17, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_deploy_archives) NOTE: recipe libffi-3.0.13-r0: task do_deploy_archives: Started NOTE: recipe libffi-3.0.13-r0: task do_deploy_archives: Succeeded NOTE: Running noexec task 726 of 728 (ID: 11, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_build) NOTE: Running task 727 of 728 (ID: 12, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_rm_work) NOTE: recipe libffi-3.0.13-r0: task do_rm_work: Started NOTE: recipe libffi-3.0.13-r0: task do_rm_work: Succeeded NOTE: Running task 728 of 728 (ID: 13, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_rm_work_all) NOTE: recipe libffi-3.0.13-r0: task do_rm_work_all: Started NOTE: recipe libffi-3.0.13-r0: task do_rm_work_all: Succeeded NOTE: Tasks Summary: Attempted 728 tasks of which 110 didn't need to be rerun and all succeeded. NOTE: Writing buildhistory Summary: There were 4 WARNING messages shown. Normally this should reuse everything from sstate and build only kernel and very few MACHINE_ARCH recipes. conf/local.conf: INHERIT += "rm_work" INHERIT += "archiver" ARCHIVER_MODE[src] = "prepatch" ARCHIVER_MODE[diff] = "1" ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches no yes" ARCHIVER_MODE[dumpdata] = "1" ARCHIVER_MODE[recipe] = "1" ARCHIVER_MODE[log] = "1" > >>> I've tested the archiver.bbclass with rm_work.bbclasss again, it works for me, > >>> it seems that you only downloaded the archiver.bbclass was not enough, would > >>> you please try this PULL? It is based on oe-core and on oe-core-contrib now:-) > >>> And if the error occurs, would you please show the configuration and error > >>> logs ? > >> > >> I've tried to enable "everything" to test it properly, so my config > >> looks like this (BTW: it would be nice to show in local.conf.sample > >> what are the default values): > >> > >> # Create source archive for complience with the licenses > >> # Enable all to properly test newer archiver backported from oe-core/master > >> INHERIT_DISTRO += "archiver" > > > > I changed it to INHERIT += "archiver" since the INHERIT_DISTRO will override > > this: > > > > meta/conf/distro/defaultsetup.conf:INHERIT_DISTRO ?= "debian devshell sstate > > license" > > > > Everything works fine to me, I'm afraid that set INHERIT_DISTRO += "archiver" > > may override other INHERIT_DISTRO's settings. > > > > I also added INHERIT += "rm_work" > > > > And please see my comments below. > > > >> ARCHIVER_MODE[src] = "prepatch" > >> ARCHIVER_MODE[diff] = "1" > >> ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches" > >> ARCHIVER_MODE[dumpdata] = "1" > >> ARCHIVER_MODE[recipe] = "1" > >> ARCHIVER_MODE[log] = "1" > >> > >> And error in do_patch: > >> > >> NOTE: recipe perf-1.0-r8: task do_patch: Started > >> ERROR: Error executing a python function in > >> /OE/openembedded-core/meta/recipes-kernel/perf/perf.bb: > >> OSError: [Errno 2] No such file or directory: > >> '/OE/tmp-eglibc/sysroots/qemux86/usr/src' > > > > The perf's ${S} is sysroots/qemux86/usr/src/kernel, but the problem is that why > > the /OE/tmp-eglibc/sysroots/qemux86/usr/src doesn't exist ? The rm_work doesn't > > remove it, no matter how many times I tried, it still works. > > > > So, please: > > > > 1) Check whether your other INHERIT_DISTRO settings have been overrided or not, > > and try to use INHERIT rather than INHERIT_DISTRO ? > > > > 2) If it still fails, would you please provide more info ? For example, how to > > reproduce it, exactly ? > > > > 3) It would be great if we can talk oneline, how about gtalk ? I will try to > > reach you there. > > > > 4) It would be great if others can help me to test it. > > > > // Robert > > > >> > >> ERROR: The stack trace of python calls that resulted in this exception/failure > >> was: > >> ERROR: File "do_ar_prepatch", line 6, in <module> > >> ERROR: > >> ERROR: File "do_ar_prepatch", line 3, in do_ar_prepatch > >> ERROR: > >> ERROR: File "archiver.bbclass", line 17, in create_tarball > >> ERROR: > >> ERROR: The code that was being executed was: > >> ERROR: 0002:def do_ar_prepatch(d): > >> ERROR: 0003: create_tarball(d, d.getVar('S', True), 'prepatch') > >> ERROR: 0004: > >> ERROR: 0005: > >> ERROR: *** 0006:do_ar_prepatch(d) > >> ERROR: 0007: > >> ERROR: [From file: 'do_ar_prepatch', lineno: 6, function: <module>] > >> ERROR: 0001: > >> ERROR: 0002:def do_ar_prepatch(d): > >> ERROR: *** 0003: create_tarball(d, d.getVar('S', True), 'prepatch') > >> ERROR: 0004: > >> ERROR: 0005: > >> ERROR: 0006:do_ar_prepatch(d) > >> ERROR: 0007: > >> ERROR: [From file: 'do_ar_prepatch', lineno: 3, function: do_ar_prepatch] > >> ERROR: 0013: srcdir = srcdir.rstrip('/') > >> ERROR: 0014: dirname = os.path.dirname(srcdir) > >> ERROR: 0015: basename = os.path.basename(srcdir) > >> ERROR: 0016: olddir = os.getcwd() > >> ERROR: *** 0017: os.chdir(dirname) > >> ERROR: 0018: bb.note("Creating %s" % tarname) > >> ERROR: 0019: if (len(os.listdir(basename))) != 0: > >> ERROR: 0020: tar = tarfile.open(tarname, "w:gz") > >> ERROR: 0021: tar.add(basename) > >> ERROR: [From file: 'archiver.bbclass', lineno: 17, function: create_tarball] > >> ERROR: Function failed: do_ar_prepatch > >> ERROR: Logfile of failure stored in: > >> /OE/tmp-eglibc/work/i586-oe-linux/perf/1.0-r8/temp/log.do_patch.28364 > >> NOTE: recipe perf-1.0-r8: task do_patch: Failed > >> ERROR: Task 20159 (/OE/openembedded-core/meta/recipes-kernel/perf/perf.bb, > >> do_patch) failed with exit code '1' > >> > >>> git://git.openembedded.org/openembedded-core-contrib rbt/archiver > >>> > >>> // Robert > >>> > >>> On 01/18/2014 08:26 PM, Martin Jansa wrote: > >>>> On Thu, Jan 16, 2014 at 05:33:20PM +0800, Robert Yang wrote: > >>>>> > >>>>> > >>>>> On 01/16/2014 05:25 PM, Martin Jansa wrote: > >>>>>> On Thu, Jan 16, 2014 at 10:14:30AM +0800, Robert Yang wrote: > >>>>>>> > >>>>>>> > >>>>>>> On 01/16/2014 12:55 AM, Martin Jansa wrote: > >>>>>>>> On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote: > >>>>>>>>> * The archive*.bbclass didn't work, and there were a few problems, for > >>>>>>>>> example: > >>>>>>>>> 1) There were a few duplicated code > >>>>>>>>> 2) There was no src_dir.org (or orig), but the diff command still > >>>>>>>>> use > >>>>>>>>> it, and it is not easy to fix this issue if we don't change a lot > >>>>>>>>> of the code. > >>>>>>>>> 3) It didn't archive the source for the native or gcc > >>>>>>>>> 4) The work flow is not very well > >>>>>>>>> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed > >>>>>>>>> 6) And others ... > >>>>>>>>> > >>>>>>>>> * So that we have to refactor it, the benefits are: > >>>>>>>>> 1) Fix the problems and make it work well. > >>>>>>>>> 2) Reduce more than 400 lines in total. > >>>>>>>>> 3) Make it easy to use. > >>>>>>>>> > >>>>>>>>> // Robert > >>>>>>>>> > >>>>>>>>> The following changes since commit > >>>>>>>>> 1b636173ca88e5ccca1992f9a12367a1189fa674: > >>>>>>>>> > >>>>>>>>> bitbake: toaster: Toaster GUI, generic search, filter and order > >>>>>>>>> (2014-01-10 15:20:26 +0000) > >>>>>>>>> > >>>>>>>>> are available in the git repository at: > >>>>>>>>> > >>>>>>>>> git://git.pokylinux.org/poky-contrib rbt/archiver > >>>>>>>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver > >>>>>>>> > >>>>>>>> Can you send pull-request for oe-core? > >>>>>>>> > >>>>>>> > >>>>>>> Here it is: > >>>>>>> git://git.pokylinux.org/poky-contrib rbt/ar_oe > >>>>>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rbt/ar_oe > >>>>>> > >>>>>> This still looks like poky-contrib (so it will download whole > >>>>>> poky-contrib when I add it as remote to oe-core repo). > >>>>>> > >>>>> > >>>>> Sorry, I can't create repo on oe-contrib, seems that I lost my key. > >>>> > >>>> That's sad, you should send new one to khem to replace it, before > >>>> someone else finds your old key. > >>>> > >>>>> How about: > >>>>> > >>>>> git fetch git://git.pokylinux.org/poky-contrib rbt/ar_oe > >>>> > >>>> That still fetches whole poky repo I'm not interested in. > >>>> > >>>> So I've downloaded just archiver.bbclass manually and tested it on my > >>>> use-case, I can confirm that it doesn't remove required file anymore, > >>>> but it seems that there is something terribly broken when used in > >>>> combination with rm_work or multimachine builds. > >>>> > >>>> I haven't debugged it yet, but 2nd build (in the same directory) with > >>>> this new archiver included almost every recipe failed in do_patch task. > >>>> > >>>>>>> // Robert > >>>>>>> > >>>>>>>>> Robert Yang (5): > >>>>>>>>> classes/archive*.bbclass: remove archive-*-source.bbclass > >>>>>>>>> archiver.bbclass: refactor it > >>>>>>>>> package_rpm.bbclass: archive the source to srpm package > >>>>>>>>> archiver.bbclass: move a few code to copyleft_compliance.bbclass > >>>>>>>>> local.conf.sample.extended: update for the archiver > >>>>>>>>> > >>>>>>>>> meta-yocto/conf/local.conf.sample.extended | 72 +-- > >>>>>>>>> meta/classes/archive-configured-source.bbclass | 65 --- > >>>>>>>>> meta/classes/archive-original-source.bbclass | 65 --- > >>>>>>>>> meta/classes/archive-patched-source.bbclass | 65 --- > >>>>>>>>> meta/classes/archiver.bbclass | 723 > >>>>>>>>> ++++++++----------------- > >>>>>>>>> meta/classes/copyleft_compliance.bbclass | 55 +- > >>>>>>>>> meta/classes/package_rpm.bbclass | 31 +- > >>>>>>>>> 7 files changed, 316 insertions(+), 760 deletions(-) > >>>>>>>>> delete mode 100644 meta/classes/archive-configured-source.bbclass > >>>>>>>>> delete mode 100644 meta/classes/archive-original-source.bbclass > >>>>>>>>> delete mode 100644 meta/classes/archive-patched-source.bbclass > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> 1.8.3.1 > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> Openembedded-core mailing list > >>>>>>>>> Openembedded-core@lists.openembedded.org > >>>>>>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core > >>>>>>>> > >>>>>> > >>>> > >> > > _______________________________________________ > > Openembedded-core mailing list > > Openembedded-core@lists.openembedded.org > > http://lists.openembedded.org/mailman/listinfo/openembedded-core > > > > -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-01-13 16:20 [PATCH 0/5] refactor the archive*.bbcalss Robert Yang ` (6 preceding siblings ...) 2014-01-15 16:55 ` [PATCH 0/5] refactor the archive*.bbcalss Martin Jansa @ 2014-02-13 9:54 ` Robert Yang 2014-02-13 10:14 ` Henning Heinold 7 siblings, 1 reply; 24+ messages in thread From: Robert Yang @ 2014-02-13 9:54 UTC (permalink / raw) To: openembedded-core Please hold on this, the recent changes of the prefuncs and postfuncs on bitbake makes it doesn't work on gcc-*, I will send a V2. // Robert On 01/14/2014 12:20 AM, Robert Yang wrote: > * The archive*.bbclass didn't work, and there were a few problems, for > example: > 1) There were a few duplicated code > 2) There was no src_dir.org (or orig), but the diff command still use > it, and it is not easy to fix this issue if we don't change a lot > of the code. > 3) It didn't archive the source for the native or gcc > 4) The work flow is not very well > 5) The "subprocess.call('fakeroot cp xxxx'" should be removed > 6) And others ... > > * So that we have to refactor it, the benefits are: > 1) Fix the problems and make it work well. > 2) Reduce more than 400 lines in total. > 3) Make it easy to use. > > // Robert > > The following changes since commit 1b636173ca88e5ccca1992f9a12367a1189fa674: > > bitbake: toaster: Toaster GUI, generic search, filter and order (2014-01-10 15:20:26 +0000) > > are available in the git repository at: > > git://git.pokylinux.org/poky-contrib rbt/archiver > http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver > > Robert Yang (5): > classes/archive*.bbclass: remove archive-*-source.bbclass > archiver.bbclass: refactor it > package_rpm.bbclass: archive the source to srpm package > archiver.bbclass: move a few code to copyleft_compliance.bbclass > local.conf.sample.extended: update for the archiver > > meta-yocto/conf/local.conf.sample.extended | 72 +-- > meta/classes/archive-configured-source.bbclass | 65 --- > meta/classes/archive-original-source.bbclass | 65 --- > meta/classes/archive-patched-source.bbclass | 65 --- > meta/classes/archiver.bbclass | 723 ++++++++----------------- > meta/classes/copyleft_compliance.bbclass | 55 +- > meta/classes/package_rpm.bbclass | 31 +- > 7 files changed, 316 insertions(+), 760 deletions(-) > delete mode 100644 meta/classes/archive-configured-source.bbclass > delete mode 100644 meta/classes/archive-original-source.bbclass > delete mode 100644 meta/classes/archive-patched-source.bbclass > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/5] refactor the archive*.bbcalss 2014-02-13 9:54 ` Robert Yang @ 2014-02-13 10:14 ` Henning Heinold 0 siblings, 0 replies; 24+ messages in thread From: Henning Heinold @ 2014-02-13 10:14 UTC (permalink / raw) To: openembedded-core On Thu, Feb 13, 2014 at 05:54:50PM +0800, Robert Yang wrote: > > Please hold on this, the recent changes of the prefuncs and postfuncs > on bitbake makes it doesn't work on gcc-*, I will send a V2. > > // Robert > Hi Robert, do not forget the sstate problems which of course happens on dora branch too. Bye Henning ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2014-02-13 10:15 UTC | newest] Thread overview: 24+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-01-13 16:20 [PATCH 0/5] refactor the archive*.bbcalss Robert Yang 2014-01-13 10:21 ` Martin Jansa 2014-01-13 10:47 ` Robert Yang 2014-01-13 11:52 ` Henning Heinold 2014-01-13 11:58 ` Martin Jansa 2014-01-14 2:13 ` Robert Yang 2014-01-13 16:20 ` [PATCH 1/5] classes/archive*.bbclass: remove archive-*-source.bbclass Robert Yang 2014-01-13 16:20 ` [PATCH 2/5] archiver.bbclass: refactor it Robert Yang 2014-01-13 16:20 ` [PATCH 3/5] package_rpm.bbclass: archive the source to srpm package Robert Yang 2014-01-13 16:20 ` [PATCH 4/5] archiver.bbclass: move a few code to copyleft_compliance.bbclass Robert Yang 2014-01-13 16:20 ` [PATCH 5/5] local.conf.sample.extended: update for the archiver Robert Yang 2014-01-15 16:55 ` [PATCH 0/5] refactor the archive*.bbcalss Martin Jansa 2014-01-16 2:14 ` Robert Yang 2014-01-16 9:25 ` Martin Jansa 2014-01-16 9:33 ` Robert Yang 2014-01-18 12:26 ` Martin Jansa 2014-01-21 10:40 ` Robert Yang 2014-01-21 10:41 ` Robert Yang 2014-01-21 12:37 ` Martin Jansa 2014-01-22 2:59 ` Robert Yang 2014-01-22 10:13 ` Robert Yang 2014-01-31 15:57 ` Martin Jansa 2014-02-13 9:54 ` Robert Yang 2014-02-13 10:14 ` Henning Heinold
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox