[PATCH 0/3] Realize archiver.bbclass

[PATCH 0/3] Realize archiver.bbclass

[Xiaofeng Yan]

From: Xiaofeng Yan <xiaofeng.yan@windriver.com>

archiver.bbclass is for archiving sources , patches and logs including .bb and .inc files by configuring  archiver_configure.bbclass which control what a archiving package should include.
For archiving a source rpm package, I add some functions to package_rpm.bbclass.
I also merge source_pkg.bbclass to archiver.bbclass due to their similar functions. 

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: xiaofeng/1977
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=xiaofeng/1977

Thanks,
    Xiaofeng Yan <xiaofeng.yan@windriver.com>
---


Xiaofeng Yan (3):
  package_rpm.bbclass: Add srpm function
  archiver.bbclass: archive sources, patches,logs to tarball
  archiver_configure.bbclass: configure the content for archiving
    package

 meta/classes/archiver.bbclass           |  393 +++++++++++++++++++++++++++++++
 meta/classes/archiver_configure.bbclass |   35 +++
 meta/classes/package_rpm.bbclass        |   54 ++++-
 3 files changed, 479 insertions(+), 3 deletions(-)
 create mode 100644 meta/classes/archiver.bbclass
 create mode 100644 meta/classes/archiver_configure.bbclass

[PATCH 1/3] package_rpm.bbclass: Add srpm function

[Xiaofeng Yan]

From: Xiaofeng Yan <xiaofeng.yan@windriver.com>

Add a new function to archive source, patches and logs to a source rpm
package. Every source rpm package  will be deployed to
${DEPLOY}/sources/deploy-srpm.

[YOCTO #1977]

Signed-off-by: Xiaofeng Yan <xiaofeng.yan@windriver.com>
---
 meta/classes/package_rpm.bbclass |   54 +++++++++++++++++++++++++++++++++++--
 1 files changed, 51 insertions(+), 3 deletions(-)

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index a813c8b..3804aa6 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -4,7 +4,7 @@ IMAGE_PKGTYPE ?= "rpm"
 
 RPM="rpm"
 RPMBUILD="rpmbuild"
-
+PKGWRITEDIRSRPM = "${DEPLOY_DIR}/sources/deploy-srpm" 
 PKGWRITEDIRRPM = "${WORKDIR}/deploy-rpms"
 
 python package_rpm_fn () {
@@ -475,6 +475,37 @@ python write_specfile () {
 	import textwrap
 	import oe.packagedata
 
+	# append information for logs and patches to %prep 
+	def add_prep(d,spec_files_bottom):
+		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+			spec_files_bottom.append('%%prep -n %s' % d.getVar('PN', True) )
+			spec_files_bottom.append('%s' % "echo \"include logs and patches, Please check them in SOURCES\"")
+			spec_files_bottom.append('')
+
+	# get the name of tarball for sources, patches and logs  
+	def get_tarballs(d):
+		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+			sourcelist=[]
+			workdir = d.getVar('WORKDIR',True)
+			for source in os.listdir(workdir):
+				if 'tar.gz' in source:
+					sourcelist.append(source)
+			return sourcelist
+
+	# append the name of tarball to key word 'SOURCE' in xxx.spec.
+	def tail_source(d,source_list=[],patch_list=None):
+		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+			source_number = 0
+			patch_number = 0
+			for source in source_list:
+				spec_preamble_top.append('Source' + str(source_number) + ': %s' % source)
+				source_number += 1
+			if patch_list:
+				for patch in patch_list:
+					print_deps(patch, "Patch" + str(patch_number), spec_preamble_top, d)
+					patch_number += 1
+
+
 	# We need a simple way to remove the MLPREFIX from the package name,
 	# and dependency information...
 	def strip_multilib(name, d):
@@ -786,6 +817,7 @@ python write_specfile () {
 		del localdata
 		bb.utils.unlockfile(lf)
 
+	add_prep(d,spec_files_bottom)
 	spec_preamble_top.append('Summary: %s' % srcsummary)
 	spec_preamble_top.append('Name: %s' % srcname)
 	spec_preamble_top.append('Version: %s' % srcversion)
@@ -796,7 +828,9 @@ python write_specfile () {
 	spec_preamble_top.append('Group: %s' % srcsection)
 	spec_preamble_top.append('Packager: %s' % srcmaintainer)
 	spec_preamble_top.append('URL: %s' % srchomepage)
-
+	
+	source_list = get_tarballs(d)
+	tail_source(d,source_list,None)
 	# Replaces == Obsoletes && Provides
 	if srcrreplaces and srcrreplaces.strip() != "":
 		for dep in srcrreplaces.split(','):
@@ -899,7 +933,13 @@ python write_specfile () {
 
 python do_package_rpm () {
 	import os
-
+	
+	def creat_srpm_dir(d):
+		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+			pkgwritesrpmdir = bb.data.expand('${PKGWRITEDIRSRPM}/${PACKAGE_ARCH_EXTEND}', d)
+			bb.mkdirhier(pkgwritesrpmdir)
+			os.chmod(pkgwritesrpmdir, 0755)
+		return pkgwritesrpmdir
 	# We need a simple way to remove the MLPREFIX from the package name,
 	# and dependency information...
 	def strip_multilib(name, d):
@@ -1015,8 +1055,16 @@ python do_package_rpm () {
 	cmd = cmd + " --define 'debug_package %{nil}'"
 	cmd = cmd + " --define '_rpmfc_magic_path " + magicfile + "'"
 	cmd = cmd + " --define '_tmppath " + workdir + "'"
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+		cmdsrpm = cmd + " --define '_sourcedir " + workdir + "' --define '_srcrpmdir " + creat_srpm_dir(d) + "'"
+		cmdsrpm = 'fakeroot ' + cmdsrpm + " -bs " + outspecfile
 	cmd = cmd + " -bb " + outspecfile
 
+	# Build the source rpm package !
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+		d.setVar('SBUILDSPEC', cmdsrpm + "\n")
+		d.setVarFlag('SBUILDSPEC', 'func', '1')
+		bb.build.exec_func('SBUILDSPEC', d)
 	# Build the rpm package!
 	d.setVar('BUILDSPEC', cmd + "\n")
 	d.setVarFlag('BUILDSPEC', 'func', '1')
-- 
1.7.0.4

[PATCH 2/3] archiver.bbclass: archive sources, patches, logs to tarball

[Xiaofeng Yan]

From: Xiaofeng Yan <xiaofeng.yan@windriver.com>

Support the following functions in this bbclass:

1 Archive sources in ${S} in the different stage to tarball (do_unpack,do_patch,do_configure).
2 Archive patches including series to tarball
3 Archive logs including scripts (.bb and .inc files) to tarball
4 dump environment resources which show all variable and functions to be
  used to xxx.showdata.dump when running a task
5 dump all content in 's' including patches to file xxx.diff.gz

All of tarballs will be deployed to ${DEPLOY_DIR}/sources/

[#YOCTO 1977]

Signed-off-by: Xiaofeng Yan <xiaofeng.yan@windriver.com>
---
 meta/classes/archiver.bbclass |  393 +++++++++++++++++++++++++++++++++++++++++
 1 files changed, 393 insertions(+), 0 deletions(-)
 create mode 100644 meta/classes/archiver.bbclass

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
new file mode 100644
index 0000000..5313535
--- /dev/null
+++ b/meta/classes/archiver.bbclass
@@ -0,0 +1,393 @@
+# This file is used for archiving sources ,patches,and logs to tarball.
+# It also output building environment to xxx.dump.data and create xxx.diff.gz to record 
+# all content in ${S} to a diff file. 
+
+EXCLUDE_FROM ?= ".pc autom4te.cache"
+ARCHIVE_TYPE ?= "TAR SRPM"
+DISTRO ?= "poky"
+
+def parse_var(d,var):
+	''' parse variable like ${PV}  in "require xxx_${PV}.inc"  to a real value. for example, change "require xxx_${PV}.inc" to "require xxx_1.2.inc" '''
+	import re 
+	pat = re.compile('.*\$({(.*)}).*')
+	if '$' not in var and '/' not in var:
+		return var
+	else:
+		if '/' in var:
+			return [i for i in var.split('/') if i.endswith('.inc')][0]
+		elif '$' in  var:
+			m = pat.match(var)
+			patstr = '\$' + m.group(1)
+			var_str = m.group(2)
+			return re.sub(patstr,d.getVar(var_str,True),var)
+		else:
+			return var
+		
+
+def get_bb_inc(d):
+	'''create a directory "script-logs" including .bb and .inc file in ${WORKDIR}'''
+	import re
+	import os
+	import shutil
+	
+	bbinc = []
+	pat=re.compile('require\s*([^\s]*\.*)(.*)')
+	file_dir = d.getVar('FILE', True)
+	bbdir = os.path.dirname(file_dir)
+	work_dir = d.getVar('WORKDIR', True)
+	os.chdir(work_dir)
+	bb.mkdirhier("script-logs")
+	os.chdir(bbdir)
+	bbfile = os.path.basename(file_dir)
+	bbinc.append(bbfile)
+	
+	def get_inc (file):
+		f = open(file,'r')
+		for line in f.readlines():
+			if 'require' not  in line:
+				bbinc.append(file)
+			else:
+				try:
+					incfile = pat.match(line).group(1)
+					incfile = parse_var(d,incfile)
+					bbinc.append(incfile)
+					get_inc(incfile)
+				except AttributeError:
+					pass
+	get_inc(bbfile)
+	os.chdir(work_dir)
+	for root, dirs, files in os.walk(bbdir):
+		for file in bbinc:
+			if file in files:
+				shutil.copy(root + '/' + file,'script-logs')
+	oe.path.copytree('temp', 'script-logs')
+	return work_dir + '/script-logs'
+
+def get_all_patches(d):
+	'''copy patches and series file to a pointed directory which will be archived to tarball in ${WORKDIR}'''
+	import shutil
+	
+	src_patches=[]
+	pf = d.getVar('PF', True)
+	work_dir = d.getVar('WORKDIR', True)
+	dest = os.path.join(work_dir, pf + '-patches') 
+	shutil.rmtree(dest, ignore_errors=True)
+	bb.mkdirhier(dest)
+	
+	src_uri = d.getVar('SRC_URI', 1).split()
+	fetch = bb.fetch2.Fetch(src_uri, d)
+	locals = (fetch.localpath(url) for url in fetch.urls)
+	for local in locals:
+		src_patches.append(local)
+	for patch in src_patches[1:]:
+		 shutil.copy(patch,dest)
+	return dest
+
+def get_applying_patches(d):
+	"""only copy applying patches to a pointed directory which will be archived to tarball"""
+	import os
+	import shutil
+
+
+	pf = d.getVar('PF', True)
+	work_dir = d.getVar('WORKDIR', True)
+	dest = os.path.join(work_dir, pf + '-patches') 
+	shutil.rmtree(dest, ignore_errors=True)
+	bb.mkdirhier(dest)
+
+
+	patches = src_patches(d)
+	for patch in patches:
+		_, _, local, _, _, parm = bb.decodeurl(patch)
+		if local:
+			 shutil.copy(local,dest)
+	return dest
+
+def not_tarball(d):
+	'''packages including key words 'work-shared','native', 'task-' will be passed'''
+	import os
+
+	workdir = d.getVar('WORKDIR',True)
+	s = d.getVar('S',True)
+	if 'work-shared' in s or 'task-' in workdir or 'native' in workdir:
+		return True
+	else:
+		return False
+
+def archive_sources(d,middle_name): 
+	'''archive sources codes tree to tarball'''
+	import tarfile
+	import shutil
+	
+	s = d.getVar('S',True)
+	workdir=d.getVar('WORKDIR', True)
+	PF = d.getVar('PF',True)
+	tarname = PF + '-' + middle_name + ".tar.gz"
+
+	if os.path.exists(s) and s is not workdir:
+		sourcedir = os.path.basename(s)
+		tarbase = os.path.dirname(s)
+		if not sourcedir or os.path.dirname(tarbase) == workdir:
+			sourcedir = os.path.basename(os.path.dirname(s))
+			tarbase = os.path.dirname(os.path.dirname(s))
+		os.chdir(tarbase)
+	else:
+		sourcedir = os.path.basename(s)
+		if not os.path.exists(sourcedir):
+			os.mkdir(sourcedir)
+		try:
+			for file in os.listdir(s):
+				if file is not 'temp' and file is not sourcedir:
+					shutil.copy(file,sourcedir)
+		except (IOError,OSError):
+			pass
+
+
+	if (len(os.listdir(sourcedir))) != 0:
+		tar = tarfile.open( tarname, "w:gz")
+		tar.add(sourcedir)
+		tar.close()
+		if cmp(workdir,os.path.dirname(s)) and not os.path.exists(workdir + '/' + tarname):
+			shutil.move(os.path.dirname(s) + '/' + tarname,workdir)
+	else:
+		return
+	return tarname
+
+def archive_patches(d,patchdir,series):
+	'''archive patches to tarball and also include series files if 'series' is True'''
+	import tarfile
+	import shutil
+
+	s = d.getVar('S',True)
+	work_dir = d.getVar('WORKDIR', True)
+	os.chdir(work_dir)
+	patch_dir = os.path.basename(patchdir)
+	tarname = patch_dir + ".tar.gz"
+	if series  == 'all' and os.path.exists(s + '/patches/series'):
+			shutil.copy(s + '/patches/series',patch_dir)
+	tar = tarfile.open(tarname, "w:gz")
+	tar.add(patch_dir)
+	tar.close()
+	shutil.rmtree(patch_dir, ignore_errors=True)
+	return tarname
+
+def select_archive_patches(d,option):
+	'''select to archive all patches including non-applying and series or applying patches '''
+	if option == "all":
+		patchdir = get_all_patches(d)
+	elif option == "applying":
+		patchdir = get_applying_patches(d)
+	try:
+		os.rmdir(patchdir)
+	except OSError:
+			tarpatch = archive_patches(d,patchdir,option)
+			return tarpatch
+	return
+
+def archive_logs(d,logdir,bbinc=False):
+	'''archive logs in temp to tarball and .bb and .inc files if bbinc is True '''
+	import tarfile
+	import shutil
+
+	log_dir = os.path.basename(logdir)
+	pf = d.getVar('PF',True)
+	tarname = pf + '-' + log_dir + ".tar.gz"
+	tar = tarfile.open(tarname, "w:gz")
+	tar.add(log_dir)
+	tar.close()
+	if bbinc:
+		shutil.rmtree(log_dir, ignore_errors=True)
+	return tarname 
+
+def move_tarball_deploy(d,tarball_list):
+	'''move tarball in location to ${DEPLOY_DIR}/sources'''
+	import shutil
+	
+	if tarball_list is []:
+		return
+	target_sys = d.getVar('TARGET_SYS', True)
+	pf = d.getVar('PF', True)
+	tar_sources = d.getVar('DEPLOY_DIR', True) + '/sources/' + target_sys + '/'  + pf
+	if not os.path.exists(tar_sources):
+		bb.mkdirhier(tar_sources)
+	for source in tarball_list:
+		if source:
+			if os.path.exists(tar_sources + '/' + source):
+				os.remove(tar_sources + '/' + source)
+			shutil.move(source,tar_sources)
+
+def verify_var(d):
+	'''check the type for archiving package('tar' or 'srpm')'''
+	try:
+		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in d.getVar('ARCHIVE_TYPE', True).split():
+			raise AttributeError	
+	except AttributeError:
+			bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" is \'tar\' or \'srpm\', no other types") 
+
+def archive_copyleft(d):
+	'''check if a package to archive is copy-left'''
+	if bb.data.inherits_class('copyleft_compliance', d):
+		included, reason = copyleft_should_include(d)
+		if not included:
+			return False
+		return True
+def archive_sources_patches(d,middle_name):
+	'''archive sources and patches to tarball. middle_name will append strings ${middle_name} to ${PR} as middle name. for example, zlib-1.4.6-prepatch(middle_name).tar.gz '''
+	verify_var(d)	
+	if not_tarball(d):
+		return
+	
+	source_tar_name = archive_sources(d,middle_name)
+	if middle_name == "prepatch":
+		if d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'TRUE':
+			patch_tar_name = select_archive_patches(d,"all")
+		elif d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'FALSE':
+			patch_tar_name = select_archive_patches(d,"applying")
+		else:
+			bb.fatal("Please define 'PATCHES_ARCHIVE_WITH_SERIES' is strings 'True' or 'False' ")
+	else:
+		patch_tar_name = ''
+
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 'SRPM':
+		move_tarball_deploy(d,[source_tar_name,patch_tar_name])
+
+def archive_sources_patches_logs_copyleft(d,middle_name):
+	'''archive source, patches and logs according to the variable "COPYLEFT_COMPLIANCE", If this variable is 'True', then archive the packages for copy-left, or else archive all packages'''
+	verify_var(d)	
+	if not_tarball(d):
+		return
+	copyleft_compliance = d.getVar('COPYLEFT_COMPLIANCE', True)
+	if copyleft_compliance is None:
+		archive_sources_patches(d,middle_name)
+	elif copyleft_compliance.upper() == 'TRUE' and archive_copyleft(d):
+		archive_sources_patches(d,middle_name)
+	
+	
+def archive_scripts_logs(d):
+	'''archive scripts and logs. scripts include .bb and .inc files and logs include stuff in "temp".'''
+	verify_var(d)
+	if not d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) or not_tarball(d):
+		return
+
+	s = d.getVar('WORKDIR', True)
+	os.chdir(s)
+	source_archive_log_with_scripts = d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True)
+	if source_archive_log_with_scripts == 'logs_with_scripts':
+		logdir = get_bb_inc(d)
+		tarlog = archive_logs(d,logdir,True)
+	elif source_archive_log_with_scripts == 'logs':
+		if os.path.exists('temp'):
+			archive_logs(d,'temp',False)
+	else:
+		return
+		
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 'SRPM':
+		move_tarball_deploy(d,[tarlog])
+
+def dumpdata(d):
+	'''dump environment to "${P}-${PR}.showdata.dump" including all kinds of varibale and functions when running a task'''
+	workdir = bb.data.getVar('WORKDIR', d, 1)
+	distro = bb.data.getVar('DISTRO', d, 1)
+	s = d.getVar('S', True)
+	pf = d.getVar('PF', True)
+	target_sys = d.getVar('TARGET_SYS', True)
+	dumpdir = d.getVar('DEPLOY_DIR', True) + '/sources/' + target_sys + '/'  + pf
+	if not os.path.exists(dumpdir):
+		bb.mkdirhier(dumpdir)
+
+	dumpfile = os.path.join(dumpdir, bb.data.expand("${P}-${PR}.showdata.dump",d))
+
+	bb.note("Dumping metadata into '%s'" % dumpfile)
+	f = open(dumpfile, "w")
+	# emit variables and shell functions
+        bb.data.emit_env(f, d, True)
+	# emit the metadata which isnt valid shell
+	for e in d.keys():
+		if bb.data.getVarFlag(e, 'python', d):
+			f.write("\npython %s () {\n%s}\n" % (e, bb.data.getVar(e, d, 1)))
+	f.close()
+
+
+def create_diff_gz(d):
+	'''creating .diff.gz in ${DEPLOY_DIR_SRC}/${P}-${PR}.diff.g gz for mapping all content in 's' including patches to  xxx.diff.gz'''
+	import shutil
+
+	work_dir = d.getVar('WORKDIR', True)
+	exclude_from = d.getVar('EXCLUDE_FROM', True).split()
+	pf = d.getVar('PF', True)
+	target_sys = d.getVar('TARGET_SYS', True)
+	diff_dir = d.getVar('DEPLOY_DIR', True) + '/sources/' + target_sys + '/'  + pf
+	diff_file = os.path.join(diff_dir, bb.data.expand("${P}-${PR}.diff.gz",d))
+	os.chdir(work_dir)
+	f = open('temp/exclude-from-file', 'a')
+	for i in exclude_from:
+		f.write(i)
+		f.write("\n")
+	f.close()
+
+
+	s=d.getVar('S', True)
+	distro = d.getVar('DISTRO',True)
+	dest = s + '/' + distro + '/files'
+	if not os.path.exists(dest):
+		bb.mkdirhier(dest)
+	for i in os.listdir(os.getcwd()):
+		if os.path.isfile(i):
+			shutil.copy(i, dest)
+	
+	bb.note("Creating .diff.gz in ${DEPLOY_DIR_SRC}/${P}-${PR}.diff.gz")
+	cmd = "LC_ALL=C TZ=UTC0 diff --exclude-from=" + work_dir + "/temp/exclude-from-file -Naur " + s + '.org' + ' ' +  s + " | gzip -c > " + diff_file
+	d.setVar('DIFF', cmd + "\n")
+	d.setVarFlag('DIFF', 'func', '1') 
+	bb.build.exec_func('DIFF', d)
+	shutil.rmtree(s + '.org', ignore_errors=True)
+
+# This function will run when user want to get tarball for sources and patches after do_unpack
+python do_archive_original_sources_patches(){
+	archive_sources_patches_logs_copyleft(d,'prepatch')
+}
+
+# This function will run when user want to get tarball for patched sources after do_patch
+python do_archive_patched_sources(){
+	archive_sources_patches_logs_copyleft(d,'patched')
+}
+
+# This function will run when user want to get tarball for configured sources after do_configure
+python do_archive_configured_sources(){
+	archive_sources_patches_logs_copyleft(d,'configured')
+}
+
+# This function will run when user want to get tarball for logs or both logs and scripts(.bb and .inc files)
+python do_archive_scripts_logs(){
+	archive_scripts_logs(d)
+}
+
+# This function will run when user want to know what variable and functions in a running task are and also can get a diff file including 
+# all content a package should include.
+python do_dumpdata_create_diff_gz(){
+	dumpdata(d)
+	create_diff_gz(d)
+}
+
+# This functions prepare for archiving "linux-yocto" because this package create directory 's' before do_patch instead of after do_unpack.
+# This is special control for archiving linux-yocto only.
+python do_archive_linux_yocto(){
+	s = d.getVar('S', True)
+	if 'linux-yocto' in s:
+		source_tar_name = archive_sources(d,'')
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 'SRPM':
+		move_tarball_deploy(d,[source_tar_name,''])
+}
+do_kernel_checkout[postfuncs] += "do_archive_linux_yocto "
+
+# remove tarball for sources, patches and logs after creating srpm.
+python do_remove_tarball(){
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+		work_dir = d.getVar('WORKDIR', True)
+		os.chdir(work_dir)
+		for file in os.listdir(os.getcwd()):
+			if '.tar.gz' in file:
+				os.remove(file)
+}
+do_remove_taball[deptask] = "do_archive_scripts_logs"
+do_package_write_rpm[postfuncs] += "do_remove_tarball "
-- 
1.7.0.4

[PATCH 3/3] archiver_configure.bbclass: configure the content for archiving package

[Xiaofeng Yan]

From: Xiaofeng Yan <xiaofeng.yan@windriver.com>

This is configuration file to point to what content a archive package
should include in the different stage of task by pointing to different variable and
stage.

[YOCTO #1977]

Signed-off-by: Xiaofeng Yan <xiaofeng.yan@windriver.com>

Signed-off-by: Xiaofeng Yan <xiaofeng.yan@windriver.com>
---
 meta/classes/archiver_configure.bbclass |   35 +++++++++++++++++++++++++++++++
 1 files changed, 35 insertions(+), 0 deletions(-)
 create mode 100644 meta/classes/archiver_configure.bbclass

diff --git a/meta/classes/archiver_configure.bbclass b/meta/classes/archiver_configure.bbclass
new file mode 100644
index 0000000..c9892d3
--- /dev/null
+++ b/meta/classes/archiver_configure.bbclass
@@ -0,0 +1,35 @@
+# This file is for getting tarball in different stage for sources, patches, and logs by configuring 
+# the following variable and tasks
+
+inherit archiver
+
+# SOURCE_ARCHIVE_PACKAGE_TYPE = {'tar','srpm'}
+SOURCE_ARCHIVE_PACKAGE_TYPE = 'tar'
+
+# SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = {'logs_with_scripts', 'logs'}
+# String 'logs_with_scripts' include temp directory and .bb and .inc file
+# String 'logs' only include temp 
+SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = 'logs_with_scripts'
+
+# PATCHES_ARCHIVE_WITH_SERIES = {'true', 'false'}
+# Strings 'true' means that patches including series files(series + non-applying)
+# String 'false' means that no series and only archive applying patches
+PATCHES_ARCHIVE_WITH_SERIES = 'true'
+
+# Archive packages for copy-left
+#COPYLEFT_COMPLIANCE = 'true'
+
+# Open this item when you want to get original sources tarball with patches
+do_unpack[postfuncs] += "do_archive_original_sources_patches "
+
+# Open this item when you want to get tarball for patched sources including patches
+#do_patch[postfuncs] += "do_archive_patched_sources "
+
+# Open this item when you want to get tarball for configured sources including patches
+#do_configure[postfuncs] += "do_archive_configured_sources "
+
+# Open this item when you want to get tarball for logs tarball
+do_package_write_rpm[prefuncs] += "do_archive_scripts_logs "
+
+# Get dump date and create diff file 
+do_package_write_rpm[postfuncs] += "do_dumpdata_create_diff_gz "
-- 
1.7.0.4

Re: [PATCH 1/3] package_rpm.bbclass: Add srpm function

[Colin Walters]

On Sat, 2012-03-03 at 18:54 +0800, Xiaofeng Yan wrote:
> From: Xiaofeng Yan <xiaofeng.yan@windriver.com>
> 
> Add a new function to archive source, patches and logs to a source rpm
> package. Every source rpm package  will be deployed to
> ${DEPLOY}/sources/deploy-srpm.

What's the goal of this?  Just to be a "one file" collection of
sources?  Or do we actually expect people to build from this somehow?

If the intention is to eventually allow building, but it's not actually
used to build right now, it seems very likely to "bitrot" as the core
changes.

Re: [PATCH 3/3] archiver_configure.bbclass: configure the content for archiving package

[Saul Wold]

On 03/03/2012 02:54 AM, Xiaofeng Yan wrote:
> From: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>
> This is configuration file to point to what content a archive package
> should include in the different stage of task by pointing to different variable and
> stage.
>
Xiaogeng,

You are on the right path with this one, but what was originally talked 
about was having multiple archiver bbclasses that did the configuration 
and set when the archive would take place.

Than one could just inherit the bbclass that matches their needs, so 
there should be a bbclass that's equivalent to the current copyleft 
bbclass. And as we defined in earlier emails the 3-4 archives, we should 
have matching bbclasses for those.

1) Original Source & Patches w/ Series file & other files
2) Patched & Configured Source & temp dir
3) Copyleft Equivlant (so we can remove that class also)

Sau!


> [YOCTO #1977]
>
> Signed-off-by: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>
> Signed-off-by: Xiaofeng Yan<xiaofeng.yan@windriver.com>
> ---
>   meta/classes/archiver_configure.bbclass |   35 +++++++++++++++++++++++++++++++
>   1 files changed, 35 insertions(+), 0 deletions(-)
>   create mode 100644 meta/classes/archiver_configure.bbclass
>
> diff --git a/meta/classes/archiver_configure.bbclass b/meta/classes/archiver_configure.bbclass
> new file mode 100644
> index 0000000..c9892d3
> --- /dev/null
> +++ b/meta/classes/archiver_configure.bbclass
> @@ -0,0 +1,35 @@
> +# This file is for getting tarball in different stage for sources, patches, and logs by configuring
> +# the following variable and tasks
> +
> +inherit archiver
> +
> +# SOURCE_ARCHIVE_PACKAGE_TYPE = {'tar','srpm'}
> +SOURCE_ARCHIVE_PACKAGE_TYPE = 'tar'
> +
> +# SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = {'logs_with_scripts', 'logs'}
> +# String 'logs_with_scripts' include temp directory and .bb and .inc file
> +# String 'logs' only include temp
> +SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = 'logs_with_scripts'
> +
> +# PATCHES_ARCHIVE_WITH_SERIES = {'true', 'false'}
> +# Strings 'true' means that patches including series files(series + non-applying)
> +# String 'false' means that no series and only archive applying patches
> +PATCHES_ARCHIVE_WITH_SERIES = 'true'
> +
> +# Archive packages for copy-left
> +#COPYLEFT_COMPLIANCE = 'true'
> +
> +# Open this item when you want to get original sources tarball with patches
> +do_unpack[postfuncs] += "do_archive_original_sources_patches "
> +
> +# Open this item when you want to get tarball for patched sources including patches
> +#do_patch[postfuncs] += "do_archive_patched_sources "
> +
> +# Open this item when you want to get tarball for configured sources including patches
> +#do_configure[postfuncs] += "do_archive_configured_sources "
> +
> +# Open this item when you want to get tarball for logs tarball
> +do_package_write_rpm[prefuncs] += "do_archive_scripts_logs "
> +
> +# Get dump date and create diff file
> +do_package_write_rpm[postfuncs] += "do_dumpdata_create_diff_gz "

Re: [PATCH 0/3] Realize archiver.bbclass

[Saul Wold]

On 03/03/2012 02:53 AM, Xiaofeng Yan wrote:
> From: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>
> archiver.bbclass is for archiving sources , patches and logs including .bb and .inc files by configuring  archiver_configure.bbclass which control what a archiving package should include.
> For archiving a source rpm package, I add some functions to package_rpm.bbclass.
> I also merge source_pkg.bbclass to archiver.bbclass due to their similar functions.
>
Are you going to deprecate/remove the other classes sourcepkg.bbclass, 
src_distribute.bbclass, src_distribute_local.bbclass and 
copyleft_compliance.bbclass

I am still reviewing this and would like to get ACKs from Chris and Mark.

This is going to require some review and testing.  I also sent and email 
about the archiver_configure class.

Can you please let us know how you tested, this is an important change 
and we would like to get it close to correct the first time through.

Thanks for your effort on this.

Sau!

> Pull URL: git://git.pokylinux.org/poky-contrib.git
>    Branch: xiaofeng/1977
>    Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=xiaofeng/1977
>
> Thanks,
>      Xiaofeng Yan<xiaofeng.yan@windriver.com>
> ---
>
>
> Xiaofeng Yan (3):
>    package_rpm.bbclass: Add srpm function
>    archiver.bbclass: archive sources, patches,logs to tarball
>    archiver_configure.bbclass: configure the content for archiving
>      package
>
>   meta/classes/archiver.bbclass           |  393 +++++++++++++++++++++++++++++++
>   meta/classes/archiver_configure.bbclass |   35 +++
>   meta/classes/package_rpm.bbclass        |   54 ++++-
>   3 files changed, 479 insertions(+), 3 deletions(-)
>   create mode 100644 meta/classes/archiver.bbclass
>   create mode 100644 meta/classes/archiver_configure.bbclass
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [PATCH 3/3] archiver_configure.bbclass: configure the content for archiving package

[Xiaofeng Yan]

[-- Attachment #1: Type: text/plain, Size: 5218 bytes --]

On 2012?03?06? 06:18, Saul Wold wrote:
> On 03/03/2012 02:54 AM, Xiaofeng Yan wrote:
>> From: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>>
>> This is configuration file to point to what content a archive package
>> should include in the different stage of task by pointing to 
>> different variable and
>> stage.
>>
> Xiaogeng,
>
> You are on the right path with this one, but what was originally 
> talked about was having multiple archiver bbclasses that did the 
> configuration and set when the archive would take place.
>
> Than one could just inherit the bbclass that matches their needs, so 
> there should be a bbclass that's equivalent to the current copyleft 
> bbclass. And as we defined in earlier emails the 3-4 archives, we 
> should have matching bbclasses for those.
>
Thanks for your reply.
> 1) Original Source & Patches w/ Series file & other files
This needs two bbclass named for 
"tar_original_source_patch_serires_scripts_logs.bblcass" and 
"srpm_original_source_patch_serires_scripts_logs.bblcass". 
archiver.bbclass should be inherited in two bbclass.

"scripts" include .bb and .inc files.
"logs" include temp.
I take zlib for an example with tar type:
step 1: inherit tar_original_source_patch_serires_scripts_logs in 
pakcage_rpm.bbclass.
step 2:
/$bitbake zlib/
step 3:
/$ ls  tmp/deploy/sources/i586-poky-linux/Zlib/zlib-1.2.6-r1/
zlib-1.2.6-r1-patches.tar.gz  zlib-1.2.6-r1-prepatch.tar.gz  
zlib-1.2.6-r1-script-logs.tar.gz/

I organize the directory to deploy archiving package with the following 
mode.
tar_sources = d.getVar('DEPLOY_DIR', True) + '/sources/' + target_sys + 
'/' + licenses + '/' + pf
here:

target_sys is  i586-poky-linux
Zlib is license of package zlib
pf is "zlib-1.2.6-r1"
//zlib-1.2.6-r1-prepatch.tar.gz  include original sources
zlib-1.2.6-r1-patches.tar.gz include patches and series
zlib-1.2.6-r1-script-logs.tar.gz include temp with .bb and .inc files
/
/if we want to get srpm package. we should inherit 
srpm_original_source_patch_serires_scripts_logs.bblcass in 
package_rpm.bbclass
$bitbake zlib
$ls /tmp/deploy/sources/deploy-srpm/i586/Zlib//
zlib-1.2.6-r1.src.rpm
/
/
> 2) Patched & Configured Source & temp dir
This needs four bbclass
tar_patched_sources_logs.bbclass
tar_configured_sources_logs.bbclass
srpm_patched_sources_logs.bbclass
srpm_configured_sources_logs.bbclass

> 3) Copyleft Equivlant (so we can remove that class also)
I only inherit this class in archiver.bbclass in V1 without merging it 
to archiver.bbclass.  I can merge this class if you want.

  I think we can define some variables like SOURCE_ARCHIVE_PACKAGE_TYPE 
= {'tar','srpm'} and SOURCE_ARCHIVE_LOGS_WITH_SCRIPTS = 'True'
SOURCE_ARCHIVE_LOGS = 'True' in conf/local.conf to reduce the number of 
archive bbclass.

I will send you the V2 according to your reply.
**
I do apologize about my design can't meet your requirement.
>
> Sau!
>
>
>> [YOCTO #1977]
>>
>> Signed-off-by: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>>
>> Signed-off-by: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>> ---
>>   meta/classes/archiver_configure.bbclass |   35 
>> +++++++++++++++++++++++++++++++
>>   1 files changed, 35 insertions(+), 0 deletions(-)
>>   create mode 100644 meta/classes/archiver_configure.bbclass
>>
>> diff --git a/meta/classes/archiver_configure.bbclass 
>> b/meta/classes/archiver_configure.bbclass
>> new file mode 100644
>> index 0000000..c9892d3
>> --- /dev/null
>> +++ b/meta/classes/archiver_configure.bbclass
>> @@ -0,0 +1,35 @@
>> +# This file is for getting tarball in different stage for sources, 
>> patches, and logs by configuring
>> +# the following variable and tasks
>> +
>> +inherit archiver
>> +
>> +# SOURCE_ARCHIVE_PACKAGE_TYPE = {'tar','srpm'}
>> +SOURCE_ARCHIVE_PACKAGE_TYPE = 'tar'
>> +
>> +# SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = {'logs_with_scripts', 'logs'}
>> +# String 'logs_with_scripts' include temp directory and .bb and .inc 
>> file
>> +# String 'logs' only include temp
>> +SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = 'logs_with_scripts'
>> +
>> +# PATCHES_ARCHIVE_WITH_SERIES = {'true', 'false'}
>> +# Strings 'true' means that patches including series files(series + 
>> non-applying)
>> +# String 'false' means that no series and only archive applying patches
>> +PATCHES_ARCHIVE_WITH_SERIES = 'true'
>> +
>> +# Archive packages for copy-left
>> +#COPYLEFT_COMPLIANCE = 'true'
>> +
>> +# Open this item when you want to get original sources tarball with 
>> patches
>> +do_unpack[postfuncs] += "do_archive_original_sources_patches "
>> +
>> +# Open this item when you want to get tarball for patched sources 
>> including patches
>> +#do_patch[postfuncs] += "do_archive_patched_sources "
>> +
>> +# Open this item when you want to get tarball for configured sources 
>> including patches
>> +#do_configure[postfuncs] += "do_archive_configured_sources "
>> +
>> +# Open this item when you want to get tarball for logs tarball
>> +do_package_write_rpm[prefuncs] += "do_archive_scripts_logs "
>> +
>> +# Get dump date and create diff file
>> +do_package_write_rpm[postfuncs] += "do_dumpdata_create_diff_gz "
>


[-- Attachment #2: Type: text/html, Size: 8591 bytes --]

Re: [PATCH 0/3] Realize archiver.bbclass

[Xiaofeng Yan]

On 2012年03月06日 06:28, Saul Wold wrote:
> On 03/03/2012 02:53 AM, Xiaofeng Yan wrote:
>> From: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>>
>> archiver.bbclass is for archiving sources , patches and logs 
>> including .bb and .inc files by configuring 
>> archiver_configure.bbclass which control what a archiving package 
>> should include.
>> For archiving a source rpm package, I add some functions to 
>> package_rpm.bbclass.
>> I also merge source_pkg.bbclass to archiver.bbclass due to their 
>> similar functions.
>>
> Are you going to deprecate/remove the other classes sourcepkg.bbclass, 
> src_distribute.bbclass, src_distribute_local.bbclass and 
> copyleft_compliance.bbclass
>
sourcepkg.bbclass has been merged into archiver.bbclass. others can't be 
merged into archiver.bbclass.

I will merge them to archiver.bbclass in V2 and remove tem.
> I am still reviewing this and would like to get ACKs from Chris and Mark.
>
> This is going to require some review and testing. I also sent and 
> email about the archiver_configure class.
>
> Can you please let us know how you tested, this is an important change 
> and we would like to get it close to correct the first time through.
>
inherit archiver_configure.bbclass in a global bbclass file, for example 
package_rpm.bbclass and then run "bitbake core-image-sato" or "bitbake 
package_name".

archiving package should be deployed in tmp/deploy/sources/ when 
finishing this building.

> Thanks for your effort on this.
>
> Sau!
>
>> Pull URL: git://git.pokylinux.org/poky-contrib.git
>> Branch: xiaofeng/1977
>> Browse: 
>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=xiaofeng/1977
>>
>> Thanks,
>> Xiaofeng Yan<xiaofeng.yan@windriver.com>
>> ---
>>
>>
>> Xiaofeng Yan (3):
>> package_rpm.bbclass: Add srpm function
>> archiver.bbclass: archive sources, patches,logs to tarball
>> archiver_configure.bbclass: configure the content for archiving
>> package
>>
>> meta/classes/archiver.bbclass | 393 +++++++++++++++++++++++++++++++
>> meta/classes/archiver_configure.bbclass | 35 +++
>> meta/classes/package_rpm.bbclass | 54 ++++-
>> 3 files changed, 479 insertions(+), 3 deletions(-)
>> create mode 100644 meta/classes/archiver.bbclass
>> create mode 100644 meta/classes/archiver_configure.bbclass
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>

Re: [PATCH 2/3] archiver.bbclass: archive sources, patches, logs to tarball

[Mark Hatle]

Apologies for the very late review...  see comments below.  All in all it looks 
good.  But I have not applied it and tried it yet.

On 3/3/12 4:54 AM, Xiaofeng Yan wrote:
> From: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>
> Support the following functions in this bbclass:
>
> 1 Archive sources in ${S} in the different stage to tarball (do_unpack,do_patch,do_configure).
> 2 Archive patches including series to tarball
> 3 Archive logs including scripts (.bb and .inc files) to tarball
> 4 dump environment resources which show all variable and functions to be
>    used to xxx.showdata.dump when running a task
> 5 dump all content in 's' including patches to file xxx.diff.gz
>
> All of tarballs will be deployed to ${DEPLOY_DIR}/sources/
>
> [#YOCTO 1977]
>
> Signed-off-by: Xiaofeng Yan<xiaofeng.yan@windriver.com>
> ---
>   meta/classes/archiver.bbclass |  393 +++++++++++++++++++++++++++++++++++++++++
>   1 files changed, 393 insertions(+), 0 deletions(-)
>   create mode 100644 meta/classes/archiver.bbclass
>
...

> +def verify_var(d):
> +	'''check the type for archiving package('tar' or 'srpm')'''
> +	try:
> +		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in d.getVar('ARCHIVE_TYPE', True).split():
> +			raise AttributeError	
> +	except AttributeError:
> +			bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" is \'tar\' or \'srpm\', no other types")

Error message should be something like:

bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" should be \'tar\' or '\srpm\'")

...

> +def archive_sources_patches(d,middle_name):
> +	'''archive sources and patches to tarball. middle_name will append strings ${middle_name} to ${PR} as middle name. for example, zlib-1.4.6-prepatch(middle_name).tar.gz '''
> +	verify_var(d)	
> +	if not_tarball(d):
> +		return
> +	
> +	source_tar_name = archive_sources(d,middle_name)
> +	if middle_name == "prepatch":
> +		if d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'TRUE':
> +			patch_tar_name = select_archive_patches(d,"all")
> +		elif d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'FALSE':
> +			patch_tar_name = select_archive_patches(d,"applying")
> +		else:
> +			bb.fatal("Please define 'PATCHES_ARCHIVE_WITH_SERIES' is strings 'True' or 'False' ")

Instead of an explicit TRUE/FALSE setting, does it make sense for one to be the 
default?  I'd suspect in this case "True" is the better default, but I'm not 
completely sure.

> +	else:
> +		patch_tar_name = ''
> +
> +	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 'SRPM':
> +		move_tarball_deploy(d,[source_tar_name,patch_tar_name])
> +
> +def archive_sources_patches_logs_copyleft(d,middle_name):
> +	'''archive source, patches and logs according to the variable "COPYLEFT_COMPLIANCE", If this variable is 'True', then archive the packages for copy-left, or else archive all packages'''
> +	verify_var(d)	
> +	if not_tarball(d):
> +		return
> +	copyleft_compliance = d.getVar('COPYLEFT_COMPLIANCE', True)
> +	if copyleft_compliance is None:
> +		archive_sources_patches(d,middle_name)
> +	elif copyleft_compliance.upper() == 'TRUE' and archive_copyleft(d):
> +		archive_sources_patches(d,middle_name)

I'm not sure I understand what is happening in this patch.  The way I read it:

If COPYLEFT_COMPLIANCE is -not- set, then we archive..  otherwise if it is set 
(true), and the copyleft item was inherited we also archive?

But I'm not sure I understand why this set of checks.


...

> +def dumpdata(d):
> +	'''dump environment to "${P}-${PR}.showdata.dump" including all kinds of varibale and functions when running a task'''

Simple typo above, should be "variable".

...

> +# This functions prepare for archiving "linux-yocto" because this package create directory 's' before do_patch instead of after do_unpack.
> +# This is special control for archiving linux-yocto only.
> +python do_archive_linux_yocto(){
> +	s = d.getVar('S', True)
> +	if 'linux-yocto' in s:
> +		source_tar_name = archive_sources(d,'')
> +	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 'SRPM':
> +		move_tarball_deploy(d,[source_tar_name,''])
> +}

Is there something we can change in linux-yocto to make the standard methods 
work?  I'm hesitant to put special logic in the archiver class for a single 
package.  (Note, I'm more likely to think it's reasonable for the kernel's 
package then a random userspace package!)

> +do_kernel_checkout[postfuncs] += "do_archive_linux_yocto "
> +

Is the real issue that we want to run something just before do_patch, but we 
also want to let any arbitrary tasks between do_unpack and do_patch to run 
first?  Is there an alternative way to state this?  [or at least detect a 
situation where we haven't waited?]

> +# remove tarball for sources, patches and logs after creating srpm.
> +python do_remove_tarball(){
> +	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
> +		work_dir = d.getVar('WORKDIR', True)
> +		os.chdir(work_dir)
> +		for file in os.listdir(os.getcwd()):
> +			if '.tar.gz' in file:
> +				os.remove(file)
> +}
> +do_remove_taball[deptask] = "do_archive_scripts_logs"
> +do_package_write_rpm[postfuncs] += "do_remove_tarball "

Finally do we really need to remove the tarball?  It can likely just stay 
around, when the user wipes out/cleans the WORKDIR it will go away on it's own?

--Mark

Re: [PATCH 3/3] archiver_configure.bbclass: configure the content for archiving package

[Mark Hatle]

A few quick comments to match up with the patch 2/3 comments..

On 3/3/12 4:54 AM, Xiaofeng Yan wrote:
> From: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>
> This is configuration file to point to what content a archive package
> should include in the different stage of task by pointing to different variable and
> stage.
>
> [YOCTO #1977]
>
> Signed-off-by: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>
> Signed-off-by: Xiaofeng Yan<xiaofeng.yan@windriver.com>
> ---
>   meta/classes/archiver_configure.bbclass |   35 +++++++++++++++++++++++++++++++
>   1 files changed, 35 insertions(+), 0 deletions(-)
>   create mode 100644 meta/classes/archiver_configure.bbclass
>
> diff --git a/meta/classes/archiver_configure.bbclass b/meta/classes/archiver_configure.bbclass
> new file mode 100644
> index 0000000..c9892d3
> --- /dev/null
> +++ b/meta/classes/archiver_configure.bbclass

This seems odd that configuration items would be in a .bbclass file.  I would 
have expected the configuration to inherit the archiver, and then either a 
default set of values be specified in the archive.bbclass or in the bitbake.conf 
(or similar)....

As for the defaults below, I think they are fine....

> @@ -0,0 +1,35 @@
> +# This file is for getting tarball in different stage for sources, patches, and logs by configuring
> +# the following variable and tasks
> +
> +inherit archiver
> +
> +# SOURCE_ARCHIVE_PACKAGE_TYPE = {'tar','srpm'}
> +SOURCE_ARCHIVE_PACKAGE_TYPE = 'tar'
> +
> +# SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = {'logs_with_scripts', 'logs'}
> +# String 'logs_with_scripts' include temp directory and .bb and .inc file
> +# String 'logs' only include temp
> +SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = 'logs_with_scripts'
> +
> +# PATCHES_ARCHIVE_WITH_SERIES = {'true', 'false'}
> +# Strings 'true' means that patches including series files(series + non-applying)
> +# String 'false' means that no series and only archive applying patches
> +PATCHES_ARCHIVE_WITH_SERIES = 'true'
> +
> +# Archive packages for copy-left
> +#COPYLEFT_COMPLIANCE = 'true'
> +
> +# Open this item when you want to get original sources tarball with patches
> +do_unpack[postfuncs] += "do_archive_original_sources_patches "

Instead of do_unpack[postfuncs] would a do_patch[prefuncs] be better?  This 
would ensure that do_unpack and all other tasks between do_unpack and do_patch 
were run first... then do we can archive the results..

> +
> +# Open this item when you want to get tarball for patched sources including patches
> +#do_patch[postfuncs] += "do_archive_patched_sources "

Perhaps the same here between do_patch and do_configure...

--Mark

> +
> +# Open this item when you want to get tarball for configured sources including patches
> +#do_configure[postfuncs] += "do_archive_configured_sources "
> +
> +# Open this item when you want to get tarball for logs tarball
> +do_package_write_rpm[prefuncs] += "do_archive_scripts_logs "
> +
> +# Get dump date and create diff file
> +do_package_write_rpm[postfuncs] += "do_dumpdata_create_diff_gz "
> -- 1.7.0.4 _______________________________________________ Openembedded-core
> mailing list Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [PATCH 3/3] archiver_configure.bbclass: configure the content for archiving package

[Xiaofeng Yan]

On 2012年03月13日 02:58, Mark Hatle wrote:
> A few quick comments to match up with the patch 2/3 comments..
>
> On 3/3/12 4:54 AM, Xiaofeng Yan wrote:
>> From: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>>
>> This is configuration file to point to what content a archive package
>> should include in the different stage of task by pointing to 
>> different variable and
>> stage.
>>
>> [YOCTO #1977]
>>
>> Signed-off-by: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>>
>> Signed-off-by: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>> ---
>> meta/classes/archiver_configure.bbclass | 35 
>> +++++++++++++++++++++++++++++++
>> 1 files changed, 35 insertions(+), 0 deletions(-)
>> create mode 100644 meta/classes/archiver_configure.bbclass
>>
>> diff --git a/meta/classes/archiver_configure.bbclass 
>> b/meta/classes/archiver_configure.bbclass
>> new file mode 100644
>> index 0000000..c9892d3
>> --- /dev/null
>> +++ b/meta/classes/archiver_configure.bbclass
>
> This seems odd that configuration items would be in a .bbclass file. I 
> would have expected the configuration to inherit the archiver, and 
> then either a default set of values be specified in the 
> archive.bbclass or in the bitbake.conf (or similar)....
>
> As for the defaults below, I think they are fine....
>
I will define a default set in local.conf.sample in V2
>> @@ -0,0 +1,35 @@
>> +# This file is for getting tarball in different stage for sources, 
>> patches, and logs by configuring
>> +# the following variable and tasks
>> +
>> +inherit archiver
>> +
>> +# SOURCE_ARCHIVE_PACKAGE_TYPE = {'tar','srpm'}
>> +SOURCE_ARCHIVE_PACKAGE_TYPE = 'tar'
>> +
>> +# SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = {'logs_with_scripts', 'logs'}
>> +# String 'logs_with_scripts' include temp directory and .bb and .inc 
>> file
>> +# String 'logs' only include temp
>> +SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = 'logs_with_scripts'
>> +
>> +# PATCHES_ARCHIVE_WITH_SERIES = {'true', 'false'}
>> +# Strings 'true' means that patches including series files(series + 
>> non-applying)
>> +# String 'false' means that no series and only archive applying patches
>> +PATCHES_ARCHIVE_WITH_SERIES = 'true'
>> +
>> +# Archive packages for copy-left
>> +#COPYLEFT_COMPLIANCE = 'true'
>> +
>> +# Open this item when you want to get original sources tarball with 
>> patches
>> +do_unpack[postfuncs] += "do_archive_original_sources_patches "
>
> Instead of do_unpack[postfuncs] would a do_patch[prefuncs] be better? 
> This would ensure that do_unpack and all other tasks between do_unpack 
> and do_patch were run first... then do we can archive the results..
>
Some bb files do some chages after do_unpack before do_patch. The 
following bb files are the case. Perhaps some other bb files could 
change some stuff before do_patch. So I archive sources codes in 
do_unpack[postfuncs].

meta$ grep "do_unpack_append" . -nr
./recipes-core/eglibc/eglibc_2.15.bb:76:do_unpack_append() {
./recipes-core/eglibc/eglibc_2.13.bb:73:do_unpack_append() {
./recipes-sato/web/web_git.bb:19:do_unpack_append () {
./recipes-extended/ltp/ltp_20120104.bb:46:do_unpack_append() {



>> +
>> +# Open this item when you want to get tarball for patched sources 
>> including patches
>> +#do_patch[postfuncs] += "do_archive_patched_sources "
>
> Perhaps the same here between do_patch and do_configure...
>
> --Mark
>
>> +
>> +# Open this item when you want to get tarball for configured sources 
>> including patches
>> +#do_configure[postfuncs] += "do_archive_configured_sources "
>> +
>> +# Open this item when you want to get tarball for logs tarball
>> +do_package_write_rpm[prefuncs] += "do_archive_scripts_logs "
>> +
>> +# Get dump date and create diff file
>> +do_package_write_rpm[postfuncs] += "do_dumpdata_create_diff_gz "
>> -- 1.7.0.4 _______________________________________________ 
>> Openembedded-core
>> mailing list Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [PATCH 2/3] archiver.bbclass: archive sources, patches, logs to tarball

[Xiaofeng Yan]

Hi Mark,

Thanks for your detailed comment.
On 2012年03月13日 02:54, Mark Hatle wrote:
> Apologies for the very late review... see comments below. All in all 
> it looks good. But I have not applied it and tried it yet.
>
> On 3/3/12 4:54 AM, Xiaofeng Yan wrote:
>> From: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>>
>> Support the following functions in this bbclass:
>>
>> 1 Archive sources in ${S} in the different stage to tarball 
>> (do_unpack,do_patch,do_configure).
>> 2 Archive patches including series to tarball
>> 3 Archive logs including scripts (.bb and .inc files) to tarball
>> 4 dump environment resources which show all variable and functions to be
>> used to xxx.showdata.dump when running a task
>> 5 dump all content in 's' including patches to file xxx.diff.gz
>>
>> All of tarballs will be deployed to ${DEPLOY_DIR}/sources/
>>
>> [#YOCTO 1977]
>>
>> Signed-off-by: Xiaofeng Yan<xiaofeng.yan@windriver.com>
>> ---
>> meta/classes/archiver.bbclass | 393 
>> +++++++++++++++++++++++++++++++++++++++++
>> 1 files changed, 393 insertions(+), 0 deletions(-)
>> create mode 100644 meta/classes/archiver.bbclass
>>
> ...
>
>> +def verify_var(d):
>> + '''check the type for archiving package('tar' or 'srpm')'''
>> + try:
>> + if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 
>> d.getVar('ARCHIVE_TYPE', True).split():
>> + raise AttributeError
>> + except AttributeError:
>> + bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" is \'tar\' or \'srpm\', 
>> no other types")
>
> Error message should be something like:
>
> bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" should be \'tar\' or '\srpm\'")
>
Thanks for correcting my grammar.

> ...
>
>> +def archive_sources_patches(d,middle_name):
>> + '''archive sources and patches to tarball. middle_name will append 
>> strings ${middle_name} to ${PR} as middle name. for example, 
>> zlib-1.4.6-prepatch(middle_name).tar.gz '''
>> + verify_var(d)
>> + if not_tarball(d):
>> + return
>> +
>> + source_tar_name = archive_sources(d,middle_name)
>> + if middle_name == "prepatch":
>> + if d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'TRUE':
>> + patch_tar_name = select_archive_patches(d,"all")
>> + elif d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'FALSE':
>> + patch_tar_name = select_archive_patches(d,"applying")
>> + else:
>> + bb.fatal("Please define 'PATCHES_ARCHIVE_WITH_SERIES' is strings 
>> 'True' or 'False' ")
>
> Instead of an explicit TRUE/FALSE setting, does it make sense for one 
> to be the default? I'd suspect in this case "True" is the better 
> default, but I'm not completely sure.
>
I can move this setting to local.conf.sample and TRUE is default setting.
>> + else:
>> + patch_tar_name = ''
>> +
>> + if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 
>> 'SRPM':
>> + move_tarball_deploy(d,[source_tar_name,patch_tar_name])
>> +
>> +def archive_sources_patches_logs_copyleft(d,middle_name):
>> + '''archive source, patches and logs according to the variable 
>> "COPYLEFT_COMPLIANCE", If this variable is 'True', then archive the 
>> packages for copy-left, or else archive all packages'''
>> + verify_var(d)
>> + if not_tarball(d):
>> + return
>> + copyleft_compliance = d.getVar('COPYLEFT_COMPLIANCE', True)
>> + if copyleft_compliance is None:
>> + archive_sources_patches(d,middle_name)
>> + elif copyleft_compliance.upper() == 'TRUE' and archive_copyleft(d):
>> + archive_sources_patches(d,middle_name)
>
> I'm not sure I understand what is happening in this patch. The way I 
> read it:
>
> If COPYLEFT_COMPLIANCE is -not- set, then we archive.. otherwise if it 
> is set (true), and the copyleft item was inherited we also archive?
>
If COPYLEFT_COMPLIANCE is -not- set, then we archive all resources 
without considering copyrights problem which is typed in 
copyleft_compliance.bbclass.
otherwise if it is set (true), and the copyleft item was inherited we 
only archive packages which be allowed by copyleft.
> But I'm not sure I understand why this set of checks.
>
>
> ...
>
>> +def dumpdata(d):
>> + '''dump environment to "${P}-${PR}.showdata.dump" including all 
>> kinds of varibale and functions when running a task'''
>
> Simple typo above, should be "variable".
>
Thanks for your detailed reviewing.
> ...
>
>> +# This functions prepare for archiving "linux-yocto" because this 
>> package create directory 's' before do_patch instead of after do_unpack.
>> +# This is special control for archiving linux-yocto only.
>> +python do_archive_linux_yocto(){
>> + s = d.getVar('S', True)
>> + if 'linux-yocto' in s:
>> + source_tar_name = archive_sources(d,'')
>> + if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 
>> 'SRPM':
>> + move_tarball_deploy(d,[source_tar_name,''])
>> +}
>
> Is there something we can change in linux-yocto to make the standard 
> methods work? I'm hesitant to put special logic in the archiver class 
> for a single package. (Note, I'm more likely to think it's reasonable 
> for the kernel's package then a random userspace package!)
>
because the 's' is created after do_kernel_checkout (after do_unpack 
before do_patch), so we will not get original sources for kernel if we 
archive 's' after do_unpack.
Some bb files do some changes after do_unpack before do_patch actually . 
The following bb files are the case. Perhaps some other bb files could 
also change some stuff before do_patch in future. So I archive sources 
codes in do_unpack[postfuncs] instead of do_patch[prefuncs].

meta$ grep "do_unpack_append" . -nr
./recipes-core/eglibc/eglibc_2.15.bb:76:do_unpack_append() {
./recipes-core/eglibc/eglibc_2.13.bb:73:do_unpack_append() {
./recipes-sato/web/web_git.bb:19:do_unpack_append () {
./recipes-extended/ltp/ltp_20120104.bb:46:do_unpack_append() {

>> +do_kernel_checkout[postfuncs] += "do_archive_linux_yocto "
>> +
>
> Is the real issue that we want to run something just before do_patch, 
> but we also want to let any arbitrary tasks between do_unpack and 
> do_patch to run first? Is there an alternative way to state this? [or 
> at least detect a situation where we haven't waited?]
>
My reasons why doing this is same as above
>> +# remove tarball for sources, patches and logs after creating srpm.
>> +python do_remove_tarball(){
>> + if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
>> + work_dir = d.getVar('WORKDIR', True)
>> + os.chdir(work_dir)
>> + for file in os.listdir(os.getcwd()):
>> + if '.tar.gz' in file:
>> + os.remove(file)
>> +}
>> +do_remove_taball[deptask] = "do_archive_scripts_logs"
>> +do_package_write_rpm[postfuncs] += "do_remove_tarball "
>
> Finally do we really need to remove the tarball? It can likely just 
> stay around, when the user wipes out/cleans the WORKDIR it will go 
> away on it's own?
>
These archiving packages will waste much disk space, so I will remove 
them after srpm package generating. but that just seemed excessive, right?

Thanks for your comment again.
> --Mark
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>