Re: [PATCH 6/6 v2] package_rpm: Allow translation of requirement to package name

[Mark Hatle <mark.hatle@windriver.com>]

On 11/15/13, 8:02 AM, Richard Purdie wrote:
> On Tue, 2013-11-12 at 19:59 -0600, Mark Hatle wrote:
>> In the translate oe to smart function, we only translated package names.
>> However, it's allowed that people can put in a dependency name in the
>> IMAGE_INSTALL.
>
> For what its worth, I find this description hard to understand, I think
> as you leave the keyword "provide" right until the end. A mention of
> RPROVIDES earlier here would help a lot.

Ok.

>>    So on a failure to translate a package name, we fall back
>> and attempt to resolve based on a package's provide.
>>
>> Note: it may be possible to generate an unsolvable install solution.  If the
>> dependency is provided by one or more things that conflict with something else
>> set to be installed.  We can't determine this until smart is run.
>>
>> If this occurs, file a bug and we'll have to identify a way to deal with the
>> RCONFLICTS and RREPLACES.  As a workaround replace the conflict REQUIRES with
>> actual package names.
>>
>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>> ---
>>   meta/classes/package_rpm.bbclass | 79 ++++++++++++++++++++++++++++++++++++++--
>>   1 file changed, 75 insertions(+), 4 deletions(-)
>>
>> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
>> index 31265d9..9fe0e6c 100644
>> --- a/meta/classes/package_rpm.bbclass
>> +++ b/meta/classes/package_rpm.bbclass
>> @@ -166,6 +166,7 @@ translate_oe_to_smart() {
>>   	[ ! -e ${target_rootfs}/install/tmp/fullpkglist.query ] && smart --data-dir=${target_rootfs}/var/lib/smart query --output ${target_rootfs}/install/tmp/fullpkglist.query
>>
>>   	pkgs_to_install=""
>> +	not_found=""
>>   	for pkg in "$@" ; do
>>   		new_pkg="$pkg"
>>   		if [ -z "$sdk_mode" ]; then
>> @@ -184,7 +185,6 @@ translate_oe_to_smart() {
>>   				fi
>>   				subst=${pkg#${mlib}-}
>>   				if [ "$subst" != "$pkg" ]; then
>> -					feeds=$@
>>   					while [ -n "$1" ]; do
>>   						arch="$1"
>>   						arch=`echo "$arch" | tr - _`
>> @@ -197,6 +197,76 @@ translate_oe_to_smart() {
>>   					done
>>   					if [ "$pkg" = "$new_pkg" ]; then
>>   						# Failed to translate, package not found!
>> +						not_found="$not_found $pkg"
>> +						continue
>> +					fi
>> +				fi
>> +			done
>> +		fi
>> +		# Apparently not a multilib package...
>> +		if [ "$pkg" = "$new_pkg" ]; then
>> +			default_archs_fixed=`echo "$default_archs" | tr - _`
>> +			for arch in $default_archs_fixed ; do
>> +				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$' ${target_rootfs}/install/tmp/fullpkglist.query ; then
>> +					new_pkg="$pkg@$arch"
>> +					# First found is best match
>> +					break
>> +				fi
>> +			done
>> +			if [ "$pkg" = "$new_pkg" ]; then
>> +				# Failed to translate, package not found!
>> +				not_found="$not_found $pkg"
>> +				continue
>> +			fi
>> +		fi
>> +		#echo "$pkg -> $new_pkg" >&2
>> +		pkgs_to_install="${pkgs_to_install} ${new_pkg}"
>> +	done
>> +
>> +	# Parse the not_found items and see if they were dependencies (RPROVIDES)
>> +	# Follow the parsing example above...
>> +	for pkg in $not_found ; do
>> +		new_pkg="$pkg"
>> +		smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$pkg --output ${target_rootfs}/install/tmp/provide.query
>> +		grep '^[^@ ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' > ${target_rootfs}/install/tmp/provide.query.list || :
>> +		prov=`echo $pkgs_to_install | xargs -n 1 echo | grep -f ${target_rootfs}/install/tmp/provide.query.list || :`
>> +		if [ -n "$prov" ]; then
>> +			# Nothing to do, already in the list
>> +			#echo "Skipping $pkg -> $prov, already in install set" >&2
>> +			continue
>> +		fi
>> +		if [ -z "$sdk_mode" ]; then
>> +			for i in ${MULTILIB_PREFIX_LIST} ; do
>> +				old_IFS="$IFS"
>> +				IFS=":"
>> +				set $i
>> +				IFS="$old_IFS"
>> +				mlib="$1"
>> +				shift
>> +				if [ "$mlib" = "default" ]; then
>> +					if [ -z "$default_archs" ]; then
>> +						default_archs=$@
>> +					fi
>> +					continue
>> +				fi
>> +				subst=${pkg#${mlib}-}
>> +				if [ "$subst" != "$pkg" ]; then
>> +					feeds=$@
>> +					smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$subst --output ${target_rootfs}/install/tmp/provide.query
>> +					grep '^[^@ ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' > ${target_rootfs}/install/tmp/provide.query.list || :
>> +					while [ -n "$1" ]; do
>> +						arch="$1"
>> +						arch=`echo "$arch" | tr - _`
>> +						shift
>> +						# Select first found, we don't know if one is better then another...
>> +						prov=`grep '^[^@ ]*@'$arch'$' ${target_rootfs}/install/tmp/provide.query.list | head -n 1`
>> +						if [ -n "$prov" ]; then
>> +							new_pkg=$prov
>> +							break
>> +						fi
>> +					done
>> +					if [ "$pkg" = "$new_pkg" ]; then
>> +						# Failed to translate, package not found!
>>   						echo "$attemptonly: $pkg not found in the $mlib feeds ($feeds)." >&2
>>   						if [ "$attemptonly" = "Error" ]; then
>>   							exit 1
>> @@ -210,9 +280,10 @@ translate_oe_to_smart() {
>>   		if [ "$pkg" = "$new_pkg" ]; then
>>   			default_archs_fixed=`echo "$default_archs" | tr - _`
>>   			for arch in $default_archs_fixed ; do
>> -				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$' ${target_rootfs}/install/tmp/fullpkglist.query ; then
>> -					new_pkg="$pkg@$arch"
>> -					# First found is best match
>> +				# Select first found, we don't know if one is better then another...
>> +				prov=`grep '^[^@ ]*@'$arch'$' ${target_rootfs}/install/tmp/provide.query.list | head -n 1`
>> +				if [ -n "$prov" ]; then
>> +					new_pkg=$prov
>>   					break
>>   				fi
>>   			done
>
> This 'shell calling rpm commands' code with all the mangling is near
> impossible to read/understand. This patch just makes the mess more
> complicated and harder to follow.

The root of this is the namespace issue.  In RPM, the architecture (and arch 
compatibility), as well as file contents all provide the namespace and conflict 
resolution.  While in opkg/deb the name of the package itself is where the 
namespace is defined.  (i.e. lib32-...)

The translation is necessary due to this multilib support, otherwise we wouldn't 
need to do any of it.  Somehow we have to translate one namespace to another -- 
and since the translation can't occur until the packages are built, it'd be 
difficult to do it 'earlier'.

> What is the plan for turning this into something better?
>
> My instinct says more functions and a language better at data
> manipulation (python instead of shell) might start to resolve this.
>
> One of the reasons for using smart was that we could probably enhance it
> to better suit our needs. Can we consider that too?

The only thing we can do is build a python (or similar) app that takes the 
inputs of the translation 'namespace' IDs and the corresponding architectures, 
and RPM feed.  Then translate from one to the other.  My biggest concern with 
this translation is to cover all of the corner cases we could start getting more 
and more complex.

> I really don't want to see more patches like this one piling on top of
> already crumbling foundations.

Easiest way to avoid this is to make dependencies in the IMAGE/PACKAGE_INSTALL 
forbidden and do the translation at a higher level generically.  But frankly I 
don't see that happening -- so the solution becomes RPM specific and we end up 
with code like this.  I'm not sure if we can move this code into a python 
function -- or if we need to simply create an external translation program...

As far as plans, frankly I don't have any plans to focus on this work for the 
time being.  There are too many other things I need to get working properly 
first.  If someone else wants to step in and work on a solution, I'm more then 
happy to help them out.

--Mark

> Cheers,
>
> Richard
>