Re: [PATCH 6/6 v2] package_rpm: Allow translation of requirement to package name

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Tue, 2013-11-12 at 19:59 -0600, Mark Hatle wrote:
> In the translate oe to smart function, we only translated package names.
> However, it's allowed that people can put in a dependency name in the
> IMAGE_INSTALL.

For what its worth, I find this description hard to understand, I think
as you leave the keyword "provide" right until the end. A mention of
RPROVIDES earlier here would help a lot.

>   So on a failure to translate a package name, we fall back
> and attempt to resolve based on a package's provide.
> 
> Note: it may be possible to generate an unsolvable install solution.  If the
> dependency is provided by one or more things that conflict with something else
> set to be installed.  We can't determine this until smart is run.
> 
> If this occurs, file a bug and we'll have to identify a way to deal with the
> RCONFLICTS and RREPLACES.  As a workaround replace the conflict REQUIRES with
> actual package names.
> 
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
>  meta/classes/package_rpm.bbclass | 79 ++++++++++++++++++++++++++++++++++++++--
>  1 file changed, 75 insertions(+), 4 deletions(-)
> 
> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
> index 31265d9..9fe0e6c 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -166,6 +166,7 @@ translate_oe_to_smart() {
>  	[ ! -e ${target_rootfs}/install/tmp/fullpkglist.query ] && smart --data-dir=${target_rootfs}/var/lib/smart query --output ${target_rootfs}/install/tmp/fullpkglist.query
>  
>  	pkgs_to_install=""
> +	not_found=""
>  	for pkg in "$@" ; do
>  		new_pkg="$pkg"
>  		if [ -z "$sdk_mode" ]; then
> @@ -184,7 +185,6 @@ translate_oe_to_smart() {
>  				fi
>  				subst=${pkg#${mlib}-}
>  				if [ "$subst" != "$pkg" ]; then
> -					feeds=$@
>  					while [ -n "$1" ]; do
>  						arch="$1"
>  						arch=`echo "$arch" | tr - _`
> @@ -197,6 +197,76 @@ translate_oe_to_smart() {
>  					done
>  					if [ "$pkg" = "$new_pkg" ]; then
>  						# Failed to translate, package not found!
> +						not_found="$not_found $pkg"
> +						continue
> +					fi
> +				fi
> +			done
> +		fi
> +		# Apparently not a multilib package...
> +		if [ "$pkg" = "$new_pkg" ]; then
> +			default_archs_fixed=`echo "$default_archs" | tr - _`
> +			for arch in $default_archs_fixed ; do
> +				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$' ${target_rootfs}/install/tmp/fullpkglist.query ; then
> +					new_pkg="$pkg@$arch"
> +					# First found is best match
> +					break
> +				fi
> +			done
> +			if [ "$pkg" = "$new_pkg" ]; then
> +				# Failed to translate, package not found!
> +				not_found="$not_found $pkg"
> +				continue
> +			fi
> +		fi
> +		#echo "$pkg -> $new_pkg" >&2
> +		pkgs_to_install="${pkgs_to_install} ${new_pkg}"
> +	done
> +
> +	# Parse the not_found items and see if they were dependencies (RPROVIDES)
> +	# Follow the parsing example above...
> +	for pkg in $not_found ; do
> +		new_pkg="$pkg"
> +		smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$pkg --output ${target_rootfs}/install/tmp/provide.query
> +		grep '^[^@ ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' > ${target_rootfs}/install/tmp/provide.query.list || :
> +		prov=`echo $pkgs_to_install | xargs -n 1 echo | grep -f ${target_rootfs}/install/tmp/provide.query.list || :`
> +		if [ -n "$prov" ]; then
> +			# Nothing to do, already in the list
> +			#echo "Skipping $pkg -> $prov, already in install set" >&2
> +			continue
> +		fi
> +		if [ -z "$sdk_mode" ]; then
> +			for i in ${MULTILIB_PREFIX_LIST} ; do
> +				old_IFS="$IFS"
> +				IFS=":"
> +				set $i
> +				IFS="$old_IFS"
> +				mlib="$1"
> +				shift
> +				if [ "$mlib" = "default" ]; then
> +					if [ -z "$default_archs" ]; then
> +						default_archs=$@
> +					fi
> +					continue
> +				fi
> +				subst=${pkg#${mlib}-}
> +				if [ "$subst" != "$pkg" ]; then
> +					feeds=$@
> +					smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$subst --output ${target_rootfs}/install/tmp/provide.query
> +					grep '^[^@ ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' > ${target_rootfs}/install/tmp/provide.query.list || :
> +					while [ -n "$1" ]; do
> +						arch="$1"
> +						arch=`echo "$arch" | tr - _`
> +						shift
> +						# Select first found, we don't know if one is better then another...
> +						prov=`grep '^[^@ ]*@'$arch'$' ${target_rootfs}/install/tmp/provide.query.list | head -n 1`
> +						if [ -n "$prov" ]; then
> +							new_pkg=$prov
> +							break
> +						fi
> +					done
> +					if [ "$pkg" = "$new_pkg" ]; then
> +						# Failed to translate, package not found!
>  						echo "$attemptonly: $pkg not found in the $mlib feeds ($feeds)." >&2
>  						if [ "$attemptonly" = "Error" ]; then
>  							exit 1
> @@ -210,9 +280,10 @@ translate_oe_to_smart() {
>  		if [ "$pkg" = "$new_pkg" ]; then
>  			default_archs_fixed=`echo "$default_archs" | tr - _`
>  			for arch in $default_archs_fixed ; do
> -				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$' ${target_rootfs}/install/tmp/fullpkglist.query ; then
> -					new_pkg="$pkg@$arch"
> -					# First found is best match
> +				# Select first found, we don't know if one is better then another...
> +				prov=`grep '^[^@ ]*@'$arch'$' ${target_rootfs}/install/tmp/provide.query.list | head -n 1`
> +				if [ -n "$prov" ]; then
> +					new_pkg=$prov
>  					break
>  				fi
>  			done

This 'shell calling rpm commands' code with all the mangling is near
impossible to read/understand. This patch just makes the mess more
complicated and harder to follow.

What is the plan for turning this into something better?

My instinct says more functions and a language better at data
manipulation (python instead of shell) might start to resolve this.

One of the reasons for using smart was that we could probably enhance it
to better suit our needs. Can we consider that too?

I really don't want to see more patches like this one piling on top of
already crumbling foundations.

Cheers,

Richard