From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail1.windriver.com (mail1.windriver.com [147.11.146.13]) by mail.openembedded.org (Postfix) with ESMTP id 106A2766E5 for ; Fri, 7 Aug 2015 02:34:28 +0000 (UTC) Received: from ALA-HCB.corp.ad.wrs.com (ala-hcb.corp.ad.wrs.com [147.11.189.41]) by mail1.windriver.com (8.15.1/8.15.1) with ESMTPS id t772YSaS016464 (version=TLSv1 cipher=AES128-SHA bits=128 verify=FAIL) for ; Thu, 6 Aug 2015 19:34:28 -0700 (PDT) Received: from [128.224.162.200] (128.224.162.200) by ALA-HCB.corp.ad.wrs.com (147.11.189.41) with Microsoft SMTP Server id 14.3.235.1; Thu, 6 Aug 2015 19:34:28 -0700 Message-ID: <55C41932.8010005@windriver.com> Date: Fri, 7 Aug 2015 10:34:26 +0800 From: Robert Yang User-Agent: Mozilla/5.0 (X11; Linux i686; rv:31.0) Gecko/20100101 Thunderbird/31.8.0 MIME-Version: 1.0 To: Mark Hatle , Patches and discussions about the oe-core layer References: <55C1B11A.7060606@windriver.com> <55C20B52.9090400@windriver.com> <55C2C773.6030506@windriver.com> <55C3CA37.5070703@windriver.com> In-Reply-To: <55C3CA37.5070703@windriver.com> Subject: Re: About multilib packages rpm pkg name X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Aug 2015 02:34:29 -0000 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit On 08/07/2015 04:57 AM, Mark Hatle wrote: > On 8/5/15 9:33 PM, Robert Yang wrote: >> >> Hi Mark, >> >> Thank you very much, this makes things more clear now, I've tried >> to change packagegroup-core-standalone-sdk-target's PACKAGE_ARCH >> to MACHINE_ARCH or TUNE_PKGARCH, it didn't work (the rdepends lib32 >> are still not installed), I will try to use read_subpkgdata() to get >> the required pkgs and fix the problem. > > During package creation we strip off the multilib prefix (since it's not > needed). Then during dependency calculations things often favor a 'shortest > path' to the dependency which may not give you the 32-bit version you are expecting. > > What I'd like to see is a patch to smart where you can say "Use this ARCH or > word size, unless explicitly selected". That would help eliminate -some- of > these issues. Thanks, sounds good, I will try smart later, at the moment, I made a simple patch to make the populate_sdk pull in multilib toolchains works, there are still other issues, I will fix them and send patches. diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass index a9e9bd7..c2491ea 100644 --- a/meta/classes/populate_sdk_base.bbclass +++ b/meta/classes/populate_sdk_base.bbclass @@ -83,8 +83,40 @@ POPULATE_SDK_POST_HOST_COMMAND_append = " write_host_sdk_manifest; " fakeroot python do_populate_sdk() { from oe.sdk import populate_sdk from oe.manifest import create_manifest, Manifest + import oe.packagedata pn = d.getVar('PN', True) + # Help rpm to pull in multilib requires since it may not pull in + # them correctly, for example, + # lib32-packagegroup-core-standalone-sdk-target requires + # lib32-libc6, rpm may pull in libc6 rather than lib32-libc6, but + # directly use + # TOOLCHAIN_TARGET_TASK_append = " lib32-libc6" works well. + for pkg in (d.getVar('TOOLCHAIN_TARGET_TASK', True) or "").split(): + sub_data = oe.packagedata.read_subpkgdata(pkg, d) + sub_rdep = sub_data.get("RDEPENDS_" + pkg) + if not sub_rdep: + continue + done = sub_rdep.split() + next = done + # Find all the rdepends on the dependency chain + while next: + new = [] + for sub_pkg in next: + sub_data = oe.packagedata.read_subpkgdata(sub_pkg, d) + sub_pkg_rdep = sub_data.get("RDEPENDS_" + sub_pkg) + if not sub_pkg_rdep: + continue + for p in sub_pkg_rdep.split(): + if p in done: + continue + if not p.startswith('(') and \ + oe.packagedata.has_subpkgdata(p, d): + # It's a new rdep + done.append(p) + new.append(p) + next = new + d.appendVar('TOOLCHAIN_TARGET_TASK', ' ' + " ".join(done)) // Robert > > The other side of the issue then is for things that do have explicit word size > requirements (like gcc and some of the build tools) we need to fix the packages > to have additions RDEPENDS and RPROVIDES that dictate the word size they need > and the word size they provide. libpam already does this, and it resolved the > issues there. > > --Mark > >> // Robert >> >> On 08/05/2015 09:10 PM, Mark Hatle wrote: >>> On 8/5/15 1:45 AM, Robert Yang wrote: >>>> >>>> Hello, >>>> >>>> For the multilib package like lib32-bash, its rpm package name >>>> is bash-4.3.30-r0.lib32_x86.rpm, but for ipk and deb, its name is >>>> lib32-bash.ipk and lib32-bash.deb, there is a side effect for >>>> the naming of rpm, for example, if packagegroup-xx rdepends on bash, >>>> the lib32-packagegroup-xx should rdepend on lib32-bash, but >>>> the its rpm pkg knows nothing about lib32-bash, so it still rdepends >>>> on bash, this causes 32bit toolchain libs not installed when export >>>> multilib sdk, for example: >>>> >>>> Configured multilib: >>>> MACHINE ?= "qemux86-64" >>>> require conf/multilib.conf >>>> MULTILIBS = "multilib:lib32" >>>> DEFAULTTUNE_virtclass-multilib-lib32 = "x86" >>>> >>>> $ bitbake core-image-minimal -cpopulate_sdk >>>> >>>> When PACKAGE_CLASSES = "package_rpm", the lib32-libgcc, >>>> lib32-libgcc-dev or other lib32-xx packages are not installed, >>>> so that the 32bit toolchain doesn't work in SDK, this is because >>>> lib32-packagegroup-core-standalone-sdk-target can't pull in the >>>> lib32-xx correctly. >>>> >>>> When PACKAGE_CLASSES = "package_ipk", it works well. >>>> >>>> Does anyone why we don't use the name like lib32-bash.rpm, please ? >>>> Can we use lib32-bash, please ? >>> >>> Because for something like bash, there is absolutely no reason to choose a >>> 64-bit or 32-bit version over the other. They have equivalent functionality for >>> the end user, and only one version can be installed at a time. >>> >>> This is the RPM behavior that is being copied from Red Hat and other places, and >>> what our customers are expecting. >>> >>> RPM supports multiple packages, with different arch fields, being installed at >>> the same time. ipkg and deb have a limitation that the package name MUST be >>> unique. So this is the first reason why we have different style names. >>> >>> The idea here being that you can build the .i586 packages once, and re-use them >>> on an i586 system and an x86-64 system. (Note, this doesn't work in practice in >>> the Yocto Project due to sstate issues... but that is the underlying design from >>> a package management perspective.) >>> >>> Dependency processing is the second reason, see below for an explanation of the >>> difference and how it affects us. >>> >>> There are different types of dependencies within the system. >>> >>> * Package level dependency >>> This is of the format 'bash' or 'lib32-bash'. The system requires a specific >>> package. All three package formats support this. >>> >>> * Virtual dependency >>> This is similar, in that the format can look like a package, or be something >>> like 'virtual/foo'. Again all three package formats support this. >>> >>> * Library SONAME >>> This is a correlation of the library SONAME into the dependency set, such as >>> 'libz.so(64)'. This is pulled directly from any ELF binaries in the system >>> during processing and added as a direct dependency to other shared libraries. >>> It is significantly more exact then the package or virtual dependencies above. >>> HOWEVER, it only works in the RPM case. >>> >>> This is the second reason why deb and ipkg need a unique 'lib32' style name for >>> things. They don't have a mechanism for a package to say "I need the 64-bit >>> version of libz.so" to run properly, while that is built into RPM itself. >>> >>> * Filename dependency >>> These are dependencies that start with '/'. This is unique to RPM as well, and >>> allows RPM to express dependencies on '/bin/bash', vs deb/ipkg having >>> dependencies on 'bash'. The difference is that it puts a hard marker in place >>> that if 'bash' doesn't provide '/bin/bash', it's not enough. >>> >>> This doesn't directly affect the naming, but occasionally has some indirect >>> adjustments on things. >>> >>> >>> So when building a system using RPM as the package type, the rootfs construction >>> may be slightly different because the package manager rules are different. >>> >>> RPM (smart) will look at the overall dependency tree and attempt to pick the >>> shortest path so that all of the dependencies are correct. You can 'weight' the >>> dependency resolver to favor one style of package over the other. This is a >>> place in smart where we likely need to change the behavior to make it a bit more >>> friendly for the Yocto Project. >>> >>> The dependency generator is already configured to weigh a package of the same >>> type (arch) as a bit higher then others. The problem is that a 'noarch' (all) >>> package is of a different type then say i586. So when an all package says "give >>> me bash". There is no 'bash.all', so it looks at the list and finds 'bash' of >>> the highest priority, IF it had not previously been selected by something else. >>> (How would it have previously been selected? If the image had selected >>> 'lib32-bash' explicitly, or the more likely case, a lib32-* package that >>> requires bash had been selected by the image, and there were not explicit >>> requirements for 'bash' [main lib version]). >>> >>> >>> RPM itself has a setting that defines which ABI type becomes the default if you >>> attempt to choose multiple packages that have the same executables listed. The >>> setting here is: >>> >>> # Set RPM_PREFER_ELF_ARCH to configure preferred ABI when using rpm packaging >>> # backend to generate a rootfs, choices are: >>> # 1: ELF32 wins >>> # 2: ELF64 wins >>> # 4: ELF64 N32 wins (for mips64 or mips64el only) >>> >>> Note, this doesn't prevent multiple packages of the same name, different arch >>> from being installed -- what this does is selects which one wins. When not set, >>> I believe it's last-in wins. >>> >>> >>> This setting should also help weigh the smart dep generator, but I'm not sure it >>> does. (Pretty easy to test though.. set the value, build a multilib rootfs, and >>> select something (all package) that requires 'bash' and see which version is >>> selected.) >>> >>> But for most things in the system this really doesn't matter at all. For things >>> where a 32-bit or 64-bit does matter (executable and/or libraries that are not >>> implicitly linked elsewhere), we really do need to add dependencies into the >>> system so that the correct set of items are selected. >>> >>> PAM, Perl, Python, and the on-target toolchains are things that adding a few >>> runtime dependencies are needed. Basically anything in the system that may have >>> runtime loadable libraries of a specific ABI, using something like dlopen. >>> (Remember SONAME dependencies are automatic as mentioned above.) >>> >>> These new runtime dependencies will not harm deb/ipkg resolution (and may help >>> it), but they give RPM the additional information it needs to select the >>> explicit set that is required. For example PAM: >>> >>> RPROVIDES_${PN} += "${PN}-${libpam_suffix}" >>> RPROVIDES_${PN}-runtime += "${PN}-runtime-${libpam_suffix}" >>> >>> RDEPENDS_${PN}-runtime = "${PN}-${libpam_suffix} \ >>> ${MLPREFIX}pam-plugin-deny-${libpam_suffix} \ >>> ${MLPREFIX}pam-plugin-permit-${libpam_suffix} \ >>> ${MLPREFIX}pam-plugin-warn-${libpam_suffix} \ >>> ${MLPREFIX}pam-plugin-unix-${libpam_suffix} \ >>> " >>> >>> (other code later constructs the RPROVIDES for the plugins as well) >>> >>> The MLPREFIX will be filtered off when RPM constructs it's packages, as it >>> doesn't want/need them. However the suffix won't be filtered. >>> >>> libpam_suffix = "suffix${@get_multilib_bit(d)}" >>> >>> This way the -runtime package ensures the right other packages (with their >>> specific PROVIDES sufffixes) are included in all cases. >>> >>> This is a unique case, but should be pretty easy to help fix. >>> >>> For the case where you want to specify 'lib32-packagegroup' and expect to ONLY >>> get 32-bit packages... While we can't ensure it with the RPM mechanism, we can >>> weigh things to more likely happens -if- the lib32-packagegroup is of the same >>> .arch as the packages it's installing, instead of being 'all' packages. >>> >>> Hopefully this explains a bit more how all of these items fit together, and the >>> choice made when RPM was implemented to follow more of an expected multilib >>> workflow. (This won't affect you if you don't use multilibs in your design!) >>> >>> --Mark >>> >>> > > >