Re(vival?): Missing files in rootfs when adding links /bin ->/usr/bin and /lib -> /usr/lib in recipe base-files

["Jan Lindåker" <jan.lindaker@axis.com>]

> When changing package base-files, so that the /bin directory is soft linked to /usr/bin and the /lib directory is soft linked to /usr/lib, the > resulting image/rootfs directory structure is missing a lot of files, most of which are unrelated to any of the  directories mentioned
> above. E.g. file://etc/fstab is missing. When inspecting the base-files .rpm it contains the file structure​ that is expected. For some
> reason the package install order changes when building rootfs when the links from /lib and /usr are present.
>
> I know too little about Yocto/kernel image building to be able to debug this effectively, since there is no errors reported and the build
> process is quite complex. Any solutions or help with where to continue my investigations would be highly appreciated.
>
> Actions to reproduce:
> I first detected the problem in an adaptation of Yocto, but I have reproduced it in the project described in the quick start:
> http://www.yoctoproject.org/docs/1.8/yocto-project-qs/yocto-project-qs.html
>
> 1. Set up a new Yocto project as described in the link above
> 2. Set up a new layer with a .bbappendfile as follows
> 
> -- ../recepies-linkdir/base-files/base-files_%.bbappend -------------------
> dirs755_remove = "/bin /lib"
> 
> do_install_extra_links() {
>    ln -srvf ${D}/usr/bin ${D}/bin
>    ln -srvf ${D}/usr/lib ${D}/lib
>}
>
> addtask install_extra_links after do_install before do_populate_sysroot do_packag
>
> -- ../meta-linkdir/conf/layer.conf ----------------------------------------
> # We have a conf and classes directory, add to BBPATH
> BBPATH =. "${LAYERDIR}:" 
> # We have a packages directory, add to BBFILES
> BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
>            ${LAYERDIR}/recipes-*/*/*.bbappend"
>
> BBFILE_COLLECTIONS += "linkdir" 
> BBFILE_PATTERN_linkdir := "^${LAYERDIR}/" 
> BBFILE_PRIORITY_linkdir := "6"
>
> ---------------------------------------------------------------------------
>
> 3. Build the software
> $ bitbake core-image-sato
> 
> 4. Inspect the .rpm
> rpm -lvpq ./tmp/work/qemux86-poky-linux/base-files/3.0.14-r89/deploy-rpms/qemux86/base-files-dev-3.0.14-r89.qemux86.rpm
> Note that: /bin -> /usr/bin, /lib -> /user/lib and that /etc/fstab is present
>
> 5. Save the generated file-strucutre
> $ pushd ./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs
> $ LC_ALL=C find . -exec "stat -c '%N' {} \;" > /tmp/yocto_link_fs.txt
> $ popd
>
> 6. Save the build log from rootfs build
> $ cp .tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/temp/log.do_rootfs /tmp/log.do_rootfs.link
>
> 7. To be able to compare with the original filesystem, disable the changes in the .bb-append file, i.e. edit it to look like this:
> -- ../recepies-linkdir/base-files/base-files_%.bbappend -------------------
> # dirs755_remove = "/bin /lib"
>
> do_install_extra_links() {
>    ln -srvf ${D}/usr/bin ${D}/bin
>    ln -srvf ${D}/usr/lib ${D}/lib
> }
>
> # addtask install_extra_links after do_install before do_populate_sysroot do_packag
>
> 8. Rebuild the project
> $ bitbake -c cleanall base-files
> $ bitbake core-image-sato
>
> 9. Inspect the .rpm
> rpm -lvpq ./tmp/work/qemux86-poky-linux/base-files/3.0.14-r89/deploy-rpms/qemux86/base-files-dev-3.0.14-r89.qemux86.rpm
> Note that: /bin and /lib are normal directories and that /etc/fstab is present
>
> 10. Save the generated file-strucutre
> $ pushd ./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs
> $ LC_ALL=C find . -exec "stat -c '%N' {} \;" > /tmp/yocto_orig_fs.txt
> $ popd
>
> 11. Save the build log from rootfs build
> $ cp .tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/temp/log.do_rootfs /tmp/log.do_rootfs.orig
>
> 12. Compare the difference and note that files are exclusively missing from the build where the soft-links where added.
> $  diff /tmp/yocto_orig_fs.txt /tmp/yocto_links_fs.txt
> 2d1
> < './proc'
> 16d14
> < './usr/share/info'
> 3643d3640
> < './usr/share/dict'
> 5068,5070d5064
> < './usr/share/dbus-1/interfaces/org.freedesktop.ConsoleKit.Seat.xml'
> < './usr/share/dbus-1/interfaces/org.freedesktop.ConsoleKit.Session.xml'
> < './usr/share/dbus-1/interfaces/org.freedesktop.ConsoleKit.Manager.xml'
> 5075d5068
> < './usr/share/dbus-1/system-services/org.freedesktop.ConsoleKit.service'
> 5382d5374
> < './usr/share/man'
> 5386d5377
> < './usr/src'
> 5488d5478
> < './usr/bin/ck-launch-session'
> 5603d5592
> < './usr/bin/ck-list-sessions'
> 5674d5662
> < './usr/bin/ck-history'
> 5708d5695
> < './usr/sbin/ck-log-system-restart'
> 5710d5696
> < './usr/sbin/ck-log-system-stop'
> 5758d5743
> < './usr/sbin/ck-log-system-start'
> 5771d5755
> < './usr/sbin/console-kit-daemon'
> 6119d6102
> < './usr/lib/libck-connector.so.0.0.0'
> 6287d6269
> < './usr/lib/libck-connector.so.0' -> 'libck-connector.so.0.0.0'
> 7570,7571d7551
> < './usr/lib/ConsoleKit/scripts/ck-system-stop'
> < './usr/lib/ConsoleKit/scripts/ck-system-restart'
> 7597,7599d7576
> < './usr/lib/consolekit/ck-get-x11-display-device'
> < './usr/lib/consolekit/ck-get-x11-server-pid'
> < './usr/lib/consolekit/ck-collect-session-info'
> 7739,7740d7715
> < './dev'
> < './run'
> 7812,7813d7786
> < './home'
> < './home/root'
> 7818d7790
> < './etc/nsswitch.conf'
> 7823d7794
> < './etc/default/usbd'
> 7910d7880
> < './etc/fstab'
> 7936d7905
> < './etc/hostname'
> 7983d7951
> < './etc/motd'
> 8020d7987
> < './etc/profile'
> 8086d8052
> < './etc/mtab' -> '/proc/mounts'
> 8098d8063
> < './etc/filesystems'
> 8138d8102
> < './etc/issue.net'
> 8163d8126
> < './etc/host.conf'
> 8227d8189
> < './etc/dbus-1/system.d/ConsoleKit.conf'
> 8434d8395
> < './etc/issue'
> 8446d8406
> < './etc/ConsoleKit/seats.d/00-primary.seat'
> 8460,8465d8419
> < './etc/inputrc'
> < './etc/skel'
> < './etc/skel/.bashrc'
> < './etc/skel/.profile'
> < './tmp'
> < './sys'
> 8533d8486
> < './media'
> 8535d8487
> < './var/lock' -> '../run/lock'
> 8544,8549c8496
> < './var/volatile'
> < './var/run' -> '../run'
> < './var/tmp' -> 'volatile/tmp'
> < './var/backups'
> < './var/local'
> < './var/log' -> 'volatile/log'
> ---
> > './var/log'
> 8562d8508
> < './var/lib/misc'
> 
> 13. Note that the package install order (for some strange reason) has changed by comparing the rootfs-logs
> diff /tmp/log.do_rootfs
> .tmp/work/q1775-poky-linux/axis-image-cvp/1.0-r0/temp/log.do_rootfs
> 
> 
> Conclusion/Expected results
> ==========
> 
> The resulting file-system is not the intended with many missing files (regardless of if there are bugs in Yocto, the packages, or that 
> the change made is not possible to do for technical reasons). Regardless of the root cause of the bad file-system, no  error
> messages are presented by Yocto or the package installation process. It is unclear for me whether the latter depends on bugs in the 
> rpm packaging system, the packages or that Yocto eats the error reporting from rpm.
> 
> /Best regards Jan
> 
> 
> 

Hi, this question has been laying here for quite some time like a dead fish. I would at least expect feedback along the following lines:
* We need more info (a, b, c, ...) to reproduce
* This is currently impossible to do in Yocto for reason (x, y, z, ...)
* Your approach/desire does not make sens at all.

/Jan