[PATCH] glibc: move ld.so.conf back to main package

[PATCH] glibc: move ld.so.conf back to main package

[Rasmus Villemoes]

There are cases where one doesn't want ldconfig on target (e.g. for
read-only root filesystems, it's rather pointless), yet one still
needs ld.so.conf to be present at image build time:

When some recipe installs libraries to a non-standard location, and
dutifully drops in a file in /etc/ld.so.conf.d/foo.conf, we need the
ld.so.conf containing the

  include /etc/ld.so.conf.d/*.conf

stanza to get those other locations picked up.

So change the packaging logic so that there's always an ld.so.conf
present when the build-time ldconfig runs, then delete it (and the .d
directory) when they would serve no purpose at run-time.

While here, fix a typo in the bb.note so one can just copy-paste the
line from the log-file and redo the command.

Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
---
 meta/lib/oe/rootfs.py                     | 11 ++++++++++-
 meta/recipes-core/glibc/glibc-package.inc |  4 ++--
 2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index cd65e62030..0ff4278d15 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -297,9 +297,18 @@ class Rootfs(object, metaclass=ABCMeta):
 
     def _run_ldconfig(self):
         if self.d.getVar('LDCONFIGDEPEND'):
-            bb.note("Executing: ldconfig -r" + self.image_rootfs + "-c new -v")
+            bb.note("Executing: ldconfig -r " + self.image_rootfs + " -c new -v")
             self._exec_shell_cmd(['ldconfig', '-r', self.image_rootfs, '-c',
                                   'new', '-v'])
+        # If DISTRO_FEATURES doesn't contain ldconfig, and the
+        # ldconfig binary hasn't been pulled in explicitly, remove the
+        # configuration file(s) from the image.
+        keep_conf = bb.utils.contains("DISTRO_FEATURES", "ldconfig", True, False, self.d)
+        keep_conf = keep_conf or os.path.exists(os.path.join(self.image_rootfs, self.d.getVar('base_sbindir'), 'ldconfig'))
+        if not keep_conf:
+            sysconfdir = os.path.join(self.image_rootfs, self.d.getVar('sysconfdir'))
+            bb.utils.remove(os.path.join(sysconfdir, 'ld.so.conf'))
+            bb.utils.remove(os.path.join(sysconfdir, 'ld.so.conf.d'), recurse=True)
 
     def _check_for_kernel_modules(self, modules_dir):
         for root, dirs, files in os.walk(modules_dir, topdown=True):
diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
index aa8e059216..387e90a0ab 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -23,9 +23,9 @@ ARCH_DYNAMIC_LOADER_aarch64 = "ld-linux-${TARGET_ARCH}.so.1"
 libc_baselibs_append = " ${@oe.utils.conditional('ARCH_DYNAMIC_LOADER', '', '', '${root_prefix}/lib/${ARCH_DYNAMIC_LOADER}', d)}"
 INSANE_SKIP_${PN}_append_aarch64 = " libdir"
 
-FILES_${PN} = "${libc_baselibs} ${libexecdir}/*"
+FILES_${PN} = "${libc_baselibs} ${libexecdir}/* ${sysconfdir}/ld.so.conf"
 RRECOMMENDS_${PN} = "${@bb.utils.filter('DISTRO_FEATURES', 'ldconfig', d)}"
-FILES_ldconfig = "${base_sbindir}/ldconfig ${sysconfdir}/ld.so.conf"
+FILES_ldconfig = "${base_sbindir}/ldconfig"
 FILES_ldd = "${bindir}/ldd"
 FILES_libsegfault = "${base_libdir}/libSegFault*"
 FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*"
-- 
2.23.0

Re: [PATCH] glibc: move ld.so.conf back to main package

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3946 bytes --]

This won't work, because as soon as glibc is upgraded with package
management, the ldconfig files will be restored and there won't be any
do_rootfs hook to remove it.

Why isn't ldconfig installed in your setup in first place? RRECOMMENDS
should be enough to pull it into every image by default.

On Mon, May 18, 2020 at 1:36 PM Rasmus Villemoes <rasmus.villemoes@prevas.dk>
wrote:

> There are cases where one doesn't want ldconfig on target (e.g. for
> read-only root filesystems, it's rather pointless), yet one still
> needs ld.so.conf to be present at image build time:
>
> When some recipe installs libraries to a non-standard location, and
> dutifully drops in a file in /etc/ld.so.conf.d/foo.conf, we need the
> ld.so.conf containing the
>
>   include /etc/ld.so.conf.d/*.conf
>
> stanza to get those other locations picked up.
>
> So change the packaging logic so that there's always an ld.so.conf
> present when the build-time ldconfig runs, then delete it (and the .d
> directory) when they would serve no purpose at run-time.
>
> While here, fix a typo in the bb.note so one can just copy-paste the
> line from the log-file and redo the command.
>
> Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
> ---
>  meta/lib/oe/rootfs.py                     | 11 ++++++++++-
>  meta/recipes-core/glibc/glibc-package.inc |  4 ++--
>  2 files changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
> index cd65e62030..0ff4278d15 100644
> --- a/meta/lib/oe/rootfs.py
> +++ b/meta/lib/oe/rootfs.py
> @@ -297,9 +297,18 @@ class Rootfs(object, metaclass=ABCMeta):
>
>      def _run_ldconfig(self):
>          if self.d.getVar('LDCONFIGDEPEND'):
> -            bb.note("Executing: ldconfig -r" + self.image_rootfs + "-c
> new -v")
> +            bb.note("Executing: ldconfig -r " + self.image_rootfs + " -c
> new -v")
>              self._exec_shell_cmd(['ldconfig', '-r', self.image_rootfs,
> '-c',
>                                    'new', '-v'])
> +        # If DISTRO_FEATURES doesn't contain ldconfig, and the
> +        # ldconfig binary hasn't been pulled in explicitly, remove the
> +        # configuration file(s) from the image.
> +        keep_conf = bb.utils.contains("DISTRO_FEATURES", "ldconfig",
> True, False, self.d)
> +        keep_conf = keep_conf or
> os.path.exists(os.path.join(self.image_rootfs,
> self.d.getVar('base_sbindir'), 'ldconfig'))
> +        if not keep_conf:
> +            sysconfdir = os.path.join(self.image_rootfs,
> self.d.getVar('sysconfdir'))
> +            bb.utils.remove(os.path.join(sysconfdir, 'ld.so.conf'))
> +            bb.utils.remove(os.path.join(sysconfdir, 'ld.so.conf.d'),
> recurse=True)
>
>      def _check_for_kernel_modules(self, modules_dir):
>          for root, dirs, files in os.walk(modules_dir, topdown=True):
> diff --git a/meta/recipes-core/glibc/glibc-package.inc
> b/meta/recipes-core/glibc/glibc-package.inc
> index aa8e059216..387e90a0ab 100644
> --- a/meta/recipes-core/glibc/glibc-package.inc
> +++ b/meta/recipes-core/glibc/glibc-package.inc
> @@ -23,9 +23,9 @@ ARCH_DYNAMIC_LOADER_aarch64 =
> "ld-linux-${TARGET_ARCH}.so.1"
>  libc_baselibs_append = " ${@oe.utils.conditional('ARCH_DYNAMIC_LOADER',
> '', '', '${root_prefix}/lib/${ARCH_DYNAMIC_LOADER}', d)}"
>  INSANE_SKIP_${PN}_append_aarch64 = " libdir"
>
> -FILES_${PN} = "${libc_baselibs} ${libexecdir}/*"
> +FILES_${PN} = "${libc_baselibs} ${libexecdir}/* ${sysconfdir}/ld.so.conf"
>  RRECOMMENDS_${PN} = "${@bb.utils.filter('DISTRO_FEATURES', 'ldconfig',
> d)}"
> -FILES_ldconfig = "${base_sbindir}/ldconfig ${sysconfdir}/ld.so.conf"
> +FILES_ldconfig = "${base_sbindir}/ldconfig"
>  FILES_ldd = "${bindir}/ldd"
>  FILES_libsegfault = "${base_libdir}/libSegFault*"
>  FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*"
> --
> 2.23.0
>
>

[-- Attachment #2: Type: text/html, Size: 4880 bytes --]

Re: [PATCH] glibc: move ld.so.conf back to main package

[Rasmus Villemoes]

On 18/05/2020 13.55, Martin Jansa wrote:
> This won't work, because as soon as glibc is upgraded with package
> management, the ldconfig files will be restored and there won't be any
> do_rootfs hook to remove it.
> 
> Why isn't ldconfig installed in your setup in first place? RRECOMMENDS
> should be enough to pull it into every image by default.

[At the risk of sounding like the lady in the Monty Python spam sketch:]
I don't _want_ it in my setup - it's a gigantic 700K binary that is
useless on a squashfs rootfs where for obvious reasons no new libraries
(or new versions) will ever appear. So no, I don't have ldconfig in my
distro features (so it's not even rrecommended), and I also don't
install recommended packages.

I just need a ld.so.cache built at image build time. Which works just
fine, except that the drop-in dir gets ignored due to lack of
ld.so.conf. After ld.so.cache has been built, the configuration files
are welcome to go the way of the dodo.

And I'm not removing the ldconfig files if ldconfig is in distro
features (or if someone has explicitly RDEPEND'ed on the ldconfig
package), so there's no issue of those files getting "restored" - if you
have ldconfig on target, they never disappear, so them getting replaced
during a package upgrade is not a problem.

I'm certainly open to other ways of solving this. But can we agree that
it is a bug that the ldconfig done at build-time does not take
/etc/ld.so.conf.d/* into account, and that that should not depend on
whether one has ldconfig-the-binary on target?

Rasmus

Re: [OE-core] [PATCH] glibc: move ld.so.conf back to main package

[Phil Blundell]

On Mon, May 18, 2020 at 02:12:43PM +0200, Rasmus Villemoes wrote:
> I'm certainly open to other ways of solving this. But can we agree that
> it is a bug that the ldconfig done at build-time does not take
> /etc/ld.so.conf.d/* into account, and that that should not depend on
> whether one has ldconfig-the-binary on target?

Yes, nowadays that's probably true.

When the USE_LDCONFIG flag (predecessor to the ldconfig DISTRO_FEATURE)
was originally added, the primary usecase was distros that didn't want
any of the ld.so.conf mechanism at all because they knew that the
libraries would always be installed in the first place that ld.so
would look for them.  In fact the commit message from when the
DISTRO_FEATURE flag was added basically says as much:

    USE_LDCONFIG could previously be set to 0 by distros which do not
    require ldconfig or ld.so.conf on the target. Since more and more
    recipes may need to respect that option, replace the ad-hoc variable
    with a distro feature.

But you're right that there's also a perfectly valid usecase for
folks who want to be able to use ld.so.conf to control the search paths
but don't actually want the ldconfig binary in the rootfs.  I guess at
that point the question becomes whether setting both read-only-rootfs
and ldconfig should cause the actual ldconfig binary to be left out
on the grounds that it can't do anything useful on a read-only rootfs,
or whether there ought to be a slightly different flag to represent
the usecase you're interested in.

p.

Re: [PATCH] glibc: move ld.so.conf back to main package

[Andreas Oberritter]

Hello Rasmus,

On Mon, 18 May 2020 14:12:43 +0200
Rasmus Villemoes <rasmus.villemoes@prevas.dk> wrote:

> I'm certainly open to other ways of solving this. But can we agree that
> it is a bug that the ldconfig done at build-time does not take
> /etc/ld.so.conf.d/* into account, and that that should not depend on
> whether one has ldconfig-the-binary on target?

have you tried installing ldconfig and adding it to ROOTFS_RO_UNNEEDED? It might be an improvement to include it in ROOTFS_RO_UNNEEDED by default.

Please consider use-cases where writable filesystems ship without ldconfig, but a user installs it from a package feed when needed.

Best regards,
Andreas

Re: [PATCH] glibc: move ld.so.conf back to main package

[Rasmus Villemoes]

On 18/05/2020 14.29, Andreas Oberritter wrote:
> Hello Rasmus,
> 
> On Mon, 18 May 2020 14:12:43 +0200
> Rasmus Villemoes <rasmus.villemoes@prevas.dk> wrote:
> 
>> I'm certainly open to other ways of solving this. But can we agree that
>> it is a bug that the ldconfig done at build-time does not take
>> /etc/ld.so.conf.d/* into account, and that that should not depend on
>> whether one has ldconfig-the-binary on target?
> 
> have you tried installing ldconfig and adding it to ROOTFS_RO_UNNEEDED? It might be an improvement to include it in ROOTFS_RO_UNNEEDED by default.

Thanks, but I don't think that will work:

        self._uninstall_unneeded()

        if self.progress_reporter:
            self.progress_reporter.next_stage()

        self._insert_feed_uris()

        self._run_ldconfig()

so if the ldconfig package (including the ld.so.conf file) is just added
to ROOTFS_RO_UNNEEDED, it will be gone by the time we get to doing the
build-time ld.so.cache generation. And I think it has to be done in this
order - if some of the packages removed by _uninstall_unneeded remove
shared libraries, one doesn't want stale entries in ld.so.cache
referring to those.

> Please consider use-cases where writable filesystems ship without ldconfig, but a user installs it from a package feed when needed.

Do you mean I should leave /etc/ld.so.conf.d/ alone? I can do that. But
I'd say that also in that case the current behaviour is buggy - if any
package in the rootfs ships with an /etc/ld.so.conf.d/ entry, that entry
should be taken into account at image build time, regardless of whether
ldconfig is there at image build time or can usefully be added later via
a package installer.

Rasmus

Re: [PATCH] glibc: move ld.so.conf back to main package

[Andreas Oberritter]

On Mon, 18 May 2020 15:25:19 +0200
Rasmus Villemoes <rasmus.villemoes@prevas.dk> wrote:

> On 18/05/2020 14.29, Andreas Oberritter wrote:
> > Hello Rasmus,
> > 
> > On Mon, 18 May 2020 14:12:43 +0200
> > Rasmus Villemoes <rasmus.villemoes@prevas.dk> wrote:
> > 
> >> I'm certainly open to other ways of solving this. But can we agree that
> >> it is a bug that the ldconfig done at build-time does not take
> >> /etc/ld.so.conf.d/* into account, and that that should not depend on
> >> whether one has ldconfig-the-binary on target?
> > 
> > have you tried installing ldconfig and adding it to ROOTFS_RO_UNNEEDED? It might be an improvement to include it in ROOTFS_RO_UNNEEDED by default.
> 
> Thanks, but I don't think that will work:
> 
>         self._uninstall_unneeded()
> 
>         if self.progress_reporter:
>             self.progress_reporter.next_stage()
> 
>         self._insert_feed_uris()
> 
>         self._run_ldconfig()
> 
> so if the ldconfig package (including the ld.so.conf file) is just added
> to ROOTFS_RO_UNNEEDED, it will be gone by the time we get to doing the
> build-time ld.so.cache generation. And I think it has to be done in this
> order - if some of the packages removed by _uninstall_unneeded remove
> shared libraries, one doesn't want stale entries in ld.so.cache
> referring to those.

That's not ideal, but we could special-case ldconfig and uninstall it after this step.

> > Please consider use-cases where writable filesystems ship without ldconfig, but a user installs it from a package feed when needed.
> 
> Do you mean I should leave /etc/ld.so.conf.d/ alone? I can do that. But
> I'd say that also in that case the current behaviour is buggy - if any
> package in the rootfs ships with an /etc/ld.so.conf.d/ entry, that entry
> should be taken into account at image build time, regardless of whether
> ldconfig is there at image build time or can usefully be added later via
> a package installer.

What I meant to say is that, in general, we can't remove files from rootfs manually if they are managed by a package manager (without causing difficulties like having to reinstall glibc in this case to restore these files when needed by someone installing ldconfig from a feed). So you'd have to leave both ld.so.conf and ld.so.conf.d alone.

Best regards,
Andreas

[PATCH v2] glibc: move ld.so.conf back to main package

[Rasmus Villemoes]

There are cases where one doesn't want ldconfig on target (e.g. for
read-only root filesystems, it's rather pointless), yet one still
needs ld.so.conf to be present at image build time:

When some recipe installs libraries to a non-standard location, and
dutifully drops in a file in /etc/ld.so.conf.d/foo.conf, we need the
ld.so.conf containing the

  include /etc/ld.so.conf.d/*.conf

stanza to get those other locations picked up.

So change the packaging logic so that there's always an ld.so.conf
present when the build-time ldconfig runs.

The ld.so.conf and ld.so.conf.d/*.conf files don't take up much
room (at least not compared to the 700K binary ldconfig), and they
might be needed in case ldconfig is installable, so leave them
alone.

In case of a read-only rootfs, one could add some logic to remove them
if one really wants to shave those few dozens of bytes off.

While here, fix typos in the bb.note (add spaces) so one can just
copy-paste the line from the log-file and redo the command.

Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
---
v2: don't delete the configuration file(s) - that's also a much
simpler patch.

 meta/lib/oe/rootfs.py                     | 2 +-
 meta/recipes-core/glibc/glibc-package.inc | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index cd65e62030..a0ac33ada6 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -297,7 +297,7 @@ class Rootfs(object, metaclass=ABCMeta):
 
     def _run_ldconfig(self):
         if self.d.getVar('LDCONFIGDEPEND'):
-            bb.note("Executing: ldconfig -r" + self.image_rootfs + "-c new -v")
+            bb.note("Executing: ldconfig -r " + self.image_rootfs + " -c new -v")
             self._exec_shell_cmd(['ldconfig', '-r', self.image_rootfs, '-c',
                                   'new', '-v'])
 
diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
index aa8e059216..387e90a0ab 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -23,9 +23,9 @@ ARCH_DYNAMIC_LOADER_aarch64 = "ld-linux-${TARGET_ARCH}.so.1"
 libc_baselibs_append = " ${@oe.utils.conditional('ARCH_DYNAMIC_LOADER', '', '', '${root_prefix}/lib/${ARCH_DYNAMIC_LOADER}', d)}"
 INSANE_SKIP_${PN}_append_aarch64 = " libdir"
 
-FILES_${PN} = "${libc_baselibs} ${libexecdir}/*"
+FILES_${PN} = "${libc_baselibs} ${libexecdir}/* ${sysconfdir}/ld.so.conf"
 RRECOMMENDS_${PN} = "${@bb.utils.filter('DISTRO_FEATURES', 'ldconfig', d)}"
-FILES_ldconfig = "${base_sbindir}/ldconfig ${sysconfdir}/ld.so.conf"
+FILES_ldconfig = "${base_sbindir}/ldconfig"
 FILES_ldd = "${bindir}/ldd"
 FILES_libsegfault = "${base_libdir}/libSegFault*"
 FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*"
-- 
2.23.0

✗ patchtest: failure for glibc: move ld.so.conf back to main package (rev2)

[Patchwork]

== Series Details ==

Series: glibc: move ld.so.conf back to main package (rev2)
Revision: 2
URL   : https://patchwork.openembedded.org/series/24198/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at 982b7f98b8)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe