RFC: magic libtool .la removal

RFC: magic libtool .la removal

[Burton, Ross]

Hi,

Looking at another recipe that deletes .la files reminded me that
every six months or so I look at this and never actually get anything
done.  Let's try attempt three!

My proposal is that we integrate .la-removal into a core class.
autotools.bbclass should cover 99% of instances as libtool is normally
used in conjunction with auto*.  Have a variable, REMOVE_LIBTOOL_LA,
which controls the behaviour post-install.

"none".  For this recipe, don't remove any .la files.

"libdir".  For this recipe, remove .la files in $base_libdir and
$libdir non-recursively, keeping the .la files in any subdirectories.
This is for packages that use libltdt to load modules, which I believe
needs .la files to work.

"all".  For this recipe, remove all .la files.

I've not written any code yet, but it shouldn't be hard.  Any thoughts?

Ross

Re: RFC: magic libtool .la removal

[Phil Blundell]

On Tue, 2013-06-18 at 15:31 +0100, Burton, Ross wrote:
> My proposal is that we integrate .la-removal into a core class.
> autotools.bbclass should cover 99% of instances as libtool is normally
> used in conjunction with auto*.  Have a variable, REMOVE_LIBTOOL_LA,
> which controls the behaviour post-install.

FWIW, I still have the patch from:

http://lists.openembedded.org/pipermail/openembedded-core/2012-October/069912.html

in my local tree and it seems to be working fine (although I don't have
any particular interest in libltdt so it's conceivable that this might
be broken).

Looking back at that old thread it seems that you asked me to split into
two patches, I said "ok", but apparently I never did anything further
about it.  It's possible I might have been put off by Paul not liking
the name or something, I don't really recall.

p.

Re: RFC: magic libtool .la removal

[Burton, Ross]

On 18 June 2013 15:42, Phil Blundell <pb@pbcl.net> wrote:
> FWIW, I still have the patch from:
>
> http://lists.openembedded.org/pipermail/openembedded-core/2012-October/069912.html
>
> in my local tree and it seems to be working fine (although I don't have
> any particular interest in libltdt so it's conceivable that this might
> be broken).

I also remember discussion with Colin Walters about ostree, which at
one point only removed from $libdir itself (my "libdir" argument) but
now removes all .la files.  Colin, the commit that changed the la
killing to recurse didn't have a rationale - can you clarify this?  If
ostree can get away with removing all, them maybe we don't need the
"libdir" option at all.

Ross

Re: RFC: magic libtool .la removal

[Colin Walters]

On Tue, 2013-06-18 at 15:56 +0100, Burton, Ross wrote:

> I also remember discussion with Colin Walters about ostree, which at
> one point only removed from $libdir itself (my "libdir" argument) but
> now removes all .la files.  Colin, the commit that changed the la
> killing to recurse didn't have a rationale - can you clarify this?  If
> ostree can get away with removing all, them maybe we don't need the
> "libdir" option at all.

The relevant data I have on hand are:

https://bugzilla.gnome.org/show_bug.cgi?id=654013
https://git.gnome.org/browse/jhbuild/commit/?id=965c8d5ceda9d1c5d6021ef2c534e0a7f68ca976

I think the executive summary is that libltdl knows how to load .so
files directly (at least currently), so there's no reason to have even
${libdir}/modulename/plugins/foo.la.

Re: RFC: magic libtool .la removal

[Burton, Ross]

On 18 June 2013 16:00, Colin Walters <walters@verbum.org> wrote:
> The relevant data I have on hand are:
>
> https://bugzilla.gnome.org/show_bug.cgi?id=654013
> https://git.gnome.org/browse/jhbuild/commit/?id=965c8d5ceda9d1c5d6021ef2c534e0a7f68ca976
>
> I think the executive summary is that libltdl knows how to load .so
> files directly (at least currently), so there's no reason to have even
> ${libdir}/modulename/plugins/foo.la.

Thanks Colin.  Let's ignore the "libdir" option so this is a
per-package keep-or-wipe option.  I'd like to default it to removing
all by default after a verification that the build results are
identical.

Ross

Re: RFC: magic libtool .la removal

[Richard Purdie]

On Tue, 2013-06-18 at 16:05 +0100, Burton, Ross wrote:
> On 18 June 2013 16:00, Colin Walters <walters@verbum.org> wrote:
> > The relevant data I have on hand are:
> >
> > https://bugzilla.gnome.org/show_bug.cgi?id=654013
> > https://git.gnome.org/browse/jhbuild/commit/?id=965c8d5ceda9d1c5d6021ef2c534e0a7f68ca976
> >
> > I think the executive summary is that libltdl knows how to load .so
> > files directly (at least currently), so there's no reason to have even
> > ${libdir}/modulename/plugins/foo.la.
> 
> Thanks Colin.  Let's ignore the "libdir" option so this is a
> per-package keep-or-wipe option.  I'd like to default it to removing
> all by default after a verification that the build results are
> identical.

The thing which really worries me about this is that we'll start to
deviate quite massively with how upstream expect us to use autotools.

As things stand, we have a number of sysroot fixes for the sysroot
support in libtool which is basically broken out the box. I have tried
discussing them with upstream and was ignored, mainly as we patch
libtool and we're supposed to use it unpatched. 

I worry that if we go this route, the builds will stop working without
the .la file removal and that we'll lose any chance of being able to
resolve our patchset with libtool upstream. We might as well throw away
libtool at that point and save the overhead. It also means we will have
bigger problems working on something like darwin (which I've had work in
the past).

So I don't see the pressing need to set us off down a path on our own.
Yes the .la files are annoying but they're not that much of a problem,
are they?

Cheers,

Richard

Re: RFC: magic libtool .la removal

[Burton, Ross]

On 18 June 2013 16:47, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> So I don't see the pressing need to set us off down a path on our own.
> Yes the .la files are annoying but they're not that much of a problem,
> are they?

Whilst the separate build directory work is a massive improvement,
there's plenty of packages that don't do B!=S and use libtool.  We're
not alone, at least Fedora and OSTree both wipe out all .la files.

As B!=S should solve this problem and others I can see the argument
for spending time making that work more comprehensively though.

Ross

Re: RFC: magic libtool .la removal

[Phil Blundell]

On Tue, 2013-06-18 at 16:47 +0100, Richard Purdie wrote:
> We might as well throw away libtool

That sounds like an excellent plan to me.  I wonder how hard it would be
to invent a libtool-dummy script which took the same arguments but
basically just invoked the compiler and linker without any of the extra
craziness.

>Yes the .la files are annoying but they're not that much of a problem,
>are they?

I do recall that I was moved to write my original patch because the .la
files were causing me an actual problem rather than just being a
nuisance.  I can't remember the details offhand unfortunately, though I
guess I could try re-enabling them again locally and see what breaks.

Also, the patch that I posted in the first place did use a
DISTRO_FEATURE to control whether you get .la files or not (and the
default was still to have them) so you would be welcome to keep them on
for poky if you wanted.

p.

Re: RFC: magic libtool .la removal

[Burton, Ross]

On 18 June 2013 16:54, Phil Blundell <pb@pbcl.net> wrote:
> On Tue, 2013-06-18 at 16:47 +0100, Richard Purdie wrote:
>> We might as well throw away libtool
>
> That sounds like an excellent plan to me.  I wonder how hard it would be
> to invent a libtool-dummy script which took the same arguments but
> basically just invoked the compiler and linker without any of the extra
> craziness.

That would be DOLT.

>>Yes the .la files are annoying but they're not that much of a problem,
>>are they?
>
> I do recall that I was moved to write my original patch because the .la
> files were causing me an actual problem rather than just being a
> nuisance.  I can't remember the details offhand unfortunately, though I
> guess I could try re-enabling them again locally and see what breaks.

FWIW, I've never seen a .la break anything where a clean doesn't solve
it, the problems are always due to rebuilds.

Ross

Re: RFC: magic libtool .la removal

[Colin Walters]

On Tue, 2013-06-18 at 16:47 +0100, Richard Purdie wrote:

> The thing which really worries me about this is that we'll start to
> deviate quite massively with how upstream expect us to use autotools.

I just consider upstream wrong, and so do others:
http://wiki.debian.org/ReleaseGoals/LAFileRemoval
http://fedoraproject.org/wiki/Packaging:Guidelines (search for .la)
http://blog.flameeyes.eu/2008/04/what-about-those-la-files (the one mostly sane Gentoo guy)

> As things stand, we have a number of sysroot fixes for the sysroot
> support in libtool which is basically broken out the box. I have tried
> discussing them with upstream and was ignored, mainly as we patch
> libtool and we're supposed to use it unpatched. 

Yeah, I dunno...maybe someone needs to fork libtool.

> I worry that if we go this route, the builds will stop working without
> the .la file removal and that we'll lose any chance of being able to
> resolve our patchset with libtool upstream. We might as well throw away
> libtool at that point

Or that...

>  and save the overhead. It also means we will have
> bigger problems working on something like darwin (which I've had work in
> the past).

I don't have any experience with Darwin myself.

> So I don't see the pressing need to set us off down a path on our own.
> Yes the .la files are annoying but they're not that much of a problem,
> are they?

Depends; I don't have a lot of first-hand experience with problems they
cause in the OE context.  But .la files completely break jhbuild.
Certainly, one could call jhbuild a hack, and it is.  But it's a tool
with absolutely essential properties for people contributing to GNOME.  

Basically jhbuild allows one to e.g.:
$ jhbuild buildone gtk+
$ jhbuild run gedit

Now you have your system version of /usr/bin/gedit linked against the
latest gtk+ from git in /opt/gnome/lib/libgtk.so.  This "two layer"
model is essential for allowing developers to test new code without
risking their /.

The root "feature" of .la files where it can say "if you link against
me, also link against these other libraries" that causes this is much
better implemented by pkg-config.

Re: RFC: magic libtool .la removal

[Colin Walters]

On Tue, 2013-06-18 at 12:05 -0400, Colin Walters wrote:

> Yeah, I dunno...maybe someone needs to fork libtool.

I should follow up to this; the thing is, libtool is at the intersection
of so many cross-cutting issues:

* RPM-style multilib vs Debian-style multiarch
* Supporting libraries that use pkg-config vs those that still
  sadly don't
* Windows vs Darwin vs GNU/Linux (in all its variations)
* Cross vs native builds
* "sysroot" support
* Plugins: libltdl  (and how that interacts with other cross-platform
  module loaders like e.g. GModule)
* Component-internal build system vs external interaction; specifically
  libtool makes it easy to run *uninstalled* binaries, which is
  quite useful, and to do that inside the tree does require
  extra metadata in .la files

So when I say .la files are worthless and broken, I really mean just for
GNU/Linux systems (generally native builds, but likely also cross), and
where things I care about know how to find .so files instead of .la,
and only for *external* build system interaction; having .la files
*inside* the build tree for a single component is fine, etc.

I can't say for sure myself that having libtool unilaterally stop
installing .la files wouldn't break anything; maybe there's some library
out there that doesn't use pkg-config and relies on consumers using
dependency_libs.  But I do think it's at best unnecessary for the use
case above; so maybe it should come down to an option, or something.