non-upstreamed patches

non-upstreamed patches

[Colin Walters]

So I've switched back to looking at Yocto again for a project, and the
Edison release works fine.  Thanks!

I work on GNOME (almost all of it), and I see quite a number of patches
in the Yocto tree that I think have been inappropriately categorized as
Upstream-Status: Inappropriate.

For example,
recipies-core/glib-2.0/glib-2.0/remove.test.for.qsort_r.patch

We definitely want upstream GLib to work for cross builds.  It was my
understanding that the better way to handle autoconf checks that require
running code was to pass a precomputed cache file:
http://www.gnu.org/s/hello/manual/autoconf/Site-Defaults.html

I think this makes sense to do for fundamental libraries like GLib;
hacking up the configure.ac isn't a good long term plan.

Another thing that seems to be proliferating is gtk-doc workarounds.  If
it isn't working for you guys to --disable-gtk-doc, just tell me why and
I'll fix it.

(The -Werror bit there is a total fuckup on our part, I will eventually
convince Owen to remove it...)

librsvg/librsvg-2.31.1/doc_Makefile.patch is also TOTALLY appropriate
upstream.  Why would you think it isn't?

gtk+/gtk+-2.24.6/run-iconcache.patch is another cross build issue that
I'm pretty sure we can solve better upstream.  gtk+ for example already
has an automake conditional:
AM_CONDITIONAL(CROSS_COMPILING, test $cross_compiling = yes)

We can just not run this code if that's true.

(And I am totally fine with just copying that conditional into every
single GNOME module).

hardcoded_libtool.patch is another one I think we can get upstream in a
better way; shouldn't it just be using $LIBTOOL ?

Finally, on the Upstream-Status: pending patches, can you guys please
add links to bugzilla?

Thanks!

Re: non-upstreamed patches

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 347 bytes --]


Op 13 nov. 2011, om 01:57 heeft Colin Walters het volgende geschreven:

> So I've switched back to looking at Yocto again for a project, and the
> Edison release works fine.  Thanks!
> 
> I work on GNOME (almost all of it)

Have a look at the GNOME layer as well http://cgit.openembedded.org/meta-openembedded/tree/ :)

regards,

Koen

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

Re: non-upstreamed patches

[Colin Walters]

On Sun, 2011-11-13 at 10:28 +0100, Koen Kooi wrote:
> Op 13 nov. 2011, om 01:57 heeft Colin Walters het volgende geschreven:
> 
> > So I've switched back to looking at Yocto again for a project, and the
> > Edison release works fine.  Thanks!
> > 
> > I work on GNOME (almost all of it)
> 
> Have a look at the GNOME layer as well http://cgit.openembedded.org/meta-openembedded/tree/ :)

What's the high level relationship of this tree versus the Yocto one?
The git log looks very similar, are Yocto commits cherry-picked into it?

Re: non-upstreamed patches

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 808 bytes --]


Op 13 nov. 2011, om 18:16 heeft Colin Walters het volgende geschreven:

> On Sun, 2011-11-13 at 10:28 +0100, Koen Kooi wrote:
>> Op 13 nov. 2011, om 01:57 heeft Colin Walters het volgende geschreven:
>> 
>>> So I've switched back to looking at Yocto again for a project, and the
>>> Edison release works fine.  Thanks!
>>> 
>>> I work on GNOME (almost all of it)
>> 
>> Have a look at the GNOME layer as well http://cgit.openembedded.org/meta-openembedded/tree/ :)
> 
> What's the high level relationship of this tree versus the Yocto one?

There's no such thing as a yocto tree, only a poky tree, which consists of openembedded-core, bitbake and some other stuff

> The git log looks very similar, are Yocto commits cherry-picked into it?

No, oe-core generally picks up fixes from here.

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

Re: non-upstreamed patches

[Richard Purdie]

On Sat, 2011-11-12 at 19:57 -0500, Colin Walters wrote:
> So I've switched back to looking at Yocto again for a project, and the
> Edison release works fine.  Thanks!

Pleased to hear it! :)

> I work on GNOME (almost all of it), and I see quite a number of patches
> in the Yocto tree that I think have been inappropriately categorized as
> Upstream-Status: Inappropriate.

The Upstream-Status fields are relatively new and I think some of the
categorizations are perhaps not quite right or need to be interpreted
correctly. 

http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Patch_Header_Recommendations

> For example,
> recipies-core/glib-2.0/glib-2.0/remove.test.for.qsort_r.patch
> 
> We definitely want upstream GLib to work for cross builds.

Taking this one as an example, someone has run into problems and fixed
it in a way that is certainly not appropriate for upstream as the patch
stands.

>   It was my
> understanding that the better way to handle autoconf checks that require
> running code was to pass a precomputed cache file:
> http://www.gnu.org/s/hello/manual/autoconf/Site-Defaults.html
>
> I think this makes sense to do for fundamental libraries like GLib;
> hacking up the configure.ac isn't a good long term plan.

Totally agreed. We do have site cache files an a much better way of
handling that issue would be to add the value to the site cache. In an
ideal world I've have caught that patch and asked it to be a site entry,
it looks like its slipped through. I'll certainly take a site file entry
and change to drop the patch.

> Another thing that seems to be proliferating is gtk-doc workarounds.  If
> it isn't working for you guys to --disable-gtk-doc, just tell me why and
> I'll fix it.
> 
> (The -Werror bit there is a total fuckup on our part, I will eventually
> convince Owen to remove it...)
> 
> librsvg/librsvg-2.31.1/doc_Makefile.patch is also TOTALLY appropriate
> upstream.  Why would you think it isn't?

This is a mess of our own making :/. You'd probably get a bit of a shock
if you look at our gtk-doc recipe. Since we reautoconf files, we need
the gtk-doc .m4 file. We just provide a static copy of it. To handle the
makefile, we just touch that.

I have a suspicion that EXTRA_DIST exists if you use the gtk-doc
makefile and hence appending to it is a good idea and hence that patch
truly is inappropriate for upstream.

In many cases we've learnt from our experiences and instead of touching
the makefile, we echo EXTRA_DIST = "" to it instead (or similar).

Ultimately I'd like to see a fixed gtk-doc recipe. It was so long ago I
last tried that I can't remember what issues we found.

> gtk+/gtk+-2.24.6/run-iconcache.patch is another cross build issue that
> I'm pretty sure we can solve better upstream.  gtk+ for example already
> has an automake conditional:
> AM_CONDITIONAL(CROSS_COMPILING, test $cross_compiling = yes)
> 
> We can just not run this code if that's true.

That would be a nice change to get upstream. The current patch is
inappropriate but I'm sure we could create one that could be accepted.

> (And I am totally fine with just copying that conditional into every
> single GNOME module).
> 
> hardcoded_libtool.patch is another one I think we can get upstream in a
> better way; shouldn't it just be using $LIBTOOL ?

Good question, I think I might tried that and had some kind of problem
but if that did would it would be great to see it upstream too.

> Finally, on the Upstream-Status: pending patches, can you guys please
> add links to bugzilla?

If its been filed there should be links. "Pending" can also mean "needs
to be submitted upstream". As I said, this is a relatively new thing, we
used to have no information in the patches at all so it is getting
better. We're making progress although never as fast as we would in an
ideal world! :)

Any help would be appreciated, the reminders above are helpful as there
does appear to be some low hanging fruit there. Also the pointers on
which ones might make it into gnome upstream are useful!

Cheers,

Richard

Re: non-upstreamed patches

[Colin Walters]

On Sun, 2011-11-13 at 09:47 +0000, Richard Purdie wrote:

> Totally agreed. We do have site cache files an a much better way of
> handling that issue would be to add the value to the site cache. In an
> ideal world I've have caught that patch and asked it to be a site entry,
> it looks like its slipped through. I'll certainly take a site file entry
> and change to drop the patch.

Ok, as I move forward I'll look at draining the patches here.  It'd be
nice if we could try harder to avoid adding new ones though - that was
my main goal with the mail.

> > Another thing that seems to be proliferating is gtk-doc workarounds.  If
> > it isn't working for you guys to --disable-gtk-doc, just tell me why and
> > I'll fix it.
> > 
> This is a mess of our own making :/. You'd probably get a bit of a shock
> if you look at our gtk-doc recipe. Since we reautoconf files, we need
> the gtk-doc .m4 file. We just provide a static copy of it. To handle the
> makefile, we just touch that.

Ah, I see.  Have you seen people.gnome.org/~walters/docs/build-api.txt ?

One of the reasons I wrote it is because in GNOME we need ./autogen.sh
scripts because there's no easy way to teach the autotools about
external dependencies.

What would you think about a patch to autotools.bbclass to just check
for autogen.sh and run it (instead of the big mess of autoreconf/etc.
hacks) if it exists?

Re: non-upstreamed patches

[Richard Purdie]

On Mon, 2011-11-14 at 14:06 -0500, Colin Walters wrote:
> On Sun, 2011-11-13 at 09:47 +0000, Richard Purdie wrote:
> 
> > Totally agreed. We do have site cache files an a much better way of
> > handling that issue would be to add the value to the site cache. In an
> > ideal world I've have caught that patch and asked it to be a site entry,
> > it looks like its slipped through. I'll certainly take a site file entry
> > and change to drop the patch.
> 
> Ok, as I move forward I'll look at draining the patches here.  It'd be
> nice if we could try harder to avoid adding new ones though - that was
> my main goal with the mail.

Believe me, we're trying *very* hard to get the number of patches down!

> > > Another thing that seems to be proliferating is gtk-doc workarounds.  If
> > > it isn't working for you guys to --disable-gtk-doc, just tell me why and
> > > I'll fix it.
> > > 
> > This is a mess of our own making :/. You'd probably get a bit of a shock
> > if you look at our gtk-doc recipe. Since we reautoconf files, we need
> > the gtk-doc .m4 file. We just provide a static copy of it. To handle the
> > makefile, we just touch that.
> 
> Ah, I see.  Have you seen people.gnome.org/~walters/docs/build-api.txt ?
> 
> One of the reasons I wrote it is because in GNOME we need ./autogen.sh
> scripts because there's no easy way to teach the autotools about
> external dependencies.
> 
> What would you think about a patch to autotools.bbclass to just check
> for autogen.sh and run it (instead of the big mess of autoreconf/etc.
> hacks) if it exists?

The number of people who write cross safe autogen.sh scripts which do
everything we need is unfortunately low. I do appreciate the gnome
community may do better than others and things are improving over time.
I'd not yet at a point where I'd trust even 10% of the autogen.sh
scripts out there though.

I'm prepared to stick my neck out and state that the reautoconf we do
doesn't cause that much of a problem at this point and that class works
pretty well. The process for reautoconf'ing a package is pretty standard
when you get down to it. We did used to have a lot more hacky recipe
specific do_configure functions and we've managed to just have
the .bbclass code in most cases now too which is an achievement in
itself.

Cheers,

Richard

Re: non-upstreamed patches

[Colin Walters]

On Mon, 2011-11-14 at 19:56 +0000, Richard Purdie wrote:

> The number of people who write cross safe autogen.sh scripts which do
> everything we need is unfortunately low.

Can you elaborate on this?  What are some common cross-unsafe problems?

So concretely here with gtk-doc, since we don't actually want to build
the documentation (because gtk-doc is native only for the exact same
reasons gobject-introspection is, and we've talked about that one).

The GNOME autogen.sh scripts will run "gtkdocize", which pretty much
just copies in gtk-doc.make and gtk-doc.m4.  We could reimplement that
with dummy copies, or build a lobotomized gtk-doc (this will require
perl-native but that's about it I think).

I'm experimenting with this now.

>  I do appreciate the gnome
> community may do better than others and things are improving over time.
> I'd not yet at a point where I'd trust even 10% of the autogen.sh
> scripts out there though.

The other thing we could do I guess is make it opt-in, like
OE_RUN_AUTOGEN=1 or something in individual recipes using
autotools.bbclass?

Re: non-upstreamed patches

[Richard Purdie]

On Mon, 2011-11-14 at 15:48 -0500, Colin Walters wrote:
> On Mon, 2011-11-14 at 19:56 +0000, Richard Purdie wrote:
> 
> > The number of people who write cross safe autogen.sh scripts which do
> > everything we need is unfortunately low.
> 
> Can you elaborate on this?  What are some common cross-unsafe problems?

Well, I went and looked at the gnome-common autogen stuff. Its not a
cross specific problem in this case but it does seem to
reinvent/reimplement pieces of what autoconf/pkgconfig is supposed to
do. Compare/check autoconf version? automake version? pkg-config
dependencies present? Why? The whole point of autoconf was to avoid
having these scripts :(

Examples of cross unsafe issues we've seen are libtool macros not being
updated and old versions used for the reconf. Forcibly running configure
from the script with options we don't want (or not including options we
do need). We do have some autotools* flags we pass in ourselves too
since the autotools we're using are slightly tweaked to avoid certain
cross bugs. There is also confusion about whether to force copies of
file or not (we always do).

Using those scripts just adds so much unpredictability to the
system :(. 

> So concretely here with gtk-doc, since we don't actually want to build
> the documentation (because gtk-doc is native only for the exact same
> reasons gobject-introspection is, and we've talked about that one).
> 
> The GNOME autogen.sh scripts will run "gtkdocize", which pretty much
> just copies in gtk-doc.make and gtk-doc.m4.  We could reimplement that
> with dummy copies, or build a lobotomized gtk-doc (this will require
> perl-native but that's about it I think).

Our current hack is really the former approach - add dummy copies in.
Its just a little incomplete and we should enhance it deal with that so
we can run gtkdocize.

> I'm experimenting with this now.
> 
> >  I do appreciate the gnome
> > community may do better than others and things are improving over time.
> > I'd not yet at a point where I'd trust even 10% of the autogen.sh
> > scripts out there though.
> 
> The other thing we could do I guess is make it opt-in, like
> OE_RUN_AUTOGEN=1 or something in individual recipes using
> autotools.bbclass?

We could do that, yes. What is the advantage over our current class code
(which I'll be the first to admit has faults but does seem to work)?

Cheers,

Richard

Re: non-upstreamed patches

[Colin Walters]

On Mon, 2011-11-14 at 22:44 +0000, Richard Purdie wrote:

> Well, I went and looked at the gnome-common autogen stuff. Its not a
> cross specific problem in this case but it does seem to
> reinvent/reimplement pieces of what autoconf/pkgconfig is supposed to
> do. Compare/check autoconf version? automake version? pkg-config
> dependencies present? Why? The whole point of autoconf was to avoid
> having these scripts :(

They serve a few purposes; one is simply helping people who check out a
given module from git know they have compatible versions of the
autotools.  This used to be a major problem, nowadays it's been a long
time since any kind serious of autotools incompatibilities.

The still-relevant issue though is that autoreconf isn't extensible - in
GNOME for better or worse we have two things that follow the auto* style
(gtk-doc and intltool mainly) where they copy data into your system if
building from git, and get shipped in tarballs so you don't need them at
build time when building from tarballs.

But we have no way to tell autoreconf to run these tools currently.
Maybe the right thing is to extend autoreconf but even if we did that,
we'd have to wait some amount of time before assuming everyone had the
modified autoreconf.

> Our current hack is really the former approach - add dummy copies in.
> Its just a little incomplete and we should enhance it deal with that so
> we can run gtkdocize.

Hmmm...so we'd be patching autotools.bbclass to grep for e.g.
GTK_DOC_CHECK in configure.ac and if detected, copy in a dummy
gtk-doc.make and gtk-doc.m4?

(And the same for intltool?)

I wouldn't object too much to that I guess, but I'd still prefer the
solution where we opt-in to running autogen.sh for GNOME modules which
use gtk-doc and/or intltool.

Re: non-upstreamed patches

[Richard Purdie]

On Mon, 2011-11-14 at 19:05 -0500, Colin Walters wrote:
> On Mon, 2011-11-14 at 22:44 +0000, Richard Purdie wrote:
> 
> > Well, I went and looked at the gnome-common autogen stuff. Its not a
> > cross specific problem in this case but it does seem to
> > reinvent/reimplement pieces of what autoconf/pkgconfig is supposed to
> > do. Compare/check autoconf version? automake version? pkg-config
> > dependencies present? Why? The whole point of autoconf was to avoid
> > having these scripts :(
> 
> They serve a few purposes; one is simply helping people who check out a
> given module from git know they have compatible versions of the
> autotools.  This used to be a major problem, nowadays it's been a long
> time since any kind serious of autotools incompatibilities.

I certainly remember those days and can appreciate why this kind of
thing exists (we have the same problem with the OE codebase in many
ways).

> The still-relevant issue though is that autoreconf isn't extensible - in
> GNOME for better or worse we have two things that follow the auto* style
> (gtk-doc and intltool mainly) where they copy data into your system if
> building from git, and get shipped in tarballs so you don't need them at
> build time when building from tarballs.
> 
> But we have no way to tell autoreconf to run these tools currently.
> Maybe the right thing is to extend autoreconf but even if we did that,
> we'd have to wait some amount of time before assuming everyone had the
> modified autoreconf.
> 
> > Our current hack is really the former approach - add dummy copies in.
> > Its just a little incomplete and we should enhance it deal with that so
> > we can run gtkdocize.
> 
> Hmmm...so we'd be patching autotools.bbclass to grep for e.g.
> GTK_DOC_CHECK in configure.ac and if detected, copy in a dummy
> gtk-doc.make and gtk-doc.m4?
> 
> (And the same for intltool?)
> 
> I wouldn't object too much to that I guess, but I'd still prefer the
> solution where we opt-in to running autogen.sh for GNOME modules which
> use gtk-doc and/or intltool.

FWIW, we already do this for intltool:

	if grep "^[[:space:]]*[AI][CT]_PROG_INTLTOOL" $CONFIGURE_AC >/dev/null; then
	  bbnote Executing intltoolize --copy --force --automake
	  intltoolize --copy --force --automake
	fi

so we might even use gtkdocize, but have modified files for it to place
into the system. Strange as it might sound I'd actually like to be able
to generate the docs if possible. It is completely the same as the
introspection case where we need to be able to run target binaries or
could a set of -native binaries get us through of most of the difficult
bits?

The reason I mention docs is that we generate -doc packages. I have an
idea of being able to generate the technical reference that would go
alongside a given image. Obviously the gtkdoc pieces would be an
important part of that. Creating the -doc packages is part of that, the
remaining tricky part would then be indexing them in a sane way that was
useful and accessible.

Cheers,

Richard

Re: non-upstreamed patches

[Colin Walters]

>> On Mon, 2011-11-14 at 22:44 +0000, Richard Purdie wrote:

> > Our current hack is really the former approach - add dummy copies in.
> > Its just a little incomplete and we should enhance it deal with that so
> > we can run gtkdocize.

Revisiting this thread - so I've now created in GNOME git a tiny new
module "gtk-doc-stub" which contains enough bits to build modules from
git with --disable-gtk-doc.

http://git.gnome.org/browse/gtk-doc-stub

It should be enough to add this to Yocto, and then hacking up GNOME
modules to remove GTK_DOC_CHECK from configure.ac etc. should be
unnecessary.  I'm using it now in my self-hosting OS build tool to avoid
wading into Docbook dependency hell.