[PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[Richard Purdie]

Unfortunately whilst reruning configure and make against a project will mostly
work there are situations where it does not correctly do the right thing.

In particular, eglibc and gcc will fail out with errors where settings
do not match a previously built configuration. It could be argued they are
broken but the situation is what it is. There is the possibility of more subtle
errors too.

This patch adds a "make distclean" call to recipes where configure is
rerunning and the sstate checksum for do_configure has changed. We could
simply use a stamp but saving out the previous configuration checksum
adds some data at no real overhead.

If we find there are things out there which don't have a "distclean" target,
we can disable this behaviour with CONFIGURESTAMPFILE = "" in the recipe,
or users could disable it globally.

[YOCTO #2774]
[YOCTO #2848]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/meta/classes/autotools.bbclass b/meta/classes/autotools.bbclass
index 4c4bf87..1ab2e0c 100644
--- a/meta/classes/autotools.bbclass
+++ b/meta/classes/autotools.bbclass
@@ -89,7 +89,16 @@ oe_runconf () {
 
 AUTOTOOLS_AUXDIR ?= "${S}"
 
+CONFIGURESTAMPFILE = "${WORKDIR}/configure.sstate"
+
 autotools_do_configure() {
+	if [ -n "${CONFIGURESTAMPFILE}" -a -e "${CONFIGURESTAMPFILE}" ]; then
+		if [ "`cat ${CONFIGURESTAMPFILE}`" != "${BB_TASKHASH}" ]; then
+			echo "Previously configured build detected, running make distclean"
+			oe_runmake distclean
+		fi
+	fi
+
 	case ${PN} in
 	autoconf*)
 	;;
@@ -180,6 +189,9 @@ autotools_do_configure() {
 	else
 		bbnote "nothing to configure"
 	fi
+	if [ -n "${CONFIGURESTAMPFILE}" ]; then
+		echo ${BB_TASKHASH} > ${CONFIGURESTAMPFILE}
+	fi
 }
 
 autotools_do_install() {

Re: [PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[Mark Hatle]

I'm curious, is there any [easy] way we can force a rerun of configure as a test 
pass over the system?

I'd like a way to verify that both this patch works as expected, and future 
recipes work as expected.  (It would also be nice to test the things that don't 
use the autotools.bbclass..)

--Mark

On 9/7/12 11:05 AM, Richard Purdie wrote:
> Unfortunately whilst reruning configure and make against a project will mostly
> work there are situations where it does not correctly do the right thing.
>
> In particular, eglibc and gcc will fail out with errors where settings
> do not match a previously built configuration. It could be argued they are
> broken but the situation is what it is. There is the possibility of more subtle
> errors too.
>
> This patch adds a "make distclean" call to recipes where configure is
> rerunning and the sstate checksum for do_configure has changed. We could
> simply use a stamp but saving out the previous configuration checksum
> adds some data at no real overhead.
>
> If we find there are things out there which don't have a "distclean" target,
> we can disable this behaviour with CONFIGURESTAMPFILE = "" in the recipe,
> or users could disable it globally.
>
> [YOCTO #2774]
> [YOCTO #2848]
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/meta/classes/autotools.bbclass b/meta/classes/autotools.bbclass
> index 4c4bf87..1ab2e0c 100644
> --- a/meta/classes/autotools.bbclass
> +++ b/meta/classes/autotools.bbclass
> @@ -89,7 +89,16 @@ oe_runconf () {
>
>   AUTOTOOLS_AUXDIR ?= "${S}"
>
> +CONFIGURESTAMPFILE = "${WORKDIR}/configure.sstate"
> +
>   autotools_do_configure() {
> +	if [ -n "${CONFIGURESTAMPFILE}" -a -e "${CONFIGURESTAMPFILE}" ]; then
> +		if [ "`cat ${CONFIGURESTAMPFILE}`" != "${BB_TASKHASH}" ]; then
> +			echo "Previously configured build detected, running make distclean"
> +			oe_runmake distclean
> +		fi
> +	fi
> +
>   	case ${PN} in
>   	autoconf*)
>   	;;
> @@ -180,6 +189,9 @@ autotools_do_configure() {
>   	else
>   		bbnote "nothing to configure"
>   	fi
> +	if [ -n "${CONFIGURESTAMPFILE}" ]; then
> +		echo ${BB_TASKHASH} > ${CONFIGURESTAMPFILE}
> +	fi
>   }
>
>   autotools_do_install() {
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[Richard Purdie]

On Fri, 2012-09-07 at 11:32 -0500, Mark Hatle wrote:
> I'm curious, is there any [easy] way we can force a rerun of configure as a test 
> pass over the system?

rm tmp/stamps/*/*do_configure*

along with a rename of the sstate cache so it doesn't get used.

> I'd like a way to verify that both this patch works as expected, and future 
> recipes work as expected.  (It would also be nice to test the things that don't 
> use the autotools.bbclass..)

Cheers,

Richard

Re: [PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[Burton, Ross]

On 7 September 2012 17:32, Mark Hatle <mark.hatle@windriver.com> wrote:
> I'm curious, is there any [easy] way we can force a rerun of configure as a
> test pass over the system?
>
> I'd like a way to verify that both this patch works as expected, and future
> recipes work as expected.  (It would also be nice to test the things that
> don't use the autotools.bbclass..)

Yeah, I expect we'll discover some cases when upstream just don't
expect a distclean.  By generally taking from git and re-running the
entire autotools we *should* be okay, but...

Ross

Re: [PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[Richard Purdie]

On Fri, 2012-09-07 at 17:42 +0100, Burton, Ross wrote:
> On 7 September 2012 17:32, Mark Hatle <mark.hatle@windriver.com> wrote:
> > I'm curious, is there any [easy] way we can force a rerun of configure as a
> > test pass over the system?
> >
> > I'd like a way to verify that both this patch works as expected, and future
> > recipes work as expected.  (It would also be nice to test the things that
> > don't use the autotools.bbclass..)
> 
> Yeah, I expect we'll discover some cases when upstream just don't
> expect a distclean.  By generally taking from git and re-running the
> entire autotools we *should* be okay, but...

Further testing suggests we either going to need a whitelist or a
blacklist for this :/

The key places people get bitten are eglibc and gcc so those should be
straight forward to test, the question is how widely to deploy this
initially. I think the mechanism is good, its now just a question of the
implementation detail.

FWIW, libgpg-error fails with checksum issues (checksumming a generated
file?!) and libtool has issues about cleaning directories that have
makefiles that were never generated...

Cheers,

Richard

Re: [PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[Phil Blundell]

On Fri, 2012-09-07 at 18:09 +0100, Richard Purdie wrote:
> The key places people get bitten are eglibc and gcc so those should be
> straight forward to test, the question is how widely to deploy this
> initially. I think the mechanism is good, its now just a question of the
> implementation detail.

Eglibc and gcc, at least, support building with ${B} != ${S} so it might
be easier/quicker to just blow away the whole ${B} tree rather than
trying to distclean it.

I imagine that most modern-ish autotools-based packages will also build
fine in that configuration, though there are bound to be some that
don't.  It's hard to say whether there are likely to be more or fewer of
those than there are ones where "make distclean" fails.

p.

Re: [PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[Khem Raj]

On (07/09/12 18:15), Phil Blundell wrote:
> On Fri, 2012-09-07 at 18:09 +0100, Richard Purdie wrote:
> > The key places people get bitten are eglibc and gcc so those should be
> > straight forward to test, the question is how widely to deploy this
> > initially. I think the mechanism is good, its now just a question of the
> > implementation detail.
> 
> Eglibc and gcc, at least, support building with ${B} != ${S} so it might
> be easier/quicker to just blow away the whole ${B} tree rather than
> trying to distclean it.

yeah and I think separating B from S in general has merits in long run
> 
> I imagine that most modern-ish autotools-based packages will also build
> fine in that configuration, though there are bound to be some that
> don't.  It's hard to say whether there are likely to be more or fewer of
> those than there are ones where "make distclean" fails.
> 

either way there seems to be same amount of uncertainity.

> p.
> 
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

-- 
-Khem

Re: [PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[Mark Hatle]

On 9/8/12 3:05 AM, Khem Raj wrote:
> On (07/09/12 18:15), Phil Blundell wrote:
>> On Fri, 2012-09-07 at 18:09 +0100, Richard Purdie wrote:
>>> The key places people get bitten are eglibc and gcc so those should be
>>> straight forward to test, the question is how widely to deploy this
>>> initially. I think the mechanism is good, its now just a question of the
>>> implementation detail.
>>
>> Eglibc and gcc, at least, support building with ${B} != ${S} so it might
>> be easier/quicker to just blow away the whole ${B} tree rather than
>> trying to distclean it.
>
> yeah and I think separating B from S in general has merits in long run

In principal I agree completely, however in practice at least 20-30% of the 
packages I try won't deal w/ the B/S separation properly.

A good optimization is likely when B != S, blow away B and start over... if they 
are the same, the distclean is likely the best approach..

>>
>> I imagine that most modern-ish autotools-based packages will also build
>> fine in that configuration, though there are bound to be some that
>> don't.  It's hard to say whether there are likely to be more or fewer of
>> those than there are ones where "make distclean" fails.
>>
>
> either way there seems to be same amount of uncertainity.
>
>> p.
>>
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[Colin Walters]

On Sat, 2012-09-08 at 08:30 -0500, Mark Hatle wrote:

> In principal I agree completely, however in practice at least 20-30% of the 
> packages I try won't deal w/ the B/S separation properly.

So what I did in my most recent build system[1] is:

1) Default to srcdir != builddir
2) Tag modules which don't support that

See: http://people.gnome.org/~walters/docs/build-api.txt

If a module doesn't support srcdir != builddir, I simply copy the source
tree to the builddir.  This is can be a confusing trap from a developer
perspective; I've caught myself editing the copied source files, and
then had them blown away during a build. 

But in general we should pressure upstreams to support srcdir !=
builddir, because the benefits are great and the drawbacks small.

It'd be awesome if you guys considered having the bitbake autotools
class implement the same semantics.

[1]
http://git.gnome.org/browse/gnome-ostree/tree/src/ostbuild/pyostbuild/builtin_compile_one.py#n132

Re: [PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[Colin Walters]

On Fri, 2012-09-07 at 17:05 +0100, Richard Purdie wrote:
> Unfortunately whilst reruning configure and make against a project will mostly
> work there are situations where it does not correctly do the right thing.
> 
> In particular, eglibc and gcc will fail out with errors where settings
> do not match a previously built configuration. It could be argued they are
> broken but the situation is what it is. There is the possibility of more subtle
> errors too.
> 
> This patch adds a "make distclean" call to recipes where configure is
> rerunning and the sstate checksum for do_configure has changed. We could
> simply use a stamp but saving out the previous configuration checksum
> adds some data at no real overhead.

The major problem with distclean is that it completely falls over in the
scenario where the source code has changed; the new distclean won't know
about *old* object files it no longer builds, files renamed, etc.

In GNOME we just this cycle landed this patch to use "git clean -dfx"
instead of "make distclean" if possible:
https://bugzilla.gnome.org/show_bug.cgi?id=656081

It might be interesting to have an option to run:
"git init; git add .; git commit -a -m auto-import" on each tarball
build in OE.  I've been considering doing this for jhbuild.

(My latest build system *only* builds from git repositories, so
 it works there =) )

Re: [PATCH] autotools.bbclass: Add functionality to force a distclean when reconfiguring

[Mark Hatle]

On 9/8/12 10:44 AM, Colin Walters wrote:
> On Fri, 2012-09-07 at 17:05 +0100, Richard Purdie wrote:
>> Unfortunately whilst reruning configure and make against a project will mostly
>> work there are situations where it does not correctly do the right thing.
>>
>> In particular, eglibc and gcc will fail out with errors where settings
>> do not match a previously built configuration. It could be argued they are
>> broken but the situation is what it is. There is the possibility of more subtle
>> errors too.
>>
>> This patch adds a "make distclean" call to recipes where configure is
>> rerunning and the sstate checksum for do_configure has changed. We could
>> simply use a stamp but saving out the previous configuration checksum
>> adds some data at no real overhead.
>
> The major problem with distclean is that it completely falls over in the
> scenario where the source code has changed; the new distclean won't know
> about *old* object files it no longer builds, files renamed, etc.
>
> In GNOME we just this cycle landed this patch to use "git clean -dfx"
> instead of "make distclean" if possible:
> https://bugzilla.gnome.org/show_bug.cgi?id=656081
>
> It might be interesting to have an option to run:
> "git init; git add .; git commit -a -m auto-import" on each tarball
> build in OE.  I've been considering doing this for jhbuild.
>
> (My latest build system *only* builds from git repositories, so
>   it works there =) )

While we're not managing the patching w/ git.. we are with quilt.  Does quilt 
have any facilities for restoring to the pristine last copy vs whatever is in 
the current tree?  (I don't think it does BTW.)

--Mark

>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>