[kvm-unit-tests PATCH] build: retain intermediate .aux.o targets

[kvm-unit-tests PATCH] build: retain intermediate .aux.o targets

[Nicholas Piggin]

arm, powerpc, riscv, build .aux.o targets with implicit pattern rules
in dependency chains that cause them to be made as intermediate files,
which get removed when make finishes. This results in unnecessary
partial rebuilds. If make is run again, this time the .aux.o targets
are not intermediate, possibly due to being made via different
dependencies.

Adding .aux.o files to .PRECIOUS prevents them being removed and solves
the rebuild problem.

s390x does not have the problem because .SECONDARY prevents dependancies
from being built as intermediate. However the same change is made for
s390x, for consistency.

Suggested-by: Marc Hartmayer <mhartmay@linux.ibm.com>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arm/Makefile.common     | 2 +-
 powerpc/Makefile.common | 2 +-
 riscv/Makefile          | 2 +-
 s390x/Makefile          | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/arm/Makefile.common b/arm/Makefile.common
index f828dbe01..0b26a92a6 100644
--- a/arm/Makefile.common
+++ b/arm/Makefile.common
@@ -31,7 +31,7 @@ CFLAGS += -O2
 CFLAGS += -I $(SRCDIR)/lib -I $(SRCDIR)/lib/libfdt -I lib
 
 # We want to keep intermediate files
-.PRECIOUS: %.elf %.o
+.PRECIOUS: %.elf %.o %.aux.o
 
 asm-offsets = lib/$(ARCH)/asm-offsets.h
 include $(SRCDIR)/scripts/asm-offsets.mak
diff --git a/powerpc/Makefile.common b/powerpc/Makefile.common
index b98f71c2f..16f14577e 100644
--- a/powerpc/Makefile.common
+++ b/powerpc/Makefile.common
@@ -30,7 +30,7 @@ CFLAGS += -I $(SRCDIR)/lib -I $(SRCDIR)/lib/libfdt -I lib
 CFLAGS += -Wa,-mregnames
 
 # We want to keep intermediate files
-.PRECIOUS: %.o
+.PRECIOUS: %.o %.aux.o
 
 asm-offsets = lib/$(ARCH)/asm-offsets.h
 include $(SRCDIR)/scripts/asm-offsets.mak
diff --git a/riscv/Makefile b/riscv/Makefile
index 919a3ebb5..7207ff988 100644
--- a/riscv/Makefile
+++ b/riscv/Makefile
@@ -53,7 +53,7 @@ AUXFLAGS ?= 0x0
 KEEP_FRAME_POINTER := y
 
 # We want to keep intermediate files
-.PRECIOUS: %.elf %.o
+.PRECIOUS: %.elf %.o %.aux.o
 
 define arch_elf_check =
 	$(if $(shell ! $(READELF) -rW $(1) >&/dev/null && echo "nok"),
diff --git a/s390x/Makefile b/s390x/Makefile
index 23342bd64..d436c6e9a 100644
--- a/s390x/Makefile
+++ b/s390x/Makefile
@@ -85,7 +85,7 @@ CFLAGS += -fno-delete-null-pointer-checks
 LDFLAGS += -Wl,--build-id=none
 
 # We want to keep intermediate files
-.PRECIOUS: %.o %.lds
+.PRECIOUS: %.o %.aux.o %.lds
 
 asm-offsets = lib/$(ARCH)/asm-offsets.h
 include $(SRCDIR)/scripts/asm-offsets.mak
-- 
2.45.1

Re: [kvm-unit-tests PATCH] build: retain intermediate .aux.o targets

[Marc Hartmayer]

On Wed, Jun 12, 2024 at 02:42 PM +1000, Nicholas Piggin <npiggin@gmail.com> wrote:
> arm, powerpc, riscv, build .aux.o targets with implicit pattern rules
> in dependency chains that cause them to be made as intermediate files,
> which get removed when make finishes. This results in unnecessary
> partial rebuilds. If make is run again, this time the .aux.o targets
> are not intermediate, possibly due to being made via different
> dependencies.
>
> Adding .aux.o files to .PRECIOUS prevents them being removed and solves
> the rebuild problem.
>
> s390x does not have the problem because .SECONDARY prevents dependancies
> from being built as intermediate. However the same change is made for
> s390x, for consistency.
>
> Suggested-by: Marc Hartmayer <mhartmay@linux.ibm.com>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---

[…snip…]

> @@ -85,7 +85,7 @@ CFLAGS += -fno-delete-null-pointer-checks
>  LDFLAGS += -Wl,--build-id=none
>  
>  # We want to keep intermediate files
> -.PRECIOUS: %.o %.lds
> +.PRECIOUS: %.o %.aux.o %.lds
>  
>  asm-offsets = lib/$(ARCH)/asm-offsets.h
>  include $(SRCDIR)/scripts/asm-offsets.mak
> -- 
> 2.45.1
>
>

Thanks for fixing this!

Reviewed-by: Marc Hartmayer <mhartmay@linux.ibm.com>

-- 
Kind regards / Beste Grüße
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Wolfgang Wendt
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

Re: [kvm-unit-tests PATCH] build: retain intermediate .aux.o targets

[Andrew Jones]

On Wed, Jun 12, 2024 at 02:42:32PM GMT, Nicholas Piggin wrote:
> arm, powerpc, riscv, build .aux.o targets with implicit pattern rules
> in dependency chains that cause them to be made as intermediate files,
> which get removed when make finishes. This results in unnecessary
> partial rebuilds. If make is run again, this time the .aux.o targets
> are not intermediate, possibly due to being made via different
> dependencies.
> 
> Adding .aux.o files to .PRECIOUS prevents them being removed and solves
> the rebuild problem.
> 
> s390x does not have the problem because .SECONDARY prevents dependancies
> from being built as intermediate. However the same change is made for
> s390x, for consistency.
> 
> Suggested-by: Marc Hartmayer <mhartmay@linux.ibm.com>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
>  arm/Makefile.common     | 2 +-
>  powerpc/Makefile.common | 2 +-
>  riscv/Makefile          | 2 +-
>  s390x/Makefile          | 2 +-
>  4 files changed, 4 insertions(+), 4 deletions(-)
>

Acked-by: Andrew Jones <andrew.jones@linux.dev>

Re: [kvm-unit-tests PATCH] build: retain intermediate .aux.o targets

[Segher Boessenkool]

On Wed, Jun 12, 2024 at 02:42:32PM +1000, Nicholas Piggin wrote:
> arm, powerpc, riscv, build .aux.o targets with implicit pattern rules
> in dependency chains that cause them to be made as intermediate files,
> which get removed when make finishes. This results in unnecessary
> partial rebuilds. If make is run again, this time the .aux.o targets
> are not intermediate, possibly due to being made via different
> dependencies.
> 
> Adding .aux.o files to .PRECIOUS prevents them being removed and solves
> the rebuild problem.
> 
> s390x does not have the problem because .SECONDARY prevents dependancies
> from being built as intermediate. However the same change is made for
> s390x, for consistency.

This is exactly what .SECONDARY is for, as its documentation says,
even.  Wouldn't it be better to just add a .SECONDARY to the other
targets as well?


Segher

Re: [kvm-unit-tests PATCH] build: retain intermediate .aux.o targets

[Nicholas Piggin]

On Wed Jun 12, 2024 at 6:28 PM AEST, Segher Boessenkool wrote:
> On Wed, Jun 12, 2024 at 02:42:32PM +1000, Nicholas Piggin wrote:
> > arm, powerpc, riscv, build .aux.o targets with implicit pattern rules
> > in dependency chains that cause them to be made as intermediate files,
> > which get removed when make finishes. This results in unnecessary
> > partial rebuilds. If make is run again, this time the .aux.o targets
> > are not intermediate, possibly due to being made via different
> > dependencies.
> > 
> > Adding .aux.o files to .PRECIOUS prevents them being removed and solves
> > the rebuild problem.
> > 
> > s390x does not have the problem because .SECONDARY prevents dependancies
> > from being built as intermediate. However the same change is made for
> > s390x, for consistency.
>
> This is exactly what .SECONDARY is for, as its documentation says,
> even.  Wouldn't it be better to just add a .SECONDARY to the other
> targets as well?

Yeah we were debating that and agreed .PRECIOUS may not be the
cleanest fix but since we already use that it's okay for a
minimal fix.

Thanks,
Nick

Re: [kvm-unit-tests PATCH] build: retain intermediate .aux.o targets

[Segher Boessenkool]

On Fri, Jun 14, 2024 at 10:43:39AM +1000, Nicholas Piggin wrote:
> On Wed Jun 12, 2024 at 6:28 PM AEST, Segher Boessenkool wrote:
> > On Wed, Jun 12, 2024 at 02:42:32PM +1000, Nicholas Piggin wrote:
> > > arm, powerpc, riscv, build .aux.o targets with implicit pattern rules
> > > in dependency chains that cause them to be made as intermediate files,
> > > which get removed when make finishes. This results in unnecessary
> > > partial rebuilds. If make is run again, this time the .aux.o targets
> > > are not intermediate, possibly due to being made via different
> > > dependencies.
> > > 
> > > Adding .aux.o files to .PRECIOUS prevents them being removed and solves
> > > the rebuild problem.
> > > 
> > > s390x does not have the problem because .SECONDARY prevents dependancies
> > > from being built as intermediate. However the same change is made for
> > > s390x, for consistency.
> >
> > This is exactly what .SECONDARY is for, as its documentation says,
> > even.  Wouldn't it be better to just add a .SECONDARY to the other
> > targets as well?
> 
> Yeah we were debating that and agreed .PRECIOUS may not be the
> cleanest fix but since we already use that it's okay for a
> minimal fix.

But why add it to s390x then?  It is not a fix there at all!


Segher

Re: [kvm-unit-tests PATCH] build: retain intermediate .aux.o targets

[Nicholas Piggin]

On Fri Jun 14, 2024 at 11:08 AM AEST, Segher Boessenkool wrote:
> On Fri, Jun 14, 2024 at 10:43:39AM +1000, Nicholas Piggin wrote:
> > On Wed Jun 12, 2024 at 6:28 PM AEST, Segher Boessenkool wrote:
> > > On Wed, Jun 12, 2024 at 02:42:32PM +1000, Nicholas Piggin wrote:
> > > > arm, powerpc, riscv, build .aux.o targets with implicit pattern rules
> > > > in dependency chains that cause them to be made as intermediate files,
> > > > which get removed when make finishes. This results in unnecessary
> > > > partial rebuilds. If make is run again, this time the .aux.o targets
> > > > are not intermediate, possibly due to being made via different
> > > > dependencies.
> > > > 
> > > > Adding .aux.o files to .PRECIOUS prevents them being removed and solves
> > > > the rebuild problem.
> > > > 
> > > > s390x does not have the problem because .SECONDARY prevents dependancies
> > > > from being built as intermediate. However the same change is made for
> > > > s390x, for consistency.
> > >
> > > This is exactly what .SECONDARY is for, as its documentation says,
> > > even.  Wouldn't it be better to just add a .SECONDARY to the other
> > > targets as well?
> > 
> > Yeah we were debating that and agreed .PRECIOUS may not be the
> > cleanest fix but since we already use that it's okay for a
> > minimal fix.
>
> But why add it to s390x then?  It is not a fix there at all!

Eh, not a big deal. I mentioned that in the changelog it doesn't seem to
pracicaly fix something. And I rather the makefiles converge as much as
possible rather than diverge more.

.SECONDARY was added independently and not to fix this problem in
s390x. And s390x has .SECONDARY slightly wrong AFAIKS. It mentions
.SECONDARY: twice in a way that looks like it was meant to depend on
specific targets, it actually gives it no dependencies and the
resulting semantics are that all intermediate files in the build are
treated as secondary. So somethig there should be cleaned up. If the
.SECONDARY was changed to only depend on the .gobj and .hdr.obj then
suddenly that would break .aux.o if I don't make the change.

So I'm meaning to work out what to do with all that, i.e., whether to
add blanket .SECONDARY for all and trim or remove the .PRECIOUS files,
or remove s390x's secondary, or make it more specific, or something
else. But it takes a while for me to do makefile work.

Thanks,
Nick

Re: [kvm-unit-tests PATCH] build: retain intermediate .aux.o targets

[Nicholas Piggin]

On Fri Jun 14, 2024 at 6:38 PM AEST, Nicholas Piggin wrote:
> On Fri Jun 14, 2024 at 11:08 AM AEST, Segher Boessenkool wrote:
> > On Fri, Jun 14, 2024 at 10:43:39AM +1000, Nicholas Piggin wrote:
> > > On Wed Jun 12, 2024 at 6:28 PM AEST, Segher Boessenkool wrote:
> > > > On Wed, Jun 12, 2024 at 02:42:32PM +1000, Nicholas Piggin wrote:
> > > > > arm, powerpc, riscv, build .aux.o targets with implicit pattern rules
> > > > > in dependency chains that cause them to be made as intermediate files,
> > > > > which get removed when make finishes. This results in unnecessary
> > > > > partial rebuilds. If make is run again, this time the .aux.o targets
> > > > > are not intermediate, possibly due to being made via different
> > > > > dependencies.
> > > > > 
> > > > > Adding .aux.o files to .PRECIOUS prevents them being removed and solves
> > > > > the rebuild problem.
> > > > > 
> > > > > s390x does not have the problem because .SECONDARY prevents dependancies
> > > > > from being built as intermediate. However the same change is made for
> > > > > s390x, for consistency.
> > > >
> > > > This is exactly what .SECONDARY is for, as its documentation says,
> > > > even.  Wouldn't it be better to just add a .SECONDARY to the other
> > > > targets as well?
> > > 
> > > Yeah we were debating that and agreed .PRECIOUS may not be the
> > > cleanest fix but since we already use that it's okay for a
> > > minimal fix.
> >
> > But why add it to s390x then?  It is not a fix there at all!
>
> Eh, not a big deal. I mentioned that in the changelog it doesn't seem to
> pracicaly fix something. And I rather the makefiles converge as much as
> possible rather than diverge more.
>
> .SECONDARY was added independently and not to fix this problem in
> s390x. And s390x has .SECONDARY slightly wrong AFAIKS. It mentions
> .SECONDARY: twice in a way that looks like it was meant to depend on
> specific targets, it actually gives it no dependencies and the
> resulting semantics are that all intermediate files in the build are
> treated as secondary. So somethig there should be cleaned up. If the
> .SECONDARY was changed to only depend on the .gobj and .hdr.obj then
> suddenly that would break .aux.o if I don't make the change.
>
> So I'm meaning to work out what to do with all that, i.e., whether to
> add blanket .SECONDARY for all and trim or remove the .PRECIOUS files,
> or remove s390x's secondary, or make it more specific, or something
> else. But it takes a while for me to do makefile work.

Hi Thomas,

Ping on this patch?

Thanks,
Nick

Re: [kvm-unit-tests PATCH] build: retain intermediate .aux.o targets

[Thomas Huth]

On 26/07/2024 06.15, Nicholas Piggin wrote:
> On Fri Jun 14, 2024 at 6:38 PM AEST, Nicholas Piggin wrote:
>> On Fri Jun 14, 2024 at 11:08 AM AEST, Segher Boessenkool wrote:
>>> On Fri, Jun 14, 2024 at 10:43:39AM +1000, Nicholas Piggin wrote:
>>>> On Wed Jun 12, 2024 at 6:28 PM AEST, Segher Boessenkool wrote:
>>>>> On Wed, Jun 12, 2024 at 02:42:32PM +1000, Nicholas Piggin wrote:
>>>>>> arm, powerpc, riscv, build .aux.o targets with implicit pattern rules
>>>>>> in dependency chains that cause them to be made as intermediate files,
>>>>>> which get removed when make finishes. This results in unnecessary
>>>>>> partial rebuilds. If make is run again, this time the .aux.o targets
>>>>>> are not intermediate, possibly due to being made via different
>>>>>> dependencies.
>>>>>>
>>>>>> Adding .aux.o files to .PRECIOUS prevents them being removed and solves
>>>>>> the rebuild problem.
>>>>>>
>>>>>> s390x does not have the problem because .SECONDARY prevents dependancies
>>>>>> from being built as intermediate. However the same change is made for
>>>>>> s390x, for consistency.
>>>>>
>>>>> This is exactly what .SECONDARY is for, as its documentation says,
>>>>> even.  Wouldn't it be better to just add a .SECONDARY to the other
>>>>> targets as well?
>>>>
>>>> Yeah we were debating that and agreed .PRECIOUS may not be the
>>>> cleanest fix but since we already use that it's okay for a
>>>> minimal fix.
>>>
>>> But why add it to s390x then?  It is not a fix there at all!
>>
>> Eh, not a big deal. I mentioned that in the changelog it doesn't seem to
>> pracicaly fix something. And I rather the makefiles converge as much as
>> possible rather than diverge more.
>>
>> .SECONDARY was added independently and not to fix this problem in
>> s390x. And s390x has .SECONDARY slightly wrong AFAIKS. It mentions
>> .SECONDARY: twice in a way that looks like it was meant to depend on
>> specific targets, it actually gives it no dependencies and the
>> resulting semantics are that all intermediate files in the build are
>> treated as secondary. So somethig there should be cleaned up. If the
>> .SECONDARY was changed to only depend on the .gobj and .hdr.obj then
>> suddenly that would break .aux.o if I don't make the change.
>>
>> So I'm meaning to work out what to do with all that, i.e., whether to
>> add blanket .SECONDARY for all and trim or remove the .PRECIOUS files,
>> or remove s390x's secondary, or make it more specific, or something
>> else. But it takes a while for me to do makefile work.
> 
> Hi Thomas,
> 
> Ping on this patch?

I assumed that Marc would chime in on the s390x part here?

  Thomas