[PATCH] modpost: Fix secondary errors seen if a single module build fails

[PATCH] modpost: Fix secondary errors seen if a single module build fails

[Guenter Roeck]

Commit ea4054a23 (modpost: handle huge numbers of modules) added
support for building a large number of modules.

Unfortunately, the commit changed the semantics of the makefile: Instead of
passing only existing object files to modpost, make now passes all expected
object files. If make was started with option -i, this results in a modpost
error if a single file failed to build.

Example with the current btrfs build falure on m68k:

fs/btrfs/btrfs.o: No such file or directory
make[1]: [__modpost] Error 1 (ignored)

This error is followed by lots of errors such as:

m68k-linux-gcc: error: arch/m68k/emu/nfcon.mod.c: No such file or directory
m68k-linux-gcc: fatal error: no input files
compilation terminated.
make[1]: [arch/m68k/emu/nfcon.mod.o] Error 1 (ignored)

This doesn't matter much for normal builds, but it is annoying for builds
started with "make -i" due to the large number of secondary errors.
Those errors unnececessarily clog any error log and make it difficult
to find the real errors in the build.

Fix the problem by only passing existing object files to modpost.

Cc: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Guenter Roeck <linux@roeck-us.net>
---
 scripts/Makefile.modpost |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
index 8dcdca2..387c806 100644
--- a/scripts/Makefile.modpost
+++ b/scripts/Makefile.modpost
@@ -81,7 +81,8 @@ modpost = scripts/mod/modpost                    \
 
 # We can go over command line length here, so be careful.
 quiet_cmd_modpost = MODPOST $(words $(filter-out vmlinux FORCE, $^)) modules
-      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$$/.o/' | $(modpost) -s -T -
+      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$$/.o/' | \
+	while read a; do [ -f $$a ] && echo $$a; done | $(modpost) -s -T -
 
 PHONY += __modpost
 __modpost: $(modules:.ko=.o) FORCE
-- 
1.7.9.7

Re: [PATCH] modpost: Fix secondary errors seen if a single module build fails

[Geert Uytterhoeven]

On Sun, Sep 15, 2013 at 6:39 AM, Guenter Roeck <linux@roeck-us.net> wrote:
> -      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$/.o/' | $(modpost) -s -T -
> +      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$/.o/' | \
> +       while read a; do [ -f $a ] && echo $a; done | $(modpost) -s -T -

I'm wondering whether this can be filtered without using a shell while loop?
E.g. using "ls"?

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH] modpost: Fix secondary errors seen if a single module build fails

[Guenter Roeck]

On 09/15/2013 01:15 AM, Geert Uytterhoeven wrote:
> On Sun, Sep 15, 2013 at 6:39 AM, Guenter Roeck <linux@roeck-us.net> wrote:
>> -      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$/.o/' | $(modpost) -s -T -
>> +      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$/.o/' | \
>> +       while read a; do [ -f $a ] && echo $a; done | $(modpost) -s -T -
>
> I'm wondering whether this can be filtered without using a shell while loop?
> E.g. using "ls"?
>

An alternative would be "$(MODLISTCMD) | sed 's/\.ko$/.o/' | xargs -r ls 2>/dev/null | ..."

I was a bit concerned about side effects of unexpected ls output.
But I'll be happy to change it along that line if people think
that it is better (and/or less ugly).

Guenter

Re: [PATCH] modpost: Fix secondary errors seen if a single module build fails

[Guenter Roeck]

On Sun, Sep 15, 2013 at 10:15:47AM +0200, Geert Uytterhoeven wrote:
> On Sun, Sep 15, 2013 at 6:39 AM, Guenter Roeck <linux@roeck-us.net> wrote:
> > -      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$/.o/' | $(modpost) -s -T -
> > +      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$/.o/' | \
> > +       while read a; do [ -f $a ] && echo $a; done | $(modpost) -s -T -
> 
> I'm wondering whether this can be filtered without using a shell while loop?
> E.g. using "ls"?
> 
I sent another version using 'ls'. Using 'find' would be possible as well,
though both need an error redirect. Hope one of the versions is acceptable.

If anyone has a better idea, please let me know.

Thanks,
Guenter

Re: [PATCH] modpost: Fix secondary errors seen if a single module build fails

[Michal Marek]

Dne 15.9.2013 06:39, Guenter Roeck napsal(a):
> Commit ea4054a23 (modpost: handle huge numbers of modules) added
> support for building a large number of modules.
> 
> Unfortunately, the commit changed the semantics of the makefile: Instead of
> passing only existing object files to modpost, make now passes all expected
> object files. If make was started with option -i, this results in a modpost
> error if a single file failed to build.
> 
> Example with the current btrfs build falure on m68k:
> 
> fs/btrfs/btrfs.o: No such file or directory
> make[1]: [__modpost] Error 1 (ignored)
> 
> This error is followed by lots of errors such as:
> 
> m68k-linux-gcc: error: arch/m68k/emu/nfcon.mod.c: No such file or directory
> m68k-linux-gcc: fatal error: no input files
> compilation terminated.
> make[1]: [arch/m68k/emu/nfcon.mod.o] Error 1 (ignored)
> 
> This doesn't matter much for normal builds, but it is annoying for builds
> started with "make -i" due to the large number of secondary errors.
> Those errors unnececessarily clog any error log and make it difficult
> to find the real errors in the build.
> 
> Fix the problem by only passing existing object files to modpost.
> 
> Cc: Rusty Russell <rusty@rustcorp.com.au>
> Signed-off-by: Guenter Roeck <linux@roeck-us.net>
> ---
>  scripts/Makefile.modpost |    3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
> index 8dcdca2..387c806 100644
> --- a/scripts/Makefile.modpost
> +++ b/scripts/Makefile.modpost
> @@ -81,7 +81,8 @@ modpost = scripts/mod/modpost                    \
>  
>  # We can go over command line length here, so be careful.
>  quiet_cmd_modpost = MODPOST $(words $(filter-out vmlinux FORCE, $^)) modules
> -      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$$/.o/' | $(modpost) -s -T -
> +      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$$/.o/' | \
> +	while read a; do [ -f $$a ] && echo $$a; done | $(modpost) -s -T -

Can you do this filtering only if make -i is used ('i' is present in
$(MAKEFLAGS)), to not hide potential buildsystem bugs? Regarding shell
loop vs. ls, maybe the cleanest way would be to add an option to modpost
to ignore missing files.

Thanks,
Michal

Re: [PATCH] modpost: Fix secondary errors seen if a single module build fails

[Guenter Roeck]

On 09/17/2013 05:34 AM, Michal Marek wrote:
> Dne 15.9.2013 06:39, Guenter Roeck napsal(a):
>> Commit ea4054a23 (modpost: handle huge numbers of modules) added
>> support for building a large number of modules.
>>
>> Unfortunately, the commit changed the semantics of the makefile: Instead of
>> passing only existing object files to modpost, make now passes all expected
>> object files. If make was started with option -i, this results in a modpost
>> error if a single file failed to build.
>>
>> Example with the current btrfs build falure on m68k:
>>
>> fs/btrfs/btrfs.o: No such file or directory
>> make[1]: [__modpost] Error 1 (ignored)
>>
>> This error is followed by lots of errors such as:
>>
>> m68k-linux-gcc: error: arch/m68k/emu/nfcon.mod.c: No such file or directory
>> m68k-linux-gcc: fatal error: no input files
>> compilation terminated.
>> make[1]: [arch/m68k/emu/nfcon.mod.o] Error 1 (ignored)
>>
>> This doesn't matter much for normal builds, but it is annoying for builds
>> started with "make -i" due to the large number of secondary errors.
>> Those errors unnececessarily clog any error log and make it difficult
>> to find the real errors in the build.
>>
>> Fix the problem by only passing existing object files to modpost.
>>
>> Cc: Rusty Russell <rusty@rustcorp.com.au>
>> Signed-off-by: Guenter Roeck <linux@roeck-us.net>
>> ---
>>   scripts/Makefile.modpost |    3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
>> index 8dcdca2..387c806 100644
>> --- a/scripts/Makefile.modpost
>> +++ b/scripts/Makefile.modpost
>> @@ -81,7 +81,8 @@ modpost = scripts/mod/modpost                    \
>>
>>   # We can go over command line length here, so be careful.
>>   quiet_cmd_modpost = MODPOST $(words $(filter-out vmlinux FORCE, $^)) modules
>> -      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$$/.o/' | $(modpost) -s -T -
>> +      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$$/.o/' | \
>> +	while read a; do [ -f $$a ] && echo $$a; done | $(modpost) -s -T -
>
> Can you do this filtering only if make -i is used ('i' is present in
> $(MAKEFLAGS)), to not hide potential buildsystem bugs? Regarding shell
> loop vs. ls, maybe the cleanest way would be to add an option to modpost
> to ignore missing files.
>

Possibly, but I don't really see the point, as the rest of the makefile
does and always did the same filtering already (using $(wildcard ...)
and it doesn't make sense to even try to run modpost on a file that
does not exist.

Sure, it might be a possibility to drop all the wildcard and other filtering
conditionally and only keep it if -i was specified, but I think that should
be a separate patch as it would add its own risks and complexities.

Guenter

Re: [PATCH] modpost: Fix secondary errors seen if a single module build fails

[Guenter Roeck]

On Tue, Sep 17, 2013 at 02:34:58PM +0200, Michal Marek wrote:
> Dne 15.9.2013 06:39, Guenter Roeck napsal(a):
> > Commit ea4054a23 (modpost: handle huge numbers of modules) added
> > support for building a large number of modules.
> > 
> > Unfortunately, the commit changed the semantics of the makefile: Instead of
> > passing only existing object files to modpost, make now passes all expected
> > object files. If make was started with option -i, this results in a modpost
> > error if a single file failed to build.
> > 
> > Example with the current btrfs build falure on m68k:
> > 
> > fs/btrfs/btrfs.o: No such file or directory
> > make[1]: [__modpost] Error 1 (ignored)
> > 
> > This error is followed by lots of errors such as:
> > 
> > m68k-linux-gcc: error: arch/m68k/emu/nfcon.mod.c: No such file or directory
> > m68k-linux-gcc: fatal error: no input files
> > compilation terminated.
> > make[1]: [arch/m68k/emu/nfcon.mod.o] Error 1 (ignored)
> > 
> > This doesn't matter much for normal builds, but it is annoying for builds
> > started with "make -i" due to the large number of secondary errors.
> > Those errors unnececessarily clog any error log and make it difficult
> > to find the real errors in the build.
> > 
> > Fix the problem by only passing existing object files to modpost.
> > 
> > Cc: Rusty Russell <rusty@rustcorp.com.au>
> > Signed-off-by: Guenter Roeck <linux@roeck-us.net>
> > ---
> >  scripts/Makefile.modpost |    3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
> > index 8dcdca2..387c806 100644
> > --- a/scripts/Makefile.modpost
> > +++ b/scripts/Makefile.modpost
> > @@ -81,7 +81,8 @@ modpost = scripts/mod/modpost                    \
> >  
> >  # We can go over command line length here, so be careful.
> >  quiet_cmd_modpost = MODPOST $(words $(filter-out vmlinux FORCE, $^)) modules
> > -      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$$/.o/' | $(modpost) -s -T -
> > +      cmd_modpost = $(MODLISTCMD) | sed 's/\.ko$$/.o/' | \
> > +	while read a; do [ -f $$a ] && echo $$a; done | $(modpost) -s -T -
> 
> Can you do this filtering only if make -i is used ('i' is present in
> $(MAKEFLAGS)), to not hide potential buildsystem bugs? Regarding shell
> loop vs. ls, maybe the cleanest way would be to add an option to modpost
> to ignore missing files.
> 
To follow up on this - are you at the Linux Plumbers conference ?
If yes maybe we can meet briefly and discuss how to proceed.
I'd volunteer to create a patch to add the above functionality if
people think it is valuable, but I would like to keep it as
separate patch.

Thanks,
Guenter

Re: [PATCH] modpost: Fix secondary errors seen if a single module build fails

[Michal Marek]

On 18.9.2013 16:05, Guenter Roeck wrote:
> To follow up on this - are you at the Linux Plumbers conference ?
> If yes maybe we can meet briefly and discuss how to proceed.
> I'd volunteer to create a patch to add the above functionality if
> people think it is valuable, but I would like to keep it as
> separate patch.

Sorry, I was not at the LPC and had vacation at that time. I see Rusty
already merged your patch, that's great.

Michal