make headers_install fail when path is too long

make headers_install fail when path is too long

[Nicolas Dichtel]

This problem has probably already been reported, but I don't find the fix in a 
kernel.org tree and I don't understand why.

Here is the problem:

root@schnaps:/home/rbb/averyveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryverylongpath/linux(master)>make 
headers_install INSTALL_HDR_PATH=/tmp/tmp/
   CHK     include/generated/uapi/linux/version.h
make[1]: Nothing to be done for `all'.
make[1]: Nothing to be done for `relocs'.
make[2]: execvp: /bin/sh: Argument list too long
make[2]: *** [/tmp/tmp//include/linux/.install] Error 127
make[1]: *** [linux] Error 2
make: *** [headers_install] Error 2

I see that this problem has been reported several times on the web, but I do not 
find why it's not fixed upstream.
Some patches are available: 
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=53cc748b93e8af584557d6db5309c3e955182c5c

Any comment/suggestion is welcome.

Regards,
Nicolas

Re: make headers_install fail when path is too long

[Sam Ravnborg]

On Wed, Mar 06, 2013 at 12:16:33PM +0100, Nicolas Dichtel wrote:
> This problem has probably already been reported, but I don't find the fix 
> in a kernel.org tree and I don't understand why.

First time I see this bug described - but I may missed it here at kbuild.
If a patch exists then please include the patch in your mail so we can
see it and comment on it.

And make sure to have the attribution correct if you consider it to
be applied.

Thanks,
	Sam

Re: make headers_install fail when path is too long

[Michal Marek]

Dne 6.3.2013 17:10, Sam Ravnborg napsal(a):
> On Wed, Mar 06, 2013 at 12:16:33PM +0100, Nicolas Dichtel wrote:
>> This problem has probably already been reported, but I don't find the fix 
>> in a kernel.org tree and I don't understand why.
> 
> First time I see this bug described - but I may missed it here at kbuild.
> If a patch exists then please include the patch in your mail so we can
> see it and comment on it.

Yes. If it's an obvious bug fix like this, simply send it to lkml and
linux-kbuild.

Michal

Re: make headers_install fail when path is too long

[Nicolas Dichtel]

Le 06/03/2013 17:10, Sam Ravnborg a écrit :
> On Wed, Mar 06, 2013 at 12:16:33PM +0100, Nicolas Dichtel wrote:
>> This problem has probably already been reported, but I don't find the fix
>> in a kernel.org tree and I don't understand why.
>
> First time I see this bug described - but I may missed it here at kbuild.
It's probably why I didn't find this on a linux kernel mailing list ;-)

> If a patch exists then please include the patch in your mail so we can
> see it and comment on it.
>
> And make sure to have the attribution correct if you consider it to
> be applied.
I'm not the author of the patch, I've found it on the web. Here it is:

 From cb9c811b8f23277de95dc687e87c6859308e68e6 Mon Sep 17 00:00:00 2001
From: Bruce Ashfield <bruce.ashfield@windriver.com>
Date: Wed, 6 Mar 2013 17:23:51 +0100
Subject: [PATCH] scripts/Makefile.headersinst: install headers from scratch
  file

If headers_install is executed from a deep/long directory structure, the
shell's maximum argument length can be execeeded, which breaks the operation
with:

| make[2]: execvp: /bin/sh: Argument list too long
| make[2]: ***

By dumping the input files to a scratch file and using xargs to read the
input list from the scratch file, we can avoid blowing out the maximum
argument size and install headers in a long path name environment.

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
  scripts/Makefile.headersinst | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst
index 25f216a..c481986 100644
--- a/scripts/Makefile.headersinst
+++ b/scripts/Makefile.headersinst
@@ -71,7 +71,7 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@))
  quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
                              file$(if $(word 2, $(all-files)),s))
        cmd_install = \
-        $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \
+        xargs $(PERL) $< $(installdir) $(SRCARCH) < 
$(INSTALL_HDR_PATH)/.input-files; \
          for F in $(wrapper-files); do                                   \
                  echo "\#include <asm-generic/$$F>" > $(installdir)/$$F;    \
          done;                                                           \
@@ -100,7 +100,9 @@ targets += $(install-file)
  $(install-file): scripts/headers_install.pl $(input-files) FORCE
  	$(if $(unwanted),$(call cmd,remove),)
  	$(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@)))
+	@echo $(input-files) > $(INSTALL_HDR_PATH)/.input-files
  	$(call if_changed,install)
+	@rm $(INSTALL_HDR_PATH)/.input-files

  else
  __headerscheck: $(subdirs) $(check-file)
-- 
1.8.0.1

Re: make headers_install fail when path is too long

[Bruce Ashfield]

On 13-03-06 11:27 AM, Nicolas Dichtel wrote:
> Le 06/03/2013 17:10, Sam Ravnborg a écrit :
>> On Wed, Mar 06, 2013 at 12:16:33PM +0100, Nicolas Dichtel wrote:
>>> This problem has probably already been reported, but I don't find the
>>> fix
>>> in a kernel.org tree and I don't understand why.
>>
>> First time I see this bug described - but I may missed it here at kbuild.
> It's probably why I didn't find this on a linux kernel mailing list ;-)

And it's been on my TODO list to float this as a RFC to the kbuild lists
before 3.9 arrived .. a goal that I managed to miss (again).

The patch in question has us up and installing headers on Yocto kernel/
kernel header packages greater than 3.7, so it has received a decent
amount of testing.

As for it being the perfect approach, I only claim that it works in
the environment that I've been using and it is functional.

If this change turns out to be acceptable, that's great. I can also
spend time re-working as appropriate .. or someone else's solution
can be adopted.

In the end it's a win and a fix, and the thread has prompted me to
get the change floated :)

Cheers,

Bruce

>
>> If a patch exists then please include the patch in your mail so we can
>> see it and comment on it.
>>
>> And make sure to have the attribution correct if you consider it to
>> be applied.
> I'm not the author of the patch, I've found it on the web. Here it is:
>
>  From cb9c811b8f23277de95dc687e87c6859308e68e6 Mon Sep 17 00:00:00 2001
> From: Bruce Ashfield <bruce.ashfield@windriver.com>
> Date: Wed, 6 Mar 2013 17:23:51 +0100
> Subject: [PATCH] scripts/Makefile.headersinst: install headers from scratch
>   file
>
> If headers_install is executed from a deep/long directory structure, the
> shell's maximum argument length can be execeeded, which breaks the
> operation
> with:
>
> | make[2]: execvp: /bin/sh: Argument list too long
> | make[2]: ***
>
> By dumping the input files to a scratch file and using xargs to read the
> input list from the scratch file, we can avoid blowing out the maximum
> argument size and install headers in a long path name environment.
>
> Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
> ---
>   scripts/Makefile.headersinst | 4 +++-
>   1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst
> index 25f216a..c481986 100644
> --- a/scripts/Makefile.headersinst
> +++ b/scripts/Makefile.headersinst
> @@ -71,7 +71,7 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@))
>   quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
>                               file$(if $(word 2, $(all-files)),s))
>         cmd_install = \
> -        $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \
> +        xargs $(PERL) $< $(installdir) $(SRCARCH) <
> $(INSTALL_HDR_PATH)/.input-files; \
>           for F in $(wrapper-files); do                                   \
>                   echo "\#include <asm-generic/$$F>" >
> $(installdir)/$$F;    \
>           done;                                                           \
> @@ -100,7 +100,9 @@ targets += $(install-file)
>   $(install-file): scripts/headers_install.pl $(input-files) FORCE
>       $(if $(unwanted),$(call cmd,remove),)
>       $(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@)))
> +    @echo $(input-files) > $(INSTALL_HDR_PATH)/.input-files
>       $(call if_changed,install)
> +    @rm $(INSTALL_HDR_PATH)/.input-files
>
>   else
>   __headerscheck: $(subdirs) $(check-file)

Re: make headers_install fail when path is too long

[Michal Marek]

Dne 6.3.2013 17:27, Nicolas Dichtel napsal(a):
> -        $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \
[...]
> +    @echo $(input-files) > $(INSTALL_HDR_PATH)/.input-files

Are you sure this is a reliable fix? What make does is to spawn

/bin/sh -c 'echo <long list of files> > usr/include/.input-files'

here. So I guess that it works for you just because "sh -c" is shorter
than "sh -c 'perl scripts/headers_install.pl...'".

Michal

Re: make headers_install fail when path is too long

[Bruce Ashfield]

On 13-03-07 03:26 AM, Michal Marek wrote:
> Dne 6.3.2013 17:27, Nicolas Dichtel napsal(a):
>> -        $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \
> [...]
>> +    @echo $(input-files) > $(INSTALL_HDR_PATH)/.input-files
>
> Are you sure this is a reliable fix? What make does is to spawn
>
> /bin/sh -c 'echo <long list of files> > usr/include/.input-files'

It's what I was referencing in my original email. It works here,
and fixes our install of headers in previously failing environments.
But yes, it does shuffle the args enough to get around the limit, but
there's still a way to have even deeper and longer path names that
could cause failures.

I experimented with loops, and other options as well. But any
construct like "for f in $(input-files)", is both slow and explodes
on the argument length limits just like the original.

>
> here. So I guess that it works for you just because "sh -c" is shorter
> than "sh -c 'perl scripts/headers_install.pl...'".

Partly, yes, but we are more than a few characters over the limit
in my testing. So it shouldn't be the whole story.

Cheers,

Bruce

>
> Michal
>

[PATCH] fix make headers_install when path is too long

[Nicolas Dichtel]

If headers_install is executed from a deep/long directory structure, the
shell's maximum argument length can be execeeded, which breaks the operation
with:

| make[2]: execvp: /bin/sh: Argument list too long
| make[2]: ***

Instead of passing each files name with the entire path, I give only the file
name without the source path and give this path as a new argument to
headers_install.pl.

Because there is three possible path, I have tree input-files list, one per
path.

Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
---

I come back to this issue. Here is another proposal to fix this pb.
Comments are welcome.

 scripts/Makefile.headersinst | 20 ++++++++++++++------
 scripts/headers_install.pl   |  8 ++++----
 2 files changed, 18 insertions(+), 10 deletions(-)

diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst
index 477d137..fbadfc3 100644
--- a/scripts/Makefile.headersinst
+++ b/scripts/Makefile.headersinst
@@ -47,18 +47,24 @@ header-y      := $(filter-out $(generic-y), $(header-y))
 all-files     := $(header-y) $(genhdr-y) $(wrapper-files)
 output-files  := $(addprefix $(installdir)/, $(all-files))
 
-input-files   := $(foreach hdr, $(header-y), \
+input-files1  := $(foreach hdr, $(header-y), \
 		   $(if $(wildcard $(srcdir)/$(hdr)), \
-			$(wildcard $(srcdir)/$(hdr)), \
+			$(wildcard $(srcdir)/$(hdr))) \
+		   )
+input-files1-name := $(notdir $(input-files1))
+input-files2  := $(foreach hdr, $(header-y), \
+		   $(if  $(wildcard $(srcdir)/$(hdr)),, \
 			$(if $(wildcard $(oldsrcdir)/$(hdr)), \
 				$(wildcard $(oldsrcdir)/$(hdr)), \
 				$(error Missing UAPI file $(srcdir)/$(hdr))) \
-		   )) \
-		 $(foreach hdr, $(genhdr-y), \
+		   ))
+input-files2-name := $(notdir $(input-files2))
+input-files3  := $(foreach hdr, $(genhdr-y), \
 		   $(if	$(wildcard $(gendir)/$(hdr)), \
 			$(wildcard $(gendir)/$(hdr)), \
 			$(error Missing generated UAPI file $(gendir)/$(hdr)) \
 		   ))
+input-files3-name := $(notdir $(input-files3))
 
 # Work out what needs to be removed
 oldheaders    := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h))
@@ -72,7 +78,9 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@))
 quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
                             file$(if $(word 2, $(all-files)),s))
       cmd_install = \
-        $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \
+        $(PERL) $< $(installdir) $(SRCARCH) $(srcdir) $(input-files1-name); \
+        $(PERL) $< $(installdir) $(SRCARCH) $(oldsrcdir) $(input-files2-name); \
+        $(PERL) $< $(installdir) $(SRCARCH) $(gendir) $(input-files3-name); \
         for F in $(wrapper-files); do                                   \
                 echo "\#include <asm-generic/$$F>" > $(installdir)/$$F;    \
         done;                                                           \
@@ -98,7 +106,7 @@ __headersinst: $(subdirs) $(install-file)
 	@:
 
 targets += $(install-file)
-$(install-file): scripts/headers_install.pl $(input-files) FORCE
+$(install-file): scripts/headers_install.pl $(input-files1) $(input-files2) $(input-files3) FORCE
 	$(if $(unwanted),$(call cmd,remove),)
 	$(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@)))
 	$(call if_changed,install)
diff --git a/scripts/headers_install.pl b/scripts/headers_install.pl
index 581ca99..87d9700 100644
--- a/scripts/headers_install.pl
+++ b/scripts/headers_install.pl
@@ -3,7 +3,7 @@
 # headers_install prepare the listed header files for use in
 # user space and copy the files to their destination.
 #
-# Usage: headers_install.pl readdir installdir arch [files...]
+# Usage: headers_install.pl readdir installdir arch srcdir [files...]
 # installdir: dir to install the files to
 # arch:       current architecture
 #             arch is used to force a reinstallation when the arch
@@ -17,7 +17,7 @@
 
 use strict;
 
-my ($installdir, $arch, @files) = @ARGV;
+my ($installdir, $arch, $srcdir, @files) = @ARGV;
 
 my $unifdef = "scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__";
 
@@ -27,8 +27,8 @@ foreach my $filename (@files) {
 
 	my $tmpfile = "$installdir/$file.tmp";
 
-	open(my $in, '<', $filename)
-	    or die "$filename: $!\n";
+	open(my $in, '<', "$srcdir/$filename")
+	    or die "$srcdir/$filename: $!\n";
 	open(my $out, '>', $tmpfile)
 	    or die "$tmpfile: $!\n";
 	while (my $line = <$in>) {
-- 
1.8.2.1

Re: [PATCH] fix make headers_install when path is too long

[Bruce Ashfield]

On 13-04-26 12:36 PM, Nicolas Dichtel wrote:
> If headers_install is executed from a deep/long directory structure, the
> shell's maximum argument length can be execeeded, which breaks the operation
> with:
>
> | make[2]: execvp: /bin/sh: Argument list too long
> | make[2]: ***
>
> Instead of passing each files name with the entire path, I give only the file
> name without the source path and give this path as a new argument to
> headers_install.pl.

This will work. I considered the same option when I solved the similar
problem for Yocto installs, but managed to avoid it with a shorter
term fixed.

>
> Because there is three possible path, I have tree input-files list, one per
> path.
>
> Signed-off-by: Nicolas Dichtel<nicolas.dichtel@6wind.com>

Tested-by: Bruce Ashfield <bruce.ashfield@windriver.com>

Bruce

> ---
>
> I come back to this issue. Here is another proposal to fix this pb.
> Comments are welcome.
>
>   scripts/Makefile.headersinst | 20 ++++++++++++++------
>   scripts/headers_install.pl   |  8 ++++----
>   2 files changed, 18 insertions(+), 10 deletions(-)
>
> diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst
> index 477d137..fbadfc3 100644
> --- a/scripts/Makefile.headersinst
> +++ b/scripts/Makefile.headersinst
> @@ -47,18 +47,24 @@ header-y      := $(filter-out $(generic-y), $(header-y))
>   all-files     := $(header-y) $(genhdr-y) $(wrapper-files)
>   output-files  := $(addprefix $(installdir)/, $(all-files))
>
> -input-files   := $(foreach hdr, $(header-y), \
> +input-files1  := $(foreach hdr, $(header-y), \
>   		   $(if $(wildcard $(srcdir)/$(hdr)), \
> -			$(wildcard $(srcdir)/$(hdr)), \
> +			$(wildcard $(srcdir)/$(hdr))) \
> +		   )
> +input-files1-name := $(notdir $(input-files1))
> +input-files2  := $(foreach hdr, $(header-y), \
> +		   $(if  $(wildcard $(srcdir)/$(hdr)),, \
>   			$(if $(wildcard $(oldsrcdir)/$(hdr)), \
>   				$(wildcard $(oldsrcdir)/$(hdr)), \
>   				$(error Missing UAPI file $(srcdir)/$(hdr))) \
> -		   )) \
> -		 $(foreach hdr, $(genhdr-y), \
> +		   ))
> +input-files2-name := $(notdir $(input-files2))
> +input-files3  := $(foreach hdr, $(genhdr-y), \
>   		   $(if	$(wildcard $(gendir)/$(hdr)), \
>   			$(wildcard $(gendir)/$(hdr)), \
>   			$(error Missing generated UAPI file $(gendir)/$(hdr)) \
>   		   ))
> +input-files3-name := $(notdir $(input-files3))
>
>   # Work out what needs to be removed
>   oldheaders    := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h))
> @@ -72,7 +78,9 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@))
>   quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
>                               file$(if $(word 2, $(all-files)),s))
>         cmd_install = \
> -        $(PERL) $<  $(installdir) $(SRCARCH) $(input-files); \
> +        $(PERL) $<  $(installdir) $(SRCARCH) $(srcdir) $(input-files1-name); \
> +        $(PERL) $<  $(installdir) $(SRCARCH) $(oldsrcdir) $(input-files2-name); \
> +        $(PERL) $<  $(installdir) $(SRCARCH) $(gendir) $(input-files3-name); \
>           for F in $(wrapper-files); do                                   \
>                   echo "\#include<asm-generic/$$F>">  $(installdir)/$$F;    \
>           done;                                                           \
> @@ -98,7 +106,7 @@ __headersinst: $(subdirs) $(install-file)
>   	@:
>
>   targets += $(install-file)
> -$(install-file): scripts/headers_install.pl $(input-files) FORCE
> +$(install-file): scripts/headers_install.pl $(input-files1) $(input-files2) $(input-files3) FORCE
>   	$(if $(unwanted),$(call cmd,remove),)
>   	$(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@)))
>   	$(call if_changed,install)
> diff --git a/scripts/headers_install.pl b/scripts/headers_install.pl
> index 581ca99..87d9700 100644
> --- a/scripts/headers_install.pl
> +++ b/scripts/headers_install.pl
> @@ -3,7 +3,7 @@
>   # headers_install prepare the listed header files for use in
>   # user space and copy the files to their destination.
>   #
> -# Usage: headers_install.pl readdir installdir arch [files...]
> +# Usage: headers_install.pl readdir installdir arch srcdir [files...]
>   # installdir: dir to install the files to
>   # arch:       current architecture
>   #             arch is used to force a reinstallation when the arch
> @@ -17,7 +17,7 @@
>
>   use strict;
>
> -my ($installdir, $arch, @files) = @ARGV;
> +my ($installdir, $arch, $srcdir, @files) = @ARGV;
>
>   my $unifdef = "scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__";
>
> @@ -27,8 +27,8 @@ foreach my $filename (@files) {
>
>   	my $tmpfile = "$installdir/$file.tmp";
>
> -	open(my $in, '<', $filename)
> -	    or die "$filename: $!\n";
> +	open(my $in, '<', "$srcdir/$filename")
> +	    or die "$srcdir/$filename: $!\n";
>   	open(my $out, '>', $tmpfile)
>   	    or die "$tmpfile: $!\n";
>   	while (my $line =<$in>) {

Re: [PATCH] fix make headers_install when path is too long

[Sam Ravnborg]

On Fri, Apr 26, 2013 at 06:36:38PM +0200, Nicolas Dichtel wrote:
> If headers_install is executed from a deep/long directory structure, the
> shell's maximum argument length can be execeeded, which breaks the operation
> with:
> 
> | make[2]: execvp: /bin/sh: Argument list too long
> | make[2]: ***
> 
> Instead of passing each files name with the entire path, I give only the file
> name without the source path and give this path as a new argument to
> headers_install.pl.
> 
> Because there is three possible path, I have tree input-files list, one per
> path.
> 
> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
> ---
> 
> I come back to this issue. Here is another proposal to fix this pb.
> Comments are welcome.
> 
>  scripts/Makefile.headersinst | 20 ++++++++++++++------
>  scripts/headers_install.pl   |  8 ++++----

In the kbuild tree the perl script has been replaced by a shell script.
You should base you patch on the kbuild-tree (or -next).

	Sam

Re: [PATCH] fix make headers_install when path is too long

[Michal Marek]

On 26.4.2013 20:57, Sam Ravnborg wrote:
> On Fri, Apr 26, 2013 at 06:36:38PM +0200, Nicolas Dichtel wrote:
>> If headers_install is executed from a deep/long directory structure, the
>> shell's maximum argument length can be execeeded, which breaks the operation
>> with:
>>
>> | make[2]: execvp: /bin/sh: Argument list too long
>> | make[2]: ***
>>
>> Instead of passing each files name with the entire path, I give only the file
>> name without the source path and give this path as a new argument to
>> headers_install.pl.
>>
>> Because there is three possible path, I have tree input-files list, one per
>> path.
>>
>> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
>> ---
>>
>> I come back to this issue. Here is another proposal to fix this pb.
>> Comments are welcome.
>>
>>  scripts/Makefile.headersinst | 20 ++++++++++++++------
>>  scripts/headers_install.pl   |  8 ++++----
> 
> In the kbuild tree the perl script has been replaced by a shell script.
> You should base you patch on the kbuild-tree (or -next).

Yes. Otherwise, the patch looks good. At least until the number of
userspace headers explodes :-).

Michal

[PATCH linux-next v2] fix make headers_install when path is too long

[Nicolas Dichtel]

If headers_install is executed from a deep/long directory structure, the
shell's maximum argument length can be execeeded, which breaks the operation
with:

| make[2]: execvp: /bin/sh: Argument list too long
| make[2]: ***

Instead of passing each files name with the entire path, I give only the file
name without the source path and give this path as a new argument to
headers_install.pl.

Because there is three possible paths, I have tree input-files list, one per
path.

Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
---

v2: rebase patch on linux-next

 scripts/Makefile.headersinst | 20 ++++++++++++++------
 scripts/headers_install.sh   |  7 +++++--
 2 files changed, 19 insertions(+), 8 deletions(-)

diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst
index 182084d..8ccf830 100644
--- a/scripts/Makefile.headersinst
+++ b/scripts/Makefile.headersinst
@@ -47,18 +47,24 @@ header-y      := $(filter-out $(generic-y), $(header-y))
 all-files     := $(header-y) $(genhdr-y) $(wrapper-files)
 output-files  := $(addprefix $(installdir)/, $(all-files))
 
-input-files   := $(foreach hdr, $(header-y), \
+input-files1  := $(foreach hdr, $(header-y), \
 		   $(if $(wildcard $(srcdir)/$(hdr)), \
-			$(wildcard $(srcdir)/$(hdr)), \
+			$(wildcard $(srcdir)/$(hdr))) \
+		   )
+input-files1-name := $(notdir $(input-files1))
+input-files2  := $(foreach hdr, $(header-y), \
+		   $(if  $(wildcard $(srcdir)/$(hdr)),, \
 			$(if $(wildcard $(oldsrcdir)/$(hdr)), \
 				$(wildcard $(oldsrcdir)/$(hdr)), \
 				$(error Missing UAPI file $(srcdir)/$(hdr))) \
-		   )) \
-		 $(foreach hdr, $(genhdr-y), \
+		   ))
+input-files2-name := $(notdir $(input-files2))
+input-files3  := $(foreach hdr, $(genhdr-y), \
 		   $(if	$(wildcard $(gendir)/$(hdr)), \
 			$(wildcard $(gendir)/$(hdr)), \
 			$(error Missing generated UAPI file $(gendir)/$(hdr)) \
 		   ))
+input-files3-name := $(notdir $(input-files3))
 
 # Work out what needs to be removed
 oldheaders    := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h))
@@ -72,7 +78,9 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@))
 quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
                             file$(if $(word 2, $(all-files)),s))
       cmd_install = \
-        $(CONFIG_SHELL) $< $(installdir) $(input-files); \
+        $(CONFIG_SHELL) $< $(installdir) $(srcdir) $(input-files1-name); \
+        $(CONFIG_SHELL) $< $(installdir) $(oldsrcdir) $(input-files2-name); \
+        $(CONFIG_SHELL) $< $(installdir) $(gendir) $(input-files3-name); \
         for F in $(wrapper-files); do                                   \
                 echo "\#include <asm-generic/$$F>" > $(installdir)/$$F;    \
         done;                                                           \
@@ -98,7 +106,7 @@ __headersinst: $(subdirs) $(install-file)
 	@:
 
 targets += $(install-file)
-$(install-file): scripts/headers_install.sh $(input-files) FORCE
+$(install-file): scripts/headers_install.sh $(input-files1) $(input-files2) $(input-files3) FORCE
 	$(if $(unwanted),$(call cmd,remove),)
 	$(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@)))
 	$(call if_changed,install)
diff --git a/scripts/headers_install.sh b/scripts/headers_install.sh
index 643764f..5de5660 100644
--- a/scripts/headers_install.sh
+++ b/scripts/headers_install.sh
@@ -2,7 +2,7 @@
 
 if [ $# -lt 1 ]
 then
-	echo "Usage: headers_install.sh OUTDIR [FILES...]
+	echo "Usage: headers_install.sh OUTDIR SRCDIR [FILES...]
 	echo
 	echo "Prepares kernel header files for use by user space, by removing"
 	echo "all compiler.h definitions and #includes, removing any"
@@ -10,6 +10,7 @@ then
 	echo "asm/inline/volatile keywords."
 	echo
 	echo "OUTDIR: directory to write each userspace header FILE to."
+	echo "SRCDIR: source directory where files are picked."
 	echo "FILES:  list of header files to operate on."
 
 	exit 1
@@ -19,6 +20,8 @@ fi
 
 OUTDIR="$1"
 shift
+SRCDIR="$1"
+shift
 
 # Iterate through files listed on command line
 
@@ -34,7 +37,7 @@ do
 		-e 's/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g' \
 		-e 's/(^|[ \t(])(inline|asm|volatile)([ \t(]|$)/\1__\2__\3/g' \
 		-e 's@#(ifndef|define|endif[ \t]*/[*])[ \t]*_UAPI@#\1 @' \
-		"$i" > "$OUTDIR/$FILE.sed" || exit 1
+		"$SRCDIR/$i" > "$OUTDIR/$FILE.sed" || exit 1
 	scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__ "$OUTDIR/$FILE.sed" \
 		> "$OUTDIR/$FILE"
 	[ $? -gt 1 ] && exit 1
-- 
1.8.2.1

Re: [PATCH] fix make headers_install when path is too long

[Nicolas Dichtel]

Le 29/04/2013 14:13, Michal Marek a écrit :
> On 26.4.2013 20:57, Sam Ravnborg wrote:
>> On Fri, Apr 26, 2013 at 06:36:38PM +0200, Nicolas Dichtel wrote:
>>> If headers_install is executed from a deep/long directory structure, the
>>> shell's maximum argument length can be execeeded, which breaks the operation
>>> with:
>>>
>>> | make[2]: execvp: /bin/sh: Argument list too long
>>> | make[2]: ***
>>>
>>> Instead of passing each files name with the entire path, I give only the file
>>> name without the source path and give this path as a new argument to
>>> headers_install.pl.
>>>
>>> Because there is three possible path, I have tree input-files list, one per
>>> path.
>>>
>>> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
>>> ---
>>>
>>> I come back to this issue. Here is another proposal to fix this pb.
>>> Comments are welcome.
>>>
>>>   scripts/Makefile.headersinst | 20 ++++++++++++++------
>>>   scripts/headers_install.pl   |  8 ++++----
>>
>> In the kbuild tree the perl script has been replaced by a shell script.
>> You should base you patch on the kbuild-tree (or -next).
>
> Yes. Otherwise, the patch looks good. At least until the number of
> userspace headers explodes :-).
It should be ok for the next 2 or 3 versions ;-)

Re: [PATCH linux-next v2] fix make headers_install when path is too long

[Bruce Ashfield]

On 13-04-29 08:15 AM, Nicolas Dichtel wrote:
> If headers_install is executed from a deep/long directory structure, the
> shell's maximum argument length can be execeeded, which breaks the operation
> with:
>
> | make[2]: execvp: /bin/sh: Argument list too long
> | make[2]: ***
>
> Instead of passing each files name with the entire path, I give only the file
> name without the source path and give this path as a new argument to
> headers_install.pl.
>
> Because there is three possible paths, I have tree input-files list, one per
> path.

This version still works for my test case.

Tested-by: Bruce Ashfield <bruce.ashfield@windriver.com>

>
> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
> ---
>
> v2: rebase patch on linux-next
>
>   scripts/Makefile.headersinst | 20 ++++++++++++++------
>   scripts/headers_install.sh   |  7 +++++--
>   2 files changed, 19 insertions(+), 8 deletions(-)
>
> diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst
> index 182084d..8ccf830 100644
> --- a/scripts/Makefile.headersinst
> +++ b/scripts/Makefile.headersinst
> @@ -47,18 +47,24 @@ header-y      := $(filter-out $(generic-y), $(header-y))
>   all-files     := $(header-y) $(genhdr-y) $(wrapper-files)
>   output-files  := $(addprefix $(installdir)/, $(all-files))
>
> -input-files   := $(foreach hdr, $(header-y), \
> +input-files1  := $(foreach hdr, $(header-y), \
>   		   $(if $(wildcard $(srcdir)/$(hdr)), \
> -			$(wildcard $(srcdir)/$(hdr)), \
> +			$(wildcard $(srcdir)/$(hdr))) \
> +		   )
> +input-files1-name := $(notdir $(input-files1))
> +input-files2  := $(foreach hdr, $(header-y), \
> +		   $(if  $(wildcard $(srcdir)/$(hdr)),, \
>   			$(if $(wildcard $(oldsrcdir)/$(hdr)), \
>   				$(wildcard $(oldsrcdir)/$(hdr)), \
>   				$(error Missing UAPI file $(srcdir)/$(hdr))) \
> -		   )) \
> -		 $(foreach hdr, $(genhdr-y), \
> +		   ))
> +input-files2-name := $(notdir $(input-files2))
> +input-files3  := $(foreach hdr, $(genhdr-y), \
>   		   $(if	$(wildcard $(gendir)/$(hdr)), \
>   			$(wildcard $(gendir)/$(hdr)), \
>   			$(error Missing generated UAPI file $(gendir)/$(hdr)) \
>   		   ))
> +input-files3-name := $(notdir $(input-files3))
>
>   # Work out what needs to be removed
>   oldheaders    := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h))
> @@ -72,7 +78,9 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@))
>   quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
>                               file$(if $(word 2, $(all-files)),s))
>         cmd_install = \
> -        $(CONFIG_SHELL) $< $(installdir) $(input-files); \
> +        $(CONFIG_SHELL) $< $(installdir) $(srcdir) $(input-files1-name); \
> +        $(CONFIG_SHELL) $< $(installdir) $(oldsrcdir) $(input-files2-name); \
> +        $(CONFIG_SHELL) $< $(installdir) $(gendir) $(input-files3-name); \
>           for F in $(wrapper-files); do                                   \
>                   echo "\#include <asm-generic/$$F>" > $(installdir)/$$F;    \
>           done;                                                           \
> @@ -98,7 +106,7 @@ __headersinst: $(subdirs) $(install-file)
>   	@:
>
>   targets += $(install-file)
> -$(install-file): scripts/headers_install.sh $(input-files) FORCE
> +$(install-file): scripts/headers_install.sh $(input-files1) $(input-files2) $(input-files3) FORCE
>   	$(if $(unwanted),$(call cmd,remove),)
>   	$(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@)))
>   	$(call if_changed,install)
> diff --git a/scripts/headers_install.sh b/scripts/headers_install.sh
> index 643764f..5de5660 100644
> --- a/scripts/headers_install.sh
> +++ b/scripts/headers_install.sh
> @@ -2,7 +2,7 @@
>
>   if [ $# -lt 1 ]
>   then
> -	echo "Usage: headers_install.sh OUTDIR [FILES...]
> +	echo "Usage: headers_install.sh OUTDIR SRCDIR [FILES...]
>   	echo
>   	echo "Prepares kernel header files for use by user space, by removing"
>   	echo "all compiler.h definitions and #includes, removing any"
> @@ -10,6 +10,7 @@ then
>   	echo "asm/inline/volatile keywords."
>   	echo
>   	echo "OUTDIR: directory to write each userspace header FILE to."
> +	echo "SRCDIR: source directory where files are picked."
>   	echo "FILES:  list of header files to operate on."
>
>   	exit 1
> @@ -19,6 +20,8 @@ fi
>
>   OUTDIR="$1"
>   shift
> +SRCDIR="$1"
> +shift
>
>   # Iterate through files listed on command line
>
> @@ -34,7 +37,7 @@ do
>   		-e 's/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g' \
>   		-e 's/(^|[ \t(])(inline|asm|volatile)([ \t(]|$)/\1__\2__\3/g' \
>   		-e 's@#(ifndef|define|endif[ \t]*/[*])[ \t]*_UAPI@#\1 @' \
> -		"$i" > "$OUTDIR/$FILE.sed" || exit 1
> +		"$SRCDIR/$i" > "$OUTDIR/$FILE.sed" || exit 1
>   	scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__ "$OUTDIR/$FILE.sed" \
>   		> "$OUTDIR/$FILE"
>   	[ $? -gt 1 ] && exit 1
>

Re: [PATCH linux-next v2] fix make headers_install when path is too long

[Nicolas Dichtel]

Le 29/04/2013 14:34, Bruce Ashfield a écrit :
> On 13-04-29 08:15 AM, Nicolas Dichtel wrote:
>> If headers_install is executed from a deep/long directory structure, the
>> shell's maximum argument length can be execeeded, which breaks the operation
>> with:
>>
>> | make[2]: execvp: /bin/sh: Argument list too long
>> | make[2]: ***
>>
>> Instead of passing each files name with the entire path, I give only the file
>> name without the source path and give this path as a new argument to
>> headers_install.pl.
>>
>> Because there is three possible paths, I have tree input-files list, one per
>> path.
>
> This version still works for my test case.
>
> Tested-by: Bruce Ashfield <bruce.ashfield@windriver.com>
>
>>
>> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
>> ---
Should I do something more for this patch? It seems not included.

Re: [PATCH linux-next v2] fix make headers_install when path is too long

[Nicolas Dichtel]

Le 06/05/2013 10:19, Nicolas Dichtel a écrit :
> Le 29/04/2013 14:34, Bruce Ashfield a écrit :
>> On 13-04-29 08:15 AM, Nicolas Dichtel wrote:
>>> If headers_install is executed from a deep/long directory structure, the
>>> shell's maximum argument length can be execeeded, which breaks the operation
>>> with:
>>>
>>> | make[2]: execvp: /bin/sh: Argument list too long
>>> | make[2]: ***
>>>
>>> Instead of passing each files name with the entire path, I give only the file
>>> name without the source path and give this path as a new argument to
>>> headers_install.pl.
>>>
>>> Because there is three possible paths, I have tree input-files list, one per
>>> path.
>>
>> This version still works for my test case.
>>
>> Tested-by: Bruce Ashfield <bruce.ashfield@windriver.com>
>>
>>>
>>> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
>>> ---
> Should I do something more for this patch? It seems not included.
ping

Re: [PATCH linux-next v2] fix make headers_install when path is too long

[Michal Marek]

On Fri, May 17, 2013 at 11:12:30AM +0200, Nicolas Dichtel wrote:
> Le 06/05/2013 10:19, Nicolas Dichtel a écrit :
>> Le 29/04/2013 14:34, Bruce Ashfield a écrit :
>>> On 13-04-29 08:15 AM, Nicolas Dichtel wrote:
>>>> If headers_install is executed from a deep/long directory structure, the
>>>> shell's maximum argument length can be execeeded, which breaks the operation
>>>> with:
>>>>
>>>> | make[2]: execvp: /bin/sh: Argument list too long
>>>> | make[2]: ***
>>>>
>>>> Instead of passing each files name with the entire path, I give only the file
>>>> name without the source path and give this path as a new argument to
>>>> headers_install.pl.
>>>>
>>>> Because there is three possible paths, I have tree input-files list, one per
>>>> path.
>>>
>>> This version still works for my test case.
>>>
>>> Tested-by: Bruce Ashfield <bruce.ashfield@windriver.com>
>>>
>>>>
>>>> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
>>>> ---
>> Should I do something more for this patch? It seems not included.
> ping

I applied the patch to kbuild.git#kbuild now.

Michal