* make headers_install fail when path is too long @ 2013-03-06 11:16 Nicolas Dichtel 2013-03-06 16:10 ` Sam Ravnborg 0 siblings, 1 reply; 17+ messages in thread From: Nicolas Dichtel @ 2013-03-06 11:16 UTC (permalink / raw) To: Michal Marek, linux-kbuild This problem has probably already been reported, but I don't find the fix in a kernel.org tree and I don't understand why. Here is the problem: root@schnaps:/home/rbb/averyveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryveryverylongpath/linux(master)>make headers_install INSTALL_HDR_PATH=/tmp/tmp/ CHK include/generated/uapi/linux/version.h make[1]: Nothing to be done for `all'. make[1]: Nothing to be done for `relocs'. make[2]: execvp: /bin/sh: Argument list too long make[2]: *** [/tmp/tmp//include/linux/.install] Error 127 make[1]: *** [linux] Error 2 make: *** [headers_install] Error 2 I see that this problem has been reported several times on the web, but I do not find why it's not fixed upstream. Some patches are available: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=53cc748b93e8af584557d6db5309c3e955182c5c Any comment/suggestion is welcome. Regards, Nicolas ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: make headers_install fail when path is too long 2013-03-06 11:16 make headers_install fail when path is too long Nicolas Dichtel @ 2013-03-06 16:10 ` Sam Ravnborg 2013-03-06 16:24 ` Michal Marek 2013-03-06 16:27 ` Nicolas Dichtel 0 siblings, 2 replies; 17+ messages in thread From: Sam Ravnborg @ 2013-03-06 16:10 UTC (permalink / raw) To: Nicolas Dichtel; +Cc: Michal Marek, linux-kbuild On Wed, Mar 06, 2013 at 12:16:33PM +0100, Nicolas Dichtel wrote: > This problem has probably already been reported, but I don't find the fix > in a kernel.org tree and I don't understand why. First time I see this bug described - but I may missed it here at kbuild. If a patch exists then please include the patch in your mail so we can see it and comment on it. And make sure to have the attribution correct if you consider it to be applied. Thanks, Sam ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: make headers_install fail when path is too long 2013-03-06 16:10 ` Sam Ravnborg @ 2013-03-06 16:24 ` Michal Marek 2013-03-06 16:27 ` Nicolas Dichtel 1 sibling, 0 replies; 17+ messages in thread From: Michal Marek @ 2013-03-06 16:24 UTC (permalink / raw) To: Nicolas Dichtel; +Cc: Sam Ravnborg, linux-kbuild Dne 6.3.2013 17:10, Sam Ravnborg napsal(a): > On Wed, Mar 06, 2013 at 12:16:33PM +0100, Nicolas Dichtel wrote: >> This problem has probably already been reported, but I don't find the fix >> in a kernel.org tree and I don't understand why. > > First time I see this bug described - but I may missed it here at kbuild. > If a patch exists then please include the patch in your mail so we can > see it and comment on it. Yes. If it's an obvious bug fix like this, simply send it to lkml and linux-kbuild. Michal ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: make headers_install fail when path is too long 2013-03-06 16:10 ` Sam Ravnborg 2013-03-06 16:24 ` Michal Marek @ 2013-03-06 16:27 ` Nicolas Dichtel 2013-03-06 16:50 ` Bruce Ashfield 2013-03-07 8:26 ` Michal Marek 1 sibling, 2 replies; 17+ messages in thread From: Nicolas Dichtel @ 2013-03-06 16:27 UTC (permalink / raw) To: Sam Ravnborg; +Cc: Michal Marek, linux-kbuild, Bruce Ashfield Le 06/03/2013 17:10, Sam Ravnborg a écrit : > On Wed, Mar 06, 2013 at 12:16:33PM +0100, Nicolas Dichtel wrote: >> This problem has probably already been reported, but I don't find the fix >> in a kernel.org tree and I don't understand why. > > First time I see this bug described - but I may missed it here at kbuild. It's probably why I didn't find this on a linux kernel mailing list ;-) > If a patch exists then please include the patch in your mail so we can > see it and comment on it. > > And make sure to have the attribution correct if you consider it to > be applied. I'm not the author of the patch, I've found it on the web. Here it is: From cb9c811b8f23277de95dc687e87c6859308e68e6 Mon Sep 17 00:00:00 2001 From: Bruce Ashfield <bruce.ashfield@windriver.com> Date: Wed, 6 Mar 2013 17:23:51 +0100 Subject: [PATCH] scripts/Makefile.headersinst: install headers from scratch file If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation with: | make[2]: execvp: /bin/sh: Argument list too long | make[2]: *** By dumping the input files to a scratch file and using xargs to read the input list from the scratch file, we can avoid blowing out the maximum argument size and install headers in a long path name environment. Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com> --- scripts/Makefile.headersinst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst index 25f216a..c481986 100644 --- a/scripts/Makefile.headersinst +++ b/scripts/Makefile.headersinst @@ -71,7 +71,7 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@)) quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\ file$(if $(word 2, $(all-files)),s)) cmd_install = \ - $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \ + xargs $(PERL) $< $(installdir) $(SRCARCH) < $(INSTALL_HDR_PATH)/.input-files; \ for F in $(wrapper-files); do \ echo "\#include <asm-generic/$$F>" > $(installdir)/$$F; \ done; \ @@ -100,7 +100,9 @@ targets += $(install-file) $(install-file): scripts/headers_install.pl $(input-files) FORCE $(if $(unwanted),$(call cmd,remove),) $(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@))) + @echo $(input-files) > $(INSTALL_HDR_PATH)/.input-files $(call if_changed,install) + @rm $(INSTALL_HDR_PATH)/.input-files else __headerscheck: $(subdirs) $(check-file) -- 1.8.0.1 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: make headers_install fail when path is too long 2013-03-06 16:27 ` Nicolas Dichtel @ 2013-03-06 16:50 ` Bruce Ashfield 2013-03-07 8:26 ` Michal Marek 1 sibling, 0 replies; 17+ messages in thread From: Bruce Ashfield @ 2013-03-06 16:50 UTC (permalink / raw) To: nicolas.dichtel; +Cc: Sam Ravnborg, Michal Marek, linux-kbuild On 13-03-06 11:27 AM, Nicolas Dichtel wrote: > Le 06/03/2013 17:10, Sam Ravnborg a écrit : >> On Wed, Mar 06, 2013 at 12:16:33PM +0100, Nicolas Dichtel wrote: >>> This problem has probably already been reported, but I don't find the >>> fix >>> in a kernel.org tree and I don't understand why. >> >> First time I see this bug described - but I may missed it here at kbuild. > It's probably why I didn't find this on a linux kernel mailing list ;-) And it's been on my TODO list to float this as a RFC to the kbuild lists before 3.9 arrived .. a goal that I managed to miss (again). The patch in question has us up and installing headers on Yocto kernel/ kernel header packages greater than 3.7, so it has received a decent amount of testing. As for it being the perfect approach, I only claim that it works in the environment that I've been using and it is functional. If this change turns out to be acceptable, that's great. I can also spend time re-working as appropriate .. or someone else's solution can be adopted. In the end it's a win and a fix, and the thread has prompted me to get the change floated :) Cheers, Bruce > >> If a patch exists then please include the patch in your mail so we can >> see it and comment on it. >> >> And make sure to have the attribution correct if you consider it to >> be applied. > I'm not the author of the patch, I've found it on the web. Here it is: > > From cb9c811b8f23277de95dc687e87c6859308e68e6 Mon Sep 17 00:00:00 2001 > From: Bruce Ashfield <bruce.ashfield@windriver.com> > Date: Wed, 6 Mar 2013 17:23:51 +0100 > Subject: [PATCH] scripts/Makefile.headersinst: install headers from scratch > file > > If headers_install is executed from a deep/long directory structure, the > shell's maximum argument length can be execeeded, which breaks the > operation > with: > > | make[2]: execvp: /bin/sh: Argument list too long > | make[2]: *** > > By dumping the input files to a scratch file and using xargs to read the > input list from the scratch file, we can avoid blowing out the maximum > argument size and install headers in a long path name environment. > > Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com> > --- > scripts/Makefile.headersinst | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst > index 25f216a..c481986 100644 > --- a/scripts/Makefile.headersinst > +++ b/scripts/Makefile.headersinst > @@ -71,7 +71,7 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@)) > quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\ > file$(if $(word 2, $(all-files)),s)) > cmd_install = \ > - $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \ > + xargs $(PERL) $< $(installdir) $(SRCARCH) < > $(INSTALL_HDR_PATH)/.input-files; \ > for F in $(wrapper-files); do \ > echo "\#include <asm-generic/$$F>" > > $(installdir)/$$F; \ > done; \ > @@ -100,7 +100,9 @@ targets += $(install-file) > $(install-file): scripts/headers_install.pl $(input-files) FORCE > $(if $(unwanted),$(call cmd,remove),) > $(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@))) > + @echo $(input-files) > $(INSTALL_HDR_PATH)/.input-files > $(call if_changed,install) > + @rm $(INSTALL_HDR_PATH)/.input-files > > else > __headerscheck: $(subdirs) $(check-file) ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: make headers_install fail when path is too long 2013-03-06 16:27 ` Nicolas Dichtel 2013-03-06 16:50 ` Bruce Ashfield @ 2013-03-07 8:26 ` Michal Marek 2013-03-07 13:28 ` Bruce Ashfield 1 sibling, 1 reply; 17+ messages in thread From: Michal Marek @ 2013-03-07 8:26 UTC (permalink / raw) To: Bruce Ashfield; +Cc: nicolas.dichtel, Sam Ravnborg, linux-kbuild Dne 6.3.2013 17:27, Nicolas Dichtel napsal(a): > - $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \ [...] > + @echo $(input-files) > $(INSTALL_HDR_PATH)/.input-files Are you sure this is a reliable fix? What make does is to spawn /bin/sh -c 'echo <long list of files> > usr/include/.input-files' here. So I guess that it works for you just because "sh -c" is shorter than "sh -c 'perl scripts/headers_install.pl...'". Michal ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: make headers_install fail when path is too long 2013-03-07 8:26 ` Michal Marek @ 2013-03-07 13:28 ` Bruce Ashfield 2013-04-26 16:36 ` [PATCH] fix make headers_install " Nicolas Dichtel 0 siblings, 1 reply; 17+ messages in thread From: Bruce Ashfield @ 2013-03-07 13:28 UTC (permalink / raw) To: Michal Marek; +Cc: nicolas.dichtel, Sam Ravnborg, linux-kbuild On 13-03-07 03:26 AM, Michal Marek wrote: > Dne 6.3.2013 17:27, Nicolas Dichtel napsal(a): >> - $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \ > [...] >> + @echo $(input-files) > $(INSTALL_HDR_PATH)/.input-files > > Are you sure this is a reliable fix? What make does is to spawn > > /bin/sh -c 'echo <long list of files> > usr/include/.input-files' It's what I was referencing in my original email. It works here, and fixes our install of headers in previously failing environments. But yes, it does shuffle the args enough to get around the limit, but there's still a way to have even deeper and longer path names that could cause failures. I experimented with loops, and other options as well. But any construct like "for f in $(input-files)", is both slow and explodes on the argument length limits just like the original. > > here. So I guess that it works for you just because "sh -c" is shorter > than "sh -c 'perl scripts/headers_install.pl...'". Partly, yes, but we are more than a few characters over the limit in my testing. So it shouldn't be the whole story. Cheers, Bruce > > Michal > ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH] fix make headers_install when path is too long 2013-03-07 13:28 ` Bruce Ashfield @ 2013-04-26 16:36 ` Nicolas Dichtel 2013-04-26 18:14 ` Bruce Ashfield 2013-04-26 18:57 ` Sam Ravnborg 0 siblings, 2 replies; 17+ messages in thread From: Nicolas Dichtel @ 2013-04-26 16:36 UTC (permalink / raw) To: bruce.ashfield; +Cc: mmarek, sam, linux-kbuild, Nicolas Dichtel If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation with: | make[2]: execvp: /bin/sh: Argument list too long | make[2]: *** Instead of passing each files name with the entire path, I give only the file name without the source path and give this path as a new argument to headers_install.pl. Because there is three possible path, I have tree input-files list, one per path. Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com> --- I come back to this issue. Here is another proposal to fix this pb. Comments are welcome. scripts/Makefile.headersinst | 20 ++++++++++++++------ scripts/headers_install.pl | 8 ++++---- 2 files changed, 18 insertions(+), 10 deletions(-) diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst index 477d137..fbadfc3 100644 --- a/scripts/Makefile.headersinst +++ b/scripts/Makefile.headersinst @@ -47,18 +47,24 @@ header-y := $(filter-out $(generic-y), $(header-y)) all-files := $(header-y) $(genhdr-y) $(wrapper-files) output-files := $(addprefix $(installdir)/, $(all-files)) -input-files := $(foreach hdr, $(header-y), \ +input-files1 := $(foreach hdr, $(header-y), \ $(if $(wildcard $(srcdir)/$(hdr)), \ - $(wildcard $(srcdir)/$(hdr)), \ + $(wildcard $(srcdir)/$(hdr))) \ + ) +input-files1-name := $(notdir $(input-files1)) +input-files2 := $(foreach hdr, $(header-y), \ + $(if $(wildcard $(srcdir)/$(hdr)),, \ $(if $(wildcard $(oldsrcdir)/$(hdr)), \ $(wildcard $(oldsrcdir)/$(hdr)), \ $(error Missing UAPI file $(srcdir)/$(hdr))) \ - )) \ - $(foreach hdr, $(genhdr-y), \ + )) +input-files2-name := $(notdir $(input-files2)) +input-files3 := $(foreach hdr, $(genhdr-y), \ $(if $(wildcard $(gendir)/$(hdr)), \ $(wildcard $(gendir)/$(hdr)), \ $(error Missing generated UAPI file $(gendir)/$(hdr)) \ )) +input-files3-name := $(notdir $(input-files3)) # Work out what needs to be removed oldheaders := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h)) @@ -72,7 +78,9 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@)) quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\ file$(if $(word 2, $(all-files)),s)) cmd_install = \ - $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \ + $(PERL) $< $(installdir) $(SRCARCH) $(srcdir) $(input-files1-name); \ + $(PERL) $< $(installdir) $(SRCARCH) $(oldsrcdir) $(input-files2-name); \ + $(PERL) $< $(installdir) $(SRCARCH) $(gendir) $(input-files3-name); \ for F in $(wrapper-files); do \ echo "\#include <asm-generic/$$F>" > $(installdir)/$$F; \ done; \ @@ -98,7 +106,7 @@ __headersinst: $(subdirs) $(install-file) @: targets += $(install-file) -$(install-file): scripts/headers_install.pl $(input-files) FORCE +$(install-file): scripts/headers_install.pl $(input-files1) $(input-files2) $(input-files3) FORCE $(if $(unwanted),$(call cmd,remove),) $(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@))) $(call if_changed,install) diff --git a/scripts/headers_install.pl b/scripts/headers_install.pl index 581ca99..87d9700 100644 --- a/scripts/headers_install.pl +++ b/scripts/headers_install.pl @@ -3,7 +3,7 @@ # headers_install prepare the listed header files for use in # user space and copy the files to their destination. # -# Usage: headers_install.pl readdir installdir arch [files...] +# Usage: headers_install.pl readdir installdir arch srcdir [files...] # installdir: dir to install the files to # arch: current architecture # arch is used to force a reinstallation when the arch @@ -17,7 +17,7 @@ use strict; -my ($installdir, $arch, @files) = @ARGV; +my ($installdir, $arch, $srcdir, @files) = @ARGV; my $unifdef = "scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__"; @@ -27,8 +27,8 @@ foreach my $filename (@files) { my $tmpfile = "$installdir/$file.tmp"; - open(my $in, '<', $filename) - or die "$filename: $!\n"; + open(my $in, '<', "$srcdir/$filename") + or die "$srcdir/$filename: $!\n"; open(my $out, '>', $tmpfile) or die "$tmpfile: $!\n"; while (my $line = <$in>) { -- 1.8.2.1 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH] fix make headers_install when path is too long 2013-04-26 16:36 ` [PATCH] fix make headers_install " Nicolas Dichtel @ 2013-04-26 18:14 ` Bruce Ashfield 2013-04-26 18:57 ` Sam Ravnborg 1 sibling, 0 replies; 17+ messages in thread From: Bruce Ashfield @ 2013-04-26 18:14 UTC (permalink / raw) To: Nicolas Dichtel; +Cc: mmarek, sam, linux-kbuild On 13-04-26 12:36 PM, Nicolas Dichtel wrote: > If headers_install is executed from a deep/long directory structure, the > shell's maximum argument length can be execeeded, which breaks the operation > with: > > | make[2]: execvp: /bin/sh: Argument list too long > | make[2]: *** > > Instead of passing each files name with the entire path, I give only the file > name without the source path and give this path as a new argument to > headers_install.pl. This will work. I considered the same option when I solved the similar problem for Yocto installs, but managed to avoid it with a shorter term fixed. > > Because there is three possible path, I have tree input-files list, one per > path. > > Signed-off-by: Nicolas Dichtel<nicolas.dichtel@6wind.com> Tested-by: Bruce Ashfield <bruce.ashfield@windriver.com> Bruce > --- > > I come back to this issue. Here is another proposal to fix this pb. > Comments are welcome. > > scripts/Makefile.headersinst | 20 ++++++++++++++------ > scripts/headers_install.pl | 8 ++++---- > 2 files changed, 18 insertions(+), 10 deletions(-) > > diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst > index 477d137..fbadfc3 100644 > --- a/scripts/Makefile.headersinst > +++ b/scripts/Makefile.headersinst > @@ -47,18 +47,24 @@ header-y := $(filter-out $(generic-y), $(header-y)) > all-files := $(header-y) $(genhdr-y) $(wrapper-files) > output-files := $(addprefix $(installdir)/, $(all-files)) > > -input-files := $(foreach hdr, $(header-y), \ > +input-files1 := $(foreach hdr, $(header-y), \ > $(if $(wildcard $(srcdir)/$(hdr)), \ > - $(wildcard $(srcdir)/$(hdr)), \ > + $(wildcard $(srcdir)/$(hdr))) \ > + ) > +input-files1-name := $(notdir $(input-files1)) > +input-files2 := $(foreach hdr, $(header-y), \ > + $(if $(wildcard $(srcdir)/$(hdr)),, \ > $(if $(wildcard $(oldsrcdir)/$(hdr)), \ > $(wildcard $(oldsrcdir)/$(hdr)), \ > $(error Missing UAPI file $(srcdir)/$(hdr))) \ > - )) \ > - $(foreach hdr, $(genhdr-y), \ > + )) > +input-files2-name := $(notdir $(input-files2)) > +input-files3 := $(foreach hdr, $(genhdr-y), \ > $(if $(wildcard $(gendir)/$(hdr)), \ > $(wildcard $(gendir)/$(hdr)), \ > $(error Missing generated UAPI file $(gendir)/$(hdr)) \ > )) > +input-files3-name := $(notdir $(input-files3)) > > # Work out what needs to be removed > oldheaders := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h)) > @@ -72,7 +78,9 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@)) > quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\ > file$(if $(word 2, $(all-files)),s)) > cmd_install = \ > - $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \ > + $(PERL) $< $(installdir) $(SRCARCH) $(srcdir) $(input-files1-name); \ > + $(PERL) $< $(installdir) $(SRCARCH) $(oldsrcdir) $(input-files2-name); \ > + $(PERL) $< $(installdir) $(SRCARCH) $(gendir) $(input-files3-name); \ > for F in $(wrapper-files); do \ > echo "\#include<asm-generic/$$F>"> $(installdir)/$$F; \ > done; \ > @@ -98,7 +106,7 @@ __headersinst: $(subdirs) $(install-file) > @: > > targets += $(install-file) > -$(install-file): scripts/headers_install.pl $(input-files) FORCE > +$(install-file): scripts/headers_install.pl $(input-files1) $(input-files2) $(input-files3) FORCE > $(if $(unwanted),$(call cmd,remove),) > $(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@))) > $(call if_changed,install) > diff --git a/scripts/headers_install.pl b/scripts/headers_install.pl > index 581ca99..87d9700 100644 > --- a/scripts/headers_install.pl > +++ b/scripts/headers_install.pl > @@ -3,7 +3,7 @@ > # headers_install prepare the listed header files for use in > # user space and copy the files to their destination. > # > -# Usage: headers_install.pl readdir installdir arch [files...] > +# Usage: headers_install.pl readdir installdir arch srcdir [files...] > # installdir: dir to install the files to > # arch: current architecture > # arch is used to force a reinstallation when the arch > @@ -17,7 +17,7 @@ > > use strict; > > -my ($installdir, $arch, @files) = @ARGV; > +my ($installdir, $arch, $srcdir, @files) = @ARGV; > > my $unifdef = "scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__"; > > @@ -27,8 +27,8 @@ foreach my $filename (@files) { > > my $tmpfile = "$installdir/$file.tmp"; > > - open(my $in, '<', $filename) > - or die "$filename: $!\n"; > + open(my $in, '<', "$srcdir/$filename") > + or die "$srcdir/$filename: $!\n"; > open(my $out, '>', $tmpfile) > or die "$tmpfile: $!\n"; > while (my $line =<$in>) { ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH] fix make headers_install when path is too long 2013-04-26 16:36 ` [PATCH] fix make headers_install " Nicolas Dichtel 2013-04-26 18:14 ` Bruce Ashfield @ 2013-04-26 18:57 ` Sam Ravnborg 2013-04-29 12:13 ` Michal Marek 2013-04-29 12:15 ` [PATCH linux-next v2] " Nicolas Dichtel 1 sibling, 2 replies; 17+ messages in thread From: Sam Ravnborg @ 2013-04-26 18:57 UTC (permalink / raw) To: Nicolas Dichtel; +Cc: bruce.ashfield, mmarek, linux-kbuild On Fri, Apr 26, 2013 at 06:36:38PM +0200, Nicolas Dichtel wrote: > If headers_install is executed from a deep/long directory structure, the > shell's maximum argument length can be execeeded, which breaks the operation > with: > > | make[2]: execvp: /bin/sh: Argument list too long > | make[2]: *** > > Instead of passing each files name with the entire path, I give only the file > name without the source path and give this path as a new argument to > headers_install.pl. > > Because there is three possible path, I have tree input-files list, one per > path. > > Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com> > --- > > I come back to this issue. Here is another proposal to fix this pb. > Comments are welcome. > > scripts/Makefile.headersinst | 20 ++++++++++++++------ > scripts/headers_install.pl | 8 ++++---- In the kbuild tree the perl script has been replaced by a shell script. You should base you patch on the kbuild-tree (or -next). Sam ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH] fix make headers_install when path is too long 2013-04-26 18:57 ` Sam Ravnborg @ 2013-04-29 12:13 ` Michal Marek 2013-04-29 12:17 ` Nicolas Dichtel 2013-04-29 12:15 ` [PATCH linux-next v2] " Nicolas Dichtel 1 sibling, 1 reply; 17+ messages in thread From: Michal Marek @ 2013-04-29 12:13 UTC (permalink / raw) To: Nicolas Dichtel; +Cc: Sam Ravnborg, bruce.ashfield, linux-kbuild On 26.4.2013 20:57, Sam Ravnborg wrote: > On Fri, Apr 26, 2013 at 06:36:38PM +0200, Nicolas Dichtel wrote: >> If headers_install is executed from a deep/long directory structure, the >> shell's maximum argument length can be execeeded, which breaks the operation >> with: >> >> | make[2]: execvp: /bin/sh: Argument list too long >> | make[2]: *** >> >> Instead of passing each files name with the entire path, I give only the file >> name without the source path and give this path as a new argument to >> headers_install.pl. >> >> Because there is three possible path, I have tree input-files list, one per >> path. >> >> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com> >> --- >> >> I come back to this issue. Here is another proposal to fix this pb. >> Comments are welcome. >> >> scripts/Makefile.headersinst | 20 ++++++++++++++------ >> scripts/headers_install.pl | 8 ++++---- > > In the kbuild tree the perl script has been replaced by a shell script. > You should base you patch on the kbuild-tree (or -next). Yes. Otherwise, the patch looks good. At least until the number of userspace headers explodes :-). Michal ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH] fix make headers_install when path is too long 2013-04-29 12:13 ` Michal Marek @ 2013-04-29 12:17 ` Nicolas Dichtel 0 siblings, 0 replies; 17+ messages in thread From: Nicolas Dichtel @ 2013-04-29 12:17 UTC (permalink / raw) To: Michal Marek; +Cc: Sam Ravnborg, bruce.ashfield, linux-kbuild Le 29/04/2013 14:13, Michal Marek a écrit : > On 26.4.2013 20:57, Sam Ravnborg wrote: >> On Fri, Apr 26, 2013 at 06:36:38PM +0200, Nicolas Dichtel wrote: >>> If headers_install is executed from a deep/long directory structure, the >>> shell's maximum argument length can be execeeded, which breaks the operation >>> with: >>> >>> | make[2]: execvp: /bin/sh: Argument list too long >>> | make[2]: *** >>> >>> Instead of passing each files name with the entire path, I give only the file >>> name without the source path and give this path as a new argument to >>> headers_install.pl. >>> >>> Because there is three possible path, I have tree input-files list, one per >>> path. >>> >>> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com> >>> --- >>> >>> I come back to this issue. Here is another proposal to fix this pb. >>> Comments are welcome. >>> >>> scripts/Makefile.headersinst | 20 ++++++++++++++------ >>> scripts/headers_install.pl | 8 ++++---- >> >> In the kbuild tree the perl script has been replaced by a shell script. >> You should base you patch on the kbuild-tree (or -next). > > Yes. Otherwise, the patch looks good. At least until the number of > userspace headers explodes :-). It should be ok for the next 2 or 3 versions ;-) ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH linux-next v2] fix make headers_install when path is too long 2013-04-26 18:57 ` Sam Ravnborg 2013-04-29 12:13 ` Michal Marek @ 2013-04-29 12:15 ` Nicolas Dichtel 2013-04-29 12:34 ` Bruce Ashfield 1 sibling, 1 reply; 17+ messages in thread From: Nicolas Dichtel @ 2013-04-29 12:15 UTC (permalink / raw) To: sam; +Cc: bruce.ashfield, mmarek, linux-kbuild, Nicolas Dichtel If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation with: | make[2]: execvp: /bin/sh: Argument list too long | make[2]: *** Instead of passing each files name with the entire path, I give only the file name without the source path and give this path as a new argument to headers_install.pl. Because there is three possible paths, I have tree input-files list, one per path. Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com> --- v2: rebase patch on linux-next scripts/Makefile.headersinst | 20 ++++++++++++++------ scripts/headers_install.sh | 7 +++++-- 2 files changed, 19 insertions(+), 8 deletions(-) diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst index 182084d..8ccf830 100644 --- a/scripts/Makefile.headersinst +++ b/scripts/Makefile.headersinst @@ -47,18 +47,24 @@ header-y := $(filter-out $(generic-y), $(header-y)) all-files := $(header-y) $(genhdr-y) $(wrapper-files) output-files := $(addprefix $(installdir)/, $(all-files)) -input-files := $(foreach hdr, $(header-y), \ +input-files1 := $(foreach hdr, $(header-y), \ $(if $(wildcard $(srcdir)/$(hdr)), \ - $(wildcard $(srcdir)/$(hdr)), \ + $(wildcard $(srcdir)/$(hdr))) \ + ) +input-files1-name := $(notdir $(input-files1)) +input-files2 := $(foreach hdr, $(header-y), \ + $(if $(wildcard $(srcdir)/$(hdr)),, \ $(if $(wildcard $(oldsrcdir)/$(hdr)), \ $(wildcard $(oldsrcdir)/$(hdr)), \ $(error Missing UAPI file $(srcdir)/$(hdr))) \ - )) \ - $(foreach hdr, $(genhdr-y), \ + )) +input-files2-name := $(notdir $(input-files2)) +input-files3 := $(foreach hdr, $(genhdr-y), \ $(if $(wildcard $(gendir)/$(hdr)), \ $(wildcard $(gendir)/$(hdr)), \ $(error Missing generated UAPI file $(gendir)/$(hdr)) \ )) +input-files3-name := $(notdir $(input-files3)) # Work out what needs to be removed oldheaders := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h)) @@ -72,7 +78,9 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@)) quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\ file$(if $(word 2, $(all-files)),s)) cmd_install = \ - $(CONFIG_SHELL) $< $(installdir) $(input-files); \ + $(CONFIG_SHELL) $< $(installdir) $(srcdir) $(input-files1-name); \ + $(CONFIG_SHELL) $< $(installdir) $(oldsrcdir) $(input-files2-name); \ + $(CONFIG_SHELL) $< $(installdir) $(gendir) $(input-files3-name); \ for F in $(wrapper-files); do \ echo "\#include <asm-generic/$$F>" > $(installdir)/$$F; \ done; \ @@ -98,7 +106,7 @@ __headersinst: $(subdirs) $(install-file) @: targets += $(install-file) -$(install-file): scripts/headers_install.sh $(input-files) FORCE +$(install-file): scripts/headers_install.sh $(input-files1) $(input-files2) $(input-files3) FORCE $(if $(unwanted),$(call cmd,remove),) $(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@))) $(call if_changed,install) diff --git a/scripts/headers_install.sh b/scripts/headers_install.sh index 643764f..5de5660 100644 --- a/scripts/headers_install.sh +++ b/scripts/headers_install.sh @@ -2,7 +2,7 @@ if [ $# -lt 1 ] then - echo "Usage: headers_install.sh OUTDIR [FILES...] + echo "Usage: headers_install.sh OUTDIR SRCDIR [FILES...] echo echo "Prepares kernel header files for use by user space, by removing" echo "all compiler.h definitions and #includes, removing any" @@ -10,6 +10,7 @@ then echo "asm/inline/volatile keywords." echo echo "OUTDIR: directory to write each userspace header FILE to." + echo "SRCDIR: source directory where files are picked." echo "FILES: list of header files to operate on." exit 1 @@ -19,6 +20,8 @@ fi OUTDIR="$1" shift +SRCDIR="$1" +shift # Iterate through files listed on command line @@ -34,7 +37,7 @@ do -e 's/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g' \ -e 's/(^|[ \t(])(inline|asm|volatile)([ \t(]|$)/\1__\2__\3/g' \ -e 's@#(ifndef|define|endif[ \t]*/[*])[ \t]*_UAPI@#\1 @' \ - "$i" > "$OUTDIR/$FILE.sed" || exit 1 + "$SRCDIR/$i" > "$OUTDIR/$FILE.sed" || exit 1 scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__ "$OUTDIR/$FILE.sed" \ > "$OUTDIR/$FILE" [ $? -gt 1 ] && exit 1 -- 1.8.2.1 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH linux-next v2] fix make headers_install when path is too long 2013-04-29 12:15 ` [PATCH linux-next v2] " Nicolas Dichtel @ 2013-04-29 12:34 ` Bruce Ashfield 2013-05-06 8:19 ` Nicolas Dichtel 0 siblings, 1 reply; 17+ messages in thread From: Bruce Ashfield @ 2013-04-29 12:34 UTC (permalink / raw) To: Nicolas Dichtel; +Cc: sam, mmarek, linux-kbuild On 13-04-29 08:15 AM, Nicolas Dichtel wrote: > If headers_install is executed from a deep/long directory structure, the > shell's maximum argument length can be execeeded, which breaks the operation > with: > > | make[2]: execvp: /bin/sh: Argument list too long > | make[2]: *** > > Instead of passing each files name with the entire path, I give only the file > name without the source path and give this path as a new argument to > headers_install.pl. > > Because there is three possible paths, I have tree input-files list, one per > path. This version still works for my test case. Tested-by: Bruce Ashfield <bruce.ashfield@windriver.com> > > Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com> > --- > > v2: rebase patch on linux-next > > scripts/Makefile.headersinst | 20 ++++++++++++++------ > scripts/headers_install.sh | 7 +++++-- > 2 files changed, 19 insertions(+), 8 deletions(-) > > diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst > index 182084d..8ccf830 100644 > --- a/scripts/Makefile.headersinst > +++ b/scripts/Makefile.headersinst > @@ -47,18 +47,24 @@ header-y := $(filter-out $(generic-y), $(header-y)) > all-files := $(header-y) $(genhdr-y) $(wrapper-files) > output-files := $(addprefix $(installdir)/, $(all-files)) > > -input-files := $(foreach hdr, $(header-y), \ > +input-files1 := $(foreach hdr, $(header-y), \ > $(if $(wildcard $(srcdir)/$(hdr)), \ > - $(wildcard $(srcdir)/$(hdr)), \ > + $(wildcard $(srcdir)/$(hdr))) \ > + ) > +input-files1-name := $(notdir $(input-files1)) > +input-files2 := $(foreach hdr, $(header-y), \ > + $(if $(wildcard $(srcdir)/$(hdr)),, \ > $(if $(wildcard $(oldsrcdir)/$(hdr)), \ > $(wildcard $(oldsrcdir)/$(hdr)), \ > $(error Missing UAPI file $(srcdir)/$(hdr))) \ > - )) \ > - $(foreach hdr, $(genhdr-y), \ > + )) > +input-files2-name := $(notdir $(input-files2)) > +input-files3 := $(foreach hdr, $(genhdr-y), \ > $(if $(wildcard $(gendir)/$(hdr)), \ > $(wildcard $(gendir)/$(hdr)), \ > $(error Missing generated UAPI file $(gendir)/$(hdr)) \ > )) > +input-files3-name := $(notdir $(input-files3)) > > # Work out what needs to be removed > oldheaders := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h)) > @@ -72,7 +78,9 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@)) > quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\ > file$(if $(word 2, $(all-files)),s)) > cmd_install = \ > - $(CONFIG_SHELL) $< $(installdir) $(input-files); \ > + $(CONFIG_SHELL) $< $(installdir) $(srcdir) $(input-files1-name); \ > + $(CONFIG_SHELL) $< $(installdir) $(oldsrcdir) $(input-files2-name); \ > + $(CONFIG_SHELL) $< $(installdir) $(gendir) $(input-files3-name); \ > for F in $(wrapper-files); do \ > echo "\#include <asm-generic/$$F>" > $(installdir)/$$F; \ > done; \ > @@ -98,7 +106,7 @@ __headersinst: $(subdirs) $(install-file) > @: > > targets += $(install-file) > -$(install-file): scripts/headers_install.sh $(input-files) FORCE > +$(install-file): scripts/headers_install.sh $(input-files1) $(input-files2) $(input-files3) FORCE > $(if $(unwanted),$(call cmd,remove),) > $(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@))) > $(call if_changed,install) > diff --git a/scripts/headers_install.sh b/scripts/headers_install.sh > index 643764f..5de5660 100644 > --- a/scripts/headers_install.sh > +++ b/scripts/headers_install.sh > @@ -2,7 +2,7 @@ > > if [ $# -lt 1 ] > then > - echo "Usage: headers_install.sh OUTDIR [FILES...] > + echo "Usage: headers_install.sh OUTDIR SRCDIR [FILES...] > echo > echo "Prepares kernel header files for use by user space, by removing" > echo "all compiler.h definitions and #includes, removing any" > @@ -10,6 +10,7 @@ then > echo "asm/inline/volatile keywords." > echo > echo "OUTDIR: directory to write each userspace header FILE to." > + echo "SRCDIR: source directory where files are picked." > echo "FILES: list of header files to operate on." > > exit 1 > @@ -19,6 +20,8 @@ fi > > OUTDIR="$1" > shift > +SRCDIR="$1" > +shift > > # Iterate through files listed on command line > > @@ -34,7 +37,7 @@ do > -e 's/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g' \ > -e 's/(^|[ \t(])(inline|asm|volatile)([ \t(]|$)/\1__\2__\3/g' \ > -e 's@#(ifndef|define|endif[ \t]*/[*])[ \t]*_UAPI@#\1 @' \ > - "$i" > "$OUTDIR/$FILE.sed" || exit 1 > + "$SRCDIR/$i" > "$OUTDIR/$FILE.sed" || exit 1 > scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__ "$OUTDIR/$FILE.sed" \ > > "$OUTDIR/$FILE" > [ $? -gt 1 ] && exit 1 > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH linux-next v2] fix make headers_install when path is too long 2013-04-29 12:34 ` Bruce Ashfield @ 2013-05-06 8:19 ` Nicolas Dichtel 2013-05-17 9:12 ` Nicolas Dichtel 0 siblings, 1 reply; 17+ messages in thread From: Nicolas Dichtel @ 2013-05-06 8:19 UTC (permalink / raw) To: mmarek; +Cc: Bruce Ashfield, sam, linux-kbuild Le 29/04/2013 14:34, Bruce Ashfield a écrit : > On 13-04-29 08:15 AM, Nicolas Dichtel wrote: >> If headers_install is executed from a deep/long directory structure, the >> shell's maximum argument length can be execeeded, which breaks the operation >> with: >> >> | make[2]: execvp: /bin/sh: Argument list too long >> | make[2]: *** >> >> Instead of passing each files name with the entire path, I give only the file >> name without the source path and give this path as a new argument to >> headers_install.pl. >> >> Because there is three possible paths, I have tree input-files list, one per >> path. > > This version still works for my test case. > > Tested-by: Bruce Ashfield <bruce.ashfield@windriver.com> > >> >> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com> >> --- Should I do something more for this patch? It seems not included. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH linux-next v2] fix make headers_install when path is too long 2013-05-06 8:19 ` Nicolas Dichtel @ 2013-05-17 9:12 ` Nicolas Dichtel 2013-05-17 20:08 ` Michal Marek 0 siblings, 1 reply; 17+ messages in thread From: Nicolas Dichtel @ 2013-05-17 9:12 UTC (permalink / raw) To: mmarek; +Cc: Bruce Ashfield, sam, linux-kbuild Le 06/05/2013 10:19, Nicolas Dichtel a écrit : > Le 29/04/2013 14:34, Bruce Ashfield a écrit : >> On 13-04-29 08:15 AM, Nicolas Dichtel wrote: >>> If headers_install is executed from a deep/long directory structure, the >>> shell's maximum argument length can be execeeded, which breaks the operation >>> with: >>> >>> | make[2]: execvp: /bin/sh: Argument list too long >>> | make[2]: *** >>> >>> Instead of passing each files name with the entire path, I give only the file >>> name without the source path and give this path as a new argument to >>> headers_install.pl. >>> >>> Because there is three possible paths, I have tree input-files list, one per >>> path. >> >> This version still works for my test case. >> >> Tested-by: Bruce Ashfield <bruce.ashfield@windriver.com> >> >>> >>> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com> >>> --- > Should I do something more for this patch? It seems not included. ping ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH linux-next v2] fix make headers_install when path is too long 2013-05-17 9:12 ` Nicolas Dichtel @ 2013-05-17 20:08 ` Michal Marek 0 siblings, 0 replies; 17+ messages in thread From: Michal Marek @ 2013-05-17 20:08 UTC (permalink / raw) To: Nicolas Dichtel; +Cc: Bruce Ashfield, sam, linux-kbuild On Fri, May 17, 2013 at 11:12:30AM +0200, Nicolas Dichtel wrote: > Le 06/05/2013 10:19, Nicolas Dichtel a écrit : >> Le 29/04/2013 14:34, Bruce Ashfield a écrit : >>> On 13-04-29 08:15 AM, Nicolas Dichtel wrote: >>>> If headers_install is executed from a deep/long directory structure, the >>>> shell's maximum argument length can be execeeded, which breaks the operation >>>> with: >>>> >>>> | make[2]: execvp: /bin/sh: Argument list too long >>>> | make[2]: *** >>>> >>>> Instead of passing each files name with the entire path, I give only the file >>>> name without the source path and give this path as a new argument to >>>> headers_install.pl. >>>> >>>> Because there is three possible paths, I have tree input-files list, one per >>>> path. >>> >>> This version still works for my test case. >>> >>> Tested-by: Bruce Ashfield <bruce.ashfield@windriver.com> >>> >>>> >>>> Signed-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com> >>>> --- >> Should I do something more for this patch? It seems not included. > ping I applied the patch to kbuild.git#kbuild now. Michal ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2013-05-17 20:08 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-03-06 11:16 make headers_install fail when path is too long Nicolas Dichtel 2013-03-06 16:10 ` Sam Ravnborg 2013-03-06 16:24 ` Michal Marek 2013-03-06 16:27 ` Nicolas Dichtel 2013-03-06 16:50 ` Bruce Ashfield 2013-03-07 8:26 ` Michal Marek 2013-03-07 13:28 ` Bruce Ashfield 2013-04-26 16:36 ` [PATCH] fix make headers_install " Nicolas Dichtel 2013-04-26 18:14 ` Bruce Ashfield 2013-04-26 18:57 ` Sam Ravnborg 2013-04-29 12:13 ` Michal Marek 2013-04-29 12:17 ` Nicolas Dichtel 2013-04-29 12:15 ` [PATCH linux-next v2] " Nicolas Dichtel 2013-04-29 12:34 ` Bruce Ashfield 2013-05-06 8:19 ` Nicolas Dichtel 2013-05-17 9:12 ` Nicolas Dichtel 2013-05-17 20:08 ` Michal Marek
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).