* [PATCH 0/1] package_manager.py: fix for "Argument list too long" @ 2016-10-13 9:45 Robert Yang 2016-10-13 9:45 ` [PATCH 1/1] " Robert Yang 0 siblings, 1 reply; 7+ messages in thread From: Robert Yang @ 2016-10-13 9:45 UTC (permalink / raw) To: openembedded-core The following changes since commit 1a2311c8fa9a3703568cd390c44639fe3335023e: build-appliance-image: Update to master head revision (2016-10-11 23:43:20 +0100) are available in the git repository at: git://git.openembedded.org/openembedded-core-contrib rbt/long http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long Robert Yang (1): package_manager.py: fix for "Argument list too long" meta/lib/oe/package_manager.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) -- 2.9.0 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 1/1] package_manager.py: fix for "Argument list too long" 2016-10-13 9:45 [PATCH 0/1] package_manager.py: fix for "Argument list too long" Robert Yang @ 2016-10-13 9:45 ` Robert Yang 2016-10-13 10:39 ` Burton, Ross 0 siblings, 1 reply; 7+ messages in thread From: Robert Yang @ 2016-10-13 9:45 UTC (permalink / raw) To: openembedded-core Fixed when len(TMPDIR) = 410: $ bitbake core-image-sato-sdk [snip] Subprocess output: /bin/sh: /bin/cp: Argument list too long ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs [snip] This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does: "cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", change cwd to deploy_arch_dir can avoid the error. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta/lib/oe/package_manager.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py index 3cee973..e0e203b 100644 --- a/meta/lib/oe/package_manager.py +++ b/meta/lib/oe/package_manager.py @@ -1024,7 +1024,11 @@ class RpmPM(PackageManager): lockfilename = self.d.getVar('DEPLOY_DIR_RPM', True) + "/rpm.lock" lf = bb.utils.lockfile(lockfilename, False) - oe.path.copyhardlinktree(deploy_arch_dir, arch_channel) + # chdir() to deploy_arch_dir to avoid "Argument list too long" error + oldcwd = os.getcwd() + os.chdir(deploy_arch_dir) + oe.path.copyhardlinktree('.', arch_channel) + os.chdir(oldcwd) bb.utils.unlockfile(lf) if not arch in ch_already_added: -- 2.9.0 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 1/1] package_manager.py: fix for "Argument list too long" 2016-10-13 9:45 ` [PATCH 1/1] " Robert Yang @ 2016-10-13 10:39 ` Burton, Ross 2016-10-14 9:58 ` Robert Yang 0 siblings, 1 reply; 7+ messages in thread From: Burton, Ross @ 2016-10-13 10:39 UTC (permalink / raw) To: Robert Yang; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 535 bytes --] On 13 October 2016 at 10:45, Robert Yang <liezhi.yang@windriver.com> wrote: > This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does: > "cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is > expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 > deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", > change cwd to deploy_arch_dir can avoid the error. > Would it be better to change the implementation of copyhardlinktree so that it *can't* have this problem? Ross [-- Attachment #2: Type: text/html, Size: 1003 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 1/1] package_manager.py: fix for "Argument list too long" 2016-10-13 10:39 ` Burton, Ross @ 2016-10-14 9:58 ` Robert Yang 2016-10-14 10:18 ` Robert Yang 0 siblings, 1 reply; 7+ messages in thread From: Robert Yang @ 2016-10-14 9:58 UTC (permalink / raw) To: Burton, Ross; +Cc: OE-core On 10/13/2016 06:39 PM, Burton, Ross wrote: > > On 13 October 2016 at 10:45, Robert Yang <liezhi.yang@windriver.com > <mailto:liezhi.yang@windriver.com>> wrote: > > This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does: > "cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is > expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 > deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", > change cwd to deploy_arch_dir can avoid the error. > > > Would it be better to change the implementation of copyhardlinktree so that it > *can't* have this problem? Good idea, thanks, updated in the repo: git://git.openembedded.org/openembedded-core-contrib rbt/long http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long Author: Robert Yang <liezhi.yang@windriver.com> Date: Thu Oct 13 01:28:41 2016 -0700 oe/path.py: fix for "Argument list too long" Fixed when len(TMPDIR) = 410: $ bitbake core-image-sato-sdk [snip] Subprocess output: /bin/sh: /bin/cp: Argument list too long ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs [snip] This is because "copyhardlinktree(src, dst)" does "cp -afl src/* dst", while src/* is expanded to "src/file1 src/file2, src/file3..." which causes the "Argument list too long", change cwd to src can fix the problem. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py index 06a5af2..13821b3 100644 --- a/meta/lib/oe/path.py +++ b/meta/lib/oe/path.py @@ -80,14 +80,18 @@ def copyhardlinktree(src, dst): cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst) subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) source = '' + # chdir() to src to avoid "Argument list too long" error + oldcwd = os.getcwd() if os.path.isdir(src): + os.chdir(src) import glob - if len(glob.glob('%s/.??*' % src)) > 0: - source = '%s/.??* ' % src - source = source + '%s/*' % src + if len(glob.glob('.??*')) > 0: + source = '.??* ' + source += '*' else: source = src cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst) + os.chdir(oldcwd) subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) else: copytree(src, dst) // Robert > > Ross ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 1/1] package_manager.py: fix for "Argument list too long" 2016-10-14 9:58 ` Robert Yang @ 2016-10-14 10:18 ` Robert Yang 2016-10-16 4:26 ` Christopher Larson 0 siblings, 1 reply; 7+ messages in thread From: Robert Yang @ 2016-10-14 10:18 UTC (permalink / raw) To: Burton, Ross; +Cc: OE-core On 10/14/2016 05:58 PM, Robert Yang wrote: > > > On 10/13/2016 06:39 PM, Burton, Ross wrote: >> >> On 13 October 2016 at 10:45, Robert Yang <liezhi.yang@windriver.com >> <mailto:liezhi.yang@windriver.com>> wrote: >> >> This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does: >> "cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is >> expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 >> deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", >> change cwd to deploy_arch_dir can avoid the error. >> >> >> Would it be better to change the implementation of copyhardlinktree so that it >> *can't* have this problem? > > Good idea, thanks, updated in the repo: > > git://git.openembedded.org/openembedded-core-contrib rbt/long > http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long > Sorry, this patch is not what I wanted to paste, it should be: diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py index 06a5af2..1d9cca5 100644 --- a/meta/lib/oe/path.py +++ b/meta/lib/oe/path.py @@ -80,15 +80,20 @@ def copyhardlinktree(src, dst): cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst) subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) source = '' + oldcwd = os.getcwd() if os.path.isdir(src): + dst = os.path.realpath(dst) + # chdir() to src to avoid "Argument list too long" error + os.chdir(src) import glob - if len(glob.glob('%s/.??*' % src)) > 0: - source = '%s/.??* ' % src - source = source + '%s/*' % src + if len(glob.glob('./.??*')) > 0: + source = './.??* ' + source += './*' else: source = src cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst) subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) + os.chdir(oldcwd) else: copytree(src, dst) // Robert > > Author: Robert Yang <liezhi.yang@windriver.com> > Date: Thu Oct 13 01:28:41 2016 -0700 > > oe/path.py: fix for "Argument list too long" > > Fixed when len(TMPDIR) = 410: > $ bitbake core-image-sato-sdk > [snip] > Subprocess output: > /bin/sh: /bin/cp: Argument list too long > > ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs > [snip] > > This is because "copyhardlinktree(src, dst)" does "cp -afl src/* dst", > while src/* is expanded to "src/file1 src/file2, src/file3..." which > causes the "Argument list too long", change cwd to src can fix the > problem. > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > > > diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py > index 06a5af2..13821b3 100644 > --- a/meta/lib/oe/path.py > +++ b/meta/lib/oe/path.py > @@ -80,14 +80,18 @@ def copyhardlinktree(src, dst): > cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*' > -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' > -xf - -C %s" % (src, src, dst) > subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) > source = '' > + # chdir() to src to avoid "Argument list too long" error > + oldcwd = os.getcwd() > if os.path.isdir(src): > + os.chdir(src) > import glob > - if len(glob.glob('%s/.??*' % src)) > 0: > - source = '%s/.??* ' % src > - source = source + '%s/*' % src > + if len(glob.glob('.??*')) > 0: > + source = '.??* ' > + source += '*' > else: > source = src > cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst) > + os.chdir(oldcwd) > subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) > else: > copytree(src, dst) > > // Robert > >> >> Ross ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 1/1] package_manager.py: fix for "Argument list too long" 2016-10-14 10:18 ` Robert Yang @ 2016-10-16 4:26 ` Christopher Larson 2016-10-17 3:45 ` Robert Yang 0 siblings, 1 reply; 7+ messages in thread From: Christopher Larson @ 2016-10-16 4:26 UTC (permalink / raw) To: Robert Yang; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 2651 bytes --] On Fri, Oct 14, 2016 at 3:18 AM, Robert Yang <liezhi.yang@windriver.com> wrote: > On 10/14/2016 05:58 PM, Robert Yang wrote: > >> >> >> On 10/13/2016 06:39 PM, Burton, Ross wrote: >> >>> >>> On 13 October 2016 at 10:45, Robert Yang <liezhi.yang@windriver.com >>> <mailto:liezhi.yang@windriver.com>> wrote: >>> >>> This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" >>> does: >>> "cp -afl deploy_arch_dir/* arch_channel", while the >>> deploy_arch_dir/* is >>> expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 >>> deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", >>> change cwd to deploy_arch_dir can avoid the error. >>> >>> >>> Would it be better to change the implementation of copyhardlinktree so >>> that it >>> *can't* have this problem? >>> >> >> Good idea, thanks, updated in the repo: >> >> git://git.openembedded.org/openembedded-core-contrib rbt/long >> http://cgit.openembedded.org/cgit.cgi/openembedded-core-cont >> rib/log/?h=rbt/long >> >> > Sorry, this patch is not what I wanted to paste, it should be: > > diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py > index 06a5af2..1d9cca5 100644 > --- a/meta/lib/oe/path.py > +++ b/meta/lib/oe/path.py > @@ -80,15 +80,20 @@ def copyhardlinktree(src, dst): > cmd = "cd %s; find . -type d -print | tar --xattrs > --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar > --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst) > subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) > source = '' > + oldcwd = os.getcwd() > if os.path.isdir(src): > + dst = os.path.realpath(dst) > + # chdir() to src to avoid "Argument list too long" error > + os.chdir(src) > import glob > - if len(glob.glob('%s/.??*' % src)) > 0: > - source = '%s/.??* ' % src > - source = source + '%s/*' % src > + if len(glob.glob('./.??*')) > 0: > + source = './.??* ' > + source += './*' > else: > source = src > cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst) > subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) > + os.chdir(oldcwd) Question, why not just pass cwd=src in the check_output() call, rather than changing it in the parent python process? -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics [-- Attachment #2: Type: text/html, Size: 4096 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 1/1] package_manager.py: fix for "Argument list too long" 2016-10-16 4:26 ` Christopher Larson @ 2016-10-17 3:45 ` Robert Yang 0 siblings, 0 replies; 7+ messages in thread From: Robert Yang @ 2016-10-17 3:45 UTC (permalink / raw) To: Christopher Larson; +Cc: OE-core Hi Christopher, On 10/16/2016 12:26 PM, Christopher Larson wrote: > > On Fri, Oct 14, 2016 at 3:18 AM, Robert Yang <liezhi.yang@windriver.com > <mailto:liezhi.yang@windriver.com>> wrote: > > On 10/14/2016 05:58 PM, Robert Yang wrote: > > > > On 10/13/2016 06:39 PM, Burton, Ross wrote: > > > On 13 October 2016 at 10:45, Robert Yang <liezhi.yang@windriver.com > <mailto:liezhi.yang@windriver.com> > <mailto:liezhi.yang@windriver.com > <mailto:liezhi.yang@windriver.com>>> wrote: > > This is because "copyhardlinktree(deploy_arch_dir, > arch_channel)" does: > "cp -afl deploy_arch_dir/* arch_channel", while the > deploy_arch_dir/* is > expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 > deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", > change cwd to deploy_arch_dir can avoid the error. > > > Would it be better to change the implementation of copyhardlinktree > so that it > *can't* have this problem? > > > Good idea, thanks, updated in the repo: > > git://git.openembedded.org/openembedded-core-contrib > <http://git.openembedded.org/openembedded-core-contrib> rbt/long > > http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long > <http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long> > > > Sorry, this patch is not what I wanted to paste, it should be: > > diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py > index 06a5af2..1d9cca5 100644 > --- a/meta/lib/oe/path.py > +++ b/meta/lib/oe/path.py > @@ -80,15 +80,20 @@ def copyhardlinktree(src, dst): > cmd = "cd %s; find . -type d -print | tar --xattrs > --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar > --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst) > subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) > source = '' > + oldcwd = os.getcwd() > if os.path.isdir(src): > + dst = os.path.realpath(dst) > + # chdir() to src to avoid "Argument list too long" error > + os.chdir(src) > import glob > - if len(glob.glob('%s/.??*' % src)) > 0: > - source = '%s/.??* ' % src > - source = source + '%s/*' % src > + if len(glob.glob('./.??*')) > 0: > + source = './.??* ' > + source += './*' > else: > source = src > cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst) > subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) > + os.chdir(oldcwd) > > > Question, why not just pass cwd=src in the check_output() call, rather than > changing it in the parent python process? src can be a file or directory, so we can't simply use cwd=src, I will use os.getcwd() as cwd when it is not a directory. Updated patch in the repo. git://git.openembedded.org/openembedded-core-contrib rbt/long Author: Robert Yang <liezhi.yang@windriver.com> Date: Thu Oct 13 01:28:41 2016 -0700 oe/path.py: fix for "Argument list too long" Fixed when len(TMPDIR) = 410: $ bitbake core-image-sato-sdk [snip] Subprocess output: /bin/sh: /bin/cp: Argument list too long ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs [snip] This is because "copyhardlinktree(src, dst)" does "cp -afl src/* dst", while src/* is expanded to "src/file1 src/file2, src/file3..." which causes the "Argument list too long", use ./* as src and change cwd in subprocess.check_output() to fix the problem. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py index 06a5af2..ed7fd1e 100644 --- a/meta/lib/oe/path.py +++ b/meta/lib/oe/path.py @@ -83,12 +83,14 @@ def copyhardlinktree(src, dst): if os.path.isdir(src): import glob if len(glob.glob('%s/.??*' % src)) > 0: - source = '%s/.??* ' % src - source = source + '%s/*' % src + source = './.??* ' + source += './*' + s_dir = src else: source = src - cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst) - subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) + s_dir = os.getcwd() + cmd = 'cp -afl --preserve=xattr %s %s' % (source, os.path.realpath(dst)) + subprocess.check_output(cmd, shell=True, cwd=s_dir, stderr=subprocess.STDOUT) else: copytree(src, dst) // Robert > -- > Christopher Larson > clarson at kergoth dot com > Founder - BitBake, OpenEmbedded, OpenZaurus > Maintainer - Tslib > Senior Software Engineer, Mentor Graphics ^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-10-17 3:45 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-10-13 9:45 [PATCH 0/1] package_manager.py: fix for "Argument list too long" Robert Yang 2016-10-13 9:45 ` [PATCH 1/1] " Robert Yang 2016-10-13 10:39 ` Burton, Ross 2016-10-14 9:58 ` Robert Yang 2016-10-14 10:18 ` Robert Yang 2016-10-16 4:26 ` Christopher Larson 2016-10-17 3:45 ` Robert Yang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.