* [PATCH 1/1] sanity.bbclass: add check for creation of long filenames [not found] <cover.1291135878.git.paul.eggleton@linux.intel.com> @ 2010-11-30 16:40 ` Paul Eggleton 2010-12-02 12:21 ` Gary Thomas 0 siblings, 1 reply; 6+ messages in thread From: Paul Eggleton @ 2010-11-30 16:40 UTC (permalink / raw) To: poky Detect and fail if filesystem in use for TMPDIR or SSTATE_DIR has an unreasonably short file name length limit (eg. eCryptFS). This can cause "file name too long" errors during poky builds (e.g. when writing sstate files for packages with a git revision as the version). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- meta/classes/sanity.bbclass | 33 ++++++++++++++++++++++++++++++++- 1 files changed, 32 insertions(+), 1 deletions(-) diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass index 06aeddc..90e8911 100644 --- a/meta/classes/sanity.bbclass +++ b/meta/classes/sanity.bbclass @@ -21,6 +21,21 @@ def check_conf_exists(fn, data): return True return False +def check_create_long_filename(filepath, pathname): + testfile = os.path.join(filepath, ''.join([`num`[-1] for num in xrange(1,200)])) + try: + if not os.path.exists(filepath): + os.path.mkdir(filepath) + f = file(testfile, "w") + f.close() + os.remove(testfile) + except IOError as (errno, strerror): + if errno == 36: # ENAMETOOLONG + return "Failed to create a file with a long name in {0}. Please use a filesystem that does not unreasonably limit filename length.\n".format(pathname) + else: + return "Failed to create a file in {0}: {1}".format(pathname, strerror) + return "" + def check_sanity(e): from bb import note, error, data, __version__ @@ -163,10 +178,26 @@ def check_sanity(e): if os.path.exists('%s/libc.so.6' % lib32path) and not os.path.exists('/usr/include/gnu/stubs-32.h'): messages = messages + "You have a 32-bit libc, but no 32-bit headers. You must install the 32-bit libc headers.\n" + tmpdir = data.getVar('TMPDIR', e.data, True) + + # + # Check that TMPDIR isn't on a filesystem with limited filename length (eg. eCryptFS) + # + testmsg = check_create_long_filename(tmpdir, "TMPDIR") + if testmsg != "": + messages = messages + testmsg + # + # Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS) + # + sstatedir = data.getVar('SSTATE_DIR', e.data, True) + if sstatedir != "": + testmsg = check_create_long_filename(sstatedir, "SSTATE_DIR") + if testmsg != "": + messages = messages + testmsg + # # Check that TMPDIR hasn't changed location since the last time we were run # - tmpdir = data.getVar('TMPDIR', e.data, True) checkfile = os.path.join(tmpdir, "saved_tmpdir") if os.path.exists(checkfile): f = file(checkfile, "r") -- 1.7.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] sanity.bbclass: add check for creation of long filenames 2010-11-30 16:40 ` [PATCH 1/1] sanity.bbclass: add check for creation of long filenames Paul Eggleton @ 2010-12-02 12:21 ` Gary Thomas 2010-12-02 14:13 ` Paul Eggleton 0 siblings, 1 reply; 6+ messages in thread From: Gary Thomas @ 2010-12-02 12:21 UTC (permalink / raw) To: Paul Eggleton; +Cc: poky On 11/30/2010 09:40 AM, Paul Eggleton wrote: > Detect and fail if filesystem in use for TMPDIR or SSTATE_DIR has an > unreasonably short file name length limit (eg. eCryptFS). This can cause > "file name too long" errors during poky builds (e.g. when writing sstate > files for packages with a git revision as the version). Is this information cached? It's seems quite the burden to have to create/remove ~400 files each time on startup. > Signed-off-by: Paul Eggleton<paul.eggleton@linux.intel.com> > --- > meta/classes/sanity.bbclass | 33 ++++++++++++++++++++++++++++++++- > 1 files changed, 32 insertions(+), 1 deletions(-) > > diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass > index 06aeddc..90e8911 100644 > --- a/meta/classes/sanity.bbclass > +++ b/meta/classes/sanity.bbclass > @@ -21,6 +21,21 @@ def check_conf_exists(fn, data): > return True > return False > > +def check_create_long_filename(filepath, pathname): > + testfile = os.path.join(filepath, ''.join([`num`[-1] for num in xrange(1,200)])) > + try: > + if not os.path.exists(filepath): > + os.path.mkdir(filepath) > + f = file(testfile, "w") > + f.close() > + os.remove(testfile) > + except IOError as (errno, strerror): > + if errno == 36: # ENAMETOOLONG > + return "Failed to create a file with a long name in {0}. Please use a filesystem that does not unreasonably limit filename length.\n".format(pathname) > + else: > + return "Failed to create a file in {0}: {1}".format(pathname, strerror) > + return "" > + > def check_sanity(e): > from bb import note, error, data, __version__ > > @@ -163,10 +178,26 @@ def check_sanity(e): > if os.path.exists('%s/libc.so.6' % lib32path) and not os.path.exists('/usr/include/gnu/stubs-32.h'): > messages = messages + "You have a 32-bit libc, but no 32-bit headers. You must install the 32-bit libc headers.\n" > > + tmpdir = data.getVar('TMPDIR', e.data, True) > + > + # > + # Check that TMPDIR isn't on a filesystem with limited filename length (eg. eCryptFS) > + # > + testmsg = check_create_long_filename(tmpdir, "TMPDIR") > + if testmsg != "": > + messages = messages + testmsg > + # > + # Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS) > + # > + sstatedir = data.getVar('SSTATE_DIR', e.data, True) > + if sstatedir != "": > + testmsg = check_create_long_filename(sstatedir, "SSTATE_DIR") > + if testmsg != "": > + messages = messages + testmsg > + > # > # Check that TMPDIR hasn't changed location since the last time we were run > # > - tmpdir = data.getVar('TMPDIR', e.data, True) > checkfile = os.path.join(tmpdir, "saved_tmpdir") > if os.path.exists(checkfile): > f = file(checkfile, "r") -- ------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] sanity.bbclass: add check for creation of long filenames 2010-12-02 12:21 ` Gary Thomas @ 2010-12-02 14:13 ` Paul Eggleton 2010-12-02 14:16 ` Gary Thomas 0 siblings, 1 reply; 6+ messages in thread From: Paul Eggleton @ 2010-12-02 14:13 UTC (permalink / raw) To: poky On Thursday 02 December 2010 12:21:06 Gary Thomas wrote: > Is this information cached? It's seems quite the burden > to have to create/remove ~400 files each time on startup. I'm not sure what you mean - this check (well, all sanity checks) should only occur once per invocation of bitbake. Cheers, Paul ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] sanity.bbclass: add check for creation of long filenames 2010-12-02 14:13 ` Paul Eggleton @ 2010-12-02 14:16 ` Gary Thomas 2010-12-02 14:36 ` Paul Eggleton 0 siblings, 1 reply; 6+ messages in thread From: Gary Thomas @ 2010-12-02 14:16 UTC (permalink / raw) To: Paul Eggleton; +Cc: poky On 12/02/2010 07:13 AM, Paul Eggleton wrote: > On Thursday 02 December 2010 12:21:06 Gary Thomas wrote: >> Is this information cached? It's seems quite the burden >> to have to create/remove ~400 files each time on startup. > > I'm not sure what you mean - this check (well, all sanity checks) should only occur once per invocation of bitbake. Precisely what I mean. The computed max length won't change unless you move the tmp or sstate-cache directories, so recomputing it every time you run bitbake is a horrible overhead. -- ------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] sanity.bbclass: add check for creation of long filenames 2010-12-02 14:16 ` Gary Thomas @ 2010-12-02 14:36 ` Paul Eggleton 2010-12-02 14:41 ` Gary Thomas 0 siblings, 1 reply; 6+ messages in thread From: Paul Eggleton @ 2010-12-02 14:36 UTC (permalink / raw) To: poky On Thursday 02 December 2010 14:16:35 Gary Thomas wrote: > On 12/02/2010 07:13 AM, Paul Eggleton wrote: > > On Thursday 02 December 2010 12:21:06 Gary Thomas wrote: > >> Is this information cached? It's seems quite the burden > >> to have to create/remove ~400 files each time on startup. > > > > I'm not sure what you mean - this check (well, all sanity checks) should only occur once per invocation of bitbake. > > Precisely what I mean. The computed max length won't change > unless you move the tmp or sstate-cache directories, so recomputing > it every time you run bitbake is a horrible overhead. It would also change if the filesystem where TMPDIR or SSTATE_DIR is stored changes, which could occur without the path itself changing, e.g. someone switches over their home directory to be encrypted. It would be possible to cache this check based on the filesystem and path, but is it really worth it given that we would have to read the disk to check if these had changed anyway? I do understand we have to be careful about items we add to the sanity checks, but in this instance we're talking about writing one file - something that should take a matter of milliseconds. Cheers, Paul ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] sanity.bbclass: add check for creation of long filenames 2010-12-02 14:36 ` Paul Eggleton @ 2010-12-02 14:41 ` Gary Thomas 0 siblings, 0 replies; 6+ messages in thread From: Gary Thomas @ 2010-12-02 14:41 UTC (permalink / raw) To: Paul Eggleton; +Cc: poky On 12/02/2010 07:36 AM, Paul Eggleton wrote: > On Thursday 02 December 2010 14:16:35 Gary Thomas wrote: >> On 12/02/2010 07:13 AM, Paul Eggleton wrote: >>> On Thursday 02 December 2010 12:21:06 Gary Thomas wrote: >>>> Is this information cached? It's seems quite the burden >>>> to have to create/remove ~400 files each time on startup. >>> >>> I'm not sure what you mean - this check (well, all sanity checks) should only occur once per invocation of bitbake. >> >> Precisely what I mean. The computed max length won't change >> unless you move the tmp or sstate-cache directories, so recomputing >> it every time you run bitbake is a horrible overhead. > > It would also change if the filesystem where TMPDIR or SSTATE_DIR is stored changes, which could occur without the path itself changing, e.g. someone switches over their home directory to be encrypted. It would be possible to cache this check based on the filesystem and path, but is it really worth it given that we would have to read the disk to check if these had changed anyway? I do understand we have to be careful about items we add to the sanity checks, but in this instance we're talking about writing one file - something that should take a matter of milliseconds. It looks like I misunderstood your code - it's only creating one file per directory, so that's OK. Sorry for the bother -- ------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------ ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-12-02 14:41 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <cover.1291135878.git.paul.eggleton@linux.intel.com>
2010-11-30 16:40 ` [PATCH 1/1] sanity.bbclass: add check for creation of long filenames Paul Eggleton
2010-12-02 12:21 ` Gary Thomas
2010-12-02 14:13 ` Paul Eggleton
2010-12-02 14:16 ` Gary Thomas
2010-12-02 14:36 ` Paul Eggleton
2010-12-02 14:41 ` Gary Thomas
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.