[PATCH 1/1] sanity.bbclass: add check for creation of long filenames

[PATCH 1/1] sanity.bbclass: add check for creation of long filenames

[Paul Eggleton]

Detect and fail if filesystem in use for TMPDIR or SSTATE_DIR has an
unreasonably short file name length limit (eg. eCryptFS). This can cause
"file name too long" errors during poky builds (e.g. when writing sstate
files for packages with a git revision as the version).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/sanity.bbclass |   33 ++++++++++++++++++++++++++++++++-
 1 files changed, 32 insertions(+), 1 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 06aeddc..90e8911 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -21,6 +21,21 @@ def check_conf_exists(fn, data):
 			return True
 	return False
 
+def check_create_long_filename(filepath, pathname):
+	testfile = os.path.join(filepath, ''.join([`num`[-1] for num in xrange(1,200)]))
+	try:
+		if not os.path.exists(filepath):
+			os.path.mkdir(filepath)
+		f = file(testfile, "w")
+		f.close()
+		os.remove(testfile)
+	except IOError as (errno, strerror):
+		if errno == 36: # ENAMETOOLONG
+			return "Failed to create a file with a long name in {0}. Please use a filesystem that does not unreasonably limit filename length.\n".format(pathname)
+		else:
+			return "Failed to create a file in {0}: {1}".format(pathname, strerror)
+	return ""
+
 def check_sanity(e):
 	from bb import note, error, data, __version__
 
@@ -163,10 +178,26 @@ def check_sanity(e):
 		if os.path.exists('%s/libc.so.6' % lib32path) and not os.path.exists('/usr/include/gnu/stubs-32.h'):
 			messages = messages + "You have a 32-bit libc, but no 32-bit headers.  You must install the 32-bit libc headers.\n"
 
+	tmpdir = data.getVar('TMPDIR', e.data, True)
+
+	#
+	# Check that TMPDIR isn't on a filesystem with limited filename length (eg. eCryptFS)
+	#
+	testmsg = check_create_long_filename(tmpdir, "TMPDIR")
+	if testmsg != "":
+		messages = messages + testmsg
+	#
+	# Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS)
+	#
+	sstatedir = data.getVar('SSTATE_DIR', e.data, True)
+	if sstatedir != "":
+		testmsg = check_create_long_filename(sstatedir, "SSTATE_DIR")
+		if testmsg != "":
+			messages = messages + testmsg
+
 	#
 	# Check that TMPDIR hasn't changed location since the last time we were run
 	#
-	tmpdir = data.getVar('TMPDIR', e.data, True)
 	checkfile = os.path.join(tmpdir, "saved_tmpdir")
 	if os.path.exists(checkfile):
 		f = file(checkfile, "r")
-- 
1.7.1

Re: [PATCH 1/1] sanity.bbclass: add check for creation of long filenames

[Gary Thomas]

On 11/30/2010 09:40 AM, Paul Eggleton wrote:
> Detect and fail if filesystem in use for TMPDIR or SSTATE_DIR has an
> unreasonably short file name length limit (eg. eCryptFS). This can cause
> "file name too long" errors during poky builds (e.g. when writing sstate
> files for packages with a git revision as the version).

Is this information cached?  It's seems quite the burden
to have to create/remove ~400 files each time on startup.

> Signed-off-by: Paul Eggleton<paul.eggleton@linux.intel.com>
> ---
>   meta/classes/sanity.bbclass |   33 ++++++++++++++++++++++++++++++++-
>   1 files changed, 32 insertions(+), 1 deletions(-)
>
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index 06aeddc..90e8911 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -21,6 +21,21 @@ def check_conf_exists(fn, data):
>   			return True
>   	return False
>
> +def check_create_long_filename(filepath, pathname):
> +	testfile = os.path.join(filepath, ''.join([`num`[-1] for num in xrange(1,200)]))
> +	try:
> +		if not os.path.exists(filepath):
> +			os.path.mkdir(filepath)
> +		f = file(testfile, "w")
> +		f.close()
> +		os.remove(testfile)
> +	except IOError as (errno, strerror):
> +		if errno == 36: # ENAMETOOLONG
> +			return "Failed to create a file with a long name in {0}. Please use a filesystem that does not unreasonably limit filename length.\n".format(pathname)
> +		else:
> +			return "Failed to create a file in {0}: {1}".format(pathname, strerror)
> +	return ""
> +
>   def check_sanity(e):
>   	from bb import note, error, data, __version__
>
> @@ -163,10 +178,26 @@ def check_sanity(e):
>   		if os.path.exists('%s/libc.so.6' % lib32path) and not os.path.exists('/usr/include/gnu/stubs-32.h'):
>   			messages = messages + "You have a 32-bit libc, but no 32-bit headers.  You must install the 32-bit libc headers.\n"
>
> +	tmpdir = data.getVar('TMPDIR', e.data, True)
> +
> +	#
> +	# Check that TMPDIR isn't on a filesystem with limited filename length (eg. eCryptFS)
> +	#
> +	testmsg = check_create_long_filename(tmpdir, "TMPDIR")
> +	if testmsg != "":
> +		messages = messages + testmsg
> +	#
> +	# Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS)
> +	#
> +	sstatedir = data.getVar('SSTATE_DIR', e.data, True)
> +	if sstatedir != "":
> +		testmsg = check_create_long_filename(sstatedir, "SSTATE_DIR")
> +		if testmsg != "":
> +			messages = messages + testmsg
> +
>   	#
>   	# Check that TMPDIR hasn't changed location since the last time we were run
>   	#
> -	tmpdir = data.getVar('TMPDIR', e.data, True)
>   	checkfile = os.path.join(tmpdir, "saved_tmpdir")
>   	if os.path.exists(checkfile):
>   		f = file(checkfile, "r")

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: [PATCH 1/1] sanity.bbclass: add check for creation of long filenames

[Paul Eggleton]

On Thursday 02 December 2010 12:21:06 Gary Thomas wrote:
> Is this information cached?  It's seems quite the burden
> to have to create/remove ~400 files each time on startup.

I'm not sure what you mean - this check (well, all sanity checks) should only occur once per invocation of bitbake.

Cheers,
Paul

Re: [PATCH 1/1] sanity.bbclass: add check for creation of long filenames

[Gary Thomas]

On 12/02/2010 07:13 AM, Paul Eggleton wrote:
> On Thursday 02 December 2010 12:21:06 Gary Thomas wrote:
>> Is this information cached?  It's seems quite the burden
>> to have to create/remove ~400 files each time on startup.
>
> I'm not sure what you mean - this check (well, all sanity checks) should only occur once per invocation of bitbake.

Precisely what I mean.  The computed max length won't change
unless you move the tmp or sstate-cache directories, so recomputing
it every time you run bitbake is a horrible overhead.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: [PATCH 1/1] sanity.bbclass: add check for creation of long filenames

[Paul Eggleton]

On Thursday 02 December 2010 14:16:35 Gary Thomas wrote:
> On 12/02/2010 07:13 AM, Paul Eggleton wrote:
> > On Thursday 02 December 2010 12:21:06 Gary Thomas wrote:
> >> Is this information cached?  It's seems quite the burden
> >> to have to create/remove ~400 files each time on startup.
> >
> > I'm not sure what you mean - this check (well, all sanity checks) should only occur once per invocation of bitbake.
> 
> Precisely what I mean.  The computed max length won't change
> unless you move the tmp or sstate-cache directories, so recomputing
> it every time you run bitbake is a horrible overhead.

It would also change if the filesystem where TMPDIR or SSTATE_DIR is stored changes, which could occur without the path itself changing, e.g. someone switches over their home directory to be encrypted. It would be possible to cache this check based on the filesystem and path, but is it really worth it given that we would have to read the disk to check if these had changed anyway? I do understand we have to be careful about items we add to the sanity checks, but in this instance we're talking about writing one file - something that should take a matter of milliseconds.

Cheers,
Paul

Re: [PATCH 1/1] sanity.bbclass: add check for creation of long filenames

[Gary Thomas]

On 12/02/2010 07:36 AM, Paul Eggleton wrote:
> On Thursday 02 December 2010 14:16:35 Gary Thomas wrote:
>> On 12/02/2010 07:13 AM, Paul Eggleton wrote:
>>> On Thursday 02 December 2010 12:21:06 Gary Thomas wrote:
>>>> Is this information cached?  It's seems quite the burden
>>>> to have to create/remove ~400 files each time on startup.
>>>
>>> I'm not sure what you mean - this check (well, all sanity checks) should only occur once per invocation of bitbake.
>>
>> Precisely what I mean.  The computed max length won't change
>> unless you move the tmp or sstate-cache directories, so recomputing
>> it every time you run bitbake is a horrible overhead.
>
> It would also change if the filesystem where TMPDIR or SSTATE_DIR is stored changes, which could occur without the path itself changing, e.g. someone switches over their home directory to be encrypted. It would be possible to cache this check based on the filesystem and path, but is it really worth it given that we would have to read the disk to check if these had changed anyway? I do understand we have to be careful about items we add to the sanity checks, but in this instance we're talking about writing one file - something that should take a matter of milliseconds.

It looks like I misunderstood your code - it's only creating
one file per directory, so that's OK.

Sorry for the bother

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------