[PATCH 0/1] sanity.bbclass: check for validity of TMPDIR

[PATCH 0/1] sanity.bbclass: check for validity of TMPDIR

[Qi.Chen]

From: Chen Qi <Qi.Chen@windriver.com>

This patch checks the validity of TMPDIR.

The TMPDIR must match the following regexp.
^/[a-zA-Z0-9\-_/.~]+$

Detailed tests about why I came up with this regexp are displayed at the end of this cover letter.

A major reason for why TMPDIR cannot support many special chars such as @ and # is that these
special chars are used by the sed command as its field separator in our recipes.

And here are things I want to make sure of.
1. For now, TMPDIR must be an absolute path. So do we need to make it support relative path?
2. I included the tilde (~) into the valid char set because `bitbake world' succeeded. But
   it's still possible the the tilde (~) will serve as the field separator in sed. So is it
   OK that I include it into the valid char set?


**********************  TESTING RECORDS  ****************************************
Test Case 1: use `bitbake minimal' on paths containing special chars
Here's the output.
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/`_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/~_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/<_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/=_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/>_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/|_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/ _contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/,_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/;_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/:_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/!_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/?_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/'_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/\"_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/(_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/)_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/{_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/}_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/@_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/$_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/*_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/\_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/&_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/#_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/%_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained"
RESULT: SUCCESS


Test Case 2: TMPDIR containing ^
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained"
Command:
    bitbake core-image-minimal -cpopulate_sdk
Result:
    Fail
Formatted Error Message:
    ERROR: Task 5 (virtual:nativesdk:/home/chenqi/poky/meta/recipes-core/eglibc/eglibc_2.18.bb, do_populate_sysroot) failed with exit code '1'
    | checking whether the C compiler works... no
    | configure: error: in `/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained/work/i686-nativesdk-pokysdk-linux/nativesdk-eglibc/2.18-r0/site_config_qemux86':
    | configure: error: C compiler cannot create executables
    | See `config.log' for more details


Test Case 3: TMPDIR containing +
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained"
Command:
    bitbake world
Result:
    Fail
Formatted Error Message:
    ERROR: Task 9084 (virtual:native:/home/chenqi/poky/meta/recipes-devtools/tcltk/tcl_8.6.1.bb, do_install) failed with exit code '1'
    | oe_libinstall: cd /home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained/work/i686-linux/tcl-native/8.6.1-r0/build
    | sed: -e expression #1, char 117: unknown option to `s'
Reason:
    '+' is used by the sed command as a separator in tcl recipe.
    sed -i "s+${WORKDIR}+${STAGING_INCDIR}+g" tclConfig.sh


Test Case 4: TMPDIR containing ]
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained"
Command:
    bitbake world
Result:
    Fail
Formatted Error Message:
    ERROR: Task 5 (virtual:native:/home/chenqi/poky/meta/recipes-graphics/xcb/xcb-proto_1.8.bb, do_configure) failed with exit code '1'
    | configure.ac:9: error: possibly undefined macro: AM_INIT_AUTOMAKE
    |       If this token and others are legitimate, please use m4_pattern_allow.
    |       See the Autoconf documentation.
    | configure.ac:12: error: possibly undefined macro: AM_CONDITIONAL
    | configure.ac:17: error: possibly undefined macro: AM_PATH_PYTHON
    | autoreconf: /home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained/sysroots/i686-linux/usr/bin/autoconf failed with exit status: 1
    | ERROR: autoreconf execution failed.
    | WARNING: exit code 1 from a shell command.


Test Case 5: TMPDIR containing {
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained"
Command:
    bitbake world
Result:
    Fail
Formatted Error Message:
    ERROR: Task 9237 (virtual:native:/home/chenqi/poky/meta/recipes-gnome/gnome/gnome-doc-utils_0.20.10.bb, do_compile) failed with exit code '1'
    | xsltproc -nonet -o "C/db-label.xml" \
    |      --stringparam basename "db-label" \
    |      --stringparam xsl_file "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/{_contained/work/i686-linux/gnome-doc-utils-native/0.20.10-r0/gnome-doc-utils-0.20.10/doc/xslt/../../xslt/\
docbook/common/db-label.xsl" \
    |      "./xsldoc-docbook.xsl" "C/db-label.xsldoc"
    | db-chunk: No stylesheet param for documented db.chunk.chunks
    | db-common: No named template for documented db.copyright
    | make[2]: *** [C/db-chunk.xml] Error 10


Test Case 6: TMPDIR containing @
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/@_contained"
Command:
    bitbake qt-mobility-x11
Result:
    Fail
Formatted Error Message:
    ERROR: Task 2 (/home/chenqi/poky/meta/recipes-qt/qt4/qt-mobility-x11_1.2.0.bb, do_install) failed with exit code '1'
    | DEBUG: Executing shell function do_install
    | sed: -e expression #1, char 75: unknown option to `s'


Test Case 7: TMPDIR containing ~
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained"
Command:
    bitbake world
Result:
   Success

****************************************************************************************

The following changes since commit ea92671d9823e3667d6ced7ac2af20f991da404d:

  bitbake: cooker: replace "w" file opening mode with "a" mode (2013-11-12 17:01:37 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib ChenQi/TMPDIR
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/TMPDIR

Chen Qi (1):
  sanity.bbclass: check for validity of TMPDIR

 meta/classes/sanity.bbclass |    7 +++++++
 1 file changed, 7 insertions(+)

-- 
1.7.9.5

[PATCH 1/1] sanity.bbclass: check for validity of TMPDIR

[Qi.Chen]

From: Chen Qi <Qi.Chen@windriver.com>

TMPDIR must be an absolute path, otherwise, the build will fail.
Special characters in TMPDIR will also cause build failures.

This patch enables checking for the validity of TMPDIR.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/sanity.bbclass |    7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 83378b0..e45906e 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -672,6 +672,13 @@ def check_sanity_everybuild(status, d):
         with open(checkfile, "w") as f:
             f.write(tmpdir)
 
+    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
+    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
+    import re
+    valid_pattern = re.compile(valid_tmpdir_regexp)
+    if not valid_pattern.match(tmpdir):
+        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
+
 def check_sanity(sanity_data):
     import subprocess
 
-- 
1.7.9.5

Re: [PATCH 1/1] sanity.bbclass: check for validity of TMPDIR

[Richard Purdie]

On Thu, 2013-11-14 at 21:07 +0800, Qi.Chen@windriver.com wrote:
> From: Chen Qi <Qi.Chen@windriver.com>
> 
> TMPDIR must be an absolute path, otherwise, the build will fail.
> Special characters in TMPDIR will also cause build failures.
> 
> This patch enables checking for the validity of TMPDIR.
> 
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/classes/sanity.bbclass |    7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index 83378b0..e45906e 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -672,6 +672,13 @@ def check_sanity_everybuild(status, d):
>          with open(checkfile, "w") as f:
>              f.write(tmpdir)
>  
> +    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
> +    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
> +    import re
> +    valid_pattern = re.compile(valid_tmpdir_regexp)
> +    if not valid_pattern.match(tmpdir):
> +        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
> +

I'm ok with the check itself however this does not need to be run every
build. It needs to run when tmpdir changes only.

Cheers,

Richard

Re: [PATCH 1/1] sanity.bbclass: check for validity of TMPDIR

[Gary Thomas]

On 2013-11-14 06:07, Qi.Chen@windriver.com wrote:
> From: Chen Qi <Qi.Chen@windriver.com>
>
> TMPDIR must be an absolute path, otherwise, the build will fail.
> Special characters in TMPDIR will also cause build failures.
>
> This patch enables checking for the validity of TMPDIR.
>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>   meta/classes/sanity.bbclass |    7 +++++++
>   1 file changed, 7 insertions(+)
>
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index 83378b0..e45906e 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -672,6 +672,13 @@ def check_sanity_everybuild(status, d):
>           with open(checkfile, "w") as f:
>               f.write(tmpdir)
>
> +    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
> +    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
> +    import re
> +    valid_pattern = re.compile(valid_tmpdir_regexp)
> +    if not valid_pattern.match(tmpdir):
> +        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
> +
>   def check_sanity(sanity_data):
>       import subprocess

Check the wording of the warning "The TMPDIR should match the this regexp"
should probably be "The TMPDIR should match this regexp" or maybe use a simpler
version like "The TMPDIR name should must be an absolute path and contain only letters,
digits and the characters -_"

Also, is "-" actually valid?  I seem to recall having problems when my build
tree had the hyphen ("-") in the path.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: [PATCH 1/1] sanity.bbclass: check for validity of TMPDIR

[Phil Blundell]

On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
> Also, is "-" actually valid?  I seem to recall having problems when my build
> tree had the hyphen ("-") in the path.

Paths with "-" in certainly work for me.  If there are any recipes which
break in that situation then we should just fix them. 

OE already has quite a range of baroque restrictions on what sort of
TMPDIR you are allowed to use (no nfs, no symlinks in the path, no
spaces in the name) and every new prohibition represents a loss in
usability.  This patch as proposed already forbids a whole range of
characters, including things like "+", and I think that disallowing "-"
as well would be a step too far.

p.

Re: [PATCH 1/1] sanity.bbclass: check for validity of TMPDIR

[Gary Thomas]

On 2013-11-14 06:42, Phil Blundell wrote:
> On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
>> Also, is "-" actually valid?  I seem to recall having problems when my build
>> tree had the hyphen ("-") in the path.
>
> Paths with "-" in certainly work for me.  If there are any recipes which
> break in that situation then we should just fix them.
>
> OE already has quite a range of baroque restrictions on what sort of
> TMPDIR you are allowed to use (no nfs, no symlinks in the path, no
> spaces in the name) and every new prohibition represents a loss in
> usability.  This patch as proposed already forbids a whole range of
> characters, including things like "+", and I think that disallowing "-"
> as well would be a step too far.

Fair enough.  I don't recall exactly what problems I had (in the dark
past), but I've just avoided trees with "-" in them.  I'll give it
another look if/when an opportunity comes by.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: [PATCH 1/1] sanity.bbclass: check for validity of TMPDIR

[Mark Hatle]

On 11/14/13, 7:42 AM, Phil Blundell wrote:
> On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
>> Also, is "-" actually valid?  I seem to recall having problems when my build
>> tree had the hyphen ("-") in the path.
>
> Paths with "-" in certainly work for me.  If there are any recipes which
> break in that situation then we should just fix them.

We found an issue with paths that -start- with a '-', there are apparently many 
places where paths are passed into various shell, and the initial '-' can be 
read as an argument identifier.

> OE already has quite a range of baroque restrictions on what sort of
> TMPDIR you are allowed to use (no nfs, no symlinks in the path, no
> spaces in the name) and every new prohibition represents a loss in
> usability.  This patch as proposed already forbids a whole range of
> characters, including things like "+", and I think that disallowing "-"
> as well would be a step too far.

I don't believe Qi Chen sent it to the list, but we built path names with all of 
the special characters and tried to run builds.  The specific list is based on 
the results of those tests.  The other items are already broken, and we're 
trying to be explicit with this for end users.  (Note, it's really the TMPDIR 
that matters.. the 'build' directory is a lot more flexible.)

--Mark

> p.
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

Re: [PATCH 1/1] sanity.bbclass: check for validity of TMPDIR

[Phil Blundell]

On Thu, 2013-11-14 at 09:49 -0600, Mark Hatle wrote:
> On 11/14/13, 7:42 AM, Phil Blundell wrote:
> > On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
> >> Also, is "-" actually valid?  I seem to recall having problems when my build
> >> tree had the hyphen ("-") in the path.
> >
> > Paths with "-" in certainly work for me.  If there are any recipes which
> > break in that situation then we should just fix them.
> 
> We found an issue with paths that -start- with a '-', there are apparently many 
> places where paths are passed into various shell, and the initial '-' can be 
> read as an argument identifier.

Right, I can imagine that this would break fairly badly.  But if we're
going to require TMPDIR to be an absolute path (which seems perfectly
reasonable) then by definition it can't start with - so this will
presumably become a non-issue.

> I don't believe Qi Chen sent it to the list, but we built path names with all of 
> the special characters and tried to run builds.  The specific list is based on 
> the results of those tests.  The other items are already broken, and we're 
> trying to be explicit with this for end users.  (Note, it's really the TMPDIR 
> that matters.. the 'build' directory is a lot more flexible.)

Yeah, I saw the results of those tests.  My slight reservation with that
methodology is that, if any single recipe fails with a given character
in TMPDIR, no matter how obscure that recipe might be, it seems that
this is enough to make that character verboten.  For example, the
rationale for excluding "+" seems to be that it breaks tcl; if this is
the only recipe that fails then it seems like it might be worth at least
trying to fix tcl rather than excluding "+" from TMPDIRs for ever more.

But, clearly, there are some characters in the list that are "just
silly" to have in pathnames.  I don't think anybody would be surprised
to learn that "*" and "$" cause problems and blacklisting those ones
does seem quite reasonable.

p.

Re: [PATCH 1/1] sanity.bbclass: check for validity of TMPDIR

[Saul Wold]

On 11/14/2013 07:49 AM, Mark Hatle wrote:
> On 11/14/13, 7:42 AM, Phil Blundell wrote:
>> On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
>>> Also, is "-" actually valid?  I seem to recall having problems when
>>> my build
>>> tree had the hyphen ("-") in the path.
>>
>> Paths with "-" in certainly work for me.  If there are any recipes which
>> break in that situation then we should just fix them.
>
> We found an issue with paths that -start- with a '-', there are
> apparently many places where paths are passed into various shell, and
> the initial '-' can be read as an argument identifier.
>

I think that I saw an email talking about "-D" in the TMPDIR caused some 
some recipes to pick that up as a CFLAG define and cause problems

Sau!

>> OE already has quite a range of baroque restrictions on what sort of
>> TMPDIR you are allowed to use (no nfs, no symlinks in the path, no
>> spaces in the name) and every new prohibition represents a loss in
>> usability.  This patch as proposed already forbids a whole range of
>> characters, including things like "+", and I think that disallowing "-"
>> as well would be a step too far.
>
> I don't believe Qi Chen sent it to the list, but we built path names
> with all of the special characters and tried to run builds.  The
> specific list is based on the results of those tests.  The other items
> are already broken, and we're trying to be explicit with this for end
> users.  (Note, it's really the TMPDIR that matters.. the 'build'
> directory is a lot more flexible.)
>
> --Mark
>
>> p.
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>

Re: [PATCH 1/1] sanity.bbclass: check for validity of TMPDIR

[Mark Hatle]

On 11/14/13, 10:04 AM, Phil Blundell wrote:
> On Thu, 2013-11-14 at 09:49 -0600, Mark Hatle wrote:
>> On 11/14/13, 7:42 AM, Phil Blundell wrote:
>>> On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
>>>> Also, is "-" actually valid?  I seem to recall having problems when my build
>>>> tree had the hyphen ("-") in the path.
>>>
>>> Paths with "-" in certainly work for me.  If there are any recipes which
>>> break in that situation then we should just fix them.
>>
>> We found an issue with paths that -start- with a '-', there are apparently many
>> places where paths are passed into various shell, and the initial '-' can be
>> read as an argument identifier.
>
> Right, I can imagine that this would break fairly badly.  But if we're
> going to require TMPDIR to be an absolute path (which seems perfectly
> reasonable) then by definition it can't start with - so this will
> presumably become a non-issue.

Except there are a number of packages that try to parse pieces of strings 
derived from TMPDIR.

Set your TMPDIR path to include "-Dmypath" some time.  SVN will blow up.. :P

>> I don't believe Qi Chen sent it to the list, but we built path names with all of
>> the special characters and tried to run builds.  The specific list is based on
>> the results of those tests.  The other items are already broken, and we're
>> trying to be explicit with this for end users.  (Note, it's really the TMPDIR
>> that matters.. the 'build' directory is a lot more flexible.)
>
> Yeah, I saw the results of those tests.  My slight reservation with that
> methodology is that, if any single recipe fails with a given character
> in TMPDIR, no matter how obscure that recipe might be, it seems that
> this is enough to make that character verboten.  For example, the
> rationale for excluding "+" seems to be that it breaks tcl; if this is
> the only recipe that fails then it seems like it might be worth at least
> trying to fix tcl rather than excluding "+" from TMPDIRs for ever more.

I agree, in the case above, the SVN example I've asked our engineers to look 
into trying to fix that case.  But what they're finding is that once you fix one 
case, then the next breaks.. and the next breaks.

> But, clearly, there are some characters in the list that are "just
> silly" to have in pathnames.  I don't think anybody would be surprised
> to learn that "*" and "$" cause problems and blacklisting those ones
> does seem quite reasonable.

:)

I'd like to see us blacklist the bunch, and then turn around and start opening 
the set backup.  But I don't know how practical this actually is -- unless we 
can get automated test machines running each odd-ball combination on a nightly 
basis.

--Mark

> p.
>
>

Re: [PATCH 1/1] sanity.bbclass: check for validity of TMPDIR

[ChenQi]

On 11/14/2013 09:12 PM, Richard Purdie wrote:
> On Thu, 2013-11-14 at 21:07 +0800, Qi.Chen@windriver.com wrote:
>> From: Chen Qi <Qi.Chen@windriver.com>
>>
>> TMPDIR must be an absolute path, otherwise, the build will fail.
>> Special characters in TMPDIR will also cause build failures.
>>
>> This patch enables checking for the validity of TMPDIR.
>>
>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>> ---
>>   meta/classes/sanity.bbclass |    7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
>> index 83378b0..e45906e 100644
>> --- a/meta/classes/sanity.bbclass
>> +++ b/meta/classes/sanity.bbclass
>> @@ -672,6 +672,13 @@ def check_sanity_everybuild(status, d):
>>           with open(checkfile, "w") as f:
>>               f.write(tmpdir)
>>   
>> +    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
>> +    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
>> +    import re
>> +    valid_pattern = re.compile(valid_tmpdir_regexp)
>> +    if not valid_pattern.match(tmpdir):
>> +        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
>> +
> I'm ok with the check itself however this does not need to be run every
> build. It needs to run when tmpdir changes only.
>
> Cheers,
>
> Richard
>
>
>
>

Thanks for pointing it out.

I've changed the logic a little bit.
1. Create the directory hierarchy for TMPDIR only if TMPDIR is valid
2. Handle ABI version change only if TMPDIR has been created.

So only when we had a valid TMPDIR for last build and this time the 
TMPDIR hasn't changed location, we don't check the validity for TMPDIR.

I've sent out  a V2.

Best Regards,
Chen Qi

Re: [PATCH 0/1] sanity.bbclass: check for validity of TMPDIR

[ChenQi]

Hi All,

This cover letter contains some testing records.

Best Regards,
Chen Qi

On 11/14/2013 09:07 PM, Qi.Chen@windriver.com wrote:
> From: Chen Qi <Qi.Chen@windriver.com>
>
> This patch checks the validity of TMPDIR.
>
> The TMPDIR must match the following regexp.
> ^/[a-zA-Z0-9\-_/.~]+$
>
> Detailed tests about why I came up with this regexp are displayed at the end of this cover letter.
>
> A major reason for why TMPDIR cannot support many special chars such as @ and # is that these
> special chars are used by the sed command as its field separator in our recipes.
>
> And here are things I want to make sure of.
> 1. For now, TMPDIR must be an absolute path. So do we need to make it support relative path?
> 2. I included the tilde (~) into the valid char set because `bitbake world' succeeded. But
>     it's still possible the the tilde (~) will serve as the field separator in sed. So is it
>     OK that I include it into the valid char set?
>
>
> **********************  TESTING RECORDS  ****************************************
> Test Case 1: use `bitbake minimal' on paths containing special chars
> Here's the output.
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/`_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained"
> RESULT: SUCCESS
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/~_contained"
> RESULT: SUCCESS
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/<_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/=_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/>_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/|_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/ _contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/,_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/;_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/:_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/!_contained"
> RESULT: SUCCESS
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/?_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/'_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/\"_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/(_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/)_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained"
> RESULT: SUCCESS
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/{_contained"
> RESULT: SUCCESS
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/}_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/@_contained"
> RESULT: SUCCESS
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/$_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/*_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/\_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/&_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/#_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/%_contained"
> RESULT: FAIL
> ===================
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained"
> RESULT: SUCCESS
>
>
> Test Case 2: TMPDIR containing ^
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained"
> Command:
>      bitbake core-image-minimal -cpopulate_sdk
> Result:
>      Fail
> Formatted Error Message:
>      ERROR: Task 5 (virtual:nativesdk:/home/chenqi/poky/meta/recipes-core/eglibc/eglibc_2.18.bb, do_populate_sysroot) failed with exit code '1'
>      | checking whether the C compiler works... no
>      | configure: error: in `/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained/work/i686-nativesdk-pokysdk-linux/nativesdk-eglibc/2.18-r0/site_config_qemux86':
>      | configure: error: C compiler cannot create executables
>      | See `config.log' for more details
>
>
> Test Case 3: TMPDIR containing +
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained"
> Command:
>      bitbake world
> Result:
>      Fail
> Formatted Error Message:
>      ERROR: Task 9084 (virtual:native:/home/chenqi/poky/meta/recipes-devtools/tcltk/tcl_8.6.1.bb, do_install) failed with exit code '1'
>      | oe_libinstall: cd /home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained/work/i686-linux/tcl-native/8.6.1-r0/build
>      | sed: -e expression #1, char 117: unknown option to `s'
> Reason:
>      '+' is used by the sed command as a separator in tcl recipe.
>      sed -i "s+${WORKDIR}+${STAGING_INCDIR}+g" tclConfig.sh
>
>
> Test Case 4: TMPDIR containing ]
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained"
> Command:
>      bitbake world
> Result:
>      Fail
> Formatted Error Message:
>      ERROR: Task 5 (virtual:native:/home/chenqi/poky/meta/recipes-graphics/xcb/xcb-proto_1.8.bb, do_configure) failed with exit code '1'
>      | configure.ac:9: error: possibly undefined macro: AM_INIT_AUTOMAKE
>      |       If this token and others are legitimate, please use m4_pattern_allow.
>      |       See the Autoconf documentation.
>      | configure.ac:12: error: possibly undefined macro: AM_CONDITIONAL
>      | configure.ac:17: error: possibly undefined macro: AM_PATH_PYTHON
>      | autoreconf: /home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained/sysroots/i686-linux/usr/bin/autoconf failed with exit status: 1
>      | ERROR: autoreconf execution failed.
>      | WARNING: exit code 1 from a shell command.
>
>
> Test Case 5: TMPDIR containing {
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained"
> Command:
>      bitbake world
> Result:
>      Fail
> Formatted Error Message:
>      ERROR: Task 9237 (virtual:native:/home/chenqi/poky/meta/recipes-gnome/gnome/gnome-doc-utils_0.20.10.bb, do_compile) failed with exit code '1'
>      | xsltproc -nonet -o "C/db-label.xml" \
>      |      --stringparam basename "db-label" \
>      |      --stringparam xsl_file "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/{_contained/work/i686-linux/gnome-doc-utils-native/0.20.10-r0/gnome-doc-utils-0.20.10/doc/xslt/../../xslt/\
> docbook/common/db-label.xsl" \
>      |      "./xsldoc-docbook.xsl" "C/db-label.xsldoc"
>      | db-chunk: No stylesheet param for documented db.chunk.chunks
>      | db-common: No named template for documented db.copyright
>      | make[2]: *** [C/db-chunk.xml] Error 10
>
>
> Test Case 6: TMPDIR containing @
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/@_contained"
> Command:
>      bitbake qt-mobility-x11
> Result:
>      Fail
> Formatted Error Message:
>      ERROR: Task 2 (/home/chenqi/poky/meta/recipes-qt/qt4/qt-mobility-x11_1.2.0.bb, do_install) failed with exit code '1'
>      | DEBUG: Executing shell function do_install
>      | sed: -e expression #1, char 75: unknown option to `s'
>
>
> Test Case 7: TMPDIR containing ~
> TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained"
> Command:
>      bitbake world
> Result:
>     Success
>
> ****************************************************************************************
>
> The following changes since commit ea92671d9823e3667d6ced7ac2af20f991da404d:
>
>    bitbake: cooker: replace "w" file opening mode with "a" mode (2013-11-12 17:01:37 +0000)
>
> are available in the git repository at:
>
>    git://git.pokylinux.org/poky-contrib ChenQi/TMPDIR
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/TMPDIR
>
> Chen Qi (1):
>    sanity.bbclass: check for validity of TMPDIR
>
>   meta/classes/sanity.bbclass |    7 +++++++
>   1 file changed, 7 insertions(+)
>

Re: [PATCH 1/1] sanity.bbclass: check for validity of TMPDIR

[ChenQi]

On 11/15/2013 02:01 AM, Saul Wold wrote:
> On 11/14/2013 07:49 AM, Mark Hatle wrote:
>> On 11/14/13, 7:42 AM, Phil Blundell wrote:
>>> On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
>>>> Also, is "-" actually valid?  I seem to recall having problems when
>>>> my build
>>>> tree had the hyphen ("-") in the path.
>>>
>>> Paths with "-" in certainly work for me.  If there are any recipes 
>>> which
>>> break in that situation then we should just fix them.
>>
>> We found an issue with paths that -start- with a '-', there are
>> apparently many places where paths are passed into various shell, and
>> the initial '-' can be read as an argument identifier.
>>
>
> I think that I saw an email talking about "-D" in the TMPDIR caused 
> some some recipes to pick that up as a CFLAG define and cause problems
>
> Sau!
>

Yes. It's actually caused by the following statement in subversion-native.
./build/ac-macros/neon.m4: SVN_NEON_INCLUDES=[`$PKG_CONFIG neon --cflags 
| $SED -e 's/-D[^ ]*//g'`]
./build/ac-macros/neon.m4: SVN_NEON_INCLUDES=[`$neon_config --cflags | 
$SED -e 's/-D[^ ]*//g'`]

And there's a bug filed for it. 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5458

Best Regards,
Chen Qi

>>> OE already has quite a range of baroque restrictions on what sort of
>>> TMPDIR you are allowed to use (no nfs, no symlinks in the path, no
>>> spaces in the name) and every new prohibition represents a loss in
>>> usability.  This patch as proposed already forbids a whole range of
>>> characters, including things like "+", and I think that disallowing "-"
>>> as well would be a step too far.
>>
>> I don't believe Qi Chen sent it to the list, but we built path names
>> with all of the special characters and tried to run builds.  The
>> specific list is based on the results of those tests.  The other items
>> are already broken, and we're trying to be explicit with this for end
>> users.  (Note, it's really the TMPDIR that matters.. the 'build'
>> directory is a lot more flexible.)
>>
>> --Mark
>>
>>> p.
>>>
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>