[Buildroot] Advice needed: Problem with clashing make variables in pkg-generic.mk when building perl-xml-sax package.

[Buildroot] Advice needed: Problem with clashing make variables in pkg-generic.mk when building perl-xml-sax package.

[Barbarien, Joeri]

Hello all,

When building perl-xml-sax (and its dependency perl-xml-sax-base) I noticed something strange in output/build/packages-file-list.txt:

---> perl-xml-sax-0.99,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/Base.pm
---> perl-xml-sax-0.99,./usr/lib/perl5/site_perl/5.24.1/arm-linux/auto/X
---> ML/SAX/Base/.packlist
---> perl-xml-sax-0.99,./usr/share/man/man3/XML::SAX::Base.3
---> perl-xml-sax-0.99,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/BuildSAX
---> Base.pl 
---> perl-xml-sax-0.99,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/Exceptio
---> n.pm
---> perl-xml-sax-0.99,./usr/share/man/man3/XML::SAX::Exception.3
---> perl-xml-sax-0.99,./usr/share/man/man3/XML::SAX::BuildSAXBase.3
perl-xml-sax,./usr/share/man/man3/XML::SAX::DocumentLocator.3
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/arm-linux/auto/XML/SAX/.packlist
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/Reader/UnicodeExt.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/NoUnicodeExt.pm
perl-xml-sax,./usr/share/man/man3/XML::SAX::ParserFactory.3
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/Reader/URI.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/EncodingDetect.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/DocumentLocator.pm
perl-xml-sax,./usr/share/man/man3/XML::SAX::PurePerl::Reader.3
perl-xml-sax,./usr/share/man/man3/XML::SAX::PurePerl.3
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/DTDDecls.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/Productions.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/Reader.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/Exception.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/XMLDecl.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/Reader/Stream.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/Intro.pod
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/UnicodeExt.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/Reader/String.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl.pm
perl-xml-sax,./usr/share/man/man3/XML::SAX::Intro.3
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/ParserFactory.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/Reader/NoUnicodeExt.pm
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/DebugHandler.pm
perl-xml-sax,./usr/share/man/man3/XML::SAX.3
perl-xml-sax,./usr/lib/perl5/site_perl/5.24.1/XML/SAX/PurePerl/DocType.pm

The files that are listed for perl-xml-sax-0.99 are actually those built by perl-xml-sax-base

This is due to a clash of make variables in package/pkg-generic.mk:

$(2)_BASE_NAME  = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
$(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
$(2)_DL_DIR =  $$(DL_DIR)
$(2)_DIR  =  $$(BUILD_DIR)/$$($(2)_BASE_NAME)

Due to the unfortunate naming of the packages,

$($(2)_BASE_NAME) for perl-xml-sax clashes with as $($(2)_NAME) for perl-xml-sax-base, leading to this wrong behavior.

I made a quick fix by renaming the _BASE_NAME suffix to _BR_BASE_NAME but perhaps a more comprehensive / cleaner solution is preferred.

Could you please advise on how this should be "properly" fixed? Some fast ideas:

1) Make sure common part of suffixes is in the front, e.g. _NAME, _NAME_BASE, _NAME_BASE_RAW
2) Use lowercase for internal suffixes so they cannot clash with by construction uppercase $(PKG)
3) A check on allowed package names ensuring clashes cannot happen

Kind regards,

Joeri Barbarien

[Buildroot] Advice needed: Problem with clashing make variables in pkg-generic.mk when building perl-xml-sax package.

[Arnout Vandecappelle]

On 23-10-17 16:41, Barbarien, Joeri (Nokia - BE/Antwerp) wrote:
> Hello all,
> 
> When building perl-xml-sax (and its dependency perl-xml-sax-base) I noticed something strange in output/build/packages-file-list.txt:
[snip]
> The files that are listed for perl-xml-sax-0.99 are actually those built by perl-xml-sax-base
> 
> This is due to a clash of make variables in package/pkg-generic.mk:
> 
> $(2)_BASE_NAME  = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
> $(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
> $(2)_DL_DIR =  $$(DL_DIR)
> $(2)_DIR  =  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
> 
> Due to the unfortunate naming of the packages,
> 
> $($(2)_BASE_NAME) for perl-xml-sax clashes with as $($(2)_NAME) for perl-xml-sax-base, leading to this wrong behavior.

 Wow, good catch! Incredible that this works at all...

> 
> I made a quick fix by renaming the _BASE_NAME suffix to _BR_BASE_NAME but perhaps a more comprehensive / cleaner solution is preferred.
> 
> Could you please advise on how this should be "properly" fixed? Some fast ideas:
> 
> 1) Make sure common part of suffixes is in the front, e.g. _NAME, _NAME_BASE, _NAME_BASE_RAW

 I guess that this reduces the risk, but there is still a possibility of
conflicts, no?

> 2) Use lowercase for internal suffixes so they cannot clash with by construction uppercase $(PKG)

 That would work. Or the reverse, that $(PKG) is lower case. But that is a
*huge* change to Buildroot, with changes to every package. It is not going to
happen. Note that for the complete solution, it is not only for internal
suffixes, but also for some that are set outside the infrastructure. E.g. the
_OVERRIDE_SRCDIR of package foo conflicts with the _SRCDIR of package foo-override.

> 3) A check on allowed package names ensuring clashes cannot happen

 That would be really good, but complicated because there are a lot of variables
to check. And when new variables are introduced, we have to remember to add them
to the list. Also it doesn't solve the problem with perl-xml-sax, we'd still
have to e.g. rename the perl-xml-sax-base package.

 I think that your first idea is probably the best one.

 I've done a quick check: there are 142 variables with 2 underscores used in all
the package infras (only the ones with two underscores run this risk). You can
get these with

grep -h -o ')\(_[A-Z0-9]*\)\{2,10\}' package/pkg*.mk | sort -u

 I got the following to find conflicts, but I'm not sure it is complete:

for i in $(grep -h -o ')\(_[A-Z0-9]*\)\{2,10\}' package/pkg*.mk | \
           sed s/.*_// | \
           sort -u); do \
  grep -q ")_$i\>" package/pkg*.mk && echo "$i"; \
done

 That comes up with the following potential conflicts:

_DEPENDENCIES with _FINAL_DEPENDENCIES, _FINAL_ALL_DEPENDENCIES,
_FINAL_PATCH_DEPENDENCIES

_DIR with _DEPS_DIR, _DL_DIR, _REBAR_DEPS_DIR, _REDIST_SOURCES_DIR

_ENV with _AUTORECONF_ENV, _BASE_ENV, _CONFIGURATOR_MAKE_ENV, _CONF_ENV,
_MAKE_ENV, and _REBAR_ENV. Impressive that we didn't hit this one yet :-)

_MAKE with _KCONFIG_MAKE

_NAME with _BASE_NAME and _RAW_BASE_NAME

_PATCH with _TARGET_PATH and _LIBTOOL_PATCH

_REBAR with _USE_BUNDLED_REBAR

_SITE with _ACTUAL_SOURCE_SITE

_SOURCE with _TARGET_ACTUAL_SOURCE and _TARGET_SOURCE

_SRCDIR with _OVERRIDE_SRCDIR

_TYPE with _SETUP_TYPE

_VERSION with _DL_VERSION

_WAF with _NEEDS_EXTERNAL_WAF


 Quite an impressive list...


 I really don't know how we're going to tackle this... Does anyone have an
innovative idea?

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Advice needed: Problem with clashing make variables in pkg-generic.mk when building perl-xml-sax package.

[Yann E. MORIN]

Joeri, Arnout, All,

On 2017-10-23 23:05 +0200, Arnout Vandecappelle spake thusly:
> On 23-10-17 16:41, Barbarien, Joeri (Nokia - BE/Antwerp) wrote:
> > Hello all,
> > 
> > When building perl-xml-sax (and its dependency perl-xml-sax-base) I noticed something strange in output/build/packages-file-list.txt:
> [snip]
> > The files that are listed for perl-xml-sax-0.99 are actually those built by perl-xml-sax-base
> > 
> > This is due to a clash of make variables in package/pkg-generic.mk:
> > 
> > $(2)_BASE_NAME  = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
> > $(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
> > $(2)_DL_DIR =  $$(DL_DIR)
> > $(2)_DIR  =  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
> > 
> > Due to the unfortunate naming of the packages,
> > 
> > $($(2)_BASE_NAME) for perl-xml-sax clashes with as $($(2)_NAME) for perl-xml-sax-base, leading to this wrong behavior.
> 
>  Wow, good catch! Incredible that this works at all...

Yes, this is a very good catch... Damned... :-/

I am basically in line with the analysis by Arnout: any solution we
decide on will be a huge work.

I think we really need to sleep on it for a little while, because we
definitely must be very cafreful that a solution:

  - does not introduces regressions

  - can be applied in as systematice a way as possible

  - does not need a second round to fix a fallout.

I think I will wait coming back from ELCE before I have a deep look at
this, though.

Notice that we have a similar problem for host packages: a target package
that is named foo and has a host variant could be confused with a target
package that is named host-foo, and then we'd have a clash in the
variable names as well...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Advice needed: Problem with clashing make variables in pkg-generic.mk when building perl-xml-sax package.

[Yann E. MORIN]

Joeri, Arnout, All,

On 2017-10-23 23:05 +0200, Arnout Vandecappelle spake thusly:
> On 23-10-17 16:41, Barbarien, Joeri (Nokia - BE/Antwerp) wrote:
> > Hello all,
> > 
> > When building perl-xml-sax (and its dependency perl-xml-sax-base) I noticed something strange in output/build/packages-file-list.txt:
> [snip]
> > The files that are listed for perl-xml-sax-0.99 are actually those built by perl-xml-sax-base
> > 
> > This is due to a clash of make variables in package/pkg-generic.mk:
> > 
> > $(2)_BASE_NAME  = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
> > $(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
> > $(2)_DL_DIR =  $$(DL_DIR)
> > $(2)_DIR  =  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
> > 
> > Due to the unfortunate naming of the packages,
> > 
> > $($(2)_BASE_NAME) for perl-xml-sax clashes with as $($(2)_NAME) for perl-xml-sax-base, leading to this wrong behavior.
> > 
> > I made a quick fix by renaming the _BASE_NAME suffix to _BR_BASE_NAME but perhaps a more comprehensive / cleaner solution is preferred.
> > 
> > Could you please advise on how this should be "properly" fixed? Some fast ideas:
> > 
> > 1) Make sure common part of suffixes is in the front, e.g. _NAME, _NAME_BASE, _NAME_BASE_RAW
> 
>  I guess that this reduces the risk, but there is still a possibility of
> conflicts, no?

I don;t think we can ensure we'd have no more clash, no. This is not a
viable solution long-term.

> > 2) Use lowercase for internal suffixes so they cannot clash with by construction uppercase $(PKG)
> 
>  That would work. Or the reverse, that $(PKG) is lower case. But that is a
> *huge* change to Buildroot, with changes to every package. It is not going to
> happen. Note that for the complete solution, it is not only for internal
> suffixes, but also for some that are set outside the infrastructure. E.g. the
> _OVERRIDE_SRCDIR of package foo conflicts with the _SRCDIR of package foo-override.
> 
> > 3) A check on allowed package names ensuring clashes cannot happen
>  That would be really good, but complicated because there are a lot of variables
> to check. And when new variables are introduced, we have to remember to add them
> to the list. Also it doesn't solve the problem with perl-xml-sax, we'd still
> have to e.g. rename the perl-xml-sax-base package.
> 
>  I think that your first idea is probably the best one.

Alternatively, we can change the way we serparate the "namespace", by
using something else than an underscore.

For example, we could be using a dot '.' to separate the package-side
on the LHS, vs. the Buildroot side on the RHS., like so:

    $(2).BASE_NAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))

But that is not very nice when we tell the user to reuse those
variables, like $(2)_PKGDIR for example, which would have to become
$(2).PKGDIR for consistency.

Regards,
Yann E. MORIN.

>  I've done a quick check: there are 142 variables with 2 underscores used in all
> the package infras (only the ones with two underscores run this risk). You can
> get these with
> 
> grep -h -o ')\(_[A-Z0-9]*\)\{2,10\}' package/pkg*.mk | sort -u
> 
>  I got the following to find conflicts, but I'm not sure it is complete:
> 
> for i in $(grep -h -o ')\(_[A-Z0-9]*\)\{2,10\}' package/pkg*.mk | \
>            sed s/.*_// | \
>            sort -u); do \
>   grep -q ")_$i\>" package/pkg*.mk && echo "$i"; \
> done
> 
>  That comes up with the following potential conflicts:
> 
> _DEPENDENCIES with _FINAL_DEPENDENCIES, _FINAL_ALL_DEPENDENCIES,
> _FINAL_PATCH_DEPENDENCIES
> 
> _DIR with _DEPS_DIR, _DL_DIR, _REBAR_DEPS_DIR, _REDIST_SOURCES_DIR
> 
> _ENV with _AUTORECONF_ENV, _BASE_ENV, _CONFIGURATOR_MAKE_ENV, _CONF_ENV,
> _MAKE_ENV, and _REBAR_ENV. Impressive that we didn't hit this one yet :-)
> 
> _MAKE with _KCONFIG_MAKE
> 
> _NAME with _BASE_NAME and _RAW_BASE_NAME
> 
> _PATCH with _TARGET_PATH and _LIBTOOL_PATCH
> 
> _REBAR with _USE_BUNDLED_REBAR
> 
> _SITE with _ACTUAL_SOURCE_SITE
> 
> _SOURCE with _TARGET_ACTUAL_SOURCE and _TARGET_SOURCE
> 
> _SRCDIR with _OVERRIDE_SRCDIR
> 
> _TYPE with _SETUP_TYPE
> 
> _VERSION with _DL_VERSION
> 
> _WAF with _NEEDS_EXTERNAL_WAF
> 
> 
>  Quite an impressive list...
> 
> 
>  I really don't know how we're going to tackle this... Does anyone have an
> innovative idea?
> 
>  Regards,
>  Arnout
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'