[PATCH] openssl: add a vardeps for configure on libdir

[PATCH] openssl: add a vardeps for configure on libdir

[Mark Asselstine]

Configure makes use of 'libdir' to set 'openssldir'. Unfortunately
openssl treats this as a hardcoded absolute path. The value passed in
as openssl becomes part of header files and baked in to executables.

For target builds libdir is not something that changes and for -native
builds unless you move your builds around you will see no issue with
the current implementation. If you do update libdir or if you move
your build or reuse sstate by copying it to a new build you will start
to see errors. These will be seen when you use the 'openssl' tool and
look like the following:

WARNING: can't open config file: /home/build/bitbake_build/tmp/
                        sysroots/x86_64-linux/usr/lib/ssl/openssl.cnf

Knowing that openssl treats 'openssldir' as a hard coded absolute path
and that we set this in configure we must add a 'vardeps' to ensure
that things get re-configured and built whenever 'libdir' is
modified. This will fix both target and -native builds per the
usecases described above. If 'libdir' is unchanged things continue to
function as they did before and a re-configure will not happen.

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
---
 meta/recipes-connectivity/openssl/openssl.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-connectivity/openssl/openssl.inc b/meta/recipes-connectivity/openssl/openssl.inc
index d337017..105b3fa 100644
--- a/meta/recipes-connectivity/openssl/openssl.inc
+++ b/meta/recipes-connectivity/openssl/openssl.inc
@@ -54,6 +54,8 @@ do_configure_prepend_darwin () {
 	sed -i -e '/version-script=openssl\.ld/d' Configure
 }
 
+do_configure[vardeps] += "${libdir}"
+
 do_configure () {
 	cd util
 	perl perlpath.pl ${STAGING_BINDIR_NATIVE}
-- 
2.0.1

Re: [PATCH] openssl: add a vardeps for configure on libdir

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2412 bytes --]

On Wed, Mar 25, 2015 at 9:28 AM, Mark Asselstine <
mark.asselstine@windriver.com> wrote:

> Configure makes use of 'libdir' to set 'openssldir'. Unfortunately
> openssl treats this as a hardcoded absolute path. The value passed in
> as openssl becomes part of header files and baked in to executables.
>
> For target builds libdir is not something that changes and for -native
> builds unless you move your builds around you will see no issue with
> the current implementation. If you do update libdir or if you move
> your build or reuse sstate by copying it to a new build you will start
> to see errors. These will be seen when you use the 'openssl' tool and
> look like the following:
>
> WARNING: can't open config file: /home/build/bitbake_build/tmp/
>                         sysroots/x86_64-linux/usr/lib/ssl/openssl.cnf
>
> Knowing that openssl treats 'openssldir' as a hard coded absolute path
> and that we set this in configure we must add a 'vardeps' to ensure
> that things get re-configured and built whenever 'libdir' is
> modified. This will fix both target and -native builds per the
> usecases described above. If 'libdir' is unchanged things continue to
> function as they did before and a re-configure will not happen.
>
> Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
> ---
>  meta/recipes-connectivity/openssl/openssl.inc | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/meta/recipes-connectivity/openssl/openssl.inc
> b/meta/recipes-connectivity/openssl/openssl.inc
> index d337017..105b3fa 100644
> --- a/meta/recipes-connectivity/openssl/openssl.inc
> +++ b/meta/recipes-connectivity/openssl/openssl.inc
> @@ -54,6 +54,8 @@ do_configure_prepend_darwin () {
>         sed -i -e '/version-script=openssl\.ld/d' Configure
>  }
>
> +do_configure[vardeps] += "${libdir}"
>

This really doesn't make sense. First of all, vardeps is a list of
variables, but the expanded libdir is not a variable name. Perhaps you
meant += "libdir"? Second, if libdir is used in do_configure, then changing
it will already cause do_configure to be re-run, as bitbake tracks variable
references. Only references which can't be tracked by bitbake need to be
listed explicitly.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3105 bytes --]

Re: [PATCH] openssl: add a vardeps for configure on libdir

[Mark Asselstine]

On March 25, 2015 10:34:00 Christopher Larson wrote:
> On Wed, Mar 25, 2015 at 9:28 AM, Mark Asselstine <
> 
> mark.asselstine@windriver.com> wrote:
> > Configure makes use of 'libdir' to set 'openssldir'. Unfortunately
> > openssl treats this as a hardcoded absolute path. The value passed in
> > as openssl becomes part of header files and baked in to executables.
> > 
> > For target builds libdir is not something that changes and for -native
> > builds unless you move your builds around you will see no issue with
> > the current implementation. If you do update libdir or if you move
> > your build or reuse sstate by copying it to a new build you will start
> > to see errors. These will be seen when you use the 'openssl' tool and
> > look like the following:
> > 
> > WARNING: can't open config file: /home/build/bitbake_build/tmp/
> > 
> >                         sysroots/x86_64-linux/usr/lib/ssl/openssl.cnf
> > 
> > Knowing that openssl treats 'openssldir' as a hard coded absolute path
> > and that we set this in configure we must add a 'vardeps' to ensure
> > that things get re-configured and built whenever 'libdir' is
> > modified. This will fix both target and -native builds per the
> > usecases described above. If 'libdir' is unchanged things continue to
> > function as they did before and a re-configure will not happen.
> > 
> > Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
> > ---
> > 
> >  meta/recipes-connectivity/openssl/openssl.inc | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/meta/recipes-connectivity/openssl/openssl.inc
> > b/meta/recipes-connectivity/openssl/openssl.inc
> > index d337017..105b3fa 100644
> > --- a/meta/recipes-connectivity/openssl/openssl.inc
> > +++ b/meta/recipes-connectivity/openssl/openssl.inc
> > @@ -54,6 +54,8 @@ do_configure_prepend_darwin () {
> > 
> >         sed -i -e '/version-script=openssl\.ld/d' Configure
> >  
> >  }
> > 
> > +do_configure[vardeps] += "${libdir}"
> 
> This really doesn't make sense. First of all, vardeps is a list of
> variables, but the expanded libdir is not a variable name. Perhaps you
> meant += "libdir"? 

My first attempt I had tried "libdir" and it doesn't seem to have the needed 
effect. I was hesitant to use other variables (such as those that libdir is 
derived from) as that would fork how -native and target builds would be 
handled. I had found other instances (meta/classes/package.bbclass) that 
seemed to use vardeps on expanded variables. At any rate I will circle back, 
your comment was not entirely unexpected, just the push that is needed for me 
to look more closely. I will follow up in a bit.

> Second, if libdir is used in do_configure, then changing
> it will already cause do_configure to be re-run, as bitbake tracks variable
> references. Only references which can't be tracked by bitbake need to be
> listed explicitly.

The observed behavior definitely supports the view that this is a case where 
it can't be tracked. It is definitely a different usage than you would find 
normally.

do_configure () {
...
    perl ./Configure ${EXTRA_OECONF} shared --prefix=$useprefix \
       --openssldir=${libdir}/ssl --libdir=`basename ${libdir}` $target
}

Mark


> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics

Re: [PATCH] openssl: add a vardeps for configure on libdir

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1865 bytes --]

On Wed, Mar 25, 2015 at 12:28 PM, Mark Asselstine <
mark.asselstine@windriver.com> wrote:

> My first attempt I had tried "libdir" and it doesn't seem to have the
> needed
> effect. I was hesitant to use other variables (such as those that libdir is
> derived from) as that would fork how -native and target builds would be
> handled. I had found other instances (meta/classes/package.bbclass) that
> seemed to use vardeps on expanded variables. At any rate I will circle
> back,
> your comment was not entirely unexpected, just the push that is needed for
> me
> to look more closely. I will follow up in a bit.
>

package.bbclass has variables containing / functions returning lists of
variable names. libdir isn't a list of variable names, it's a path.

> Second, if libdir is used in do_configure, then changing
> > it will already cause do_configure to be re-run, as bitbake tracks
> variable
> > references. Only references which can't be tracked by bitbake need to be
> > listed explicitly.
>
> The observed behavior definitely supports the view that this is a case
> where
> it can't be tracked. It is definitely a different usage than you would find
> normally.
>
> do_configure () {
> ...
>     perl ./Configure ${EXTRA_OECONF} shared --prefix=$useprefix \
>        --openssldir=${libdir}/ssl --libdir=`basename ${libdir}` $target
> }
>


Nope, this is exactly what bitbake knows how to track, a standard variable
reference; ${libdir}. It knows that do_configure uses libdir already. It's
only when the variable name is determined programmatically at runtime in
python that bitbake isn't able to track the reference (e.g. d.getVar(foo +
"bar", True)).
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2578 bytes --]

Re: [PATCH] openssl: add a vardeps for configure on libdir

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1411 bytes --]

On 25 March 2015 at 16:28, Mark Asselstine <mark.asselstine@windriver.com>
wrote:

> For target builds libdir is not something that changes and for -native
> builds unless you move your builds around you will see no issue with
> the current implementation. If you do update libdir or if you move
> your build or reuse sstate by copying it to a new build you will start
> to see errors. These will be seen when you use the 'openssl' tool and
> look like the following:
>
> WARNING: can't open config file: /home/build/bitbake_build/tmp/
>                         sysroots/x86_64-linux/usr/lib/ssl/openssl.cnf
>

We don't generally require that native sstate is rebuilt when the build
directory changes, as otherwise you wouldn't be able to share native sstate
between builds and users.  Instead, paths are munged where possible when
using sstate and wrapper scripts used to "fix" binaries.  Indeed
openssl.inc does this:

do_install_append_virtclass-native() {
        create_wrapper ${D}${bindir}/openssl \
            OPENSSL_CONF=${libdir}/ssl/openssl.cnf \
            SSL_CERT_DIR=${libdir}/ssl/certs \
            SSL_CERT_FILE=${libdir}/ssl/cert.pem \
            OPENSSL_ENGINES=${libdir}/ssl/engines
}

Is it possible that the binary isn't respecting the OPENSSL_CONF
environment variable since that wrapper was written? (or the wrapper wasn't
tested well enough)

Ross

[-- Attachment #2: Type: text/html, Size: 2330 bytes --]

Re: [PATCH] openssl: add a vardeps for configure on libdir

[Mark Asselstine]

On March 25, 2015 13:35:46 Christopher Larson wrote:
> On Wed, Mar 25, 2015 at 12:28 PM, Mark Asselstine <
> 
> mark.asselstine@windriver.com> wrote:
> > My first attempt I had tried "libdir" and it doesn't seem to have the
> > needed
> > effect. I was hesitant to use other variables (such as those that libdir
> > is
> > derived from) as that would fork how -native and target builds would be
> > handled. I had found other instances (meta/classes/package.bbclass) that
> > seemed to use vardeps on expanded variables. At any rate I will circle
> > back,
> > your comment was not entirely unexpected, just the push that is needed for
> > me
> > to look more closely. I will follow up in a bit.
> 
> package.bbclass has variables containing / functions returning lists of
> variable names. libdir isn't a list of variable names, it's a path.

Agreed. And this is why I was going back to look at it.

> 
> > Second, if libdir is used in do_configure, then changing
> > 
> > > it will already cause do_configure to be re-run, as bitbake tracks
> > 
> > variable
> > 
> > > references. Only references which can't be tracked by bitbake need to be
> > > listed explicitly.
> > 
> > The observed behavior definitely supports the view that this is a case
> > where
> > it can't be tracked. It is definitely a different usage than you would
> > find
> > normally.
> > 
> > do_configure () {
> > ...
> > 
> >     perl ./Configure ${EXTRA_OECONF} shared --prefix=$useprefix \
> >     
> >        --openssldir=${libdir}/ssl --libdir=`basename ${libdir}` $target
> > 
> > }
> 
> Nope, this is exactly what bitbake knows how to track, a standard variable
> reference; ${libdir}. It knows that do_configure uses libdir already. It's
> only when the variable name is determined programmatically at runtime in
> python that bitbake isn't able to track the reference (e.g. d.getVar(foo +
> "bar", True)).

Using 'bitbake-dumpsig' I can confirm that 'libdir' is listed in the 
do_configure 'Task dependencies' in the sstate. This explains why with or 
without

do_configure[vardeps] += "libdir"


the outcome didn't change, the change was essentially a nop. The variable is 
definitely at play during configure, as mentioned in my commit log it is used as 
a hard coded abs path. For instance it is substituted in the package's 
crypto/opensslconf.h file.

When the sstate is reused in a different location configure needs to be rerun. 
Similarly if libdir for the target is modified it needs to rerun configure.

Any ideas why/how libdir is listed as a task dependency yet even with it 
changing the task is being rerun?

I will keep investigating.

Mark

Re: [PATCH] openssl: add a vardeps for configure on libdir

[Mark Asselstine]

On March 25, 2015 20:44:57 Burton, Ross wrote:
> On 25 March 2015 at 16:28, Mark Asselstine <mark.asselstine@windriver.com>
> 
> wrote:
> > For target builds libdir is not something that changes and for -native
> > builds unless you move your builds around you will see no issue with
> > the current implementation. If you do update libdir or if you move
> > your build or reuse sstate by copying it to a new build you will start
> > to see errors. These will be seen when you use the 'openssl' tool and
> > look like the following:
> > 
> > WARNING: can't open config file: /home/build/bitbake_build/tmp/
> > 
> >                         sysroots/x86_64-linux/usr/lib/ssl/openssl.cnf
> 
> We don't generally require that native sstate is rebuilt when the build
> directory changes, as otherwise you wouldn't be able to share native sstate
> between builds and users.  Instead, paths are munged where possible when
> using sstate and wrapper scripts used to "fix" binaries.  Indeed
> openssl.inc does this:
> 
> do_install_append_virtclass-native() {
>         create_wrapper ${D}${bindir}/openssl \
>             OPENSSL_CONF=${libdir}/ssl/openssl.cnf \
>             SSL_CERT_DIR=${libdir}/ssl/certs \
>             SSL_CERT_FILE=${libdir}/ssl/cert.pem \
>             OPENSSL_ENGINES=${libdir}/ssl/engines
> }
> 
> Is it possible that the binary isn't respecting the OPENSSL_CONF
> environment variable since that wrapper was written? (or the wrapper wasn't
> tested well enough)

Interesting. Thanks for the heads up Ross I can definitely look at the wrapper 
for possible issues.

Mark

> 
> Ross

Re: [PATCH] openssl: add a vardeps for configure on libdir

[Mark Asselstine]

On March 25, 2015 20:37:51 Mark Asselstine wrote:
> On March 25, 2015 20:44:57 Burton, Ross wrote:
> > On 25 March 2015 at 16:28, Mark Asselstine <mark.asselstine@windriver.com>
> > 
> > wrote:
> > > For target builds libdir is not something that changes and for -native
> > > builds unless you move your builds around you will see no issue with
> > > the current implementation. If you do update libdir or if you move
> > > your build or reuse sstate by copying it to a new build you will start
> > > to see errors. These will be seen when you use the 'openssl' tool and
> > > look like the following:
> > > 
> > > WARNING: can't open config file: /home/build/bitbake_build/tmp/
> > > 
> > >                         sysroots/x86_64-linux/usr/lib/ssl/openssl.cnf
> > 
> > We don't generally require that native sstate is rebuilt when the build
> > directory changes, as otherwise you wouldn't be able to share native
> > sstate
> > between builds and users.  Instead, paths are munged where possible when
> > using sstate and wrapper scripts used to "fix" binaries.  Indeed
> > openssl.inc does this:
> > 
> > do_install_append_virtclass-native() {
> > 
> >         create_wrapper ${D}${bindir}/openssl \
> >         
> >             OPENSSL_CONF=${libdir}/ssl/openssl.cnf \
> >             SSL_CERT_DIR=${libdir}/ssl/certs \
> >             SSL_CERT_FILE=${libdir}/ssl/cert.pem \
> >             OPENSSL_ENGINES=${libdir}/ssl/engines
> > 
> > }
> > 
> > Is it possible that the binary isn't respecting the OPENSSL_CONF
> > environment variable since that wrapper was written? (or the wrapper
> > wasn't
> > tested well enough)
> 
> Interesting. Thanks for the heads up Ross I can definitely look at the
> wrapper for possible issues.

Ross,

This is exactly the clue I needed. With this I was able to determine the flaw 
in my analysis. Things are working as expected, my patch is not needed.

Sorry for the noise. Thanks Ross and Christopher for the guidance,
Mark

> 
> Mark
> 
> > Ross