runqueue: Ensure task environment is correct

runqueue: Ensure task environment is correct

[Richard Purdie]

This fixes two problems:

a) Variables which were in the parent environment but not set as "export"
   variables in the datastore could end up in the task environment

b) oe.environ.update() can't cope with the generator returned by 
   bb.data.exported_vars()

Whilst the updated code isn't as neat, it does do the expected thing,
sets the environment correctly and stops unwanted values leaking into
the task environment.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 5a4321f..72e6845 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1136,7 +1136,12 @@ class RunQueueExecute:
                 for h in self.rqdata.hash_deps:
                     the_data.setVar("BBHASHDEPS_%s" % h, self.rqdata.hash_deps[h])
 
-                os.environ.update(bb.data.exported_vars(the_data))
+                # exported_vars() returns a generator which *cannot* be passed to os.environ.update() 
+                # successfully. We also need to unset anything from the environment which shouldn't be there 
+                exports = bb.data.exported_vars(the_data)
+                bb.utils.empty_environment()
+                for e, v in exports:
+                    os.environ[e] = v
 
                 if quieterrors:
                     the_data.setVarFlag(taskname, "quieterrors", "1")

Re: runqueue: Ensure task environment is correct

[Chris Larson]

On Wed, Sep 7, 2011 at 10:43 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> This fixes two problems:
>
> a) Variables which were in the parent environment but not set as "export"
>   variables in the datastore could end up in the task environment
>
> b) oe.environ.update() can't cope with the generator returned by
>   bb.data.exported_vars()
>
> Whilst the updated code isn't as neat, it does do the expected thing,
> sets the environment correctly and stops unwanted values leaking into
> the task environment.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index 5a4321f..72e6845 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -1136,7 +1136,12 @@ class RunQueueExecute:
>                 for h in self.rqdata.hash_deps:
>                     the_data.setVar("BBHASHDEPS_%s" % h, self.rqdata.hash_deps[h])
>
> -                os.environ.update(bb.data.exported_vars(the_data))
> +                # exported_vars() returns a generator which *cannot* be passed to os.environ.update()
> +                # successfully. We also need to unset anything from the environment which shouldn't be there
> +                exports = bb.data.exported_vars(the_data)
> +                bb.utils.empty_environment()
> +                for e, v in exports:
> +                    os.environ[e] = v

os.environ.clear()
os.environ.update(bb.data.exported_vars(the_data))
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: runqueue: Ensure task environment is correct

[Richard Purdie]

On Wed, 2011-09-07 at 11:38 -0700, Chris Larson wrote:
> On Wed, Sep 7, 2011 at 10:43 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > This fixes two problems:
> >
> > a) Variables which were in the parent environment but not set as "export"
> >   variables in the datastore could end up in the task environment
> >
> > b) oe.environ.update() can't cope with the generator returned by
> >   bb.data.exported_vars()
> >
> > Whilst the updated code isn't as neat, it does do the expected thing,
> > sets the environment correctly and stops unwanted values leaking into
> > the task environment.
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> >
> > diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> > index 5a4321f..72e6845 100644
> > --- a/bitbake/lib/bb/runqueue.py
> > +++ b/bitbake/lib/bb/runqueue.py
> > @@ -1136,7 +1136,12 @@ class RunQueueExecute:
> >                 for h in self.rqdata.hash_deps:
> >                     the_data.setVar("BBHASHDEPS_%s" % h, self.rqdata.hash_deps[h])
> >
> > -                os.environ.update(bb.data.exported_vars(the_data))
> > +                # exported_vars() returns a generator which *cannot* be passed to os.environ.update()
> > +                # successfully. We also need to unset anything from the environment which shouldn't be there
> > +                exports = bb.data.exported_vars(the_data)
> > +                bb.utils.empty_environment()
> > +                for e, v in exports:
> > +                    os.environ[e] = v
> 
> os.environ.clear()
> os.environ.update(bb.data.exported_vars(the_data))

I tried this when testing and when I read os.environ back, it was not
correct. There is something odd occurring if you pass the generator into
update() directly.

Cheers,

Richard

Re: runqueue: Ensure task environment is correct

[Chris Larson]

On Wed, Sep 7, 2011 at 2:11 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Wed, 2011-09-07 at 11:38 -0700, Chris Larson wrote:
>> On Wed, Sep 7, 2011 at 10:43 AM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>> > This fixes two problems:
>> >
>> > a) Variables which were in the parent environment but not set as "export"
>> >   variables in the datastore could end up in the task environment
>> >
>> > b) oe.environ.update() can't cope with the generator returned by
>> >   bb.data.exported_vars()
>> >
>> > Whilst the updated code isn't as neat, it does do the expected thing,
>> > sets the environment correctly and stops unwanted values leaking into
>> > the task environment.
>> >
>> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>> >
>> > diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
>> > index 5a4321f..72e6845 100644
>> > --- a/bitbake/lib/bb/runqueue.py
>> > +++ b/bitbake/lib/bb/runqueue.py
>> > @@ -1136,7 +1136,12 @@ class RunQueueExecute:
>> >                 for h in self.rqdata.hash_deps:
>> >                     the_data.setVar("BBHASHDEPS_%s" % h, self.rqdata.hash_deps[h])
>> >
>> > -                os.environ.update(bb.data.exported_vars(the_data))
>> > +                # exported_vars() returns a generator which *cannot* be passed to os.environ.update()
>> > +                # successfully. We also need to unset anything from the environment which shouldn't be there
>> > +                exports = bb.data.exported_vars(the_data)
>> > +                bb.utils.empty_environment()
>> > +                for e, v in exports:
>> > +                    os.environ[e] = v
>>
>> os.environ.clear()
>> os.environ.update(bb.data.exported_vars(the_data))
>
> I tried this when testing and when I read os.environ back, it was not
> correct. There is something odd occurring if you pass the generator into
> update() directly.

Odd. It works fine in my local testing. Maybe there's a python bug
lurking -- I can't imagine it's supposed to work this way. Guess we
have to do it this way, but I really think you should consider
isolating the problem and reporting it upstream to the python project.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: runqueue: Ensure task environment is correct

[Richard Purdie]

On Wed, 2011-09-07 at 14:43 -0700, Chris Larson wrote:
> On Wed, Sep 7, 2011 at 2:11 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Wed, 2011-09-07 at 11:38 -0700, Chris Larson wrote:
> >> On Wed, Sep 7, 2011 at 10:43 AM, Richard Purdie
> >> <richard.purdie@linuxfoundation.org> wrote:
> >> > This fixes two problems:
> >> >
> >> > a) Variables which were in the parent environment but not set as "export"
> >> >   variables in the datastore could end up in the task environment
> >> >
> >> > b) oe.environ.update() can't cope with the generator returned by
> >> >   bb.data.exported_vars()
> >> >
> >> > Whilst the updated code isn't as neat, it does do the expected thing,
> >> > sets the environment correctly and stops unwanted values leaking into
> >> > the task environment.
> >> >
> >> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> >> >
> >> > diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> >> > index 5a4321f..72e6845 100644
> >> > --- a/bitbake/lib/bb/runqueue.py
> >> > +++ b/bitbake/lib/bb/runqueue.py
> >> > @@ -1136,7 +1136,12 @@ class RunQueueExecute:
> >> >                 for h in self.rqdata.hash_deps:
> >> >                     the_data.setVar("BBHASHDEPS_%s" % h, self.rqdata.hash_deps[h])
> >> >
> >> > -                os.environ.update(bb.data.exported_vars(the_data))
> >> > +                # exported_vars() returns a generator which *cannot* be passed to os.environ.update()
> >> > +                # successfully. We also need to unset anything from the environment which shouldn't be there
> >> > +                exports = bb.data.exported_vars(the_data)
> >> > +                bb.utils.empty_environment()
> >> > +                for e, v in exports:
> >> > +                    os.environ[e] = v
> >>
> >> os.environ.clear()
> >> os.environ.update(bb.data.exported_vars(the_data))
> >
> > I tried this when testing and when I read os.environ back, it was not
> > correct. There is something odd occurring if you pass the generator into
> > update() directly.
> 
> Odd. It works fine in my local testing. Maybe there's a python bug
> lurking -- I can't imagine it's supposed to work this way. Guess we
> have to do it this way, but I really think you should consider
> isolating the problem and reporting it upstream to the python project.

Agreed, it does seem to be something about my machine/python version
since this is not the first time I've had issues with os.environ :(.

Which version did you test with? I was seeing this with Ubuntu's python
"2.7.1+" whatever that is. I did try and write a small test case and
that works which is odd/worrying/annoying...

Cheers,

Richard