[PATCH] cooker: Ensure BB_CONSOLE remains correct over server resets

[PATCH] cooker: Ensure BB_CONSOLE remains correct over server resets

[Richard Purdie]

The console log data is written to is created at console initialisation
time and does not change over reset events. This ensures the
BB_CONSOLELOG value is correct over such resets by preserving it.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 bitbake/lib/bb/cooker.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 9c9d761..ba0a8fe 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -244,6 +244,11 @@ class BBCooker:
         self.state = state.initial
         self.caches_array = []
 
+        # Need to preserve BB_CONSOLELOG over resets
+        consolelog = None
+        if hasattr(self, "data"):
+            consolelog = self.data.getVar("BB_CONSOLELOG", True)
+
         if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
             self.enableDataTracking()
 
@@ -270,6 +275,8 @@ class BBCooker:
         self.data = self.databuilder.data
         self.data_hash = self.databuilder.data_hash
 
+        if consolelog:
+            self.data.setVar("BB_CONSOLELOG", consolelog)
 
         # we log all events to a file if so directed
         if self.configuration.writeeventlog:

Re: [PATCH] cooker: Ensure BB_CONSOLE remains correct over server resets

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 1877 bytes --]

Thanks for these, Richard.

On 11 October 2015 at 10:22, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> The console log data is written to is created at console initialisation
> time and does not change over reset events. This ensures the
> BB_CONSOLELOG value is correct over such resets by preserving it.
>

I've put your three patches on top of master and ran a few builds. The
result is that all of the logs for those builds are going into a single
timestamped file. Before, they went to separate files. Is this intentional,
or was the old behaviour incorrect? (Ideally, toaster needs a separate log
file for each build.)

Thanks.
Elliot




>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  bitbake/lib/bb/cooker.py | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index 9c9d761..ba0a8fe 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -244,6 +244,11 @@ class BBCooker:
>          self.state = state.initial
>          self.caches_array = []
>
> +        # Need to preserve BB_CONSOLELOG over resets
> +        consolelog = None
> +        if hasattr(self, "data"):
> +            consolelog = self.data.getVar("BB_CONSOLELOG", True)
> +
>          if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
>              self.enableDataTracking()
>
> @@ -270,6 +275,8 @@ class BBCooker:
>          self.data = self.databuilder.data
>          self.data_hash = self.databuilder.data_hash
>
> +        if consolelog:
> +            self.data.setVar("BB_CONSOLELOG", consolelog)
>
>          # we log all events to a file if so directed
>          if self.configuration.writeeventlog:
>
>
>


-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 2783 bytes --]

Re: [PATCH] cooker: Ensure BB_CONSOLE remains correct over server resets

[Richard Purdie]

On Fri, 2015-10-16 at 16:57 +0100, Smith, Elliot wrote:
> Thanks for these, Richard.
>
> On 11 October 2015 at 10:22, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>         The console log data is written to is created at console
>         initialisation
>         time and does not change over reset events. This ensures the
>         BB_CONSOLELOG value is correct over such resets by preserving
>         it.
>
> I've put your three patches on top of master and ran a few builds. The
> result is that all of the logs for those builds are going into a
> single timestamped file. Before, they went to separate files. Is this
> intentional, or was the old behaviour incorrect? (Ideally, toaster
> needs a separate log file for each build.)
> 
It should generate one log file for each "session". A session would be
one logfile for all bitbake memory resident transactions and a log file
for each commandline transaction.

It is a "console log", not an individual build log.

Cheers,

Richard

Re: [PATCH] cooker: Ensure BB_CONSOLE remains correct over server resets

[Barros Pena, Belen]

On 16/10/2015 16:59, "bitbake-devel-bounces@lists.openembedded.org on
behalf of Richard Purdie" <bitbake-devel-bounces@lists.openembedded.org on
behalf of richard.purdie@linuxfoundation.org> wrote:

>On Fri, 2015-10-16 at 16:57 +0100, Smith, Elliot wrote:
>> Thanks for these, Richard.
>>
>> On 11 October 2015 at 10:22, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>>         The console log data is written to is created at console
>>         initialisation
>>         time and does not change over reset events. This ensures the
>>         BB_CONSOLELOG value is correct over such resets by preserving
>>         it.
>>
>> I've put your three patches on top of master and ran a few builds. The
>> result is that all of the logs for those builds are going into a
>> single timestamped file. Before, they went to separate files. Is this
>> intentional, or was the old behaviour incorrect? (Ideally, toaster
>> needs a separate log file for each build.)
>> 
>It should generate one log file for each "session". A session would be
>one logfile for all bitbake memory resident transactions and a log file
>for each commandline transaction.
>
>It is a "console log", not an individual build log.

Mmmm, for Toaster we need an individual build log, not a console log :/

>
>Cheers,
>
>Richard
>
>
>
>-- 
>_______________________________________________
>bitbake-devel mailing list
>bitbake-devel@lists.openembedded.org
>http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Re: [PATCH] cooker: Ensure BB_CONSOLE remains correct over server resets

[Richard Purdie]

On Mon, 2015-10-19 at 10:32 +0000, Barros Pena, Belen wrote:
> 
> On 16/10/2015 16:59, "bitbake-devel-bounces@lists.openembedded.org on
> behalf of Richard Purdie" <bitbake-devel-bounces@lists.openembedded.org on
> behalf of richard.purdie@linuxfoundation.org> wrote:
> 
> >On Fri, 2015-10-16 at 16:57 +0100, Smith, Elliot wrote:
> >> Thanks for these, Richard.
> >>
> >> On 11 October 2015 at 10:22, Richard Purdie
> >> <richard.purdie@linuxfoundation.org> wrote:
> >>         The console log data is written to is created at console
> >>         initialisation
> >>         time and does not change over reset events. This ensures the
> >>         BB_CONSOLELOG value is correct over such resets by preserving
> >>         it.
> >>
> >> I've put your three patches on top of master and ran a few builds.

When you say "ran a few builds", was this with toaster? I'd like to
double check we are seeing the right behaviour but it depends how you
actually ran the builds as to whether this was the right result.

>  The
> >> result is that all of the logs for those builds are going into a
> >> single timestamped file. Before, they went to separate files. Is this
> >> intentional, or was the old behaviour incorrect? (Ideally, toaster
> >> needs a separate log file for each build.)
> >> 
> >It should generate one log file for each "session". A session would be
> >one logfile for all bitbake memory resident transactions and a log file
> >for each commandline transaction.
> >
> >It is a "console log", not an individual build log.
> 
> Mmmm, for Toaster we need an individual build log, not a console log :/

Since toaster records all the log information, is it not able to compute
an individual build log from that data?

The reason I'm reluctant to just add some kind of "build log"
functionality to bitbake is that at the commandline level, I think it
would mislead people. In theory, a "build" would be the output between
BuildStarted and BuildCompleted, the question is then what happens to
the output before and after these events.

Cheers,

Richard

Re: [PATCH] cooker: Ensure BB_CONSOLE remains correct over server resets

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 1715 bytes --]

On 16 October 2015 at 16:59, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Fri, 2015-10-16 at 16:57 +0100, Smith, Elliot wrote:
> > On 11 October 2015 at 10:22, Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> >         The console log data is written to is created at console
> >         initialisation
> >         time and does not change over reset events. This ensures the
> >         BB_CONSOLELOG value is correct over such resets by preserving
> >         it.
> >
> > I've put your three patches on top of master and ran a few builds. The
> > result is that all of the logs for those builds are going into a
> > single timestamped file. Before, they went to separate files. Is this
> > intentional, or was the old behaviour incorrect? (Ideally, toaster
> > needs a separate log file for each build.)
> >
> It should generate one log file for each "session". A session would be
> one logfile for all bitbake memory resident transactions and a log file
> for each commandline transaction.
>
> It is a "console log", not an individual build log.
>

Thanks for clarifying.

Will these patches be in 2.0? From the toaster point of view, it would be
better if the current behaviour remained for now, even though it is broken.

We are working on a solution which will write separate build logs from
within Toaster itself, which will mean we can keep our log download working
when your patches do go in.

However, we may not get this finished for 2.0, and it would be good if we
could keep the Toaster log download feature working in the meantime.

Thanks.
Elliot

-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 2424 bytes --]

Re: [PATCH] cooker: Ensure BB_CONSOLE remains correct over server resets

[Richard Purdie]

On Mon, 2015-10-19 at 13:27 +0100, Smith, Elliot wrote:
> Thanks for clarifying.
>
> Will these patches be in 2.0? From the toaster point of view, it would
> be better if the current behaviour remained for now, even though it is
> broken.
>
> We are working on a solution which will write separate build logs from
> within Toaster itself, which will mean we can keep our log download
> working when your patches do go in.
>
> However, we may not get this finished for 2.0, and it would be good if
> we could keep the Toaster log download feature working in the
> meantime.
> 
Given the other discussion I've seen, I'm assuming it would be ok to
merge these now?

Cheers,

Richard

Re: [PATCH] cooker: Ensure BB_CONSOLE remains correct over server resets

[Barros Pena, Belen]

On 24/10/2015 12:26, "bitbake-devel-bounces@lists.openembedded.org on
behalf of Richard Purdie" <bitbake-devel-bounces@lists.openembedded.org on
behalf of richard.purdie@linuxfoundation.org> wrote:

>On Mon, 2015-10-19 at 13:27 +0100, Smith, Elliot wrote:
>> Thanks for clarifying.
>>
>> Will these patches be in 2.0? From the toaster point of view, it would
>> be better if the current behaviour remained for now, even though it is
>> broken.
>>
>> We are working on a solution which will write separate build logs from
>> within Toaster itself, which will mean we can keep our log download
>> working when your patches do go in.
>>
>> However, we may not get this finished for 2.0, and it would be good if
>> we could keep the Toaster log download feature working in the
>> meantime.
>> 
>Given the other discussion I've seen, I'm assuming it would be ok to
>merge these now?

I think it would be ok to merge them only if this

https://lists.yoctoproject.org/pipermail/toaster/2015-October/003196.html


Makes it to 2.0. Right now is still under review in the Toaster mailing
list.

Elliot: correct me if I am wrong.

Cheers

Belén


>
>Cheers,
>
>Richard
>
>
>
>-- 
>_______________________________________________
>bitbake-devel mailing list
>bitbake-devel@lists.openembedded.org
>http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Re: [PATCH] cooker: Ensure BB_CONSOLE remains correct over server resets

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

On 24 October 2015 at 12:26, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2015-10-19 at 13:27 +0100, Smith, Elliot wrote:
> > Thanks for clarifying.
> >
> > Will these patches be in 2.0? From the toaster point of view, it would
> > be better if the current behaviour remained for now, even though it is
> > broken.
> >
> > We are working on a solution which will write separate build logs from
> > within Toaster itself, which will mean we can keep our log download
> > working when your patches do go in.
> >
> > However, we may not get this finished for 2.0, and it would be good if
> > we could keep the Toaster log download feature working in the
> > meantime.
> >
> Given the other discussion I've seen, I'm assuming it would be ok to
> merge these now?
>

Yes, as we're now writing our own logs for Toaster.

Thanks for merging.

Elliot


> Cheers,
>
> Richard
>
>
>
>


-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 1741 bytes --]

Re: [PATCH] cooker: Ensure BB_CONSOLE remains correct over server resets

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

On 27 October 2015 at 11:21, Smith, Elliot <elliot.smith@intel.com> wrote:

> On 24 October 2015 at 12:26, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> On Mon, 2015-10-19 at 13:27 +0100, Smith, Elliot wrote:
>> Given the other discussion I've seen, I'm assuming it would be ok to
>> merge these now?
>>
>
> Yes, as we're now writing our own logs for Toaster.
>
> Thanks for merging.
>

Richard, apologies, I just realised we have one patch outstanding:
https://lists.yoctoproject.org/pipermail/toaster/2015-October/003196.html

(This is the patch which actually does the work on the Toaster side.)

I thought this had been reviewed and submitted to bitbake-devel, but it
still hasn't. I will try to get this done today.

If we can get this in, as well as your patches to how bitbake sets
BB_CONSOLELOG, we won't lose any functionality in Toaster.

Thanks.
Elliot


> Elliot
>
>
>> Cheers,
>>
>> Richard
>>
>>
>>
>>
>
>
> --
> Elliot Smith
> Software Engineer
> Intel Open Source Technology Centre
>



-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 2848 bytes --]