[WIP][RFC PATCH 0/5] fix for memres bitbake

[WIP][RFC PATCH 0/5] fix for memres bitbake

[Robert Yang]

The following changes since commit 9ed748a542b520c1cb763d981969233c0f5efd4e:

  bitbake: daemonize: Always print any remaning UI events at exit (2017-08-03 11:14:13 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib rbt/bitbake
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/bitbake

Robert Yang (5):
  btiabke: main.py: remove unneeded float()
  bitbake: main: add log handler when status-only
  bitbake: process: fix disconnect when BB_SERVER_TIMEOUT
  bitbake: main: handle BB_SERVER_TIMEOUT = -1
  bitbake: main: make observe-only work without --bind

 bitbake/lib/bb/main.py           | 14 +++++---------
 bitbake/lib/bb/server/process.py | 13 ++++++-------
 2 files changed, 11 insertions(+), 16 deletions(-)

-- 
2.11.0

[WIP][RFC PATCH 1/5] btiabke: main.py: remove unneeded float()

[Robert Yang]

There is already a type=float, so the float() is not needed, which also makes
the error clearer:

$ export BB_SERVER_TIMEOUT=10000AA
With float():
$ bitbake quilt-native
[snip]
ValueError: could not convert string to float: '10000AA'

Without float():
$ bitbake quilt-native
[snip]
optparse.OptionValueError: option --idle-timeout: invalid floating-point value: '10000AA'

The second one tells clearly where is wrong.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/main.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
index 431f6f47940..62f9e1eacf1 100755
--- a/bitbake/lib/bb/main.py
+++ b/bitbake/lib/bb/main.py
@@ -259,7 +259,7 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
                           help="The name/address for the bitbake xmlrpc server to bind to.")
 
         parser.add_option("-T", "--idle-timeout", type=float, dest="server_timeout",
-                          default=float(os.environ.get("BB_SERVER_TIMEOUT", 0)) or None,
+                          default=os.environ.get("BB_SERVER_TIMEOUT", 0) or None,
                           help="Set timeout to unload bitbake server due to inactivity")
 
         parser.add_option("", "--no-setscene", action="store_true",
-- 
2.11.0

[WIP][RFC PATCH 2/5] bitbake: main: add log handler when status-only

[Robert Yang]

The ui is not a must for logger, and use logger when status-only is very good
for debugging.

e.g.:
$ bitbake --status
NOTE: bitbake server is not running.

$ bitbake --server-only -B localhost:-1
$ bitbake --status
NOTE: Reconnecting to bitbake server...

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/main.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
index 62f9e1eacf1..fb082e7e0f3 100755
--- a/bitbake/lib/bb/main.py
+++ b/bitbake/lib/bb/main.py
@@ -397,9 +397,7 @@ def bitbake_main(configParams, configuration):
 def setup_bitbake(configParams, configuration, extrafeatures=None, setup_logging=True):
     # Ensure logging messages get sent to the UI as events
     handler = bb.event.LogHandler()
-    if setup_logging and not configParams.status_only:
-        # In status only mode there are no logs and no UI
-        logger.addHandler(handler)
+    logger.addHandler(handler)
 
     # Clear away any spurious environment variables while we stoke up the cooker
     cleanedvars = bb.utils.clean_environment()
-- 
2.11.0

Re: [WIP][RFC PATCH 2/5] bitbake: main: add log handler when status-only

[Richard Purdie]

On Tue, 2017-08-08 at 02:12 -0700, Robert Yang wrote:
> The ui is not a must for logger, and use logger when status-only is
> very good
> for debugging.
> 
> e.g.:
> $ bitbake --status
> NOTE: bitbake server is not running.
> 
> $ bitbake --server-only -B localhost:-1
> $ bitbake --status
> NOTE: Reconnecting to bitbake server...
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/main.py | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
> 
> diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
> index 62f9e1eacf1..fb082e7e0f3 100755
> --- a/bitbake/lib/bb/main.py
> +++ b/bitbake/lib/bb/main.py
> @@ -397,9 +397,7 @@ def bitbake_main(configParams, configuration):
>  def setup_bitbake(configParams, configuration, extrafeatures=None,
> setup_logging=True):
>      # Ensure logging messages get sent to the UI as events
>      handler = bb.event.LogHandler()
> -    if setup_logging and not configParams.status_only:
> -        # In status only mode there are no logs and no UI
> -        logger.addHandler(handler)
> +    logger.addHandler(handler)
>  
>      # Clear away any spurious environment variables while we stoke
> up the cooker
>      cleanedvars = bb.utils.clean_environment()

The patch doesn't seem to apply, there is no setup_logging parameter on
master?

Cheers,

Richard

Re: [WIP][RFC PATCH 2/5] bitbake: main: add log handler when status-only

[Robert Yang]

On 08/08/2017 08:14 PM, Richard Purdie wrote:
> On Tue, 2017-08-08 at 02:12 -0700, Robert Yang wrote:
>> The ui is not a must for logger, and use logger when status-only is
>> very good
>> for debugging.
>>
>> e.g.:
>> $ bitbake --status
>> NOTE: bitbake server is not running.
>>
>> $ bitbake --server-only -B localhost:-1
>> $ bitbake --status
>> NOTE: Reconnecting to bitbake server...
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>  bitbake/lib/bb/main.py | 4 +---
>>  1 file changed, 1 insertion(+), 3 deletions(-)
>>
>> diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
>> index 62f9e1eacf1..fb082e7e0f3 100755
>> --- a/bitbake/lib/bb/main.py
>> +++ b/bitbake/lib/bb/main.py
>> @@ -397,9 +397,7 @@ def bitbake_main(configParams, configuration):
>>  def setup_bitbake(configParams, configuration, extrafeatures=None,
>> setup_logging=True):
>>      # Ensure logging messages get sent to the UI as events
>>      handler = bb.event.LogHandler()
>> -    if setup_logging and not configParams.status_only:
>> -        # In status only mode there are no logs and no UI
>> -        logger.addHandler(handler)
>> +    logger.addHandler(handler)
>>
>>      # Clear away any spurious environment variables while we stoke
>> up the cooker
>>      cleanedvars = bb.utils.clean_environment()
>
> The patch doesn't seem to apply, there is no setup_logging parameter on
> master?

Sorry, it does on master branch, but not no master-next. I will rebase it
to master-next.

// Robert



>
> Cheers,
>
> Richard
>

[WIP][RFC PATCH 3/5] bitbake: process: fix disconnect when BB_SERVER_TIMEOUT

[Robert Yang]

Fixed:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only
$ bitbake --status-only
[snip]
  File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
    msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor

And:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only -B localhost:-1
$ bitbake --status-only # Everything is fine in first run
$ bitbake --status-only
[snip]
  File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
    msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor

This was because self.controllersock was not set to False, so it still ran
sock.recvmsg() when sock was closed.

And also need set command_channel to Flase, otherwise the
self.command_channel.get() will always run when EOF, and cause infinite loop.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/server/process.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py
index bfd6404b738..5b8a549f94e 100644
--- a/bitbake/lib/bb/server/process.py
+++ b/bitbake/lib/bb/server/process.py
@@ -173,6 +173,7 @@ class ProcessServer(multiprocessing.Process):
                     self.event_writer.writer.close()
                     del self.event_writer
                     self.controllersock.close()
+                    self.controllersock = False
                     self.haveui = False
                     self.lastui = time.time()
                     self.cooker.clientComplete()
@@ -188,6 +189,7 @@ class ProcessServer(multiprocessing.Process):
                     command = self.command_channel.get()
                 except EOFError:
                     # Client connection shutting down
+                    self.command_channel = False
                     continue
                 if command[0] == "terminateServer":
                     self.quit = True
-- 
2.11.0

[WIP][RFC PATCH 4/5] bitbake: main: handle BB_SERVER_TIMEOUT = -1

[Robert Yang]

Make BB_SERVER_TIMEOUT = -1 mean no unload forever.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/main.py           | 6 ++++--
 bitbake/lib/bb/server/process.py | 3 ++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
index fb082e7e0f3..57b4d7688d9 100755
--- a/bitbake/lib/bb/main.py
+++ b/bitbake/lib/bb/main.py
@@ -259,8 +259,10 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
                           help="The name/address for the bitbake xmlrpc server to bind to.")
 
         parser.add_option("-T", "--idle-timeout", type=float, dest="server_timeout",
-                          default=os.environ.get("BB_SERVER_TIMEOUT", 0) or None,
-                          help="Set timeout to unload bitbake server due to inactivity")
+                          default=os.getenv("BB_SERVER_TIMEOUT"),
+                          help="Set timeout to unload bitbake server due to inactivity, "
+                                "set to -1 means no unload, "
+                                "default: Environment variable BB_SERVER_TIMEOUT.")
 
         parser.add_option("", "--no-setscene", action="store_true",
                           dest="nosetscene", default=False,
diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py
index 5b8a549f94e..338c44835e1 100644
--- a/bitbake/lib/bb/server/process.py
+++ b/bitbake/lib/bb/server/process.py
@@ -180,7 +180,8 @@ class ProcessServer(multiprocessing.Process):
                     if self.timeout is None:
                         print("No timeout, exiting.")
                         self.quit = True
-            if not self.haveui and self.lastui and self.timeout and (self.lastui + self.timeout) < time.time():
+            if not self.timeout == -1.0 and not self.haveui and self.lastui and self.timeout and \
+                    (self.lastui + self.timeout) < time.time():
                 print("Server timeout, exiting.")
                 self.quit = True
 
-- 
2.11.0

[WIP][RFC PATCH 5/5] bitbake: main: make observe-only work without --bind

[Robert Yang]

The bitbake --server-only can work without --bind, so --observe-only should also
can work without --bind or --remote-server.

And also allow the connection all the time, otherwise --observe-only doesn't
work.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/main.py           | 4 ----
 bitbake/lib/bb/server/process.py | 8 ++------
 2 files changed, 2 insertions(+), 10 deletions(-)

diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
index 57b4d7688d9..4477d9c5998 100755
--- a/bitbake/lib/bb/main.py
+++ b/bitbake/lib/bb/main.py
@@ -357,10 +357,6 @@ def bitbake_main(configParams, configuration):
                                   ("the BBSERVER environment variable" if "BBSERVER" in os.environ \
                                    else "the '--remote-server' option"))
 
-    if configParams.observe_only and not (configParams.remote_server or configParams.bind):
-        raise BBMainException("FATAL: '--observe-only' can only be used by UI clients "
-                              "connecting to a server.\n")
-
     if "BBDEBUG" in os.environ:
         level = int(os.environ["BBDEBUG"])
         if level > configuration.debug:
diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py
index 338c44835e1..7f81996246e 100644
--- a/bitbake/lib/bb/server/process.py
+++ b/bitbake/lib/bb/server/process.py
@@ -137,12 +137,8 @@ class ProcessServer(multiprocessing.Process):
         while not self.quit:
             if self.sock in ready:
                 self.controllersock, address = self.sock.accept()
-                if self.haveui:
-                    print("Dropping connection attempt as we have a UI %s" % (str(ready)))
-                    self.controllersock.close()
-                else:
-                    print("Accepting %s" % (str(ready)))
-                    fds.append(self.controllersock)
+                print("Accepting %s" % (str(ready)))
+                fds.append(self.controllersock)
             if self.controllersock in ready:
                 try:
                     print("Connecting Client")
-- 
2.11.0

Re: [WIP][RFC PATCH 5/5] bitbake: main: make observe-only work without --bind

[Richard Purdie]

On Tue, 2017-08-08 at 02:12 -0700, Robert Yang wrote:
> The bitbake --server-only can work without --bind, so --observe-only
> should also
> can work without --bind or --remote-server.
> 
> And also allow the connection all the time, otherwise --observe-only
> doesn't
> work.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/main.py           | 4 ----
>  bitbake/lib/bb/server/process.py | 8 ++------
>  2 files changed, 2 insertions(+), 10 deletions(-)

The other patches look ok but I worry a bit about this one since the
code isn't really set up to handle multiple potentially competing
connections right now. If something is an observer, we need to ensure
its marked as such and can't send control commands...

Cheers,

Richard

Re: [WIP][RFC PATCH 5/5] bitbake: main: make observe-only work without --bind

[Robert Yang]

On 08/08/2017 08:50 PM, Richard Purdie wrote:
> On Tue, 2017-08-08 at 02:12 -0700, Robert Yang wrote:
>> The bitbake --server-only can work without --bind, so --observe-only
>> should also
>> can work without --bind or --remote-server.
>>
>> And also allow the connection all the time, otherwise --observe-only
>> doesn't
>> work.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>  bitbake/lib/bb/main.py           | 4 ----
>>  bitbake/lib/bb/server/process.py | 8 ++------
>>  2 files changed, 2 insertions(+), 10 deletions(-)
>
> The other patches look ok but I worry a bit about this one since the
> code isn't really set up to handle multiple potentially competing
> connections right now. If something is an observer, we need to ensure
> its marked as such and can't send control commands...

Yes, that's a problem, I will fix it.

// Robert

>
> Cheers,
>
> Richard
>