Re: [PATCH][1.24/Dizzy] cooker: properly fix bitbake.lock handling

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Thu, 2015-08-20 at 10:48 -0700, Saul Wold wrote:
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> If the PR server or indeed any other child process takes some time to
> exit (which it sometimes does when saving its database), it can end up
> holding bitbake.lock after the UI exits, which led to errors if you ran
> bitbake commands successively - we saw this when running the PR server
> oe-selftest tests in OE-Core. The recent attempt to fix this wasn't
> quite right and ended up breaking memory resident bitbake. This time we
> close the lock file when cooker shuts down (inside the UI process)
> instead of unlocking it, and this is done in the cooker code rather than
> the actual UI code so it doesn't matter which UI is in use. Additionally
> we report that we're waiting for the lock to be released, using lsof or
> fuser if available to list the processes with the lock open.
> 
> The 'magic' in the locking is due to all spawned subprocesses of bitbake
> holding an open file descriptor to the bitbake.lock. It is automatically
> unlocked when all those fds close the file (as all the processes terminate).
> We close the UI copy of the lock explicitly, then close the server process
> copy, any remaining open copy is therefore some proess exiting.
> 
> (The reproducer for the problem is to set PRSERV_HOST = "localhost:0"
> and add a call to time.sleep(20) after self.server_close() in
> lib/prserv/serv.py, then run "bitbake -p; bitbake -p" ).
> 
> Cleanup work done by Paul Eggleton <paul.eggleton@linux.intel.com>.
> 
> This reverts bitbake commit 69ecd15aece54753154950c55d7af42f85ad8606 and
> e97a9f1528d77503b5c93e48e3de9933fbb9f3cd.
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> [sgw - merged changes from new main.py to bin/bitbake, dizzy will continue
> to use bin/bitbake and not use the new main.py (which is removed)]
> Signed-off-by: Saul Wold <sgw@linux.intel.com>
> 
> Conflicts:
> 	lib/bb/cooker.py
> 	lib/bb/main.py
> 	lib/bb/tinfoil.py
> 	lib/bb/ui/knotty.py
> ---
>  bin/bitbake         |  7 ++++++-
>  lib/bb/cooker.py    | 31 ++++++++++++++++++++++++++++++-
>  lib/bb/tinfoil.py   |  5 +++++
>  lib/bb/ui/knotty.py | 43 ++++++++++++++++++++++++-------------------
>  lib/bb/utils.py     | 29 +++++++++++++++++++++++++----
>  5 files changed, 90 insertions(+), 25 deletions(-)
> 
> diff --git a/bin/bitbake b/bin/bitbake
> index a2e8cc1..d3055fb 100755
> --- a/bin/bitbake
> +++ b/bin/bitbake
> @@ -263,13 +263,18 @@ def start_server(servermodule, configParams, configuration, features):
>                  logger.handle(event)
>          raise exc_info[1], None, exc_info[2]
>      server.detach()
> +    cooker.lock.close()
>      return server
>  
> 
> 
>  def main():
>  
> -    configParams = BitBakeConfigParameters()
> +    # Python multiprocessing requires /dev/shm on Linux
> +    if sys.platform.startswith('linux') and not os.access('/dev/shm', os.W_OK | os.X_OK):
> +        raise sys.exit("FATAL: /dev/shm does not exist or is not writable")
> +
> +    conaigParams = BitBakeConfigParameters()
>      configuration = cookerdata.CookerConfiguration()
>      configuration.setConfigParameters(configParams)
>  

conaigParams ?

I think the above hunk shouldn't be there...

Cheers,

Richard