Re: [bitbake-devel][PATCH v4 00/22] Bitbake Hash Server WebSockets, Alternate Database Backend, and User Management

[Alexandre Belloni <alexandre.belloni@bootlin.com>]

Hello Joshua,

This is causing warning on the AB:

WARNING: Error contacting Hash Equivalence Server hashserv.yocto.io:8686: Expecting value: line 1 column 1 (char 0)

https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/3522/steps/12/logs/warnings
https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/3523/steps/17/logs/warnings
https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/8384/steps/14/logs/warnings
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/7990/steps/13/logs/warnings
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/7987/steps/12/logs/warnings
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/7991/steps/13/logs/warnings
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/7988/steps/12/logs/warnings





On 31/10/2023 11:21:16-0600, Joshua Watt wrote:
> This patch series reworks the bitbake asyncrpc API to add a WebSockets
> implementation for both the client and server. The hash equivalence
> server is updated to allow using this new API (the PR server can also be
> updated in the future if desired).
> 
> In addition, the database backed for the hash equivalence server is
> abstracted so that sqlalchemy can optionally be used instead of sqlite.
> This allows using "big metal" databases as the backend, which allows the
> hash equivalence server to scale to a large number of queries.
> 
> Note that both websockets and sqlalchemy require 3rd party python
> modules to function. However, these modules are optional unless the user
> desires to use the APIs.
> 
> Also, user management is added. This allows user accounts to be
> registered with the server and users can be given permissions to do
> certain operations on the server. Users are not (necessarily) required
> to login to access the server, as permissions can granted to anonymous
> users. The default permissions will give anonymous users the same
> permissions that they would have before user accounts were added so as
> to retain backward compatibility, but server admins will likely want to
> change this.
> 
> V3: Remove RFC status; patches are ready for review
> V4: Fixed protocol breakage with mixing older and newer clients/servers
> 
> Joshua Watt (22):
>   asyncrpc: Abstract sockets
>   hashserv: Add websocket connection implementation
>   asyncrpc: Add context manager API
>   hashserv: tests: Add external database tests
>   asyncrpc: Prefix log messages with client info
>   bitbake-hashserv: Allow arguments from environment
>   hashserv: Abstract database
>   hashserv: Add SQLalchemy backend
>   hashserv: Implement read-only version of "report" RPC
>   asyncrpc: Add InvokeError
>   asyncrpc: client: Prevent double closing of loop
>   asyncrpc: client: Add disconnect API
>   hashserv: Add user permissions
>   hashserv: Add become-user API
>   hashserv: Add db-usage API
>   hashserv: Add database column query API
>   hashserv: test: Add bitbake-hashclient tests
>   bitbake-hashclient: Output stats in JSON format
>   bitbake-hashserver: Allow anonymous permissions to be space separated
>   hashserv: tests: Allow authentication for external server tests
>   hashserv: Allow self-service deletion
>   hashserv: server: Add owner if user is logged in
> 
>  bin/bitbake-hashclient        | 145 +++++-
>  bin/bitbake-hashserv          | 132 ++++-
>  lib/bb/asyncrpc/__init__.py   |  33 +-
>  lib/bb/asyncrpc/client.py     | 120 ++---
>  lib/bb/asyncrpc/connection.py | 146 ++++++
>  lib/bb/asyncrpc/exceptions.py |  21 +
>  lib/bb/asyncrpc/serv.py       | 356 ++++++++-----
>  lib/hashserv/__init__.py      | 190 +++----
>  lib/hashserv/client.py        | 147 +++++-
>  lib/hashserv/server.py        | 951 +++++++++++++++++++++-------------
>  lib/hashserv/sqlalchemy.py    | 427 +++++++++++++++
>  lib/hashserv/sqlite.py        | 391 ++++++++++++++
>  lib/hashserv/tests.py         | 736 +++++++++++++++++++++++++-
>  lib/prserv/client.py          |   8 +-
>  lib/prserv/serv.py            |  37 +-
>  15 files changed, 3034 insertions(+), 806 deletions(-)
>  create mode 100644 lib/bb/asyncrpc/connection.py
>  create mode 100644 lib/bb/asyncrpc/exceptions.py
>  create mode 100644 lib/hashserv/sqlalchemy.py
>  create mode 100644 lib/hashserv/sqlite.py
> 
> -- 
> 2.34.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15350): https://lists.openembedded.org/g/bitbake-devel/message/15350
> Mute This Topic: https://lists.openembedded.org/mt/102302326/3617179
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com