Re: [bitbake-devel] bitbake-selftest losing sqlite3 database contents!

[Jan-Simon Moeller <dl9pf@gmx.de>]

Sorry for highjacking the thread ... reply inline.

<snip>
Am Montag, 22. April 2024, 16:47:48 CEST schrieb Joshua Watt:
> On Mon, Apr 22, 2024 at 4:00 AM Jan-Simon Möller <dl9pf@gmx.de> wrote:
> > Am Samstag, 20. April 2024, 22:35:33 CEST schrieb Joshua Watt via 
lists.openembedded.org:
> > @Richard/Josh:
> > Let me add (yes, grumble a bit ;) ) : for me as *end user* of this, this
> > is all the same and would be best kept in one single server that I can
> > scale. Having it split into two just complicates things ... and now for
> > scarthgap, we need another instance of the hashserv due to the API
> > change. Please consider this from a user perspective working on multiple
> > branches and not just one master branch! The data might not be 'the same'
> > or not 'connected', but they are still linked in a way and moreover we
> > are exposing this to users to use and usage gets mandatory if we think of
> > speeding up builds and binary feeds. Don't get me wrong, this is good
> > stuff, but we make it hard for general use. Wrt scaling: - essentially
> > we're setting us up with scaling problems wrt prserv down the road when
> > it gets needed for binary feed. Heck, we're just solving this for
> > hashserv, so why not do this likewise right away - at best put both in
> > one server?
> Can you clarify what you mean by "both in one server"? There are a lot
> of different levels of varying complexity that could be done for that.
> For example, you could:
> 1) Run both servers as separate processes in e.g. a shared container
> 2) Run both servers in a shared script
> 3) Combine the API into a single server API
> 
> These are generally more complex with more caveats as you go; #1 is
> the "easiest", #2 isn't possible until the PR server is completely
> asynchronous like the hashserver (otherwise it won't scale at all),
> and #3 is basically a completely new (breaking) API.

To be really honest, only #3 makes sense from my perspective  aka 'end user 
perspective' .


#1 is what we do already for the existing pr+hashserv ... 
excuse my language, but it s**ks. You need to care for multiple API revisions 
aka double (pr+hash) the port numbers to punch through. Each breaking API 
update needs us to start more server processes and punch through more ports. 
This is such a maintenance pita.

#2 is no difference to #1 if you think of different branches being in use.

> My preference is #1 or #2, but #2 makes no sense until the PR server
> has all of the features of the hash server; this patch series adds
> websockets which is a good step, but it would also need the database
> backend re-written to be nonblocking like the hash server, otherwise
> it's just asking for scaling problems if you run them in the same
> process :)

Nack.
Just teach Hashserv to store/serv the additional values and be done ?
Thanks to your great work on scaling the hashserv, we're almost there wrt 
deployments at scale. Why do that whole setup again for prserv ?
We'd have to duplicate also all the assets we use during scale-out ?

> Also keep in mind that one of the goals of the websockets server is to
> reduce the need for the "read-only" server, since you can have your CI
> authenticate with the hash server and get rw access, but anonymous
> users can be given read-only access.

Even better, then put it in one combined pr+hashserv, on one port, so we can 
manage the whole thing easily. 
Think of it ...  why would you want to deal with auth for 2 servers (pr+hash)?
This just leads to mismatches in auth (we all make mistakes during setup) and 
having to deal with twice as much credentials ... why?
I can't think of a case where I'd r/w on one but r/o on the other. It makes no 
sense.
 
> In short, there is space to make it a little easier to stand up the
> servers in some common ways, but I'm hesitant to force everyone to run
> the servers a specific way as I don't think there is a "one size fits
> all".
In the end: This is both 'metadata' that we want to store / reuse with 
bitbake. And more so going forward. So keep it simple also for the users.
This is not PR data vs hashserv data. Both is metadata we want to keep.

One size - actually I think exactly that. This makes it easier to run a local 
instance (just one - no mismatches possible). But also scale out if required 
and not doubling the infra (and cost!)  when doing so.


Best,
Jan-Simon