Re: [bitbake-devel] bitbake-selftest losing sqlite3 database contents!

["Jan-Simon Möller" <dl9pf@gmx.de>]

Am Samstag, 20. April 2024, 22:35:33 CEST schrieb Joshua Watt via lists.openembedded.org:

> I'm not exactly sure what the purpose of the read-only prserv is in
> the first place; based on these requirements, it sort of sounds like
> there isn't anyway to make a read-only server that has actual data in
> it :)

Well what about this use-case:
An CI loop would populate using a rw PR-server , those artifacts (sources, sstate, hashserv(!), deploy dir)
are then pushed to a download server. We do not want to expose that r/w PR-serv publically.
Now for a binary feed-like setup, we'd need a matching PR and Hashserv for the downstream. No ?
Thus extra r/o instances sharing the same sqlite in the back.
Isn't this then required for a downstream user to create their own 'matching' binaries for the feed ?

Correct me, but e.g. this is what we assume is the way to expose the data:

# rw prserv on port 8181
bitbake-prserv --start --file /home/yocto/pr_hash_serv-volume/prserv.db --loglevel=DEBUG --log /home/yocto/pr_hash_serv-volume/prserv-rw.log --port 8181
# ro prserv on port 8282
bitbake-prserv --start -r --file /home/yocto/pr_hash_serv-volume/prserv.db --loglevel=DEBUG --log /home/yocto/pr_hash_serv-volume/prserv-ro.log --port 8282

# rw hashserv on port 8383
bitbake-hashserv --bind :8383 --log DEBUG --database /home/yocto/pr_hash_serv-volume/hashserv.db > /home/yocto/pr_hash_serv-volume/hashserv-rw.log 2>&1 &
# ro hashserv on port 8484
bitbake-hashserv -r --bind :8484 --log DEBUG --database /home/yocto/pr_hash_serv-volume/hashserv.db > /home/yocto/pr_hash_serv-volume/hashserv-ro.log 2>&1 &


@Richard/Josh:
Let me add (yes, grumble a bit ;) ) : for me as *end user* of this, this is all the same and would be best kept in one single server that I can scale.
Having it split into two just complicates things ... and now for scarthgap, we need another instance of the hashserv due to the API change.
Please consider this from a user perspective working on multiple branches and not just one master branch!
The data might not be 'the same' or not 'connected', but they are still linked in a way and moreover we are exposing this to users
to use and usage gets mandatory if we think of speeding up builds and binary feeds. Don't get me wrong, this is good stuff, but we make it hard for general use.
Wrt scaling: - essentially we're setting us up with scaling problems wrt prserv down the road when it gets needed for binary feed.
Heck, we're just solving this for hashserv, so why not do this likewise right away - at best put both in one server?

@Michael: the 'empty file' issue rings a bell and I've seen this back when we tried this for the first time.
Export did not work either. But I have a hard time remembering the details. Might even depend on sqlite versions back then.
I found this in my local git:
sed -i -e "s#EXCLUSIVE#IMMEDIATE#g" /home/yocto/poky/bitbake/lib/prserv/db.py
It probably helped some but might cause other issues. Let's see what the experts say.


> Ostensibly, you should be able to write a test fixture that does
> whatever a real-life PR server installation would do to populate a
> database and then make it read-only. You would then do this process on
> the test database before each test where you needed to test the read
> only functionality. You have to do it for each test because as you
> noticed, you should not ever have "carry-over" in state from one test
> to another. Having carry-over between tests is not a good idea for a
> lot of reasons, which is why the hash server tests are setup the way
> that they are with an empty database for each test.

I second Josh here.


Best,
JS