Re: Pickle vs. sqlite3 for cache?

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Tue, 2024-08-20 at 13:56 +0000, chris.laplante@agilent.com wrote:
> Apologies if this discussion has already happened (I searched the
> archives but couldn’t find anything), but I had a thought the other
> day. I wonder if the decision to use ‘pickle’ for bb_cache.dat still
> makes sense? BitBake performance has certainly been getting better
> over the years I’ve used it, however it still seems to spend an
> inordinate amount of time loading gigantic pickled data into memory.
> I wonder if it is time to dump pickle and consider sqlite3 instead?
>  
> Just wanted to get the discussion started, since there’s a chance
> there is some detail(s) I missed that will stop this idea cold :(.

I'd not be in favour of using sqlite here, it is not a good fit for
what we need. We've used it in other areas in bitbake in the past and
it never seemed to work too well. We're effectively storing python data
structures in the cache and having mappings in/out of sqlite is not
going to help performance.

We should probably have a look at what exactly is being stored in the
cache and using the most space as I strongly suspect there are ways to
optmise that. Interning strings was very effective in the past for
example as you then only have one copy of the string and everything
else is a reference to it.

Also, the hope was also that BB_SERVER_TIMEOUT (i.e. memory resident
bitbake) would avoid this pickle time in particular. I don't know how
effectively that is happening.

Right now, I think the priority needs to be to sort out the
"runcommand" async vs a sync command issues and clean up that area of
the API, then we can think more about performance.

Cheers,

Richard