Pickle vs. sqlite3 for cache?

Pickle vs. sqlite3 for cache?

[chris.laplante]

[-- Attachment #1: Type: text/plain, Size: 637 bytes --]

Hi all,

Apologies if this discussion has already happened (I searched the archives but couldn't find anything), but I had a thought the other day. I wonder if the decision to use 'pickle' for bb_cache.dat still makes sense? BitBake performance has certainly been getting better over the years I've used it, however it still seems to spend an inordinate amount of time loading gigantic pickled data into memory. I wonder if it is time to dump pickle and consider sqlite3 instead?

Just wanted to get the discussion started, since there's a chance there is some detail(s) I missed that will stop this idea cold :(.

Thanks,
Chris

[-- Attachment #2: Type: text/html, Size: 2364 bytes --]

Re: [bitbake-devel] Pickle vs. sqlite3 for cache?

[Joshua Watt]

On Tue, Aug 20, 2024 at 7:57 AM Chris Laplante via
lists.openembedded.org
<chris.laplante=agilent.com@lists.openembedded.org> wrote:
>
> Hi all,
>
>
>
> Apologies if this discussion has already happened (I searched the archives but couldn’t find anything), but I had a thought the other day. I wonder if the decision to use ‘pickle’ for bb_cache.dat still makes sense? BitBake performance has certainly been getting better over the years I’ve used it, however it still seems to spend an inordinate amount of time loading gigantic pickled data into memory. I wonder if it is time to dump pickle and consider sqlite3 instead?

It's an interesting idea. Is the concept that instead of having to
load the entire cache into memory at once you would basically amortize
the lookup over time by querying for what you need when it's actually
required? I'm wondering how much of the bb_cache.dat is used for any
given invocation of bitbake. If the entire cache is going to be
queried anyway, it seems unlikely this would be better performing,
since a sqlite query is likely going to be a much more expensive
operation (per lookup) than loading in a pickle as data that Python
can natively access. If the percentage of data needed from the
bb_cache.dat is low, I could see that as a win though.

As with many things, careful profiling would be needed to tell if it's
actually a win.

>
>
>
> Just wanted to get the discussion started, since there’s a chance there is some detail(s) I missed that will stop this idea cold :(.

I'm not aware of that being discussed before.

>
>
>
> Thanks,
>
> Chris
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#16499): https://lists.openembedded.org/g/bitbake-devel/message/16499
> Mute This Topic: https://lists.openembedded.org/mt/108000847/3616693
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [JPEWhacker@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: Pickle vs. sqlite3 for cache?

[Richard Purdie]

On Tue, 2024-08-20 at 13:56 +0000, chris.laplante@agilent.com wrote:
> Apologies if this discussion has already happened (I searched the
> archives but couldn’t find anything), but I had a thought the other
> day. I wonder if the decision to use ‘pickle’ for bb_cache.dat still
> makes sense? BitBake performance has certainly been getting better
> over the years I’ve used it, however it still seems to spend an
> inordinate amount of time loading gigantic pickled data into memory.
> I wonder if it is time to dump pickle and consider sqlite3 instead?
>  
> Just wanted to get the discussion started, since there’s a chance
> there is some detail(s) I missed that will stop this idea cold :(.

I'd not be in favour of using sqlite here, it is not a good fit for
what we need. We've used it in other areas in bitbake in the past and
it never seemed to work too well. We're effectively storing python data
structures in the cache and having mappings in/out of sqlite is not
going to help performance.

We should probably have a look at what exactly is being stored in the
cache and using the most space as I strongly suspect there are ways to
optmise that. Interning strings was very effective in the past for
example as you then only have one copy of the string and everything
else is a reference to it.

Also, the hope was also that BB_SERVER_TIMEOUT (i.e. memory resident
bitbake) would avoid this pickle time in particular. I don't know how
effectively that is happening.

Right now, I think the priority needs to be to sort out the
"runcommand" async vs a sync command issues and clean up that area of
the API, then we can think more about performance.

Cheers,

Richard

RE: Pickle vs. sqlite3 for cache?

[chris.laplante]

> On Tue, 2024-08-20 at 13:56 +0000, chris.laplante@agilent.com wrote:
> > Apologies if this discussion has already happened (I searched the
> > archives but couldn’t find anything), but I had a thought the other
> > day. I wonder if the decision to use ‘pickle’ for bb_cache.dat still
> > makes sense? BitBake performance has certainly been getting better
> > over the years I’ve used it, however it still seems to spend an
> > inordinate amount of time loading gigantic pickled data into memory.
> > I wonder if it is time to dump pickle and consider sqlite3 instead?
> >
> > Just wanted to get the discussion started, since there’s a chance
> > there is some detail(s) I missed that will stop this idea cold :(.
> 
> I'd not be in favour of using sqlite here, it is not a good fit for what we need.
> We've used it in other areas in bitbake in the past and it never seemed to work
> too well. We're effectively storing python data structures in the cache and
> having mappings in/out of sqlite is not going to help performance.
> 
> We should probably have a look at what exactly is being stored in the cache
> and using the most space as I strongly suspect there are ways to optmise that.
> Interning strings was very effective in the past for example as you then only
> have one copy of the string and everything else is a reference to it.

Understood, thanks for the background. One optimization thing that came up while I was researching this is the 'pickletools' module, specifically the 'optimize' method: https://docs.python.org/3/library/pickletools.html#pickletools.optimize. I wonder if there is a place in BitBake to run that occasionally (assuming, of course, it has the desired effect during testing).

> Also, the hope was also that BB_SERVER_TIMEOUT (i.e. memory resident
> bitbake) would avoid this pickle time in particular. I don't know how effectively
> that is happening.
>
> Right now, I think the priority needs to be to sort out the "runcommand"
> async vs a sync command issues and clean up that area of the API, then we
> can think more about performance.

Certainly. I did not mean to try to assert this as any kind of priority whatsoever :). Was just curious.

Thanks,
Chris