Re: [PATCH] bitbake: fetch2: Revalidate checksums, YOCTO #5571

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Fri, 2015-03-06 at 14:29 +0100, Martin Jansa wrote:
> On Fri, Mar 06, 2015 at 02:03:20PM +0100, Clemens Lang wrote:
> > Hi Martin,
> > 
> > On Fri, Mar 06, 2015 at 01:22:38PM +0100, Martin Jansa wrote:
> > > Using pickle is much better than counting the checksums on every
> > > do_fetch.. we have GBs of sources and we don't want to spend extra
> > > minutes on the build to re-check them if they are identical.
> > 
> > That's what I thought as well. The pickle method should scale a lot
> > better than re-running the hash calculation, especially in terms of I/O
> > ops.
> > 
> > > Maybe add the file time stamp as well, if the file was modified then
> > > .done could be invalidated and checksums re-verified.
> > 
> > I could modify the code to avoid reading the precomputed checksums if
> > the .done file is older than the downloaded file, which should cover
> > this case. Of course, the modification date will not be of much use if
> > the download is not a file, but a directory.
> > 
> > Does this sound good to you?
> 
> yes :)

Firstly, the delay in getting to reviewing this is mainly that we're
having issues in OE-Core which are causing me a lot of headaches so
sorry about that. We've long held the belief we needed to improve this
mechanism.

I am nervous about the amount and kind of code changes this is
involving. Having "binary" data format files in pickle format is
suboptimal in that the user can't easily inspect or change them and its
not clear what the contents means. Using pickle doesn't give a
dependency issue since we use it elsewhere in bitbake as you mentioned.

I was wondering about whether we should just drop to one checksum format
and simplify the problem somewhat. I understand the reasons for
supporting multiple checksum types though and if we add in a requirement
to track timestamps too, the single format doesn't buy us anything.

I guess above all, I've just been trying to push consolidation in the
fetcher, it has at various points been an unmaintainable nightmare. This
is a worthwhile improvement though.

Cheers,

Richard