Checksum failure encountered with download

Checksum failure encountered with download

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 849 bytes --]

Hi,

when fetcher fails with ChecksumError it shows error like in subject and
tries another MIRRORs

Would anyone object to keep fetched source with wrong checksum in
downloads directory (renamed so it does not conflict with file with
correct checksum downloaded later)?

Right now it just removes the file:
                        # Remove any incomplete fetch
                        m.clean(ud, self.d)

The problem with this is when upstream repacks some archive I would like
to be able to compare old tarball and new one.

My proposal is to add md5sum of downloaded file to ChecksumError and
rename it to <file>.<md5sum> in "if isinstance(e, ChecksumError):"

Does it make sense (should I send patch for this)?

FWIW: Emerge in gentoo does something similar.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [bitbake-devel] Checksum failure encountered with download

[Richard Purdie]

On Thu, 2013-04-11 at 15:30 +0200, Martin Jansa wrote:
> Hi,
> 
> when fetcher fails with ChecksumError it shows error like in subject and
> tries another MIRRORs
> 
> Would anyone object to keep fetched source with wrong checksum in
> downloads directory (renamed so it does not conflict with file with
> correct checksum downloaded later)?
> 
> Right now it just removes the file:
>                         # Remove any incomplete fetch
>                         m.clean(ud, self.d)
> 
> The problem with this is when upstream repacks some archive I would like
> to be able to compare old tarball and new one.
> 
> My proposal is to add md5sum of downloaded file to ChecksumError and
> rename it to <file>.<md5sum> in "if isinstance(e, ChecksumError):"
> 
> Does it make sense (should I send patch for this)?

I'm fine with that.

Cheers,

Richard

Re: [bitbake-devel] Checksum failure encountered with download

[Trevor Woerner]

Sorry for not noticing this sooner.

On Thu, Apr 11, 2013 at 9:30 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
> My proposal is to add md5sum of downloaded file to ChecksumError and
> rename it to <file>.<md5sum> in "if isinstance(e, ChecksumError):"

Would it be possible to make this behaviour configurable for those who
might be concerned about disk space? Or perhaps while renaming and
saving it, also move it into the $TMPDIR directory and/or tie this
feature in with rm_work?

I'm just thinking that over time one's DL_DIR could potentially become
rather cluttered with lots of failed downloads. Only some people would
be interested in seeing the failures and the rest would have to remove
them manually. I'm also thinking of the scenario where the DL_DIR is
shared/sync'ed among a group of people/computers; that could add up to
quite a bit of data to share.

Re: [bitbake-devel] Checksum failure encountered with download

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1398 bytes --]

On Fri, Apr 12, 2013 at 03:44:41PM -0400, Trevor Woerner wrote:
> Sorry for not noticing this sooner.
> 
> On Thu, Apr 11, 2013 at 9:30 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
> > My proposal is to add md5sum of downloaded file to ChecksumError and
> > rename it to <file>.<md5sum> in "if isinstance(e, ChecksumError):"
> 
> Would it be possible to make this behaviour configurable for those who
> might be concerned about disk space? Or perhaps while renaming and
> saving it, also move it into the $TMPDIR directory and/or tie this
> feature in with rm_work?
> 
> I'm just thinking that over time one's DL_DIR could potentially become
> rather cluttered with lots of failed downloads. Only some people would
> be interested in seeing the failures and the rest would have to remove
> them manually. I'm also thinking of the scenario where the DL_DIR is
> shared/sync'ed among a group of people/computers; that could add up to
> quite a bit of data to share.

I can make it configurable, but I think that bad-checksums should be
only in few exceptional cases, while package upgrades are common.

That said 90% of my downloads directory are old versions I keep just to
be able to recreate old builds with upstream sources already gone and
now I've one renamed bad-checksum tarball from evas-1.7.6.1...

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [bitbake-devel] Checksum failure encountered with download

[Richard Purdie]

On Fri, 2013-04-12 at 22:37 +0200, Martin Jansa wrote:
> On Fri, Apr 12, 2013 at 03:44:41PM -0400, Trevor Woerner wrote:
> > Sorry for not noticing this sooner.
> > 
> > On Thu, Apr 11, 2013 at 9:30 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
> > > My proposal is to add md5sum of downloaded file to ChecksumError and
> > > rename it to <file>.<md5sum> in "if isinstance(e, ChecksumError):"
> > 
> > Would it be possible to make this behaviour configurable for those who
> > might be concerned about disk space? Or perhaps while renaming and
> > saving it, also move it into the $TMPDIR directory and/or tie this
> > feature in with rm_work?
> > 
> > I'm just thinking that over time one's DL_DIR could potentially become
> > rather cluttered with lots of failed downloads. Only some people would
> > be interested in seeing the failures and the rest would have to remove
> > them manually. I'm also thinking of the scenario where the DL_DIR is
> > shared/sync'ed among a group of people/computers; that could add up to
> > quite a bit of data to share.
> 
> I can make it configurable, but I think that bad-checksums should be
> only in few exceptional cases, while package upgrades are common.
> 
> That said 90% of my downloads directory are old versions I keep just to
> be able to recreate old builds with upstream sources already gone and
> now I've one renamed bad-checksum tarball from evas-1.7.6.1...

Agreed, I'd hope this isn't something which happens that often.

Cheers,

Richard