tslib keeps failing on checksum

tslib keeps failing on checksum

[Mike Looijmans]

I get this error every time I try t build the current oe-core master:

ERROR: Checksum failure fetching 
https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
ERROR: Function failed: Fetcher failure for URL: 
'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'. 
Checksum mismatch!
File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum 
d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was 
expected

Now the weird thing is, that when I fetch the thing using wget, I get a blob 
with the right checksum:

# wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
# md5sum tslib-1.1.tar.xz
14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz

Is oe-core using some other flavour of wget or what else could cause this kind 
of fetch error?

-- 
Mike Looijmans - TOPIC Automation


Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) – (0)499 - 33.69.79
Telefax: (+31) - (0)499 - 33.69.70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Dit e-mail bericht en de eventueel daarbij behorende bijlagen zijn uitsluitend bestemd voor de geadresseerde, zoals die blijkt uit het e-mail bericht en/of de bijlagen. Er kunnen gegevens met betrekking tot een derde instaan. Indien u als niet-geadresseerde dit bericht en de bijlagen ontvangt, terwijl u niet bevoegd of gemachtigd bent om dit bericht namens de geadresseerde te ontvangen, wordt u verzocht de afzender hierover direct te informeren en het e-mail bericht met de bijlagen te vernietigen. Ieder gebruik van de inhoud van het e-mail bericht, waaronder de daarbij behorende bijlagen, door een ander dan de geadresseerde is onrechtmatig jegens ons dan wel de eventueel in het e-mail bericht of de bijlagen voorkomende andere personen. TOPIC Embedded Systems is niet aansprakelijk voor enigerlei schade voortvloeiend uit het gebruik en/of acceptatie van dit e-mail bericht of de daarbij behorende bijlagen.

The contents of this message, as well as any enclosures, are addressed personally to, and thus solely intended for the addressee. They may contain information regarding a third party. A recipient who is neither the addressee, nor empowered to receive this message on behalf of the addressee, is kindly requested to immediately inform the sender of receipt, and to destroy the message and the enclosures. Any use of the contents of this message and/or the enclosures by any other person than the addressee or person who is empowered to receive this message, is illegal towards the sender and/or the aforementioned third party. TOPIC Embedded Systems is not  liable for any damage as a result of the use and/or acceptance of this message and as well as any enclosures.

Re: tslib keeps failing on checksum

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3545 bytes --]

On Wed, Nov 20, 2013 at 11:01:36AM +0100, Mike Looijmans wrote:
> I get this error every time I try t build the current oe-core master:
> 
> ERROR: Checksum failure fetching 
> https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
> ERROR: Function failed: Fetcher failure for URL: 
> 'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'. 
> Checksum mismatch!
> File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum 
> d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was 
> expected
> 
> Now the weird thing is, that when I fetch the thing using wget, I get a blob 
> with the right checksum:
> 
> # wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
> # md5sum tslib-1.1.tar.xz
> 14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
> 
> Is oe-core using some other flavour of wget or what else could cause this kind 
> of fetch error?

Have you tried to unpack both tarballs and compare content in them?

If you have older version of bitbake, then it could be that you have
broken tarball in downloads directory.

> 
> -- 
> Mike Looijmans - TOPIC Automation
> 
> 
> Met vriendelijke groet / kind regards,
> 
> Mike Looijmans
> 
> TOPIC Embedded Systems
> Eindhovenseweg 32-C, NL-5683 KH Best
> Postbus 440, NL-5680 AK Best
> Telefoon: (+31) – (0)499 - 33.69.79
> Telefax: (+31) - (0)499 - 33.69.70
> E-mail: mike.looijmans@topic.nl
> Website: www.topic.nl
> 
> Dit e-mail bericht en de eventueel daarbij behorende bijlagen zijn uitsluitend bestemd voor de geadresseerde, zoals die blijkt uit het e-mail bericht en/of de bijlagen. Er kunnen gegevens met betrekking tot een derde instaan. Indien u als niet-geadresseerde dit bericht en de bijlagen ontvangt, terwijl u niet bevoegd of gemachtigd bent om dit bericht namens de geadresseerde te ontvangen, wordt u verzocht de afzender hierover direct te informeren en het e-mail bericht met de bijlagen te vernietigen. Ieder gebruik van de inhoud van het e-mail bericht, waaronder de daarbij behorende bijlagen, door een ander dan de geadresseerde is onrechtmatig jegens ons dan wel de eventueel in het e-mail bericht of de bijlagen voorkomende andere personen. TOPIC Embedded Systems is niet aansprakelijk voor enigerlei schade voortvloeiend uit het gebruik en/of acceptatie van dit e-mail bericht of de daarbij behorende bijlagen.
> 
> The contents of this message, as well as any enclosures, are addressed personally to, and thus solely intended for the addressee. They may contain information regarding a third party. A recipient who is neither the addressee, nor empowered to receive this message on behalf of the addressee, is kindly requested to immediately inform the sender of receipt, and to destroy the message and the enclosures. Any use of the contents of this message and/or the enclosures by any other person than the addressee or person who is empowered to receive this message, is illegal towards the sender and/or the aforementioned third party. TOPIC Embedded Systems is not  liable for any damage as a result of the use and/or acceptance of this message and as well as any enclosures.
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: tslib keeps failing on checksum

[Gary Thomas]

On 2013-11-20 03:01, Mike Looijmans wrote:
> I get this error every time I try t build the current oe-core master:
>
> ERROR: Checksum failure fetching https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
> ERROR: Function failed: Fetcher failure for URL: 'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'. Checksum mismatch!
> File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was expected
>
> Now the weird thing is, that when I fetch the thing using wget, I get a blob with the right checksum:
>
> # wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
> # md5sum tslib-1.1.tar.xz
> 14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
>
> Is oe-core using some other flavour of wget or what else could cause this kind of fetch error?
>

What version of OE-core metadata are you using?  The recipe
   meta/recipes-graphics/tslib/tslib_1.1.bb
in OE-core rev 2d8e124adcf27af524eeeae61daf1b21a1c2f27c has
the correct md5sum, so you must have incorrect/stale recipe.

The recipe was last updated in this rev:
commit 2e92d845b433f3a1805c310ccda54cfc7dd8b1e1
Author: Paul Eggleton <paul.eggleton@linux.intel.com>
Date:   Fri Aug 16 14:33:12 2013 +0100

     tslib: update to 1.1

     Drop patches merged upstream. 32bitBE-support.patch wasn't merged, but
     no longer applies and similar changes look to have been made; tslib 1.1
     works properly on qemumips without it, so this has also been dropped.

     Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
     Signed-off-by: Saul Wold <sgw@linux.intel.com>

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: tslib keeps failing on checksum

[Mike Looijmans]

On 11/20/2013 11:38 AM, Martin Jansa wrote:
> On Wed, Nov 20, 2013 at 11:01:36AM +0100, Mike Looijmans wrote:
>> I get this error every time I try t build the current oe-core master:
>>
>> ERROR: Checksum failure fetching
>> https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
>> ERROR: Function failed: Fetcher failure for URL:
>> 'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.
>> Checksum mismatch!
>> File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum
>> d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was
>> expected
>>
>> Now the weird thing is, that when I fetch the thing using wget, I get a blob
>> with the right checksum:
>>
>> # wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
>> # md5sum tslib-1.1.tar.xz
>> 14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
>>
>> Is oe-core using some other flavour of wget or what else could cause this kind
>> of fetch error?
>
> Have you tried to unpack both tarballs and compare content in them?

Nope, I just assumed they're two somewhat different copies of the same 
software (e.g. one of them has a "downloaded from here" remark or so). 
Anything in particular I should look for?

> If you have older version of bitbake, then it could be that you have
> broken tarball in downloads directory.

I seemed to be using 1.19.1 (as reported on startup), so I've upgraded to the 
master which reports to be 1.21.0, cleaned the build dir, and will try again.

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) – (0)499 - 33.69.79
Telefax: (+31) - (0)499 - 33.69.70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Dit e-mail bericht en de eventueel daarbij behorende bijlagen zijn uitsluitend bestemd voor de geadresseerde, zoals die blijkt uit het e-mail bericht en/of de bijlagen. Er kunnen gegevens met betrekking tot een derde instaan. Indien u als niet-geadresseerde dit bericht en de bijlagen ontvangt, terwijl u niet bevoegd of gemachtigd bent om dit bericht namens de geadresseerde te ontvangen, wordt u verzocht de afzender hierover direct te informeren en het e-mail bericht met de bijlagen te vernietigen. Ieder gebruik van de inhoud van het e-mail bericht, waaronder de daarbij behorende bijlagen, door een ander dan de geadresseerde is onrechtmatig jegens ons dan wel de eventueel in het e-mail bericht of de bijlagen voorkomende andere personen. TOPIC Embedded Systems is niet aansprakelijk voor enigerlei schade voortvloeiend uit het gebruik en/of acceptatie van dit e-mail bericht of de daarbij behorende bijlagen.

The contents of this message, as well as any enclosures, are addressed personally to, and thus solely intended for the addressee. They may contain information regarding a third party. A recipient who is neither the addressee, nor empowered to receive this message on behalf of the addressee, is kindly requested to immediately inform the sender of receipt, and to destroy the message and the enclosures. Any use of the contents of this message and/or the enclosures by any other person than the addressee or person who is empowered to receive this message, is illegal towards the sender and/or the aforementioned third party. TOPIC Embedded Systems is not  liable for any damage as a result of the use and/or acceptance of this message and as well as any enclosures.

Re: tslib keeps failing on checksum

[Mike Looijmans]

On 11/20/2013 12:09 PM, Mike Looijmans wrote:
> On 11/20/2013 11:38 AM, Martin Jansa wrote:
>> On Wed, Nov 20, 2013 at 11:01:36AM +0100, Mike Looijmans wrote:
>>> I get this error every time I try t build the current oe-core master:
>>>
>>> ERROR: Checksum failure fetching
>>> https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
>>>
>>> ERROR: Function failed: Fetcher failure for URL:
>>> 'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.
>>>
>>> Checksum mismatch!
>>> File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum
>>> d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was
>>> expected
>>>
>>> Now the weird thing is, that when I fetch the thing using wget, I get ablob
>>> with the right checksum:
>>>
>>> # wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
>>> # md5sum tslib-1.1.tar.xz
>>> 14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
>>>
>>> Is oe-core using some other flavour of wget or what else could cause this kind
>>> of fetch error?
>>
>> Have you tried to unpack both tarballs and compare content in them?
>
> Nope, I just assumed they're two somewhat different copies of the same
> software (e.g. one of them has a "downloaded from here" remark or so).
> Anything in particular I should look for?

After upgrading I tried again. The file as downloaded by bitbake is completely
empty. Nothing in it. The md5 sum of this empty file is
d41d8cd98f00b204e9800998ecf8427e indeed.

I'm now using bitbake 1.21.0 (current master) and OE rev
0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results
still.

(I know I can just move my own file into the download directory and get on
with it, but i'd rather actually solve this problem).

There's a direct connection to the internet, no proxy (just a router) that
might have been caching things.

Any suggestions?

Mike.




Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) – (0)499 - 33.69.79
Telefax: (+31) - (0)499 - 33.69.70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Dit e-mail bericht en de eventueel daarbij behorende bijlagen zijn uitsluitend bestemd voor de geadresseerde, zoals die blijkt uit het e-mail bericht en/of de bijlagen. Er kunnen gegevens met betrekking tot een derde instaan. Indien u als niet-geadresseerde dit bericht en de bijlagen ontvangt, terwijl u niet bevoegd of gemachtigd bent om dit bericht namens de geadresseerde te ontvangen, wordt u verzocht de afzender hierover direct te informeren en het e-mail bericht met de bijlagen te vernietigen. Ieder gebruik van de inhoud van het e-mail bericht, waaronder de daarbij behorende bijlagen, door een ander dan de geadresseerde is onrechtmatig jegens ons dan wel de eventueel in het e-mail bericht of de bijlagen voorkomende andere personen. TOPIC Embedded Systems is niet aansprakelijk voor enigerlei schade voortvloeiend uit het gebruik en/of acceptatie van dit e-mail bericht of de daarbij behorende bijlagen.

The contents of this message, as well as any enclosures, are addressed personally to, and thus solely intended for the addressee. They may contain information regarding a third party. A recipient who is neither the addressee, nor empowered to receive this message on behalf of the addressee, is kindly requested to immediately inform the sender of receipt, and to destroy the message and the enclosures. Any use of the contents of this message and/or the enclosures by any other person than the addressee or person who is empowered to receive this message, is illegal towards the sender and/or the aforementioned third party. TOPIC Embedded Systems is not  liable for any damage as a result of the use and/or acceptance of this message and as well as any enclosures.

Re: tslib keeps failing on checksum

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2933 bytes --]

On Wed, Nov 20, 2013 at 01:02:27PM +0100, Mike Looijmans wrote:
> On 11/20/2013 12:09 PM, Mike Looijmans wrote:
> > On 11/20/2013 11:38 AM, Martin Jansa wrote:
> >> On Wed, Nov 20, 2013 at 11:01:36AM +0100, Mike Looijmans wrote:
> >>> I get this error every time I try t build the current oe-core master:
> >>>
> >>> ERROR: Checksum failure fetching
> >>> https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
> >>>
> >>> ERROR: Function failed: Fetcher failure for URL:
> >>> 'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.
> >>>
> >>> Checksum mismatch!
> >>> File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum
> >>> d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was
> >>> expected
> >>>
> >>> Now the weird thing is, that when I fetch the thing using wget, I get ablob
> >>> with the right checksum:
> >>>
> >>> # wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
> >>> # md5sum tslib-1.1.tar.xz
> >>> 14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
> >>>
> >>> Is oe-core using some other flavour of wget or what else could cause this kind
> >>> of fetch error?
> >>
> >> Have you tried to unpack both tarballs and compare content in them?
> >
> > Nope, I just assumed they're two somewhat different copies of the same
> > software (e.g. one of them has a "downloaded from here" remark or so).
> > Anything in particular I should look for?

Any difference is important, sometimes you can find there HTML code from some
proxy or server which generates HTML page instead of 404 or someone
trying to sell you domain of long dead project etc

> After upgrading I tried again. The file as downloaded by bitbake is completely
> empty. Nothing in it. The md5 sum of this empty file is
> d41d8cd98f00b204e9800998ecf8427e indeed.
> 
> I'm now using bitbake 1.21.0 (current master) and OE rev
> 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results
> still.

OK, I was asking mostly because newer bitbake (1.19+ is new enough)
renames file with bad checksum, moving corrupted download aside so it
doesn't stand in way for new one.

> (I know I can just move my own file into the download directory and get on
> with it, but i'd rather actually solve this problem).
> 
> There's a direct connection to the internet, no proxy (just a router) that
> might have been caching things.
> 
> Any suggestions?

Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only
the original URL or also possible (PRE)MIRROR url from which it could
download that empty file.

You should see the exact URL in log.do_fetch.

You can also check for (PRE)MIRROR usage in bitbake -e tslib output
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: tslib keeps failing on checksum

[Mike Looijmans]

On 11/20/2013 01:29 PM, Martin Jansa wrote:
> On Wed, Nov 20, 2013 at 01:02:27PM +0100, Mike Looijmans wrote:
>> On 11/20/2013 12:09 PM, Mike Looijmans wrote:
>>> On 11/20/2013 11:38 AM, Martin Jansa wrote:
>>>> On Wed, Nov 20, 2013 at 11:01:36AM +0100, Mike Looijmans wrote:
>>>>> I get this error every time I try t build the current oe-core master:
>>>>>
>>>>> ERROR: Checksum failure fetching
>>>>> https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
>>>>>
>>>>> ERROR: Function failed: Fetcher failure for URL:
>>>>> 'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.
>>>>>
>>>>> Checksum mismatch!
>>>>> File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum
>>>>> d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was
>>>>> expected
>>>>>
>>>>> Now the weird thing is, that when I fetch the thing using wget, I get ablob
>>>>> with the right checksum:
>>>>>
>>>>> # wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
>>>>> # md5sum tslib-1.1.tar.xz
>>>>> 14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
>>>>>
>>>>> Is oe-core using some other flavour of wget or what else could cause this kind
>>>>> of fetch error?
>>>>
>>>> Have you tried to unpack both tarballs and compare content in them?
>>>
>>> Nope, I just assumed they're two somewhat different copies of the same
>>> software (e.g. one of them has a "downloaded from here" remark or so).
>>> Anything in particular I should look for?
>
> Any difference is important, sometimes you can find there HTML code from some
> proxy or server which generates HTML page instead of 404 or someone
> trying to sell you domain of long dead project etc
>
>> After upgrading I tried again. The file as downloaded by bitbake is completely
>> empty. Nothing in it. The md5 sum of this empty file is
>> d41d8cd98f00b204e9800998ecf8427e indeed.
>>
>> I'm now using bitbake 1.21.0 (current master) and OE rev
>> 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results
>> still.
>
> OK, I was asking mostly because newer bitbake (1.19+ is new enough)
> renames file with bad checksum, moving corrupted download aside so it
> doesn't stand in way for new one.
>
>> (I know I can just move my own file into the download directory and get on
>> with it, but i'd rather actually solve this problem).
>>
>> There's a direct connection to the internet, no proxy (just a router) that
>> might have been caching things.
>>
>> Any suggestions?
>
> Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only
> the original URL or also possible (PRE)MIRROR url from which it could
> download that empty file.
>
> You should see the exact URL in log.do_fetch.

The problem seems to be that I am "my own mirror". There's a http server that 
serves sstate-cache and downloads to the rest of the company. And via an NFS 
mount, that dir is also linked to my local downloads directory. So bitbake 
ended up executing:

/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -O 
/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz -P 
/home/mike/zynq-next/build/downloads 
'http://192.168.80.24/sources/tslib-1.1.tar.xz'

This would create an empty file "tslib-1.1.tar.xz" and the http server happily 
returns that empty file, and then bitbake thinks it all went well.

I removed the link to the NFS mount, and now it works.

Weird that only tslib suffers from this. Then again, it's probably just the 
only package that wasn't readily available.

Is this my mistake and don't do this again, or should there be some protection 
against fetching your own files? (for example, by NOT using the filename 
itself as a target, but download with a ".part" extension and then rename when 
done. This also helps when multible builds share the download dir).

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) – (0)499 - 33.69.79
Telefax: (+31) - (0)499 - 33.69.70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Dit e-mail bericht en de eventueel daarbij behorende bijlagen zijn uitsluitend bestemd voor de geadresseerde, zoals die blijkt uit het e-mail bericht en/of de bijlagen. Er kunnen gegevens met betrekking tot een derde instaan. Indien u als niet-geadresseerde dit bericht en de bijlagen ontvangt, terwijl u niet bevoegd of gemachtigd bent om dit bericht namens de geadresseerde te ontvangen, wordt u verzocht de afzender hierover direct te informeren en het e-mail bericht met de bijlagen te vernietigen. Ieder gebruik van de inhoud van het e-mail bericht, waaronder de daarbij behorende bijlagen, door een ander dan de geadresseerde is onrechtmatig jegens ons dan wel de eventueel in het e-mail bericht of de bijlagen voorkomende andere personen. TOPIC Embedded Systems is niet aansprakelijk voor enigerlei schade voortvloeiend uit het gebruik en/of acceptatie van dit e-mail bericht of de daarbij behorende bijlagen.

The contents of this message, as well as any enclosures, are addressed personally to, and thus solely intended for the addressee. They may contain information regarding a third party. A recipient who is neither the addressee, nor empowered to receive this message on behalf of the addressee, is kindly requested to immediately inform the sender of receipt, and to destroy the message and the enclosures. Any use of the contents of this message and/or the enclosures by any other person than the addressee or person who is empowered to receive this message, is illegal towards the sender and/or the aforementioned third party. TOPIC Embedded Systems is not  liable for any damage as a result of the use and/or acceptance of this message and as well as any enclosures.

Re: tslib keeps failing on checksum

[Richard Purdie]

On Wed, 2013-11-20 at 13:02 +0100, Mike Looijmans wrote:
> On 11/20/2013 12:09 PM, Mike Looijmans wrote:
> > On 11/20/2013 11:38 AM, Martin Jansa wrote:
> >> On Wed, Nov 20, 2013 at 11:01:36AM +0100, Mike Looijmans wrote:
> >>> I get this error every time I try t build the current oe-core master:
> >>>
> >>> ERROR: Checksum failure fetching
> >>> https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
> >>>
> >>> ERROR: Function failed: Fetcher failure for URL:
> >>> 'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.
> >>>
> >>> Checksum mismatch!
> >>> File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum
> >>> d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was
> >>> expected
> >>>
> >>> Now the weird thing is, that when I fetch the thing using wget, I get ablob
> >>> with the right checksum:
> >>>
> >>> # wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
> >>> # md5sum tslib-1.1.tar.xz
> >>> 14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
> >>>
> >>> Is oe-core using some other flavour of wget or what else could cause this kind
> >>> of fetch error?
> >>
> >> Have you tried to unpack both tarballs and compare content in them?
> >
> > Nope, I just assumed they're two somewhat different copies of the same
> > software (e.g. one of them has a "downloaded from here" remark or so).
> > Anything in particular I should look for?
> 
> After upgrading I tried again. The file as downloaded by bitbake is completely
> empty. Nothing in it. The md5 sum of this empty file is
> d41d8cd98f00b204e9800998ecf8427e indeed.
> 
> I'm now using bitbake 1.21.0 (current master) and OE rev
> 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results
> still.
> 
> (I know I can just move my own file into the download directory and get on
> with it, but i'd rather actually solve this problem).
> 
> There's a direct connection to the internet, no proxy (just a router) that
> might have been caching things.
> 
> Any suggestions?

Can you share the tslib do_fetch logs (all of them). I'm interested to
see what happened historically in the build...

Cheers,

Richard

Re: tslib keeps failing on checksum

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 6549 bytes --]

On Wed, Nov 20, 2013 at 01:45:14PM +0100, Mike Looijmans wrote:
> On 11/20/2013 01:29 PM, Martin Jansa wrote:
> > On Wed, Nov 20, 2013 at 01:02:27PM +0100, Mike Looijmans wrote:
> >> On 11/20/2013 12:09 PM, Mike Looijmans wrote:
> >>> On 11/20/2013 11:38 AM, Martin Jansa wrote:
> >>>> On Wed, Nov 20, 2013 at 11:01:36AM +0100, Mike Looijmans wrote:
> >>>>> I get this error every time I try t build the current oe-core master:
> >>>>>
> >>>>> ERROR: Checksum failure fetching
> >>>>> https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
> >>>>>
> >>>>> ERROR: Function failed: Fetcher failure for URL:
> >>>>> 'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.
> >>>>>
> >>>>> Checksum mismatch!
> >>>>> File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum
> >>>>> d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was
> >>>>> expected
> >>>>>
> >>>>> Now the weird thing is, that when I fetch the thing using wget, I get ablob
> >>>>> with the right checksum:
> >>>>>
> >>>>> # wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
> >>>>> # md5sum tslib-1.1.tar.xz
> >>>>> 14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
> >>>>>
> >>>>> Is oe-core using some other flavour of wget or what else could cause this kind
> >>>>> of fetch error?
> >>>>
> >>>> Have you tried to unpack both tarballs and compare content in them?
> >>>
> >>> Nope, I just assumed they're two somewhat different copies of the same
> >>> software (e.g. one of them has a "downloaded from here" remark or so).
> >>> Anything in particular I should look for?
> >
> > Any difference is important, sometimes you can find there HTML code from some
> > proxy or server which generates HTML page instead of 404 or someone
> > trying to sell you domain of long dead project etc
> >
> >> After upgrading I tried again. The file as downloaded by bitbake is completely
> >> empty. Nothing in it. The md5 sum of this empty file is
> >> d41d8cd98f00b204e9800998ecf8427e indeed.
> >>
> >> I'm now using bitbake 1.21.0 (current master) and OE rev
> >> 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results
> >> still.
> >
> > OK, I was asking mostly because newer bitbake (1.19+ is new enough)
> > renames file with bad checksum, moving corrupted download aside so it
> > doesn't stand in way for new one.
> >
> >> (I know I can just move my own file into the download directory and get on
> >> with it, but i'd rather actually solve this problem).
> >>
> >> There's a direct connection to the internet, no proxy (just a router) that
> >> might have been caching things.
> >>
> >> Any suggestions?
> >
> > Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only
> > the original URL or also possible (PRE)MIRROR url from which it could
> > download that empty file.
> >
> > You should see the exact URL in log.do_fetch.
> 
> The problem seems to be that I am "my own mirror". There's a http server that 
> serves sstate-cache and downloads to the rest of the company. And via an NFS 
> mount, that dir is also linked to my local downloads directory. So bitbake 
> ended up executing:
> 
> /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -O 
> /home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz -P 
> /home/mike/zynq-next/build/downloads 
> 'http://192.168.80.24/sources/tslib-1.1.tar.xz'
> 
> This would create an empty file "tslib-1.1.tar.xz" and the http server happily 
> returns that empty file, and then bitbake thinks it all went well.
> 
> I removed the link to the NFS mount, and now it works.

If I understand you correctly your DL_DIR and (PRE)MIRROR point to
physically the same directory, correct?

So there is no point to set "duplicate" (PRE)MIRROR on this builder and
then it should work fine.

> Weird that only tslib suffers from this. Then again, it's probably just the 
> only package that wasn't readily available.
> 
> Is this my mistake and don't do this again, or should there be some protection 
> against fetching your own files? (for example, by NOT using the filename 
> itself as a target, but download with a ".part" extension and then rename when 
> done. This also helps when multible builds share the download dir).
> 
> Mike.
> 
> 
> Met vriendelijke groet / kind regards,
> 
> Mike Looijmans
> 
> TOPIC Embedded Systems
> Eindhovenseweg 32-C, NL-5683 KH Best
> Postbus 440, NL-5680 AK Best
> Telefoon: (+31) – (0)499 - 33.69.79
> Telefax: (+31) - (0)499 - 33.69.70
> E-mail: mike.looijmans@topic.nl
> Website: www.topic.nl
> 
> Dit e-mail bericht en de eventueel daarbij behorende bijlagen zijn uitsluitend bestemd voor de geadresseerde, zoals die blijkt uit het e-mail bericht en/of de bijlagen. Er kunnen gegevens met betrekking tot een derde instaan. Indien u als niet-geadresseerde dit bericht en de bijlagen ontvangt, terwijl u niet bevoegd of gemachtigd bent om dit bericht namens de geadresseerde te ontvangen, wordt u verzocht de afzender hierover direct te informeren en het e-mail bericht met de bijlagen te vernietigen. Ieder gebruik van de inhoud van het e-mail bericht, waaronder de daarbij behorende bijlagen, door een ander dan de geadresseerde is onrechtmatig jegens ons dan wel de eventueel in het e-mail bericht of de bijlagen voorkomende andere personen. TOPIC Embedded Systems is niet aansprakelijk voor enigerlei schade voortvloeiend uit het gebruik en/of acceptatie van dit e-mail bericht of de daarbij behorende bijlagen.
> 
> The contents of this message, as well as any enclosures, are addressed personally to, and thus solely intended for the addressee. They may contain information regarding a third party. A recipient who is neither the addressee, nor empowered to receive this message on behalf of the addressee, is kindly requested to immediately inform the sender of receipt, and to destroy the message and the enclosures. Any use of the contents of this message and/or the enclosures by any other person than the addressee or person who is empowered to receive this message, is illegal towards the sender and/or the aforementioned third party. TOPIC Embedded Systems is not  liable for any damage as a result of the use and/or acceptance of this message and as well as any enclosures.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: tslib keeps failing on checksum

[Richard Purdie]

On Wed, 2013-11-20 at 13:45 +0100, Mike Looijmans wrote:
> On 11/20/2013 01:29 PM, Martin Jansa wrote:
> > On Wed, Nov 20, 2013 at 01:02:27PM +0100, Mike Looijmans wrote:
> >> On 11/20/2013 12:09 PM, Mike Looijmans wrote:
> >>> On 11/20/2013 11:38 AM, Martin Jansa wrote:
> >>>> On Wed, Nov 20, 2013 at 11:01:36AM +0100, Mike Looijmans wrote:
> >>>>> I get this error every time I try t build the current oe-core master:
> >>>>>
> >>>>> ERROR: Checksum failure fetching
> >>>>> https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
> >>>>>
> >>>>> ERROR: Function failed: Fetcher failure for URL:
> >>>>> 'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.
> >>>>>
> >>>>> Checksum mismatch!
> >>>>> File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum
> >>>>> d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was
> >>>>> expected
> >>>>>
> >>>>> Now the weird thing is, that when I fetch the thing using wget, I get ablob
> >>>>> with the right checksum:
> >>>>>
> >>>>> # wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
> >>>>> # md5sum tslib-1.1.tar.xz
> >>>>> 14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
> >>>>>
> >>>>> Is oe-core using some other flavour of wget or what else could cause this kind
> >>>>> of fetch error?
> >>>>
> >>>> Have you tried to unpack both tarballs and compare content in them?
> >>>
> >>> Nope, I just assumed they're two somewhat different copies of the same
> >>> software (e.g. one of them has a "downloaded from here" remark or so).
> >>> Anything in particular I should look for?
> >
> > Any difference is important, sometimes you can find there HTML code from some
> > proxy or server which generates HTML page instead of 404 or someone
> > trying to sell you domain of long dead project etc
> >
> >> After upgrading I tried again. The file as downloaded by bitbake is completely
> >> empty. Nothing in it. The md5 sum of this empty file is
> >> d41d8cd98f00b204e9800998ecf8427e indeed.
> >>
> >> I'm now using bitbake 1.21.0 (current master) and OE rev
> >> 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results
> >> still.
> >
> > OK, I was asking mostly because newer bitbake (1.19+ is new enough)
> > renames file with bad checksum, moving corrupted download aside so it
> > doesn't stand in way for new one.
> >
> >> (I know I can just move my own file into the download directory and get on
> >> with it, but i'd rather actually solve this problem).
> >>
> >> There's a direct connection to the internet, no proxy (just a router) that
> >> might have been caching things.
> >>
> >> Any suggestions?
> >
> > Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only
> > the original URL or also possible (PRE)MIRROR url from which it could
> > download that empty file.
> >
> > You should see the exact URL in log.do_fetch.
> 
> The problem seems to be that I am "my own mirror". There's a http server that 
> serves sstate-cache and downloads to the rest of the company. And via an NFS 
> mount, that dir is also linked to my local downloads directory. So bitbake 
> ended up executing:
> 
> /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -O 
> /home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz -P 
> /home/mike/zynq-next/build/downloads 
> 'http://192.168.80.24/sources/tslib-1.1.tar.xz'
> 
> This would create an empty file "tslib-1.1.tar.xz" and the http server happily 
> returns that empty file, and then bitbake thinks it all went well.
> 
> I removed the link to the NFS mount, and now it works.
> 
> Weird that only tslib suffers from this. Then again, it's probably just the 
> only package that wasn't readily available.
> 
> Is this my mistake and don't do this again, or should there be some protection 
> against fetching your own files? (for example, by NOT using the filename 
> itself as a target, but download with a ".part" extension and then rename when 
> done. This also helps when multible builds share the download dir).

We certainly don't support circular references like this. As you
mention, we can work around some parts with the .part approach but for
things like git repositories for example things are much harder to
ensure they work.

I'm not even sure how we'd detect this kind of situation to be honest,
I'm open to ideas though.

Sharing a "live" DL_DIR is probably a bad idea as files may be
incomplete, etc. The checksums do at least help catch most of the
problems.

Cheers,

Richard

Re: tslib keeps failing on checksum

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 5647 bytes --]

On Wed, Nov 20, 2013 at 01:43:45PM +0000, Richard Purdie wrote:
> On Wed, 2013-11-20 at 13:45 +0100, Mike Looijmans wrote:
> > On 11/20/2013 01:29 PM, Martin Jansa wrote:
> > > On Wed, Nov 20, 2013 at 01:02:27PM +0100, Mike Looijmans wrote:
> > >> On 11/20/2013 12:09 PM, Mike Looijmans wrote:
> > >>> On 11/20/2013 11:38 AM, Martin Jansa wrote:
> > >>>> On Wed, Nov 20, 2013 at 11:01:36AM +0100, Mike Looijmans wrote:
> > >>>>> I get this error every time I try t build the current oe-core master:
> > >>>>>
> > >>>>> ERROR: Checksum failure fetching
> > >>>>> https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
> > >>>>>
> > >>>>> ERROR: Function failed: Fetcher failure for URL:
> > >>>>> 'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.
> > >>>>>
> > >>>>> Checksum mismatch!
> > >>>>> File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum
> > >>>>> d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was
> > >>>>> expected
> > >>>>>
> > >>>>> Now the weird thing is, that when I fetch the thing using wget, I get ablob
> > >>>>> with the right checksum:
> > >>>>>
> > >>>>> # wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
> > >>>>> # md5sum tslib-1.1.tar.xz
> > >>>>> 14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
> > >>>>>
> > >>>>> Is oe-core using some other flavour of wget or what else could cause this kind
> > >>>>> of fetch error?
> > >>>>
> > >>>> Have you tried to unpack both tarballs and compare content in them?
> > >>>
> > >>> Nope, I just assumed they're two somewhat different copies of the same
> > >>> software (e.g. one of them has a "downloaded from here" remark or so).
> > >>> Anything in particular I should look for?
> > >
> > > Any difference is important, sometimes you can find there HTML code from some
> > > proxy or server which generates HTML page instead of 404 or someone
> > > trying to sell you domain of long dead project etc
> > >
> > >> After upgrading I tried again. The file as downloaded by bitbake is completely
> > >> empty. Nothing in it. The md5 sum of this empty file is
> > >> d41d8cd98f00b204e9800998ecf8427e indeed.
> > >>
> > >> I'm now using bitbake 1.21.0 (current master) and OE rev
> > >> 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results
> > >> still.
> > >
> > > OK, I was asking mostly because newer bitbake (1.19+ is new enough)
> > > renames file with bad checksum, moving corrupted download aside so it
> > > doesn't stand in way for new one.
> > >
> > >> (I know I can just move my own file into the download directory and get on
> > >> with it, but i'd rather actually solve this problem).
> > >>
> > >> There's a direct connection to the internet, no proxy (just a router) that
> > >> might have been caching things.
> > >>
> > >> Any suggestions?
> > >
> > > Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only
> > > the original URL or also possible (PRE)MIRROR url from which it could
> > > download that empty file.
> > >
> > > You should see the exact URL in log.do_fetch.
> > 
> > The problem seems to be that I am "my own mirror". There's a http server that 
> > serves sstate-cache and downloads to the rest of the company. And via an NFS 
> > mount, that dir is also linked to my local downloads directory. So bitbake 
> > ended up executing:
> > 
> > /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -O 
> > /home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz -P 
> > /home/mike/zynq-next/build/downloads 
> > 'http://192.168.80.24/sources/tslib-1.1.tar.xz'
> > 
> > This would create an empty file "tslib-1.1.tar.xz" and the http server happily 
> > returns that empty file, and then bitbake thinks it all went well.
> > 
> > I removed the link to the NFS mount, and now it works.
> > 
> > Weird that only tslib suffers from this. Then again, it's probably just the 
> > only package that wasn't readily available.
> > 
> > Is this my mistake and don't do this again, or should there be some protection 
> > against fetching your own files? (for example, by NOT using the filename 
> > itself as a target, but download with a ".part" extension and then rename when 
> > done. This also helps when multible builds share the download dir).
> 
> We certainly don't support circular references like this. As you
> mention, we can work around some parts with the .part approach but for
> things like git repositories for example things are much harder to
> ensure they work.
> 
> I'm not even sure how we'd detect this kind of situation to be honest,
> I'm open to ideas though.
> 
> Sharing a "live" DL_DIR is probably a bad idea as files may be
> incomplete, etc. The checksums do at least help catch most of the
> problems.

Aren't .lock and .done stampfiles supposed to prevent downloading
incomplete files when it's shared over NFS between multiple builders?

I haven't tried it in practise yet, but I was planing to convert all our
jenkins builders to share common live DL_DIR/SSTATE_DIR, so that first
builder to fetch or build something saves the work for other slaves.

I know that especially for sstate archives it could be too late when 2
builders are already in runqueue phase, but still it could be a bit
sooner than waiting for whole build to finish and rsync
DL_DIR/SSTATE_DIR after the build.

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: tslib keeps failing on checksum

[Richard Purdie]

On Wed, 2013-11-20 at 16:18 +0100, Martin Jansa wrote:
> On Wed, Nov 20, 2013 at 01:43:45PM +0000, Richard Purdie wrote:
> > On Wed, 2013-11-20 at 13:45 +0100, Mike Looijmans wrote:
> > > On 11/20/2013 01:29 PM, Martin Jansa wrote:
> > > > On Wed, Nov 20, 2013 at 01:02:27PM +0100, Mike Looijmans wrote:
> > > >> On 11/20/2013 12:09 PM, Mike Looijmans wrote:
> > > > Any difference is important, sometimes you can find there HTML code from some
> > > > proxy or server which generates HTML page instead of 404 or someone
> > > > trying to sell you domain of long dead project etc
> > > >
> > > >> After upgrading I tried again. The file as downloaded by bitbake is completely
> > > >> empty. Nothing in it. The md5 sum of this empty file is
> > > >> d41d8cd98f00b204e9800998ecf8427e indeed.
> > > >>
> > > >> I'm now using bitbake 1.21.0 (current master) and OE rev
> > > >> 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results
> > > >> still.
> > > >
> > > > OK, I was asking mostly because newer bitbake (1.19+ is new enough)
> > > > renames file with bad checksum, moving corrupted download aside so it
> > > > doesn't stand in way for new one.
> > > >
> > > >> (I know I can just move my own file into the download directory and get on
> > > >> with it, but i'd rather actually solve this problem).
> > > >>
> > > >> There's a direct connection to the internet, no proxy (just a router) that
> > > >> might have been caching things.
> > > >>
> > > >> Any suggestions?
> > > >
> > > > Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only
> > > > the original URL or also possible (PRE)MIRROR url from which it could
> > > > download that empty file.
> > > >
> > > > You should see the exact URL in log.do_fetch.
> > > 
> > > The problem seems to be that I am "my own mirror". There's a http server that 
> > > serves sstate-cache and downloads to the rest of the company. And via an NFS 
> > > mount, that dir is also linked to my local downloads directory. So bitbake 
> > > ended up executing:
> > > 
> > > /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -O 
> > > /home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz -P 
> > > /home/mike/zynq-next/build/downloads 
> > > 'http://192.168.80.24/sources/tslib-1.1.tar.xz'
> > > 
> > > This would create an empty file "tslib-1.1.tar.xz" and the http server happily 
> > > returns that empty file, and then bitbake thinks it all went well.
> > > 
> > > I removed the link to the NFS mount, and now it works.
> > > 
> > > Weird that only tslib suffers from this. Then again, it's probably just the 
> > > only package that wasn't readily available.
> > > 
> > > Is this my mistake and don't do this again, or should there be some protection 
> > > against fetching your own files? (for example, by NOT using the filename 
> > > itself as a target, but download with a ".part" extension and then rename when 
> > > done. This also helps when multible builds share the download dir).
> > 
> > We certainly don't support circular references like this. As you
> > mention, we can work around some parts with the .part approach but for
> > things like git repositories for example things are much harder to
> > ensure they work.
> > 
> > I'm not even sure how we'd detect this kind of situation to be honest,
> > I'm open to ideas though.
> > 
> > Sharing a "live" DL_DIR is probably a bad idea as files may be
> > incomplete, etc. The checksums do at least help catch most of the
> > problems.
> 
> Aren't .lock and .done stampfiles supposed to prevent downloading
> incomplete files when it's shared over NFS between multiple builders?

Yes, this was not an NFS case though. This was where DL_DIR was being
served out over http from the same directory and that http address was
listed in PREMIRRORS/MIRRORS. Locking doesn't work over http :)

> I haven't tried it in practise yet, but I was planing to convert all our
> jenkins builders to share common live DL_DIR/SSTATE_DIR, so that first
> builder to fetch or build something saves the work for other slaves.

Right, we try and be safe over NFS so assuming your NFS locking works,
this should work fine. We atomically move sstate files into place to
ensure that is safe.

> I know that especially for sstate archives it could be too late when 2
> builders are already in runqueue phase, but still it could be a bit
> sooner than waiting for whole build to finish and rsync
> DL_DIR/SSTATE_DIR after the build.

Agreed and this is how the Yocto Project autobuilders are setup for that
reason.

Cheers,

Richard