Possible stale tags in the download directory

Possible stale tags in the download directory

[Ulf Samuelsson]

When I look at the "downloads" location I find a lot of files called.

"11_usagi_fix.patch.done"

but no  "11_usagi_fix.patch" file in the directory.

If this is a indication of that a patch has been downloaded,
what happens if you for some reason or other delete your
OE-core directory and keep the downloads directory.

When you start from fresh, all the tags are then stale.

(Deleting the "downloads" directory seems a bad idea)

I think that if there should be tags in the "downloads" directory,
they should only reflect things which are downloaded to the
"downloads" directory, and nothing else.

As for the tags in the directory, I think a better approach
is to download a file "tarball.tar.bz2" to a different filename first
I.E: "tarball.tar.bz2.in-progress"  and if the download completes
then move the file to "tarball.tar.bz2".
Should remove a lot of clutter from the "downloads" directory.

-- 
Best Regards
Ulf Samuelsson
eMagii

Re: Possible stale tags in the download directory

[Richard Purdie]

On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote:
> When I look at the "downloads" location I find a lot of files called.
> 
> "11_usagi_fix.patch.done"
> 
> but no  "11_usagi_fix.patch" file in the directory.
> 
> If this is a indication of that a patch has been downloaded,
> what happens if you for some reason or other delete your
> OE-core directory and keep the downloads directory.
> 
> When you start from fresh, all the tags are then stale.
> 
> (Deleting the "downloads" directory seems a bad idea)
> 
> I think that if there should be tags in the "downloads" directory,
> they should only reflect things which are downloaded to the
> "downloads" directory, and nothing else.
> 
> As for the tags in the directory, I think a better approach
> is to download a file "tarball.tar.bz2" to a different filename first
> I.E: "tarball.tar.bz2.in-progress"  and if the download completes
> then move the file to "tarball.tar.bz2".
> Should remove a lot of clutter from the "downloads" directory.

What we've tried to do is simplify the fetcher code paths in fetch2.
There were some many corner/special cases and different code paths it
was near impossible to tell what was going on. One of the side effects
is that local file:// urls do touch the done stamp in the same way as
other downloads. The main reason for those files is now checksum
tracking. If the done stamps were part of tmpdir, we'd keep checksumming
the contents of the downloads at each build rather than once after the
download. The way the implementation works, there is very little risk
from stale stamps, it would just mean the checksum code wouldn't get
triggered and that is likely unneeded for a local file:// url anyway.

So yes, its not ideal but looking at the code I'd be surprised if a
build ever broke due to it.

Cheers,

Richard

Re: Possible stale tags in the download directory

[Ulf Samuelsson]

On 2011-12-07 10:16, Richard Purdie wrote:
> On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote:
>> When I look at the "downloads" location I find a lot of files called.
>>
>> "11_usagi_fix.patch.done"
>>
>> but no  "11_usagi_fix.patch" file in the directory.
>>
>> If this is a indication of that a patch has been downloaded,
>> what happens if you for some reason or other delete your
>> OE-core directory and keep the downloads directory.
>>
>> When you start from fresh, all the tags are then stale.
>>
>> (Deleting the "downloads" directory seems a bad idea)
>>
>> I think that if there should be tags in the "downloads" directory,
>> they should only reflect things which are downloaded to the
>> "downloads" directory, and nothing else.
>>
>> As for the tags in the directory, I think a better approach
>> is to download a file "tarball.tar.bz2" to a different filename first
>> I.E: "tarball.tar.bz2.in-progress"  and if the download completes
>> then move the file to "tarball.tar.bz2".
>> Should remove a lot of clutter from the "downloads" directory.
> What we've tried to do is simplify the fetcher code paths in fetch2.
> There were some many corner/special cases and different code paths it
> was near impossible to tell what was going on. One of the side effects
> is that local file:// urls do touch the done stamp in the same way as
> other downloads. The main reason for those files is now checksum
> tracking. If the done stamps were part of tmpdir, we'd keep checksumming
> the contents of the downloads at each build rather than once after the
> download. The way the implementation works, there is very little risk
> from stale stamps, it would just mean the checksum code wouldn't get
> triggered and that is likely unneeded for a local file:// url anyway.
>
> So yes, its not ideal but looking at the code I'd be surprised if a
> build ever broke due to it.
>
Maybe I am paranoid, but I think I remember seeing tarballs developing
bit rot when moved between different machines.

Could of course clean out the downloads directory, but it would be
nice if the stamps would be in a different directory.
Then I just delete the directory when I move the tarballs.

I tend to use the "downloads" directory for other build systems as well,
to avoid duplication, and having stamps in the directory could have
real bad effects.

I agree that recomputing checksums for each build is not so good,
but doing it once after moving to a new machine, or for each new user
is acceptable (at least to me).

The stamps could be stored in a central location.

     "~/.oe/downloads"  ?

Con: If you have several repositories, maybe because you
forgot to configure for your central repository,
you may be in trouble.


> Cheers,
>
> Richard
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core


-- 
Best Regards
Ulf Samuelsson
eMagii

Re: Possible stale tags in the download directory

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 2239 bytes --]


Op 7 dec. 2011, om 12:27 heeft Ulf Samuelsson het volgende geschreven:

> On 2011-12-07 10:16, Richard Purdie wrote:
>> On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote:
>>> When I look at the "downloads" location I find a lot of files called.
>>> 
>>> "11_usagi_fix.patch.done"
>>> 
>>> but no  "11_usagi_fix.patch" file in the directory.
>>> 
>>> If this is a indication of that a patch has been downloaded,
>>> what happens if you for some reason or other delete your
>>> OE-core directory and keep the downloads directory.
>>> 
>>> When you start from fresh, all the tags are then stale.
>>> 
>>> (Deleting the "downloads" directory seems a bad idea)
>>> 
>>> I think that if there should be tags in the "downloads" directory,
>>> they should only reflect things which are downloaded to the
>>> "downloads" directory, and nothing else.
>>> 
>>> As for the tags in the directory, I think a better approach
>>> is to download a file "tarball.tar.bz2" to a different filename first
>>> I.E: "tarball.tar.bz2.in-progress"  and if the download completes
>>> then move the file to "tarball.tar.bz2".
>>> Should remove a lot of clutter from the "downloads" directory.
>> What we've tried to do is simplify the fetcher code paths in fetch2.
>> There were some many corner/special cases and different code paths it
>> was near impossible to tell what was going on. One of the side effects
>> is that local file:// urls do touch the done stamp in the same way as
>> other downloads. The main reason for those files is now checksum
>> tracking. If the done stamps were part of tmpdir, we'd keep checksumming
>> the contents of the downloads at each build rather than once after the
>> download. The way the implementation works, there is very little risk
>> from stale stamps, it would just mean the checksum code wouldn't get
>> triggered and that is likely unneeded for a local file:// url anyway.
>> 
>> So yes, its not ideal but looking at the code I'd be surprised if a
>> build ever broke due to it.
>> 
> Maybe I am paranoid, but I think I remember seeing tarballs developing
> bit rot when moved between different machines.

And that's why checksums are checked before do_unpack.



[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

Re: Possible stale tags in the download directory

[Ulf Samuelsson]

[-- Attachment #1: Type: text/plain, Size: 2752 bytes --]

On 2011-12-07 13:00, Koen Kooi wrote:
> Op 7 dec. 2011, om 12:27 heeft Ulf Samuelsson het volgende geschreven:
>
>> On 2011-12-07 10:16, Richard Purdie wrote:
>>> On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote:
>>>> When I look at the "downloads" location I find a lot of files called.
>>>>
>>>> "11_usagi_fix.patch.done"
>>>>
>>>> but no  "11_usagi_fix.patch" file in the directory.
>>>>
>>>> If this is a indication of that a patch has been downloaded,
>>>> what happens if you for some reason or other delete your
>>>> OE-core directory and keep the downloads directory.
>>>>
>>>> When you start from fresh, all the tags are then stale.
>>>>
>>>> (Deleting the "downloads" directory seems a bad idea)
>>>>
>>>> I think that if there should be tags in the "downloads" directory,
>>>> they should only reflect things which are downloaded to the
>>>> "downloads" directory, and nothing else.
>>>>
>>>> As for the tags in the directory, I think a better approach
>>>> is to download a file "tarball.tar.bz2" to a different filename first
>>>> I.E: "tarball.tar.bz2.in-progress"  and if the download completes
>>>> then move the file to "tarball.tar.bz2".
>>>> Should remove a lot of clutter from the "downloads" directory.
>>> What we've tried to do is simplify the fetcher code paths in fetch2.
>>> There were some many corner/special cases and different code paths it
>>> was near impossible to tell what was going on. One of the side effects
>>> is that local file:// urls do touch the done stamp in the same way as
>>> other downloads. The main reason for those files is now checksum
>>> tracking. If the done stamps were part of tmpdir, we'd keep checksumming
>>> the contents of the downloads at each build rather than once after the
>>> download. The way the implementation works, there is very little risk
>>> from stale stamps, it would just mean the checksum code wouldn't get
>>> triggered and that is likely unneeded for a local file:// url anyway.
>>>
>>> So yes, its not ideal but looking at the code I'd be surprised if a
>>> build ever broke due to it.
>>>
>> Maybe I am paranoid, but I think I remember seeing tarballs developing
>> bit rot when moved between different machines.
> And that's why checksums are checked before do_unpack.

Richard says that if the *.done file is there, then the checksum is not 
calculated.
Or that is at least how I interpret his comment.

If the check is always there, why the tag?



>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core


-- 
Best Regards
Ulf Samuelsson
eMagii


[-- Attachment #2: Type: text/html, Size: 4027 bytes --]

Re: Possible stale tags in the download directory

[Paul Eggleton]

On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote:
> Richard says that if the *.done file is there, then the checksum is not
> calculated.
> Or that is at least how I interpret his comment.
> 
> If the check is always there, why the tag?

That's not what my reading of the bitbake source says. 
bitbake/lib/bb/fetch2/__init__.py contains the only references to .done files 
and all it does is make sure the done file gets "touched" whenever a fetch for 
the associated file occurs; it does not otherwise check for its existence.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Possible stale tags in the download directory

[Richard Purdie]

On Wed, 2011-12-07 at 14:37 +0000, Paul Eggleton wrote:
> On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote:
> > Richard says that if the *.done file is there, then the checksum is not
> > calculated.
> > Or that is at least how I interpret his comment.
> > 
> > If the check is always there, why the tag?
> 
> That's not what my reading of the bitbake source says. 
> bitbake/lib/bb/fetch2/__init__.py contains the only references to .done files 
> and all it does is make sure the done file gets "touched" whenever a fetch for 
> the associated file occurs; it does not otherwise check for its existence.

It does two things, one is an activity marker useful for seeing whether
files are in active use and the second is for the checksum calculation
which only ever happens once after download due to the file (see
verify_checksum() in that file).

Cheers,

Richard

Re: Possible stale tags in the download directory

[Khem Raj]

On (07/12/11 15:00), Richard Purdie wrote:
> On Wed, 2011-12-07 at 14:37 +0000, Paul Eggleton wrote:
> > On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote:
> > > Richard says that if the *.done file is there, then the checksum is not
> > > calculated.
> > > Or that is at least how I interpret his comment.
> > > 
> > > If the check is always there, why the tag?
> > 
> > That's not what my reading of the bitbake source says. 
> > bitbake/lib/bb/fetch2/__init__.py contains the only references to .done files 
> > and all it does is make sure the done file gets "touched" whenever a fetch for 
> > the associated file occurs; it does not otherwise check for its existence.
> 
> It does two things, one is an activity marker useful for seeing whether
> files are in active use and the second is for the checksum calculation
> which only ever happens once after download due to the file (see
> verify_checksum() in that file).

when do we create .done files? if I am in middle of download of a tar
file and I cancel it and restart it does not fetch the file again but
tried to use the corrupt file. 

Is it a big build time hit if checksums are verified all the time ?

-- 
-Khem

Re: Possible stale tags in the download directory

[Ulf Samuelsson]

On 2011-12-07 22:43, Khem Raj wrote:
> On (07/12/11 15:00), Richard Purdie wrote:
>> On Wed, 2011-12-07 at 14:37 +0000, Paul Eggleton wrote:
>>> On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote:
>>>> Richard says that if the *.done file is there, then the checksum is not
>>>> calculated.
>>>> Or that is at least how I interpret his comment.
>>>>
>>>> If the check is always there, why the tag?
>>> That's not what my reading of the bitbake source says.
>>> bitbake/lib/bb/fetch2/__init__.py contains the only references to .done files
>>> and all it does is make sure the done file gets "touched" whenever a fetch for
>>> the associated file occurs; it does not otherwise check for its existence.
>> It does two things, one is an activity marker useful for seeing whether
>> files are in active use and the second is for the checksum calculation
>> which only ever happens once after download due to the file (see
>> verify_checksum() in that file).
> when do we create .done files? if I am in middle of download of a tar
> file and I cancel it and restart it does not fetch the file again but
> tried to use the corrupt file.
>
> Is it a big build time hit if checksums are verified all the time ?
>

I have seen the same problem.

-- 
Best Regards
Ulf Samuelsson
eMagii

Re: Possible stale tags in the download directory

[Richard Purdie]

On Wed, 2011-12-07 at 12:27 +0100, Ulf Samuelsson wrote:
> On 2011-12-07 10:16, Richard Purdie wrote:
> > On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote:
> >> When I look at the "downloads" location I find a lot of files called.
> >>
> >> "11_usagi_fix.patch.done"
> >>
> >> but no  "11_usagi_fix.patch" file in the directory.
> >>
> >> If this is a indication of that a patch has been downloaded,
> >> what happens if you for some reason or other delete your
> >> OE-core directory and keep the downloads directory.
> >>
> >> When you start from fresh, all the tags are then stale.
> >>
> >> (Deleting the "downloads" directory seems a bad idea)
> >>
> >> I think that if there should be tags in the "downloads" directory,
> >> they should only reflect things which are downloaded to the
> >> "downloads" directory, and nothing else.
> >>
> >> As for the tags in the directory, I think a better approach
> >> is to download a file "tarball.tar.bz2" to a different filename first
> >> I.E: "tarball.tar.bz2.in-progress"  and if the download completes
> >> then move the file to "tarball.tar.bz2".
> >> Should remove a lot of clutter from the "downloads" directory.
> > What we've tried to do is simplify the fetcher code paths in fetch2.
> > There were some many corner/special cases and different code paths it
> > was near impossible to tell what was going on. One of the side effects
> > is that local file:// urls do touch the done stamp in the same way as
> > other downloads. The main reason for those files is now checksum
> > tracking. If the done stamps were part of tmpdir, we'd keep checksumming
> > the contents of the downloads at each build rather than once after the
> > download. The way the implementation works, there is very little risk
> > from stale stamps, it would just mean the checksum code wouldn't get
> > triggered and that is likely unneeded for a local file:// url anyway.
> >
> > So yes, its not ideal but looking at the code I'd be surprised if a
> > build ever broke due to it.
> >
> Maybe I am paranoid, but I think I remember seeing tarballs developing
> bit rot when moved between different machines.

I don't understand what you mean here. If your process for copying files
between machines corrupts files, you have bigger problems. You can just
rm *.done at the same time if that is an issue.

> Could of course clean out the downloads directory, but it would be
> nice if the stamps would be in a different directory.
> Then I just delete the directory when I move the tarballs.
> 
> I tend to use the "downloads" directory for other build systems as well,
> to avoid duplication, and having stamps in the directory could have
> real bad effects.
> 
> I agree that recomputing checksums for each build is not so good,
> but doing it once after moving to a new machine, or for each new user
> is acceptable (at least to me).
> 
> The stamps could be stored in a central location.
> 
>      "~/.oe/downloads"  ?

I believe the data the .done files represent is directly related to the
files themselves and they should be kept together.

Nothing said here is convincing me there is a real world problem...

Cheers,

Richard