* Possible stale tags in the download directory @ 2011-12-07 8:27 Ulf Samuelsson 2011-12-07 9:16 ` Richard Purdie 0 siblings, 1 reply; 10+ messages in thread From: Ulf Samuelsson @ 2011-12-07 8:27 UTC (permalink / raw) To: openembedded-core When I look at the "downloads" location I find a lot of files called. "11_usagi_fix.patch.done" but no "11_usagi_fix.patch" file in the directory. If this is a indication of that a patch has been downloaded, what happens if you for some reason or other delete your OE-core directory and keep the downloads directory. When you start from fresh, all the tags are then stale. (Deleting the "downloads" directory seems a bad idea) I think that if there should be tags in the "downloads" directory, they should only reflect things which are downloaded to the "downloads" directory, and nothing else. As for the tags in the directory, I think a better approach is to download a file "tarball.tar.bz2" to a different filename first I.E: "tarball.tar.bz2.in-progress" and if the download completes then move the file to "tarball.tar.bz2". Should remove a lot of clutter from the "downloads" directory. -- Best Regards Ulf Samuelsson eMagii ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Possible stale tags in the download directory 2011-12-07 8:27 Possible stale tags in the download directory Ulf Samuelsson @ 2011-12-07 9:16 ` Richard Purdie 2011-12-07 11:27 ` Ulf Samuelsson 0 siblings, 1 reply; 10+ messages in thread From: Richard Purdie @ 2011-12-07 9:16 UTC (permalink / raw) To: ulf, Patches and discussions about the oe-core layer On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote: > When I look at the "downloads" location I find a lot of files called. > > "11_usagi_fix.patch.done" > > but no "11_usagi_fix.patch" file in the directory. > > If this is a indication of that a patch has been downloaded, > what happens if you for some reason or other delete your > OE-core directory and keep the downloads directory. > > When you start from fresh, all the tags are then stale. > > (Deleting the "downloads" directory seems a bad idea) > > I think that if there should be tags in the "downloads" directory, > they should only reflect things which are downloaded to the > "downloads" directory, and nothing else. > > As for the tags in the directory, I think a better approach > is to download a file "tarball.tar.bz2" to a different filename first > I.E: "tarball.tar.bz2.in-progress" and if the download completes > then move the file to "tarball.tar.bz2". > Should remove a lot of clutter from the "downloads" directory. What we've tried to do is simplify the fetcher code paths in fetch2. There were some many corner/special cases and different code paths it was near impossible to tell what was going on. One of the side effects is that local file:// urls do touch the done stamp in the same way as other downloads. The main reason for those files is now checksum tracking. If the done stamps were part of tmpdir, we'd keep checksumming the contents of the downloads at each build rather than once after the download. The way the implementation works, there is very little risk from stale stamps, it would just mean the checksum code wouldn't get triggered and that is likely unneeded for a local file:// url anyway. So yes, its not ideal but looking at the code I'd be surprised if a build ever broke due to it. Cheers, Richard ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Possible stale tags in the download directory 2011-12-07 9:16 ` Richard Purdie @ 2011-12-07 11:27 ` Ulf Samuelsson 2011-12-07 12:00 ` Koen Kooi 2011-12-07 15:04 ` Richard Purdie 0 siblings, 2 replies; 10+ messages in thread From: Ulf Samuelsson @ 2011-12-07 11:27 UTC (permalink / raw) To: Patches and discussions about the oe-core layer On 2011-12-07 10:16, Richard Purdie wrote: > On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote: >> When I look at the "downloads" location I find a lot of files called. >> >> "11_usagi_fix.patch.done" >> >> but no "11_usagi_fix.patch" file in the directory. >> >> If this is a indication of that a patch has been downloaded, >> what happens if you for some reason or other delete your >> OE-core directory and keep the downloads directory. >> >> When you start from fresh, all the tags are then stale. >> >> (Deleting the "downloads" directory seems a bad idea) >> >> I think that if there should be tags in the "downloads" directory, >> they should only reflect things which are downloaded to the >> "downloads" directory, and nothing else. >> >> As for the tags in the directory, I think a better approach >> is to download a file "tarball.tar.bz2" to a different filename first >> I.E: "tarball.tar.bz2.in-progress" and if the download completes >> then move the file to "tarball.tar.bz2". >> Should remove a lot of clutter from the "downloads" directory. > What we've tried to do is simplify the fetcher code paths in fetch2. > There were some many corner/special cases and different code paths it > was near impossible to tell what was going on. One of the side effects > is that local file:// urls do touch the done stamp in the same way as > other downloads. The main reason for those files is now checksum > tracking. If the done stamps were part of tmpdir, we'd keep checksumming > the contents of the downloads at each build rather than once after the > download. The way the implementation works, there is very little risk > from stale stamps, it would just mean the checksum code wouldn't get > triggered and that is likely unneeded for a local file:// url anyway. > > So yes, its not ideal but looking at the code I'd be surprised if a > build ever broke due to it. > Maybe I am paranoid, but I think I remember seeing tarballs developing bit rot when moved between different machines. Could of course clean out the downloads directory, but it would be nice if the stamps would be in a different directory. Then I just delete the directory when I move the tarballs. I tend to use the "downloads" directory for other build systems as well, to avoid duplication, and having stamps in the directory could have real bad effects. I agree that recomputing checksums for each build is not so good, but doing it once after moving to a new machine, or for each new user is acceptable (at least to me). The stamps could be stored in a central location. "~/.oe/downloads" ? Con: If you have several repositories, maybe because you forgot to configure for your central repository, you may be in trouble. > Cheers, > > Richard > > > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core -- Best Regards Ulf Samuelsson eMagii ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Possible stale tags in the download directory 2011-12-07 11:27 ` Ulf Samuelsson @ 2011-12-07 12:00 ` Koen Kooi 2011-12-07 14:30 ` Ulf Samuelsson 2011-12-07 15:04 ` Richard Purdie 1 sibling, 1 reply; 10+ messages in thread From: Koen Kooi @ 2011-12-07 12:00 UTC (permalink / raw) To: ulf, Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 2239 bytes --] Op 7 dec. 2011, om 12:27 heeft Ulf Samuelsson het volgende geschreven: > On 2011-12-07 10:16, Richard Purdie wrote: >> On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote: >>> When I look at the "downloads" location I find a lot of files called. >>> >>> "11_usagi_fix.patch.done" >>> >>> but no "11_usagi_fix.patch" file in the directory. >>> >>> If this is a indication of that a patch has been downloaded, >>> what happens if you for some reason or other delete your >>> OE-core directory and keep the downloads directory. >>> >>> When you start from fresh, all the tags are then stale. >>> >>> (Deleting the "downloads" directory seems a bad idea) >>> >>> I think that if there should be tags in the "downloads" directory, >>> they should only reflect things which are downloaded to the >>> "downloads" directory, and nothing else. >>> >>> As for the tags in the directory, I think a better approach >>> is to download a file "tarball.tar.bz2" to a different filename first >>> I.E: "tarball.tar.bz2.in-progress" and if the download completes >>> then move the file to "tarball.tar.bz2". >>> Should remove a lot of clutter from the "downloads" directory. >> What we've tried to do is simplify the fetcher code paths in fetch2. >> There were some many corner/special cases and different code paths it >> was near impossible to tell what was going on. One of the side effects >> is that local file:// urls do touch the done stamp in the same way as >> other downloads. The main reason for those files is now checksum >> tracking. If the done stamps were part of tmpdir, we'd keep checksumming >> the contents of the downloads at each build rather than once after the >> download. The way the implementation works, there is very little risk >> from stale stamps, it would just mean the checksum code wouldn't get >> triggered and that is likely unneeded for a local file:// url anyway. >> >> So yes, its not ideal but looking at the code I'd be surprised if a >> build ever broke due to it. >> > Maybe I am paranoid, but I think I remember seeing tarballs developing > bit rot when moved between different machines. And that's why checksums are checked before do_unpack. [-- Attachment #2: Message signed with OpenPGP using GPGMail --] [-- Type: application/pgp-signature, Size: 169 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Possible stale tags in the download directory 2011-12-07 12:00 ` Koen Kooi @ 2011-12-07 14:30 ` Ulf Samuelsson 2011-12-07 14:37 ` Paul Eggleton 0 siblings, 1 reply; 10+ messages in thread From: Ulf Samuelsson @ 2011-12-07 14:30 UTC (permalink / raw) To: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 2752 bytes --] On 2011-12-07 13:00, Koen Kooi wrote: > Op 7 dec. 2011, om 12:27 heeft Ulf Samuelsson het volgende geschreven: > >> On 2011-12-07 10:16, Richard Purdie wrote: >>> On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote: >>>> When I look at the "downloads" location I find a lot of files called. >>>> >>>> "11_usagi_fix.patch.done" >>>> >>>> but no "11_usagi_fix.patch" file in the directory. >>>> >>>> If this is a indication of that a patch has been downloaded, >>>> what happens if you for some reason or other delete your >>>> OE-core directory and keep the downloads directory. >>>> >>>> When you start from fresh, all the tags are then stale. >>>> >>>> (Deleting the "downloads" directory seems a bad idea) >>>> >>>> I think that if there should be tags in the "downloads" directory, >>>> they should only reflect things which are downloaded to the >>>> "downloads" directory, and nothing else. >>>> >>>> As for the tags in the directory, I think a better approach >>>> is to download a file "tarball.tar.bz2" to a different filename first >>>> I.E: "tarball.tar.bz2.in-progress" and if the download completes >>>> then move the file to "tarball.tar.bz2". >>>> Should remove a lot of clutter from the "downloads" directory. >>> What we've tried to do is simplify the fetcher code paths in fetch2. >>> There were some many corner/special cases and different code paths it >>> was near impossible to tell what was going on. One of the side effects >>> is that local file:// urls do touch the done stamp in the same way as >>> other downloads. The main reason for those files is now checksum >>> tracking. If the done stamps were part of tmpdir, we'd keep checksumming >>> the contents of the downloads at each build rather than once after the >>> download. The way the implementation works, there is very little risk >>> from stale stamps, it would just mean the checksum code wouldn't get >>> triggered and that is likely unneeded for a local file:// url anyway. >>> >>> So yes, its not ideal but looking at the code I'd be surprised if a >>> build ever broke due to it. >>> >> Maybe I am paranoid, but I think I remember seeing tarballs developing >> bit rot when moved between different machines. > And that's why checksums are checked before do_unpack. Richard says that if the *.done file is there, then the checksum is not calculated. Or that is at least how I interpret his comment. If the check is always there, why the tag? > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core -- Best Regards Ulf Samuelsson eMagii [-- Attachment #2: Type: text/html, Size: 4027 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Possible stale tags in the download directory 2011-12-07 14:30 ` Ulf Samuelsson @ 2011-12-07 14:37 ` Paul Eggleton 2011-12-07 15:00 ` Richard Purdie 0 siblings, 1 reply; 10+ messages in thread From: Paul Eggleton @ 2011-12-07 14:37 UTC (permalink / raw) To: ulf; +Cc: openembedded-core On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote: > Richard says that if the *.done file is there, then the checksum is not > calculated. > Or that is at least how I interpret his comment. > > If the check is always there, why the tag? That's not what my reading of the bitbake source says. bitbake/lib/bb/fetch2/__init__.py contains the only references to .done files and all it does is make sure the done file gets "touched" whenever a fetch for the associated file occurs; it does not otherwise check for its existence. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Possible stale tags in the download directory 2011-12-07 14:37 ` Paul Eggleton @ 2011-12-07 15:00 ` Richard Purdie 2011-12-07 21:43 ` Khem Raj 0 siblings, 1 reply; 10+ messages in thread From: Richard Purdie @ 2011-12-07 15:00 UTC (permalink / raw) To: Patches and discussions about the oe-core layer; +Cc: ulf On Wed, 2011-12-07 at 14:37 +0000, Paul Eggleton wrote: > On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote: > > Richard says that if the *.done file is there, then the checksum is not > > calculated. > > Or that is at least how I interpret his comment. > > > > If the check is always there, why the tag? > > That's not what my reading of the bitbake source says. > bitbake/lib/bb/fetch2/__init__.py contains the only references to .done files > and all it does is make sure the done file gets "touched" whenever a fetch for > the associated file occurs; it does not otherwise check for its existence. It does two things, one is an activity marker useful for seeing whether files are in active use and the second is for the checksum calculation which only ever happens once after download due to the file (see verify_checksum() in that file). Cheers, Richard ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Possible stale tags in the download directory 2011-12-07 15:00 ` Richard Purdie @ 2011-12-07 21:43 ` Khem Raj 2011-12-07 23:46 ` Ulf Samuelsson 0 siblings, 1 reply; 10+ messages in thread From: Khem Raj @ 2011-12-07 21:43 UTC (permalink / raw) To: Patches and discussions about the oe-core layer; +Cc: ulf On (07/12/11 15:00), Richard Purdie wrote: > On Wed, 2011-12-07 at 14:37 +0000, Paul Eggleton wrote: > > On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote: > > > Richard says that if the *.done file is there, then the checksum is not > > > calculated. > > > Or that is at least how I interpret his comment. > > > > > > If the check is always there, why the tag? > > > > That's not what my reading of the bitbake source says. > > bitbake/lib/bb/fetch2/__init__.py contains the only references to .done files > > and all it does is make sure the done file gets "touched" whenever a fetch for > > the associated file occurs; it does not otherwise check for its existence. > > It does two things, one is an activity marker useful for seeing whether > files are in active use and the second is for the checksum calculation > which only ever happens once after download due to the file (see > verify_checksum() in that file). when do we create .done files? if I am in middle of download of a tar file and I cancel it and restart it does not fetch the file again but tried to use the corrupt file. Is it a big build time hit if checksums are verified all the time ? -- -Khem ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Possible stale tags in the download directory 2011-12-07 21:43 ` Khem Raj @ 2011-12-07 23:46 ` Ulf Samuelsson 0 siblings, 0 replies; 10+ messages in thread From: Ulf Samuelsson @ 2011-12-07 23:46 UTC (permalink / raw) To: Patches and discussions about the oe-core layer On 2011-12-07 22:43, Khem Raj wrote: > On (07/12/11 15:00), Richard Purdie wrote: >> On Wed, 2011-12-07 at 14:37 +0000, Paul Eggleton wrote: >>> On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote: >>>> Richard says that if the *.done file is there, then the checksum is not >>>> calculated. >>>> Or that is at least how I interpret his comment. >>>> >>>> If the check is always there, why the tag? >>> That's not what my reading of the bitbake source says. >>> bitbake/lib/bb/fetch2/__init__.py contains the only references to .done files >>> and all it does is make sure the done file gets "touched" whenever a fetch for >>> the associated file occurs; it does not otherwise check for its existence. >> It does two things, one is an activity marker useful for seeing whether >> files are in active use and the second is for the checksum calculation >> which only ever happens once after download due to the file (see >> verify_checksum() in that file). > when do we create .done files? if I am in middle of download of a tar > file and I cancel it and restart it does not fetch the file again but > tried to use the corrupt file. > > Is it a big build time hit if checksums are verified all the time ? > I have seen the same problem. -- Best Regards Ulf Samuelsson eMagii ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Possible stale tags in the download directory 2011-12-07 11:27 ` Ulf Samuelsson 2011-12-07 12:00 ` Koen Kooi @ 2011-12-07 15:04 ` Richard Purdie 1 sibling, 0 replies; 10+ messages in thread From: Richard Purdie @ 2011-12-07 15:04 UTC (permalink / raw) To: ulf, Patches and discussions about the oe-core layer On Wed, 2011-12-07 at 12:27 +0100, Ulf Samuelsson wrote: > On 2011-12-07 10:16, Richard Purdie wrote: > > On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote: > >> When I look at the "downloads" location I find a lot of files called. > >> > >> "11_usagi_fix.patch.done" > >> > >> but no "11_usagi_fix.patch" file in the directory. > >> > >> If this is a indication of that a patch has been downloaded, > >> what happens if you for some reason or other delete your > >> OE-core directory and keep the downloads directory. > >> > >> When you start from fresh, all the tags are then stale. > >> > >> (Deleting the "downloads" directory seems a bad idea) > >> > >> I think that if there should be tags in the "downloads" directory, > >> they should only reflect things which are downloaded to the > >> "downloads" directory, and nothing else. > >> > >> As for the tags in the directory, I think a better approach > >> is to download a file "tarball.tar.bz2" to a different filename first > >> I.E: "tarball.tar.bz2.in-progress" and if the download completes > >> then move the file to "tarball.tar.bz2". > >> Should remove a lot of clutter from the "downloads" directory. > > What we've tried to do is simplify the fetcher code paths in fetch2. > > There were some many corner/special cases and different code paths it > > was near impossible to tell what was going on. One of the side effects > > is that local file:// urls do touch the done stamp in the same way as > > other downloads. The main reason for those files is now checksum > > tracking. If the done stamps were part of tmpdir, we'd keep checksumming > > the contents of the downloads at each build rather than once after the > > download. The way the implementation works, there is very little risk > > from stale stamps, it would just mean the checksum code wouldn't get > > triggered and that is likely unneeded for a local file:// url anyway. > > > > So yes, its not ideal but looking at the code I'd be surprised if a > > build ever broke due to it. > > > Maybe I am paranoid, but I think I remember seeing tarballs developing > bit rot when moved between different machines. I don't understand what you mean here. If your process for copying files between machines corrupts files, you have bigger problems. You can just rm *.done at the same time if that is an issue. > Could of course clean out the downloads directory, but it would be > nice if the stamps would be in a different directory. > Then I just delete the directory when I move the tarballs. > > I tend to use the "downloads" directory for other build systems as well, > to avoid duplication, and having stamps in the directory could have > real bad effects. > > I agree that recomputing checksums for each build is not so good, > but doing it once after moving to a new machine, or for each new user > is acceptable (at least to me). > > The stamps could be stored in a central location. > > "~/.oe/downloads" ? I believe the data the .done files represent is directly related to the files themselves and they should be kept together. Nothing said here is convincing me there is a real world problem... Cheers, Richard ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2011-12-07 23:53 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-12-07 8:27 Possible stale tags in the download directory Ulf Samuelsson 2011-12-07 9:16 ` Richard Purdie 2011-12-07 11:27 ` Ulf Samuelsson 2011-12-07 12:00 ` Koen Kooi 2011-12-07 14:30 ` Ulf Samuelsson 2011-12-07 14:37 ` Paul Eggleton 2011-12-07 15:00 ` Richard Purdie 2011-12-07 21:43 ` Khem Raj 2011-12-07 23:46 ` Ulf Samuelsson 2011-12-07 15:04 ` Richard Purdie
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.