[PATCH] fetch/hg: Disable checksums for archived downloads

[PATCH] fetch/hg: Disable checksums for archived downloads

[Ash Charles]

Like the Git fetcher, the Mercurial fetcher shouldn't expect recipes to
provide a checksum.  As described [1], recipes using a mercurial
fetcher that don't provide a checksum will fail in a the repository has
previously been downloaded and archived.

Credit to Rafaël Carré for figuring out the bug.

[1] https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg41328.html

Signed-off-by: Ash Charles <ashcharles@gmail.com>
---
 bitbake/lib/bb/fetch2/hg.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py
index 81592f6..cab371f 100644
--- a/bitbake/lib/bb/fetch2/hg.py
+++ b/bitbake/lib/bb/fetch2/hg.py
@@ -43,6 +43,13 @@ class Hg(FetchMethod):
         """
         return ud.type in ['hg']
 
+    def supports_checksum(self, urldata):
+        """
+        Don't require checksums for local archives created from
+        repository checkouts.
+        """ 
+        return False
+
     def urldata_init(self, ud, d):
         """
         init hg specific variable within url data
-- 
2.1.4

Re: [oe] [PATCH] fetch/hg: Disable checksums for archived downloads

[Robert Yang]

Hello,

I think that the following patch has broken the mirror tarball of hg:

http://lists.openembedded.org/pipermail/bitbake-devel/2015-May/005725.html

I wonder how did you test the patch, please ? Did you revert the patch:
  fetch/hg: support submodules ?

Or other ways, please ?

// Robert


On 05/22/2015 11:39 PM, Ash Charles wrote:
> Like the Git fetcher, the Mercurial fetcher shouldn't expect recipes to
> provide a checksum.  As described [1], recipes using a mercurial
> fetcher that don't provide a checksum will fail in a the repository has
> previously been downloaded and archived.
>
> Credit to Rafaël Carré for figuring out the bug.
>
> [1] https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg41328.html
>
> Signed-off-by: Ash Charles <ashcharles@gmail.com>
> ---
>   bitbake/lib/bb/fetch2/hg.py | 7 +++++++
>   1 file changed, 7 insertions(+)
>
> diff --git a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py
> index 81592f6..cab371f 100644
> --- a/bitbake/lib/bb/fetch2/hg.py
> +++ b/bitbake/lib/bb/fetch2/hg.py
> @@ -43,6 +43,13 @@ class Hg(FetchMethod):
>           """
>           return ud.type in ['hg']
>
> +    def supports_checksum(self, urldata):
> +        """
> +        Don't require checksums for local archives created from
> +        repository checkouts.
> +        """
> +        return False
> +
>       def urldata_init(self, ud, d):
>           """
>           init hg specific variable within url data
>

Re: [oe] [PATCH] fetch/hg: Disable checksums for archived downloads

[Ash Charles]

[-- Attachment #1: Type: text/plain, Size: 499 bytes --]

> I wonder how did you test the patch, please ? Did you revert the patch:
>  fetch/hg: support submodules ?
Hi Robert,
I tested the patch in a Yocto environment which had a prepopulated
downloads directory. In particular, I had seen the failure described [1]
for vim.
My version of bitbake was slightly older so the submodule patch that was
breaking the mirroring hadn't yet been applied.

[1]
https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg41328.html

--Ash

[-- Attachment #2: Type: text/html, Size: 638 bytes --]

Re: [oe] [PATCH] fetch/hg: Disable checksums for archived downloads

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3899 bytes --]

FWIW: I see vim unpack failing now in world builds:

NOTE: recipe vim-7.4.481-r0: task do_unpack: Started
ERROR: Function failed: Fetcher failure: Fetch command failed with
exit code 255, output:
abort: repository
/home/jenkins/oe/world/shr-core/downloads/hg/vim.googlecode.com/hg/vim
not found!

ERROR: Logfile of failure stored in:
/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/temp/log.do_unpack.28505
Log data follows:
| DEBUG: Executing python function do_unpack
| DEBUG: Executing python function base_do_unpack
| DEBUG: Unpack: creating new hg repository in
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/vim'
| DEBUG: Running export
PATH="/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin/arm-oe-linux-gnueabi:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/qemuarm/usr/bin/crossscripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/bin:/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/bitbake/bin:/home/jenkins/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games";
export HOME="/home/jenkins"; /usr/bin/env hg init
/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/vim
| DEBUG: Unpack: updating source in
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/vim'
| DEBUG: Running export
PATH="/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin/arm-oe-linux-gnueabi:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/qemuarm/usr/bin/crossscripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/bin:/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/bitbake/bin:/home/jenkins/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games";
export HOME="/home/jenkins"; /usr/bin/env hg pull
/home/jenkins/oe/world/shr-core/downloads/hg/vim.googlecode.com/hg/vim
| DEBUG: Python function base_do_unpack finished
| DEBUG: Python function do_unpack finished
| ERROR: Function failed: Fetcher failure: Fetch command failed with
exit code 255, output:
| abort: repository
/home/jenkins/oe/world/shr-core/downloads/hg/vim.googlecode.com/hg/vim
not found!
|
NOTE: recipe vim-7.4.481-r0: task do_unpack: Failed
ERROR: Task 5789
(/home/jenkins/oe/world/shr-core/meta-openembedded/meta-oe/recipes-support/vim/vim_7.4.481.bb,
do_unpack) failed with exit code '1'



On Fri, May 29, 2015 at 7:46 AM, Ash Charles <ashcharles@gmail.com> wrote:

> > I wonder how did you test the patch, please ? Did you revert the patch:
> >  fetch/hg: support submodules ?
> Hi Robert,
> I tested the patch in a Yocto environment which had a prepopulated
> downloads directory. In particular, I had seen the failure described [1]
> for vim.
> My version of bitbake was slightly older so the submodule patch that was
> breaking the mirroring hadn't yet been applied.
>
> [1]
> https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg41328.html
>
> --Ash
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>
>

[-- Attachment #2: Type: text/html, Size: 5081 bytes --]

Re: [oe] [PATCH] fetch/hg: Disable checksums for archived downloads

[Ash Charles]

Hi,

I verified that the "fetch/hg: support submodules" [1] patch, in
combination with this [2] patch is causing do_unpack failures for me.
Based on the logs from the world build, I suspect this is also
breaking dvb-apps.  I don't have a fix so perhaps the best plan is
reverting [2] now that cleanall for hg packages actually works.

[1] http://lists.openembedded.org/pipermail/bitbake-devel/2015-May/005725.html
[2] http://lists.openembedded.org/pipermail/bitbake-devel/2015-May/005758.html

--Ash

On Fri, May 29, 2015 at 2:18 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
> FWIW: I see vim unpack failing now in world builds:
>
> NOTE: recipe vim-7.4.481-r0: task do_unpack: Started
> ERROR: Function failed: Fetcher failure: Fetch command failed with exit code
> 255, output:
> abort: repository
> /home/jenkins/oe/world/shr-core/downloads/hg/vim.googlecode.com/hg/vim not
> found!
>
> ERROR: Logfile of failure stored in:
> /home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/temp/log.do_unpack.28505
> Log data follows:
> | DEBUG: Executing python function do_unpack
> | DEBUG: Executing python function base_do_unpack
> | DEBUG: Unpack: creating new hg repository in
> '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/vim'
> | DEBUG: Running export
> PATH="/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin/arm-oe-linux-gnueabi:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/qemuarm/usr/bin/crossscripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/bin:/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/bitbake/bin:/home/jenkins/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games";
> export HOME="/home/jenkins"; /usr/bin/env hg init
> /home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/vim
> | DEBUG: Unpack: updating source in
> '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/vim'
> | DEBUG: Running export
> PATH="/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin/arm-oe-linux-gnueabi:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/qemuarm/usr/bin/crossscripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/bin:/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/bitbake/bin:/home/jenkins/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games";
> export HOME="/home/jenkins"; /usr/bin/env hg pull
> /home/jenkins/oe/world/shr-core/downloads/hg/vim.googlecode.com/hg/vim
> | DEBUG: Python function base_do_unpack finished
> | DEBUG: Python function do_unpack finished
> | ERROR: Function failed: Fetcher failure: Fetch command failed with exit
> code 255, output:
> | abort: repository
> /home/jenkins/oe/world/shr-core/downloads/hg/vim.googlecode.com/hg/vim not
> found!
> |
> NOTE: recipe vim-7.4.481-r0: task do_unpack: Failed
> ERROR: Task 5789
> (/home/jenkins/oe/world/shr-core/meta-openembedded/meta-oe/recipes-support/vim/vim_7.4.481.bb,
> do_unpack) failed with exit code '1'
>
>
>
> On Fri, May 29, 2015 at 7:46 AM, Ash Charles <ashcharles@gmail.com> wrote:
>>
>> > I wonder how did you test the patch, please ? Did you revert the patch:
>> >  fetch/hg: support submodules ?
>> Hi Robert,
>> I tested the patch in a Yocto environment which had a prepopulated
>> downloads directory. In particular, I had seen the failure described [1] for
>> vim.
>> My version of bitbake was slightly older so the submodule patch that was
>> breaking the mirroring hadn't yet been applied.
>>
>> [1]
>> https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg41328.html
>>
>> --Ash
>>
>>
>> --
>> _______________________________________________
>> bitbake-devel mailing list
>> bitbake-devel@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>>
>

Re: [oe] [PATCH] fetch/hg: Disable checksums for archived downloads

[Robert Yang]

On 06/02/2015 08:08 AM, Ash Charles wrote:
> Hi,
>
> I verified that the "fetch/hg: support submodules" [1] patch, in
> combination with this [2] patch is causing do_unpack failures for me.
> Based on the logs from the world build, I suspect this is also
> breaking dvb-apps.  I don't have a fix so perhaps the best plan is
> reverting [2] now that cleanall for hg packages actually works.
>
> [1] http://lists.openembedded.org/pipermail/bitbake-devel/2015-May/005725.html
> [2] http://lists.openembedded.org/pipermail/bitbake-devel/2015-May/005758.html
>

I'm working on fixing the issues now, will send patches sooner.

// Robert

> --Ash
>
> On Fri, May 29, 2015 at 2:18 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
>> FWIW: I see vim unpack failing now in world builds:
>>
>> NOTE: recipe vim-7.4.481-r0: task do_unpack: Started
>> ERROR: Function failed: Fetcher failure: Fetch command failed with exit code
>> 255, output:
>> abort: repository
>> /home/jenkins/oe/world/shr-core/downloads/hg/vim.googlecode.com/hg/vim not
>> found!
>>
>> ERROR: Logfile of failure stored in:
>> /home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/temp/log.do_unpack.28505
>> Log data follows:
>> | DEBUG: Executing python function do_unpack
>> | DEBUG: Executing python function base_do_unpack
>> | DEBUG: Unpack: creating new hg repository in
>> '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/vim'
>> | DEBUG: Running export
>> PATH="/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin/arm-oe-linux-gnueabi:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/qemuarm/usr/bin/crossscripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/bin:/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/bitbake/bin:/home/jenkins/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games";
>> export HOME="/home/jenkins"; /usr/bin/env hg init
>> /home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/vim
>> | DEBUG: Unpack: updating source in
>> '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/vim/7.4.481-r0/vim'
>> | DEBUG: Running export
>> PATH="/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin/arm-oe-linux-gnueabi:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/qemuarm/usr/bin/crossscripts:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/usr/bin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/sbin:/home/jenkins/oe/world/shr-core/tmp-glibc/sysroots/x86_64-linux/bin:/home/jenkins/oe/world/shr-core/openembedded-core/scripts:/home/jenkins/oe/world/shr-core/bitbake/bin:/home/jenkins/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games";
>> export HOME="/home/jenkins"; /usr/bin/env hg pull
>> /home/jenkins/oe/world/shr-core/downloads/hg/vim.googlecode.com/hg/vim
>> | DEBUG: Python function base_do_unpack finished
>> | DEBUG: Python function do_unpack finished
>> | ERROR: Function failed: Fetcher failure: Fetch command failed with exit
>> code 255, output:
>> | abort: repository
>> /home/jenkins/oe/world/shr-core/downloads/hg/vim.googlecode.com/hg/vim not
>> found!
>> |
>> NOTE: recipe vim-7.4.481-r0: task do_unpack: Failed
>> ERROR: Task 5789
>> (/home/jenkins/oe/world/shr-core/meta-openembedded/meta-oe/recipes-support/vim/vim_7.4.481.bb,
>> do_unpack) failed with exit code '1'
>>
>>
>>
>> On Fri, May 29, 2015 at 7:46 AM, Ash Charles <ashcharles@gmail.com> wrote:
>>>
>>>> I wonder how did you test the patch, please ? Did you revert the patch:
>>>>   fetch/hg: support submodules ?
>>> Hi Robert,
>>> I tested the patch in a Yocto environment which had a prepopulated
>>> downloads directory. In particular, I had seen the failure described [1] for
>>> vim.
>>> My version of bitbake was slightly older so the submodule patch that was
>>> breaking the mirroring hadn't yet been applied.
>>>
>>> [1]
>>> https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg41328.html
>>>
>>> --Ash
>>>
>>>
>>> --
>>> _______________________________________________
>>> bitbake-devel mailing list
>>> bitbake-devel@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>>>
>>
>
>

Re: [oe] [PATCH] fetch/hg: Disable checksums for archived downloads

[Steffen Sledz]

On 02.06.2015 06:38, Robert Yang wrote:
> I'm working on fixing the issues now, will send patches sooner.

Any news on that?

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058