Re: [yocto] Yocto [thud], [zeus] do_fetch and do_unpack failures with offline/online svn build! #yocto #python

["Mikko Rapeli" <mikko.rapeli@bmw.de>]

On Mon, Feb 17, 2020 at 12:56:13PM +0000, Georgi Georgiev via Lists.Yoctoproject.Org wrote:
> Another serious issue with svn fetcher, always occurring when doing incremental builds. I am putting it here because I think both issues are related. Please suggest what to do. The error we get is shown below. This time it happens always. The only workaround is to clean and build. As soon as we make modification of the recipe (PV change only) it fails:
> 
> ERROR: ww-pacman-121204-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated:
> 
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
>      0001:
>  *** 0002:base_do_fetch(d)
>      0003:
> File: '/data/home/w23698/projects/proj/build/../sources/poky/meta/classes/base.bbclass', lineno: 163, function: base_do_fetch
>      0159:        return
>      0160:
>      0161:    try:
>      0162:        fetcher = bb.fetch2.Fetch(src_uri, d)
>  *** 0163:        fetcher.download()
>      0164:    except bb.fetch2.BBFetchException as e:
>      0165:        bb.fatal(str(e))
>      0166:}
>      0167:
> File: '/data/home/w23698/projects/proj/sources/poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 1678, function: download
>      1674:                    try:
>      1675:                        if not trusted_network(self.d, ud.url):
>      1676:                            raise UntrustedUrl(ud.url)
>      1677:                        logger.debug(1, "Trying Upstream")
>  *** 1678:                        m.download(ud, self.d)
>      1679:                        if hasattr(m, "build_mirror_data"):
>      1680:                            m.build_mirror_data(ud, self.d)
>      1681:                        localpath = ud.localpath
>      1682:                        # early checksum verify, so that if checksum mismatched,
> File: '/data/home/w23698/projects/proj/sources/poky/bitbake/lib/bb/fetch2/svn.py', lineno: 150, function: download
>      0146:            if not ("externals" in ud.parm and ud.parm["externals"] == "nowarn"):
>      0147:                # Warn the user if this had externals (won't catch them all)
>      0148:                output = runfetchcmd("svn propget svn:externals || true", d, workdir=ud.moddir)
>      0149:                if output:
>  *** 0150:                    if "--ignore-externals" in svnfetchcmd.split():
>      0151:                        bb.warn("%s contains svn:externals." % ud.url)
>      0152:                        bb.warn("These should be added to the recipe SRC_URI as necessary.")
>      0153:                        bb.warn("svn fetch has ignored externals:\n%s" % output)
>      0154:                        bb.warn("To disable this warning add ';externals=nowarn' to the url.")
> Exception: UnboundLocalError: local variable 'svnfetchcmd' referenced before assignment
> 
> ERROR: Logfile of failure stored in: /data/home/w23698/projects/proj/build/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121204-r0/temp/log.do_fetch.18853

Hmm. Rings a bell. I run builds with a local non-upstreamed patch:

Author:     Mikko Rapeli <mikko.rapeli@bmw.de>
AuthorDate: Fri Sep 6 14:15:20 2019 +0200
Commit:     Mikko Rapeli <mikko.rapeli@bmw.de>
CommitDate: Fri Sep 6 14:25:15 2019 +0200

    svn fetcher: allow "svn propget svn:externals" to fail

    Not all servers and repositories have this property set
    which results in failures like this when actual svn checkout
    command succeeded:

    svn: warning: W200017: Property 'svn:externals' not found on ''
    svn: E200000: A problem occurred; see other errors for details

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>

--- a/bitbake/lib/bb/fetch2/svn.py
+++ b/bitbake/lib/bb/fetch2/svn.py
@@ -145,7 +145,7 @@ class Svn(FetchMethod):
 
                 if not ("externals" in ud.parm and ud.parm["externals"] == "nowarn"):
                     # Warn the user if this had externals (won't catch them all)
-                    output = runfetchcmd("svn propget svn:externals", d, workdir=ud.moddir)
+                    output = runfetchcmd("svn propget svn:externals || true", d, workdir=ud.moddir)
                     if output:
                         if "--ignore-externals" in svnfetchcmd.split():
                             bb.warn("%s contains svn:externals." % ud.url)

I think this would help.

Cheers,

-Mikko


> -----Original Message-----
> From: Georgi Georgiev 
> Sent: Wednesday, February 05, 2020 6:57 PM
> To: Georgi Georgiev <Georgi.Georgiev@woodward.com>; Khem Raj <raj.khem@gmail.com>; yocto@lists.yoctoproject.org
> Subject: Re: [yocto] Yocto [thud], [zeus] do_fetch and do_unpack failures with offline/online svn build! #yocto #python
> 
> Sorry Khem,
> With esc character '\' before & it can't take the full path. So briefly:
> 
> Yocto build with char '\' before & in SRC_URI:
> 	online - OK
> 	offline - Error - svn: E170013: Unable to connect to a repository...
> Yocto build without '\' in SRC_URI:
> 	online and offline same error - do_unpack: Unpack failure for URL - "package".tar.gz is present in DL_DIR
> 
> Thanks
> 
> 
> 
> -----Original Message-----
> From: yocto@lists.yoctoproject.org [mailto:yocto@lists.yoctoproject.org] On Behalf Of Georgi Georgiev via Lists.Yoctoproject.Org
> Sent: Wednesday, February 05, 2020 5:26 PM
> To: Khem Raj <raj.khem@gmail.com>; yocto@lists.yoctoproject.org
> Cc: yocto@lists.yoctoproject.org
> Subject: Re: [EXTERNAL] Re: [yocto] Yocto [thud], [zeus] do_fetch and do_unpack failures with offline/online svn build! #yocto #python
> 
> Hi Khem,
> 
> Yes, no issues with that. When I am connected to network it fetches the code with and without escape character before & in the path.
> svn co " https://urldefense.proofpoint.com/v2/url?u=http-3A__cocosubversion_svn_Embedded_Valve-26Actuator_DVPII_trunk_SOCPACManEnvEngKeys&d=DwIGaQ&c=y6L7g950KfMp92YmLM0QlMdXcRn6b-Cq4AApnSJOenE&r=kHtJrQGzfH1ZmfsNkJpYuH-jtNpv8yMDkqAmsRP99mc&m=vXia-xlcxuLr2fSmBogZpMIaOntXLKWv3mkxcRgtUc4&s=UYH9lueDIeMGZndMxgbZgb76A5EfqN-MR58DBv1iriI&e= " works fine and fetches the package. 
> 
> Georgi
> 
> -----Original Message-----
> From: Khem Raj [mailto:raj.khem@gmail.com]
> Sent: Friday, January 31, 2020 8:33 PM
> To: Georgi Georgiev <Georgi.Georgiev@woodward.com>; yocto@lists.yoctoproject.org
> Subject: [EXTERNAL] Re: [yocto] Yocto [thud], [zeus] do_fetch and do_unpack failures with offline/online svn build! #yocto #python
> 
> On 1/31/20 4:02 AM, georgi.georgiev via Lists.Yoctoproject.Org wrote:
> > Hello Community,
> > This is the third time I am asking for support on this issue. This 
> > time I decided to use the web form.
> > In our project we have a requirement to be able to build the project 
> > offline. E.g. on the field without any network connection. When we are 
> > connected with the recipe mentioned below we don't have issues:
> > 
> 
> 
> svn: E170013: Unable to connect to a repository at URL 'https://urldefense.proofpoint.com/v2/url?u=http-3A__cocosubversion_svn_Embedded_Valve-26Actuator_DVPII_trunk_SOCPACManEnvEngKeys_trunk&d=DwIGaQ&c=y6L7g950KfMp92YmLM0QlMdXcRn6b-Cq4AApnSJOenE&r=kHtJrQGzfH1ZmfsNkJpYuH-jtNpv8yMDkqAmsRP99mc&m=ocx6BoVSrDQGWUvUUVvgKtuJbT7eH7jFSjCy1Ys73Vw&s=OTunnmsi_tCUt3JytmU2Hs7i7Xnqhl8-2CMsqmzIl90&e= '
> svn: E670003: Temporary failure in name resolution
> 
> can you try checking out the repo out side of fetcher and see if the machine can fetch the url you are using in SRC_URI ?
> 
> > *SUMMARY = "PACMan - Parameter And Configuration MANager"* *LICENSE =
> > "CLOSED"* *inherit systemd useradd* *REQUIRED_DISTRO_FEATURES =
> > "systemd"* *# SVN revision* *PV = "121026"* *# Name of SVN project*
> > *PACMAN_PROJ_NAME="SOCPACManEnvEngKeys"*
> > *SRC_URI =
> > "svn://cocosubversion/svn/Embedded/Valve%5C&Actuator/DVPII/trunk/$%7BP
> > ACMAN_PROJ_NAME%7D;module=trunk;protocol=http;externals=allowed;rev=$%
> > 7BPV%7D%22* *SRC_URI += "file://ww-authpacman.service%22* *SRC_URI +=
> > "file://ww-pacman.service%22* *S = "${WORKDIR}/trunk/"* *# ${PN}-sys: 
> > content related to system, which goes to base rootfs (only .service 
> > file and symlinks)* *# ${PN}:      real content which may go to 
> > separate partition* *PACKAGES =+ " ${PN}-sys"* .........
> > When disconnect the network, erase sstate-cache, cache and tmp I see 
> > log file attached (log.do_fetch.32757) and the following output:
> > 
> > *ERROR: ww-pacman-121026-r0 do_fetch: Fetcher failure: Fetch command 
> > export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; export 
> > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export 
> > SSH_AGENT_PID="11412"; export 
> > SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export 
> > PATH="/home/w23698/projects/proj_dvp2/build_dvp2/tmp/sysroots-uninativ
> > e/x86_64-linux/usr/bin:/home/w23698/projects/proj_dvp2/sources/poky/sc
> > ripts:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-n
> > eon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/usr/b
> > in/arm-poky-linux-gnueabi:/home/w23698/projects/proj_dvp2/build_dvp2/t
> > mp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-
> > sysroot/usr/bin/crossscripts:/home/w23698/projects/proj_dvp2/build_dvp
> > 2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/reci
> > pe-sysroot-native/usr/sbin:/home/w23698/projects/proj_dvp2/build_dvp2/
> > tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe
> > -sysroot-native/usr/bin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp
> > /work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sy
> > sroot-native/sbin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/
> > cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-
> > native/bin:/home/w23698/projects/proj_dvp2/sources/poky/bitbake/bin:/h
> > ome/w23698/projects/proj_dvp2/build_dvp2/tmp/hosttools";
> > export HOME="/home/w23698"; /usr/bin/env svn --non-interactive 
> > --trust-server-cert update --no-auth-cache -r 121026 failed with exit 
> > code 1, output:* *Updating '.':*
> > *svn: E170013: Unable to connect to a repository at URL 
> > 'https://urldefense.proofpoint.com/v2/url?u=http-3A__cocosubversion_sv
> > n_Embedded_Valve-26Actuator_DVPII_trunk_SOCPACManEnvEngKeys_trunk-27-2
> > A&d=DwIGaQ&c=y6L7g950KfMp92YmLM0QlMdXcRn6b-Cq4AApnSJOenE&r=kHtJrQGzfH1
> > ZmfsNkJpYuH-jtNpv8yMDkqAmsRP99mc&m=ocx6BoVSrDQGWUvUUVvgKtuJbT7eH7jFSjC
> > y1Ys73Vw&s=Ffv1JU1QYBh4g49fmoLnsDSFgMMBc_5MbOpy59QUS18&e=
> > *svn: E670003: Temporary failure in name resolution*
> > *ERROR: ww-pacman-121026-r0 do_fetch: Fetcher failure for URL: 
> > 'svn://cocosubversion/svn/Embedded/Valve%5C&Actuator/DVPII/trunk/SOCPACManEnvEngKeys;module=trunk;protocol=http;externals=allowed;rev=121026'. 
> > Unable to fetch URL from any source.*
> > *ERROR: Logfile of failure stored in: 
> > /home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-po
> > ky-linux-gnueabi/ww-pacman/121026-r0/temp/log.do_fetch.32757*
> > *ERROR: Task
> > (/home/w23698/projects/proj_dvp2/build_dvp2/../sources/meta-ww-dvp2/re
> > cipes-ww/ww-pacman/ww-pacman.bb:do_fetch)
> > failed with exit code '1'*
> > 
> > When remove the '\' character in SRC_URI, e.g. to become:
> > 
> > *SRC_URI =
> > "svn://cocosubversion/svn/Embedded/Valve&Actuator/DVPII/trunk/$%7BPACM
> > AN_PROJ_NAME%7D;module=trunk;protocol=http;externals=allowed;rev=$%7BP
> > V%7D%22*
> > 
> > In connected and not connected to network do_fetch() passes 
> > successfully but I see one and same error (log.do_unpack.25226) output:
> > 
> > *ERROR: ww-pacman-121026-r0 do_unpack: Unpack failure for URL: 
> > 'svn://cocosubversion/svn/Embedded/Valve&Actuator/DVPII/trunk/SOCPACManEnvEngKeys;module=trunk;protocol=http;externals=allowed;rev=121026'. 
> > Unpack command
> > PATH="/home/w23698/projects/proj_dvp2/build_dvp2/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/w23698/projects/proj_dvp2/sources/poky/scripts:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot/usr/bin/crossscripts:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/usr/sbin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/usr/bin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/sbin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/bin:/home/w23698/projects/proj_dvp2/sources/poky/bitbake/bin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/hosttools" 
> > tar xz --no-same-owner -f
> > /home/w23698/projects/proj_dvp2/build_dvp2/downloads/trunk_cocosubvers
> > ion_.svn.Embedded.Valve&Actuator.DVPII.trunk.SOCPACManEnvEngKeys_12102
> > 6_.tar.gz
> > failed with return value 127*
> > *ERROR: Logfile of failure stored in: 
> > /home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-po
> > ky-linux-gnueabi/ww-pacman/121026-r0/temp/log.do_unpack.25226*
> > *ERROR: Task
> > (/home/w23698/projects/proj_dvp2/build_dvp2/../sources/meta-ww-dvp2/re
> > cipes-ww/ww-pacman/ww-pacman.bb:do_unpack)
> > failed with exit code '1'
> > *
> > I don't it this matters but, the build machine is baremetal Ubuntu
> > 18.04.3 LTS. In all cases the packed tar.gz remain in downloads 
> > directory with one name!
> > 
> > Cordially,
> > Georgi
> > 
> > 
> > 
> > 
> 
> 
> ***
> The information in this email is confidential and intended solely for the individual or entity to whom it is addressed.  If you have received this email in error please notify the sender by return e-mail, delete this email, and refrain from any disclosure or action based on the information.
> ***
> 
> ***
> The information in this email is confidential and intended solely for the individual or entity to whom it is addressed.  If you have received this email in error please notify the sender by return e-mail, delete this email, and refrain from any disclosure or action based on the information.
> ***

>