bitbake git fetcher aborts during do_unpack with UnicodeDecodeError

bitbake git fetcher aborts during do_unpack with UnicodeDecodeError

[Klauer, Daniel]

[-- Attachment #1: Type: text/plain, Size: 3288 bytes --]

Hello,

we're using Yocto (jethro) with some custom recipes that retrieve source code from Git and use AUTOREV, for example:

SRC_URI = "git://url/project.git;protocol=ssh"
SRCREV = "${AUTOREV}"

Building the image with bitbake works on one machine, but fails on another with an error like this (full error attached):

File: '.../poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 812, function: runfetchcmd
     0808:
    0809:    for var in exportvars:
     0810:        val = d.getVar(var, True)
     0811:        if val:
 *** 0812:            cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd)
     0813:
     0814:    logger.debug(1, "Running %s", cmd)
     0815:
     0816:    success = False
Exception: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 17: ordinal not in range(128)

It appears that bitbake's git fetcher is prepending shell export commands for certain environment variables (HOME, PATH, but also others - see [1]) to every shell command it runs via runfetchcmd(). Apparently in our case sometimes at least one of these contains non-ASCII bytes (e.g. UTF8 user names).

This by itself is probably ok, but these byte strings are added to the cmd variable, which sometimes is a unicode string - thus causing a decoding to Unicode. It happens because FetchMethod.latest_revision() caches the HEAD revision as a string in an SQL database (bb.persist_data.SQLTable) using the Python sqlite3 module, which returns unicode strings when querying text (by default, see [2]). Then this unicode string variable holding the HEAD rev trickles down to runfetchcmd() where it (sometimes) triggers the UnicodeDecodeError.


Reproducing the issue seems to be as simple as:

    1. $ git clone -b jethro git://git.yoctoproject.org/poky.git
    2. $ cd poky
    3. $ mkdir meta/recipes-support/test
    4. create recipe meta/recipes-support/test/testgit.bb:
        # just a test recipe
        LICENSE = "CLOSED"
        SRC_URI = "git://github.com/schacon/simplegit.git;protocol=https"
        SRCREV = "${AUTOREV}"
    5. $ source oe-init-build-env
    6. $ SOCKS5_USER=ü bitbake testgit

I.e. setting one of the environment variables handled by runfetchcmd() to something containing non-ASCII UTF8 bytes, and building a recipe that uses Git and AUTOREV.


Now I'm wondering, how to best solve this problem? I don't have much experience with bitbake or even Python for that matter. The Python sqlite3 module documentation suggests setting text_factory = str to get byte strings instead of unicode strings. It seems to solve the problem here, but I have no idea if it's the right solution:

--- a/bitbake/lib/bb/persist_data.py
+++ b/bitbake/lib/bb/persist_data.py
@@ -201,6 +201,7 @@ class PersistData(object):
 def connect(database):
     connection = sqlite3.connect(database, timeout=5, isolation_level=None)
     connection.execute("pragma synchronous = off;")
+    connection.text_factory = str
     return connection
 
 def persist(domain, d):


Best regards,
Daniel Klauer

[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/__init__.py?h=jethro&id=2fb7ee2628e23d7efc9b041bb9daae7c4a8de541#n789
[2] https://docs.python.org/2/library/sqlite3.html#sqlite-and-python-types

[-- Attachment #2: full-bitbake-error.txt --]
[-- Type: text/plain, Size: 2297 bytes --]

ERROR: Error executing a python function in .../project/project.bb:

The stack trace of python calls that resulted in this exception/failure was:
File: 'base_do_unpack', lineno: 23, function: <module>
     0019:    except bb.fetch2.BBFetchException as e:
     0020:        raise bb.build.FuncFailed(e)
     0021:
     0022:
 *** 0023:base_do_unpack(d)
     0024:
File: 'base_do_unpack', lineno: 18, function: base_do_unpack
     0014:    bb.utils.remove(p_dir, True)
     0015:
     0016:    try:
     0017:        fetcher = bb.fetch2.Fetch(src_uri, d)
 *** 0018:        fetcher.unpack(rootdir)
     0019:    except bb.fetch2.BBFetchException as e:
     0020:        raise bb.build.FuncFailed(e)
     0021:
     0022:
File: '.../poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 1695, function: unpack
     1691:
     1692:            if ud.lockfile:
     1693:                lf = bb.utils.lockfile(ud.lockfile)
     1694:
 *** 1695:            ud.method.unpack(ud, root, self.d)
     1696:
     1697:            if ud.lockfile:
     1698:                bb.utils.unlockfile(lf)
     1699:
File: '.../poky/bitbake/lib/bb/fetch2/git.py', lineno: 291, function: unpack
     0287:                runfetchcmd("%s checkout-index -q -f -a" % ud.basecmd, d)
     0288:            elif not ud.nobranch:
     0289:                branchname =  ud.branches[ud.names[0]]
     0290:                runfetchcmd("%s checkout -B %s %s" % (ud.basecmd, branchname, \
 *** 0291:                            ud.revisions[ud.names[0]]), d)
     0292:                runfetchcmd("%s branch --set-upstream %s origin/%s" % (ud.basecmd, branchname, \
     0293:                            branchname), d)
     0294:            else:
     0295:                runfetchcmd("%s checkout %s" % (ud.basecmd, ud.revisions[ud.names[0]]), d)
File: '.../poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 812, function: runfetchcmd
     0808:
    0809:    for var in exportvars:
     0810:        val = d.getVar(var, True)
     0811:        if val:
 *** 0812:            cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd)
     0813:
     0814:    logger.debug(1, "Running %s", cmd)
     0815:
     0816:    success = False
Exception: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 17: ordinal not in range(128)