Git, AUTOREV and SSH

Git, AUTOREV and SSH

[Andreas Oberritter]

Hi,

there's a problem with the Git fetcher when used with AUTOREV, at least
with SSH URIs. "git ls-remote" may output warnings on stderr, e.g.:

X11 forwarding request failed on channel 0

When this happens, "X11" becomes the SRCREV and the fetcher subsequently
fails. Certainly, other warnings by Git, SSH or a different SCM or
transport could appear under other circumstances. Even localized strings
may appear. Fetch2 merges stderr into stdout on purpose [1], so I'm not
sure a proper fix should look like.

Regards,
Andreas

[1] See runfetchcmd() in lib/bb/fetch2/__init__.py

Re: Git, AUTOREV and SSH

[Richard Purdie]

On Tue, 2012-02-28 at 12:52 +0100, Andreas Oberritter wrote:
> Hi,
> 
> there's a problem with the Git fetcher when used with AUTOREV, at least
> with SSH URIs. "git ls-remote" may output warnings on stderr, e.g.:
> 
> X11 forwarding request failed on channel 0
> 
> When this happens, "X11" becomes the SRCREV and the fetcher subsequently
> fails. Certainly, other warnings by Git, SSH or a different SCM or
> transport could appear under other circumstances. Even localized strings
> may appear. Fetch2 merges stderr into stdout on purpose [1], so I'm not
> sure a proper fix should look like.
> 
> Regards,
> Andreas
> 
> [1] See runfetchcmd() in lib/bb/fetch2/__init__.py

Would the change in
http://lists.linuxtogo.org/pipermail/bitbake-devel/2012-January/001771.html help?

I've still not had a chance to rebase and apply that patch :(

Cheers,

Richard

Re: Git, AUTOREV and SSH

[Andreas Oberritter]

On 28.02.2012 13:03, Richard Purdie wrote:
> On Tue, 2012-02-28 at 12:52 +0100, Andreas Oberritter wrote:
>> Hi,
>>
>> there's a problem with the Git fetcher when used with AUTOREV, at least
>> with SSH URIs. "git ls-remote" may output warnings on stderr, e.g.:
>>
>> X11 forwarding request failed on channel 0
>>
>> When this happens, "X11" becomes the SRCREV and the fetcher subsequently
>> fails. Certainly, other warnings by Git, SSH or a different SCM or
>> transport could appear under other circumstances. Even localized strings
>> may appear. Fetch2 merges stderr into stdout on purpose [1], so I'm not
>> sure a proper fix should look like.
>>
>> Regards,
>> Andreas
>>
>> [1] See runfetchcmd() in lib/bb/fetch2/__init__.py
> 
> Would the change in
> http://lists.linuxtogo.org/pipermail/bitbake-devel/2012-January/001771.html help?
> 
> I've still not had a chance to rebase and apply that patch :(

Yes, this patch did help. See below for a rebased version. Thanks!

Regards,
Andreas

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 07aac4c..42fef69 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -392,6 +392,9 @@ def runfetchcmd(cmd, d, quiet = False, cleanup = []):
     Optionally remove the files/directories listed in cleanup upon failure
     """
 
+    import bb.process
+    import subprocess
+
     # Need to export PATH as binary could be in metadata paths
     # rather than host provided
     # Also include some other variables.
@@ -409,36 +412,27 @@ def runfetchcmd(cmd, d, quiet = False, cleanup = []):
 
     logger.debug(1, "Running %s", cmd)
 
-    # redirect stderr to stdout
-    stdout_handle = os.popen(cmd + " 2>&1", "r")
-    output = ""
-
-    while True:
-        line = stdout_handle.readline()
-        if not line:
-            break
-        if not quiet:
-            print(line, end=' ')
-        output += line
-
-    status = stdout_handle.close() or 0
-    signal = os.WTERMSIG(status)
-    if os.WIFEXITED(status):
-        exitstatus = os.WEXITSTATUS(status)
-    else:
-        exitstatus = 0
+    success = False
+    error_message = ""
+
+    try:
+        (output, errors) = bb.process.run(cmd, shell=True, stderr=subprocess.PIPE)
+        success = True
+    except bb.process.NotFoundError as e:
+        error_message = "Fetch command %s" % (e.command)
+    except bb.process.CmdError as e:
+        error_message = "Fetch command %s could not be run:\n%s" % (e.command, e.msg)
+    except bb.process.ExecutionError as e:
+        error_message = "Fetch command %s failed with exit code %s, output:\n%s" % (e.command, e.exitcode, e.stderr)
 
-    if (signal or status != 0):
+    if not success:
         for f in cleanup:
             try:
                 bb.utils.remove(f, True)
             except OSError:
                 pass
 
-        if signal:
-            raise FetchError("Fetch command %s failed with signal %s, output:\n%s" % (cmd, signal, output))
-        elif exitstatus:
-            raise FetchError("Fetch command %s failed with exit code %s, output:\n%s" % (cmd, exitstatus, output))
+        raise FetchError(error_message)
 
     return output
 

Re: Git, AUTOREV and SSH

[Richard Purdie]

On Tue, 2012-02-28 at 17:32 +0100, Andreas Oberritter wrote:
> On 28.02.2012 13:03, Richard Purdie wrote:
> > On Tue, 2012-02-28 at 12:52 +0100, Andreas Oberritter wrote:
> >> Hi,
> >>
> >> there's a problem with the Git fetcher when used with AUTOREV, at least
> >> with SSH URIs. "git ls-remote" may output warnings on stderr, e.g.:
> >>
> >> X11 forwarding request failed on channel 0
> >>
> >> When this happens, "X11" becomes the SRCREV and the fetcher subsequently
> >> fails. Certainly, other warnings by Git, SSH or a different SCM or
> >> transport could appear under other circumstances. Even localized strings
> >> may appear. Fetch2 merges stderr into stdout on purpose [1], so I'm not
> >> sure a proper fix should look like.
> >>
> >> Regards,
> >> Andreas
> >>
> >> [1] See runfetchcmd() in lib/bb/fetch2/__init__.py
> > 
> > Would the change in
> > http://lists.linuxtogo.org/pipermail/bitbake-devel/2012-January/001771.html help?
> > 
> > I've still not had a chance to rebase and apply that patch :(
> 
> Yes, this patch did help. See below for a rebased version. Thanks!

Thanks very much, I applied this.

Cheers,

Richard