Re: [osstest test] 60719: tolerable FAIL - PUSHED

[Ian Campbell <ian.campbell@citrix.com>]

On Thu, 2015-09-03 at 00:38 -0600, Jim Fehlig wrote:
> AFAICT, this error means the source libvirtd cannot open a tcp connection to the 
> destination libvirtd during the 'perform' phase of migration. In the preceding 
> 'prepare' phase, the destination libvirtd opened a socket to listen for the 
> incoming migration, and passed the connection details back to the source 
> libvirtd. The connection details (hostname:port) are generated on the 
> destination libvirtd with
> 
> virGetHostname():virPortAllocatorAcquire()
> 
> virPortAllocatorAcquire() grabs the next available port in a range of ports. 
> virGetHostName() attempts to get the FQDN of the host
> 
> http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/util/virutil.c;h=cddc78a700c12a4f786a1f6544b92b8ee19c85f5;hb=HEAD#l632
>
> Seems the source libvirtd cannot connect to the hostname:port created by the 
> destination libvirtd.

Indeed. I've now got two boxes setup to do this and in the libvirtd.log of
the source host I see:

2015-09-03 10:03:56.154+0000: 3440: error : virNetSocketNewConnectTCP:578 : unable to connect to server at 'lace-bug.xs.citrite.net:49154': Connection refused
2015-09-03 10:03:56.154+0000: 3440: error : libxlDomainMigrationPerform:501 : unable to connect to 'lace-bug.xs.citrite.net:49154': Invalid argument

It seems like libxlDomainMigrationPerform is clobbering the errno from
virNetSocketNewConnectTCP. I sent a patch for that:

http://lists.xen.org/archives/html/xen-devel/2015-09/msg00320.html

Looking further at the test failure on the destination host I see:

2015-09-03 10:03:56.133+0000: 3463: info : virNetSocketNew:277 : RPC_SOCKET_NEW: sock=0x7fbb768807a0 fd=28 errfd=-1 pid=0 localAddr=127.0.1.1;49154, remoteAddr=<null>

Notice that it has bound to 127.0.1.1 and not to 10.80.228.77!

I suspect this is down to:

    root@lace-bug    :/etc/libvirt# cat /etc/hosts
    127.0.0.1    	    localhost
    127.0.1.1    	    lace-bug.xs.citrite.net    	    lace-bug

    # The following lines are desirable for IPv6 capable hosts
    ::1     localhost ip6-localhost ip6-loopback
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters

And in particular the line associating 127.0.1.1 with lace
-bug.xs.citrite.net.

This seems to be a Debian thing, possibly the installer I'm not sure.

https://lists.debian.org/debian-devel/2013/07/msg00809.html looks relevant.

Overall I'm not sure what to do here. The Debian config seems a bit odd,
but I'm not sure if it is actually "wrong". OTOH I'm not sure how libvirt
could be changed to work in this scenario.

Ian.