Re: [osstest test] 60719: tolerable FAIL - PUSHED

[Jim Fehlig <jfehlig@suse.com>]

On 09/01/2015 07:14 AM, Ian Campbell wrote:
> On Tue, 2015-09-01 at 13:47 +0100, Ian Jackson wrote:
>> Jim Fehlig writes ("Re: [osstest test] 60719: tolerable FAIL - PUSHED"):
>>> This sounds a bit like an issue discussed in the Redhat libvirt
>>> troubleshooting FAQ
>>>
>>> https://access.redhat.com/documentation/en
>>> -US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Admin
>>> istration_Guide/sect-Troubleshooting
>>> -Common_libvirt_errors_and_troubleshooting.html#sect
>>> -Migration_fails_with_Error_unable_to_resolve_address
>>> Right. If it is a DNS issue, error handling in the libvirt libxl
>>> migration code needs improving.
>> I booked out a test host, and (as I expected) forward DNS works, but
>> reverse DNS on test box IP addresses does not:
> As discussed IRL I was also investigating this using the Cambridge
> instance, which does have correct reverse DNS:
>
>      root@moss-bug    :~# host moss-bug.xs.citrite.net
>      moss-bug.xs.citrite.net has address 10.80.229.144
>      root@moss-bug    :~# host -i 10.80.229.144
>      144.229.80.10.in-addr.arpa domain name pointer moss-bug.xs.citrite.net.
>      root@moss-bug    :~# domainname -f
>      moss-bug.xs.citrite.net
>      root@moss-bug    :~# cat /etc/hosts
>      127.0.0.1    	    localhost
>      127.0.1.1    	    moss-bug.xs.citrite.net    	    moss-bug
>
>      # The following lines are desirable for IPv6 capable hosts
>      ::1     localhost ip6-localhost ip6-loopback
>      ff02::1 ip6-allnodes
>      ff02::2 ip6-allrouters
>      root@moss-bug    :~#
>
> (previously these machines had a wrong idea about their own FQDN, that is
> now fixed)
>
> I am now seeing the same error as the production instance:
>
> 2015-09-01 12:10:00 Z executing ssh ... root@10.80.229.144 virsh --debug 0 migrate --live debian.guest.osstest xen+ssh://10.80.228.77
> migrate: live(bool): (none)
> migrate: domain(optdata): debian.guest.osstest
> migrate: desturi(optdata): xen+ssh://10.80.228.77
> migrate: found option <domain>: debian.guest.osstest
> migrate: <domain> trying as domain NAME
> migrate: found option <domain>: debian.guest.osstest
> migrate: <domain> trying as domain NAME
> error: unable to connect to 'lace-bug.xs.citrite.net:49152': Invalid argument

AFAICT, this error means the source libvirtd cannot open a tcp connection to the 
destination libvirtd during the 'perform' phase of migration. In the preceding 
'prepare' phase, the destination libvirtd opened a socket to listen for the 
incoming migration, and passed the connection details back to the source 
libvirtd. The connection details (hostname:port) are generated on the 
destination libvirtd with

virGetHostname():virPortAllocatorAcquire()

virPortAllocatorAcquire() grabs the next available port in a range of ports. 
virGetHostName() attempts to get the FQDN of the host

http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/util/virutil.c;h=cddc78a700c12a4f786a1f6544b92b8ee19c85f5;hb=HEAD#l632

Seems the source libvirtd cannot connect to the hostname:port created by the 
destination libvirtd.

Regards,
Jim