[Qemu-devel] Fail to start 2nd guest

[Qemu-devel] Fail to start 2nd guest

[Xiong Zhou]

Hi,

It worked fine on Linus tree commit:
  7bb0338 Merge tag 'rodata-v4.11-rc1' of git://git.kernel.org/pub/scm/..

failed to start 2nd domain on this commit:
37c8596 Merge tag 'tty-4.11-rc1' of git://git.kernel.org/pub/scm/..
(this commit probably is not the first bad, i didn't do the bisecting)


sh-4.2# uname -r
4.10.0-master-37c8596+
sh-4.2# rpm -qv qemu
qemu-2.0.0-1.el7.6.x86_64
sh-4.2# ps ajxf | grep qemu
11441 11462 11461 10972 pts/0    11461 S+       0   0:00              \_ grep qemu
    1 11065 11064 11064 ?           -1 Sl     107   2:03 /usr/libexec/qemu-kvm -name 73h -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell,-hle,-rtm -m 8000 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid aab09e0f-d48a-4223-9b81-35d4718d59fc -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-73h/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/rhel73.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/var/lib/libvirt/images/73h.qcow2,format=qcow2,if=none,id=drive-ide0-0-1 -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -drive file=/root/RHEL-7.2-20151030.0-Server-x86_64-dvd1.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2 -netdev tap,fd=25,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:83:12:7b,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -global qxl-vga.vgamem_mb=16 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
sh-4.2# virsh list
 Id    Name                           State
----------------------------------------------------
 1     73h                            running

sh-4.2# virsh start 73us
error: Failed to start domain 73us
error: internal error: qemu unexpectedly closed the monitor: ((null):11497): Spice-Warning **: reds.c:2499:reds_init_socket: listen: Address already in use
2017-02-27T09:33:42.335708Z qemu-kvm: failed to initialize spice server

sh-4.2# 

Re: [Qemu-devel] Fail to start 2nd guest

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4112 bytes --]

On Mon, Feb 27, 2017 at 05:40:50PM +0800, Xiong Zhou wrote:
> It worked fine on Linus tree commit:
>   7bb0338 Merge tag 'rodata-v4.11-rc1' of git://git.kernel.org/pub/scm/..
> 
> failed to start 2nd domain on this commit:
> 37c8596 Merge tag 'tty-4.11-rc1' of git://git.kernel.org/pub/scm/..
> (this commit probably is not the first bad, i didn't do the bisecting)
> 
> 
> sh-4.2# uname -r
> 4.10.0-master-37c8596+
> sh-4.2# rpm -qv qemu
> qemu-2.0.0-1.el7.6.x86_64
> sh-4.2# ps ajxf | grep qemu
> 11441 11462 11461 10972 pts/0    11461 S+       0   0:00              \_ grep qemu
>     1 11065 11064 11064 ?           -1 Sl     107   2:03 /usr/libexec/qemu-kvm -name 73h -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell,-hle,-rtm -m 8000 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid aab09e0f-d48a-4223-9b81-35d4718d59fc -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-73h/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/rhel73.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/var/lib/libvirt/images/73h.qcow2,format=qcow2,if=none,id=drive-ide0-0-1 -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -drive file=/root/RHEL-7.2-20151030.0-Server-x86_64-dvd1.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2 -netdev tap,fd=25,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:83:12:7b,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -global qxl-vga.vgamem_mb=16 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
> sh-4.2# virsh list
>  Id    Name                           State
> ----------------------------------------------------
>  1     73h                            running
> 
> sh-4.2# virsh start 73us
> error: Failed to start domain 73us
> error: internal error: qemu unexpectedly closed the monitor: ((null):11497): Spice-Warning **: reds.c:2499:reds_init_socket: listen: Address already in use
> 2017-02-27T09:33:42.335708Z qemu-kvm: failed to initialize spice server

The error message says that the spice remote desktop cannot listen on
-spice port=5900,addr=127.0.0.1.

Did you hardcode port 5900 in the domain XML?  That could explain why
the second guest fails to launch - you need to use unique port numbers
or let libvirt automatically assign them.  Check the domain XML:

  <graphics type='spice' port='-1' tlsPort='-1' autoport='yes'>

Another possibility is that a process running on the host is already
using port 5900.  Perhaps a guest or VNC server that was launched
outside of libvirt?  You can check this with:

  netstat -alpn | grep 5900

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Fail to start 2nd guest

[Xiong Zhou]

On Mon, Feb 27, 2017 at 10:11:04AM +0000, Stefan Hajnoczi wrote:
> On Mon, Feb 27, 2017 at 05:40:50PM +0800, Xiong Zhou wrote:
> > It worked fine on Linus tree commit:
> >   7bb0338 Merge tag 'rodata-v4.11-rc1' of git://git.kernel.org/pub/scm/..
> > 
> > failed to start 2nd domain on this commit:
> > 37c8596 Merge tag 'tty-4.11-rc1' of git://git.kernel.org/pub/scm/..
> > (this commit probably is not the first bad, i didn't do the bisecting)
> > 
> > 
> > sh-4.2# uname -r
> > 4.10.0-master-37c8596+
> > sh-4.2# rpm -qv qemu
> > qemu-2.0.0-1.el7.6.x86_64
> > sh-4.2# ps ajxf | grep qemu
> > ...
> > sh-4.2# virsh list
> >  Id    Name                           State
> > ----------------------------------------------------
> >  1     73h                            running
> > 
> > sh-4.2# virsh start 73us
> > error: Failed to start domain 73us
> > error: internal error: qemu unexpectedly closed the monitor: ((null):11497): Spice-Warning **: reds.c:2499:reds_init_socket: listen: Address already in use
> > 2017-02-27T09:33:42.335708Z qemu-kvm: failed to initialize spice server
> 
> The error message says that the spice remote desktop cannot listen on
> -spice port=5900,addr=127.0.0.1.
> 
> Did you hardcode port 5900 in the domain XML?  That could explain why
No.
> the second guest fails to launch - you need to use unique port numbers
> or let libvirt automatically assign them.  Check the domain XML:
> 
>   <graphics type='spice' port='-1' tlsPort='-1' autoport='yes'>

It looks like:

    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>


> 
> Another possibility is that a process running on the host is already
> using port 5900.  Perhaps a guest or VNC server that was launched
> outside of libvirt?  You can check this with:
> 
>   netstat -alpn | grep 5900
# netstat -alpn | grep 5900
tcp        0      0 127.0.0.1:5900          0.0.0.0:*               LISTEN      11065/qemu-kvm      
> 
> Stefan

Re: [Qemu-devel] Fail to start 2nd guest

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2440 bytes --]

On Mon, Feb 27, 2017 at 08:50:03PM +0800, Xiong Zhou wrote:
> On Mon, Feb 27, 2017 at 10:11:04AM +0000, Stefan Hajnoczi wrote:
> > On Mon, Feb 27, 2017 at 05:40:50PM +0800, Xiong Zhou wrote:
> > > It worked fine on Linus tree commit:
> > >   7bb0338 Merge tag 'rodata-v4.11-rc1' of git://git.kernel.org/pub/scm/..
> > > 
> > > failed to start 2nd domain on this commit:
> > > 37c8596 Merge tag 'tty-4.11-rc1' of git://git.kernel.org/pub/scm/..
> > > (this commit probably is not the first bad, i didn't do the bisecting)
> > > 
> > > 
> > > sh-4.2# uname -r
> > > 4.10.0-master-37c8596+
> > > sh-4.2# rpm -qv qemu
> > > qemu-2.0.0-1.el7.6.x86_64
> > > sh-4.2# ps ajxf | grep qemu
> > > ...
> > > sh-4.2# virsh list
> > >  Id    Name                           State
> > > ----------------------------------------------------
> > >  1     73h                            running
> > > 
> > > sh-4.2# virsh start 73us
> > > error: Failed to start domain 73us
> > > error: internal error: qemu unexpectedly closed the monitor: ((null):11497): Spice-Warning **: reds.c:2499:reds_init_socket: listen: Address already in use
> > > 2017-02-27T09:33:42.335708Z qemu-kvm: failed to initialize spice server
> > 
> > The error message says that the spice remote desktop cannot listen on
> > -spice port=5900,addr=127.0.0.1.
> > 
> > Did you hardcode port 5900 in the domain XML?  That could explain why
> No.
> > the second guest fails to launch - you need to use unique port numbers
> > or let libvirt automatically assign them.  Check the domain XML:
> > 
> >   <graphics type='spice' port='-1' tlsPort='-1' autoport='yes'>
> 
> It looks like:
> 
>     <graphics type='spice' autoport='yes'>
>       <listen type='address'/>
>       <image compression='off'/>
>     </graphics>
> 
> 
> > 
> > Another possibility is that a process running on the host is already
> > using port 5900.  Perhaps a guest or VNC server that was launched
> > outside of libvirt?  You can check this with:
> > 
> >   netstat -alpn | grep 5900
> # netstat -alpn | grep 5900
> tcp        0      0 127.0.0.1:5900          0.0.0.0:*               LISTEN      11065/qemu-kvm      

Please check that 11065/qemu-kvm was launched by the same libvirtd and
its domain XML also uses autoport='yes'.

I have CCed the libvirt mailing list because they may be able to explain
why there is a collision on TCP port 5900.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [libvirt] Fail to start 2nd guest

[Peter Krempa]

[-- Attachment #1: Type: text/plain, Size: 2010 bytes --]

On Mon, Feb 27, 2017 at 13:31:30 +0000, Stefan Hajnoczi wrote:
> On Mon, Feb 27, 2017 at 08:50:03PM +0800, Xiong Zhou wrote:
> > On Mon, Feb 27, 2017 at 10:11:04AM +0000, Stefan Hajnoczi wrote:
> > > On Mon, Feb 27, 2017 at 05:40:50PM +0800, Xiong Zhou wrote:

[...]

> > > > sh-4.2# virsh start 73us
> > > > error: Failed to start domain 73us
> > > > error: internal error: qemu unexpectedly closed the monitor: ((null):11497): Spice-Warning **: reds.c:2499:reds_init_socket: listen: Address already in use
> > > > 2017-02-27T09:33:42.335708Z qemu-kvm: failed to initialize spice server
> > > 
> > > The error message says that the spice remote desktop cannot listen on
> > > -spice port=5900,addr=127.0.0.1.
> > > 
> > > Did you hardcode port 5900 in the domain XML?  That could explain why
> > No.
> > > the second guest fails to launch - you need to use unique port numbers
> > > or let libvirt automatically assign them.  Check the domain XML:
> > > 
> > >   <graphics type='spice' port='-1' tlsPort='-1' autoport='yes'>
> > 
> > It looks like:
> > 
> >     <graphics type='spice' autoport='yes'>
> >       <listen type='address'/>
> >       <image compression='off'/>
> >     </graphics>
> > 
> > 
> > > 
> > > Another possibility is that a process running on the host is already
> > > using port 5900.  Perhaps a guest or VNC server that was launched
> > > outside of libvirt?  You can check this with:
> > > 
> > >   netstat -alpn | grep 5900
> > # netstat -alpn | grep 5900
> > tcp        0      0 127.0.0.1:5900          0.0.0.0:*               LISTEN      11065/qemu-kvm      
> 
> Please check that 11065/qemu-kvm was launched by the same libvirtd and
> its domain XML also uses autoport='yes'.
> 
> I have CCed the libvirt mailing list because they may be able to explain
> why there is a collision on TCP port 5900.

There was a bug in the code. There could be a race between startup of
two VMs:
https://bugzilla.redhat.com/show_bug.cgi?id=1397440

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]