[Cluster-devel] [fence-virt PATCH] fix handling of VIR_DOMAIN_EVENT_STARTED

[Cluster-devel] [fence-virt PATCH] fix handling of VIR_DOMAIN_EVENT_STARTED

[Kazunori INOUE]

Hi all,

I am using fence-virt.
When using a serial listener, fence_virtd does not receive a request
from VM which started after fence_virtd start.

Step 1) Start fence_virtd.
         # fence_virtd -d5 -F
Step 2) And then start VM.
         # virsh start srv-a1
Step 3) A request sent from the VM,
         # fence_virt -D/dev/ttyS1 -oXXX
        but fence_virtd not receive it.

Below, x3650f is a host, srv-XX is a VM.

[root at x3650f ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.1 (Santiago)
[root at x3650f ~]#
[root at x3650f ~]# rpm -qa | egrep '^libvirt|^qemu' | sort
libvirt-0.8.7-18.el6.x86_64
libvirt-client-0.8.7-18.el6.x86_64
libvirt-devel-0.8.7-18.el6.x86_64
libvirt-java-0.4.7-1.el6.noarch
libvirt-java-devel-0.4.7-1.el6.noarch
libvirt-python-0.8.7-18.el6.x86_64
libvirt-qpid-0.2.22-6.el6.x86_64
qemu-img-0.12.1.2-2.160.el6.x86_64
qemu-kvm-0.12.1.2-2.160.el6.x86_64
[root at x3650f ~]#

And I used following fence-virt.
http://fence-virt.git.sourceforge.net/git/gitweb.cgi?p=fence-virt/fence-virt;a=commit;h=53a2b293d12991beb6c2927f78586257bff5173a

[root at x3650f ~]# tar xfz fence-virt-53a2b29.tar.gz && cd fence-virt-53a2b29 && ./build
[root at x3650f ~]# cd && mkdir plugins && cd plugins
[root at x3650f plugins]# ln -s /root/fence-virt-53a2b29/server/serial.so .
[root at x3650f plugins]# ln -s /root/fence-virt-53a2b29/server/libvirt.so .
[root at x3650f plugins]# ls -l
lrwxrwxrwx 1 root root 42 Sep 13 10:28 libvirt.so -> /root/fence-virt-53a2b29/server/libvirt.so
lrwxrwxrwx 1 root root 41 Sep 13 10:28 serial.so -> /root/fence-virt-53a2b29/server/serial.so
[root at x3650f plugins]#

I attached /etc/fence_virt.conf and /etc/libvirt/qemu/srv-a1.xml,
please refer to it.

Step 1)
  Start fence_virtd.

  [root at x3650f ~]# ./fence-virt-53a2b29/server/fence_virtd -d5 -F
  Background mode disabled
  Debugging threshold is now 5
  (snip)
  Backend plugin: libvirt
  Listener plugin: serial
  Searching /root/plugins for plugins...
  Searching for plugins in /root/plugins
  Loading plugin from /root/plugins/serial.so
  Failed to map backend_plugin_version
  Registered listener plugin serial 0.4
  Loading plugin from /root/plugins/libvirt.so
  Registered backend plugin libvirt 0.1
  2 plugins found
  Available backends:
      libvirt 0.1
  Available listeners:
      serial 0.4
  Debugging threshold is now 5
  Using qemu:///system
  Debugging threshold is now 5
  Got /var/lib/libvirt/qemu for uri
  Got serial for mode
  Libvirt event listener starting
   * Socket path: /var/lib/libvirt/qemu
   * Mode: Serial
  myEventAddHandleFunc:128: Add handle 6 1 0x37558aef80 0x7f68cc000920 (nil)
  myEventAddTimeoutFunc:160: Adding Timeout -1 0x37558a7f90 0x7f68cc000920
  event_thread:448 :: Registering domain event cbs
  myEventUpdateHandleFunc:140: Updated Handle 0 0
  myEventUpdateHandleFunc:140: Updated Handle 0 1
  myEventUpdateHandleFunc:140: Updated Handle 0 0
  myEventUpdateHandleFunc:140: Updated Handle 0 1
                                                   <- Step 2) I started VM here.
  myEventUpdateHandleFunc:140: Updated Handle 0 0
  myEventUpdateHandleFunc:140: Updated Handle 0 1
  Registered 621d8c61-1070-7aab-6158-3889d68470ab on 9
                                                   <- Step 3) I run fence_virt here.
  (snip)

Step 3)
  A request sent from the VM, but fence_virtd not receive it.

  [root at srv-a1 ~]# ./fence-virt-53a2b29/client/fence_virt -D/dev/ttyS1 -olist
  Unknown response (255)
  [root at srv-a1 ~]#

I attached the patch for this problem.

* Modified to send a SIGHUP to the primary thread from the secondary
  thread detects the start of the VM, and
  the primary thread is modified so as to select() the requests from
  the VM which started.
* Also, when four VM(s) are started simultaneously,
  poll() <in event_thread()@server/virt-serial.c> could not handle
  all the boot.
  Therefore, I revised it to call domainStarted()/domainStopped() in
  myDomainEventCallback1().

Regards,
Kazunori INOUE
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fix-handling-VIR_DOMAIN_EVENT_STARTED.patch
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20110913/d11987f2/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fence_virt.conf
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20110913/d11987f2/attachment.conf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: srv-a1.xml
Type: text/xml
Size: 2530 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20110913/d11987f2/attachment.xml>

[Cluster-devel] [fence-virt PATCH] fix handling of VIR_DOMAIN_EVENT_STARTED

[Lon Hohberger]

On 09/13/2011 04:44 AM, Kazunori INOUE wrote:

> I attached the patch for this problem.
> 
> * Modified to send a SIGHUP to the primary thread from the secondary
>    thread detects the start of the VM, and
>    the primary thread is modified so as to select() the requests from
>    the VM which started.

That's certainly one way of doing it.  I might have used
'pthread_cond_singal'.

> * Also, when four VM(s) are started simultaneously,
>    poll()<in event_thread()@server/virt-serial.c>  could not handle
>    all the boot.
>    Therefore, I revised it to call domainStarted()/domainStopped() in
>    myDomainEventCallback1().

You're using pthread_kill, so what I suspect happens is 4 SIGHUPs are
coming in very quickly, causing more modification to be required.

I'll look at this a bit more with Marek.

-- Lon

[Cluster-devel] [fence-virt PATCH] fix handling of VIR_DOMAIN_EVENT_STARTED

[Lon Hohberger]

On 09/19/2011 04:32 PM, Lon Hohberger wrote:
> On 09/13/2011 04:44 AM, Kazunori INOUE wrote:
> 
>> I attached the patch for this problem.
>>
>> * Modified to send a SIGHUP to the primary thread from the secondary
>>     thread detects the start of the VM, and
>>     the primary thread is modified so as to select() the requests from
>>     the VM which started.
> 
> That's certainly one way of doing it.  I might have used
> 'pthread_cond_singal'.

I am sorry for the delay.

I have a fix based on yours that uses a pipe to wake up select() rather
than pthread_kill; I'll attach it here.  This ends up being a bit lower
impact, as well; it doesn't require changing the main program - only the
serial/vmchannel plugin.

Apart from that, the patch here is largely the same.

Let me know what you think at your convenience.

-- Lon
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 0001-Fix-serial-domain-handling.patch
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111025/d7b5bd33/attachment.ksh>

[Cluster-devel] [fence-virt PATCH] fix handling of VIR_DOMAIN_EVENT_STARTED

[Kazunori INOUE]

(2011/10/26 8:30), Lon Hohberger wrote:
> On 09/19/2011 04:32 PM, Lon Hohberger wrote:
>> On 09/13/2011 04:44 AM, Kazunori INOUE wrote:
>>
>>> I attached the patch for this problem.
>>>
>>> * Modified to send a SIGHUP to the primary thread from the secondary
>>>      thread detects the start of the VM, and
>>>      the primary thread is modified so as to select() the requests from
>>>      the VM which started.
>>
>> That's certainly one way of doing it.  I might have used
>> 'pthread_cond_singal'.
> 
> I am sorry for the delay.
> 
> I have a fix based on yours that uses a pipe to wake up select() rather
> than pthread_kill; I'll attach it here.  This ends up being a bit lower
> impact, as well; it doesn't require changing the main program - only the
> serial/vmchannel plugin.
> 
> Apart from that, the patch here is largely the same.
> 
> Let me know what you think at your convenience.
> 
> -- Lon

Hi, Lon

Thank you for your reply.
I tested this patch, and confirmed that the problem was solved.
(It was good also when four VM (s) was started simultaneously.)

Many thanks!

[Cluster-devel] [fence-virt PATCH] fix handling of VIR_DOMAIN_EVENT_STARTED

[Kazunori INOUE]

(11.10.27 16:23), Kazunori INOUE wrote:
> (2011/10/26 8:30), Lon Hohberger wrote:
>> On 09/19/2011 04:32 PM, Lon Hohberger wrote:
>>> On 09/13/2011 04:44 AM, Kazunori INOUE wrote:
>>>
>>>> I attached the patch for this problem.
>>>>
>>>> * Modified to send a SIGHUP to the primary thread from the secondary
>>>>       thread detects the start of the VM, and
>>>>       the primary thread is modified so as to select() the requests from
>>>>       the VM which started.
>>>
>>> That's certainly one way of doing it.  I might have used
>>> 'pthread_cond_singal'.
>>
>> I am sorry for the delay.
>>
>> I have a fix based on yours that uses a pipe to wake up select() rather
>> than pthread_kill; I'll attach it here.  This ends up being a bit lower
>> impact, as well; it doesn't require changing the main program - only the
>> serial/vmchannel plugin.
>>
>> Apart from that, the patch here is largely the same.
>>
>> Let me know what you think at your convenience.
>>
>> -- Lon
> 
> Hi, Lon
> 
> Thank you for your reply.
> I tested this patch, and confirmed that the problem was solved.
> (It was good also when four VM (s) was started simultaneously.)
> 
> Many thanks!

Hi,

Could you please merge?

Regards,
Kazunori INOUE

[Cluster-devel] [fence-virt PATCH] fix handling of VIR_DOMAIN_EVENT_STARTED

[Lon Hohberger]

On 09/13/2011 04:44 AM, Kazunori INOUE wrote:

[all late and wrong]

Merged in to 0.3.0

-- Lon