[Cluster-devel] [fence-virt PATCH] backend plugin for monitoring a host's status

[Cluster-devel] [fence-virt PATCH] backend plugin for monitoring a host's status

[Kazunori INOUE]

Hi all,

I think that the communication function of fence-virt is flexible,
so I want to use it more effectively.

Therefore I made backend plugin for a guest to get the host's status
using the communication facility of fence-virt,
and I changed to allow specifying one more backend (for fencing, and
replying the host's status).

I created the backend "pm-monitor" which has met the following
configurations / requirements.
- Both hosts and VMs, cluster (Pacemaker) have been configured.

Here's an overview of function. Please refer to attached 'overview.png'.
(*) pingd resource notifies the status of connection with a specific
    host to pacemaker, and pacemaker manages the result.
(1) resource (vm-client) which requires the host's status is executed.
(2) vm-client requests 'host_status (result of pingd)' to the host
    with fence_virt.
(3) use the serial listener,
(4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from
    pacemaker and answers it after conversion.
    - the conversion rule is set in /etc/pm-monitor.conf

Here's a description of the attached files.
* add_general_backend.patch
  - add the server/pm-fence.c
  - change the configure.in and server/Makefile.in
* overview.png
  - figure of the overview.
* vm-client
  - Resource Agent for VM.
  - I'm going to post this RA to the appropriate community
    (https://github.com/ClusterLabs/resource-agents).
* fence_virt.conf
  - sample configuration.
* pm-monitor.conf
  - sample configuration for pm-monitor.so

* host.cli, mon-host.txt
  - sample configuration file of a host cluster,
    and mon-host.txt is an output of the crm_mon command.
* VM.cli, mon-VM.txt
  - sample configuration file of a VM cluster,
    and mon-VM.txt is an output of the crm_mon command.

Best Regards
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: add_general_backend.patch
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment.ksh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: overview.PNG
Type: image/png
Size: 23810 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment.png>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: vm-client
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment-0001.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fence_virt.conf
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment.conf>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: pm-monitor.conf
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment-0001.conf>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: host.cli
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment-0002.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: VM.cli
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment-0003.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mon-host.txt
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mon-VM.txt
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment-0001.txt>

[Cluster-devel] [fence-virt PATCH] backend plugin for monitoring a host's status

[Andrew Beekhof]

Was making host connectivity information available to the guests the
goal here or just something to demonstrate the concept?
Interesting idea :-)

2011/10/5 Kazunori INOUE <inouekazu@intellilink.co.jp>:
> Hi all,
>
> I think that the communication function of fence-virt is flexible,
> so I want to use it more effectively.
>
> Therefore I made backend plugin for a guest to get the host's status
> using the communication facility of fence-virt,
> and I changed to allow specifying one more backend (for fencing, and
> replying the host's status).
>
> I created the backend "pm-monitor" which has met the following
> configurations / requirements.
> - Both hosts and VMs, cluster (Pacemaker) have been configured.
>
> Here's an overview of function. Please refer to attached 'overview.png'.
> (*) pingd resource notifies the status of connection with a specific
> ? ?host to pacemaker, and pacemaker manages the result.
> (1) resource (vm-client) which requires the host's status is executed.
> (2) vm-client requests 'host_status (result of pingd)' to the host
> ? ?with fence_virt.
> (3) use the serial listener,
> (4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from
> ? ?pacemaker and answers it after conversion.
> ? ?- the conversion rule is set in /etc/pm-monitor.conf
>
> Here's a description of the attached files.
> * add_general_backend.patch
> ?- add the server/pm-fence.c
> ?- change the configure.in and server/Makefile.in
> * overview.png
> ?- figure of the overview.
> * vm-client
> ?- Resource Agent for VM.
> ?- I'm going to post this RA to the appropriate community
> ? ?(https://github.com/ClusterLabs/resource-agents).
> * fence_virt.conf
> ?- sample configuration.
> * pm-monitor.conf
> ?- sample configuration for pm-monitor.so
>
> * host.cli, mon-host.txt
> ?- sample configuration file of a host cluster,
> ? ?and mon-host.txt is an output of the crm_mon command.
> * VM.cli, mon-VM.txt
> ?- sample configuration file of a VM cluster,
> ? ?and mon-VM.txt is an output of the crm_mon command.
>
> Best Regards
>

[Cluster-devel] [fence-virt PATCH] backend plugin for monitoring a host's status

[Kazunori INOUE]

Hi, Andrew

We (NTT) will participate in the HA Mini-Conference,
please let we explain at that time.

Best Regards

(2011/10/12 10:40), Andrew Beekhof wrote:
> Was making host connectivity information available to the guests the
> goal here or just something to demonstrate the concept?
> Interesting idea :-)
>
> 2011/10/5 Kazunori INOUE<inouekazu@intellilink.co.jp>:
>> Hi all,
>>
>> I think that the communication function of fence-virt is flexible,
>> so I want to use it more effectively.
>>
>> Therefore I made backend plugin for a guest to get the host's status
>> using the communication facility of fence-virt,
>> and I changed to allow specifying one more backend (for fencing, and
>> replying the host's status).
>>
>> I created the backend "pm-monitor" which has met the following
>> configurations / requirements.
>> - Both hosts and VMs, cluster (Pacemaker) have been configured.
>>
>> Here's an overview of function. Please refer to attached 'overview.png'.
>> (*) pingd resource notifies the status of connection with a specific
>>     host to pacemaker, and pacemaker manages the result.
>> (1) resource (vm-client) which requires the host's status is executed.
>> (2) vm-client requests 'host_status (result of pingd)' to the host
>>     with fence_virt.
>> (3) use the serial listener,
>> (4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from
>>     pacemaker and answers it after conversion.
>>     - the conversion rule is set in /etc/pm-monitor.conf
>>
>> Here's a description of the attached files.
>> * add_general_backend.patch
>>   - add the server/pm-fence.c
>>   - change the configure.in and server/Makefile.in
>> * overview.png
>>   - figure of the overview.
>> * vm-client
>>   - Resource Agent for VM.
>>   - I'm going to post this RA to the appropriate community
>>     (https://github.com/ClusterLabs/resource-agents).
>> * fence_virt.conf
>>   - sample configuration.
>> * pm-monitor.conf
>>   - sample configuration for pm-monitor.so
>>
>> * host.cli, mon-host.txt
>>   - sample configuration file of a host cluster,
>>     and mon-host.txt is an output of the crm_mon command.
>> * VM.cli, mon-VM.txt
>>   - sample configuration file of a VM cluster,
>>     and mon-VM.txt is an output of the crm_mon command.
>>
>> Best Regards

[Cluster-devel] [fence-virt PATCH] backend plugin for monitoring a host's status

[Andrew Beekhof]

On Thu, Oct 13, 2011 at 7:52 PM, Kazunori INOUE
<inouekazu@intellilink.co.jp> wrote:
> Hi, Andrew
>
> We (NTT) will participate in the HA Mini-Conference,
> please let we explain at that time.

Sounds good :-)


> Best Regards
>
> (2011/10/12 10:40), Andrew Beekhof wrote:
>>
>> Was making host connectivity information available to the guests the
>> goal here or just something to demonstrate the concept?
>> Interesting idea :-)
>>
>> 2011/10/5 Kazunori INOUE<inouekazu@intellilink.co.jp>:
>>>
>>> Hi all,
>>>
>>> I think that the communication function of fence-virt is flexible,
>>> so I want to use it more effectively.
>>>
>>> Therefore I made backend plugin for a guest to get the host's status
>>> using the communication facility of fence-virt,
>>> and I changed to allow specifying one more backend (for fencing, and
>>> replying the host's status).
>>>
>>> I created the backend "pm-monitor" which has met the following
>>> configurations / requirements.
>>> - Both hosts and VMs, cluster (Pacemaker) have been configured.
>>>
>>> Here's an overview of function. Please refer to attached 'overview.png'.
>>> (*) pingd resource notifies the status of connection with a specific
>>> ? ?host to pacemaker, and pacemaker manages the result.
>>> (1) resource (vm-client) which requires the host's status is executed.
>>> (2) vm-client requests 'host_status (result of pingd)' to the host
>>> ? ?with fence_virt.
>>> (3) use the serial listener,
>>> (4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from
>>> ? ?pacemaker and answers it after conversion.
>>> ? ?- the conversion rule is set in /etc/pm-monitor.conf
>>>
>>> Here's a description of the attached files.
>>> * add_general_backend.patch
>>> ?- add the server/pm-fence.c
>>> ?- change the configure.in and server/Makefile.in
>>> * overview.png
>>> ?- figure of the overview.
>>> * vm-client
>>> ?- Resource Agent for VM.
>>> ?- I'm going to post this RA to the appropriate community
>>> ? ?(https://github.com/ClusterLabs/resource-agents).
>>> * fence_virt.conf
>>> ?- sample configuration.
>>> * pm-monitor.conf
>>> ?- sample configuration for pm-monitor.so
>>>
>>> * host.cli, mon-host.txt
>>> ?- sample configuration file of a host cluster,
>>> ? ?and mon-host.txt is an output of the crm_mon command.
>>> * VM.cli, mon-VM.txt
>>> ?- sample configuration file of a VM cluster,
>>> ? ?and mon-VM.txt is an output of the crm_mon command.
>>>
>>> Best Regards
>
>

[Cluster-devel] [fence-virt PATCH] backend plugin for monitoring a host's status

[Lon Hohberger]

On 10/05/2011 04:56 AM, Kazunori INOUE wrote:
> Hi all,
> 
> I think that the communication function of fence-virt is flexible,
> so I want to use it more effectively.
> 
> Therefore I made backend plugin for a guest to get the host's status
> using the communication facility of fence-virt,
> and I changed to allow specifying one more backend (for fencing, and
> replying the host's status).
> 
> I created the backend "pm-monitor" which has met the following
> configurations / requirements.
> - Both hosts and VMs, cluster (Pacemaker) have been configured.
> 
> Here's an overview of function. Please refer to attached 'overview.png'.
> (*) pingd resource notifies the status of connection with a specific
>      host to pacemaker, and pacemaker manages the result.
> (1) resource (vm-client) which requires the host's status is executed.
> (2) vm-client requests 'host_status (result of pingd)' to the host
>      with fence_virt.
> (3) use the serial listener,
> (4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from
>      pacemaker and answers it after conversion.
>      - the conversion rule is set in /etc/pm-monitor.conf

Originally, the devstatus callback was supposed to be what provided the
answer to the question "is my fencing [device|host] operational?" - but
your patch is more than that, it looks like a generalized way to do
arbitrary request/response pacemaker resource monitoring from within VMs.

That kind of monitoring is interesting.

> Here's a description of the attached files.
> * add_general_backend.patch
>    - add the server/pm-fence.c
>    - change the configure.in and server/Makefile.in

I think I understand how it operates; your explanation is very detailed.

* I don't quite understand all of the benefits.  Presumably, one uses
the serial configuration to prevent guests/hosts from sharing the same
networks - i.e., it's designed for environments where guests and hosts
are not allowed to talk over the network to each other.  So, presumably,
we're using this to indirectly monitor an IP on the host network, using
fence-virt as a bridge.  What I don't understand is the benefit to the
virtual machine cluster in doing this.

* Do you see other uses besides monitoring pingd sets?

* It may be that this new project may be a better "backbone" for this
sort of general request/response than fence-virt; it's a much more
general communications medium for guest<->host communication:

    http://git.fedorahosted.org/git/?p=vios-proxy.git

(although it is written in C++ ;) ).


Notes about the patch itself:

 * no support for multicast listener (this looks intentional -
   is it?)

 * this will break on-wire compatibility; we need to be very careful
   here.

 * some duplicate build system changes with the other
   patch (not a big deal)

 * [not directly related to your patch] the vmchannel/serial support
   in fence-virt probably should be replaced with more recent
   technology in libvirt

-- Lon

[Cluster-devel] [fence-virt PATCH] backend plugin for monitoring a host's status

[Kazunori INOUE]

Hi, Lon

(2011/10/26 10:35), Lon Hohberger wrote:
> On 10/05/2011 04:56 AM, Kazunori INOUE wrote:
>> Hi all,
>>
>> I think that the communication function of fence-virt is flexible,
>> so I want to use it more effectively.
>>
>> Therefore I made backend plugin for a guest to get the host's status
>> using the communication facility of fence-virt,
>> and I changed to allow specifying one more backend (for fencing, and
>> replying the host's status).
>>
>> I created the backend "pm-monitor" which has met the following
>> configurations / requirements.
>> - Both hosts and VMs, cluster (Pacemaker) have been configured.
>>
>> Here's an overview of function. Please refer to attached 'overview.png'.
>> (*) pingd resource notifies the status of connection with a specific
>>       host to pacemaker, and pacemaker manages the result.
>> (1) resource (vm-client) which requires the host's status is executed.
>> (2) vm-client requests 'host_status (result of pingd)' to the host
>>       with fence_virt.
>> (3) use the serial listener,
>> (4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from
>>       pacemaker and answers it after conversion.
>>       - the conversion rule is set in /etc/pm-monitor.conf
> 
> Originally, the devstatus callback was supposed to be what provided the
> answer to the question "is my fencing [device|host] operational?" - but
> your patch is more than that, it looks like a generalized way to do
> arbitrary request/response pacemaker resource monitoring from within VMs.
> 
> That kind of monitoring is interesting.
> 
>> Here's a description of the attached files.
>> * add_general_backend.patch
>>     - add the server/pm-fence.c
>>     - change the configure.in and server/Makefile.in
> 
> I think I understand how it operates; your explanation is very detailed.
> 
> * I don't quite understand all of the benefits.  Presumably, one uses
> the serial configuration to prevent guests/hosts from sharing the same
> networks - i.e., it's designed for environments where guests and hosts
> are not allowed to talk over the network to each other.  So, presumably,
> we're using this to indirectly monitor an IP on the host network, using
> fence-virt as a bridge.  What I don't understand is the benefit to the
> virtual machine cluster in doing this.
> 
> * Do you see other uses besides monitoring pingd sets?
> 
Yes, VM can get the status of host's hardware (status of disk, status
of CPU, etc.) if the following RAs are running on the host.
 - HealthCPU
 - SysInfo
 - diskd (disk monitor RA - published only in Japanese community.
          http://hg.sourceforge.jp/view/linux-ha/pm_diskd/)

Since such RA writes status of hardware into the CIB(*), vm-client can
specify the attribute and can get it's value.
(*) example of the CIB
    # cibadmin -Q
    <cib>
      <status>
        <node_state ...>
          <transient_attributes ...>
            <instance_attributes ...>
              <nvpair id="..." name="default_ping_set" value="100"/>
              <nvpair id="..." name="#health-cpu" value="green"/>
              <nvpair id="..." name="diskd" value="normal"/>
                :

Our final purpose is to get the status of host's hardware who cannot
perform acquisition directly from a guest by this way.

> * It may be that this new project may be a better "backbone" for this
> sort of general request/response than fence-virt; it's a much more
> general communications medium for guest<->host communication:
> 
>      http://git.fedorahosted.org/git/?p=vios-proxy.git
> 
> (although it is written in C++ ;) ).
> 
OK, thank you. We will also see it.

> 
> Notes about the patch itself:
> 
>   * no support for multicast listener (this looks intentional -
>     is it?)
> 
Yes. This patch is only support for serial listener.

Best Regards,
Kazunori INOUE

>   * this will break on-wire compatibility; we need to be very careful
>     here.
> 
>   * some duplicate build system changes with the other
>     patch (not a big deal)
> 
>   * [not directly related to your patch] the vmchannel/serial support
>     in fence-virt probably should be replaced with more recent
>     technology in libvirt
> 
> -- Lon