Re: [Qemu-devel] [PATCH RFC v2 1/6] docs/block-replication: Add description for shared-disk case

[Hailiang Zhang <zhang.zhanghailiang@huawei.com>]

On 2017/1/20 0:41, Stefan Hajnoczi wrote:
> On Thu, Jan 19, 2017 at 10:50:19AM +0800, Hailiang Zhang wrote:
>> On 2017/1/13 21:41, Stefan Hajnoczi wrote:
>>> On Mon, Dec 05, 2016 at 04:34:59PM +0800, zhanghailiang wrote:
>>>> +Issue qmp command:
>>>> +  { 'execute': 'blockdev-add',
>>>> +    'arguments': {
>>>> +        'driver': 'replication',
>>>> +        'node-name': 'rep',
>>>> +        'mode': 'primary',
>>>> +        'shared-disk-id': 'primary_disk0',
>>>> +        'shared-disk': true,
>>>> +        'file': {
>>>> +            'driver': 'nbd',
>>>> +            'export': 'hidden_disk0',
>>>> +            'server': {
>>>> +                'type': 'inet',
>>>> +                'data': {
>>>> +                    'host': 'xxx.xxx.xxx.xxx',
>>>> +                    'port': 'yyy'
>>>> +                }
>>>> +            }
>>>
>>> block/nbd.c does have good error handling and recovery in case there is
>>> a network issue.  There are no reconnection attempts or timeouts that
>>> deal with a temporary loss of network connectivity.
>>>
>>> This is a general problem with block/nbd.c and not something to solve in
>>> this patch series.  I'm just mentioning it because it may affect COLO
>>> replication.
>>>
>>> I'm sure these limitations in block/nbd.c can be fixed but it will take
>>> some effort.  Maybe block/sheepdog.c, net/socket.c, and other network
>>> code could also benefit from generic network connection recovery.
>>>
>>
>> Hmm, good suggestion, but IMHO, here, COLO is a little different from
>> other scenes, if the reconnection method has been implemented,
>> it still needs a mechanism to identify the temporary loss of network
>> connection or real broken in network connection.
>>
>> I did a simple test, just ifconfig down the network card that be used
>> by block replication, It seems that NBD in qemu doesn't has a ability to
>> find the connection has been broken, there was no error reports
>> and COLO just got stuck in vm_stop() where it called aio_poll().
>
> Yes, this is the vm_stop() problem again.  There is no reliable way to
> cancel I/O requests so instead QEMU waits...forever.  A solution is
> needed so COLO doesn't hang on network failure.
>

Yes, COLO needs to detect this situation and cancel the requests in a proper
way.

> I'm not sure how to solve the problem.  The secondary still has the last
> successful checkpoint so it could resume instead of waiting for the
> current checkpoint to commit.
>
> There may still be NBD I/O in flight, so the would need to drain it or
> fence storage to prevent interference once the secondary VM is running.
>

Agreed, we need to think this carefully. We'll put these reliabilities
developing in future after COLO's basic function completed.

Thanks,
Hailiang

> Stefan
>