Question: data consistency on fail-over using shared disk

Question: data consistency on fail-over using shared disk

[Takuya Yoshikawa]

Hi,


We are now checking about what we should do on vm fail-over.

Concerning this, does anybody know about any danger about data
consistency when we are using shared disk?


What I'm concerning is if crashed VM-side host is still holding
buffered data, starting a new VM instance on another node may
result in file system corruption.

  This problem may similar to live-migration but little bit different
  in the sense that VM is crashed -> cannot do anything from that point.


How about the combination of old or new guest OS and the following
settings?

 - writethrough
 - writeback
 - none

If needed, we'll do sync by HA-side scripts before starting a new VM
instance.


Thanks,
  Takuya

Re: Question: data consistency on fail-over using shared disk

[Avi Kivity]

On 07/21/2010 09:27 AM, Takuya Yoshikawa wrote:
> Hi,
>
>
> We are now checking about what we should do on vm fail-over.
>
> Concerning this, does anybody know about any danger about data
> consistency when we are using shared disk?
>
>
> What I'm concerning is if crashed VM-side host is still holding
> buffered data, starting a new VM instance on another node may
> result in file system corruption.
>
>   This problem may similar to live-migration but little bit different
>   in the sense that VM is crashed -> cannot do anything from that point.
>   

If the VM can't do anything, you're safe. Of course a crash doesn't mean
a VM can't do anything, you typically need to isolate it by resetting
the host or having a switch disconnect its storage and network.

>
> How about the combination of old or new guest OS and the following
> settings?
>
>  - writethrough
>  - writeback
>  - none
>
> If needed, we'll do sync by HA-side scripts before starting a new VM
> instance.
>   

I believe nothing special is needed, as long as a crash means both the
qemu and host kernel crashed. If only qemu crashed, then the writeback
case needs buffers flushed.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Question: data consistency on fail-over using shared disk

[Takuya Yoshikawa]

(2010/07/21 16:27), Avi Kivity wrote:
> On 07/21/2010 09:27 AM, Takuya Yoshikawa wrote:
>> Hi,
>>
>>
>> We are now checking about what we should do on vm fail-over.
>>
>> Concerning this, does anybody know about any danger about data
>> consistency when we are using shared disk?
>>
>>
>> What I'm concerning is if crashed VM-side host is still holding
>> buffered data, starting a new VM instance on another node may
>> result in file system corruption.
>>
>>    This problem may similar to live-migration but little bit different
>>    in the sense that VM is crashed ->  cannot do anything from that point.
>>
> 
> If the VM can't do anything, you're safe. Of course a crash doesn't mean
> a VM can't do anything, you typically need to isolate it by resetting
> the host or having a switch disconnect its storage and network.
> 

Thanks,
We are starting to manage the "only VM crashed" case by HA.
So yes, we need to reset the host.

>>
>> How about the combination of old or new guest OS and the following
>> settings?
>>
>>   - writethrough
>>   - writeback
>>   - none
>>
>> If needed, we'll do sync by HA-side scripts before starting a new VM
>> instance.
>>
> 
> I believe nothing special is needed, as long as a crash means both the
> qemu and host kernel crashed. If only qemu crashed, then the writeback
> case needs buffers flushed.
> 

My main concern is only qemu crashed and host is alive case.

Host memory is typically large, so we want to eliminate flushing everything
if possible.

So we don't need to flush in the case of writethrough = RHEL default ?

Re: Question: data consistency on fail-over using shared disk

[Avi Kivity]

On 07/21/2010 10:43 AM, Takuya Yoshikawa wrote:
>
>> If the VM can't do anything, you're safe. Of course a crash doesn't mean
>> a VM can't do anything, you typically need to isolate it by resetting
>> the host or having a switch disconnect its storage and network.
>>
>>     
> Thanks,
> We are starting to manage the "only VM crashed" case by HA.
> So yes, we need to reset the host.
>   

It may be sufficient to issue a sync(2).


>> I believe nothing special is needed, as long as a crash means both the
>> qemu and host kernel crashed. If only qemu crashed, then the writeback
>> case needs buffers flushed.
>>
>>     
> My main concern is only qemu crashed and host is alive case.
>
> Host memory is typically large, so we want to eliminate flushing everything
> if possible.
>
> So we don't need to flush in the case of writethrough = RHEL default ?
>   

No. But you do need to unmount the underlying host filesystem.

I recommend using cache=none and direct volume access, this is easier to
guarantee integrity.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Question: data consistency on fail-over using shared disk

[Fernando Luis Vázquez Cao]

On Wed, 2010-07-21 at 13:46 +0300, Avi Kivity wrote:
> On 07/21/2010 10:43 AM, Takuya Yoshikawa wrote:
> >
> >> If the VM can't do anything, you're safe. Of course a crash doesn't mean
> >> a VM can't do anything, you typically need to isolate it by resetting
> >> the host or having a switch disconnect its storage and network.
> >>
> >>     
> > Thanks,
> > We are starting to manage the "only VM crashed" case by HA.
> > So yes, we need to reset the host.
> >   
> 
> It may be sufficient to issue a sync(2).
> 
> 
> >> I believe nothing special is needed, as long as a crash means both the
> >> qemu and host kernel crashed. If only qemu crashed, then the writeback
> >> case needs buffers flushed.
> >>
> >>     
> > My main concern is only qemu crashed and host is alive case.
> >
> > Host memory is typically large, so we want to eliminate flushing everything
> > if possible.
> >
> > So we don't need to flush in the case of writethrough = RHEL default ?
> >   
> 
> No. But you do need to unmount the underlying host filesystem.
> 
> I recommend using cache=none and direct volume access, this is easier to
> guarantee integrity.

By the way, if we used cache=none we would still need to invoke sync(2)
to flush the physical device's write cache.