[Qemu-devel] Using the one disk image file on 2 virtual machines at the same time

[Qemu-devel] Using the one disk image file on 2 virtual machines at the same time

[Manjong Han]

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]

Hi,

I was facing a weird behavior when I used the one disk image file on 2
virtual machines at the same time.

I made the instance of a virtual machine, using the below command.
$ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm

When the OS(Ubuntu 14.04 64bit) was booted up, I made an another one, using
same command.
$ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm

Then, I had 2 virtual machines, using same disk image file.

When I made a directory on first virtual machine, using the command like
below.
$ mkdir test1

The "test1" directory wasn't showed another virtual machine. And, I made a
directory on second virtual machine, using the command like below.
$ mkdir test2

The "test2" directory wasn't showed another virtual machine, too.

Then, I shut the all virtual machines down. I tried to mount the disk image
file on my host machine.
$ sudo modprobe nbd
$ sudo qemu-nbd -c /dev/nbd0
$ sudo mount /dev/nbd0p1 /mnt/10G

Then, I checked what the file is showed. The result was that "test2"
directory only lived.

Where is the "test1" directory? This is the weird behavior on my test.

Before my test, I created a disk image file and installed the OS on the
disk image, using the commands like below.
$ qemu-img create -f qcow2 -o preallocation=metadata 10G.qcow2 10G
$ qemu-system-x86_64 -smp 2 -m 2048 -hda 10G.qcow2 -cdrom
../iso/ubuntu-1404.2-desktop-x86_64.iso -enable-kvm

[-- Attachment #2: Type: text/html, Size: 1851 bytes --]

[Qemu-devel] Using the one disk image file on 2 virtual machines at the same time

[Manjong Han]

[-- Attachment #1: Type: text/plain, Size: 1452 bytes --]

Hi,

I was facing a weird behavior when I used the one disk image file on 2
virtual machines at the same time.

I made the instance of a virtual machine, using the below command.
$ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm

When the OS(Ubuntu 14.04 64bit) was booted up, I made an another one, using
the same command.
$ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm

Then, I had 2 virtual machines using the same disk image file.

When I made a directory on first virtual machine, using the command like
below.
$ mkdir test1

The "test1" directory wasn't showed on another virtual machine. And, I made
a directory on second virtual machine, using the command like below.
$ mkdir test2

The "test2" directory wasn't showed on another virtual machine, either.

Then, I shut the all virtual machines down. I tried to mount the disk image
file on my host machine.
$ sudo modprobe nbd
$ sudo qemu-nbd -c /dev/nbd0 ./10G.qcow2
$ sudo mount /dev/nbd0p1 /mnt/10G

Then, I checked what the file is showed. The result was that "test2"
directory was only showed.

Where is the "test1" directory? This is the weird behavior on my test.

Before my test, I created a disk image file and installed the OS on the
disk image, using the commands like below.

$ qemu-img create -f qcow2 -o preallocation=metadata 10G.qcow2 10G
$ qemu-system-x86_64 -smp 2 -m 2048 -hda 10G.qcow2 -cdrom
../iso/ubuntu-1404.2-desktop-x86_64.iso -enable-kvm

[-- Attachment #2: Type: text/html, Size: 2671 bytes --]

[Qemu-devel] Using the one disk image file on 2 virtual machines at the same time

[한만종]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

Hi,

I'm facing a weird behavior when I used the one disk image file on 2 virtual machines at the same time.

I made the instance of a virtual machine, using the below command.
$ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm

When the OS(Ubuntu 14.04 64bit) was booted up, I made an another one, using the same command.
$ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm

Then, I had 2 virtual machines using the same disk image file.

When I made a directory on first virtual machine, using the command like below.
$ mkdir test1

The "test1" directory wasn't showed on another virtual machine. And, I made a directory on second virtual machine, using the command like below.
$ mkdir test2

The "test2" directory wasn't showed on another virtual machine, either.

Then, I shut the all virtual machines down. I tried to mount the disk image file on my host machine.
$ sudo modprobe nbd
$ sudo qemu-nbd -c /dev/nbd0 ./10G.qcow2
$ sudo mount /dev/nbd0p1 /mnt/10G

Then, I checked what the file is showed. The result was that "test2" directory was only showed.

Where is the "test1" directory? This is the weird behavior on my test.

Before my test, I created a disk image file and installed the OS on the disk image, using the commands like below.

$ qemu-img create -f qcow2 -o preallocation=metadata 10G.qcow2 10G
$ qemu-system-x86_64 -smp 2 -m 2048 -hda 10G.qcow2 -cdrom ../iso/ubuntu-1404.2-desktop-x86_64.iso -enable-kvm

[-- Attachment #2: Type: text/html, Size: 6621 bytes --]

Re: [Qemu-devel] Using the one disk image file on 2 virtual machines at the same time

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1610 bytes --]

On 07/28/2015 11:06 AM, 한만종 wrote:
> Hi,
> 
> I'm facing a weird behavior when I used the one disk image file on 2 virtual machines at the same time.
> 
> I made the instance of a virtual machine, using the below command.
> $ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm
> 
> When the OS(Ubuntu 14.04 64bit) was booted up, I made an another one, using the same command.
> $ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm
> 
> Then, I had 2 virtual machines using the same disk image file.

Bad idea.  You should NEVER have more than one qemu or qemu-img opening
an image read-write at the same time (even having multiple read-only
qemu-img visitors visiting a file opened read-write by qemu is
dangerous).  You are very likely to cause fatal corruption to the point
that neither guest will be able to see data.

> 
> When I made a directory on first virtual machine, using the command like below.
> $ mkdir test1
> 
> The "test1" directory wasn't showed on another virtual machine. And, I made a directory on second virtual machine, using the command like below.
> $ mkdir test2
> 
> The "test2" directory wasn't showed on another virtual machine, either.

And you should NOT expect it to work.  qcow2 images are NOT shared file
systems.  If you want a shared file system, then do something like
having both your guests mount common storage via NFS or glusterfs or
some other protocol designed to be used as a shared file systems.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] Using the one disk image file on 2 virtual machines at the same time

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1241 bytes --]

On Wed, Jul 29, 2015 at 12:57:30AM +0900, Manjong Han wrote:
> I was facing a weird behavior when I used the one disk image file on 2
> virtual machines at the same time.
> 
> I made the instance of a virtual machine, using the below command.
> $ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm
> 
> When the OS(Ubuntu 14.04 64bit) was booted up, I made an another one, using
> same command.
> $ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm
> 
> Then, I had 2 virtual machines, using same disk image file.

This configuration is invalid.  It's similar to using the same physical
disk or iSCSI LUN from two machines at the same time.

Standard file systems (ext4, xfs) and volume managers (LVM) are not
cluster-aware by default.  They must only be accessed from one machine
at a time.  Otherwise you risk data corruption.

You should probably use qcow2 backing files instead:

  10G.qcow2 <-- vm001.qcow2
            ^-- vm002.qcow2

The command to create these files is:

  qemu-img create -f qcow2 -o backing_file=10G.qcow2 vm001.qcow2.

Both VMs share the data in 10G.qcow2.  All writes go to vm001.qcow2 or
vm002.qcow2, respectively, so they don't corrupt each other.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] Using the one disk image file on 2 virtual machines at the same time

[Kashyap Chamarthy]

On Wed, Jul 29, 2015 at 09:46:53AM +0100, Stefan Hajnoczi wrote:
> On Wed, Jul 29, 2015 at 12:57:30AM +0900, Manjong Han wrote:
> > I was facing a weird behavior when I used the one disk image file on 2
> > virtual machines at the same time.
> > 
> > I made the instance of a virtual machine, using the below command.
> > $ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm
> > 
> > When the OS(Ubuntu 14.04 64bit) was booted up, I made an another one, using
> > same command.
> > $ qemu-system-x86_64 -smp 2 -m 1024 -hda 10G.qcow2 -enable-kvm
> > 
> > Then, I had 2 virtual machines, using same disk image file.
> 
> This configuration is invalid.  It's similar to using the same physical
> disk or iSCSI LUN from two machines at the same time.
> 
> Standard file systems (ext4, xfs) and volume managers (LVM) are not
> cluster-aware by default.  They must only be accessed from one machine
> at a time.  Otherwise you risk data corruption.
> 
> You should probably use qcow2 backing files instead:
> 
>   10G.qcow2 <-- vm001.qcow2
>             ^-- vm002.qcow2
> 
> The command to create these files is:
> 
>   qemu-img create -f qcow2 -o backing_file=10G.qcow2 vm001.qcow2.
> 
> Both VMs share the data in 10G.qcow2.  All writes go to vm001.qcow2 or
> vm002.qcow2, respectively, so they don't corrupt each other.

As an addendum, when using a management library like libvirt, it
provides a convenient daemon called 'virtlockd'[1] (which uses the POSIX
fcntl(2) mechanism),  which takes care of invalid configs like above.

>From my notes (from a FOSDEM 2014 talk by Dan Berange), virtlockd
operates thus:

    - The QEMU driver inside libvirt daemon, just talks to the virtlockd
      daemon using an RPC mechanism. So, whenever you first start a
      guest, the first thing it does is it talks to the virtlockd daemon
      and acquire locks for all of these disk images -- only if this
      succeeds, will the QEMU process will be started

    - These locks are also released and reaquired whenever you paused
      the virtual machines -- which is the key to make migrations work.


[1] https://libvirt.org/locking-lockd.html

-- 
/kashyap

Re: [Qemu-devel] Using the one disk image file on 2 virtual machines at the same time

[Manjong Han]

Thanks, Stefan.

2015-07-29 17:46 GMT+09:00 Stefan Hajnoczi <stefanha@gmail.com>:
>
> You should probably use qcow2 backing files instead:
>
>   10G.qcow2 <-- vm001.qcow2
>             ^-- vm002.qcow2
>
> The command to create these files is:
>
>   qemu-img create -f qcow2 -o backing_file=10G.qcow2 vm001.qcow2.
>
> Both VMs share the data in 10G.qcow2.  All writes go to vm001.qcow2 or
> vm002.qcow2, respectively, so they don't corrupt each other.
>

I tried to create a backing files, using the commands which you told.

$ qemu-img create -f qcow2 -o backing_file=10G.qcow2 vm001.qcow2
$ qemu-img create -f qcow2 -o backing_file=10G.qcow2 vm002.qcow2

And, I used these backing files on each virtual machines.
But, new files weren't written on original disk image(10G.qcow2)..
The backing files were working each other.

> Standard file systems (ext4, xfs) and volume managers (LVM) are not
> cluster-aware by default.  They must only be accessed from one machine
> at a time.  Otherwise you risk data corruption.
>

I think that I must probably use a shared file system like NFS..

Re: [Qemu-devel] Using the one disk image file on 2 virtual machines at the same time

[John Snow]

On 07/29/2015 01:29 PM, Manjong Han wrote:
> Thanks, Stefan.
> 
> 2015-07-29 17:46 GMT+09:00 Stefan Hajnoczi <stefanha@gmail.com>:
>>
>> You should probably use qcow2 backing files instead:
>>
>>   10G.qcow2 <-- vm001.qcow2
>>             ^-- vm002.qcow2
>>
>> The command to create these files is:
>>
>>   qemu-img create -f qcow2 -o backing_file=10G.qcow2 vm001.qcow2.
>>
>> Both VMs share the data in 10G.qcow2.  All writes go to vm001.qcow2 or
>> vm002.qcow2, respectively, so they don't corrupt each other.
>>
> 
> I tried to create a backing files, using the commands which you told.
> 
> $ qemu-img create -f qcow2 -o backing_file=10G.qcow2 vm001.qcow2
> $ qemu-img create -f qcow2 -o backing_file=10G.qcow2 vm002.qcow2
> 
> And, I used these backing files on each virtual machines.
> But, new files weren't written on original disk image(10G.qcow2)..
> The backing files were working each other.
> 
>> Standard file systems (ext4, xfs) and volume managers (LVM) are not
>> cluster-aware by default.  They must only be accessed from one machine
>> at a time.  Otherwise you risk data corruption.
>>
> 
> I think that I must probably use a shared file system like NFS..
> 

Yes, any files written using the backing files like outlined above will
put new files in the overlays (e.g. vm001.qcow2 or vm002.qcow2) and NOT
into the backing file (10G.qcow2)

this is a safe way to share a base image for an OS, but it's not a
method of accomplishing a concurrent fileshare.

You'll want to configure an NFS or CIFS share (etc) in the base image
and then allow the multiple VMs to utilize that share.

--js

Re: [Qemu-devel] Using the one disk image file on 2 virtual machines at the same time

[Christopher Covington]

On 07/29/2015 01:46 PM, John Snow wrote:
> 
> 
> On 07/29/2015 01:29 PM, Manjong Han wrote:
>> Thanks, Stefan.
>>
>> 2015-07-29 17:46 GMT+09:00 Stefan Hajnoczi <stefanha@gmail.com>:
>>>
>>> You should probably use qcow2 backing files instead:
>>>
>>>   10G.qcow2 <-- vm001.qcow2
>>>             ^-- vm002.qcow2
>>>
>>> The command to create these files is:
>>>
>>>   qemu-img create -f qcow2 -o backing_file=10G.qcow2 vm001.qcow2.
>>>
>>> Both VMs share the data in 10G.qcow2.  All writes go to vm001.qcow2 or
>>> vm002.qcow2, respectively, so they don't corrupt each other.
>>>
>>
>> I tried to create a backing files, using the commands which you told.
>>
>> $ qemu-img create -f qcow2 -o backing_file=10G.qcow2 vm001.qcow2
>> $ qemu-img create -f qcow2 -o backing_file=10G.qcow2 vm002.qcow2
>>
>> And, I used these backing files on each virtual machines.
>> But, new files weren't written on original disk image(10G.qcow2)..
>> The backing files were working each other.
>>
>>> Standard file systems (ext4, xfs) and volume managers (LVM) are not
>>> cluster-aware by default.  They must only be accessed from one machine
>>> at a time.  Otherwise you risk data corruption.
>>>
>>
>> I think that I must probably use a shared file system like NFS..
>>
> 
> Yes, any files written using the backing files like outlined above will
> put new files in the overlays (e.g. vm001.qcow2 or vm002.qcow2) and NOT
> into the backing file (10G.qcow2)
> 
> this is a safe way to share a base image for an OS, but it's not a
> method of accomplishing a concurrent fileshare.
> 
> You'll want to configure an NFS or CIFS share (etc) in the base image
> and then allow the multiple VMs to utilize that share.

Using a VirtIO-9P based passthrough filesystem on both might produce the same
result with less host-side configuration required.

http://wiki.qemu.org/Documentation/9psetup

Chris

-- 
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project