[Qemu-devel] qemu-img problem when create a file larger than fs's size

[Qemu-devel] qemu-img problem when create a file larger than fs's size

[yuxh]

Hello all,

I have to consult you a qemu-img's problem.

Is this reasonable to create a file which is larger than the available 
size of the fs by qemu-img cmd ?

When I use qemu-img create a file which is larger than the available 
size of the fs, the creation is completed succesfully.

However when I use this file in guest as a guest's disk, and write 
beyond the size the host file can provides, the guest was paused by 
qemu-kvm or libvirt and was in maybe a infinite circle where the guest 
just can't be used except I detach the disk from guest or destroy the guest.

I read the qemu-img's code and found it just create the file as we 
required and doesn't check if the size we specify is reasonable.But this 
may let the guest in a risk of meeting the problem I describe above.

Exp:
[root@build mytest]# df -ah /mytest/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb2             2.0G  3.1M  1.9G   1% /mytest
[root@build mytest]# qemu-img create -f raw test.raw 3G
Formatting 'test.raw', fmt=raw size=3221225472
[root@build mytest]# ls -l test.raw
-rw-r--r--. 1 root root 3221225472 May  8 12:23 test.raw
[root@build mytest]#

Thank you.

Best Regards
Xinghai Yu

Re: [Qemu-devel] qemu-img problem when create a file larger than fs's size

[Fam Zheng]

I think it's the system admin to be responsible for the risk of over
provisioning. We have host sparse file[1] (as your example) for
preallocated image (for example, -f raw), as well as sparse image (as
supported in qcow2, vmdk, etc.). There are cases that host file system
is extended or the vm disk is moved to a larger storage when the actual
data grows closer to full, so it's not very practical to limit the
creating size, just as this is quietly valid in linux, no matter how
small your /tmp is.

    # df /tmp/ -h
    Filesystem      Size  Used Avail Use% Mounted on
    tmpfs           3.8G   14M  3.8G   1% /tmp
    # touch /tmp/a
    # truncate /tmp/a --size 10000T
    # ls -l /tmp/a
    -rw-rw-r--. 1 fam fam 10995116277760000 May  8 14:33 /tmp/a

[1]: http://en.wikipedia.org/wiki/Sparse_file

On Wed, 05/08 13:18, yuxh wrote:
> Hello all,
> 
> I have to consult you a qemu-img's problem.
> 
> Is this reasonable to create a file which is larger than the
> available size of the fs by qemu-img cmd ?
> 
> When I use qemu-img create a file which is larger than the available
> size of the fs, the creation is completed succesfully.
> 
> However when I use this file in guest as a guest's disk, and write
> beyond the size the host file can provides, the guest was paused by
> qemu-kvm or libvirt and was in maybe a infinite circle where the
> guest just can't be used except I detach the disk from guest or
> destroy the guest.
> 
> I read the qemu-img's code and found it just create the file as we
> required and doesn't check if the size we specify is reasonable.But
> this may let the guest in a risk of meeting the problem I describe
> above.
> 
> Exp:
> [root@build mytest]# df -ah /mytest/
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sdb2             2.0G  3.1M  1.9G   1% /mytest
> [root@build mytest]# qemu-img create -f raw test.raw 3G
> Formatting 'test.raw', fmt=raw size=3221225472
> [root@build mytest]# ls -l test.raw
> -rw-r--r--. 1 root root 3221225472 May  8 12:23 test.raw
> [root@build mytest]#
> 
> Thank you.
> 
> Best Regards
> Xinghai Yu
> 

-- 
Fam

Re: [Qemu-devel] qemu-img problem when create a file larger than fs's size

[yuxh]

Thank you for your reply. I agreed what you said.
And do you think we shall print a prompt to user when the size user 
specified is larger than available space ?

On 05/08/2013 02:36 PM, Fam Zheng wrote:
> I think it's the system admin to be responsible for the risk of over
> provisioning. We have host sparse file[1] (as your example) for
> preallocated image (for example, -f raw), as well as sparse image (as
> supported in qcow2, vmdk, etc.). There are cases that host file system
> is extended or the vm disk is moved to a larger storage when the actual
> data grows closer to full, so it's not very practical to limit the
> creating size, just as this is quietly valid in linux, no matter how
> small your /tmp is.
>
>      # df /tmp/ -h
>      Filesystem      Size  Used Avail Use% Mounted on
>      tmpfs           3.8G   14M  3.8G   1% /tmp
>      # touch /tmp/a
>      # truncate /tmp/a --size 10000T
>      # ls -l /tmp/a
>      -rw-rw-r--. 1 fam fam 10995116277760000 May  8 14:33 /tmp/a
>
> [1]: http://en.wikipedia.org/wiki/Sparse_file
>
> On Wed, 05/08 13:18, yuxh wrote:
>> Hello all,
>>
>> I have to consult you a qemu-img's problem.
>>
>> Is this reasonable to create a file which is larger than the
>> available size of the fs by qemu-img cmd ?
>>
>> When I use qemu-img create a file which is larger than the available
>> size of the fs, the creation is completed succesfully.
>>
>> However when I use this file in guest as a guest's disk, and write
>> beyond the size the host file can provides, the guest was paused by
>> qemu-kvm or libvirt and was in maybe a infinite circle where the
>> guest just can't be used except I detach the disk from guest or
>> destroy the guest.
>>
>> I read the qemu-img's code and found it just create the file as we
>> required and doesn't check if the size we specify is reasonable.But
>> this may let the guest in a risk of meeting the problem I describe
>> above.
>>
>> Exp:
>> [root@build mytest]# df -ah /mytest/
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/sdb2             2.0G  3.1M  1.9G   1% /mytest
>> [root@build mytest]# qemu-img create -f raw test.raw 3G
>> Formatting 'test.raw', fmt=raw size=3221225472
>> [root@build mytest]# ls -l test.raw
>> -rw-r--r--. 1 root root 3221225472 May  8 12:23 test.raw
>> [root@build mytest]#
>>
>> Thank you.
>>
>> Best Regards
>> Xinghai Yu
>>
>

-- 
以上

第一软件事业部 第一开发部 driver组 于星海
Best Regards
--------------------------------------------------
Yu Xinghai
Development Dept.I
Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST)
No.6 Wenzhu Road, Nanjing, 210012, China
TEL: +86+25-86630566-8533
FUJITSU INTERNAL: 7998-8533
FAX: +86+25-83317685
MAIL: yuxinghai@cn.fujitsu.com
--------------------------------------------------

Re: [Qemu-devel] qemu-img problem when create a file larger than fs's size

[Fam Zheng]

On Wed, 05/08 15:34, yuxh wrote:
> 
> Thank you for your reply. I agreed what you said.
> And do you think we shall print a prompt to user when the size user
> specified is larger than available space ?

Might be nice, but it's hard to define "available space", and "available
space" at creating time promises nothing. Even if the user creates a
small image which seems safe, risk is still there: when other
applications' data fills up the "available space", the use still
experiences the same problem.

> 
> On 05/08/2013 02:36 PM, Fam Zheng wrote:
> >I think it's the system admin to be responsible for the risk of over
> >provisioning. We have host sparse file[1] (as your example) for
> >preallocated image (for example, -f raw), as well as sparse image (as
> >supported in qcow2, vmdk, etc.). There are cases that host file system
> >is extended or the vm disk is moved to a larger storage when the actual
> >data grows closer to full, so it's not very practical to limit the
> >creating size, just as this is quietly valid in linux, no matter how
> >small your /tmp is.
> >
> >     # df /tmp/ -h
> >     Filesystem      Size  Used Avail Use% Mounted on
> >     tmpfs           3.8G   14M  3.8G   1% /tmp
> >     # touch /tmp/a
> >     # truncate /tmp/a --size 10000T
> >     # ls -l /tmp/a
> >     -rw-rw-r--. 1 fam fam 10995116277760000 May  8 14:33 /tmp/a
> >
> >[1]: http://en.wikipedia.org/wiki/Sparse_file
> >
> >On Wed, 05/08 13:18, yuxh wrote:
> >>Hello all,
> >>
> >>I have to consult you a qemu-img's problem.
> >>
> >>Is this reasonable to create a file which is larger than the
> >>available size of the fs by qemu-img cmd ?
> >>
> >>When I use qemu-img create a file which is larger than the available
> >>size of the fs, the creation is completed succesfully.
> >>
> >>However when I use this file in guest as a guest's disk, and write
> >>beyond the size the host file can provides, the guest was paused by
> >>qemu-kvm or libvirt and was in maybe a infinite circle where the
> >>guest just can't be used except I detach the disk from guest or
> >>destroy the guest.
> >>
> >>I read the qemu-img's code and found it just create the file as we
> >>required and doesn't check if the size we specify is reasonable.But
> >>this may let the guest in a risk of meeting the problem I describe
> >>above.
> >>
> >>Exp:
> >>[root@build mytest]# df -ah /mytest/
> >>Filesystem            Size  Used Avail Use% Mounted on
> >>/dev/sdb2             2.0G  3.1M  1.9G   1% /mytest
> >>[root@build mytest]# qemu-img create -f raw test.raw 3G
> >>Formatting 'test.raw', fmt=raw size=3221225472
> >>[root@build mytest]# ls -l test.raw
> >>-rw-r--r--. 1 root root 3221225472 May  8 12:23 test.raw
> >>[root@build mytest]#
> >>
> >>Thank you.
> >>
> >>Best Regards
> >>Xinghai Yu
> >>
> >
> 
> -- 
> 以上
> 
> 第一软件事业部 第一开发部 driver组 于星海
> Best Regards
> --------------------------------------------------
> Yu Xinghai
> Development Dept.I
> Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST)
> No.6 Wenzhu Road, Nanjing, 210012, China
> TEL: +86+25-86630566-8533
> FUJITSU INTERNAL: 7998-8533
> FAX: +86+25-83317685
> MAIL: yuxinghai@cn.fujitsu.com
> --------------------------------------------------

-- 
Fam

Re: [Qemu-devel] qemu-img problem when create a file larger than fs's size

[yuxh]

On 05/08/2013 04:20 PM, Fam Zheng wrote:
> On Wed, 05/08 15:34, yuxh wrote:
>>
>> Thank you for your reply. I agreed what you said.
>> And do you think we shall print a prompt to user when the size user
>> specified is larger than available space ?
>
> Might be nice, but it's hard to define "available space", and "available
> space" at creating time promises nothing. Even if the user creates a
> small image which seems safe, risk is still there: when other
> applications' data fills up the "available space", the use still
> experiences the same problem.
>

Yes. That means the prompt means nothing. Thank you very much!

yuxh

>>
>> On 05/08/2013 02:36 PM, Fam Zheng wrote:
>>> I think it's the system admin to be responsible for the risk of over
>>> provisioning. We have host sparse file[1] (as your example) for
>>> preallocated image (for example, -f raw), as well as sparse image (as
>>> supported in qcow2, vmdk, etc.). There are cases that host file system
>>> is extended or the vm disk is moved to a larger storage when the actual
>>> data grows closer to full, so it's not very practical to limit the
>>> creating size, just as this is quietly valid in linux, no matter how
>>> small your /tmp is.
>>>
>>>      # df /tmp/ -h
>>>      Filesystem      Size  Used Avail Use% Mounted on
>>>      tmpfs           3.8G   14M  3.8G   1% /tmp
>>>      # touch /tmp/a
>>>      # truncate /tmp/a --size 10000T
>>>      # ls -l /tmp/a
>>>      -rw-rw-r--. 1 fam fam 10995116277760000 May  8 14:33 /tmp/a
>>>
>>> [1]: http://en.wikipedia.org/wiki/Sparse_file
>>>
>>> On Wed, 05/08 13:18, yuxh wrote:
>>>> Hello all,
>>>>
>>>> I have to consult you a qemu-img's problem.
>>>>
>>>> Is this reasonable to create a file which is larger than the
>>>> available size of the fs by qemu-img cmd ?
>>>>
>>>> When I use qemu-img create a file which is larger than the available
>>>> size of the fs, the creation is completed succesfully.
>>>>
>>>> However when I use this file in guest as a guest's disk, and write
>>>> beyond the size the host file can provides, the guest was paused by
>>>> qemu-kvm or libvirt and was in maybe a infinite circle where the
>>>> guest just can't be used except I detach the disk from guest or
>>>> destroy the guest.
>>>>
>>>> I read the qemu-img's code and found it just create the file as we
>>>> required and doesn't check if the size we specify is reasonable.But
>>>> this may let the guest in a risk of meeting the problem I describe
>>>> above.
>>>>
>>>> Exp:
>>>> [root@build mytest]# df -ah /mytest/
>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>> /dev/sdb2             2.0G  3.1M  1.9G   1% /mytest
>>>> [root@build mytest]# qemu-img create -f raw test.raw 3G
>>>> Formatting 'test.raw', fmt=raw size=3221225472
>>>> [root@build mytest]# ls -l test.raw
>>>> -rw-r--r--. 1 root root 3221225472 May  8 12:23 test.raw
>>>> [root@build mytest]#
>>>>
>>>> Thank you.
>>>>
>>>> Best Regards
>>>> Xinghai Yu
>>>>
>>>
>>
>> --
>> 以上
>>
>> 第一软件事业部 第一开发部 driver组 于星海
>> Best Regards
>> --------------------------------------------------
>> Yu Xinghai
>> Development Dept.I
>> Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST)
>> No.6 Wenzhu Road, Nanjing, 210012, China
>> TEL: +86+25-86630566-8533
>> FUJITSU INTERNAL: 7998-8533
>> FAX: +86+25-83317685
>> MAIL: yuxinghai@cn.fujitsu.com
>> --------------------------------------------------
>

-- 
以上

第一软件事业部 第一开发部 driver组 于星海
Best Regards
--------------------------------------------------
Yu Xinghai
Development Dept.I
Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST)
No.6 Wenzhu Road, Nanjing, 210012, China
TEL: +86+25-86630566-8533
FUJITSU INTERNAL: 7998-8533
FAX: +86+25-83317685
MAIL: yuxinghai@cn.fujitsu.com
--------------------------------------------------

Re: [Qemu-devel] qemu-img problem when create a file larger than fs's size

[Stefan Hajnoczi]

On Wed, May 08, 2013 at 01:18:17PM +0800, yuxh wrote:
> I have to consult you a qemu-img's problem.
> 
> Is this reasonable to create a file which is larger than the
> available size of the fs by qemu-img cmd ?
> 
> When I use qemu-img create a file which is larger than the available
> size of the fs, the creation is completed succesfully.
> 
> However when I use this file in guest as a guest's disk, and write
> beyond the size the host file can provides, the guest was paused by
> qemu-kvm or libvirt and was in maybe a infinite circle where the
> guest just can't be used except I detach the disk from guest or
> destroy the guest.

You can change the ENOSPC policy with -drive werror=,rerror=.  See the
QEMU man page.

The default behavior is to pause the guest so the host admin can free up
or add space.  Then the guest can be continued - this will retry the
I/O.

But you can also tell QEMU to pass the error through to the guest using
the -drive werror=,rerror= options.

Stefan

Re: [Qemu-devel] qemu-img problem when create a file larger than fs's size

[yuxh]

On 05/09/2013 03:44 PM, Stefan Hajnoczi wrote:
> On Wed, May 08, 2013 at 01:18:17PM +0800, yuxh wrote:
>> I have to consult you a qemu-img's problem.
>>
>> Is this reasonable to create a file which is larger than the
>> available size of the fs by qemu-img cmd ?
>>
>> When I use qemu-img create a file which is larger than the available
>> size of the fs, the creation is completed succesfully.
>>
>> However when I use this file in guest as a guest's disk, and write
>> beyond the size the host file can provides, the guest was paused by
>> qemu-kvm or libvirt and was in maybe a infinite circle where the
>> guest just can't be used except I detach the disk from guest or
>> destroy the guest.
>
> You can change the ENOSPC policy with -drive werror=,rerror=.  See the
> QEMU man page.
>
> The default behavior is to pause the guest so the host admin can free up
> or add space.  Then the guest can be continued - this will retry the
> I/O.
>
> But you can also tell QEMU to pass the error through to the guest using
> the -drive werror=,rerror= options.
>
> Stefan
>
>

Oh, understand now.
Thank you so much for your explanation.

Yu