[Qemu-devel] hd-geo-test creates 4GB files on FSes that don't support sparse images, doesn't delete them on error

[Qemu-devel] hd-geo-test creates 4GB files on FSes that don't support sparse images, doesn't delete them on error

[Peter Maydell]

I've noticed recently that tests/hd-geo-test.c creates test disk
images which are 4GB in size, which is a problem if the filesystem
on the host doesn't support sparse files. In particular, OSX's HFS+
doesn't have sparse file support, and Windows probably doesn't either.
Worse, if the test fails an assertion somewhere the test doesn't
clean up after itself and leaves a 4GB file lying around in /tmp/.

It would be nice if we could skip these tests on filesystems that
don't have sparse file support...

thanks
-- PMM

Re: [Qemu-devel] hd-geo-test creates 4GB files on FSes that don't support sparse images, doesn't delete them on error

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1260 bytes --]

On 08/27/2015 09:17 AM, Peter Maydell wrote:
> I've noticed recently that tests/hd-geo-test.c creates test disk
> images which are 4GB in size, which is a problem if the filesystem
> on the host doesn't support sparse files. In particular, OSX's HFS+
> doesn't have sparse file support, and Windows probably doesn't either.

Windows NTFS supports sparse files (minimum hole size of 64k), but it
can be a pain to set up, and while it saves disk space, it may actually
slow your program down.

[At one point cygwin created sparse files on windows by default, but
because it was demonstrated to hurt performance in dealing with sparse
files, because Windows doesn't handle sparse files efficiently, the
cygwin defaults were switched so that it now requires an explicit opt-in
mount option before even attempting sparse files]

> Worse, if the test fails an assertion somewhere the test doesn't
> clean up after itself and leaves a 4GB file lying around in /tmp/.
> 
> It would be nice if we could skip these tests on filesystems that
> don't have sparse file support...

Or even where sparse files are supported but not default.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] hd-geo-test creates 4GB files on FSes that don't support sparse images, doesn't delete them on error

[John Snow]

On 08/27/2015 11:29 AM, Eric Blake wrote:
> On 08/27/2015 09:17 AM, Peter Maydell wrote:
>> I've noticed recently that tests/hd-geo-test.c creates test disk
>> images which are 4GB in size, which is a problem if the filesystem
>> on the host doesn't support sparse files. In particular, OSX's HFS+
>> doesn't have sparse file support, and Windows probably doesn't either.
> 
> Windows NTFS supports sparse files (minimum hole size of 64k), but it
> can be a pain to set up, and while it saves disk space, it may actually
> slow your program down.
> 
> [At one point cygwin created sparse files on windows by default, but
> because it was demonstrated to hurt performance in dealing with sparse
> files, because Windows doesn't handle sparse files efficiently, the
> cygwin defaults were switched so that it now requires an explicit opt-in
> mount option before even attempting sparse files]
> 
>> Worse, if the test fails an assertion somewhere the test doesn't
>> clean up after itself and leaves a 4GB file lying around in /tmp/.
>>
>> It would be nice if we could skip these tests on filesystems that
>> don't have sparse file support...
> 
> Or even where sparse files are supported but not default.
> 

Does this test *require* the raw format?

Use tests/libqos/libqos.c mkqcow2 instead. I'll send a patch.

--js

Re: [Qemu-devel] hd-geo-test creates 4GB files on FSes that don't support sparse images, doesn't delete them on error

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 08/27/2015 11:29 AM, Eric Blake wrote:
>> On 08/27/2015 09:17 AM, Peter Maydell wrote:
>>> I've noticed recently that tests/hd-geo-test.c creates test disk
>>> images which are 4GB in size, which is a problem if the filesystem
>>> on the host doesn't support sparse files. In particular, OSX's HFS+
>>> doesn't have sparse file support, and Windows probably doesn't either.
>> 
>> Windows NTFS supports sparse files (minimum hole size of 64k), but it
>> can be a pain to set up, and while it saves disk space, it may actually
>> slow your program down.
>> 
>> [At one point cygwin created sparse files on windows by default, but
>> because it was demonstrated to hurt performance in dealing with sparse
>> files, because Windows doesn't handle sparse files efficiently, the
>> cygwin defaults were switched so that it now requires an explicit opt-in
>> mount option before even attempting sparse files]
>> 
>>> Worse, if the test fails an assertion somewhere the test doesn't
>>> clean up after itself and leaves a 4GB file lying around in /tmp/.
>>>
>>> It would be nice if we could skip these tests on filesystems that
>>> don't have sparse file support...
>> 
>> Or even where sparse files are supported but not default.
>> 
>
> Does this test *require* the raw format?

If memory serves, the test doesn't require a specific format, only the
size and the contents of the MBR matters.

> Use tests/libqos/libqos.c mkqcow2 instead. I'll send a patch.

Go right ahead.

Re: [Qemu-devel] hd-geo-test creates 4GB files on FSes that don't support sparse images, doesn't delete them on error

[John Snow]

On 08/28/2015 09:06 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 08/27/2015 11:29 AM, Eric Blake wrote:
>>> On 08/27/2015 09:17 AM, Peter Maydell wrote:
>>>> I've noticed recently that tests/hd-geo-test.c creates test disk
>>>> images which are 4GB in size, which is a problem if the filesystem
>>>> on the host doesn't support sparse files. In particular, OSX's HFS+
>>>> doesn't have sparse file support, and Windows probably doesn't either.
>>>
>>> Windows NTFS supports sparse files (minimum hole size of 64k), but it
>>> can be a pain to set up, and while it saves disk space, it may actually
>>> slow your program down.
>>>
>>> [At one point cygwin created sparse files on windows by default, but
>>> because it was demonstrated to hurt performance in dealing with sparse
>>> files, because Windows doesn't handle sparse files efficiently, the
>>> cygwin defaults were switched so that it now requires an explicit opt-in
>>> mount option before even attempting sparse files]
>>>
>>>> Worse, if the test fails an assertion somewhere the test doesn't
>>>> clean up after itself and leaves a 4GB file lying around in /tmp/.
>>>>
>>>> It would be nice if we could skip these tests on filesystems that
>>>> don't have sparse file support...
>>>
>>> Or even where sparse files are supported but not default.
>>>
>>
>> Does this test *require* the raw format?
> 
> If memory serves, the test doesn't require a specific format, only the
> size and the contents of the MBR matters.
> 
>> Use tests/libqos/libqos.c mkqcow2 instead. I'll send a patch.
> 
> Go right ahead.
> 

Oh, taking a look at it, it needs to writethat MBR data to the file
before it opens it. We don't have an existing qemu-io dependency here to
use.

I could add it, but the line between iotest and qtest starts to get
pretty fuzzy.

Thoughts?

--js

Re: [Qemu-devel] hd-geo-test creates 4GB files on FSes that don't support sparse images, doesn't delete them on error

[Peter Maydell]

On 31 August 2015 at 19:54, John Snow <jsnow@redhat.com> wrote:
> Oh, taking a look at it, it needs to writethat MBR data to the file
> before it opens it. We don't have an existing qemu-io dependency here to
> use.
>
> I could add it, but the line between iotest and qtest starts to get
> pretty fuzzy.

I don't care whether we call this test an iotest or a qtest,
but if we can avoid creating a 4GB test file then we should...

thanks
-- PMM

Re: [Qemu-devel] hd-geo-test creates 4GB files on FSes that don't support sparse images, doesn't delete them on error

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 08/28/2015 09:06 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> On 08/27/2015 11:29 AM, Eric Blake wrote:
>>>> On 08/27/2015 09:17 AM, Peter Maydell wrote:
>>>>> I've noticed recently that tests/hd-geo-test.c creates test disk
>>>>> images which are 4GB in size, which is a problem if the filesystem
>>>>> on the host doesn't support sparse files. In particular, OSX's HFS+
>>>>> doesn't have sparse file support, and Windows probably doesn't either.
>>>>
>>>> Windows NTFS supports sparse files (minimum hole size of 64k), but it
>>>> can be a pain to set up, and while it saves disk space, it may actually
>>>> slow your program down.
>>>>
>>>> [At one point cygwin created sparse files on windows by default, but
>>>> because it was demonstrated to hurt performance in dealing with sparse
>>>> files, because Windows doesn't handle sparse files efficiently, the
>>>> cygwin defaults were switched so that it now requires an explicit opt-in
>>>> mount option before even attempting sparse files]
>>>>
>>>>> Worse, if the test fails an assertion somewhere the test doesn't
>>>>> clean up after itself and leaves a 4GB file lying around in /tmp/.
>>>>>
>>>>> It would be nice if we could skip these tests on filesystems that
>>>>> don't have sparse file support...
>>>>
>>>> Or even where sparse files are supported but not default.
>>>>
>>>
>>> Does this test *require* the raw format?
>> 
>> If memory serves, the test doesn't require a specific format, only the
>> size and the contents of the MBR matters.
>> 
>>> Use tests/libqos/libqos.c mkqcow2 instead. I'll send a patch.
>> 
>> Go right ahead.
>> 
>
> Oh, taking a look at it, it needs to writethat MBR data to the file
> before it opens it. We don't have an existing qemu-io dependency here to
> use.
>
> I could add it, but the line between iotest and qtest starts to get
> pretty fuzzy.
>
> Thoughts?

Let's go back to the original problem for a bit:

(1) The test needs a big image, contents beyond the MBR doesn't matter.
Image format doesn't matter, either.

(2) On losing filesystem, this image takes up significant space, because
we use "raw".

(3) We fail to clean it up on failure.  This is a general problem with
tests creating files.

Independently of what else we do, (3) is worth fixing.  Make each test
run in its own directory, create files only there, and have the test
harness remove the directory.

John suggested to attack (2) by using "qcow2", but that turns out to be
a bit harder than anticipated: the infrastructure for creating such
images isn't there.  Options:

(a) Create the infrastructure in qtests.

(b) Move the test to iotests.

(c) Create the image once and commit it to git.

(d) Ignore the problem.  Needing 8GiB for tests if your filesystem sucks
isn't exactly horrible.   But (3) really needs fixing then.

We could try to attack (1) to mitigate.  I don't remember why I picked
8GiB.  Perhaps we can make the image somewhat smaller without
compromising the test.  Need to examine the geometry guessing code to
figure that out.