QEMU policy for real file tests

QEMU policy for real file tests

[Christian Schoenebeck]

Hi,

is there a QEMU policy for test cases that create/write/read/delete real files 
and directories? E.g. should they be situated at a certain location and is any 
measure of sandboxing required?

Best regards,
Christian Schoenebeck

Re: QEMU policy for real file tests

[Alex Bennée]

Christian Schoenebeck <qemu_oss@crudebyte.com> writes:

> Hi,
>
> is there a QEMU policy for test cases that create/write/read/delete real files 
> and directories? E.g. should they be situated at a certain location and is any 
> measure of sandboxing required?

I don't think we have a hard and fast policy. It also depends on what
you are doing the test in - but ideally you should use a secure mktempd
(that can't clash) and clean-up after you are finished. This is a bit
easier in python than shell I think.

For example iotests end up in $BUILD_DIR/scratch (driven by TEST_DIR
being set somewhere) whereas the gdb stub tests use socket_dir =
TemporaryDirectory("qemu-gdbstub") which get auto-cleaned when it
finishes.


-- 
Alex Bennée

Re: QEMU policy for real file tests

[Daniel P. Berrangé]

On Thu, Sep 17, 2020 at 10:26:36AM +0100, Alex Bennée wrote:
> 
> Christian Schoenebeck <qemu_oss@crudebyte.com> writes:
> 
> > Hi,
> >
> > is there a QEMU policy for test cases that create/write/read/delete real files 
> > and directories? E.g. should they be situated at a certain location and is any 
> > measure of sandboxing required?
> 
> I don't think we have a hard and fast policy. It also depends on what
> you are doing the test in - but ideally you should use a secure mktempd
> (that can't clash) and clean-up after you are finished. This is a bit
> easier in python than shell I think.

mktempd will end up on /tmp usually which can be tmpfs and size limited,
so be mindful of the size of files you create. Don't assume you can
create multi-GB sized files !  Creating a temp dir underneath the build
dir (effectively CWD of the test) is a reasonable alternative.

> For example iotests end up in $BUILD_DIR/scratch (driven by TEST_DIR
> being set somewhere) whereas the gdb stub tests use socket_dir =
> TemporaryDirectory("qemu-gdbstub") which get auto-cleaned when it
> finishes.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: QEMU policy for real file tests

[Thomas Huth]

On 17/09/2020 11.37, Daniel P. Berrangé wrote:
> On Thu, Sep 17, 2020 at 10:26:36AM +0100, Alex Bennée wrote:
>>
>> Christian Schoenebeck <qemu_oss@crudebyte.com> writes:
>>
>>> Hi,
>>>
>>> is there a QEMU policy for test cases that create/write/read/delete real files 
>>> and directories? E.g. should they be situated at a certain location and is any 
>>> measure of sandboxing required?
>>
>> I don't think we have a hard and fast policy. It also depends on what
>> you are doing the test in - but ideally you should use a secure mktempd
>> (that can't clash) and clean-up after you are finished. This is a bit
>> easier in python than shell I think.
> 
> mktempd will end up on /tmp usually which can be tmpfs and size limited,
> so be mindful of the size of files you create. Don't assume you can
> create multi-GB sized files !  Creating a temp dir underneath the build
> dir (effectively CWD of the test) is a reasonable alternative.

Another thing to consider: If you want to create Unix sockets in your
tests, make sure that the file name does not get too long, since there
are limits on certain systems - i.e. socket files should be created in a
/tmp subdirectory, indeed.

 Thomas

Re: QEMU policy for real file tests

[Christian Schoenebeck]

On Donnerstag, 17. September 2020 11:55:00 CEST Thomas Huth wrote:
> On 17/09/2020 11.37, Daniel P. Berrangé wrote:
> > On Thu, Sep 17, 2020 at 10:26:36AM +0100, Alex Bennée wrote:
> >> Christian Schoenebeck <qemu_oss@crudebyte.com> writes:
> >>> Hi,
> >>> 
> >>> is there a QEMU policy for test cases that create/write/read/delete real
> >>> files and directories? E.g. should they be situated at a certain
> >>> location and is any measure of sandboxing required?
> >> 
> >> I don't think we have a hard and fast policy. It also depends on what
> >> you are doing the test in - but ideally you should use a secure mktempd
> >> (that can't clash) and clean-up after you are finished. This is a bit
> >> easier in python than shell I think.
> > 
> > mktempd will end up on /tmp usually which can be tmpfs and size limited,
> > so be mindful of the size of files you create. Don't assume you can
> > create multi-GB sized files !  Creating a temp dir underneath the build
> > dir (effectively CWD of the test) is a reasonable alternative.
> 
> Another thing to consider: If you want to create Unix sockets in your
> tests, make sure that the file name does not get too long, since there
> are limits on certain systems - i.e. socket files should be created in a
> /tmp subdirectory, indeed.
> 
>  Thomas

These answers already cover everything I need right now. Thanks!

Final question: if at some later point one large file needs to be created for 
some test case, is there some approximate size limit to stay below for not 
causing issues with free CI cloud services?

Best regards,
Christian Schoenebeck

Re: QEMU policy for real file tests

[Daniel P. Berrangé]

On Thu, Sep 17, 2020 at 02:06:33PM +0200, Christian Schoenebeck wrote:
> On Donnerstag, 17. September 2020 11:55:00 CEST Thomas Huth wrote:
> > On 17/09/2020 11.37, Daniel P. Berrangé wrote:
> > > On Thu, Sep 17, 2020 at 10:26:36AM +0100, Alex Bennée wrote:
> > >> Christian Schoenebeck <qemu_oss@crudebyte.com> writes:
> > >>> Hi,
> > >>> 
> > >>> is there a QEMU policy for test cases that create/write/read/delete real
> > >>> files and directories? E.g. should they be situated at a certain
> > >>> location and is any measure of sandboxing required?
> > >> 
> > >> I don't think we have a hard and fast policy. It also depends on what
> > >> you are doing the test in - but ideally you should use a secure mktempd
> > >> (that can't clash) and clean-up after you are finished. This is a bit
> > >> easier in python than shell I think.
> > > 
> > > mktempd will end up on /tmp usually which can be tmpfs and size limited,
> > > so be mindful of the size of files you create. Don't assume you can
> > > create multi-GB sized files !  Creating a temp dir underneath the build
> > > dir (effectively CWD of the test) is a reasonable alternative.
> > 
> > Another thing to consider: If you want to create Unix sockets in your
> > tests, make sure that the file name does not get too long, since there
> > are limits on certain systems - i.e. socket files should be created in a
> > /tmp subdirectory, indeed.
> > 
> >  Thomas
> 
> These answers already cover everything I need right now. Thanks!
> 
> Final question: if at some later point one large file needs to be created for 
> some test case, is there some approximate size limit to stay below for not 
> causing issues with free CI cloud services?

A GIT source tree checkout of QEMU is approx 1 GB in size right now.
A build with just one target enable takes another 1 GB.
So if creating files in the source tree, or a build tree then, I'd
suggest a rule of thumb is to stay below that level as a rough
order of magnitude.

If creating in /tmp then stay below 100 MB, as it can be surprisingly
space constrained in some cases.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: QEMU policy for real file tests

[Peter Maydell]

On Thu, 17 Sep 2020 at 13:24, Daniel P. Berrangé <berrange@redhat.com> wrote:
> If creating in /tmp then stay below 100 MB, as it can be surprisingly
> space constrained in some cases.

Also if creating in /tmp/ it's nice to use an obviously qemu-testing
related filename. If a 'make check' run falls over partway through it
is prone to leaving files un-cleaned-up in /tmp/ and it's much easier
to know you can just delete them if they're named 'qtest*' rather
than just 'tmp*'...

(Ideally we'd put in some more consistent framework for temp files
used by tests to put them all in a single subdir or something, for
convenience in cleaning up afterwards.)

thanks
-- PMM

Re: QEMU policy for real file tests

[Thomas Huth]

On 17/09/2020 14.06, Christian Schoenebeck wrote:
[...]
> Final question: if at some later point one large file needs to be created for 
> some test case, is there some approximate size limit to stay below for not 
> causing issues with free CI cloud services?

FWIW, I know that 4G is already too big for some containers on Travis,
see commit 178d383f10e15f5e5a7.

 Thomas

Re: QEMU policy for real file tests

[Peter Maydell]

On Thu, 17 Sep 2020 at 14:04, Thomas Huth <thuth@redhat.com> wrote:
>
> On 17/09/2020 14.06, Christian Schoenebeck wrote:
> [...]
> > Final question: if at some later point one large file needs to be created for
> > some test case, is there some approximate size limit to stay below for not
> > causing issues with free CI cloud services?
>
> FWIW, I know that 4G is already too big for some containers on Travis,
> see commit 178d383f10e15f5e5a7.

Yes. Also "it's sparse" doesn't always help -- eg on OSX there is
no sparse-file support so a 4GB file really does take 4GB even
if it's mostly zeroes...

thanks
-- PMM

Re: QEMU policy for real file tests

[Christian Schoenebeck]

On Donnerstag, 17. September 2020 14:40:42 CEST Peter Maydell wrote:
> (Ideally we'd put in some more consistent framework for temp files
> used by tests to put them all in a single subdir or something, for
> convenience in cleaning up afterwards.)

Yes, a unified interface for creating and auto wiping these, as well as 
communicating limits for them would be nice to allow test cases adapting to 
low quota vs. generous environments automatically.

For now I take Thomas' advice of 4GB as very rough barrier to stay away from.

On Donnerstag, 17. September 2020 15:11:18 CEST Peter Maydell wrote:
> On Thu, 17 Sep 2020 at 14:04, Thomas Huth <thuth@redhat.com> wrote:
> > On 17/09/2020 14.06, Christian Schoenebeck wrote:
> > [...]
> > 
> > > Final question: if at some later point one large file needs to be
> > > created for some test case, is there some approximate size limit to
> > > stay below for not causing issues with free CI cloud services?
> > 
> > FWIW, I know that 4G is already too big for some containers on Travis,
> > see commit 178d383f10e15f5e5a7.
> 
> Yes. Also "it's sparse" doesn't always help -- eg on OSX there is
> no sparse-file support so a 4GB file really does take 4GB even
> if it's mostly zeroes...

While I agree not using sparse files for test cases, as many environments 
don't support them; as a side note though: macOS actually supports sparse 
files for a long time, both on APFS and HPFS+ volumes. There is F_PUNCHHOLE 
and F_PREALLOCATE for that purpose and st_size reflects the logical file size, 
whereas st_blocks*S_BLKSIZE returns the actual allocated physical file size.

Best regards,
Christian Schoenebeck