[PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

[PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

[Andrey Drobyshev]

There're tests whose logic implies running without O_DIRECT set,
otherwise they fail when running iotests in '-nocache' mode.  For these
tests let's add _require_no_o_direct() helper which can be put in the
preabmle and which makes sure '-nocache' isn't set.  Use it to skip
running the following tests:

  * 271: creates files with unaligned sizes, thus producing multiple
    errors like:

qemu-io: can't open device /path/to/t.qcow2.raw: Cannot get 'write'
permission without 'resize': Image size is not a multiple of request alignment

  * 308, file-io-error: use fuse exports.  Though fuse does have
    'direct-io' mode (see https://docs.kernel.org/filesystems/fuse-io.html)
    we aren't using it yet, thus getting errors like:

qemu-io: can't open device /path/to/t.qcow2.fuse: Could not open
'/path/to/t.qcow2.fuse': filesystem does not support O_DIRECT

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 tests/qemu-iotests/271                 | 1 +
 tests/qemu-iotests/308                 | 2 ++
 tests/qemu-iotests/common.rc           | 7 +++++++
 tests/qemu-iotests/tests/file-io-error | 1 +
 4 files changed, 11 insertions(+)

diff --git a/tests/qemu-iotests/271 b/tests/qemu-iotests/271
index 59a6fafa2f..1424b6954d 100755
--- a/tests/qemu-iotests/271
+++ b/tests/qemu-iotests/271
@@ -44,6 +44,7 @@ _supported_fmt qcow2
 _supported_proto file nfs
 _supported_os Linux
 _unsupported_imgopts extended_l2 compat=0.10 cluster_size data_file refcount_bits=1[^0-9]
+_require_no_o_direct
 
 l2_offset=$((0x40000))
 
diff --git a/tests/qemu-iotests/308 b/tests/qemu-iotests/308
index de12b2b1b9..535455e5b1 100755
--- a/tests/qemu-iotests/308
+++ b/tests/qemu-iotests/308
@@ -52,6 +52,8 @@ _unsupported_fmt vpc
 _supported_proto file # We create the FUSE export manually
 _supported_os Linux # We need /dev/urandom
 
+_require_no_o_direct
+
 # $1: Export ID
 # $2: Options (beyond the node-name and ID)
 # $3: Expected return value (defaults to 'return')
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index 95c12577dd..f61eae73b4 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -857,6 +857,13 @@ _check_o_direct()
     [[ "$out" != *"O_DIRECT"* ]]
 }
 
+_require_no_o_direct()
+{
+    if [ $CACHEMODE == "none" ] || [ $CACHEMODE == "directsync" ]; then
+        _notrun "not suitable for cache mode: $CACHEMODE (implies O_DIRECT)"
+    fi
+}
+
 _require_o_direct()
 {
     if ! _check_o_direct; then
diff --git a/tests/qemu-iotests/tests/file-io-error b/tests/qemu-iotests/tests/file-io-error
index 88ee5f670c..2b8dc7f009 100755
--- a/tests/qemu-iotests/tests/file-io-error
+++ b/tests/qemu-iotests/tests/file-io-error
@@ -40,6 +40,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Format-agnostic (we do not use any), but we do test the file protocol
 _supported_proto file
 _require_drivers blkdebug null-co
+_require_no_o_direct
 
 if [ "$IMGOPTSSYNTAX" = "true" ]; then
     # We need `$QEMU_IO -f file` to work; IMGOPTSSYNTAX uses --image-opts,
-- 
2.39.3

Re: [PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

[Andrey Drobyshev]

On 12/11/23 15:32, Andrey Drobyshev wrote:
> There're tests whose logic implies running without O_DIRECT set,
> otherwise they fail when running iotests in '-nocache' mode.  For these
> tests let's add _require_no_o_direct() helper which can be put in the
> preabmle and which makes sure '-nocache' isn't set.  Use it to skip
> running the following tests:
> 
>   * 271: creates files with unaligned sizes, thus producing multiple
>     errors like:
> 
> qemu-io: can't open device /path/to/t.qcow2.raw: Cannot get 'write'
> permission without 'resize': Image size is not a multiple of request alignment
> 
>   * 308, file-io-error: use fuse exports.  Though fuse does have
>     'direct-io' mode (see https://docs.kernel.org/filesystems/fuse-io.html)
>     we aren't using it yet, thus getting errors like:
> 
> qemu-io: can't open device /path/to/t.qcow2.fuse: Could not open
> '/path/to/t.qcow2.fuse': filesystem does not support O_DIRECT
> 
> Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
> ---
>  tests/qemu-iotests/271                 | 1 +
>  tests/qemu-iotests/308                 | 2 ++
>  tests/qemu-iotests/common.rc           | 7 +++++++
>  tests/qemu-iotests/tests/file-io-error | 1 +
>  4 files changed, 11 insertions(+)
> 
> diff --git a/tests/qemu-iotests/271 b/tests/qemu-iotests/271
> index 59a6fafa2f..1424b6954d 100755
> --- a/tests/qemu-iotests/271
> +++ b/tests/qemu-iotests/271
> @@ -44,6 +44,7 @@ _supported_fmt qcow2
>  _supported_proto file nfs
>  _supported_os Linux
>  _unsupported_imgopts extended_l2 compat=0.10 cluster_size data_file refcount_bits=1[^0-9]
> +_require_no_o_direct
>  
>  l2_offset=$((0x40000))
>  
> diff --git a/tests/qemu-iotests/308 b/tests/qemu-iotests/308
> index de12b2b1b9..535455e5b1 100755
> --- a/tests/qemu-iotests/308
> +++ b/tests/qemu-iotests/308
> @@ -52,6 +52,8 @@ _unsupported_fmt vpc
>  _supported_proto file # We create the FUSE export manually
>  _supported_os Linux # We need /dev/urandom
>  
> +_require_no_o_direct
> +
>  # $1: Export ID
>  # $2: Options (beyond the node-name and ID)
>  # $3: Expected return value (defaults to 'return')
> diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
> index 95c12577dd..f61eae73b4 100644
> --- a/tests/qemu-iotests/common.rc
> +++ b/tests/qemu-iotests/common.rc
> @@ -857,6 +857,13 @@ _check_o_direct()
>      [[ "$out" != *"O_DIRECT"* ]]
>  }
>  
> +_require_no_o_direct()
> +{
> +    if [ $CACHEMODE == "none" ] || [ $CACHEMODE == "directsync" ]; then
> +        _notrun "not suitable for cache mode: $CACHEMODE (implies O_DIRECT)"
> +    fi
> +}
> +
>  _require_o_direct()
>  {
>      if ! _check_o_direct; then
> diff --git a/tests/qemu-iotests/tests/file-io-error b/tests/qemu-iotests/tests/file-io-error
> index 88ee5f670c..2b8dc7f009 100755
> --- a/tests/qemu-iotests/tests/file-io-error
> +++ b/tests/qemu-iotests/tests/file-io-error
> @@ -40,6 +40,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
>  # Format-agnostic (we do not use any), but we do test the file protocol
>  _supported_proto file
>  _require_drivers blkdebug null-co
> +_require_no_o_direct
>  
>  if [ "$IMGOPTSSYNTAX" = "true" ]; then
>      # We need `$QEMU_IO -f file` to work; IMGOPTSSYNTAX uses --image-opts,

Ping

Re: [PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

[Andrey Drobyshev]

On 1/11/24 14:53, Andrey Drobyshev wrote:
> On 12/11/23 15:32, Andrey Drobyshev wrote:
>> There're tests whose logic implies running without O_DIRECT set,
>> otherwise they fail when running iotests in '-nocache' mode.  For these
>> tests let's add _require_no_o_direct() helper which can be put in the
>> preabmle and which makes sure '-nocache' isn't set.  Use it to skip
>> running the following tests:
>>
>>   * 271: creates files with unaligned sizes, thus producing multiple
>>     errors like:
>>
>> qemu-io: can't open device /path/to/t.qcow2.raw: Cannot get 'write'
>> permission without 'resize': Image size is not a multiple of request alignment
>>
>>   * 308, file-io-error: use fuse exports.  Though fuse does have
>>     'direct-io' mode (see https://docs.kernel.org/filesystems/fuse-io.html)
>>     we aren't using it yet, thus getting errors like:
>>
>> qemu-io: can't open device /path/to/t.qcow2.fuse: Could not open
>> '/path/to/t.qcow2.fuse': filesystem does not support O_DIRECT
>>
>> Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
>> ---
>>  tests/qemu-iotests/271                 | 1 +
>>  tests/qemu-iotests/308                 | 2 ++
>>  tests/qemu-iotests/common.rc           | 7 +++++++
>>  tests/qemu-iotests/tests/file-io-error | 1 +
>>  4 files changed, 11 insertions(+)
>>
>> diff --git a/tests/qemu-iotests/271 b/tests/qemu-iotests/271
>> index 59a6fafa2f..1424b6954d 100755
>> --- a/tests/qemu-iotests/271
>> +++ b/tests/qemu-iotests/271
>> @@ -44,6 +44,7 @@ _supported_fmt qcow2
>>  _supported_proto file nfs
>>  _supported_os Linux
>>  _unsupported_imgopts extended_l2 compat=0.10 cluster_size data_file refcount_bits=1[^0-9]
>> +_require_no_o_direct
>>  
>>  l2_offset=$((0x40000))
>>  
>> diff --git a/tests/qemu-iotests/308 b/tests/qemu-iotests/308
>> index de12b2b1b9..535455e5b1 100755
>> --- a/tests/qemu-iotests/308
>> +++ b/tests/qemu-iotests/308
>> @@ -52,6 +52,8 @@ _unsupported_fmt vpc
>>  _supported_proto file # We create the FUSE export manually
>>  _supported_os Linux # We need /dev/urandom
>>  
>> +_require_no_o_direct
>> +
>>  # $1: Export ID
>>  # $2: Options (beyond the node-name and ID)
>>  # $3: Expected return value (defaults to 'return')
>> diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
>> index 95c12577dd..f61eae73b4 100644
>> --- a/tests/qemu-iotests/common.rc
>> +++ b/tests/qemu-iotests/common.rc
>> @@ -857,6 +857,13 @@ _check_o_direct()
>>      [[ "$out" != *"O_DIRECT"* ]]
>>  }
>>  
>> +_require_no_o_direct()
>> +{
>> +    if [ $CACHEMODE == "none" ] || [ $CACHEMODE == "directsync" ]; then
>> +        _notrun "not suitable for cache mode: $CACHEMODE (implies O_DIRECT)"
>> +    fi
>> +}
>> +
>>  _require_o_direct()
>>  {
>>      if ! _check_o_direct; then
>> diff --git a/tests/qemu-iotests/tests/file-io-error b/tests/qemu-iotests/tests/file-io-error
>> index 88ee5f670c..2b8dc7f009 100755
>> --- a/tests/qemu-iotests/tests/file-io-error
>> +++ b/tests/qemu-iotests/tests/file-io-error
>> @@ -40,6 +40,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
>>  # Format-agnostic (we do not use any), but we do test the file protocol
>>  _supported_proto file
>>  _require_drivers blkdebug null-co
>> +_require_no_o_direct
>>  
>>  if [ "$IMGOPTSSYNTAX" = "true" ]; then
>>      # We need `$QEMU_IO -f file` to work; IMGOPTSSYNTAX uses --image-opts,
> 
> Ping

Ping

Re: [PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

[Eric Blake]

On Mon, Dec 11, 2023 at 03:32:23PM +0200, Andrey Drobyshev wrote:
> There're tests whose logic implies running without O_DIRECT set,
> otherwise they fail when running iotests in '-nocache' mode.  For these
> tests let's add _require_no_o_direct() helper which can be put in the
> preabmle and which makes sure '-nocache' isn't set.  Use it to skip

preamble

> running the following tests:
> 
>   * 271: creates files with unaligned sizes, thus producing multiple
>     errors like:
> 
> qemu-io: can't open device /path/to/t.qcow2.raw: Cannot get 'write'
> permission without 'resize': Image size is not a multiple of request alignment

I wonder if we can instead tweak the test to use larger sizes such
that all accesses ARE aligned even with O_DIRECT.

/me goes and reads the test...

    # Note that the image size is not a multiple of the cluster size
    _reset_img 2083k

Ah - we really DO have a test that depends on odd sizing; where
changing it to be more nicely aligned will break other assumptions in
the test.

> 
>   * 308, file-io-error: use fuse exports.  Though fuse does have
>     'direct-io' mode (see https://docs.kernel.org/filesystems/fuse-io.html)
>     we aren't using it yet, thus getting errors like:
> 
> qemu-io: can't open device /path/to/t.qcow2.fuse: Could not open
> '/path/to/t.qcow2.fuse': filesystem does not support O_DIRECT

And I agree that this one is beyond our control, so adding skip
support makes sense.

> 
> Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
> ---
>  tests/qemu-iotests/271                 | 1 +
>  tests/qemu-iotests/308                 | 2 ++
>  tests/qemu-iotests/common.rc           | 7 +++++++
>  tests/qemu-iotests/tests/file-io-error | 1 +
>  4 files changed, 11 insertions(+)

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

Re: [PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

[Kevin Wolf]

Am 11.12.2023 um 14:32 hat Andrey Drobyshev geschrieben:
> There're tests whose logic implies running without O_DIRECT set,
> otherwise they fail when running iotests in '-nocache' mode.  For these
> tests let's add _require_no_o_direct() helper which can be put in the
> preabmle and which makes sure '-nocache' isn't set.  Use it to skip
> running the following tests:
> 
>   * 271: creates files with unaligned sizes, thus producing multiple
>     errors like:
> 
> qemu-io: can't open device /path/to/t.qcow2.raw: Cannot get 'write'
> permission without 'resize': Image size is not a multiple of request alignment
> 
>   * 308, file-io-error: use fuse exports.  Though fuse does have
>     'direct-io' mode (see https://docs.kernel.org/filesystems/fuse-io.html)
>     we aren't using it yet, thus getting errors like:
> 
> qemu-io: can't open device /path/to/t.qcow2.fuse: Could not open
> '/path/to/t.qcow2.fuse': filesystem does not support O_DIRECT
> 
> Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>

How are you running qemu-iotests to make these tests fail? I tried to
reproduce, but they just pass for me:

$ tests/qemu-iotests/check -qcow2 -nocache 271 308 file-io-error
[...]
271             pass       [11:20:50] [11:21:11]   21.1s  (last: 20.4s) 
308             pass       [11:21:11] [11:21:14]   3.3s   (last: 3.3s)  
file-io-error   pass       [11:21:14] [11:21:14]   0.3s   (last: 0.3s)  
Passed all 3 iotests

$ tests/qemu-iotests/check -raw -nocache 271 308 file-io-error
271             not run    [11:21:20] [11:21:21]   ...                  not suitable for this image format: raw
308             pass       [11:21:21] [11:21:24]   3.8s   (last: 2.8s)  
file-io-error   pass       [11:21:24] [11:21:25]   0.3s   (last: 0.3s)  
Not run: 271
Passed all 2 iotests

Kevin

Re: [PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

[Andrey Drobyshev]

On 1/26/24 12:24, Kevin Wolf wrote:
> Am 11.12.2023 um 14:32 hat Andrey Drobyshev geschrieben:
>> There're tests whose logic implies running without O_DIRECT set,
>> otherwise they fail when running iotests in '-nocache' mode.  For these
>> tests let's add _require_no_o_direct() helper which can be put in the
>> preabmle and which makes sure '-nocache' isn't set.  Use it to skip
>> running the following tests:
>>
>>   * 271: creates files with unaligned sizes, thus producing multiple
>>     errors like:
>>
>> qemu-io: can't open device /path/to/t.qcow2.raw: Cannot get 'write'
>> permission without 'resize': Image size is not a multiple of request alignment
>>
>>   * 308, file-io-error: use fuse exports.  Though fuse does have
>>     'direct-io' mode (see https://docs.kernel.org/filesystems/fuse-io.html)
>>     we aren't using it yet, thus getting errors like:
>>
>> qemu-io: can't open device /path/to/t.qcow2.fuse: Could not open
>> '/path/to/t.qcow2.fuse': filesystem does not support O_DIRECT
>>
>> Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
> 
> How are you running qemu-iotests to make these tests fail? I tried to
> reproduce, but they just pass for me:
> 
> $ tests/qemu-iotests/check -qcow2 -nocache 271 308 file-io-error
> [...]
> 271             pass       [11:20:50] [11:21:11]   21.1s  (last: 20.4s) 
> 308             pass       [11:21:11] [11:21:14]   3.3s   (last: 3.3s)  
> file-io-error   pass       [11:21:14] [11:21:14]   0.3s   (last: 0.3s)  
> Passed all 3 iotests
> 
> $ tests/qemu-iotests/check -raw -nocache 271 308 file-io-error
> 271             not run    [11:21:20] [11:21:21]   ...                  not suitable for this image format: raw
> 308             pass       [11:21:21] [11:21:24]   3.8s   (last: 2.8s)  
> file-io-error   pass       [11:21:24] [11:21:25]   0.3s   (last: 0.3s)  
> Not run: 271
> Passed all 2 iotests
> 
> Kevin
> 

As for the test 271, I imagine this might be caused by different request
alignment.  The failure occurs in block.c, bdrv_node_refresh_perm().  If
I print the alignment out explicitly, I get:

qemu-io: can't open device /path/to/t.qcow2.raw: Cannot get 'write'
permission without 'resize': Image size is not a multiple of request
alignment: 4096

For the record, I'm running tests on ext4.

I'm not sure about the fuse tests though.  Could it also have smth to do
with the underlying fs?

Andrey

Re: [PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

[Kevin Wolf]

Am 26.01.2024 um 12:04 hat Andrey Drobyshev geschrieben:
> On 1/26/24 12:24, Kevin Wolf wrote:
> > Am 11.12.2023 um 14:32 hat Andrey Drobyshev geschrieben:
> >> There're tests whose logic implies running without O_DIRECT set,
> >> otherwise they fail when running iotests in '-nocache' mode.  For these
> >> tests let's add _require_no_o_direct() helper which can be put in the
> >> preabmle and which makes sure '-nocache' isn't set.  Use it to skip
> >> running the following tests:
> >>
> >>   * 271: creates files with unaligned sizes, thus producing multiple
> >>     errors like:
> >>
> >> qemu-io: can't open device /path/to/t.qcow2.raw: Cannot get 'write'
> >> permission without 'resize': Image size is not a multiple of request alignment
> >>
> >>   * 308, file-io-error: use fuse exports.  Though fuse does have
> >>     'direct-io' mode (see https://docs.kernel.org/filesystems/fuse-io.html)
> >>     we aren't using it yet, thus getting errors like:
> >>
> >> qemu-io: can't open device /path/to/t.qcow2.fuse: Could not open
> >> '/path/to/t.qcow2.fuse': filesystem does not support O_DIRECT
> >>
> >> Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
> > 
> > How are you running qemu-iotests to make these tests fail? I tried to
> > reproduce, but they just pass for me:
> > 
> > $ tests/qemu-iotests/check -qcow2 -nocache 271 308 file-io-error
> > [...]
> > 271             pass       [11:20:50] [11:21:11]   21.1s  (last: 20.4s) 
> > 308             pass       [11:21:11] [11:21:14]   3.3s   (last: 3.3s)  
> > file-io-error   pass       [11:21:14] [11:21:14]   0.3s   (last: 0.3s)  
> > Passed all 3 iotests
> > 
> > $ tests/qemu-iotests/check -raw -nocache 271 308 file-io-error
> > 271             not run    [11:21:20] [11:21:21]   ...                  not suitable for this image format: raw
> > 308             pass       [11:21:21] [11:21:24]   3.8s   (last: 2.8s)  
> > file-io-error   pass       [11:21:24] [11:21:25]   0.3s   (last: 0.3s)  
> > Not run: 271
> > Passed all 2 iotests
> > 
> > Kevin
> > 
> 
> As for the test 271, I imagine this might be caused by different request
> alignment.  The failure occurs in block.c, bdrv_node_refresh_perm().  If
> I print the alignment out explicitly, I get:
> 
> qemu-io: can't open device /path/to/t.qcow2.raw: Cannot get 'write'
> permission without 'resize': Image size is not a multiple of request
> alignment: 4096

Oh, do you have a 4k sector size? I think for me O_DIRECT only enforces
512 byte alignment. I can later try again on a 4k sector size loopback
device or something.

If so, I think that's worth mentioning in the commit message.

> For the record, I'm running tests on ext4.

I don't expect that this makes the difference, but xfs for me.

> I'm not sure about the fuse tests though.  Could it also have smth to do
> with the underlying fs?

Probably the same request alignment thing then.

Kevin