blktests failures with v6.1

blktests failures with v6.1

[Shinichiro Kawasaki]

Hi all,

I ran the latest blktests (git hash: b35866f2d0df) with v6.1 kernel and
observed 5 test cases fail. I call for support to fix them.

List of failure cases
=====================
#1: block/011
#2: block/024
#3: nvme/002
#4: nvme/016
#5: nvme/017

Failure description
===================

#1: block/011

   The test case fails with NVME devices due to lockdep WARNING "possible
   circular locking dependency detected". Reported in September [1] and solution
   was discussed. A kernel fix patch was posted [2]. It may not be the best fix.

   [1] https://lore.kernel.org/linux-block/20220930001943.zdbvolc3gkekfmcv@shindev/
   [2] https://lore.kernel.org/linux-nvme/20221012080318.705449-1-shinichiro.kawasaki@wdc.com/

   This test case shows different failure symptom with HDDs. Need further
   investigation.

#2: block/024

   Fails on slow machines. I suspect test case needs improvement.
   Need further investigation.

   block/024 (do I/O faster than a jiffy and check iostats times) [failed]
    runtime    ...  4.347s
    --- tests/block/024.out     2022-12-06 20:51:41.525066605 +0900
    +++ /home/shin/kts/kernel-test-suite/sets/blktests/log/runlog/nodev/block/024.out.bad       2022-12-07 12:51:03.610924521 +0900
    @@ -6,5 +6,5 @@
     read 1 s
     write 1 s
     read 2 s
    -write 3 s
    +write 4 s
     Test complete

#3: nvme/002
#4: nvme/016
#5: nvme/017

   The test cases fail with similar messages below. Reported in June [3].
   Fixes in the test cases are expected.

   [3] https://lore.kernel.org/linux-nvme/20220609235329.4jbz4wr3eg2nmzqa@shindev/

   nvme/002 (create many subsystems and test discovery)         [failed]
    runtime    ...  119.796s
    --- tests/nvme/002.out      2022-12-13 12:42:53.694306873 +0900
    +++ /home/shin/kts/kernel-test-suite/sets/blktests/log/runlog/nodev/nvme/002.out.bad        2022-12-13 14:06:35.864050520 +0900
    @@ -1,3003 +1,3006 @@
     Running nvme/002
    -Discovery Log Number of Records 1000, Generation counter X
    +Discovery Log Number of Records 1001, Generation counter X
     =====Discovery Log Entry 0======
     trtype:  loop
    -subnqn:  blktests-subsystem-0
    +subnqn:  nqn.2014-08.org.nvmexpress.discovery
    ...
    (Run 'diff -u tests/nvme/002.out /home/shin/kts/kernel-test-suite/sets/blktests/log/runlog/nodev/nvme/002.out.bad' to see the entire diff)

   nvme/016 (create/delete many NVMeOF block device-backed ns and test discovery) [failed]
    runtime    ...  46.695s
    --- tests/nvme/016.out      2022-12-13 12:42:53.706306913 +0900
    +++ /home/shin/kts/kernel-test-suite/sets/blktests/log/runlog/nodev/nvme/016.out.bad        2022-12-13 14:12:30.566239949 +0900
    @@ -1,6 +1,9 @@
     Running nvme/016
    -Discovery Log Number of Records 1, Generation counter X
    +Discovery Log Number of Records 2, Generation counter X
     =====Discovery Log Entry 0======
     trtype:  loop
    +subnqn:  nqn.2014-08.org.nvmexpress.discovery
    +=====Discovery Log Entry 1======
    ...
    (Run 'diff -u tests/nvme/016.out /home/shin/kts/kernel-test-suite/sets/blktests/log/runlog/nodev/nvme/016.out.bad' to see the entire diff)

   nvme/017 (create/delete many file-ns and test discovery)     [failed]
    runtime    ...  81.274s
    --- tests/nvme/017.out      2022-12-13 12:42:53.707306917 +0900
    +++ /home/shin/kts/kernel-test-suite/sets/blktests/log/runlog/nodev/nvme/017.out.bad        2022-12-13 14:13:52.828515799 +0900
    @@ -1,6 +1,9 @@
     Running nvme/017
    -Discovery Log Number of Records 1, Generation counter X
    +Discovery Log Number of Records 2, Generation counter X
     =====Discovery Log Entry 0======
     trtype:  loop
    +subnqn:  nqn.2014-08.org.nvmexpress.discovery
    +=====Discovery Log Entry 1======
    ...
    (Run 'diff -u tests/nvme/017.out /home/shin/kts/kernel-test-suite/sets/blktests/log/runlog/nodev/nvme/017.out.bad' to see the entire diff)

-- 
Shin'ichiro Kawasaki

Re: blktests failures with v6.1

[Christoph Hellwig]

On Fri, Dec 23, 2022 at 04:50:42AM +0000, Shinichiro Kawasaki wrote:
>    The test case fails with NVME devices due to lockdep WARNING "possible
>    circular locking dependency detected". Reported in September [1] and solution
>    was discussed. A kernel fix patch was posted [2]. It may not be the best fix.
> 
>    [1] https://lore.kernel.org/linux-block/20220930001943.zdbvolc3gkekfmcv@shindev/
>    [2] https://lore.kernel.org/linux-nvme/20221012080318.705449-1-shinichiro.kawasaki@wdc.com/
> 
>    This test case shows different failure symptom with HDDs. Need further
>    investigation.

Yes.  Not a huge fan of the fix, but we need to look into this.
Sorry for delaying this so long, I'll try to look into this.

> #2: block/024
> 
>    Fails on slow machines. I suspect test case needs improvement.
>    Need further investigation.
> 
>    block/024 (do I/O faster than a jiffy and check iostats times) [failed]
>     runtime    ...  4.347s
>     --- tests/block/024.out     2022-12-06 20:51:41.525066605 +0900
>     +++ /home/shin/kts/kernel-test-suite/sets/blktests/log/runlog/nodev/block/024.out.bad       2022-12-07 12:51:03.610924521 +0900
>     @@ -6,5 +6,5 @@
>      read 1 s
>      write 1 s
>      read 2 s
>     -write 3 s
>     +write 4 s

Yes, I've been seeing this for a while as well.

> #3: nvme/002
> #4: nvme/016
> #5: nvme/017
> 
>    The test cases fail with similar messages below. Reported in June [3].
>    Fixes in the test cases are expected.

I think this is related to the current NQN that Hannes added.
Hannes, can you look into this?

Re: blktests failures with v6.1

[Chaitanya Kulkarni]

Hannes,

>> #3: nvme/002
>> #4: nvme/016
>> #5: nvme/017
>>
>>     The test cases fail with similar messages below. Reported in June [3].
>>     Fixes in the test cases are expected.
> 
> I think this is related to the current NQN that Hannes added.
> Hannes, can you look into this?
> 

These testcases are still failing on latest nvme-6.3 HEAD:-

commit baff6491448b487e920faaa117e432989cbafa89 (HEAD -> nvme-6.3, 
origin/nvme-6.3)
Author: Keith Busch <kbusch@kernel.org>
Date:   Fri Jan 27 08:56:20 2023 -0800

     nvme: mask CSE effects for security receive

can you please look into these ?

-ck

Re: blktests failures with v6.1

[Hannes Reinecke]

On 2/6/23 08:14, Chaitanya Kulkarni wrote:
> Hannes,
> 
>>> #3: nvme/002
>>> #4: nvme/016
>>> #5: nvme/017
>>>
>>>      The test cases fail with similar messages below. Reported in June [3].
>>>      Fixes in the test cases are expected.
>>
>> I think this is related to the current NQN that Hannes added.
>> Hannes, can you look into this?
>>
> 
> These testcases are still failing on latest nvme-6.3 HEAD:-
> 
> commit baff6491448b487e920faaa117e432989cbafa89 (HEAD -> nvme-6.3,
> origin/nvme-6.3)
> Author: Keith Busch <kbusch@kernel.org>
> Date:   Fri Jan 27 08:56:20 2023 -0800
> 
>       nvme: mask CSE effects for security receive
> 
> can you please look into these ?
> 
Indeed, these _are_ the current NQN changes; but they do work as 
designed (main change was to add an entry for the discovery subsystem in 
the discovery log page).
And as we're checking the out of the discovery log we found that things 
changed; which they have.
I'll be sending a patch for blktests.

Cheers,

Hannes

Re: blktests failures with v6.1

[Shinichiro Kawasaki]

On Feb 06, 2023 / 14:06, Hannes Reinecke wrote:
[...]
> I'll be sending a patch for blktests.

The patch has got merged [1]. I'm happy to see them fixed :)
Thank you Hannes, Christoph and Chaitanya.

[1] https://github.com/osandov/blktests/pull/111

-- 
Shin'ichiro Kawasaki