[LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Chaitanya Kulkarni]

Hi,

Since discussion of the storage stack and device driver at the
LSFMM 2017 (https://lwn.net/Articles/717699/), Omar Sandoval introduced 
a new framework "blktests" dedicated for Linux Kernel Block layer
testing that is maintained by Shinichiro Kawasaki :-

https://lwn.net/Articles/722785/
https://github.com/osandov/blktests

As Linux Kernel Block layer is central to the various file systems and
underlying low-level device drivers it is important to have a 
centralized testing framework and make sure it grows with the latest 
block layer changes which are being added based on the different device 
features from different device types (e.g. NVMe devices with Zoned
Namespace support).

Since then blktests has grown and became go-to framework where we have
integrated different stand-alone test suites like SRP-tests, NVMFTESTS,
NVMe Multipath tests, zone block device tests, into one central
framework, which has made an overall block layer testing and development
much easier than having to configure and execute different test cases
for each kernel release for different subsystems such as FS, NVMe, Zone
Block devices, etc).

Here is the list of the existing test categories:-
     block        :28
      loop        :8
      meta        :15
       nbd        :4
      nvme        :44
nvmeof-mp        :9
      scsi        :6
       srp        :16
       zbd        :10
----------------------------------------------------------------
9 Categories     :141 Tests

This project has gathered much attention and storage stack community is
actively participating and adding new test cases with different
categories to the framework.

Since addition of this framework we are consistently finding bugs
proactively with the help of blktests testcases.

For storage track, I would like to propose a session dedicated to
blktests. It is a great opportunity for the storage developers to gather
and have a discussion about:-

1. Current status of the blktests framework.
2. Any new/missing features that we want to add in the blktests.
3. Any new kernel features that could be used to make testing easier?
4. DM/MD Testcases.

E.g. Implementing new features in the null_blk.c in order to have device
independent complete test coverage. (e.g. adding discard command for
null_blk or any other specific REQ_OP). Discussion about having any new
tracepoint events in the block layer.

4. Any new test cases/categories which are lacking in the blktests
framework.

-ck

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Shinichiro Kawasaki]

CC+: Mike, dm-devel,

Hi Chaitanya, thanks for bringing this up! I definitely want to join and learn
from the discussions. Here I note my comments about them.

On Jan 18, 2023 / 23:52, Chaitanya Kulkarni wrote:
[...]
> For storage track, I would like to propose a session dedicated to
> blktests. It is a great opportunity for the storage developers to gather
> and have a discussion about:-
> 
> 1. Current status of the blktests framework.

In the session, I can talk shortly about recent blktests improvements and
failure cases.

> 2. Any new/missing features that we want to add in the blktests.

A feature I wish is to mark dangerous test cases which cause system crash, in
similar way as fstests does. I think they should be skipped by default not to
confuse new blktests users.

I remember that dmesg logging was discussed at the last LSFMMBPF, but it is not
yet implemented. It may worth revisit.

> 3. Any new kernel features that could be used to make testing easier?
> 4. DM/MD Testcases.

I took a liberty to add Mike and dm-devel to CC. Recently, a patch was posted to
add 'dm' test category [1]. I hope it will help DM/MD developers to add more
tests in the category. I would like to discuss if it is a good start, or if
anything is missing in blktests to support DM/MD testing.

[1] https://lore.kernel.org/linux-block/20221230065424.19998-1-yukuai1@huaweicloud.com/#t

> 
> E.g. Implementing new features in the null_blk.c in order to have device
> independent complete test coverage. (e.g. adding discard command for
> null_blk or any other specific REQ_OP). Discussion about having any new
> tracepoint events in the block layer.
> 
> 4. Any new test cases/categories which are lacking in the blktests
> framework.

One thing in my mind is character device. From recent discussions [2][3], it
looks worth adding some basic test cases for NVME character devices and SG
devices.

[2] https://lore.kernel.org/linux-block/20221221103441.3216600-1-mcgrof@kernel.org/
[3] https://lore.kernel.org/linux-scsi/Y77J%2Fw0gf2nIDMd%2F@x1-carbon/

-- 
Shin'ichiro Kawasaki

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Chaitanya Kulkarni]

On 1/18/23 18:20, Shinichiro Kawasaki wrote:
> CC+: Mike, dm-devel,
> 
> Hi Chaitanya, thanks for bringing this up! I definitely want to join and learn
> from the discussions. Here I note my comments about them.
> 
> On Jan 18, 2023 / 23:52, Chaitanya Kulkarni wrote:
> [...]
>> For storage track, I would like to propose a session dedicated to
>> blktests. It is a great opportunity for the storage developers to gather
>> and have a discussion about:-
>>
>> 1. Current status of the blktests framework.
> 
> In the session, I can talk shortly about recent blktests improvements and
> failure cases.
> 
>> 2. Any new/missing features that we want to add in the blktests.
> 
> A feature I wish is to mark dangerous test cases which cause system crash, in
> similar way as fstests does. I think they should be skipped by default not to
> confuse new blktests users.
> 
> I remember that dmesg logging was discussed at the last LSFMMBPF, but it is not
> yet implemented. It may worth revisit.
> 
>> 3. Any new kernel features that could be used to make testing easier?
>> 4. DM/MD Testcases.
> 
> I took a liberty to add Mike and dm-devel to CC. Recently, a patch was posted to
> add 'dm' test category [1]. I hope it will help DM/MD developers to add more
> tests in the category. I would like to discuss if it is a good start, or if
> anything is missing in blktests to support DM/MD testing.
> 
> [1] https://lore.kernel.org/linux-block/20221230065424.19998-1-yukuai1@huaweicloud.com/#t

we really need to sort out the dm testcases, without dm testcases it
not allowing us to create baseline correctness for block layer,
I've already discussed that in the last LSF.

> 
>>
>> E.g. Implementing new features in the null_blk.c in order to have device
>> independent complete test coverage. (e.g. adding discard command for
>> null_blk or any other specific REQ_OP). Discussion about having any new
>> tracepoint events in the block layer.
>>
>> 4. Any new test cases/categories which are lacking in the blktests
>> framework.
> 
> One thing in my mind is character device. From recent discussions [2][3], it
> looks worth adding some basic test cases for NVME character devices and SG
> devices.
> 

Agree

-ck