[LSF/MM TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[LSF/MM TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Chaitanya Kulkarni]

Hi,

Since discussion of the storage stack and device driver at the LSFMM 2017
(https://lwn.net/Articles/717699/),  Omar Sandoval introduced a new framework
"blktests" dedicated for Linux Kernel Block layer testing.
(https://lwn.net/Articles/722785/, https://github.com/osandov/blktests).
 
As Linux Kernel Block layer is central to the various file systems and underlying
low-level device drivers it is important to have a centralized testing framework and
make sure it grows with the latest block layer changed which are being added based
on the different device features from different device types
(e.g. NVMe devices with Zoned Namespace support).

Since then blktests has grown and became go-to framework where we have integrated
different stand-alone test suites like SRP-tests, NVMFTESTS, NVMe Multipath tests,
zone block device tests, into one central framework, which has made an overall block layer
testing and development much easier than having to configure and execute different
test cases for each kernel release for different subsystems such as FS, NVMe,
Zone Block devices, etc). 

Here is the list of the existing test categories:-

├── block                                           28 Tests
├── loop                                             07 Tests
├── meta                                            12 Tests
├── nbd                                              02 Tests
├── nvme                                           28 Tests
├── nvmeof-mp                                  12 Tests
├── scsi                                              06 Tests
├── srp                                               13 Tests
└── zbd                                              05 Tests
---------------------------------------------------------------- 
           9 Categories                            ~110 Tests

This project has gathered much attention and storage stack community is actively
participating and adding new test cases with different categories to the framework. 

For storage track, we would like to propose a session dedicated to blktests. It is a great
opportunity for the storage developers to gather and have a discussion about:-

1. Current status of the blktests framework.
2. Any new/missing features that we want to add in the blktests.
3. Any new kernel features that could be used to make testing easier?
E.g. Implementing new features in the null_blk.c in order to have device
independent complete test coverage. (e.g. adding discard command for null_blk or any
other specific REQ_OP). Discussion about having any new tracepoint events in the block layer.
4. Any new test cases/categories which are lacking in the blktests framework.
 
Regards,
Chaitanya

Re: [LSF/MM TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Johannes Thumshirn]

On 06/02/2019 06:21, Chaitanya Kulkarni wrote:
> Hi,
> 
> Since discussion of the storage stack and device driver at the LSFMM 2017
> (https://lwn.net/Articles/717699/),  Omar Sandoval introduced a new framework
> "blktests" dedicated for Linux Kernel Block layer testing.
> (https://lwn.net/Articles/722785/, https://github.com/osandov/blktests).
>  
> As Linux Kernel Block layer is central to the various file systems and underlying
> low-level device drivers it is important to have a centralized testing framework and
> make sure it grows with the latest block layer changed which are being added based
> on the different device features from different device types
> (e.g. NVMe devices with Zoned Namespace support).
> 
> Since then blktests has grown and became go-to framework where we have integrated
> different stand-alone test suites like SRP-tests, NVMFTESTS, NVMe Multipath tests,
> zone block device tests, into one central framework, which has made an overall block layer
> testing and development much easier than having to configure and execute different
> test cases for each kernel release for different subsystems such as FS, NVMe,
> Zone Block devices, etc). 
> 
> Here is the list of the existing test categories:-
> 
> ├── block                                           28 Tests
> ├── loop                                             07 Tests
> ├── meta                                            12 Tests
> ├── nbd                                              02 Tests
> ├── nvme                                           28 Tests
> ├── nvmeof-mp                                  12 Tests
> ├── scsi                                              06 Tests
> ├── srp                                               13 Tests
> └── zbd                                              05 Tests
> ---------------------------------------------------------------- 
>            9 Categories                            ~110 Tests
> 
> This project has gathered much attention and storage stack community is actively
> participating and adding new test cases with different categories to the framework. 
> 
> For storage track, we would like to propose a session dedicated to blktests. It is a great
> opportunity for the storage developers to gather and have a discussion about:-
> 
> 1. Current status of the blktests framework.
> 2. Any new/missing features that we want to add in the blktests.
> 3. Any new kernel features that could be used to make testing easier?
> E.g. Implementing new features in the null_blk.c in order to have device
> independent complete test coverage. (e.g. adding discard command for null_blk or any
> other specific REQ_OP). Discussion about having any new tracepoint events in the block layer.
> 4. Any new test cases/categories which are lacking in the blktests framework.

One thing I'd love to see is more hardware/driver specific tests. I'm
sure Broadcom, Marvell, Huawei and all the others out there do have test
suites for their HBA drivers but not a single one of these tests is
publicly available.

We're also lacking tests for things like ioprio, persistent reservation,
bcache and so on.

Adding support for collecting gcov information after running a test case
would also be awesome (this is missing in xfstests as well).

So I think a session on blktests can help us get the gap closed.

Byte,
	Johannes
-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: [LSF/MM TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Damien Le Moal]

On 2019/02/06 19:32, Johannes Thumshirn wrote:
> On 06/02/2019 06:21, Chaitanya Kulkarni wrote:
>> Hi,
>>
>> Since discussion of the storage stack and device driver at the LSFMM 2017
>> (https://lwn.net/Articles/717699/),  Omar Sandoval introduced a new framework
>> "blktests" dedicated for Linux Kernel Block layer testing.
>> (https://lwn.net/Articles/722785/, https://github.com/osandov/blktests).
>>  
>> As Linux Kernel Block layer is central to the various file systems and underlying
>> low-level device drivers it is important to have a centralized testing framework and
>> make sure it grows with the latest block layer changed which are being added based
>> on the different device features from different device types
>> (e.g. NVMe devices with Zoned Namespace support).
>>
>> Since then blktests has grown and became go-to framework where we have integrated
>> different stand-alone test suites like SRP-tests, NVMFTESTS, NVMe Multipath tests,
>> zone block device tests, into one central framework, which has made an overall block layer
>> testing and development much easier than having to configure and execute different
>> test cases for each kernel release for different subsystems such as FS, NVMe,
>> Zone Block devices, etc). 
>>
>> Here is the list of the existing test categories:-
>>
>> ├── block                                           28 Tests
>> ├── loop                                             07 Tests
>> ├── meta                                            12 Tests
>> ├── nbd                                              02 Tests
>> ├── nvme                                           28 Tests
>> ├── nvmeof-mp                                  12 Tests
>> ├── scsi                                              06 Tests
>> ├── srp                                               13 Tests
>> └── zbd                                              05 Tests
>> ---------------------------------------------------------------- 
>>            9 Categories                            ~110 Tests
>>
>> This project has gathered much attention and storage stack community is actively
>> participating and adding new test cases with different categories to the framework. 
>>
>> For storage track, we would like to propose a session dedicated to blktests. It is a great
>> opportunity for the storage developers to gather and have a discussion about:-
>>
>> 1. Current status of the blktests framework.
>> 2. Any new/missing features that we want to add in the blktests.
>> 3. Any new kernel features that could be used to make testing easier?
>> E.g. Implementing new features in the null_blk.c in order to have device
>> independent complete test coverage. (e.g. adding discard command for null_blk or any
>> other specific REQ_OP). Discussion about having any new tracepoint events in the block layer.
>> 4. Any new test cases/categories which are lacking in the blktests framework.
> 
> One thing I'd love to see is more hardware/driver specific tests. I'm
> sure Broadcom, Marvell, Huawei and all the others out there do have test
> suites for their HBA drivers but not a single one of these tests is
> publicly available.
> 
> We're also lacking tests for things like ioprio, persistent reservation,
> bcache and so on.

+1 for ioprio discussion. I mentioned my interest in discussing this in my
invite request. Having it as a topic would be great. Since we are in the middle
of blktest improvements for zoned devices, I can try to put together a proposal
as a discussion base.

> 
> Adding support for collecting gcov information after running a test case
> would also be awesome (this is missing in xfstests as well).
> 
> So I think a session on blktests can help us get the gap closed.
> 
> Byte,
> 	Johannes
> 


-- 
Damien Le Moal
Western Digital Research

Re: [LSF/MM TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Lee Duncan]

On 2/6/19 2:32 AM, Johannes Thumshirn wrote:
> On 06/02/2019 06:21, Chaitanya Kulkarni wrote:
>> Hi,
>>
>> Since discussion of the storage stack and device driver at the LSFMM 2017
>> (https://lwn.net/Articles/717699/),  Omar Sandoval introduced a new framework
>> "blktests" dedicated for Linux Kernel Block layer testing.
>> (https://lwn.net/Articles/722785/, https://github.com/osandov/blktests).
>>  
>> As Linux Kernel Block layer is central to the various file systems and underlying
>> low-level device drivers it is important to have a centralized testing framework and
>> make sure it grows with the latest block layer changed which are being added based
>> on the different device features from different device types
>> (e.g. NVMe devices with Zoned Namespace support).
>>
>> Since then blktests has grown and became go-to framework where we have integrated
>> different stand-alone test suites like SRP-tests, NVMFTESTS, NVMe Multipath tests,
>> zone block device tests, into one central framework, which has made an overall block layer
>> testing and development much easier than having to configure and execute different
>> test cases for each kernel release for different subsystems such as FS, NVMe,
>> Zone Block devices, etc). 
>>
>> Here is the list of the existing test categories:-
>>
>> ├── block                                           28 Tests
>> ├── loop                                             07 Tests
>> ├── meta                                            12 Tests
>> ├── nbd                                              02 Tests
>> ├── nvme                                           28 Tests
>> ├── nvmeof-mp                                  12 Tests
>> ├── scsi                                              06 Tests
>> ├── srp                                               13 Tests
>> └── zbd                                              05 Tests
>> ---------------------------------------------------------------- 
>>            9 Categories                            ~110 Tests
>>
>> This project has gathered much attention and storage stack community is actively
>> participating and adding new test cases with different categories to the framework. 
>>
>> For storage track, we would like to propose a session dedicated to blktests. It is a great
>> opportunity for the storage developers to gather and have a discussion about:-
>>
>> 1. Current status of the blktests framework.
>> 2. Any new/missing features that we want to add in the blktests.
>> 3. Any new kernel features that could be used to make testing easier?
>> E.g. Implementing new features in the null_blk.c in order to have device
>> independent complete test coverage. (e.g. adding discard command for null_blk or any
>> other specific REQ_OP). Discussion about having any new tracepoint events in the block layer.
>> 4. Any new test cases/categories which are lacking in the blktests framework.
> 
> One thing I'd love to see is more hardware/driver specific tests. I'm
> sure Broadcom, Marvell, Huawei and all the others out there do have test
> suites for their HBA drivers but not a single one of these tests is
> publicly available.

I said this same thing 3 (?) years ago, but then I was told to go out
and solve it. :) Dealing with multiple manufactures could be a full-time
job.

> 
> We're also lacking tests for things like ioprio, persistent reservation,
> bcache and so on.

I have a test suite for persistent reservations, but would love help in
adding it to this test suite.

> 
> Adding support for collecting gcov information after running a test case
> would also be awesome (this is missing in xfstests as well).
> 
> So I think a session on blktests can help us get the gap closed.
> 
> Byte,
> 	Johannes
> 

I also would love to attend such a session.
-- 
Lee Duncan

Re: [LSF/MM TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Bart Van Assche]

On Wed, 2019-02-06 at 05:21 +0000, Chaitanya Kulkarni wrote:
> For storage track, we would like to propose a session dedicated to blktests. It is a great
> opportunity for the storage developers to gather and have a discussion about:-
> 
> 1. Current status of the blktests framework.
> 2. Any new/missing features that we want to add in the blktests.
> 3. Any new kernel features that could be used to make testing easier?
> E.g. Implementing new features in the null_blk.c in order to have device
> independent complete test coverage. (e.g. adding discard command for null_blk or any
> other specific REQ_OP). Discussion about having any new tracepoint events in the block layer.
> 4. Any new test cases/categories which are lacking in the blktests framework.

Hi Chaitanya,

Thanks for having proposed this topic. I'd like to add a fifth item to the
agenda, namely blktests maintainership. The following could e.g. be discussed:
- How many maintainers should the blktests project have? A single maintainer
  or also one or more co-maintainers?
- Is it acceptable that patches get accepted in the blktests repository that
  break the continuous integration tests? If so, why do we even have continuous
  integration tests? See also "[PATCH] Unbreak the continuous integration build"
  (https://marc.info/?l=linux-block&m=154990323618159).
- How long should it take before a blktests maintainer provides feedback on
  blktests patches and pull requests? Is it considered acceptable that it takes
  more than four weeks to process a pull request that is in perfect shape? See
  e.g. https://github.com/osandov/blktests/pull/44.

Bart.

Re: [LSF/MM TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Omar Sandoval]

On Wed, Feb 13, 2019 at 10:11:14AM -0800, Bart Van Assche wrote:
> On Wed, 2019-02-06 at 05:21 +0000, Chaitanya Kulkarni wrote:
> > For storage track, we would like to propose a session dedicated to blktests. It is a great
> > opportunity for the storage developers to gather and have a discussion about:-
> > 
> > 1. Current status of the blktests framework.
> > 2. Any new/missing features that we want to add in the blktests.
> > 3. Any new kernel features that could be used to make testing easier?
> > E.g. Implementing new features in the null_blk.c in order to have device
> > independent complete test coverage. (e.g. adding discard command for null_blk or any
> > other specific REQ_OP). Discussion about having any new tracepoint events in the block layer.
> > 4. Any new test cases/categories which are lacking in the blktests framework.
> 
> Hi Chaitanya,
> 
> Thanks for having proposed this topic. I'd like to add a fifth item to the
> agenda, namely blktests maintainership. The following could e.g. be discussed:
> - How many maintainers should the blktests project have? A single maintainer
>   or also one or more co-maintainers?
> - Is it acceptable that patches get accepted in the blktests repository that
>   break the continuous integration tests? If so, why do we even have continuous
>   integration tests? See also "[PATCH] Unbreak the continuous integration build"
>   (https://marc.info/?l=linux-block&m=154990323618159).

To be honest, I've never used travis, so I don't even know where to find
the results. https://travis-ci.org/osandov/blktests doesn't point to
anything. Can we add a build status badge to the README like other
projects have?

> - How long should it take before a blktests maintainer provides feedback on
>   blktests patches and pull requests? Is it considered acceptable that it takes
>   more than four weeks to process a pull request that is in perfect shape? See
>   e.g. https://github.com/osandov/blktests/pull/44.

Nope, that's not acceptable, sorry about that :( We can talk about a
reasonable SLA for review and merging.

> Bart.

Re: [LSF/MM TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Bart Van Assche]

On Wed, 2019-02-13 at 10:43 -0800, Omar Sandoval wrote:
> On Wed, Feb 13, 2019 at 10:11:14AM -0800, Bart Van Assche wrote:
> > - Is it acceptable that patches get accepted in the blktests repository that
> >   break the continuous integration tests? If so, why do we even have continuous
> >   integration tests? See also "[PATCH] Unbreak the continuous integration build"
> >   (https://marc.info/?l=linux-block&m=154990323618159).
> 
> To be honest, I've never used travis, so I don't even know where to find
> the results. https://travis-ci.org/osandov/blktests doesn't point to
> anything. Can we add a build status badge to the README like other
> projects have?

Hi Omar,

What is a build status badge? Anyway, enabling Travis CI is easy:
* Navigate to https://travis-ci.org/ and click on "Sign in with github".
* In the left column, click on "+" (Add New Repository).
* For the blktests repository, enable continuous integration. This will cause a
  continuous integration test to be started after every git push and also every
  time a pull request is submitted. The rdma-core project uses Travis CI not only
  to compile-test pull requests but also to verify whether new code in pull
  requests passes building with sparse. This is useful for the rdma-core project
  since a lot of endianness conversions happen in that code and sparse can
  verify whether these conversions have been annotated correctly. See also
  https://github.com/linux-rdma/rdma-core.

Bart.

Re: [LSF/MM TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Omar Sandoval]

On Wed, Feb 13, 2019 at 10:54:04AM -0800, Bart Van Assche wrote:
> On Wed, 2019-02-13 at 10:43 -0800, Omar Sandoval wrote:
> > On Wed, Feb 13, 2019 at 10:11:14AM -0800, Bart Van Assche wrote:
> > > - Is it acceptable that patches get accepted in the blktests repository that
> > >   break the continuous integration tests? If so, why do we even have continuous
> > >   integration tests? See also "[PATCH] Unbreak the continuous integration build"
> > >   (https://marc.info/?l=linux-block&m=154990323618159).
> > 
> > To be honest, I've never used travis, so I don't even know where to find
> > the results. https://travis-ci.org/osandov/blktests doesn't point to
> > anything. Can we add a build status badge to the README like other
> > projects have?
> 
> Hi Omar,
> 
> What is a build status badge?

I just added it, see
https://github.com/osandov/blktests/commit/a61aa7fcce0bad9094b0e7646f3a8299c30afa6a

Anyway, enabling Travis CI is easy:
> * Navigate to https://travis-ci.org/ and click on "Sign in with github".
> * In the left column, click on "+" (Add New Repository).
> * For the blktests repository, enable continuous integration. This will cause a
>   continuous integration test to be started after every git push and also every
>   time a pull request is submitted. The rdma-core project uses Travis CI not only
>   to compile-test pull requests but also to verify whether new code in pull
>   requests passes building with sparse. This is useful for the rdma-core project
>   since a lot of endianness conversions happen in that code and sparse can
>   verify whether these conversions have been annotated correctly. See also
>   https://github.com/linux-rdma/rdma-core.

Thanks, I got it set up now.

Re: [LSF/MM TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Bart Van Assche]

On Wed, 2019-02-13 at 11:56 -0800, Omar Sandoval wrote:
> On Wed, Feb 13, 2019 at 10:54:04AM -0800, Bart Van Assche wrote:
> > What is a build status badge?
> 
> I just added it, see
> https://github.com/osandov/blktests/commit/a61aa7fcce0bad9094b0e7646f3a8299c30afa6a
> 
> Anyway, enabling Travis CI is easy:
> > * Navigate to https://travis-ci.org/ and click on "Sign in with github".
> > * In the left column, click on "+" (Add New Repository).
> > * For the blktests repository, enable continuous integration. This will cause a
> >   continuous integration test to be started after every git push and also every
> >   time a pull request is submitted. The rdma-core project uses Travis CI not only
> >   to compile-test pull requests but also to verify whether new code in pull
> >   requests passes building with sparse. This is useful for the rdma-core project
> >   since a lot of endianness conversions happen in that code and sparse can
> >   verify whether these conversions have been annotated correctly. See also
> >   https://github.com/linux-rdma/rdma-core.
> 
> Thanks, I got it set up now.

Thanks!

Bart.

Re: [LSF/MM TOPIC] : blktests: status, an expansion plan for the storage stack test framework

[Chaitanya Kulkarni]

On 2/13/19 10:11 AM, Bart Van Assche wrote:
> On Wed, 2019-02-06 at 05:21 +0000, Chaitanya Kulkarni wrote:
>> For storage track, we would like to propose a session dedicated to blktests. It is a great
>> opportunity for the storage developers to gather and have a discussion about:-
>>
>> 1. Current status of the blktests framework.
>> 2. Any new/missing features that we want to add in the blktests.
>> 3. Any new kernel features that could be used to make testing easier?
>> E.g. Implementing new features in the null_blk.c in order to have device
>> independent complete test coverage. (e.g. adding discard command for null_blk or any
>> other specific REQ_OP). Discussion about having any new tracepoint events in the block layer.
>> 4. Any new test cases/categories which are lacking in the blktests framework.
> 
> Hi Chaitanya,
> 
> Thanks for having proposed this topic. I'd like to add a fifth item to the
> agenda, namely blktests maintainership. The following could e.g. be discussed:
> - How many maintainers should the blktests project have? A single maintainer
>    or also one or more co-maintainers?
> - Is it acceptable that patches get accepted in the blktests repository that
>    break the continuous integration tests? If so, why do we even have continuous
>    integration tests? See also "[PATCH] Unbreak the continuous integration build"
>    (https://marc.info/?l=linux-block&m=154990323618159).
> - How long should it take before a blktests maintainer provides feedback on
>    blktests patches and pull requests? Is it considered acceptable that it takes
>    more than four weeks to process a pull request that is in perfect shape? See
>    e.g. https://github.com/osandov/blktests/pull/44.
> 
> Bart.
> 

Thanks Bart for the suggestions, it will be great to have these topics 
for the overall discussion, it can help us to have a concrete plan 
moving forward.