[LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Chaitanya Kulkarni]

Hi all,

   Since the discussion at the LSFMM 2017 [1], Omar Sandoval introduced 
the new
   framework "blktests" dedicated for Linux Kernel Block layer testing.

   Blktests serves as the centralized testing framework. It has grown 
with the
   latest block layer changes and successfully integrated various 
stand-alone
   test suites like SRP-tests, NVMFTESTS, NVMe Multipath tests, zone 
block device
   tests. This integration has significantly simplified the process of 
block layer
   testing and development, eliminating the need to configure and 
execute test
   cases for each kernel release.

   The storage stack community is actively engaged, contributing and 
adding new
   test cases across diverse categories to the framework. Since the 
beginning, we
   are consistently finding bugs proactively with the help of blktests 
testcases.

   Below is a summary of the existing test categories and their test 
cases as of
   February 2026.

   block        :  41
   blktrace     :   2
   dm           :   3
   loop         :  11
   md           :   4
   meta         :  24
   nbd          :   4
   nvme         :  59
   rnbd         :   2
   scsi         :  10
   srp          :  15
   throtl       :   7
   ublk         :   6
   zbd          :  14
   ------------------

   14 Categories   : 202 Tests

   For the storage track at LSFMMBPF2026, I propose a session dedicated to
   blktests to discuss expansion plan and CI integration progress.

-ck

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Daniel Wagner]

On Wed, Feb 11, 2026 at 08:35:30PM +0000, Chaitanya Kulkarni wrote:
>    For the storage track at LSFMMBPF2026, I propose a session dedicated to
>    blktests to discuss expansion plan and CI integration progress.

Thanks for proposing this topic.

Just a few random topics which come to mind we could discuss:

- blktests has gain a bit of traction and some folks run on regular
  basis these tests. Can we gather feedback from them, what is working
  good, what is not? Are there feature wishes?
- Do we need some sort of configuration tool which allows to setup a
  config? I'd still have a TODO to provide a config example with all
  knobs which influence blktests, but I wonder if we should go a step
  further here, e.g. something like kdevops has?
- Which area do we lack tests? Should we just add an initial simple
  tests for the missing areas, so the basic infra is there and thus
  lowering the bar for adding new tests?
- The recent addition of kmemleak shows it's a great idea to enable more
  of the kernel test infrastructure when running the tests. Are there
  more such things we could/should enable?
- I would like to hear from Shin'ichiro if he is happy how things
  are going? :)

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Johannes Thumshirn]

On 2/12/26 8:52 AM, Daniel Wagner wrote:
> On Wed, Feb 11, 2026 at 08:35:30PM +0000, Chaitanya Kulkarni wrote:
>>     For the storage track at LSFMMBPF2026, I propose a session dedicated to
>>     blktests to discuss expansion plan and CI integration progress.
>>
>> - The recent addition of kmemleak shows it's a great idea to enable more
>>    of the kernel test infrastructure when running the tests. Are there
>>    more such things we could/should enable?

One thing that comes to my mind (and that I always wanted to do for 
fstests but didn't for $REASONS) is adding per-test code coverage 
information.

Something like the per test kmemleak and dmesg output. This way one can 
check that the test case is actually executing the code it intended to 
test. Also its a good way to see which areas of code lack proper testing.

Just my $.05.

Byte,

     Johannes

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Bart Van Assche]

On 2/11/26 11:57 PM, Johannes Thumshirn wrote:
> One thing that comes to my mind (and that I always wanted to do for
> fstests but didn't for $REASONS) is adding per-test code coverage
> information.

Code coverage information is useful but it's important to keep in mind
that 100% code coverage (which is very hard to achieve) does not
guarantee code correctness. There are many state machines in the block
layer and also in block drivers. Code coverage information does not
reveal what percentage of the states of state machines has been
triggered.

Bart.

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[James Bottomley]

On Fri, 2026-02-13 at 09:30 -0800, Bart Van Assche wrote:
> On 2/11/26 11:57 PM, Johannes Thumshirn wrote:
> > One thing that comes to my mind (and that I always wanted to do for
> > fstests but didn't for $REASONS) is adding per-test code coverage
> > information.
> 
> Code coverage information is useful but it's important to keep in
> mind that 100% code coverage (which is very hard to achieve) does not
> guarantee code correctness. There are many state machines in the
> block layer and also in block drivers. Code coverage information does
> not reveal what percentage of the states of state machines has been
> triggered.

This is not an either/or.  Usually our functional tests try to cover
the state machine (although often requiring error injection).  However,
a lot of our bugs hide in error legs and code coverage at least assures
us we've looked for them.

Regards,

James

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Shinichiro Kawasaki]

On Feb 12, 2026 / 08:52, Daniel Wagner wrote:
> On Wed, Feb 11, 2026 at 08:35:30PM +0000, Chaitanya Kulkarni wrote:
> >    For the storage track at LSFMMBPF2026, I propose a session dedicated to
> >    blktests to discuss expansion plan and CI integration progress.
> 
> Thanks for proposing this topic.

Chaitanya, my thank also goes to you.

> Just a few random topics which come to mind we could discuss:
> 
> - blktests has gain a bit of traction and some folks run on regular
>   basis these tests. Can we gather feedback from them, what is working
>   good, what is not? Are there feature wishes?

Good topic, I also would like to hear about it.

FYI, from the past LSFMM sessions and hallway talks, major feedbacks I had
received are these two:

 1. blktests CI infra looks missing (other than CKI by Redhat)
    -> Some activities are ongoing to start blktests CI service.
       I hope the status are shared at the session.

 2. blktests are rather difficult to start using for some new users
    -> I think config example is demanded, so that new users can
       just copy it to start the first run, and understand the
       config options easily.

> - Do we need some sort of configuration tool which allows to setup a
>   config? I'd still have a TODO to provide a config example with all
>   knobs which influence blktests, but I wonder if we should go a step
>   further here, e.g. something like kdevops has?

Do you mean the "make menuconfig" style? Most of the blktests users are
familiar with menuconfig, so that would be an idea. If users really want
it, we can think of it. IMO, blktests still do not have so many options,
then config.example would be simpler and more appropriate, probably.

> - Which area do we lack tests? Should we just add an initial simple
>   tests for the missing areas, so the basic infra is there and thus
>   lowering the bar for adding new tests?

To identify the uncovered area, I think code coverage will be useful. A few
years ago, I measured it and shared in LSFMM, but that measurement was done for
each source tree directory. The coverage ratio by source file will be more
helpful to identify the missing area. I don't have time slot to measure it,
so if anyone can do it and share the result, it will be appreciated. Once we
know the missing areas, it sounds a good idea to add initial samples for each
of the areas.

> - The recent addition of kmemleak shows it's a great idea to enable more
>   of the kernel test infrastructure when running the tests.

Completely agreed.

>   Are there more such things we could/should enable?

I'm also interested in this question :)

> - I would like to hear from Shin'ichiro if he is happy how things
>   are going? :)

More importantly, I would like to listen to voices from storage sub-system
developers to see if they are happy or not, especially the maintainers.

From my view, blktests keep on finding kernel bugs. I think it demonstrates the
value of this community effort, and I'm happy about it. Said that, I find what
blktests can improve more, of course. Here I share the list of improvement
opportunities from my view point (I already mentioned the first three items).

 1. We can have more CI infra to make the most of blktests
 2. We can add config examples to help new users
 3. We can measure code coverage to identify missing test areas
 4. Long standing failures make test result reports dirty
    - I feel lockdep WARNs are tend to be left unfixed rather long period.
      How can we gather effort to fix them?
 5. We can refactor and clean up blktests framework for ease of maintainance
      (e.g. trap handling)
 6. Some users run blktests with built-in kernel modules, which makes a number
    of test cases skipped. We can add more built-in kernel modules support to
    expand test coverage for such use case.

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Haris Iqbal]

On Fri, Feb 13, 2026 at 12:25 PM Shinichiro Kawasaki
<shinichiro.kawasaki@wdc.com> wrote:
>
> On Feb 12, 2026 / 08:52, Daniel Wagner wrote:
> > On Wed, Feb 11, 2026 at 08:35:30PM +0000, Chaitanya Kulkarni wrote:
> > >    For the storage track at LSFMMBPF2026, I propose a session dedicated to
> > >    blktests to discuss expansion plan and CI integration progress.

I am interested in this topic.

> >
> > Thanks for proposing this topic.
>
> Chaitanya, my thank also goes to you.
>
> > Just a few random topics which come to mind we could discuss:
> >
> > - blktests has gain a bit of traction and some folks run on regular
> >   basis these tests. Can we gather feedback from them, what is working
> >   good, what is not? Are there feature wishes?
>
> Good topic, I also would like to hear about it.
>
> FYI, from the past LSFMM sessions and hallway talks, major feedbacks I had
> received are these two:
>
>  1. blktests CI infra looks missing (other than CKI by Redhat)
>     -> Some activities are ongoing to start blktests CI service.
>        I hope the status are shared at the session.
>
>  2. blktests are rather difficult to start using for some new users
>     -> I think config example is demanded, so that new users can
>        just copy it to start the first run, and understand the
>        config options easily.

+1 to this.

>
> > - Do we need some sort of configuration tool which allows to setup a
> >   config? I'd still have a TODO to provide a config example with all
> >   knobs which influence blktests, but I wonder if we should go a step
> >   further here, e.g. something like kdevops has?
>
> Do you mean the "make menuconfig" style? Most of the blktests users are
> familiar with menuconfig, so that would be an idea. If users really want
> it, we can think of it. IMO, blktests still do not have so many options,
> then config.example would be simpler and more appropriate, probably.
>
> > - Which area do we lack tests? Should we just add an initial simple
> >   tests for the missing areas, so the basic infra is there and thus
> >   lowering the bar for adding new tests?
>
> To identify the uncovered area, I think code coverage will be useful. A few
> years ago, I measured it and shared in LSFMM, but that measurement was done for
> each source tree directory. The coverage ratio by source file will be more
> helpful to identify the missing area. I don't have time slot to measure it,
> so if anyone can do it and share the result, it will be appreciated. Once we
> know the missing areas, it sounds a good idea to add initial samples for each
> of the areas.
>
> > - The recent addition of kmemleak shows it's a great idea to enable more
> >   of the kernel test infrastructure when running the tests.
>
> Completely agreed.
>
> >   Are there more such things we could/should enable?
>
> I'm also interested in this question :)
>
> > - I would like to hear from Shin'ichiro if he is happy how things
> >   are going? :)
>
> More importantly, I would like to listen to voices from storage sub-system
> developers to see if they are happy or not, especially the maintainers.

I like the idea of blktests.
We have internal tests which we run for RNBD (and RTRS).
And I plan to port the RNBD ones to blktests.

>
> From my view, blktests keep on finding kernel bugs. I think it demonstrates the
> value of this community effort, and I'm happy about it. Said that, I find what
> blktests can improve more, of course. Here I share the list of improvement
> opportunities from my view point (I already mentioned the first three items).
>
>  1. We can have more CI infra to make the most of blktests
>  2. We can add config examples to help new users
>  3. We can measure code coverage to identify missing test areas
>  4. Long standing failures make test result reports dirty
>     - I feel lockdep WARNs are tend to be left unfixed rather long period.
>       How can we gather effort to fix them?
>  5. We can refactor and clean up blktests framework for ease of maintainance
>       (e.g. trap handling)
>  6. Some users run blktests with built-in kernel modules, which makes a number
>     of test cases skipped. We can add more built-in kernel modules support to
>     expand test coverage for such use case.

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Nilay Shroff]

On 2/13/26 4:53 PM, Shinichiro Kawasaki wrote:
> On Feb 12, 2026 / 08:52, Daniel Wagner wrote:
>> On Wed, Feb 11, 2026 at 08:35:30PM +0000, Chaitanya Kulkarni wrote:
>>>    For the storage track at LSFMMBPF2026, I propose a session dedicated to
>>>    blktests to discuss expansion plan and CI integration progress.
>>
>> Thanks for proposing this topic.
> 
> Chaitanya, my thank also goes to you.
> 
Yes thanks for proposing this!

>> Just a few random topics which come to mind we could discuss:
>>
>> - blktests has gain a bit of traction and some folks run on regular
>>   basis these tests. Can we gather feedback from them, what is working
>>   good, what is not? Are there feature wishes?
> 
> Good topic, I also would like to hear about it.
> 
One improvement I’d like to highlight is related to how blktests are executed
today. So far, we’ve been running blktests serially, but if it's possible to 
run tests in parallel to improve test turnaround time and make large-scale or
CI-based testing more efficient? For instance, adding parallel_safe Tags: Marking tests
that don't modify global kernel state so they can be safely offloaded to parallel
workers. Marking parallel_safe tags would allow the runner to distinguish:

Safe Tests: Tests that only perform I/O on a specific, non-shared device or 
check static kernel parameters.

Unsafe Tests: Tests that reload kernel modules, modify global /sys or /proc entries,
or require exclusive access to specific hardware addresses.

Yes adding parallel execution support shall require framework/design changes. 

> FYI, from the past LSFMM sessions and hallway talks, major feedbacks I had
> received are these two:
> 
>  1. blktests CI infra looks missing (other than CKI by Redhat)
>     -> Some activities are ongoing to start blktests CI service.
>        I hope the status are shared at the session.
> 
>  2. blktests are rather difficult to start using for some new users
>     -> I think config example is demanded, so that new users can
>        just copy it to start the first run, and understand the
>        config options easily.
> 
>> - Do we need some sort of configuration tool which allows to setup a
>>   config? I'd still have a TODO to provide a config example with all
>>   knobs which influence blktests, but I wonder if we should go a step
>>   further here, e.g. something like kdevops has?
> 
> Do you mean the "make menuconfig" style? Most of the blktests users are
> familiar with menuconfig, so that would be an idea. If users really want
> it, we can think of it. IMO, blktests still do not have so many options,
> then config.example would be simpler and more appropriate, probably.
> 
>> - Which area do we lack tests? Should we just add an initial simple
>>   tests for the missing areas, so the basic infra is there and thus
>>   lowering the bar for adding new tests?
> 
> To identify the uncovered area, I think code coverage will be useful. A few
> years ago, I measured it and shared in LSFMM, but that measurement was done for
> each source tree directory. The coverage ratio by source file will be more
> helpful to identify the missing area. I don't have time slot to measure it,
> so if anyone can do it and share the result, it will be appreciated. Once we
> know the missing areas, it sounds a good idea to add initial samples for each
> of the areas.
> 
>> - The recent addition of kmemleak shows it's a great idea to enable more
>>   of the kernel test infrastructure when running the tests.
> 
> Completely agreed.
> 
>>   Are there more such things we could/should enable?
> 
> I'm also interested in this question :)
> 
>> - I would like to hear from Shin'ichiro if he is happy how things
>>   are going? :)
> 
> More importantly, I would like to listen to voices from storage sub-system
> developers to see if they are happy or not, especially the maintainers.
> 
> From my view, blktests keep on finding kernel bugs. I think it demonstrates the
> value of this community effort, and I'm happy about it. Said that, I find what
> blktests can improve more, of course. Here I share the list of improvement
> opportunities from my view point (I already mentioned the first three items).
> 
>  1. We can have more CI infra to make the most of blktests
>  2. We can add config examples to help new users
>  3. We can measure code coverage to identify missing test areas
>  4. Long standing failures make test result reports dirty
>     - I feel lockdep WARNs are tend to be left unfixed rather long period.
>       How can we gather effort to fix them?

I agree regarding lockdep; recently we did see quite a few lockdep splats.
That said, I believe the number has dropped significantly and only a small
set remains. From what I can tell, most of the outstanding lockdep issues
are related to fs-reclaim paths recursing into the block layer while the
queue is frozen. We should be able to resolve most of these soon, or at
least before the conference. If anything is still outstanding after that,
we can discuss it during the conference and work toward addressing it as
quickly as possible.

>  5. We can refactor and clean up blktests framework for ease of maintainance
>       (e.g. trap handling)
>  6. Some users run blktests with built-in kernel modules, which makes a number
>     of test cases skipped. We can add more built-in kernel modules support to
>     expand test coverage for such use case.

Thanks,
--Nilay

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Shin'ichiro Kawasaki]

On Feb 16, 2026 / 00:08, Nilay Shroff wrote:
> 
> 
> On 2/13/26 4:53 PM, Shinichiro Kawasaki wrote:
> > On Feb 12, 2026 / 08:52, Daniel Wagner wrote:
> >> On Wed, Feb 11, 2026 at 08:35:30PM +0000, Chaitanya Kulkarni wrote:
> >>>    For the storage track at LSFMMBPF2026, I propose a session dedicated to
> >>>    blktests to discuss expansion plan and CI integration progress.
> >>
> >> Thanks for proposing this topic.
> > 
> > Chaitanya, my thank also goes to you.
> > 
> Yes thanks for proposing this!
> 
> >> Just a few random topics which come to mind we could discuss:
> >>
> >> - blktests has gain a bit of traction and some folks run on regular
> >>   basis these tests. Can we gather feedback from them, what is working
> >>   good, what is not? Are there feature wishes?
> > 
> > Good topic, I also would like to hear about it.
> > 
> One improvement I’d like to highlight is related to how blktests are executed
> today. So far, we’ve been running blktests serially, but if it's possible to 
> run tests in parallel to improve test turnaround time and make large-scale or
> CI-based testing more efficient? For instance, adding parallel_safe Tags: Marking tests
> that don't modify global kernel state so they can be safely offloaded to parallel
> workers. Marking parallel_safe tags would allow the runner to distinguish:
> 
> Safe Tests: Tests that only perform I/O on a specific, non-shared device or 
> check static kernel parameters.
> 
> Unsafe Tests: Tests that reload kernel modules, modify global /sys or /proc entries,
> or require exclusive access to specific hardware addresses.
> 
> Yes adding parallel execution support shall require framework/design changes.

Hi Nilay, thanks for the idea. I understand that shorter test time will make CI
cycles faster and improve the development efficiency.

Said that, the safe/unsafe testing idea may not be enough. I think majority of
test case does kernel module set up using null_blk, scsi_debug, or nvme target
drivers. Then I foresee the majority of the test cases will be "unsafe", and
cannot be run in parallel.

Also, parallel runs on single system will affect dmesg or kmemleak checking.
We cannot tell which run caused a dmesg message or a memory leak.

For the runtime reduction by parallel runs, I guess blktests run on VMs might be
the good approach as Haris pointed out. Anyway, this topic will need more
discussion.

[...]

> >  4. Long standing failures make test result reports dirty
> >     - I feel lockdep WARNs are tend to be left unfixed rather long period.
> >       How can we gather effort to fix them?
> 
> I agree regarding lockdep; recently we did see quite a few lockdep splats.
> That said, I believe the number has dropped significantly and only a small
> set remains. From what I can tell, most of the outstanding lockdep issues
> are related to fs-reclaim paths recursing into the block layer while the
> queue is frozen. We should be able to resolve most of these soon, or at
> least before the conference. If anything is still outstanding after that,
> we can discuss it during the conference and work toward addressing it as
> quickly as possible.

Taking this chance, I'd like to express my appreciation for the effort to
resolve the lockdep issues. It is great that a number of lockdeps are already
fixed. Said that, two lockdep issues are still observed with v7.0 kernel at
nvme/005 and nbd/002 [1]. I would like to gather attentions to the failures.

[1] https://lore.kernel.org/linux-block/ynmi72x5wt5ooljjafebhcarit3pvu6axkslqenikb2p5txe57@ldytqa2t4i2x/

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Nilay Shroff]

On 4/21/26 11:49 AM, Shin'ichiro Kawasaki wrote:
> On Feb 16, 2026 / 00:08, Nilay Shroff wrote:
>>
>>
>> On 2/13/26 4:53 PM, Shinichiro Kawasaki wrote:
>>> On Feb 12, 2026 / 08:52, Daniel Wagner wrote:
>>>> On Wed, Feb 11, 2026 at 08:35:30PM +0000, Chaitanya Kulkarni wrote:
>>>>>     For the storage track at LSFMMBPF2026, I propose a session dedicated to
>>>>>     blktests to discuss expansion plan and CI integration progress.
>>>>
>>>> Thanks for proposing this topic.
>>>
>>> Chaitanya, my thank also goes to you.
>>>
>> Yes thanks for proposing this!
>>
>>>> Just a few random topics which come to mind we could discuss:
>>>>
>>>> - blktests has gain a bit of traction and some folks run on regular
>>>>    basis these tests. Can we gather feedback from them, what is working
>>>>    good, what is not? Are there feature wishes?
>>>
>>> Good topic, I also would like to hear about it.
>>>
>> One improvement I’d like to highlight is related to how blktests are executed
>> today. So far, we’ve been running blktests serially, but if it's possible to
>> run tests in parallel to improve test turnaround time and make large-scale or
>> CI-based testing more efficient? For instance, adding parallel_safe Tags: Marking tests
>> that don't modify global kernel state so they can be safely offloaded to parallel
>> workers. Marking parallel_safe tags would allow the runner to distinguish:
>>
>> Safe Tests: Tests that only perform I/O on a specific, non-shared device or
>> check static kernel parameters.
>>
>> Unsafe Tests: Tests that reload kernel modules, modify global /sys or /proc entries,
>> or require exclusive access to specific hardware addresses.
>>
>> Yes adding parallel execution support shall require framework/design changes.
> 
> Hi Nilay, thanks for the idea. I understand that shorter test time will make CI
> cycles faster and improve the development efficiency.
> 
> Said that, the safe/unsafe testing idea may not be enough. I think majority of
> test case does kernel module set up using null_blk, scsi_debug, or nvme target
> drivers. Then I foresee the majority of the test cases will be "unsafe", and
> cannot be run in parallel.
> 
> Also, parallel runs on single system will affect dmesg or kmemleak checking.
> We cannot tell which run caused a dmesg message or a memory leak.
> 
> For the runtime reduction by parallel runs, I guess blktests run on VMs might be
> the good approach as Haris pointed out. Anyway, this topic will need more
> discussion.
> 
> [...]

Alright, see if we may discuss this during LSFMM.

> 
>>>   4. Long standing failures make test result reports dirty
>>>      - I feel lockdep WARNs are tend to be left unfixed rather long period.
>>>        How can we gather effort to fix them?
>>
>> I agree regarding lockdep; recently we did see quite a few lockdep splats.
>> That said, I believe the number has dropped significantly and only a small
>> set remains. From what I can tell, most of the outstanding lockdep issues
>> are related to fs-reclaim paths recursing into the block layer while the
>> queue is frozen. We should be able to resolve most of these soon, or at
>> least before the conference. If anything is still outstanding after that,
>> we can discuss it during the conference and work toward addressing it as
>> quickly as possible.
> 
> Taking this chance, I'd like to express my appreciation for the effort to
> resolve the lockdep issues. It is great that a number of lockdeps are already
> fixed. Said that, two lockdep issues are still observed with v7.0 kernel at
> nvme/005 and nbd/002 [1]. I would like to gather attentions to the failures.
> 
> [1] https://lore.kernel.org/linux-block/ynmi72x5wt5ooljjafebhcarit3pvu6axkslqenikb2p5txe57@ldytqa2t4i2x/
> 
I think nvme/005 and nbd/002 failures shall be addressed with this
patch: https://lore.kernel.org/all/20260413171628.6204-1-kch@nvidia.com/

It's currently applied to nvme-7.1 and not there yet to mainline kernel.

Thanks,
--Nilay

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Daniel Wagner]

On Thu, Apr 23, 2026 at 01:35:47PM +0530, Nilay Shroff wrote:
> > Taking this chance, I'd like to express my appreciation for the effort to
> > resolve the lockdep issues. It is great that a number of lockdeps are already
> > fixed. Said that, two lockdep issues are still observed with v7.0 kernel at
> > nvme/005 and nbd/002 [1]. I would like to gather attentions to the failures.
> > 
> > [1] https://lore.kernel.org/linux-block/ynmi72x5wt5ooljjafebhcarit3pvu6axkslqenikb2p5txe57@ldytqa2t4i2x/
> > 
> I think nvme/005 and nbd/002 failures shall be addressed with this
> patch: https://lore.kernel.org/all/20260413171628.6204-1-kch@nvidia.com/
> 
> It's currently applied to nvme-7.1 and not there yet to mainline kernel.

FWIW, the nightly CI runs for nvme-cli with blktests/master and
linux/master are all green except for the above mentioned nvme/005 tcp
case. Thanks everyone for helping out!

https://github.com/linux-nvme/nvme-cli/actions/runs/24816292836

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Shin'ichiro Kawasaki]

On Apr 23, 2026 / 13:35, Nilay Shroff wrote:
> On 4/21/26 11:49 AM, Shin'ichiro Kawasaki wrote:
> > On Feb 16, 2026 / 00:08, Nilay Shroff wrote:
> > > On 2/13/26 4:53 PM, Shinichiro Kawasaki wrote:
[...]
> > > >   4. Long standing failures make test result reports dirty
> > > >      - I feel lockdep WARNs are tend to be left unfixed rather long period.
> > > >        How can we gather effort to fix them?
> > > 
> > > I agree regarding lockdep; recently we did see quite a few lockdep splats.
> > > That said, I believe the number has dropped significantly and only a small
> > > set remains. From what I can tell, most of the outstanding lockdep issues
> > > are related to fs-reclaim paths recursing into the block layer while the
> > > queue is frozen. We should be able to resolve most of these soon, or at
> > > least before the conference. If anything is still outstanding after that,
> > > we can discuss it during the conference and work toward addressing it as
> > > quickly as possible.
> > 
> > Taking this chance, I'd like to express my appreciation for the effort to
> > resolve the lockdep issues. It is great that a number of lockdeps are already
> > fixed. Said that, two lockdep issues are still observed with v7.0 kernel at
> > nvme/005 and nbd/002 [1]. I would like to gather attentions to the failures.
> > 
> > [1] https://lore.kernel.org/linux-block/ynmi72x5wt5ooljjafebhcarit3pvu6axkslqenikb2p5txe57@ldytqa2t4i2x/
> > 
> I think nvme/005 and nbd/002 failures shall be addressed with this
> patch: https://lore.kernel.org/all/20260413171628.6204-1-kch@nvidia.com/
> 
> It's currently applied to nvme-7.1 and not there yet to mainline kernel.

Ah, I missed that patch. Thanks a lot, Chaitanya!

Today, I applied the nvme fix patch on top of v7.1-rc1, and ran nvme/005 with
tcp transport. Unfortunately, I still observe the lockdep splat for
&q->elevator_lock, &q->q_usage_counter(io) and set->srcu [*]. This time, the
call chain looks a bit different (cpu_hotplug_lock is involved?).

I also still observe the nbd/002 failure. The nvme fix patch does not affect
nbd, then I think it is expected that the nbd/002 failure is still there.


[*]

Apr 27 20:32:07 testnode1 unknown: run blktests nvme/005 at 2026-04-27 20:32:07
Apr 27 20:32:08 testnode1 kernel: loop0: detected capacity change from 0 to 2097152
Apr 27 20:32:08 testnode1 kernel: nvmet: adding nsid 1 to subsystem blktests-subsystem-1
Apr 27 20:32:08 testnode1 kernel: nvmet_tcp: enabling port 0 (127.0.0.1:4420)
Apr 27 20:32:08 testnode1 kernel: nvmet: Created nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
Apr 27 20:32:08 testnode1 kernel: nvme nvme5: creating 4 I/O queues.
Apr 27 20:32:08 testnode1 kernel: nvme nvme5: mapped 4/0/0 default/read/poll queues.
Apr 27 20:32:08 testnode1 kernel: nvme nvme5: new ctrl: NQN "blktests-subsystem-1", addr 127.0.0.1:4420, hostnqn: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
Apr 27 20:32:08 testnode1 kernel: nvmet: Created nvm controller 2 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
Apr 27 20:32:08 testnode1 kernel: nvme nvme5: creating 4 I/O queues.
Apr 27 20:32:08 testnode1 kernel: nvme nvme5: mapped 4/0/0 default/read/poll queues.
Apr 27 20:32:08 testnode1 kernel: nvme nvme5: Removing ctrl: NQN "blktests-subsystem-1"
Apr 27 20:32:08 testnode1 kernel: 
Apr 27 20:32:08 testnode1 kernel: ======================================================
Apr 27 20:32:08 testnode1 kernel: WARNING: possible circular locking dependency detected
Apr 27 20:32:08 testnode1 kernel: 7.1.0-rc1+ #3 Not tainted
Apr 27 20:32:08 testnode1 kernel: ------------------------------------------------------
Apr 27 20:32:08 testnode1 kernel: nvme/1171 is trying to acquire lock:
Apr 27 20:32:08 testnode1 kernel: ffff888121e8bb98 (set->srcu){.+.+}-{0:0}, at: __synchronize_srcu+0x21/0x2b0
Apr 27 20:32:08 testnode1 kernel: 
                                  but task is already holding lock:
Apr 27 20:32:08 testnode1 kernel: ffff88812ab7bd68 (&q->elevator_lock){+.+.}-{4:4}, at: elevator_change+0x188/0x4f0
Apr 27 20:32:08 testnode1 kernel: 
                                  which lock already depends on the new lock.
Apr 27 20:32:08 testnode1 kernel: 
                                  the existing dependency chain (in reverse order) is:
Apr 27 20:32:08 testnode1 kernel: 
                                  -> #5 (&q->elevator_lock){+.+.}-{4:4}:
Apr 27 20:32:08 testnode1 kernel:        __mutex_lock+0x1ae/0x2600
Apr 27 20:32:08 testnode1 kernel:        elevator_change+0x188/0x4f0
Apr 27 20:32:08 testnode1 kernel:        elv_iosched_store+0x308/0x390
Apr 27 20:32:08 testnode1 kernel:        queue_attr_store+0x23b/0x360
Apr 27 20:32:08 testnode1 kernel:        kernfs_fop_write_iter+0x3d6/0x5e0
Apr 27 20:32:08 testnode1 kernel:        vfs_write+0x52c/0xf80
Apr 27 20:32:08 testnode1 kernel:        ksys_write+0xfb/0x200
Apr 27 20:32:08 testnode1 kernel:        do_syscall_64+0xdd/0x14c0
Apr 27 20:32:08 testnode1 kernel:        entry_SYSCALL_64_after_hwframe+0x76/0x7e
Apr 27 20:32:08 testnode1 kernel: 
                                  -> #4 (&q->q_usage_counter(io)){++++}-{0:0}:
Apr 27 20:32:08 testnode1 kernel:        blk_alloc_queue+0x5b3/0x730
Apr 27 20:32:08 testnode1 kernel:        blk_mq_alloc_queue+0x13f/0x250
Apr 27 20:32:08 testnode1 kernel:        scsi_alloc_sdev+0x84e/0xca0
Apr 27 20:32:08 testnode1 kernel:        scsi_probe_and_add_lun+0x63f/0xc30
Apr 27 20:32:08 testnode1 kernel:        __scsi_add_device+0x1be/0x1f0
Apr 27 20:32:08 testnode1 kernel:        ata_scsi_scan_host+0x139/0x3a0
Apr 27 20:32:08 testnode1 kernel:        async_run_entry_fn+0x93/0x550
Apr 27 20:32:08 testnode1 kernel:        process_one_work+0x8b4/0x1640
Apr 27 20:32:08 testnode1 kernel:        worker_thread+0x606/0xff0
Apr 27 20:32:08 testnode1 kernel:        kthread+0x368/0x460
Apr 27 20:32:08 testnode1 kernel:        ret_from_fork+0x653/0x9d0
Apr 27 20:32:08 testnode1 kernel:        ret_from_fork_asm+0x1a/0x30
Apr 27 20:32:08 testnode1 kernel: 
                                  -> #3 (fs_reclaim){+.+.}-{0:0}:
Apr 27 20:32:08 testnode1 kernel:        fs_reclaim_acquire+0xd5/0x120
Apr 27 20:32:08 testnode1 kernel:        __kmalloc_cache_node_noprof+0x51/0x740
Apr 27 20:32:08 testnode1 kernel:        create_worker+0xfb/0x710
Apr 27 20:32:08 testnode1 kernel:        workqueue_prepare_cpu+0x87/0xe0
Apr 27 20:32:08 testnode1 kernel:        cpuhp_invoke_callback+0x2a7/0x1230
Apr 27 20:32:08 testnode1 kernel:        __cpuhp_invoke_callback_range+0xbd/0x1f0
Apr 27 20:32:08 testnode1 kernel:        _cpu_up+0x2ec/0x700
Apr 27 20:32:08 testnode1 kernel:        cpu_up+0x111/0x190
Apr 27 20:32:08 testnode1 kernel:        cpuhp_bringup_mask+0xd3/0x110
Apr 27 20:32:08 testnode1 kernel:        bringup_nonboot_cpus+0x139/0x170
Apr 27 20:32:08 testnode1 kernel:        smp_init+0x27/0xe0
Apr 27 20:32:08 testnode1 kernel:        kernel_init_freeable+0x445/0x6f0
Apr 27 20:32:08 testnode1 kernel:        kernel_init+0x18/0x150
Apr 27 20:32:08 testnode1 kernel:        ret_from_fork+0x653/0x9d0
Apr 27 20:32:08 testnode1 kernel:        ret_from_fork_asm+0x1a/0x30
Apr 27 20:32:08 testnode1 kernel: 
                                  -> #2 (cpu_hotplug_lock){++++}-{0:0}:
Apr 27 20:32:08 testnode1 kernel:        cpus_read_lock+0x3c/0xe0
Apr 27 20:32:08 testnode1 kernel:        static_key_disable+0x12/0x30
Apr 27 20:32:08 testnode1 kernel:        __inet_hash_connect+0x10f7/0x1a50
Apr 27 20:32:08 testnode1 kernel:        tcp_v4_connect+0xcb0/0x18b0
Apr 27 20:32:08 testnode1 kernel:        __inet_stream_connect+0x349/0xf00
Apr 27 20:32:08 testnode1 kernel:        inet_stream_connect+0x55/0xb0
Apr 27 20:32:08 testnode1 kernel:        kernel_connect+0xdf/0x140
Apr 27 20:32:08 testnode1 kernel:        nvme_tcp_alloc_queue+0xa48/0x1b60 [nvme_tcp]
Apr 27 20:32:08 testnode1 kernel:        nvme_tcp_alloc_admin_queue+0xff/0x440 [nvme_tcp]
Apr 27 20:32:08 testnode1 kernel:        nvme_tcp_setup_ctrl+0x8a/0x830 [nvme_tcp]
Apr 27 20:32:08 testnode1 kernel:        nvme_tcp_create_ctrl+0x834/0xb90 [nvme_tcp]
Apr 27 20:32:08 testnode1 kernel:        nvmf_dev_write+0x3e3/0x800 [nvme_fabrics]
Apr 27 20:32:08 testnode1 kernel:        vfs_write+0x1cc/0xf80
Apr 27 20:32:08 testnode1 kernel:        ksys_write+0xfb/0x200
Apr 27 20:32:08 testnode1 kernel:        do_syscall_64+0xdd/0x14c0
Apr 27 20:32:08 testnode1 kernel:        entry_SYSCALL_64_after_hwframe+0x76/0x7e
Apr 27 20:32:08 testnode1 kernel: 
                                  -> #1 (sk_lock-AF_INET-NVME){+.+.}-{0:0}:
Apr 27 20:32:08 testnode1 kernel:        lock_sock_nested+0x32/0xf0
Apr 27 20:32:08 testnode1 kernel:        tcp_sendmsg+0x1c/0x50
Apr 27 20:32:08 testnode1 kernel:        sock_sendmsg+0x2bd/0x370
Apr 27 20:32:08 testnode1 kernel:        nvme_tcp_try_send_cmd_pdu+0x57f/0xbd0 [nvme_tcp]
Apr 27 20:32:08 testnode1 kernel:        nvme_tcp_try_send+0x1b3/0x9c0 [nvme_tcp]
Apr 27 20:32:08 testnode1 kernel:        nvme_tcp_queue_rq+0xf77/0x1970 [nvme_tcp]
Apr 27 20:32:08 testnode1 kernel:        blk_mq_dispatch_rq_list+0x39b/0x2340
Apr 27 20:32:08 testnode1 kernel:        __blk_mq_sched_dispatch_requests+0x1dd/0x1510
Apr 27 20:32:08 testnode1 kernel:        blk_mq_sched_dispatch_requests+0xa8/0x150
Apr 27 20:32:08 testnode1 kernel:        blk_mq_run_work_fn+0x127/0x2c0
Apr 27 20:32:08 testnode1 kernel:        process_one_work+0x8b4/0x1640
Apr 27 20:32:08 testnode1 kernel:        worker_thread+0x606/0xff0
Apr 27 20:32:08 testnode1 kernel:        kthread+0x368/0x460
Apr 27 20:32:08 testnode1 kernel:        ret_from_fork+0x653/0x9d0
Apr 27 20:32:08 testnode1 kernel:        ret_from_fork_asm+0x1a/0x30
Apr 27 20:32:08 testnode1 kernel: 
                                  -> #0 (set->srcu){.+.+}-{0:0}:
Apr 27 20:32:08 testnode1 kernel:        __lock_acquire+0x14a6/0x2230
Apr 27 20:32:08 testnode1 kernel:        lock_sync+0xbd/0x120
Apr 27 20:32:08 testnode1 kernel:        __synchronize_srcu+0xa1/0x2b0
Apr 27 20:32:08 testnode1 kernel:        elevator_switch+0x2a5/0x680
Apr 27 20:32:08 testnode1 kernel:        elevator_change+0x2d8/0x4f0
Apr 27 20:32:08 testnode1 kernel:        elevator_set_none+0x87/0xd0
Apr 27 20:32:08 testnode1 kernel:        blk_unregister_queue+0x13f/0x2b0
Apr 27 20:32:08 testnode1 kernel:        __del_gendisk+0x263/0x9e0
Apr 27 20:32:08 testnode1 kernel:        del_gendisk+0x102/0x190
Apr 27 20:32:08 testnode1 kernel:        nvme_ns_remove+0x32a/0x900 [nvme_core]
Apr 27 20:32:08 testnode1 kernel:        nvme_remove_namespaces+0x263/0x3b0 [nvme_core]
Apr 27 20:32:08 testnode1 kernel:        nvme_do_delete_ctrl+0xf5/0x160 [nvme_core]
Apr 27 20:32:08 testnode1 kernel:        nvme_delete_ctrl_sync.cold+0x8/0xd [nvme_core]
Apr 27 20:32:08 testnode1 kernel:        nvme_sysfs_delete+0x96/0xc0 [nvme_core]
Apr 27 20:32:08 testnode1 kernel:        kernfs_fop_write_iter+0x3d6/0x5e0
Apr 27 20:32:08 testnode1 kernel:        vfs_write+0x52c/0xf80
Apr 27 20:32:08 testnode1 kernel:        ksys_write+0xfb/0x200
Apr 27 20:32:08 testnode1 kernel:        do_syscall_64+0xdd/0x14c0
Apr 27 20:32:08 testnode1 kernel:        entry_SYSCALL_64_after_hwframe+0x76/0x7e
Apr 27 20:32:08 testnode1 kernel: 
                                  other info that might help us debug this:
Apr 27 20:32:08 testnode1 kernel: Chain exists of:
                                    set->srcu --> &q->q_usage_counter(io) --> &q->elevator_lock
Apr 27 20:32:08 testnode1 kernel:  Possible unsafe locking scenario:
Apr 27 20:32:08 testnode1 kernel:        CPU0                    CPU1
Apr 27 20:32:08 testnode1 kernel:        ----                    ----
Apr 27 20:32:08 testnode1 kernel:   lock(&q->elevator_lock);
Apr 27 20:32:08 testnode1 kernel:                                lock(&q->q_usage_counter(io));
Apr 27 20:32:08 testnode1 kernel:                                lock(&q->elevator_lock);
Apr 27 20:32:08 testnode1 kernel:   sync(set->srcu);
Apr 27 20:32:08 testnode1 kernel: 
                                   *** DEADLOCK ***
Apr 27 20:32:08 testnode1 kernel: 5 locks held by nvme/1171:
Apr 27 20:32:08 testnode1 kernel:  #0: ffff88810868e410 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xfb/0x200
Apr 27 20:32:08 testnode1 kernel:  #1: ffff88814e03f080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x257/0x5e0
Apr 27 20:32:08 testnode1 kernel:  #2: ffff88814e3f84b8 (kn->active#140){++++}-{0:0}, at: sysfs_remove_file_self+0x61/0xb0
Apr 27 20:32:08 testnode1 kernel:  #3: ffff8881073281c8 (&set->update_nr_hwq_lock){++++}-{4:4}, at: del_gendisk+0xfa/0x190
Apr 27 20:32:08 testnode1 kernel:  #4: ffff88812ab7bd68 (&q->elevator_lock){+.+.}-{4:4}, at: elevator_change+0x188/0x4f0
Apr 27 20:32:08 testnode1 kernel: 
                                  stack backtrace:
Apr 27 20:32:08 testnode1 kernel: CPU: 3 UID: 0 PID: 1171 Comm: nvme Not tainted 7.1.0-rc1+ #3 PREEMPT(full) 
Apr 27 20:32:08 testnode1 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-9.fc43 06/10/2025
Apr 27 20:32:08 testnode1 kernel: Call Trace:
Apr 27 20:32:08 testnode1 kernel:  <TASK>
Apr 27 20:32:08 testnode1 kernel:  dump_stack_lvl+0x6a/0x90
Apr 27 20:32:08 testnode1 kernel:  print_circular_bug.cold+0x185/0x1d0
Apr 27 20:32:08 testnode1 kernel:  check_noncircular+0x148/0x170
Apr 27 20:32:08 testnode1 kernel:  __lock_acquire+0x14a6/0x2230
Apr 27 20:32:08 testnode1 kernel:  lock_sync+0xbd/0x120
Apr 27 20:32:08 testnode1 kernel:  ? __synchronize_srcu+0x21/0x2b0
Apr 27 20:32:08 testnode1 kernel:  ? __synchronize_srcu+0x21/0x2b0
Apr 27 20:32:08 testnode1 kernel:  __synchronize_srcu+0xa1/0x2b0
Apr 27 20:32:08 testnode1 kernel:  ? __pfx___synchronize_srcu+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  ? kvm_clock_get_cycles+0x14/0x30
Apr 27 20:32:08 testnode1 kernel:  ? ktime_get_mono_fast_ns+0x193/0x490
Apr 27 20:32:08 testnode1 kernel:  ? lockdep_hardirqs_on+0x88/0x130
Apr 27 20:32:08 testnode1 kernel:  ? _raw_spin_unlock_irqrestore+0x4c/0x60
Apr 27 20:32:08 testnode1 kernel:  elevator_switch+0x2a5/0x680
Apr 27 20:32:08 testnode1 kernel:  elevator_change+0x2d8/0x4f0
Apr 27 20:32:08 testnode1 kernel:  elevator_set_none+0x87/0xd0
Apr 27 20:32:08 testnode1 kernel:  ? __pfx_elevator_set_none+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  ? kobject_put+0x5a/0x4e0
Apr 27 20:32:08 testnode1 kernel:  blk_unregister_queue+0x13f/0x2b0
Apr 27 20:32:08 testnode1 kernel:  __del_gendisk+0x263/0x9e0
Apr 27 20:32:08 testnode1 kernel:  ? down_read+0x13b/0x480
Apr 27 20:32:08 testnode1 kernel:  ? __pfx___del_gendisk+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  ? __pfx_down_read+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  ? up_write+0x294/0x510
Apr 27 20:32:08 testnode1 kernel:  del_gendisk+0x102/0x190
Apr 27 20:32:08 testnode1 kernel:  nvme_ns_remove+0x32a/0x900 [nvme_core]
Apr 27 20:32:08 testnode1 kernel:  nvme_remove_namespaces+0x263/0x3b0 [nvme_core]
Apr 27 20:32:08 testnode1 kernel:  ? __pfx_nvme_remove_namespaces+0x10/0x10 [nvme_core]
Apr 27 20:32:08 testnode1 kernel:  nvme_do_delete_ctrl+0xf5/0x160 [nvme_core]
Apr 27 20:32:08 testnode1 kernel:  nvme_delete_ctrl_sync.cold+0x8/0xd [nvme_core]
Apr 27 20:32:08 testnode1 kernel:  nvme_sysfs_delete+0x96/0xc0 [nvme_core]
Apr 27 20:32:08 testnode1 kernel:  ? __pfx_sysfs_kf_write+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  kernfs_fop_write_iter+0x3d6/0x5e0
Apr 27 20:32:08 testnode1 kernel:  ? __pfx_kernfs_fop_write_iter+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  vfs_write+0x52c/0xf80
Apr 27 20:32:08 testnode1 kernel:  ? __pfx_vfs_write+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  ? kasan_save_free_info+0x37/0x70
Apr 27 20:32:08 testnode1 kernel:  ? __kasan_slab_free+0x67/0x80
Apr 27 20:32:08 testnode1 kernel:  ? kmem_cache_free+0x14c/0x670
Apr 27 20:32:08 testnode1 kernel:  ? do_sys_openat2+0xfd/0x170
Apr 27 20:32:08 testnode1 kernel:  ? __x64_sys_openat+0x10a/0x210
Apr 27 20:32:08 testnode1 kernel:  ksys_write+0xfb/0x200
Apr 27 20:32:08 testnode1 kernel:  ? __pfx_ksys_write+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  do_syscall_64+0xdd/0x14c0
Apr 27 20:32:08 testnode1 kernel:  ? kasan_quarantine_put+0xff/0x220
Apr 27 20:32:08 testnode1 kernel:  ? lockdep_hardirqs_on+0x88/0x130
Apr 27 20:32:08 testnode1 kernel:  ? kasan_quarantine_put+0xff/0x220
Apr 27 20:32:08 testnode1 kernel:  ? kasan_quarantine_put+0xff/0x220
Apr 27 20:32:08 testnode1 kernel:  ? do_sys_openat2+0xfd/0x170
Apr 27 20:32:08 testnode1 kernel:  ? kmem_cache_free+0x14c/0x670
Apr 27 20:32:08 testnode1 kernel:  ? do_sys_openat2+0xfd/0x170
Apr 27 20:32:08 testnode1 kernel:  ? __pfx_do_sys_openat2+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  ? kmem_cache_free+0x14c/0x670
Apr 27 20:32:08 testnode1 kernel:  ? __x64_sys_openat+0x10a/0x210
Apr 27 20:32:08 testnode1 kernel:  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
Apr 27 20:32:08 testnode1 kernel:  ? __pfx___x64_sys_openat+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  ? rcu_is_watching+0x11/0xb0
Apr 27 20:32:08 testnode1 kernel:  ? do_syscall_64+0x1ea/0x14c0
Apr 27 20:32:08 testnode1 kernel:  ? lockdep_hardirqs_on+0x88/0x130
Apr 27 20:32:08 testnode1 kernel:  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
Apr 27 20:32:08 testnode1 kernel:  ? do_syscall_64+0x208/0x14c0
Apr 27 20:32:08 testnode1 kernel:  ? __pfx___x64_sys_openat+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  ? __pfx___x64_sys_openat+0x10/0x10
Apr 27 20:32:08 testnode1 kernel:  ? rcu_is_watching+0x11/0xb0
Apr 27 20:32:08 testnode1 kernel:  ? do_syscall_64+0x1ea/0x14c0
Apr 27 20:32:08 testnode1 kernel:  ? lockdep_hardirqs_on+0x88/0x130
Apr 27 20:32:08 testnode1 kernel:  ? do_syscall_64+0x208/0x14c0
Apr 27 20:32:08 testnode1 kernel:  ? do_syscall_64+0x32/0x14c0
Apr 27 20:32:08 testnode1 kernel:  ? preempt_count_add+0x7f/0x190
Apr 27 20:32:08 testnode1 kernel:  ? do_syscall_64+0x5d/0x14c0
Apr 27 20:32:08 testnode1 kernel:  ? do_syscall_64+0x8d/0x14c0
Apr 27 20:32:08 testnode1 kernel:  ? irqentry_exit+0xf1/0x720
Apr 27 20:32:08 testnode1 kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Apr 27 20:32:08 testnode1 kernel: RIP: 0033:0x7f245cf99c5e
Apr 27 20:32:08 testnode1 kernel: Code: 4d 89 d8 e8 34 bd 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 <c9> c3 83 e2 39 83 fa 08 75 e7 e8 13 ff ff ff 0f 1f 00 f3 0f 1e fa
Apr 27 20:32:08 testnode1 kernel: RSP: 002b:00007ffca6d9f6a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
Apr 27 20:32:08 testnode1 kernel: RAX: ffffffffffffffda RBX: 00007f245d1639a6 RCX: 00007f245cf99c5e
Apr 27 20:32:08 testnode1 kernel: RDX: 0000000000000001 RSI: 00007f245d1639a6 RDI: 0000000000000003
Apr 27 20:32:08 testnode1 kernel: RBP: 00007ffca6d9f6b0 R08: 0000000000000000 R09: 0000000000000000
Apr 27 20:32:08 testnode1 kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 000000003d0f6860
Apr 27 20:32:08 testnode1 kernel: R13: 000000003d0f8580 R14: 000000003d0f6680 R15: 0000000000000000
Apr 27 20:32:08 testnode1 kernel:  </TASK>

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Haris Iqbal]

On Fri, Feb 13, 2026 at 12:25 PM Shinichiro Kawasaki
<shinichiro.kawasaki@wdc.com> wrote:
>
> On Feb 12, 2026 / 08:52, Daniel Wagner wrote:
> > On Wed, Feb 11, 2026 at 08:35:30PM +0000, Chaitanya Kulkarni wrote:
> > >    For the storage track at LSFMMBPF2026, I propose a session dedicated to
> > >    blktests to discuss expansion plan and CI integration progress.
> >
> > Thanks for proposing this topic.
>
> Chaitanya, my thank also goes to you.
>
> > Just a few random topics which come to mind we could discuss:
> >
> > - blktests has gain a bit of traction and some folks run on regular
> >   basis these tests. Can we gather feedback from them, what is working
> >   good, what is not? Are there feature wishes?
>
> Good topic, I also would like to hear about it.
>
> FYI, from the past LSFMM sessions and hallway talks, major feedbacks I had
> received are these two:
>
>  1. blktests CI infra looks missing (other than CKI by Redhat)
>     -> Some activities are ongoing to start blktests CI service.
>        I hope the status are shared at the session.
>
>  2. blktests are rather difficult to start using for some new users
>     -> I think config example is demanded, so that new users can
>        just copy it to start the first run, and understand the
>        config options easily.
>
> > - Do we need some sort of configuration tool which allows to setup a
> >   config? I'd still have a TODO to provide a config example with all
> >   knobs which influence blktests, but I wonder if we should go a step
> >   further here, e.g. something like kdevops has?
>
> Do you mean the "make menuconfig" style? Most of the blktests users are
> familiar with menuconfig, so that would be an idea. If users really want
> it, we can think of it. IMO, blktests still do not have so many options,
> then config.example would be simpler and more appropriate, probably.
>
> > - Which area do we lack tests? Should we just add an initial simple
> >   tests for the missing areas, so the basic infra is there and thus
> >   lowering the bar for adding new tests?
>
> To identify the uncovered area, I think code coverage will be useful. A few
> years ago, I measured it and shared in LSFMM, but that measurement was done for
> each source tree directory. The coverage ratio by source file will be more
> helpful to identify the missing area. I don't have time slot to measure it,
> so if anyone can do it and share the result, it will be appreciated. Once we
> know the missing areas, it sounds a good idea to add initial samples for each
> of the areas.
>
> > - The recent addition of kmemleak shows it's a great idea to enable more
> >   of the kernel test infrastructure when running the tests.
>
> Completely agreed.
>
> >   Are there more such things we could/should enable?
>
> I'm also interested in this question :)
>
> > - I would like to hear from Shin'ichiro if he is happy how things
> >   are going? :)
>
> More importantly, I would like to listen to voices from storage sub-system
> developers to see if they are happy or not, especially the maintainers.
>
> From my view, blktests keep on finding kernel bugs. I think it demonstrates the
> value of this community effort, and I'm happy about it. Said that, I find what
> blktests can improve more, of course. Here I share the list of improvement
> opportunities from my view point (I already mentioned the first three items).

A possible feature for blktest could be integration with something
like virtme-ng.
Running on VM can be versatile and fast. The run can be made parallel
too, by spawning multiple VMs simultaneously.

>
>  1. We can have more CI infra to make the most of blktests
>  2. We can add config examples to help new users
>  3. We can measure code coverage to identify missing test areas
>  4. Long standing failures make test result reports dirty
>     - I feel lockdep WARNs are tend to be left unfixed rather long period.
>       How can we gather effort to fix them?
>  5. We can refactor and clean up blktests framework for ease of maintainance
>       (e.g. trap handling)
>  6. Some users run blktests with built-in kernel modules, which makes a number
>     of test cases skipped. We can add more built-in kernel modules support to
>     expand test coverage for such use case.

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Chaitanya Kulkarni]

On 2/15/26 13:18, Haris Iqbal wrote:
> On Fri, Feb 13, 2026 at 12:25 PM Shinichiro Kawasaki
> <shinichiro.kawasaki@wdc.com> wrote:
>> On Feb 12, 2026 / 08:52, Daniel Wagner wrote:
>>> On Wed, Feb 11, 2026 at 08:35:30PM +0000, Chaitanya Kulkarni wrote:
>>>>     For the storage track at LSFMMBPF2026, I propose a session dedicated to
>>>>     blktests to discuss expansion plan and CI integration progress.
>>> Thanks for proposing this topic.
>> Chaitanya, my thank also goes to you.
>>
>>> Just a few random topics which come to mind we could discuss:
>>>
>>> - blktests has gain a bit of traction and some folks run on regular
>>>    basis these tests. Can we gather feedback from them, what is working
>>>    good, what is not? Are there feature wishes?
>> Good topic, I also would like to hear about it.
>>
>> FYI, from the past LSFMM sessions and hallway talks, major feedbacks I had
>> received are these two:
>>
>>   1. blktests CI infra looks missing (other than CKI by Redhat)
>>      -> Some activities are ongoing to start blktests CI service.
>>         I hope the status are shared at the session.
>>
>>   2. blktests are rather difficult to start using for some new users
>>      -> I think config example is demanded, so that new users can
>>         just copy it to start the first run, and understand the
>>         config options easily.
>>
>>> - Do we need some sort of configuration tool which allows to setup a
>>>    config? I'd still have a TODO to provide a config example with all
>>>    knobs which influence blktests, but I wonder if we should go a step
>>>    further here, e.g. something like kdevops has?
>> Do you mean the "make menuconfig" style? Most of the blktests users are
>> familiar with menuconfig, so that would be an idea. If users really want
>> it, we can think of it. IMO, blktests still do not have so many options,
>> then config.example would be simpler and more appropriate, probably.
>>
>>> - Which area do we lack tests? Should we just add an initial simple
>>>    tests for the missing areas, so the basic infra is there and thus
>>>    lowering the bar for adding new tests?
>> To identify the uncovered area, I think code coverage will be useful. A few
>> years ago, I measured it and shared in LSFMM, but that measurement was done for
>> each source tree directory. The coverage ratio by source file will be more
>> helpful to identify the missing area. I don't have time slot to measure it,
>> so if anyone can do it and share the result, it will be appreciated. Once we
>> know the missing areas, it sounds a good idea to add initial samples for each
>> of the areas.
>>
>>> - The recent addition of kmemleak shows it's a great idea to enable more
>>>    of the kernel test infrastructure when running the tests.
>> Completely agreed.
>>
>>>    Are there more such things we could/should enable?
>> I'm also interested in this question 🙂
>>
>>> - I would like to hear from Shin'ichiro if he is happy how things
>>>    are going? 🙂
>> More importantly, I would like to listen to voices from storage sub-system
>> developers to see if they are happy or not, especially the maintainers.
>>
>>  From my view, blktests keep on finding kernel bugs. I think it demonstrates the
>> value of this community effort, and I'm happy about it. Said that, I find what
>> blktests can improve more, of course. Here I share the list of improvement
>> opportunities from my view point (I already mentioned the first three items).
> A possible feature for blktest could be integration with something
> like virtme-ng.
> Running on VM can be versatile and fast. The run can be made parallel
> too, by spawning multiple VMs simultaneously.

This is my goal and I had proposed this topic few yeard back
to have blktests integrated with VM. I've spent sometime on a initial
setup but never got it to finish it.

If someone is working on it I'll be happy to help and review also test the
implementation.

-ck

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Johannes Thumshirn]

On 2/15/26 10:18 PM, Haris Iqbal wrote:
>>  From my view, blktests keep on finding kernel bugs. I think it demonstrates the
>> value of this community effort, and I'm happy about it. Said that, I find what
>> blktests can improve more, of course. Here I share the list of improvement
>> opportunities from my view point (I already mentioned the first three items).
> A possible feature for blktest could be integration with something
> like virtme-ng.
> Running on VM can be versatile and fast. The run can be made parallel
> too, by spawning multiple VMs simultaneously.

This is actually rather trivial to solve I have some pre-made things for 
fstests and that can be adopted for blktests as well:

vng \
     --user=root -v --name vng-tcmu-runner \
     -a loglevel=3 \
     --run $KDIR \
     --cpus=8 --memory=8G \
     --exec "~johannes/src/ci/run-fstests.sh" \
     --qemu-opts="-device virtio-scsi,id=scsi0 -drive 
file=/dev/sda,format=raw,if=none,id=zbc0 -device 
scsi-block,bus=scsi0.0,drive=zbc0" \
     --qemu-opts="-device virtio-scsi,id=scsi1 -drive 
file=/dev/sdb,format=raw,if=none,id=zbc1 -device 
scsi-block,bus=scsi1.0,drive=zbc1"

and run-fstests.sh is:

#!/bin/sh
# SPDX-License-Identifier: GPL-2.0

DIR="/tmp/"
MKFS="mkfs.btrfs -f"
FSTESTS_DIR="/home/johannes/src/fstests"
HOSTCONF="$FSTESTS_DIR/configs/$(hostname -s)"
TESTDEV="$(grep TEST_DEV $HOSTCONF | cut -d '=' -f 2)"

mkdir -p $DIR/{test,scratch,results}
$MKFS $TESTDEV

cd $FSTESTS_DIR
./check -x raid

I'm not sure it'll make sense to include this into blktests other than 
maybe providing an example in the README.


Byte,

     Johannes

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Haris Iqbal]

On Mon, Feb 23, 2026 at 8:44 AM Johannes Thumshirn
<Johannes.Thumshirn@wdc.com> wrote:
>
> On 2/15/26 10:18 PM, Haris Iqbal wrote:
> >>  From my view, blktests keep on finding kernel bugs. I think it demonstrates the
> >> value of this community effort, and I'm happy about it. Said that, I find what
> >> blktests can improve more, of course. Here I share the list of improvement
> >> opportunities from my view point (I already mentioned the first three items).
> > A possible feature for blktest could be integration with something
> > like virtme-ng.
> > Running on VM can be versatile and fast. The run can be made parallel
> > too, by spawning multiple VMs simultaneously.
>
> This is actually rather trivial to solve I have some pre-made things for
> fstests and that can be adopted for blktests as well:
>
> vng \
>      --user=root -v --name vng-tcmu-runner \
>      -a loglevel=3 \
>      --run $KDIR \
>      --cpus=8 --memory=8G \
>      --exec "~johannes/src/ci/run-fstests.sh" \
>      --qemu-opts="-device virtio-scsi,id=scsi0 -drive
> file=/dev/sda,format=raw,if=none,id=zbc0 -device
> scsi-block,bus=scsi0.0,drive=zbc0" \
>      --qemu-opts="-device virtio-scsi,id=scsi1 -drive
> file=/dev/sdb,format=raw,if=none,id=zbc1 -device
> scsi-block,bus=scsi1.0,drive=zbc1"
>
> and run-fstests.sh is:
>
> #!/bin/sh
> # SPDX-License-Identifier: GPL-2.0
>
> DIR="/tmp/"
> MKFS="mkfs.btrfs -f"
> FSTESTS_DIR="/home/johannes/src/fstests"
> HOSTCONF="$FSTESTS_DIR/configs/$(hostname -s)"
> TESTDEV="$(grep TEST_DEV $HOSTCONF | cut -d '=' -f 2)"
>
> mkdir -p $DIR/{test,scratch,results}
> $MKFS $TESTDEV
>
> cd $FSTESTS_DIR
> ./check -x raid
>
> I'm not sure it'll make sense to include this into blktests other than
> maybe providing an example in the README.

You're right. It is pretty trivial to run on VMs, but only after
everything is set up.
Adding it to blktests would allow this setup to be done (and run tests
after that) on any system by running just a couple of commands.

>
>
> Byte,
>
>      Johannes
>

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Shin'ichiro Kawasaki]

[-- Attachment #1: Type: text/plain, Size: 2250 bytes --]

On Feb 23, 2026 / 07:44, Johannes Thumshirn wrote:
> On 2/15/26 10:18 PM, Haris Iqbal wrote:
> >>  From my view, blktests keep on finding kernel bugs. I think it demonstrates the
> >> value of this community effort, and I'm happy about it. Said that, I find what
> >> blktests can improve more, of course. Here I share the list of improvement
> >> opportunities from my view point (I already mentioned the first three items).
> > A possible feature for blktest could be integration with something
> > like virtme-ng.
> > Running on VM can be versatile and fast. The run can be made parallel
> > too, by spawning multiple VMs simultaneously.
> 
> This is actually rather trivial to solve I have some pre-made things for 
> fstests and that can be adopted for blktests as well:
> 
> vng \
>      --user=root -v --name vng-tcmu-runner \
>      -a loglevel=3 \
>      --run $KDIR \
>      --cpus=8 --memory=8G \
>      --exec "~johannes/src/ci/run-fstests.sh" \
>      --qemu-opts="-device virtio-scsi,id=scsi0 -drive 
> file=/dev/sda,format=raw,if=none,id=zbc0 -device 
> scsi-block,bus=scsi0.0,drive=zbc0" \
>      --qemu-opts="-device virtio-scsi,id=scsi1 -drive 
> file=/dev/sdb,format=raw,if=none,id=zbc1 -device 
> scsi-block,bus=scsi1.0,drive=zbc1"
> 
> and run-fstests.sh is:
> 
> #!/bin/sh
> # SPDX-License-Identifier: GPL-2.0
> 
> DIR="/tmp/"
> MKFS="mkfs.btrfs -f"
> FSTESTS_DIR="/home/johannes/src/fstests"
> HOSTCONF="$FSTESTS_DIR/configs/$(hostname -s)"
> TESTDEV="$(grep TEST_DEV $HOSTCONF | cut -d '=' -f 2)"
> 
> mkdir -p $DIR/{test,scratch,results}
> $MKFS $TESTDEV
> 
> cd $FSTESTS_DIR
> ./check -x raid
> 
> I'm not sure it'll make sense to include this into blktests other than 
> maybe providing an example in the README.

I guess the example can be added in contrib/. This idea interested me, so I did
some quick scripting and created the patch attached. I did some blktests runs
with virtme-ng, and found that:

- virtme-ng allows to skip kernel installation step. Fast and useful as Haris
  pointed out.
- systemd does not look working well with virtme-ng, even when I specify the
  --systemd option. This will be a condition difference from normal
  blktests runs.

I hope that the patch will help discussion at the conference.

[-- Attachment #2: 0001-contrib-add-scripts-to-run-blktests-with-virtme-ng.patch --]
[-- Type: text/plain, Size: 5517 bytes --]

From 4a310c98f674e65218fb7e33e3b176a85d897333 Mon Sep 17 00:00:00 2001
From: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Date: Tue, 21 Apr 2026 14:26:01 +0900
Subject: [PATCH blktests RFC] contrib: add scripts to run blktests with
 virtme-ng

It takes rather long time to run blktests repeatedly for debug target
kernels because of the time to install the test target kernel to the
test target system. To reduce the turn around time, virtme-ng [1] is
useful. Add helper scripts to run blktests using virtme-ng per
suggestion by Johaness [2].

After building a kernel in ~/linux, the command lines below will run
blktests on the built kernel. With this, kernel installation step is no
longer required.

 $ KDIR=~/linux contrib/run-vng loop/010
 virtme: waiting for virtiofsd to start
 virtme: use 'microvm' QEMU architecture
 early console in setup code
 Probing EDD (edd=off to disable)... ok
 [    0.000000][    T0] Linux version 7.0.0 (shin@shinmob) (gcc (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7), GNU ld version 2.45.1-4.fc43) #22 SMP PREEMPT_DYNAMIC Mon Apr 13 15:44:07 JST 2026
 [    0.000000][    T0] Command line: virtme_hostname=vng-blktests-runner nr_open=2147483584 virtme_link_mods=/home/shin/linux/.virtme_mods/lib/modules/0.0.0 virtme_initmount0=tmp virtme_rw_overlay0=/etc virtme_rw_overlay1=/lib virtme_rw_overlay2=/home virtme_rw_overlay3=/opt virtme_rw_overlay4=/srv virtme_rw_overlay5=/usr virtme_rw_overlay6=/var virtme_rw_overlay7=/tmp virtme_console=ttyS0 console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1 virtme.exec=`L2hvbWUvc2hpbi9CbGt0ZXN0cy9ibGt0ZXN0cy9jb250cmliL3J1bi1ibGt0ZXN0cy1xdWljayBsb29wLzAwMQ==` virtme.ssh virtme_ssh_channel=vsock virtme_ssh_cache=/home/shin/.cache/virtme-ng/.ssh virtme_chdir=home/shin/Blktests/blktests/contrib debug loglevel=3 init=/usr/lib/python3.14/site-packages/virtme/guest/bin/virtme-ng-init
 [    0.000000][    T0] x86/split lock detection: #DB: warning on user-space bus_locks
 [    0.000000][    T0] BIOS-provided physical RAM map:
 [    0.000000][    T0] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff]  System RAM
 [    0.000000][    T0] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff]  device reserved
 [    0.000000][    T0] BIOS-e820: [gap 0x00000000000a0000-0x00000000000effff]
 [    0.000000][    T0] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff]  device reserved
 [    0.000000][    T0] BIOS-e820: [mem 0x0000000000100000-0x00000000bfffefff]  System RAM
 [    0.000000][    T0] BIOS-e820: [mem 0x00000000bffff000-0x00000000bfffffff]  device reserved
 [    0.000000][    T0] BIOS-e820: [gap 0x00000000c0000000-0x00000000feffbfff]
 [    0.000000][    T0] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff]  device reserved
 [    0.000000][    T0] BIOS-e820: [gap 0x00000000ff000000-0x00000000fffbffff]
 [    0.000000][    T0] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff]  device reserved
 [    0.000000][    T0] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff]  System RAM
 [    0.000000][    T0] printk: legacy bootconsole [earlyser0] enabled
 Poking KASLR using RDRAND RDTSC...
 Clean up /tmp/results ...
 loop/001 (scan loop device partitions)
 loop/001 (scan loop device partitions)                       [passed]
     runtime    ...  14.610s
 Run logs:
 [   90.927921][    T1] reboot: Power down

When I ran the script on my Fedora 43 testnode for some test groups, I
observed hangs at nvme/010 and nvme/012. Also I observed a failure at
loop/010. Some fixes will be required in blktests and kernel to run the
new script in stable manner.

Of note is that the script does not support test target block device
pass-through yet.

[1] https://github.com/arighi/virtme-ng
[2] https://lore.kernel.org/linux-block/ae47ef06-3f66-4aab-b4ab-f3ae2b634f87@wdc.com/

Suggested-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 contrib/run-blktests-quick | 27 +++++++++++++++++++++++++++
 contrib/run-vng            | 18 ++++++++++++++++++
 2 files changed, 45 insertions(+)
 create mode 100755 contrib/run-blktests-quick
 create mode 100755 contrib/run-vng

diff --git a/contrib/run-blktests-quick b/contrib/run-blktests-quick
new file mode 100755
index 0000000..9324c38
--- /dev/null
+++ b/contrib/run-blktests-quick
@@ -0,0 +1,27 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2026 Western Digital Corporation or its affiliates.
+
+declare DIR LOGDIR BLKTESTS_DIR
+
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
+BLKTESTS_DIR="${DIR}/../"
+
+LOGDIR="/tmp/results"
+
+if [[ -e $LOGDIR ]]; then
+	echo "Clean up ${LOGDIR} ..."
+	rm -rf $LOGDIR
+fi
+mkdir -p "$LOGDIR"
+
+cd "$BLKTESTS_DIR" || exit
+
+cat > config << EOF
+QUICK_RUN=1
+TIMEOUT=10
+EOF
+
+./check -o "$LOGDIR" "$@"
+echo "Run logs: ${LOGIDR}"
+
diff --git a/contrib/run-vng b/contrib/run-vng
new file mode 100755
index 0000000..9aa4b91
--- /dev/null
+++ b/contrib/run-vng
@@ -0,0 +1,18 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2026 Western Digital Corporation or its affiliates.
+
+declare DIR
+declare -a BLKTESTS_OPTS
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
+
+if (($#)); then
+	BLKTESTS_OPTS=("$@")
+else
+	BLKTESTS_OPTS=(block nvme loop scsi)
+fi
+
+vng \
+ --user=root --verbose --name vng-blktests-runner --cpus=4 --memory=4G \
+ --append loglevel=3 --run "$KDIR" --ssh \
+ --rwdir=/tmp --exec "$DIR/run-blktests-quick ${BLKTESTS_OPTS[*]}"
-- 
2.53.0

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Bart Van Assche]

On 2/15/26 1:18 PM, Haris Iqbal wrote:
> A possible feature for blktest could be integration with something
> like virtme-ng.
> Running on VM can be versatile and fast. The run can be made parallel
> too, by spawning multiple VMs simultaneously.
Hmm ... this probably would break tests that measure performance and
also tests that modify data or reservations of a physical storage
device.

Bart.

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Chaitanya Kulkarni]

On 2/23/26 09:08, Bart Van Assche wrote:
> On 2/15/26 1:18 PM, Haris Iqbal wrote:
>> A possible feature for blktest could be integration with something
>> like virtme-ng.
>> Running on VM can be versatile and fast. The run can be made parallel
>> too, by spawning multiple VMs simultaneously.
> Hmm ... this probably would break tests that measure performance and
> also tests that modify data or reservations of a physical storage
> device.
>
> Bart.

We can always make a flag and make test that are parallel compatible ?
so we don't have to for the parallel execution by default.

WDYT ?

-ck

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Haris Iqbal]

On Mon, Feb 23, 2026 at 6:08 PM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 2/15/26 1:18 PM, Haris Iqbal wrote:
> > A possible feature for blktest could be integration with something
> > like virtme-ng.
> > Running on VM can be versatile and fast. The run can be made parallel
> > too, by spawning multiple VMs simultaneously.
> Hmm ... this probably would break tests that measure performance and
> also tests that modify data or reservations of a physical storage
> device.

Performance related tests can be skipped when running in a virtual environment.
Regarding data modification, if the tests do not involve any crash or
reboot, then the VMs can be started in "snapshot" mode. This gives a
number of advantages.
a) Data modifications will not persist once the VM is shut down. This
means the disk will be clean for the next test cycle.
b) Using just a single set of qcow files, one can bring up any number
of VMs in snapshot mode. The data written while the VM is running can
be safely read/modified, but it disappears after a reboot.

>
> Bart.

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Bart Van Assche]

On 2/25/26 2:07 AM, Haris Iqbal wrote:
> Regarding data modification, if the tests do not involve any crash or
> reboot, then the VMs can be started in "snapshot" mode.
I'm not sure that proposal makes sense. If e.g. an NVMe device is
specified in the blktests config file, it probably is the intention of
the person who runs the test to test the NVMe driver and/or the NVMe
device. Using any method to create a "snapshot" of the device and to
run blktests against that snapshot changes the kernel driver and also
the physical device that are tested. Not modifying the kernel driver
or physical device that are tested implies using PCIe passthrough. And
the PCIe passthrough mechanism can only be used by one VM at a time as
far as I know.

Bart.

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Shin'ichiro Kawasaki]

On Feb 13, 2026 / 11:23, Shinichiro Kawasaki wrote:
[...]
> Here I share the list of improvement
> opportunities from my view point (I already mentioned the first three items).
> 
>  1. We can have more CI infra to make the most of blktests
>  2. We can add config examples to help new users
>  3. We can measure code coverage to identify missing test areas
>  4. Long standing failures make test result reports dirty
>     - I feel lockdep WARNs are tend to be left unfixed rather long period.
>       How can we gather effort to fix them?
>  5. We can refactor and clean up blktests framework for ease of maintainance
>       (e.g. trap handling)
>  6. Some users run blktests with built-in kernel modules, which makes a number
>     of test cases skipped. We can add more built-in kernel modules support to
>     expand test coverage for such use case.

I read through this e-mail thread again, and revised the list of improvement
opportunities. Hope this list helps the discussion at the session.

 1. configuration example, documentation, or tool for ease of new users
 2. blktests framework capabilitly
   - more bug detection capability (such as kmemleak)
     - code coverage measurement per test case
   - VM integration (virtme-ng? valid use case?)
   - parallel execution for shorter run
   - built-in module testing
   - multi device testing
 3. blktests CI
 4. longstanding failures
   - nvme/005 tcp transport, nbd/002
 5. test area coverage expansion (which area do we lack tests?)
   - code coverage information as the guide
   - new category candidates (rnbd?)
 6. blktests framework refactoring
   - trap handling


One thing I added was "multi device testing". Most of the test cases use
single test device for single test run, while md/003 and bcache/001 require
multiple devices for single test run. There was a discussion how to specify
multiple devices in the config file [*].

[*] https://lore.kernel.org/linux-block/aXHbPWCf5SfycpBX@shinmob/

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Shin'ichiro Kawasaki]

On Feb 11, 2026 / 20:35, Chaitanya Kulkarni wrote:
> Hi all,
> 
>    Since the discussion at the LSFMM 2017 [1], Omar Sandoval introduced 
> the new
>    framework "blktests" dedicated for Linux Kernel Block layer testing.

[...]

>    For the storage track at LSFMMBPF2026, I propose a session dedicated to
>    blktests to discuss expansion plan and CI integration progress.

My thank goes to the session attendees. Here I share my memorandom about what
was discussed there. If I overlook anything, amendments will be appreciated.

----------------------------------------------------------------------------

blktests CI
===========

- Regular test runs for branches (nightly or -rcX)
  - It will be helpful to run blktests for -rcX releases and report.
  - Ted has resources to run blktests regularly, but it requires someone to
    look into the test results [*]

- Repeated test case run
  - fstests has the feature to repeat failed test cases 10(x) times.

- kdevops has the feature to find out the failure recreated by 1000x times
  repeat
   -> This will be a 2nd step topic and looks too early currently.

- Kernel config for CI runs
  - Please enable DEBUG_ATOMIC_SLEEP [**]
  - Can we specify kernel config for each patch author?
    -> It will require too much resource

- PatchWork for linux-nvme
  - Good for patch status tracking also
  - Should ask kernel.org

- e-mail reporting
  - e-mail address should be available at kernel.org
  - Do not report the "ALL PASS" case. It will be noisy.
    -> Will send out report only for failure cases.

blktests improvement ideas
==========================

- For easier config:
  - Ted has the config generation tool.
  -> I'm interested in it, and would like to take a look.

- VM integration
  - VM integration script will help new users to try out blktests
  - Haris thinks this is useful: he created own VM runner like virtme-ng
  -> Will introudce virtme-ng script


[*] I would like to volunteer to take the role to look into the results.
[**] I checked later and confirmed that DEBUG_ATOMIC_SLEEP is already enabled.
     Ref: current config:
      https://github.com/linux-blktests/blktests-ci/blob/be781155bacf49bf318ee8859572fb7ca1f72aea/playbooks/roles/kernel-builder-k8s-job/templates/build-kernel.sh#L22

----------------------------------------------------------------------------

I also had good talks in hallways, and got two new ideas to improve blktests:
1) "machine readable result reports" and 2) "command tracing" for easier test
case debug. I plan to work on them.

Thanks again for all of the discussion!