Re: [PATCH blktests v2 00/11] support test case repeat by different conditions

Sagi Grimberg
2024-04-18T09:40:35Z

On 16/04/2024 13:31, Shin'ichiro Kawasaki wrote:
> In the recent discussion for nvme test group [1], two pain points were mentioned
> regarding the test case runs.
>
> 1) Several test cases in nvme test group do exactly the same test except the
> NVME transport backend set up condition difference (device vs. file). This
> results in duplicate test script codes. It is desired to unify the test cases
> and run them repeatedly with the different conditions.
>
> 2) NVME transport types can be specified with nvme_trtype parameter so that the
> same tests can be run for various transport types. However, some test cases
> do not depend on the transport types. They are repeated in multiple runs for
> the various transport types under the exact same conditions. It is desired to
> repeat the test cases only when such repetition is required.
>
> [1] https://lore.kernel.org/linux-block/w2eaegjopbah5qbjsvpnrwln2t5dr7mv3v4n2e63m5tjqiochm@uonrjm2i2g72/
>
> One idea to address these pain points is to add the test repeat feature to the
> nvme test group. However, Daniel questioned if the feature could be implemented
> in the blktests framework. Actually, a similar feature has already been
> implemented to repeat some test cases for non-zoned block devices and zoned
> block devices. However, this feature is implemented only for the zoned and non-
> zoned device conditions. It can not fulfill the desires for nvme test group.
>
> This series proposes to generalize the feature in the blktests framework to
> repeat the test cases with different conditions. Introduce a new function
> set_conditions() that each test case can define and instruct the framework to
> repeat the test case. The first four patches introduce the feature and apply it
> to the repetition for non-zoned and zoned block devices. The following seven
> patches apply the feature to nvme test group so that the test cases can be
> repeated for NVME transport types and backend types in the ideal way. Two of the
> seven patches are reused from the work by Daniel. The all patches are listed in
> the order that does not lose the test coverage with the default set up.

Nice idea!

> This series introduces new config parameters NVMET_TRTYPES and
> NVMET_BLKDEV_TYPES, which can take multiple values with space separators. When
> they are defined in the config file as follows,
>
> NVMET_TRTYPES="loop rdma tcp"
> NVMET_BLKDEV_TYPES="device file"
>
> the test cases which depend on these parameters are repeated 3 x 2 = 6 times.
> For example, nvme/006 is repeated as follows.
>
> nvme/006 (nvmet bd=device tr=loop) (create an NVMeOF target) [passed]
> runtime 0.148s ... 0.165s
> nvme/006 (nvmet bd=device tr=rdma) (create an NVMeOF target) [passed]
> runtime 0.273s ... 0.235s
> nvme/006 (nvmet bd=device tr=tcp) (create an NVMeOF target) [passed]
> runtime 0.162s ... 0.164s
> nvme/006 (nvmet bd=file tr=loop) (create an NVMeOF target) [passed]
> runtime 0.138s ... 0.134s
> nvme/006 (nvmet bd=file tr=rdma) (create an NVMeOF target) [passed]
> runtime 0.216s ... 0.201s
> nvme/006 (nvmet bd=file tr=tcp) (create an NVMeOF target) [passed]
> runtime 0.154s ... 0.146s
>
>
> Changes from v1:
> * Renamed NVMET_TR_TYPES to NVMET_TRTYPES
> * 1st patch: reflected comments on the list and added Reviewed-by tag
> * 5th patch: changed NVMET_TRTYPES from array to variable
> * 7th patch: changed NVMET_BLKDEV_TYPES from array to variable
> * Reflected other comments on the list
>
>
> Daniel Wagner (3):
> nvme/rc: add blkdev type environment variable
> nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type
> tests
> nvme/{021,022,025,026,027,028}: do not hard code target blkdev type
>
> Shin'ichiro Kawasaki (8):
> check: factor out _run_test()
> check: support test case repeat by different conditions
> check: use set_conditions() for the CAN_BE_ZONED test cases
> meta/{016,017}: add test cases to check repeated test case runs
> nvme/rc: introduce NVMET_TRTYPES
> nvme/rc: introduce NVMET_BLKDEV_TYPES
> nvme/{002-031,033-038,040-045,047,048}: support NMVET_TRTYPES
> nvme/{006,008,010,012,014,019,023}: support NVMET_BLKDEV_TYPES
>
> Documentation/running-tests.md | 16 +++-
> Makefile | 3 +-
> check | 129 ++++++++++++++++++++++-----------
> common/shellcheck | 2 +-
> common/zoned | 22 ++++++
> new | 21 ++++++
> tests/meta/016 | 29 ++++++++
> tests/meta/016.out | 2 +
> tests/meta/017 | 29 ++++++++
> tests/meta/017.out | 2 +
> tests/nvme/002 | 4 +
> tests/nvme/003 | 4 +
> tests/nvme/004 | 4 +
> tests/nvme/005 | 4 +
> tests/nvme/006 | 9 ++-
> tests/nvme/007 | 28 -------
> tests/nvme/007.out | 2 -
> tests/nvme/008 | 8 +-
> tests/nvme/009 | 36 ---------
> tests/nvme/009.out | 3 -
> tests/nvme/010 | 8 +-
> tests/nvme/011 | 39 ----------
> tests/nvme/011.out | 3 -
> tests/nvme/012 | 8 +-
> tests/nvme/013 | 43 -----------
> tests/nvme/013.out | 3 -
> tests/nvme/014 | 8 +-
> tests/nvme/015 | 48 ------------
> tests/nvme/015.out | 4 -
> tests/nvme/016 | 4 +
> tests/nvme/017 | 4 +
> tests/nvme/018 | 4 +
> tests/nvme/019 | 8 +-
> tests/nvme/020 | 40 ----------
> tests/nvme/020.out | 4 -
> tests/nvme/021 | 10 ++-
> tests/nvme/022 | 10 ++-
> tests/nvme/023 | 8 +-
> tests/nvme/024 | 40 ----------
> tests/nvme/024.out | 2 -
> tests/nvme/025 | 10 ++-
> tests/nvme/026 | 10 ++-
> tests/nvme/027 | 10 ++-
> tests/nvme/028 | 10 ++-
> tests/nvme/029 | 4 +
> tests/nvme/030 | 4 +
> tests/nvme/031 | 4 +
> tests/nvme/033 | 4 +
> tests/nvme/034 | 4 +
> tests/nvme/035 | 4 +
> tests/nvme/036 | 4 +
> tests/nvme/037 | 4 +
> tests/nvme/038 | 4 +
> tests/nvme/040 | 4 +
> tests/nvme/041 | 3 +
> tests/nvme/042 | 3 +
> tests/nvme/043 | 3 +
> tests/nvme/044 | 3 +
> tests/nvme/045 | 3 +
> tests/nvme/047 | 4 +
> tests/nvme/048 | 4 +
> tests/nvme/rc | 58 ++++++++++++++-
> 62 files changed, 437 insertions(+), 379 deletions(-)
> create mode 100644 common/zoned
> create mode 100755 tests/meta/016
> create mode 100644 tests/meta/016.out
> create mode 100755 tests/meta/017
> create mode 100644 tests/meta/017.out
> delete mode 100755 tests/nvme/007
> delete mode 100644 tests/nvme/007.out
> delete mode 100755 tests/nvme/009
> delete mode 100644 tests/nvme/009.out
> delete mode 100755 tests/nvme/011
> delete mode 100644 tests/nvme/011.out
> delete mode 100755 tests/nvme/013
> delete mode 100644 tests/nvme/013.out
> delete mode 100755 tests/nvme/015
> delete mode 100644 tests/nvme/015.out
> delete mode 100755 tests/nvme/020
> delete mode 100644 tests/nvme/020.out
> delete mode 100755 tests/nvme/024
> delete mode 100644 tests/nvme/024.out
>