Re: [PATCH blktests 5/5] nvme/070: test for ABA ghost writes on a multipath fabrics namespace

[Mohamed Khalfella <mkhalfella@purestorage.com>]

On Tue 2026-09-22 13:07:04 -0400, Jesse Taube wrote:
> On Wed, Sep 16, 2026 at 10:09 PM Mohamed Khalfella
> <mkhalfella@purestorage.com> wrote:
> >
> > An unacknowledged write that is retried on another path can still be
> > alive in the fabric. If it reaches the target after a later write to the
> > same LBA has landed, it overwrites it, and a read returns stale data.
> > Nothing in the tree exercises that window.
> >
> > Add a test that builds it deliberately. A ublk loop device backs an
> > nvmet namespace exported through two ports, and the host connects to
> > both, so nvme-multipath has a second path to fail over to. io_timeout on
> > the subsystem drops to 2 seconds, then miniublk's inject command holds
> > one write in the backstore for 4 seconds. The host times that write out,
> > retries it on the other path, and the held write completes at the target
> > afterwards.
> >
> > nvme-ghost-write-detector then writes distinct patterns to a single LBA
> > and reads the block back, so a resurfaced write shows up as the wrong
> > pattern.
> >
> > The test requires nvme_core.multipath=Y and a fabrics transport. As of
> > today it passes on loop, which defines no timeout callback and so never
> > times the write out and never retries it, and fails on tcp, rdma and fc.
> 
> Can you share a tcp or fc variant of this test that demonstrates failure?

Okay, will do that in next revision.

> 
> >
> > Signed-off-by: Mohamed Khalfella <mkhalfella@purestorage.com>
> 
> Reviewed-by: Jesse Taube <jtaubepe@redhat.com>
> Tested-by: Jesse Taube <jtaubepe@redhat.com>

Thanks for testing and reviewing the change.

> 
> > ---
> >  tests/nvme/070     | 98 ++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/nvme/070.out | 35 +++++++++++++++++
> >  2 files changed, 133 insertions(+)
> >  create mode 100755 tests/nvme/070
> >  create mode 100644 tests/nvme/070.out
> 
> btw this needs to be bumped to 071 as 070 got added recently.

Noted.

> 
> Thanks,
> Jesse Taube
> 
> >
> > diff --git a/tests/nvme/070 b/tests/nvme/070
> > new file mode 100755
> > index 0000000..e9a4690
> > --- /dev/null
> > +++ b/tests/nvme/070
> > @@ -0,0 +1,98 @@
> > +#!/bin/bash
> > +# SPDX-License-Identifier: GPL-3.0+
> > +# Copyright (C) 2026 Mohamed Khalfella
> > +
> > +. tests/nvme/rc
> > +. common/ublk
> > +
> > +DESCRIPTION="Test injecting delay on nvme-target backstore and expect no corruption"
> > +
> > +requires() {
> > +       _nvme_requires
> > +       _have_loop
> > +       _have_ublk
> > +       _have_module_param_value nvme_core multipath Y
> > +       _require_nvme_trtype_is_fabrics
> > +       _have_src_program nvme-ghost-write-detector
> > +}
> > +
> > +set_conditions() {
> > +       _set_nvme_trtype "$@"
> > +}
> > +
> > +count_paths_to_subsystem() {
> > +       local subsysnqn="$1"
> > +       local dev count
> > +
> > +       count=0
> > +       for dev in /sys/class/nvme/nvme*; do
> > +               [[ -e "${dev}/subsysnqn" ]] || continue
> > +               [[ "$(cat "${dev}/subsysnqn")" == "${subsysnqn}" ]] || continue
> > +               count=$(( count + 1 ))
> > +       done
> > +       echo "${count}"
> > +}
> > +
> > +set_io_timeout_of_subsystem() {
> > +       local subsysnqn="$1"
> > +       local timeout="$2"
> > +       local dev
> > +
> > +       for dev in /sys/class/nvme/nvme*; do
> > +               [[ -e "${dev}/subsysnqn" ]] || continue
> > +               [[ "$(cat "${dev}/subsysnqn")" == "${subsysnqn}" ]] || continue
> > +               if ! echo "${timeout}" > "${dev}/io_timeout" 2> /dev/null; then
> > +                       echo "FAIL: can not set io_timeout on ${dev##*/}"
> > +                       return 1
> > +               fi
> > +       done
> > +}
> > +
> > +test() {
> > +       echo "Running ${TEST_NAME}"
> > +
> > +       local ns port nr_paths
> > +       local -a ports
> > +
> > +       if ! _init_ublk; then
> > +               return 1
> > +       fi
> > +
> > +       truncate -s "${NVME_IMG_SIZE}" "${TMPDIR}/ublk-img"
> > +       if ! ${UBLK_PROG} add -t loop -f "${TMPDIR}/ublk-img" -n 0 > "$FULL" 2>&1; then
> > +               echo "fail to add ublk device"
> > +               _exit_ublk
> > +               return 1
> > +       fi
> > +       udevadm settle
> > +
> > +       _setup_nvmet
> > +       _nvmet_target_setup --ports 2 --blkdev none
> > +       _create_nvmet_ns --blkdev /dev/ublkb0 \
> > +               --uuid "${def_subsys_uuid}" > /dev/null
> > +
> > +       _get_nvmet_ports "${def_subsysnqn}" ports
> > +       echo "Target ports: ${#ports[@]}"
> > +       for port in "${ports[@]}"; do
> > +               _nvme_connect_subsys --port "${port}"
> > +       done
> > +
> > +       nr_paths=$(count_paths_to_subsystem "${def_subsysnqn}")
> > +       if ((  nr_paths != 2 )); then
> > +               echo "FAIL: expected 2 paths, found ${nr_paths}"
> > +       fi
> > +
> > +       set_io_timeout_of_subsystem "${def_subsysnqn}" 2000
> > +
> > +       if ! ${UBLK_PROG} inject -n 0 -o write -d 4 -c 1 >> "$FULL" 2>&1; then
> > +               echo "FAIL: can not inject write delay"
> > +       fi
> > +
> > +       ns=$(_find_nvme_ns "${def_subsys_uuid}")
> > +       "$SRCDIR/nvme-ghost-write-detector" "/dev/${ns}"
> > +
> > +       _nvme_disconnect_subsys
> > +       _nvmet_target_cleanup
> > +       _exit_ublk
> > +       echo "Test complete"
> > +}
> > diff --git a/tests/nvme/070.out b/tests/nvme/070.out
> > new file mode 100644
> > index 0000000..b43fec7
> > --- /dev/null
> > +++ b/tests/nvme/070.out
> > @@ -0,0 +1,35 @@
> > +Running nvme/070
> > +Target ports: 2
> > +starting nvme-ghost-write-detector test program
> > +iteration number 0, writing data
> > +validating written data
> > +successfully validated
> > +iteration number 1, writing data
> > +validating written data
> > +successfully validated
> > +iteration number 2, writing data
> > +validating written data
> > +successfully validated
> > +iteration number 3, writing data
> > +validating written data
> > +successfully validated
> > +iteration number 4, writing data
> > +validating written data
> > +successfully validated
> > +iteration number 5, writing data
> > +validating written data
> > +successfully validated
> > +iteration number 6, writing data
> > +validating written data
> > +successfully validated
> > +iteration number 7, writing data
> > +validating written data
> > +successfully validated
> > +iteration number 8, writing data
> > +validating written data
> > +successfully validated
> > +iteration number 9, writing data
> > +validating written data
> > +successfully validated
> > +finished nvme-ghost-write-detector test program
> > +Test complete
> > --
> > 2.55.0
> >
>