Re: Flush warning

Sagi Grimberg
2017-08-07T01:06:38Z

> Hey guys,

Hey Steve,

> We're seeing a WARNING happening when running an fio test on a single NVMF
> attached ramdisk over iw_cxgb4. While the fio test is running, the NVMF host is
> also killing the controller via writing to
> /sys/block/nvme*/device/reset_controller. Here is the script:
>
> ----
> [root@trinitycraft ~]# cat fio_issue.sh
> num=0
>
> fio --rw=randrw --name=random --norandommap --ioengine=libaio --size=400m
> --group_reporting --exitall --fsync_on_close=1 --invalidate=1 --direct=1
> --filename=/dev/nvme0n1 --time_based --runtime=30 --iodepth=32 --numjobs=8
> --unit_base=1 --bs=4k --kb_base=1000 &
>
> sleep 2
> while [ $num -lt 30 ]
> do
> echo 1 >/sys/block/nvme0n1/device/reset_controller
> [ $? -eq 1 ] && echo "reset_controller operation failed: $num" && exit 1
> ((num++))
> sleep 0.5
> done
> -----
>
> The WARNING seems to be due to nvmet_rdma_queue_connect() calling
> flush_scheduled_work() while in the upcall from the RDMA_CM. It I running on
> the iw_cm event workqueue, which is created with WQ_MEM_RECLAIM set. I'm not
> sure what this WARNING is telling me. Does the iw_cm workqueue NOT need
> WQ_MEM_RECLAIM set? Or is there some other issue with the nvmet/rdma code doing
> work flushing in the iw_cm workq context?

This flush is designed to prevent nvmet-rdma from having too much
inflight resources in case of a high pace of controller teardown and
establishment (like you trigger in your test).

queue teardowns are run on system_wq, does iw_cm needs memory
reclamation protection?