Re: broken CRCs at NVMeF target with SIW & NVMe/TCP transports

From: Krishnamraju Eraparaju <krishna2@chelsio.com>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: linux-rdma@vger.kernel.org,
	Potnuri Bharat Teja <bharat@chelsio.com>,
	Nirranjan Kirubaharan <nirranjan@chelsio.com>,
	linux-nvme@lists.infradead.org,
	Bernard Metzler <BMT@zurich.ibm.com>,
	Christoph Hellwig <hch@lst.de>
Subject: Re: broken CRCs at NVMeF target with SIW & NVMe/TCP transports
Date: Sat, 21 Mar 2020 09:32:09 +0530	[thread overview]
Message-ID: <20200321040207.GA11659@chelsio.com> (raw)
In-Reply-To: <87bfe03d-baad-1166-14a1-6eba1739fde4@grimberg.me>

On Friday, March 03/20/20, 2020 at 13:49:25 -0700, Sagi Grimberg wrote:
> 
> >>I assume this happens with iSCSI as well? There is nothing special
> >>we are doing with respect to digest.
> >
> >I don't see this issue with iscsi-tcp.
> >
> >May be blk-mq is causing this issue? I assume iscsi-tcp does not have
> >blk_mq support yet upstream to verify with blk_mq enabled.
> >I tried on Ubuntu 19.10(which is based on Linux kernel 5.3), note that
> >RHEL does not support DataDigest.
> >
> >The reason that I'm seeing this issue only with NVMe(tcp/softiwarp) &
> >iSER(softiwarp) is becuase of NVMeF&ISER using blk-mq?
> >
> >Anyhow, I see the content of the page is being updated by upper layers
> >while the tranport driver is computing CRC on that page content and
> >this needs a fix.
> 
> Krishna, do you happen to run with nvme multipath enabled?

Yes Sagi, issue occurs with nvme multipath enabled also..

dmesg at initiator:
[ +10.671996] EXT4-fs (nvme0n1): mounting ext3 file system using the
ext4 subsystem
[  +0.004643] EXT4-fs (nvme0n1): mounted filesystem with ordered data
mode. Opts: (null)
[ +15.955424] block nvme0n1: no usable path - requeuing I/O
[  +0.000142] block nvme0n1: no usable path - requeuing I/O
[  +0.000135] block nvme0n1: no usable path - requeuing I/O
[  +0.000119] block nvme0n1: no usable path - requeuing I/O
[  +0.000108] block nvme0n1: no usable path - requeuing I/O
[  +0.000111] block nvme0n1: no usable path - requeuing I/O
[  +0.000118] block nvme0n1: no usable path - requeuing I/O
[  +0.000158] block nvme0n1: no usable path - requeuing I/O
[  +0.000130] block nvme0n1: no usable path - requeuing I/O
[  +0.000138] block nvme0n1: no usable path - requeuing I/O
[  +0.011754] nvme nvme0: Reconnecting in 10 seconds...
[ +10.261223] nvme_ns_head_make_request: 5 callbacks suppressed
[  +0.000002] block nvme0n1: no usable path - requeuing I/O
[  +0.000240] block nvme0n1: no usable path - requeuing I/O
[  +0.000107] block nvme0n1: no usable path - requeuing I/O
[  +0.000107] block nvme0n1: no usable path - requeuing I/O
[  +0.000107] block nvme0n1: no usable path - requeuing I/O
[  +0.000108] block nvme0n1: no usable path - requeuing I/O
[  +0.000132] block nvme0n1: no usable path - requeuing I/O
[  +0.000010] nvme nvme0: creating 12 I/O queues.
[  +0.000110] block nvme0n1: no usable path - requeuing I/O
[  +0.000232] block nvme0n1: no usable path - requeuing I/O
[  +0.000122] block nvme0n1: no usable path - requeuing I/O
[  +0.008407] nvme nvme0: Successfully reconnected (1 attempt)

dmesg at target:
[Mar21 09:24] nvmet_tcp: queue 3: cmd 38 pdu (6) data digest error: recv
0x21e59730 expected 0x2b88fed0
[  +0.000029] nvmet: ctrl 1 fatal error occurred!
[ +10.280101] nvmet: creating controller 1 for subsystem nvme-ram0 for
NQN nqn.2014-08.org.nvmexpress.chelsio.

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme