Re: broken CRCs at NVMeF target with SIW & NVMe/TCP transports

From: Krishnamraju Eraparaju <krishna2@chelsio.com>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: linux-rdma@vger.kernel.org,
	Potnuri Bharat Teja <bharat@chelsio.com>,
	Nirranjan Kirubaharan <nirranjan@chelsio.com>,
	linux-nvme@lists.infradead.org,
	Bernard Metzler <BMT@zurich.ibm.com>,
	Christoph Hellwig <hch@lst.de>
Subject: Re: broken CRCs at NVMeF target with SIW & NVMe/TCP transports
Date: Fri, 20 Mar 2020 20:05:47 +0530	[thread overview]
Message-ID: <20200320143544.GA5539@chelsio.com> (raw)
In-Reply-To: <3f42f881-0309-b86a-4b70-af23c58960fc@grimberg.me>

On Wednesday, March 03/18/20, 2020 at 09:49:07 -0700, Sagi Grimberg wrote:
> 
> >>Thanks Krishna,
> >>
> >>I assume that this makes the issue go away?
> >>--
> >>diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> >>index 11e10fe1760f..cc93e1949b2c 100644
> >>--- a/drivers/nvme/host/tcp.c
> >>+++ b/drivers/nvme/host/tcp.c
> >>@@ -889,7 +889,7 @@ static int nvme_tcp_try_send_data(struct
> >>nvme_tcp_request *req)
> >>                         flags |= MSG_MORE;
> >>
> >>                 /* can't zcopy slab pages */
> >>-               if (unlikely(PageSlab(page))) {
> >>+               if (unlikely(PageSlab(page)) || queue->data_digest) {
> >>                         ret = sock_no_sendpage(queue->sock, page,
> >>offset, len,
> >>                                         flags);
> >>                 } else {
> >>--
> >
> >Unfortunately, issue is still occuring with this patch also.
> >
> >Looks like the integrity of the data buffer right after the CRC
> >computation(data digest) is what causing this issue, despite the
> >buffer being sent via sendpage or no_sendpage.
> 
> I assume this happens with iSCSI as well? There is nothing special
> we are doing with respect to digest.

I don't see this issue with iscsi-tcp.

May be blk-mq is causing this issue? I assume iscsi-tcp does not have
blk_mq support yet upstream to verify with blk_mq enabled.
I tried on Ubuntu 19.10(which is based on Linux kernel 5.3), note that
RHEL does not support DataDigest.

The reason that I'm seeing this issue only with NVMe(tcp/softiwarp) &
iSER(softiwarp) is becuase of NVMeF&ISER using blk-mq? 

Anyhow, I see the content of the page is being updated by upper layers
while the tranport driver is computing CRC on that page content and
this needs a fix.

one could very easily recreate this issue running the below simple program over
NVMe/TCP.
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

int main() {
	int i;
	char* line1 = "123";
	FILE* fp;
	while(1) {
		fp = fopen("/mnt/tmp.txt", "w");
		setvbuf(fp, NULL, _IONBF, 0);
		for (i=0; i<100000; i++)
		     if ((fwrite(line1, 1, strlen(line1), fp) !=
strlen(line1)))
			exit(1);

		if (fclose(fp) != 0)
			exit(1);
	}
return 0;
}

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme