All of lore.kernel.org
 help / color / mirror / Atom feed
From: Krishnamraju Eraparaju <krishna2@chelsio.com>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: linux-rdma@vger.kernel.org,
	Potnuri Bharat Teja <bharat@chelsio.com>,
	Nirranjan Kirubaharan <nirranjan@chelsio.com>,
	linux-nvme@lists.infradead.org,
	Bernard Metzler <BMT@zurich.ibm.com>,
	Christoph Hellwig <hch@lst.de>
Subject: Re: broken CRCs at NVMeF target with SIW & NVMe/TCP transports
Date: Wed, 18 Mar 2020 02:01:57 +0530	[thread overview]
Message-ID: <20200317203152.GA14946@chelsio.com> (raw)
In-Reply-To: <38f79fb7-841a-9faa-e1f8-2de4b9f21118@grimberg.me>

On Tuesday, March 03/17/20, 2020 at 12:33:44 -0700, Sagi Grimberg wrote:
> 
> >>>>>For TCP we can set BDI_CAP_STABLE_WRITES.  For RDMA I don't think
> >>>>that
> >>>>>is a good idea as pretty much all RDMA block drivers rely on the
> >>>>>DMA behavior above.  The answer is to bounce buffer the data in
> >>>>>SoftiWARP / SoftRoCE.
> >>>>
> >>>>We already do, see nvme_alloc_ns.
> >>>>
> >>>>
> >>>
> >>>Krishna was getting the issue when testing TCP/NVMeF with -G
> >>>during connect. That enables data digest and STABLE_WRITES
> >>>I think. So to me it seems we don't get stable pages, but
> >>>pages which are touched after handover to the provider.
> >>
> >>Non of the transports modifies the data at any point, both will
> >>scan it to compute crc. So surely this is coming from the fs,
> >>Krishna does this happen with xfs as well?
> >Yes, but rare(took ~15min to recreate), whereas with ext3/4
> >its almost immediate. Here is the error log for NVMe/TCP with xfs.
> 
> Thanks Krishna,
> 
> I assume that this makes the issue go away?
> --
> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> index 11e10fe1760f..cc93e1949b2c 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -889,7 +889,7 @@ static int nvme_tcp_try_send_data(struct
> nvme_tcp_request *req)
>                         flags |= MSG_MORE;
> 
>                 /* can't zcopy slab pages */
> -               if (unlikely(PageSlab(page))) {
> +               if (unlikely(PageSlab(page)) || queue->data_digest) {
>                         ret = sock_no_sendpage(queue->sock, page,
> offset, len,
>                                         flags);
>                 } else {
> --

Unfortunately, issue is still occuring with this patch also.

Looks like the integrity of the data buffer right after the CRC
computation(data digest) is what causing this issue, despite the
buffer being sent via sendpage or no_sendpage.

Thanks,
Krishna.

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

WARNING: multiple messages have this Message-ID (diff)
From: Krishnamraju Eraparaju <krishna2@chelsio.com>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: Bernard Metzler <BMT@zurich.ibm.com>,
	Christoph Hellwig <hch@lst.de>,
	linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
	Nirranjan Kirubaharan <nirranjan@chelsio.com>,
	Potnuri Bharat Teja <bharat@chelsio.com>
Subject: Re: broken CRCs at NVMeF target with SIW & NVMe/TCP transports
Date: Wed, 18 Mar 2020 02:01:57 +0530	[thread overview]
Message-ID: <20200317203152.GA14946@chelsio.com> (raw)
In-Reply-To: <38f79fb7-841a-9faa-e1f8-2de4b9f21118@grimberg.me>

On Tuesday, March 03/17/20, 2020 at 12:33:44 -0700, Sagi Grimberg wrote:
> 
> >>>>>For TCP we can set BDI_CAP_STABLE_WRITES.  For RDMA I don't think
> >>>>that
> >>>>>is a good idea as pretty much all RDMA block drivers rely on the
> >>>>>DMA behavior above.  The answer is to bounce buffer the data in
> >>>>>SoftiWARP / SoftRoCE.
> >>>>
> >>>>We already do, see nvme_alloc_ns.
> >>>>
> >>>>
> >>>
> >>>Krishna was getting the issue when testing TCP/NVMeF with -G
> >>>during connect. That enables data digest and STABLE_WRITES
> >>>I think. So to me it seems we don't get stable pages, but
> >>>pages which are touched after handover to the provider.
> >>
> >>Non of the transports modifies the data at any point, both will
> >>scan it to compute crc. So surely this is coming from the fs,
> >>Krishna does this happen with xfs as well?
> >Yes, but rare(took ~15min to recreate), whereas with ext3/4
> >its almost immediate. Here is the error log for NVMe/TCP with xfs.
> 
> Thanks Krishna,
> 
> I assume that this makes the issue go away?
> --
> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> index 11e10fe1760f..cc93e1949b2c 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -889,7 +889,7 @@ static int nvme_tcp_try_send_data(struct
> nvme_tcp_request *req)
>                         flags |= MSG_MORE;
> 
>                 /* can't zcopy slab pages */
> -               if (unlikely(PageSlab(page))) {
> +               if (unlikely(PageSlab(page)) || queue->data_digest) {
>                         ret = sock_no_sendpage(queue->sock, page,
> offset, len,
>                                         flags);
>                 } else {
> --

Unfortunately, issue is still occuring with this patch also.

Looks like the integrity of the data buffer right after the CRC
computation(data digest) is what causing this issue, despite the
buffer being sent via sendpage or no_sendpage.

Thanks,
Krishna.

  reply	other threads:[~2020-03-17 20:32 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-16 16:20 broken CRCs at NVMeF target with SIW & NVMe/TCP transports Krishnamraju Eraparaju
2020-03-16 16:20 ` Krishnamraju Eraparaju
2020-03-17  9:31 ` Bernard Metzler
2020-03-17  9:31   ` Bernard Metzler
2020-03-17 12:26   ` Tom Talpey
2020-03-17 12:26     ` Tom Talpey
2020-03-17 12:45 ` Christoph Hellwig
2020-03-17 12:45   ` Christoph Hellwig
2020-03-17 13:17   ` Bernard Metzler
2020-03-17 13:17     ` Bernard Metzler
2020-03-17 16:03   ` Sagi Grimberg
2020-03-17 16:03     ` Sagi Grimberg
2020-03-17 16:29     ` Bernard Metzler
2020-03-17 16:29       ` Bernard Metzler
2020-03-17 16:39       ` Sagi Grimberg
2020-03-17 16:39         ` Sagi Grimberg
2020-03-17 19:17         ` Krishnamraju Eraparaju
2020-03-17 19:17           ` Krishnamraju Eraparaju
2020-03-17 19:33           ` Sagi Grimberg
2020-03-17 19:33             ` Sagi Grimberg
2020-03-17 20:31             ` Krishnamraju Eraparaju [this message]
2020-03-17 20:31               ` Krishnamraju Eraparaju
2020-03-18 16:49               ` Sagi Grimberg
2020-03-18 16:49                 ` Sagi Grimberg
2020-03-20 14:35                 ` Krishnamraju Eraparaju
2020-03-20 14:35                   ` Krishnamraju Eraparaju
2020-03-20 20:49                   ` Sagi Grimberg
2020-03-20 20:49                     ` Sagi Grimberg
2020-03-21  4:02                     ` Krishnamraju Eraparaju
2020-03-21  4:02                       ` Krishnamraju Eraparaju

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200317203152.GA14946@chelsio.com \
    --to=krishna2@chelsio.com \
    --cc=BMT@zurich.ibm.com \
    --cc=bharat@chelsio.com \
    --cc=hch@lst.de \
    --cc=linux-nvme@lists.infradead.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=nirranjan@chelsio.com \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.