From: jgunthorpe@obsidianresearch.com (Jason Gunthorpe)
Subject: Unexpected issues with 2 NVME initiators using the same target
Date: Tue, 27 Jun 2017 10:28:00 -0600 [thread overview]
Message-ID: <20170627162800.GA22592@obsidianresearch.com> (raw)
In-Reply-To: <52ad3547-efcf-f428-6b39-117efda3379f@grimberg.me>
On Tue, Jun 27, 2017@07:07:08PM +0300, Sagi Grimberg wrote:
>
> >Go back and browse the end of the thread: there's no need to change
> >xprtrdma, and maybe no need to change the others either.
>
> I think there is, even with inline, xprtrdma dma maps the immediate
> buffers, also the message head and tail so unmapping these buffers
> without waiting for the send completion would trigger a IOMMU access
> error (the HCA (re)tries to access an already unmapped buffer).
Yes, that is an excellent observation. When using the local rdma lkey
you still need to ensure the linux API DMA map remains until
completion.
send completion mitigation is still possible, if it is OK for the
backing pages to remain, but I think a more sophisticated strategy is
needed - eg maybe push some kind of NOP through the send q after a
timer or only complete when the last available work is stuffed or
something.
Jason
WARNING: multiple messages have this Message-ID (diff)
From: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
To: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>,
Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>,
Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org>,
Marta Rybczynska
<mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org>,
Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>,
"Gruher,
Joseph R"
<joseph.r.gruher-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>,
"shahar.salzman"
<shahar.salzman-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
Laurence Oberman
<loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
"Riches Jr,
Robert M"
<robert.m.riches.jr-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>,
linux-rdma <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
Liran Liss <liranl-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
Bart Van Assche
<bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Subject: Re: Unexpected issues with 2 NVME initiators using the same target
Date: Tue, 27 Jun 2017 10:28:00 -0600 [thread overview]
Message-ID: <20170627162800.GA22592@obsidianresearch.com> (raw)
In-Reply-To: <52ad3547-efcf-f428-6b39-117efda3379f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
On Tue, Jun 27, 2017 at 07:07:08PM +0300, Sagi Grimberg wrote:
>
> >Go back and browse the end of the thread: there's no need to change
> >xprtrdma, and maybe no need to change the others either.
>
> I think there is, even with inline, xprtrdma dma maps the immediate
> buffers, also the message head and tail so unmapping these buffers
> without waiting for the send completion would trigger a IOMMU access
> error (the HCA (re)tries to access an already unmapped buffer).
Yes, that is an excellent observation. When using the local rdma lkey
you still need to ensure the linux API DMA map remains until
completion.
send completion mitigation is still possible, if it is OK for the
backing pages to remain, but I think a more sophisticated strategy is
needed - eg maybe push some kind of NOP through the send q after a
timer or only complete when the last available work is stuffed or
something.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2017-06-27 16:28 UTC|newest]
Thread overview: 171+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-02-21 19:38 Unexpected issues with 2 NVME initiators using the same target shahar.salzman
2017-02-21 22:50 ` Sagi Grimberg
2017-02-21 22:50 ` Sagi Grimberg
2017-02-22 16:52 ` Laurence Oberman
2017-02-22 16:52 ` Laurence Oberman
2017-02-22 19:39 ` Sagi Grimberg
2017-02-22 19:39 ` Sagi Grimberg
2017-02-26 8:03 ` shahar.salzman
2017-02-26 8:03 ` shahar.salzman
2017-02-26 17:58 ` Gruher, Joseph R
2017-02-27 20:33 ` Sagi Grimberg
2017-02-27 20:33 ` Sagi Grimberg
2017-02-27 20:57 ` Gruher, Joseph R
2017-03-05 18:23 ` Leon Romanovsky
2017-03-06 0:07 ` Max Gurtovoy
2017-03-06 11:28 ` Sagi Grimberg
2017-03-06 11:28 ` Sagi Grimberg
2017-03-07 9:27 ` Max Gurtovoy
2017-03-07 13:41 ` Sagi Grimberg
2017-03-07 13:41 ` Sagi Grimberg
2017-03-09 12:18 ` shahar.salzman
2017-03-09 12:18 ` shahar.salzman
2017-03-12 12:33 ` Vladimir Neyelov
2017-03-13 9:43 ` Sagi Grimberg
2017-03-13 9:43 ` Sagi Grimberg
2017-03-14 8:55 ` Max Gurtovoy
2017-03-14 19:57 ` Gruher, Joseph R
2017-03-14 23:42 ` Gruher, Joseph R
2017-03-16 0:03 ` Gruher, Joseph R
2017-03-17 18:37 ` Gruher, Joseph R
2017-03-17 19:49 ` Max Gurtovoy
[not found] ` <DE927C68B458BE418D582EC97927A928550391C2@ORSMSX113.amr.corp.intel.com>
2017-03-24 18:30 ` Gruher, Joseph R
2017-03-27 14:17 ` Max Gurtovoy
2017-03-27 15:39 ` Gruher, Joseph R
2017-03-28 8:38 ` Max Gurtovoy
2017-03-28 10:21 ` shahar.salzman
2017-03-28 11:34 ` Sagi Grimberg
2017-03-28 11:34 ` Sagi Grimberg
2017-04-10 11:40 ` Marta Rybczynska
2017-04-10 11:40 ` Marta Rybczynska
2017-04-10 14:09 ` Max Gurtovoy
2017-04-11 12:47 ` Marta Rybczynska
2017-04-11 12:47 ` Marta Rybczynska
2017-04-20 10:18 ` Sagi Grimberg
2017-04-20 10:18 ` Sagi Grimberg
2017-04-26 11:56 ` Max Gurtovoy
2017-04-26 14:45 ` Sagi Grimberg
2017-04-26 14:45 ` Sagi Grimberg
2017-05-12 19:20 ` Gruher, Joseph R
2017-05-15 12:00 ` Sagi Grimberg
2017-05-15 12:00 ` Sagi Grimberg
2017-05-15 13:31 ` Leon Romanovsky
2017-05-15 13:31 ` Leon Romanovsky
2017-05-15 13:43 ` Sagi Grimberg
2017-05-15 13:43 ` Sagi Grimberg
2017-05-15 14:36 ` Leon Romanovsky
2017-05-15 14:36 ` Leon Romanovsky
2017-05-15 14:59 ` Christoph Hellwig
2017-05-15 14:59 ` Christoph Hellwig
2017-05-15 17:05 ` Leon Romanovsky
2017-05-15 17:05 ` Leon Romanovsky
2017-05-17 12:56 ` Marta Rybczynska
2017-05-17 12:56 ` Marta Rybczynska
2017-05-18 13:34 ` Leon Romanovsky
2017-05-18 13:34 ` Leon Romanovsky
2017-06-19 17:21 ` Robert LeBlanc
2017-06-19 17:21 ` Robert LeBlanc
2017-06-20 6:39 ` Sagi Grimberg
2017-06-20 6:39 ` Sagi Grimberg
2017-06-20 7:46 ` Leon Romanovsky
2017-06-20 7:46 ` Leon Romanovsky
2017-06-20 7:58 ` Sagi Grimberg
2017-06-20 7:58 ` Sagi Grimberg
2017-06-20 8:33 ` Leon Romanovsky
2017-06-20 8:33 ` Leon Romanovsky
2017-06-20 9:33 ` Sagi Grimberg
2017-06-20 9:33 ` Sagi Grimberg
2017-06-20 10:31 ` Max Gurtovoy
2017-06-20 22:58 ` Robert LeBlanc
2017-06-20 22:58 ` Robert LeBlanc
2017-06-27 7:16 ` Sagi Grimberg
2017-06-27 7:16 ` Sagi Grimberg
2017-06-20 12:02 ` Sagi Grimberg
2017-06-20 12:02 ` Sagi Grimberg
2017-06-20 13:28 ` Max Gurtovoy
2017-06-20 17:01 ` Chuck Lever
2017-06-20 17:01 ` Chuck Lever
2017-06-20 17:12 ` Sagi Grimberg
2017-06-20 17:12 ` Sagi Grimberg
2017-06-20 17:35 ` Jason Gunthorpe
2017-06-20 17:35 ` Jason Gunthorpe
2017-06-20 18:17 ` Chuck Lever
2017-06-20 18:17 ` Chuck Lever
2017-06-20 19:27 ` Jason Gunthorpe
2017-06-20 19:27 ` Jason Gunthorpe
2017-06-20 20:56 ` Chuck Lever
2017-06-20 20:56 ` Chuck Lever
2017-06-20 21:19 ` Jason Gunthorpe
2017-06-20 21:19 ` Jason Gunthorpe
2017-06-27 7:37 ` Sagi Grimberg
2017-06-27 7:37 ` Sagi Grimberg
2017-06-27 14:42 ` Chuck Lever
2017-06-27 14:42 ` Chuck Lever
2017-06-27 16:07 ` Sagi Grimberg
2017-06-27 16:07 ` Sagi Grimberg
2017-06-27 16:28 ` Jason Gunthorpe [this message]
2017-06-27 16:28 ` Jason Gunthorpe
2017-06-28 7:03 ` Sagi Grimberg
2017-06-28 7:03 ` Sagi Grimberg
2017-06-27 16:28 ` Chuck Lever
2017-06-27 16:28 ` Chuck Lever
2017-06-28 7:08 ` Sagi Grimberg
2017-06-28 7:08 ` Sagi Grimberg
2017-06-28 16:11 ` Chuck Lever
2017-06-28 16:11 ` Chuck Lever
2017-06-29 5:35 ` Sagi Grimberg
2017-06-29 5:35 ` Sagi Grimberg
2017-06-29 14:55 ` Chuck Lever
2017-06-29 14:55 ` Chuck Lever
2017-07-02 9:45 ` Sagi Grimberg
2017-07-02 9:45 ` Sagi Grimberg
2017-07-02 18:17 ` Chuck Lever
2017-07-02 18:17 ` Chuck Lever
2017-07-09 16:47 ` Jason Gunthorpe
2017-07-09 16:47 ` Jason Gunthorpe
2017-07-10 19:03 ` Chuck Lever
2017-07-10 19:03 ` Chuck Lever
2017-07-10 20:05 ` Jason Gunthorpe
2017-07-10 20:05 ` Jason Gunthorpe
2017-07-10 20:51 ` Chuck Lever
2017-07-10 20:51 ` Chuck Lever
2017-07-10 21:14 ` Jason Gunthorpe
2017-07-10 21:14 ` Jason Gunthorpe
2017-07-10 21:24 ` Jason Gunthorpe
2017-07-10 21:24 ` Jason Gunthorpe
2017-07-10 21:29 ` Chuck Lever
2017-07-10 21:29 ` Chuck Lever
2017-07-10 21:32 ` Jason Gunthorpe
2017-07-10 21:32 ` Jason Gunthorpe
2017-07-10 22:04 ` Chuck Lever
2017-07-10 22:04 ` Chuck Lever
2017-07-10 22:09 ` Jason Gunthorpe
2017-07-10 22:09 ` Jason Gunthorpe
2017-07-11 3:57 ` Chuck Lever
2017-07-11 3:57 ` Chuck Lever
2017-07-11 13:23 ` Tom Talpey
2017-07-11 13:23 ` Tom Talpey
2017-07-11 14:55 ` Chuck Lever
2017-07-11 14:55 ` Chuck Lever
2017-06-27 18:08 ` Bart Van Assche
2017-06-27 18:08 ` Bart Van Assche
2017-06-27 18:14 ` Jason Gunthorpe
2017-06-27 18:14 ` Jason Gunthorpe
2017-06-28 7:16 ` Sagi Grimberg
2017-06-28 7:16 ` Sagi Grimberg
2017-06-28 9:43 ` Bart Van Assche
2017-06-28 9:43 ` Bart Van Assche
2017-06-20 17:08 ` Robert LeBlanc
2017-06-20 17:08 ` Robert LeBlanc
2017-06-20 17:19 ` Sagi Grimberg
2017-06-20 17:19 ` Sagi Grimberg
2017-06-20 17:28 ` Robert LeBlanc
2017-06-20 17:28 ` Robert LeBlanc
2017-06-27 7:22 ` Sagi Grimberg
2017-06-27 7:22 ` Sagi Grimberg
2017-06-20 14:43 ` Robert LeBlanc
2017-06-20 14:43 ` Robert LeBlanc
2017-06-20 14:41 ` Robert LeBlanc
2017-06-20 14:41 ` Robert LeBlanc
2017-02-27 20:13 ` Sagi Grimberg
2017-02-27 20:13 ` Sagi Grimberg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170627162800.GA22592@obsidianresearch.com \
--to=jgunthorpe@obsidianresearch.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.