From: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
To: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>,
Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>,
Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org>,
Marta Rybczynska
<mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org>,
Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>,
"Gruher,
Joseph R"
<joseph.r.gruher-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>,
"shahar.salzman"
<shahar.salzman-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
Laurence Oberman
<loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
"Riches Jr,
Robert M"
<robert.m.riches.jr-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>,
linux-rdma <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
Liran Liss <liranl-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
Bart Van Assche
<bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Subject: Re: Unexpected issues with 2 NVME initiators using the same target
Date: Tue, 27 Jun 2017 10:28:00 -0600 [thread overview]
Message-ID: <20170627162800.GA22592@obsidianresearch.com> (raw)
In-Reply-To: <52ad3547-efcf-f428-6b39-117efda3379f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
On Tue, Jun 27, 2017 at 07:07:08PM +0300, Sagi Grimberg wrote:
>
> >Go back and browse the end of the thread: there's no need to change
> >xprtrdma, and maybe no need to change the others either.
>
> I think there is, even with inline, xprtrdma dma maps the immediate
> buffers, also the message head and tail so unmapping these buffers
> without waiting for the send completion would trigger a IOMMU access
> error (the HCA (re)tries to access an already unmapped buffer).
Yes, that is an excellent observation. When using the local rdma lkey
you still need to ensure the linux API DMA map remains until
completion.
send completion mitigation is still possible, if it is OK for the
backing pages to remain, but I think a more sophisticated strategy is
needed - eg maybe push some kind of NOP through the send q after a
timer or only complete when the last available work is stuffed or
something.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2017-06-27 16:28 UTC|newest]
Thread overview: 74+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <08131a05-1f56-ef61-990a-7fff04eea095@gmail.com>
[not found] ` <08131a05-1f56-ef61-990a-7fff04eea095-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2017-02-21 22:50 ` Unexpected issues with 2 NVME initiators using the same target Sagi Grimberg
[not found] ` <de1a559a-bf24-0d73-5fc7-148d6cd4d4e0-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-02-22 16:52 ` Laurence Oberman
[not found] ` <1848296658.37025722.1487782361271.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2017-02-22 19:39 ` Sagi Grimberg
2017-02-26 8:03 ` shahar.salzman
[not found] ` <1554c1d1-6bf4-9ca2-12d4-a0125d8c5715-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2017-02-27 20:13 ` Sagi Grimberg
[not found] ` <DE927C68B458BE418D582EC97927A92854655137@ORSMSX112.amr.corp.intel.com>
[not found] ` <DE927C68B458BE418D582EC97927A92854655137-8oqHQFITsIFcIJlls4ac1rfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2017-02-27 20:33 ` Sagi Grimberg
[not found] ` <26912d0c-578f-26e9-490d-94fc95bdf259@mellanox.com>
[not found] ` <26912d0c-578f-26e9-490d-94fc95bdf259-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2017-03-06 11:28 ` Sagi Grimberg
[not found] ` <fbd647dd-3a16-8155-107d-f98e8326cc63@mellanox.com>
[not found] ` <fbd647dd-3a16-8155-107d-f98e8326cc63-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2017-03-07 13:41 ` Sagi Grimberg
[not found] ` <c5a6f55b-a633-53ca-4378-7a7eaf7f77bd-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-03-09 12:18 ` shahar.salzman
[not found] ` <AM4PR0501MB278621363209E177A738D75FCB220@AM4PR0501MB2786.eurprd05.prod.outlook.com>
[not found] ` <AM4PR0501MB278621363209E177A738D75FCB220-dp/nxUn679hhbxXPg6FtWcDSnupUy6xnnBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>
2017-03-13 9:43 ` Sagi Grimberg
[not found] ` <DE927C68B458BE418D582EC97927A92855031724@ORSMSX113.amr.corp.intel.com>
[not found] ` <809f87ab-b787-9d40-5840-07500d12e81a@mellanox.com>
[not found] ` <DE927C68B458BE418D582EC97927A928550391C2@ORSMSX113.amr.corp.intel.com>
[not found] ` <DE927C68B458BE418D582EC97927A928550419FA@ORSMSX113.amr.corp.intel.com>
[not found] ` <DE927C68B458BE418D582EC97927A928550419FA-8oqHQFITsIFQxe9IK+vIArfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2017-03-28 11:34 ` Sagi Grimberg
[not found] ` <809f87ab-b787-9d40-5840-07500d12e81a-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2017-04-10 11:40 ` Marta Rybczynska
[not found] ` <33e2cc35-f147-d4a4-9a42-8f1245e35842@mellanox.com>
[not found] ` <33e2cc35-f147-d4a4-9a42-8f1245e35842-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2017-04-11 12:47 ` Marta Rybczynska
2017-04-20 10:18 ` Sagi Grimberg
[not found] ` <af9044c2-fb7c-e8b3-d8fc-4874cfd1bb67@mellanox.com>
[not found] ` <af9044c2-fb7c-e8b3-d8fc-4874cfd1bb67-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2017-04-26 14:45 ` Sagi Grimberg
[not found] ` <DE927C68B458BE418D582EC97927A92855088C6F@ORSMSX113.amr.corp.intel.com>
[not found] ` <DE927C68B458BE418D582EC97927A92855088C6F-8oqHQFITsIFQxe9IK+vIArfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2017-05-15 12:00 ` Sagi Grimberg
[not found] ` <82dd5b24-5657-ae5e-8a33-646fddd8b75b-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-05-15 13:31 ` Leon Romanovsky
[not found] ` <20170515133122.GG3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-05-15 13:43 ` Sagi Grimberg
[not found] ` <9465cd0c-83db-b058-7615-5626ef60dbb0-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-05-15 14:36 ` Leon Romanovsky
[not found] ` <20170515143632.GH3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-05-15 14:59 ` Christoph Hellwig
[not found] ` <20170515145952.GA7871-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2017-05-15 17:05 ` Leon Romanovsky
[not found] ` <20170515170506.GK3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-05-17 12:56 ` Marta Rybczynska
[not found] ` <779753075.36035391.1495025796237.JavaMail.zimbra-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org>
2017-05-18 13:34 ` Leon Romanovsky
[not found] ` <20170518133439.GD3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-06-19 17:21 ` Robert LeBlanc
[not found] ` <CAANLjFrCLpX3nb3q7LpFPpLJKciU+1Hvmt_hxyTovQJM2-zQmg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-06-20 6:39 ` Sagi Grimberg
[not found] ` <6073e553-e8c2-6d14-ba5d-c2bd5aff15eb-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-06-20 7:46 ` Leon Romanovsky
[not found] ` <20170620074639.GP17846-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-06-20 7:58 ` Sagi Grimberg
[not found] ` <1c706958-992e-b104-6bae-4a6616c0a9f9-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-06-20 8:33 ` Leon Romanovsky
[not found] ` <20170620083309.GQ17846-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-06-20 9:33 ` Sagi Grimberg
[not found] ` <bd0b986f-9bed-3dfa-7454-0661559a527b-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-06-20 12:02 ` Sagi Grimberg
[not found] ` <614481c7-22dd-d93b-e97e-52f868727ec3-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-06-20 17:01 ` Chuck Lever
[not found] ` <59FF0C04-2BFB-4F66-81BA-A598A9A087FC-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-06-20 17:12 ` Sagi Grimberg
2017-06-20 17:35 ` Jason Gunthorpe
[not found] ` <20170620173532.GA827-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-06-20 18:17 ` Chuck Lever
[not found] ` <D3DC49A2-FFC9-4F62-8876-3E6AD5167DE5-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-06-20 19:27 ` Jason Gunthorpe
[not found] ` <20170620192742.GB827-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-06-20 20:56 ` Chuck Lever
[not found] ` <C14B071E-F1B2-466A-82CF-4E20BFAD9DC1-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-06-20 21:19 ` Jason Gunthorpe
[not found] ` <20170620211958.GA5574-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-06-27 7:37 ` Sagi Grimberg
[not found] ` <4f0812f1-0067-4e63-e383-b913ee1f319d-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-06-27 14:42 ` Chuck Lever
[not found] ` <28F6F58E-B6F4-4114-8DFF-B72353CE814B-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-06-27 16:07 ` Sagi Grimberg
[not found] ` <52ad3547-efcf-f428-6b39-117efda3379f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-06-27 16:28 ` Jason Gunthorpe [this message]
[not found] ` <20170627162800.GA22592-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-06-28 7:03 ` Sagi Grimberg
2017-06-27 16:28 ` Chuck Lever
[not found] ` <9990B5CB-E0FF-481E-9F34-21EACF0E796E-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-06-28 7:08 ` Sagi Grimberg
[not found] ` <f1f1a68c-90db-e6bf-e35e-55c4b469c339-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-06-28 16:11 ` Chuck Lever
[not found] ` <7D1C540B-FEA0-4101-8B58-87BCB7DB5492-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-06-29 5:35 ` Sagi Grimberg
[not found] ` <66b1b8be-e506-50b8-c01f-fa0e3cea98a4-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-06-29 14:55 ` Chuck Lever
[not found] ` <9D8C7BC8-7E18-405A-9017-9DB23A6B5C15-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-07-02 9:45 ` Sagi Grimberg
[not found] ` <11aa1a24-9f0b-dbb8-18eb-ad357c7727b2-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-07-02 18:17 ` Chuck Lever
[not found] ` <9E30754F-464A-4B62-ADE7-F6B2F6D95763-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-07-09 16:47 ` Jason Gunthorpe
[not found] ` <20170709164755.GB3058-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-10 19:03 ` Chuck Lever
[not found] ` <A7C8C159-E916-4060-9FD1-8726D816B3C0-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-07-10 20:05 ` Jason Gunthorpe
[not found] ` <20170710200522.GA19293-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-10 20:51 ` Chuck Lever
[not found] ` <C04891DF-5B3B-4156-9E04-9E18B238864A-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-07-10 21:14 ` Jason Gunthorpe
2017-07-10 21:24 ` Jason Gunthorpe
[not found] ` <20170710212430.GA21721-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-10 21:29 ` Chuck Lever
[not found] ` <C142385D-3A54-44C9-BA3D-0AABBC5E9E7B-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-07-10 21:32 ` Jason Gunthorpe
[not found] ` <20170710213251.GA21908-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-10 22:04 ` Chuck Lever
[not found] ` <A342254B-1ECB-4644-8D68-328A52940C52-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-07-10 22:09 ` Jason Gunthorpe
[not found] ` <20170710220905.GA22589-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-11 3:57 ` Chuck Lever
[not found] ` <718A099D-3597-4262-9A33-0BA7EFE5461F-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2017-07-11 13:23 ` Tom Talpey
[not found] ` <8dd77b19-3846-96ea-f50f-22182989e941-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org>
2017-07-11 14:55 ` Chuck Lever
2017-06-27 18:08 ` Bart Van Assche
[not found] ` <1498586933.14963.1.camel-Sjgp3cTcYWE@public.gmane.org>
2017-06-27 18:14 ` Jason Gunthorpe
2017-06-28 7:16 ` Sagi Grimberg
[not found] ` <47bbf598-6c82-610f-dc1d-706a1d869b8d-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-06-28 9:43 ` Bart Van Assche
2017-06-20 17:08 ` Robert LeBlanc
[not found] ` <CAANLjFrpHzapvqgBajUu7QpgNNPvNjThMZrsXGcKt58E+6siMA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-06-20 17:19 ` Sagi Grimberg
[not found] ` <3f3830eb-68c5-f862-58c7-7021e6462f6f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-06-20 17:28 ` Robert LeBlanc
[not found] ` <CAANLjFo_1E5nPkXMmG0VxtXpQ2m6=WNd0OtvG4rd-__1TD0-VQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-06-27 7:22 ` Sagi Grimberg
2017-06-20 14:43 ` Robert LeBlanc
[not found] ` <78b2c1db-6ece-0274-c4c9-5ee1f7c88469@mellanox.com>
[not found] ` <78b2c1db-6ece-0274-c4c9-5ee1f7c88469-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2017-06-20 22:58 ` Robert LeBlanc
2017-06-27 7:16 ` Sagi Grimberg
2017-06-20 14:41 ` Robert LeBlanc
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170627162800.GA22592@obsidianresearch.com \
--to=jgunthorpe-epgobjl8dl3ta4ec/59zmfatqe2ktcn/@public.gmane.org \
--cc=bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org \
--cc=chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org \
--cc=hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org \
--cc=joseph.r.gruher-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \
--cc=leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org \
--cc=linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org \
--cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=liranl-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
--cc=loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \
--cc=maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
--cc=mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org \
--cc=robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org \
--cc=robert.m.riches.jr-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \
--cc=sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org \
--cc=shahar.salzman-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox