From: Matthew Wilcox <willy@linux.intel.com>
To: Stephen Bates <stephen.bates@pmcs.com>
Cc: haggaie@mellanox.com, javier@cnexlabs.com,
linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org,
sagig@mellanox.com, linux-mm@kvack.org, artemyko@mellanox.com,
hch@infradead.org, leonro@mellanox.com,
jgunthorpe@obsidianresearch.com
Subject: Re: [PATCH RFC 1/1] Add support for ZONE_DEVICE IO memory with struct pages.
Date: Mon, 14 Mar 2016 17:23:44 -0400 [thread overview]
Message-ID: <20160314212344.GC23727@linux.intel.com> (raw)
In-Reply-To: <1457979277-26791-1-git-send-email-stephen.bates@pmcs.com>
On Mon, Mar 14, 2016 at 12:14:37PM -0600, Stephen Bates wrote:
> 3. Coherency Issues. When IOMEM is written from both the CPU and a PCIe
> peer there is potential for coherency issues and for writes to occur out
> of order. This is something that users of this feature need to be
> cognizant of and may necessitate the use of CONFIG_EXPERT. Though really,
> this isn't much different than the existing situation with RDMA: if
> userspace sets up an MR for remote use, they need to be careful about
> using that memory region themselves.
There's more to the coherency problem than this. As I understand it, on
x86, memory in a PCI BAR does not participate in the coherency protocol.
So you can get a situation where CPU A stores 4 bytes to offset 8 in a
cacheline, then CPU B stores 4 bytes to offset 16 in the same cacheline,
and CPU A's write mysteriously goes missing.
I may have misunderstood the exact details when this was explained to me a
few years ago, but the details were horrible enough to run away screaming.
Pretending PCI BARs are real memory? Just Say No.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
parent reply other threads:[~2016-03-14 21:23 UTC|newest]
Thread overview: expand[flat|nested] mbox.gz Atom feed
[parent not found: <1457979277-26791-1-git-send-email-stephen.bates@pmcs.com>]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160314212344.GC23727@linux.intel.com \
--to=willy@linux.intel.com \
--cc=artemyko@mellanox.com \
--cc=haggaie@mellanox.com \
--cc=hch@infradead.org \
--cc=javier@cnexlabs.com \
--cc=jgunthorpe@obsidianresearch.com \
--cc=leonro@mellanox.com \
--cc=linux-mm@kvack.org \
--cc=linux-nvdimm@lists.01.org \
--cc=linux-rdma@vger.kernel.org \
--cc=sagig@mellanox.com \
--cc=stephen.bates@pmcs.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox