From: Christoph Hellwig <hch@lst.de>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: "Christoph Hellwig" <hch@lst.de>,
"Logan Gunthorpe" <logang@deltatee.com>,
linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org,
linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
linux-nvdimm@lists.01.org, linux-block@vger.kernel.org,
"Stephen Bates" <sbates@raithlin.com>,
"Jens Axboe" <axboe@kernel.dk>,
"Keith Busch" <keith.busch@intel.com>,
"Sagi Grimberg" <sagi@grimberg.me>,
"Bjorn Helgaas" <bhelgaas@google.com>,
"Max Gurtovoy" <maxg@mellanox.com>,
"Dan Williams" <dan.j.williams@intel.com>,
"Jérôme Glisse" <jglisse@redhat.com>,
"Benjamin Herrenschmidt" <benh@kernel.crashing.org>
Subject: Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()
Date: Tue, 9 Jan 2018 17:55:49 +0100 [thread overview]
Message-ID: <20180109165549.GC12159@lst.de> (raw)
In-Reply-To: <20180108190116.GI11348@ziepe.ca>
On Mon, Jan 08, 2018 at 12:01:16PM -0700, Jason Gunthorpe wrote:
> > So I very much disagree about where to place that workaround - the
> > RDMA code is exactly the right place.
>
> But why? RDMA is using core code to do this. It uses dma_ops in struct
> device and it uses normal dma_map SG. How is it RDMA's problem that
> some PCI drivers provide strange DMA ops?
Because RDMA uses the dma_virt_ops to pretend a device does DMA when
in fact it doesn't - at least not for the exact data mapped, or
as far as I can tell often not all - e.g. the QIB/HFI devices
might do mmio access for data mapped.
This whole problem only exist because a few RDMA HCA drivers lie
with the help of the RDMA core.
WARNING: multiple messages have this Message-ID (diff)
From: Christoph Hellwig <hch@lst.de>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: "Jens Axboe" <axboe@kernel.dk>,
"Keith Busch" <keith.busch@intel.com>,
linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org,
linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-nvme@lists.infradead.org, linux-block@vger.kernel.org,
"Jérôme Glisse" <jglisse@redhat.com>,
"Benjamin Herrenschmidt" <benh@kernel.crashing.org>,
"Bjorn Helgaas" <bhelgaas@google.com>,
"Max Gurtovoy" <maxg@mellanox.com>,
"Christoph Hellwig" <hch@lst.de>
Subject: Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()
Date: Tue, 9 Jan 2018 17:55:49 +0100 [thread overview]
Message-ID: <20180109165549.GC12159@lst.de> (raw)
In-Reply-To: <20180108190116.GI11348@ziepe.ca>
On Mon, Jan 08, 2018 at 12:01:16PM -0700, Jason Gunthorpe wrote:
> > So I very much disagree about where to place that workaround - the
> > RDMA code is exactly the right place.
>
> But why? RDMA is using core code to do this. It uses dma_ops in struct
> device and it uses normal dma_map SG. How is it RDMA's problem that
> some PCI drivers provide strange DMA ops?
Because RDMA uses the dma_virt_ops to pretend a device does DMA when
in fact it doesn't - at least not for the exact data mapped, or
as far as I can tell often not all - e.g. the QIB/HFI devices
might do mmio access for data mapped.
This whole problem only exist because a few RDMA HCA drivers lie
with the help of the RDMA core.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
WARNING: multiple messages have this Message-ID (diff)
From: hch@lst.de (Christoph Hellwig)
Subject: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()
Date: Tue, 9 Jan 2018 17:55:49 +0100 [thread overview]
Message-ID: <20180109165549.GC12159@lst.de> (raw)
In-Reply-To: <20180108190116.GI11348@ziepe.ca>
On Mon, Jan 08, 2018@12:01:16PM -0700, Jason Gunthorpe wrote:
> > So I very much disagree about where to place that workaround - the
> > RDMA code is exactly the right place.
>
> But why? RDMA is using core code to do this. It uses dma_ops in struct
> device and it uses normal dma_map SG. How is it RDMA's problem that
> some PCI drivers provide strange DMA ops?
Because RDMA uses the dma_virt_ops to pretend a device does DMA when
in fact it doesn't - at least not for the exact data mapped, or
as far as I can tell often not all - e.g. the QIB/HFI devices
might do mmio access for data mapped.
This whole problem only exist because a few RDMA HCA drivers lie
with the help of the RDMA core.
WARNING: multiple messages have this Message-ID (diff)
From: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
To: Jason Gunthorpe <jgg-uk2M96/98Pc@public.gmane.org>
Cc: "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>,
"Logan Gunthorpe"
<logang-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org,
linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
"Stephen Bates" <sbates-pv7U853sEMVWk0Htik3J/w@public.gmane.org>,
"Jens Axboe" <axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org>,
"Keith Busch"
<keith.busch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>,
"Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
"Bjorn Helgaas"
<bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>,
"Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
"Dan Williams"
<dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>,
"Jérôme Glisse" <jglisse-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
"Benjamin Herrenschmidt"
<benh-XVmvHMARGAS8U2dJNN8I7kB+6BGkLq7r@public.gmane.org>
Subject: Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()
Date: Tue, 9 Jan 2018 17:55:49 +0100 [thread overview]
Message-ID: <20180109165549.GC12159@lst.de> (raw)
In-Reply-To: <20180108190116.GI11348-uk2M96/98Pc@public.gmane.org>
On Mon, Jan 08, 2018 at 12:01:16PM -0700, Jason Gunthorpe wrote:
> > So I very much disagree about where to place that workaround - the
> > RDMA code is exactly the right place.
>
> But why? RDMA is using core code to do this. It uses dma_ops in struct
> device and it uses normal dma_map SG. How is it RDMA's problem that
> some PCI drivers provide strange DMA ops?
Because RDMA uses the dma_virt_ops to pretend a device does DMA when
in fact it doesn't - at least not for the exact data mapped, or
as far as I can tell often not all - e.g. the QIB/HFI devices
might do mmio access for data mapped.
This whole problem only exist because a few RDMA HCA drivers lie
with the help of the RDMA core.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2018-01-09 16:55 UTC|newest]
Thread overview: 198+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-04 19:01 [PATCH 00/11] Copy Offload in NVMe Fabrics with P2P PCI Memory Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 01/12] pci-p2p: Support peer to peer memory Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 21:40 ` Bjorn Helgaas
2018-01-04 21:40 ` Bjorn Helgaas
2018-01-04 21:40 ` Bjorn Helgaas
2018-01-04 23:06 ` Logan Gunthorpe
2018-01-04 23:06 ` Logan Gunthorpe
2018-01-04 23:06 ` Logan Gunthorpe
2018-01-04 23:06 ` Logan Gunthorpe
2018-01-04 21:59 ` Bjorn Helgaas
2018-01-04 21:59 ` Bjorn Helgaas
2018-01-04 21:59 ` Bjorn Helgaas
2018-01-04 21:59 ` Bjorn Helgaas
2018-01-05 0:20 ` Logan Gunthorpe
2018-01-05 0:20 ` Logan Gunthorpe
2018-01-05 0:20 ` Logan Gunthorpe
2018-01-05 0:20 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 02/12] pci-p2p: Add sysfs group to display p2pmem stats Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 21:50 ` Bjorn Helgaas
2018-01-04 21:50 ` Bjorn Helgaas
2018-01-04 21:50 ` Bjorn Helgaas
2018-01-04 22:25 ` Jason Gunthorpe
2018-01-04 22:25 ` Jason Gunthorpe
2018-01-04 22:25 ` Jason Gunthorpe
2018-01-04 22:25 ` Jason Gunthorpe
2018-01-04 23:13 ` Logan Gunthorpe
2018-01-04 23:13 ` Logan Gunthorpe
2018-01-04 23:13 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 03/12] pci-p2p: Add PCI p2pmem dma mappings to adjust the bus offset Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 04/12] pci-p2p: Clear ACS P2P flags for all client devices Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 21:57 ` Bjorn Helgaas
2018-01-04 21:57 ` Bjorn Helgaas
2018-01-04 21:57 ` Bjorn Helgaas
2018-01-04 21:57 ` Bjorn Helgaas
2018-01-04 22:35 ` Alex Williamson
2018-01-04 22:35 ` Alex Williamson
2018-01-04 22:35 ` Alex Williamson
2018-01-05 0:00 ` Logan Gunthorpe
2018-01-05 0:00 ` Logan Gunthorpe
2018-01-05 0:00 ` Logan Gunthorpe
2018-01-05 0:00 ` Logan Gunthorpe
2018-01-05 1:09 ` Logan Gunthorpe
2018-01-05 1:09 ` Logan Gunthorpe
2018-01-05 1:09 ` Logan Gunthorpe
2018-01-05 3:33 ` Alex Williamson
2018-01-05 3:33 ` Alex Williamson
2018-01-05 3:33 ` Alex Williamson
2018-01-05 6:47 ` Jerome Glisse
2018-01-05 6:47 ` Jerome Glisse
2018-01-05 6:47 ` Jerome Glisse
2018-01-05 6:47 ` Jerome Glisse
2018-01-05 15:41 ` Alex Williamson
2018-01-05 15:41 ` Alex Williamson
2018-01-05 15:41 ` Alex Williamson
2018-01-05 17:10 ` Logan Gunthorpe
2018-01-05 17:10 ` Logan Gunthorpe
2018-01-05 17:10 ` Logan Gunthorpe
2018-01-05 17:10 ` Logan Gunthorpe
2018-01-05 17:18 ` Alex Williamson
2018-01-05 17:18 ` Alex Williamson
2018-01-05 17:18 ` Alex Williamson
2018-01-05 17:18 ` Alex Williamson
2018-01-04 19:01 ` [PATCH 05/12] block: Introduce PCI P2P flags for request and request queue Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]() Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:22 ` Jason Gunthorpe
2018-01-04 19:22 ` Jason Gunthorpe
2018-01-04 19:52 ` Logan Gunthorpe
2018-01-04 19:52 ` Logan Gunthorpe
2018-01-04 19:52 ` Logan Gunthorpe
2018-01-04 19:52 ` Logan Gunthorpe
2018-01-04 22:13 ` Jason Gunthorpe
2018-01-04 22:13 ` Jason Gunthorpe
2018-01-04 23:44 ` Logan Gunthorpe
2018-01-04 23:44 ` Logan Gunthorpe
2018-01-04 23:44 ` Logan Gunthorpe
2018-01-05 4:50 ` Jason Gunthorpe
2018-01-05 4:50 ` Jason Gunthorpe
2018-01-08 14:59 ` Christoph Hellwig
2018-01-08 14:59 ` Christoph Hellwig
2018-01-08 14:59 ` Christoph Hellwig
2018-01-08 18:09 ` Jason Gunthorpe
2018-01-08 18:09 ` Jason Gunthorpe
2018-01-08 18:17 ` Logan Gunthorpe
2018-01-08 18:17 ` Logan Gunthorpe
2018-01-08 18:17 ` Logan Gunthorpe
2018-01-08 18:17 ` Logan Gunthorpe
2018-01-08 18:29 ` Jason Gunthorpe
2018-01-08 18:29 ` Jason Gunthorpe
2018-01-08 18:29 ` Jason Gunthorpe
2018-01-08 18:34 ` Christoph Hellwig
2018-01-08 18:34 ` Christoph Hellwig
2018-01-08 18:34 ` Christoph Hellwig
2018-01-08 18:34 ` Christoph Hellwig
2018-01-08 18:44 ` Logan Gunthorpe
2018-01-08 18:44 ` Logan Gunthorpe
2018-01-08 18:44 ` Logan Gunthorpe
2018-01-08 18:44 ` Logan Gunthorpe
2018-01-08 18:57 ` Christoph Hellwig
2018-01-08 18:57 ` Christoph Hellwig
2018-01-08 18:57 ` Christoph Hellwig
2018-01-08 18:57 ` Christoph Hellwig
2018-01-08 19:05 ` Logan Gunthorpe
2018-01-08 19:05 ` Logan Gunthorpe
2018-01-08 19:05 ` Logan Gunthorpe
2018-01-08 19:05 ` Logan Gunthorpe
2018-01-09 16:47 ` Christoph Hellwig
2018-01-09 16:47 ` Christoph Hellwig
2018-01-09 16:47 ` Christoph Hellwig
2018-01-09 16:47 ` Christoph Hellwig
2018-01-08 19:49 ` Jason Gunthorpe
2018-01-08 19:49 ` Jason Gunthorpe
2018-01-09 16:46 ` Christoph Hellwig
2018-01-09 16:46 ` Christoph Hellwig
2018-01-09 16:46 ` Christoph Hellwig
2018-01-09 17:10 ` Jason Gunthorpe
2018-01-09 17:10 ` Jason Gunthorpe
2018-01-08 19:01 ` Jason Gunthorpe
2018-01-08 19:01 ` Jason Gunthorpe
2018-01-08 19:01 ` Jason Gunthorpe
2018-01-09 16:55 ` Christoph Hellwig [this message]
2018-01-09 16:55 ` Christoph Hellwig
2018-01-09 16:55 ` Christoph Hellwig
2018-01-09 16:55 ` Christoph Hellwig
2018-01-04 19:01 ` [PATCH 07/12] nvme-pci: clean up CMB initialization Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 08/12] nvme-pci: clean up SMBSZ bit definitions Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 09/12] nvme-pci: Use PCI p2pmem subsystem to manage the CMB Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-05 15:30 ` Marta Rybczynska
2018-01-05 15:30 ` Marta Rybczynska
2018-01-05 15:30 ` Marta Rybczynska
2018-01-05 15:30 ` Marta Rybczynska
2018-01-05 18:14 ` Logan Gunthorpe
2018-01-05 18:14 ` Logan Gunthorpe
2018-01-05 18:14 ` Logan Gunthorpe
2018-01-05 18:14 ` Logan Gunthorpe
2018-01-05 18:11 ` Keith Busch
2018-01-05 18:11 ` Keith Busch
2018-01-05 18:11 ` Keith Busch
2018-01-05 18:11 ` Keith Busch
2018-01-05 18:19 ` Logan Gunthorpe
2018-01-05 18:19 ` Logan Gunthorpe
2018-01-05 18:19 ` Logan Gunthorpe
2018-01-05 19:01 ` Keith Busch
2018-01-05 19:01 ` Keith Busch
2018-01-05 19:01 ` Keith Busch
2018-01-05 19:01 ` Keith Busch
2018-01-05 19:04 ` Logan Gunthorpe
2018-01-05 19:04 ` Logan Gunthorpe
2018-01-05 19:04 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 10/12] nvme-pci: Add support for P2P memory in requests Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 11/12] nvme-pci: Add a quirk for a pseudo CMB Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 12/12] nvmet: Optionally use PCI P2P memory Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180109165549.GC12159@lst.de \
--to=hch@lst.de \
--cc=axboe@kernel.dk \
--cc=benh@kernel.crashing.org \
--cc=bhelgaas@google.com \
--cc=dan.j.williams@intel.com \
--cc=jgg@ziepe.ca \
--cc=jglisse@redhat.com \
--cc=keith.busch@intel.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvdimm@lists.01.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=logang@deltatee.com \
--cc=maxg@mellanox.com \
--cc=sagi@grimberg.me \
--cc=sbates@raithlin.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.