* Re: [RFC PATCH 00/28] Removing struct page from P2PDMA
From: Logan Gunthorpe @ 2019-06-20 19:34 UTC (permalink / raw)
To: Dan Williams
Cc: Linux Kernel Mailing List, linux-block, linux-nvme, linux-pci,
linux-rdma, Jens Axboe, Christoph Hellwig, Bjorn Helgaas,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates
In-Reply-To: <CAPcyv4ijztOK1FUjLuFing7ps4LOHt=6z=eO=98HHWauHA+yog@mail.gmail.com>
On 2019-06-20 12:45 p.m., Dan Williams wrote:
> On Thu, Jun 20, 2019 at 9:13 AM Logan Gunthorpe <logang@deltatee.com> wrote:
>>
>> For eons there has been a debate over whether or not to use
>> struct pages for peer-to-peer DMA transactions. Pro-pagers have
>> argued that struct pages are necessary for interacting with
>> existing code like scatterlists or the bio_vecs. Anti-pagers
>> assert that the tracking of the memory is unecessary and
>> allocating the pages is a waste of memory. Both viewpoints are
>> valid, however developers working on GPUs and RDMA tend to be
>> able to do away with struct pages relatively easily
>
> Presumably because they have historically never tried to be
> inter-operable with the block layer or drivers outside graphics and
> RDMA.
Yes, but really there are three main sets of users for P2P right now:
graphics, RDMA and NVMe. And every time a patch set comes from GPU/RDMA
people they don't bother with struct page. I seem to be the only one
trying to push P2P with NVMe and it seems to be a losing battle.
> Please spell out the value, it is not immediately obvious to me
> outside of some memory capacity savings.
There are a few things:
* Have consistency with P2P efforts as most other efforts have been
avoiding struct page. Nobody else seems to want
pci_p2pdma_add_resource() or any devm_memremap_pages() call.
* Avoid all arch-specific dependencies for P2P. With struct page the IO
memory must fit in the linear mapping. This requires some work with
RISC-V and I remember some complaints from the powerpc people regarding
this. Certainly not all arches will be able to fit the IO region into
the linear mapping space.
* Remove a bunch of PCI P2PDMA special case mapping stuff from the block
layer and RDMA interface (which I've been hearing complaints over).
* Save the struct page memory that is largely unused (as you note).
>> Previously, there have been multiple attempts[1][2] to replace
>> struct page usage with pfn_t but this has been unpopular seeing
>> it creates dangerous edge cases where unsuspecting code might
>> run accross pfn_t's they are not ready for.
>
> That's not the conclusion I arrived at because pfn_t is specifically
> an opaque type precisely to force "unsuspecting" code to throw
> compiler assertions. Instead pfn_t was dealt its death blow here:
>
> https://lore.kernel.org/lkml/CA+55aFzON9617c2_Amep0ngLq91kfrPiSccdZakxir82iekUiA@mail.gmail.com/
Ok, well yes the special pages are what we've done for P2PDMA today. But
I don't think Linus's criticism really applies to what's in this RFC.
For starters, P2PDMA doesn't, and has have never, used struct page to
look up the reference count. PCI BARs have no relation to the cache so
there's no need to serialize their access but this can be done
before/after the DMA addresses are submitted to the block/rdma layer if
it was required.
In fact, the only thing the struct page is used for in the current
P2PDMA implementation is a single flag indicating it's special and needs
to be mapped in a special way.
> My primary concern with this is that ascribes a level of generality
> that just isn't there for peer-to-peer dma operations. "Peer"
> addresses are not "DMA" addresses, and the rules about what can and
> can't do peer-DMA are not generically known to the block layer.
Correct, but I don't think we should teach the block layer about these
rules. In the current code, the rules are enforced outside the block
layer before the bios are submitted and this patch set doesn't change
that. The driver orchestrating P2P will always have to check the rules
and derive addresses from them (as appropriate). With the RFC the block
layer then doesn't have to care and can just handle the DMA addresses
directly.
> At least with a side object there's a chance to describe / recall those
> restrictions as these things get passed around the I/O stack, but an
> undecorated "DMA" address passed through the block layer with no other
> benefit to any subsystem besides RDMA does not feel like it advances
> the state of the art.
>
> Again, what are the benefits of plumbing this RDMA special case?
Because I don't think it is an RDMA special case.
Logan
^ permalink raw reply
* Re: [RFC PATCH 00/28] Removing struct page from P2PDMA
From: Jason Gunthorpe @ 2019-06-20 19:33 UTC (permalink / raw)
To: Dan Williams
Cc: Logan Gunthorpe, Linux Kernel Mailing List, linux-block,
linux-nvme, linux-pci, linux-rdma, Jens Axboe, Christoph Hellwig,
Bjorn Helgaas, Sagi Grimberg, Keith Busch, Stephen Bates
In-Reply-To: <CAPcyv4ijztOK1FUjLuFing7ps4LOHt=6z=eO=98HHWauHA+yog@mail.gmail.com>
On Thu, Jun 20, 2019 at 11:45:38AM -0700, Dan Williams wrote:
> > Previously, there have been multiple attempts[1][2] to replace
> > struct page usage with pfn_t but this has been unpopular seeing
> > it creates dangerous edge cases where unsuspecting code might
> > run accross pfn_t's they are not ready for.
>
> That's not the conclusion I arrived at because pfn_t is specifically
> an opaque type precisely to force "unsuspecting" code to throw
> compiler assertions. Instead pfn_t was dealt its death blow here:
>
> https://lore.kernel.org/lkml/CA+55aFzON9617c2_Amep0ngLq91kfrPiSccdZakxir82iekUiA@mail.gmail.com/
>
> ...and I think that feedback also reads on this proposal.
I read through Linus's remarks and it he seems completely right that
anything that touches a filesystem needs a struct page, because FS's
rely heavily on that.
It is much less clear to me why a GPU BAR or a NVME CMB that never
touches a filesystem needs a struct page.. The best reason I've seen
is that it must have struct page because the block layer heavily
depends on struct page.
Since that thread was so DAX/pmem centric (and Linus did say he liked
the __pfn_t), maybe it is worth checking again, but not for DAX/pmem
users?
This P2P is quite distinct from DAX as the struct page* would point to
non-cacheable weird memory that few struct page users would even be
able to work with, while I understand DAX use cases focused on CPU
cache coherent memory, and filesystem involvement.
> My primary concern with this is that ascribes a level of generality
> that just isn't there for peer-to-peer dma operations. "Peer"
> addresses are not "DMA" addresses, and the rules about what can and
> can't do peer-DMA are not generically known to the block layer.
?? The P2P infrastructure produces a DMA bus address for the
initiating device that is is absolutely a DMA address. There is some
intermediate CPU centric representation, but after mapping it is the
same as any other DMA bus address.
The map function can tell if the device pair combination can do p2p or
not.
> Again, what are the benefits of plumbing this RDMA special case?
It is not just RDMA, this is interesting for GPU and vfio use cases
too. RDMA is just the most complete in-tree user we have today.
ie GPU people wouuld really like to do read() and have P2P
transparently happen to on-GPU pages. With GPUs having huge amounts of
memory loading file data into them is really a performance critical
thing.
Jason
^ permalink raw reply
* Re: [RFC PATCH 00/28] Removing struct page from P2PDMA
From: Dan Williams @ 2019-06-20 18:45 UTC (permalink / raw)
To: Logan Gunthorpe
Cc: Linux Kernel Mailing List, linux-block, linux-nvme, linux-pci,
linux-rdma, Jens Axboe, Christoph Hellwig, Bjorn Helgaas,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
On Thu, Jun 20, 2019 at 9:13 AM Logan Gunthorpe <logang@deltatee.com> wrote:
>
> For eons there has been a debate over whether or not to use
> struct pages for peer-to-peer DMA transactions. Pro-pagers have
> argued that struct pages are necessary for interacting with
> existing code like scatterlists or the bio_vecs. Anti-pagers
> assert that the tracking of the memory is unecessary and
> allocating the pages is a waste of memory. Both viewpoints are
> valid, however developers working on GPUs and RDMA tend to be
> able to do away with struct pages relatively easily
Presumably because they have historically never tried to be
inter-operable with the block layer or drivers outside graphics and
RDMA.
> compared to
> those wanting to work with NVMe devices through the block layer.
> So it would be of great value to be able to universally do P2PDMA
> transactions without the use of struct pages.
Please spell out the value, it is not immediately obvious to me
outside of some memory capacity savings.
> Previously, there have been multiple attempts[1][2] to replace
> struct page usage with pfn_t but this has been unpopular seeing
> it creates dangerous edge cases where unsuspecting code might
> run accross pfn_t's they are not ready for.
That's not the conclusion I arrived at because pfn_t is specifically
an opaque type precisely to force "unsuspecting" code to throw
compiler assertions. Instead pfn_t was dealt its death blow here:
https://lore.kernel.org/lkml/CA+55aFzON9617c2_Amep0ngLq91kfrPiSccdZakxir82iekUiA@mail.gmail.com/
...and I think that feedback also reads on this proposal.
> Currently, we have P2PDMA using struct pages through the block layer
> and the dangerous cases are avoided by using a queue flag that
> indicates support for the special pages.
>
> This RFC proposes a new solution: allow the block layer to take
> DMA addresses directly for queues that indicate support. This will
> provide a more general path for doing P2PDMA-like requests and will
> allow us to remove the struct pages that back P2PDMA memory thus paving
> the way to build a more uniform P2PDMA ecosystem.
My primary concern with this is that ascribes a level of generality
that just isn't there for peer-to-peer dma operations. "Peer"
addresses are not "DMA" addresses, and the rules about what can and
can't do peer-DMA are not generically known to the block layer. At
least with a side object there's a chance to describe / recall those
restrictions as these things get passed around the I/O stack, but an
undecorated "DMA" address passed through the block layer with no other
benefit to any subsystem besides RDMA does not feel like it advances
the state of the art.
Again, what are the benefits of plumbing this RDMA special case?
^ permalink raw reply
* Re: [RFC PATCH 04/28] block: Never bounce dma-direct bios
From: Logan Gunthorpe @ 2019-06-20 18:38 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma,
Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Stephen Bates
In-Reply-To: <20190620172347.GE19891@ziepe.ca>
On 2019-06-20 11:23 a.m., Jason Gunthorpe wrote:
> On Thu, Jun 20, 2019 at 10:12:16AM -0600, Logan Gunthorpe wrote:
>> It is expected the creator of the dma-direct bio will ensure the
>> target device can access the DMA address it's creating bios for.
>> It's also not possible to bounce a dma-direct bio seeing the block
>> layer doesn't have any way to access the underlying data behind
>> the DMA address.
>>
>> Thus, never bounce dma-direct bios.
>
> I wonder how feasible it would be to implement a 'dma vec' copy
> from/to?
> That is about the only operation you could safely do on P2P BAR
> memory.
>
> I wonder if a copy implementation could somehow query the iommu layer
> to get a kmap of the memory pointed at by the dma address so we don't
> need to carry struct page around?
That sounds a bit nasty. First we'd have to determine what the
dma_addr_t points to; and with P2P it may be a bus address or it may be
an IOVA address and it would probably have to be based on whether the
IOVA is reserved or not (PCI bus addresses should all be reserved).
Second, if it is an IOVA then the we'd have to get the physical address
back from the IOMMU tables and hope we can then get it back to a
sensible kernel mapping -- and if it points to a PCI bus address we'd
then have to somehow get back to the kernel mapping which could be
anywhere in the VMALLOC region as we no longer have the linear mapping
that struct page provides.
I think if we need access to the memory, then this is the wrong approach
and we should keep struct page or try pfn_t so we can map the memory in
a way that would perform better.
In theory, I could relatively easily do the same thing I did for dma_vec
but with a pfn_t_vec. Though we'd still have the problem of determining
virtual address from physical address for memory that isn't linearly
mapped. We'd probably have to introduce some arch-specific thing to
linearly map an io region or something which may be possible on some
arches on not on others (same problems we have with struct page).
Logan
^ permalink raw reply
* Re: [RFC PATCH 20/28] IB/core: Introduce API for initializing a RW ctx from a DMA address
From: Logan Gunthorpe @ 2019-06-20 18:24 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma,
Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Stephen Bates
In-Reply-To: <20190620171105.GD19891@ziepe.ca>
On 2019-06-20 11:11 a.m., Jason Gunthorpe wrote:
> On Thu, Jun 20, 2019 at 10:59:44AM -0600, Logan Gunthorpe wrote:
>>
>>
>> On 2019-06-20 10:49 a.m., Jason Gunthorpe wrote:
>>> On Thu, Jun 20, 2019 at 10:12:32AM -0600, Logan Gunthorpe wrote:
>>>> Introduce rdma_rw_ctx_dma_init() and rdma_rw_ctx_dma_destroy() which
>>>> peform the same operation as rdma_rw_ctx_init() and
>>>> rdma_rw_ctx_destroy() respectively except they operate on a DMA
>>>> address and length instead of an SGL.
>>>>
>>>> This will be used for struct page-less P2PDMA, but there's also
>>>> been opinions expressed to migrate away from SGLs and struct
>>>> pages in the RDMA APIs and this will likely fit with that
>>>> effort.
>>>>
>>>> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
>>>> drivers/infiniband/core/rw.c | 74 ++++++++++++++++++++++++++++++------
>>>> include/rdma/rw.h | 6 +++
>>>> 2 files changed, 69 insertions(+), 11 deletions(-)
>>>>
>>>> diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
>>>> index 32ca8429eaae..cefa6b930bc8 100644
>>>> +++ b/drivers/infiniband/core/rw.c
>>>> @@ -319,6 +319,39 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
>>>> }
>>>> EXPORT_SYMBOL(rdma_rw_ctx_init);
>>>>
>>>> +/**
>>>> + * rdma_rw_ctx_dma_init - initialize a RDMA READ/WRITE context from a
>>>> + * DMA address instead of SGL
>>>> + * @ctx: context to initialize
>>>> + * @qp: queue pair to operate on
>>>> + * @port_num: port num to which the connection is bound
>>>> + * @addr: DMA address to READ/WRITE from/to
>>>> + * @len: length of memory to operate on
>>>> + * @remote_addr:remote address to read/write (relative to @rkey)
>>>> + * @rkey: remote key to operate on
>>>> + * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ
>>>> + *
>>>> + * Returns the number of WQEs that will be needed on the workqueue if
>>>> + * successful, or a negative error code.
>>>> + */
>>>> +int rdma_rw_ctx_dma_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
>>>> + u8 port_num, dma_addr_t addr, u32 len, u64 remote_addr,
>>>> + u32 rkey, enum dma_data_direction dir)
>>>
>>> Why not keep the same basic signature here but replace the scatterlist
>>> with the dma vec ?
>>
>> Could do. At the moment, I had no need for dma_vec in this interface.
>
> I think that is because you only did nvme not srp/iser :)
I'm not sure that's true at least for the P2P case. With P2P we are able
to allocate one continuous region of memory for each transaction. It
would be quite weird to allocate multiple regions for a single transaction.
>>>> +{
>>>> + struct scatterlist sg;
>>>> +
>>>> + sg_dma_address(&sg) = addr;
>>>> + sg_dma_len(&sg) = len;
>>>
>>> This needs to fail if the driver is one of the few that require
>>> struct page to work..
>>
>> Yes, right. Currently P2PDMA checks for the use of dma_virt_ops. And
>> that probably should also be done here. But is that sufficient? You're
>> probably right that it'll take an audit of the RDMA tree to sort that out.
>
> For this purpose I'd be fine if you added a flag to the struct
> ib_device_ops that is set on drivers that we know are OK.. We can make
> that list bigger over time.
Ok, that would mirror what we did for the block layer. I'll look at
doing something like that in the near future.
Thanks,
Logan
^ permalink raw reply
* Re: [RFC PATCH 04/28] block: Never bounce dma-direct bios
From: Jason Gunthorpe @ 2019-06-20 17:23 UTC (permalink / raw)
To: Logan Gunthorpe
Cc: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma,
Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Stephen Bates
In-Reply-To: <20190620161240.22738-5-logang@deltatee.com>
On Thu, Jun 20, 2019 at 10:12:16AM -0600, Logan Gunthorpe wrote:
> It is expected the creator of the dma-direct bio will ensure the
> target device can access the DMA address it's creating bios for.
> It's also not possible to bounce a dma-direct bio seeing the block
> layer doesn't have any way to access the underlying data behind
> the DMA address.
>
> Thus, never bounce dma-direct bios.
I wonder how feasible it would be to implement a 'dma vec' copy
from/to?
That is about the only operation you could safely do on P2P BAR
memory.
I wonder if a copy implementation could somehow query the iommu layer
to get a kmap of the memory pointed at by the dma address so we don't
need to carry struct page around?
Jason
^ permalink raw reply
* Re: [RFC PATCH 20/28] IB/core: Introduce API for initializing a RW ctx from a DMA address
From: Jason Gunthorpe @ 2019-06-20 17:11 UTC (permalink / raw)
To: Logan Gunthorpe
Cc: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma,
Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Stephen Bates
In-Reply-To: <f9186b2b-7737-965f-2dca-25e40e566e64@deltatee.com>
On Thu, Jun 20, 2019 at 10:59:44AM -0600, Logan Gunthorpe wrote:
>
>
> On 2019-06-20 10:49 a.m., Jason Gunthorpe wrote:
> > On Thu, Jun 20, 2019 at 10:12:32AM -0600, Logan Gunthorpe wrote:
> >> Introduce rdma_rw_ctx_dma_init() and rdma_rw_ctx_dma_destroy() which
> >> peform the same operation as rdma_rw_ctx_init() and
> >> rdma_rw_ctx_destroy() respectively except they operate on a DMA
> >> address and length instead of an SGL.
> >>
> >> This will be used for struct page-less P2PDMA, but there's also
> >> been opinions expressed to migrate away from SGLs and struct
> >> pages in the RDMA APIs and this will likely fit with that
> >> effort.
> >>
> >> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
> >> drivers/infiniband/core/rw.c | 74 ++++++++++++++++++++++++++++++------
> >> include/rdma/rw.h | 6 +++
> >> 2 files changed, 69 insertions(+), 11 deletions(-)
> >>
> >> diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
> >> index 32ca8429eaae..cefa6b930bc8 100644
> >> +++ b/drivers/infiniband/core/rw.c
> >> @@ -319,6 +319,39 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
> >> }
> >> EXPORT_SYMBOL(rdma_rw_ctx_init);
> >>
> >> +/**
> >> + * rdma_rw_ctx_dma_init - initialize a RDMA READ/WRITE context from a
> >> + * DMA address instead of SGL
> >> + * @ctx: context to initialize
> >> + * @qp: queue pair to operate on
> >> + * @port_num: port num to which the connection is bound
> >> + * @addr: DMA address to READ/WRITE from/to
> >> + * @len: length of memory to operate on
> >> + * @remote_addr:remote address to read/write (relative to @rkey)
> >> + * @rkey: remote key to operate on
> >> + * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ
> >> + *
> >> + * Returns the number of WQEs that will be needed on the workqueue if
> >> + * successful, or a negative error code.
> >> + */
> >> +int rdma_rw_ctx_dma_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
> >> + u8 port_num, dma_addr_t addr, u32 len, u64 remote_addr,
> >> + u32 rkey, enum dma_data_direction dir)
> >
> > Why not keep the same basic signature here but replace the scatterlist
> > with the dma vec ?
>
> Could do. At the moment, I had no need for dma_vec in this interface.
I think that is because you only did nvme not srp/iser :)
> >> +{
> >> + struct scatterlist sg;
> >> +
> >> + sg_dma_address(&sg) = addr;
> >> + sg_dma_len(&sg) = len;
> >
> > This needs to fail if the driver is one of the few that require
> > struct page to work..
>
> Yes, right. Currently P2PDMA checks for the use of dma_virt_ops. And
> that probably should also be done here. But is that sufficient? You're
> probably right that it'll take an audit of the RDMA tree to sort that out.
For this purpose I'd be fine if you added a flag to the struct
ib_device_ops that is set on drivers that we know are OK.. We can make
that list bigger over time.
> > This is not so hard to do, as most drivers are already struct page
> > free, but is pretty much blocked on needing some way to go from the
> > block layer SGL world to the dma vec world that does not hurt storage
> > performance.
>
> Maybe I can end up helping with that if it helps push the ideas here
> through. (And assuming people think it's an acceptable approach for the
> block-layer side of things).
Let us hope for a clear decision then
Jason
^ permalink raw reply
* Re: [RFC PATCH 20/28] IB/core: Introduce API for initializing a RW ctx from a DMA address
From: Logan Gunthorpe @ 2019-06-20 16:59 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma,
Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Stephen Bates
In-Reply-To: <20190620164909.GC19891@ziepe.ca>
On 2019-06-20 10:49 a.m., Jason Gunthorpe wrote:
> On Thu, Jun 20, 2019 at 10:12:32AM -0600, Logan Gunthorpe wrote:
>> Introduce rdma_rw_ctx_dma_init() and rdma_rw_ctx_dma_destroy() which
>> peform the same operation as rdma_rw_ctx_init() and
>> rdma_rw_ctx_destroy() respectively except they operate on a DMA
>> address and length instead of an SGL.
>>
>> This will be used for struct page-less P2PDMA, but there's also
>> been opinions expressed to migrate away from SGLs and struct
>> pages in the RDMA APIs and this will likely fit with that
>> effort.
>>
>> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
>> drivers/infiniband/core/rw.c | 74 ++++++++++++++++++++++++++++++------
>> include/rdma/rw.h | 6 +++
>> 2 files changed, 69 insertions(+), 11 deletions(-)
>>
>> diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
>> index 32ca8429eaae..cefa6b930bc8 100644
>> +++ b/drivers/infiniband/core/rw.c
>> @@ -319,6 +319,39 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
>> }
>> EXPORT_SYMBOL(rdma_rw_ctx_init);
>>
>> +/**
>> + * rdma_rw_ctx_dma_init - initialize a RDMA READ/WRITE context from a
>> + * DMA address instead of SGL
>> + * @ctx: context to initialize
>> + * @qp: queue pair to operate on
>> + * @port_num: port num to which the connection is bound
>> + * @addr: DMA address to READ/WRITE from/to
>> + * @len: length of memory to operate on
>> + * @remote_addr:remote address to read/write (relative to @rkey)
>> + * @rkey: remote key to operate on
>> + * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ
>> + *
>> + * Returns the number of WQEs that will be needed on the workqueue if
>> + * successful, or a negative error code.
>> + */
>> +int rdma_rw_ctx_dma_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
>> + u8 port_num, dma_addr_t addr, u32 len, u64 remote_addr,
>> + u32 rkey, enum dma_data_direction dir)
>
> Why not keep the same basic signature here but replace the scatterlist
> with the dma vec ?
Could do. At the moment, I had no need for dma_vec in this interface.
>> +{
>> + struct scatterlist sg;
>> +
>> + sg_dma_address(&sg) = addr;
>> + sg_dma_len(&sg) = len;
>
> This needs to fail if the driver is one of the few that require
> struct page to work..
Yes, right. Currently P2PDMA checks for the use of dma_virt_ops. And
that probably should also be done here. But is that sufficient? You're
probably right that it'll take an audit of the RDMA tree to sort that out.
> Really want I want to do is to have this new 'dma vec' pushed through
> the RDMA APIs so we know that if a driver is using the dma vec
> interface it is struct page free.
Yeah, I know you were talking about heading this way during LSF/MM and
is partly what inspired this series. However, largely, my focus for this
RFC was the block layer to see this is an acceptable approach -- I just
kind of hacked RDMA for now.
> This is not so hard to do, as most drivers are already struct page
> free, but is pretty much blocked on needing some way to go from the
> block layer SGL world to the dma vec world that does not hurt storage
> performance.
Maybe I can end up helping with that if it helps push the ideas here
through. (And assuming people think it's an acceptable approach for the
block-layer side of things).
Thanks,
Logan
^ permalink raw reply
* Re: [RFC PATCH 20/28] IB/core: Introduce API for initializing a RW ctx from a DMA address
From: Jason Gunthorpe @ 2019-06-20 16:49 UTC (permalink / raw)
To: Logan Gunthorpe
Cc: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma,
Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Stephen Bates
In-Reply-To: <20190620161240.22738-21-logang@deltatee.com>
On Thu, Jun 20, 2019 at 10:12:32AM -0600, Logan Gunthorpe wrote:
> Introduce rdma_rw_ctx_dma_init() and rdma_rw_ctx_dma_destroy() which
> peform the same operation as rdma_rw_ctx_init() and
> rdma_rw_ctx_destroy() respectively except they operate on a DMA
> address and length instead of an SGL.
>
> This will be used for struct page-less P2PDMA, but there's also
> been opinions expressed to migrate away from SGLs and struct
> pages in the RDMA APIs and this will likely fit with that
> effort.
>
> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
> drivers/infiniband/core/rw.c | 74 ++++++++++++++++++++++++++++++------
> include/rdma/rw.h | 6 +++
> 2 files changed, 69 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
> index 32ca8429eaae..cefa6b930bc8 100644
> +++ b/drivers/infiniband/core/rw.c
> @@ -319,6 +319,39 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
> }
> EXPORT_SYMBOL(rdma_rw_ctx_init);
>
> +/**
> + * rdma_rw_ctx_dma_init - initialize a RDMA READ/WRITE context from a
> + * DMA address instead of SGL
> + * @ctx: context to initialize
> + * @qp: queue pair to operate on
> + * @port_num: port num to which the connection is bound
> + * @addr: DMA address to READ/WRITE from/to
> + * @len: length of memory to operate on
> + * @remote_addr:remote address to read/write (relative to @rkey)
> + * @rkey: remote key to operate on
> + * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ
> + *
> + * Returns the number of WQEs that will be needed on the workqueue if
> + * successful, or a negative error code.
> + */
> +int rdma_rw_ctx_dma_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
> + u8 port_num, dma_addr_t addr, u32 len, u64 remote_addr,
> + u32 rkey, enum dma_data_direction dir)
Why not keep the same basic signature here but replace the scatterlist
with the dma vec ?
> +{
> + struct scatterlist sg;
> +
> + sg_dma_address(&sg) = addr;
> + sg_dma_len(&sg) = len;
This needs to fail if the driver is one of the few that require
struct page to work..
Really want I want to do is to have this new 'dma vec' pushed through
the RDMA APIs so we know that if a driver is using the dma vec
interface it is struct page free.
This is not so hard to do, as most drivers are already struct page
free, but is pretty much blocked on needing some way to go from the
block layer SGL world to the dma vec world that does not hurt storage
performance.
I am hoping that the biovec dma mapping that CH has talked about will
give the missing pieces.
FWIW, rdma is one of the places that is largely struct page free, and
has few problems to natively handle a 'dma vec' from top to bottom, so
I do like this approach.
Someone would have to look carefully at siw, rxe and hfi/qib to see
how they could continue to work with a dma vec, as they do actually
seem to need to kmap the data they are transferring. However, I
thought they were using custom dma ops these days, so maybe they just
encode a struct page in their dma vec and reject p2p entirely?
Jason
^ permalink raw reply
* [RFC PATCH 28/28] memremap: Remove PCI P2PDMA page memory type
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
There are no more users of MEMORY_DEVICE_PCI_P2PDMA and
is_pci_p2pdma_page(), so remove them.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
include/linux/memremap.h | 5 -----
include/linux/mm.h | 13 -------------
2 files changed, 18 deletions(-)
diff --git a/include/linux/memremap.h b/include/linux/memremap.h
index 1732dea030b2..2e5d9fcd4d69 100644
--- a/include/linux/memremap.h
+++ b/include/linux/memremap.h
@@ -51,16 +51,11 @@ struct vmem_altmap {
* wakeup event whenever a page is unpinned and becomes idle. This
* wakeup is used to coordinate physical address space management (ex:
* fs truncate/hole punch) vs pinned pages (ex: device dma).
- *
- * MEMORY_DEVICE_PCI_P2PDMA:
- * Device memory residing in a PCI BAR intended for use with Peer-to-Peer
- * transactions.
*/
enum memory_type {
MEMORY_DEVICE_PRIVATE = 1,
MEMORY_DEVICE_PUBLIC,
MEMORY_DEVICE_FS_DAX,
- MEMORY_DEVICE_PCI_P2PDMA,
};
/*
diff --git a/include/linux/mm.h b/include/linux/mm.h
index dd0b5f4e1e45..f5fa9ec440e3 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -966,19 +966,6 @@ static inline bool is_device_public_page(const struct page *page)
page->pgmap->type == MEMORY_DEVICE_PUBLIC;
}
-#ifdef CONFIG_PCI_P2PDMA
-static inline bool is_pci_p2pdma_page(const struct page *page)
-{
- return is_zone_device_page(page) &&
- page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA;
-}
-#else /* CONFIG_PCI_P2PDMA */
-static inline bool is_pci_p2pdma_page(const struct page *page)
-{
- return false;
-}
-#endif /* CONFIG_PCI_P2PDMA */
-
#else /* CONFIG_DEV_PAGEMAP_OPS */
static inline void dev_pagemap_get_ops(void)
{
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 27/28] PCI/P2PDMA: Remove struct pages that back P2PDMA memory
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
There are no more users of the struct pages that back P2P memory,
so convert the devm_memremap_pages() call to devm_memremap() to remove
them.
The percpu_ref and completion are retained in struct p2pdma to track
when there is no memory allocated out of the genpool and it is safe
to free it.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
drivers/pci/p2pdma.c | 107 +++++++++++++------------------------------
1 file changed, 33 insertions(+), 74 deletions(-)
diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
index 9b82e13f802c..83d93911f792 100644
--- a/drivers/pci/p2pdma.c
+++ b/drivers/pci/p2pdma.c
@@ -22,10 +22,6 @@
struct pci_p2pdma {
struct gen_pool *pool;
bool p2pmem_published;
-};
-
-struct p2pdma_pagemap {
- struct dev_pagemap pgmap;
struct percpu_ref ref;
struct completion ref_done;
};
@@ -78,29 +74,12 @@ static const struct attribute_group p2pmem_group = {
.name = "p2pmem",
};
-static struct p2pdma_pagemap *to_p2p_pgmap(struct percpu_ref *ref)
-{
- return container_of(ref, struct p2pdma_pagemap, ref);
-}
-
static void pci_p2pdma_percpu_release(struct percpu_ref *ref)
{
- struct p2pdma_pagemap *p2p_pgmap = to_p2p_pgmap(ref);
+ struct pci_p2pdma *p2pdma =
+ container_of(ref, struct pci_p2pdma, ref);
- complete(&p2p_pgmap->ref_done);
-}
-
-static void pci_p2pdma_percpu_kill(struct percpu_ref *ref)
-{
- percpu_ref_kill(ref);
-}
-
-static void pci_p2pdma_percpu_cleanup(struct percpu_ref *ref)
-{
- struct p2pdma_pagemap *p2p_pgmap = to_p2p_pgmap(ref);
-
- wait_for_completion(&p2p_pgmap->ref_done);
- percpu_ref_exit(&p2p_pgmap->ref);
+ complete(&p2pdma->ref_done);
}
static void pci_p2pdma_release(void *data)
@@ -111,6 +90,10 @@ static void pci_p2pdma_release(void *data)
if (!p2pdma)
return;
+ percpu_ref_kill(&p2pdma->ref);
+ wait_for_completion(&p2pdma->ref_done);
+ percpu_ref_exit(&p2pdma->ref);
+
/* Flush and disable pci_alloc_p2p_mem() */
pdev->p2pdma = NULL;
synchronize_rcu();
@@ -128,10 +111,17 @@ static int pci_p2pdma_setup(struct pci_dev *pdev)
if (!p2p)
return -ENOMEM;
+ init_completion(&p2p->ref_done);
+
p2p->pool = gen_pool_create(PAGE_SHIFT, dev_to_node(&pdev->dev));
if (!p2p->pool)
goto out;
+ error = percpu_ref_init(&p2p->ref, pci_p2pdma_percpu_release, 0,
+ GFP_KERNEL);
+ if (error)
+ goto out_pool_destroy;
+
error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_release, pdev);
if (error)
goto out_pool_destroy;
@@ -165,8 +155,7 @@ static int pci_p2pdma_setup(struct pci_dev *pdev)
int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size,
u64 offset)
{
- struct p2pdma_pagemap *p2p_pgmap;
- struct dev_pagemap *pgmap;
+ struct resource res;
void *addr;
int error;
@@ -188,50 +177,26 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size,
return error;
}
- p2p_pgmap = devm_kzalloc(&pdev->dev, sizeof(*p2p_pgmap), GFP_KERNEL);
- if (!p2p_pgmap)
- return -ENOMEM;
+ res.start = pci_resource_start(pdev, bar) + offset;
+ res.end = res.start + size - 1;
+ res.flags = pci_resource_flags(pdev, bar);
- init_completion(&p2p_pgmap->ref_done);
- error = percpu_ref_init(&p2p_pgmap->ref,
- pci_p2pdma_percpu_release, 0, GFP_KERNEL);
- if (error)
- goto pgmap_free;
-
- pgmap = &p2p_pgmap->pgmap;
-
- pgmap->res.start = pci_resource_start(pdev, bar) + offset;
- pgmap->res.end = pgmap->res.start + size - 1;
- pgmap->res.flags = pci_resource_flags(pdev, bar);
- pgmap->ref = &p2p_pgmap->ref;
- pgmap->type = MEMORY_DEVICE_PCI_P2PDMA;
- pgmap->pci_p2pdma_bus_offset = pci_bus_address(pdev, bar) -
- pci_resource_start(pdev, bar);
- pgmap->kill = pci_p2pdma_percpu_kill;
- pgmap->cleanup = pci_p2pdma_percpu_cleanup;
-
- addr = devm_memremap_pages(&pdev->dev, pgmap);
- if (IS_ERR(addr)) {
- error = PTR_ERR(addr);
- goto pgmap_free;
- }
+ addr = devm_memremap(&pdev->dev, res.start, size, MEMREMAP_WC);
+ if (IS_ERR(addr))
+ return PTR_ERR(addr);
- error = gen_pool_add_owner(pdev->p2pdma->pool, (unsigned long)addr,
- pci_bus_address(pdev, bar) + offset,
- resource_size(&pgmap->res), dev_to_node(&pdev->dev),
- &p2p_pgmap->ref);
+ error = gen_pool_add_virt(pdev->p2pdma->pool, (unsigned long)addr,
+ pci_bus_address(pdev, bar) + offset, size,
+ dev_to_node(&pdev->dev));
if (error)
goto pages_free;
- pci_info(pdev, "added peer-to-peer DMA memory %pR\n",
- &pgmap->res);
+ pci_info(pdev, "added peer-to-peer DMA memory %pR\n", &res);
return 0;
pages_free:
- devm_memunmap_pages(&pdev->dev, pgmap);
-pgmap_free:
- devm_kfree(&pdev->dev, p2p_pgmap);
+ devm_memunmap(&pdev->dev, addr);
return error;
}
EXPORT_SYMBOL_GPL(pci_p2pdma_add_resource);
@@ -601,7 +566,6 @@ EXPORT_SYMBOL_GPL(pci_p2pmem_find_many);
void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size)
{
void *ret = NULL;
- struct percpu_ref *ref;
/*
* Pairs with synchronize_rcu() in pci_p2pdma_release() to
@@ -612,16 +576,13 @@ void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size)
if (unlikely(!pdev->p2pdma))
goto out;
- ret = (void *)gen_pool_alloc_owner(pdev->p2pdma->pool, size,
- (void **) &ref);
- if (!ret)
+ if (unlikely(!percpu_ref_tryget_live(&pdev->p2pdma->ref)))
goto out;
- if (unlikely(!percpu_ref_tryget_live(ref))) {
- gen_pool_free(pdev->p2pdma->pool, (unsigned long) ret, size);
- ret = NULL;
- goto out;
- }
+ ret = (void *)gen_pool_alloc(pdev->p2pdma->pool, size);
+ if (!ret)
+ percpu_ref_put(&pdev->p2pdma->ref);
+
out:
rcu_read_unlock();
return ret;
@@ -636,11 +597,9 @@ EXPORT_SYMBOL_GPL(pci_alloc_p2pmem);
*/
void pci_free_p2pmem(struct pci_dev *pdev, void *addr, size_t size)
{
- struct percpu_ref *ref;
+ gen_pool_free(pdev->p2pdma->pool, (uintptr_t)addr, size);
- gen_pool_free_owner(pdev->p2pdma->pool, (uintptr_t)addr, size,
- (void **) &ref);
- percpu_ref_put(ref);
+ percpu_ref_put(&pdev->p2pdma->ref);
}
EXPORT_SYMBOL_GPL(pci_free_p2pmem);
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 26/28] PCI/P2PDMA: Remove SGL helpers
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
The functions, pci_p2pmem_alloc_sgl(), pci_p2pmem_free_sgl() and
pci_p2pdma_map_sg() no longer have any callers, so remove them.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
Documentation/driver-api/pci/p2pdma.rst | 9 +--
drivers/pci/p2pdma.c | 95 -------------------------
include/linux/pci-p2pdma.h | 19 -----
3 files changed, 3 insertions(+), 120 deletions(-)
diff --git a/Documentation/driver-api/pci/p2pdma.rst b/Documentation/driver-api/pci/p2pdma.rst
index 44deb52beeb4..5b19c420d921 100644
--- a/Documentation/driver-api/pci/p2pdma.rst
+++ b/Documentation/driver-api/pci/p2pdma.rst
@@ -84,9 +84,8 @@ Client Drivers
--------------
A client driver typically only has to conditionally change its DMA map
-routine to use the mapping function :c:func:`pci_p2pdma_map_sg()` instead
-of the usual :c:func:`dma_map_sg()` function. Memory mapped in this
-way does not need to be unmapped.
+routine to use the PCI bus address with :c:func:`pci_p2pmem_virt_to_bus()`
+for the DMA address instead of the usual :c:func:`dma_map_sg()` function.
The client may also, optionally, make use of
:c:func:`is_pci_p2pdma_page()` to determine when to use the P2P mapping
@@ -117,9 +116,7 @@ returned with pci_dev_put().
Once a provider is selected, the orchestrator can then use
:c:func:`pci_alloc_p2pmem()` and :c:func:`pci_free_p2pmem()` to
-allocate P2P memory from the provider. :c:func:`pci_p2pmem_alloc_sgl()`
-and :c:func:`pci_p2pmem_free_sgl()` are convenience functions for
-allocating scatter-gather lists with P2P memory.
+allocate P2P memory from the provider.
Struct Page Caveats
-------------------
diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
index a98126ad9c3a..9b82e13f802c 100644
--- a/drivers/pci/p2pdma.c
+++ b/drivers/pci/p2pdma.c
@@ -666,60 +666,6 @@ pci_bus_addr_t pci_p2pmem_virt_to_bus(struct pci_dev *pdev, void *addr)
}
EXPORT_SYMBOL_GPL(pci_p2pmem_virt_to_bus);
-/**
- * pci_p2pmem_alloc_sgl - allocate peer-to-peer DMA memory in a scatterlist
- * @pdev: the device to allocate memory from
- * @nents: the number of SG entries in the list
- * @length: number of bytes to allocate
- *
- * Return: %NULL on error or &struct scatterlist pointer and @nents on success
- */
-struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev,
- unsigned int *nents, u32 length)
-{
- struct scatterlist *sg;
- void *addr;
-
- sg = kzalloc(sizeof(*sg), GFP_KERNEL);
- if (!sg)
- return NULL;
-
- sg_init_table(sg, 1);
-
- addr = pci_alloc_p2pmem(pdev, length);
- if (!addr)
- goto out_free_sg;
-
- sg_set_buf(sg, addr, length);
- *nents = 1;
- return sg;
-
-out_free_sg:
- kfree(sg);
- return NULL;
-}
-EXPORT_SYMBOL_GPL(pci_p2pmem_alloc_sgl);
-
-/**
- * pci_p2pmem_free_sgl - free a scatterlist allocated by pci_p2pmem_alloc_sgl()
- * @pdev: the device to allocate memory from
- * @sgl: the allocated scatterlist
- */
-void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl)
-{
- struct scatterlist *sg;
- int count;
-
- for_each_sg(sgl, sg, INT_MAX, count) {
- if (!sg)
- break;
-
- pci_free_p2pmem(pdev, sg_virt(sg), sg->length);
- }
- kfree(sgl);
-}
-EXPORT_SYMBOL_GPL(pci_p2pmem_free_sgl);
-
/**
* pci_p2pmem_publish - publish the peer-to-peer DMA memory for use by
* other devices with pci_p2pmem_find()
@@ -738,47 +684,6 @@ void pci_p2pmem_publish(struct pci_dev *pdev, bool publish)
}
EXPORT_SYMBOL_GPL(pci_p2pmem_publish);
-/**
- * pci_p2pdma_map_sg - map a PCI peer-to-peer scatterlist for DMA
- * @dev: device doing the DMA request
- * @sg: scatter list to map
- * @nents: elements in the scatterlist
- * @dir: DMA direction
- *
- * Scatterlists mapped with this function should not be unmapped in any way.
- *
- * Returns the number of SG entries mapped or 0 on error.
- */
-int pci_p2pdma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
- enum dma_data_direction dir)
-{
- struct dev_pagemap *pgmap;
- struct scatterlist *s;
- phys_addr_t paddr;
- int i;
-
- /*
- * p2pdma mappings are not compatible with devices that use
- * dma_virt_ops. If the upper layers do the right thing
- * this should never happen because it will be prevented
- * by the check in pci_p2pdma_add_client()
- */
- if (WARN_ON_ONCE(IS_ENABLED(CONFIG_DMA_VIRT_OPS) &&
- dev->dma_ops == &dma_virt_ops))
- return 0;
-
- for_each_sg(sg, s, nents, i) {
- pgmap = sg_page(s)->pgmap;
- paddr = sg_phys(s);
-
- s->dma_address = paddr - pgmap->pci_p2pdma_bus_offset;
- sg_dma_len(s) = s->length;
- }
-
- return nents;
-}
-EXPORT_SYMBOL_GPL(pci_p2pdma_map_sg);
-
/**
* pci_p2pdma_enable_store - parse a configfs/sysfs attribute store
* to enable p2pdma
diff --git a/include/linux/pci-p2pdma.h b/include/linux/pci-p2pdma.h
index bca9bc3e5be7..4a75a3f43444 100644
--- a/include/linux/pci-p2pdma.h
+++ b/include/linux/pci-p2pdma.h
@@ -26,12 +26,7 @@ struct pci_dev *pci_p2pmem_find_many(struct device **clients, int num_clients);
void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size);
void pci_free_p2pmem(struct pci_dev *pdev, void *addr, size_t size);
pci_bus_addr_t pci_p2pmem_virt_to_bus(struct pci_dev *pdev, void *addr);
-struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev,
- unsigned int *nents, u32 length);
-void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl);
void pci_p2pmem_publish(struct pci_dev *pdev, bool publish);
-int pci_p2pdma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
- enum dma_data_direction dir);
int pci_p2pdma_enable_store(const char *page, struct pci_dev **p2p_dev,
bool *use_p2pdma);
ssize_t pci_p2pdma_enable_show(char *page, struct pci_dev *p2p_dev,
@@ -69,23 +64,9 @@ static inline pci_bus_addr_t pci_p2pmem_virt_to_bus(struct pci_dev *pdev,
{
return 0;
}
-static inline struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev,
- unsigned int *nents, u32 length)
-{
- return NULL;
-}
-static inline void pci_p2pmem_free_sgl(struct pci_dev *pdev,
- struct scatterlist *sgl)
-{
-}
static inline void pci_p2pmem_publish(struct pci_dev *pdev, bool publish)
{
}
-static inline int pci_p2pdma_map_sg(struct device *dev,
- struct scatterlist *sg, int nents, enum dma_data_direction dir)
-{
- return 0;
-}
static inline int pci_p2pdma_enable_store(const char *page,
struct pci_dev **p2p_dev, bool *use_p2pdma)
{
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 25/28] IB/core: Remove P2PDMA mapping support in rdma_rw_ctx
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
There are no longer any users submitting P2PDMA struct pages to
rdma_rw_ctx. So it can be removed.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
drivers/infiniband/core/rw.c | 11 ++---------
1 file changed, 2 insertions(+), 9 deletions(-)
diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
index cefa6b930bc8..350b9b730ddc 100644
--- a/drivers/infiniband/core/rw.c
+++ b/drivers/infiniband/core/rw.c
@@ -4,7 +4,6 @@
*/
#include <linux/moduleparam.h>
#include <linux/slab.h>
-#include <linux/pci-p2pdma.h>
#include <rdma/mr_pool.h>
#include <rdma/rw.h>
@@ -271,11 +270,7 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
struct ib_device *dev = qp->pd->device;
int ret;
- if (is_pci_p2pdma_page(sg_page(sg)))
- ret = pci_p2pdma_map_sg(dev->dma_device, sg, sg_cnt, dir);
- else
- ret = ib_dma_map_sg(dev, sg, sg_cnt, dir);
-
+ ret = ib_dma_map_sg(dev, sg, sg_cnt, dir);
if (!ret)
return -ENOMEM;
sg_cnt = ret;
@@ -635,9 +630,7 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
{
__rdma_rw_ctx_destroy(ctx, qp);
- /* P2PDMA contexts do not need to be unmapped */
- if (!is_pci_p2pdma_page(sg_page(sg)))
- ib_dma_unmap_sg(qp->pd->device, sg, sg_cnt, dir);
+ ib_dma_unmap_sg(qp->pd->device, sg, sg_cnt, dir);
}
EXPORT_SYMBOL(rdma_rw_ctx_destroy);
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 24/28] block: Remove PCI_P2PDMA queue flag
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
This flag has been superseded by the DMA_DIRECT functionality.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
include/linux/blkdev.h | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index a5b856324276..9ea800645cf5 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -615,8 +615,7 @@ struct request_queue {
#define QUEUE_FLAG_REGISTERED 22 /* queue has been registered to a disk */
#define QUEUE_FLAG_SCSI_PASSTHROUGH 23 /* queue supports SCSI commands */
#define QUEUE_FLAG_QUIESCED 24 /* queue has been quiesced */
-#define QUEUE_FLAG_PCI_P2PDMA 25 /* device supports PCI p2p requests */
-#define QUEUE_FLAG_DMA_DIRECT 26 /* device supports dma-addr requests */
+#define QUEUE_FLAG_DMA_DIRECT 25 /* device supports dma-addr requests */
#define QUEUE_FLAG_MQ_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \
(1 << QUEUE_FLAG_SAME_COMP))
@@ -641,8 +640,6 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q);
#define blk_queue_dax(q) test_bit(QUEUE_FLAG_DAX, &(q)->queue_flags)
#define blk_queue_scsi_passthrough(q) \
test_bit(QUEUE_FLAG_SCSI_PASSTHROUGH, &(q)->queue_flags)
-#define blk_queue_pci_p2pdma(q) \
- test_bit(QUEUE_FLAG_PCI_P2PDMA, &(q)->queue_flags)
#define blk_queue_dma_direct(q) \
test_bit(QUEUE_FLAG_DMA_DIRECT, &(q)->queue_flags)
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 23/28] nvme-pci: Remove support for PCI_P2PDMA requests
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
These requests have been superseded by dma-direct requests and are
therefore no longer needed.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
drivers/nvme/host/core.c | 2 --
drivers/nvme/host/nvme.h | 3 +--
drivers/nvme/host/pci.c | 27 ++++++++++-----------------
3 files changed, 11 insertions(+), 21 deletions(-)
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 8e876417c44b..63d132c478b4 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3257,8 +3257,6 @@ static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
}
blk_queue_flag_set(QUEUE_FLAG_NONROT, ns->queue);
- if (ctrl->ops->flags & NVME_F_PCI_P2PDMA)
- blk_queue_flag_set(QUEUE_FLAG_PCI_P2PDMA, ns->queue);
if (ctrl->ops->flags & NVME_F_DMA_DIRECT)
blk_queue_flag_set(QUEUE_FLAG_DMA_DIRECT, ns->queue);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index f1dddc95c6a8..d103cecc14dd 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -361,8 +361,7 @@ struct nvme_ctrl_ops {
unsigned int flags;
#define NVME_F_FABRICS (1 << 0)
#define NVME_F_METADATA_SUPPORTED (1 << 1)
-#define NVME_F_PCI_P2PDMA (1 << 2)
-#define NVME_F_DMA_DIRECT (1 << 3)
+#define NVME_F_DMA_DIRECT (1 << 2)
int (*reg_read32)(struct nvme_ctrl *ctrl, u32 off, u32 *val);
int (*reg_write32)(struct nvme_ctrl *ctrl, u32 off, u32 val);
int (*reg_read64)(struct nvme_ctrl *ctrl, u32 off, u64 *val);
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 5957f3a4f261..7f806e76230a 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -564,9 +564,8 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req)
WARN_ON_ONCE(!iod->nents);
- /* P2PDMA requests do not need to be unmapped */
- if (!is_pci_p2pdma_page(sg_page(iod->sg)) &&
- !blk_rq_is_dma_direct(req))
+ /* DMA direct requests do not need to be unmapped */
+ if (!blk_rq_is_dma_direct(req))
dma_unmap_sg(dev->dev, iod->sg, iod->nents, rq_dma_dir(req));
@@ -828,16 +827,14 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
if (blk_rq_nr_phys_segments(req) == 1 && !blk_rq_is_dma_direct(req)) {
struct bio_vec bv = req_bvec(req);
- if (!is_pci_p2pdma_page(bv.bv_page)) {
- if (bv.bv_offset + bv.bv_len <= dev->ctrl.page_size * 2)
- return nvme_setup_prp_simple(dev, req,
- &cmnd->rw, &bv);
+ if (bv.bv_offset + bv.bv_len <= dev->ctrl.page_size * 2)
+ return nvme_setup_prp_simple(dev, req,
+ &cmnd->rw, &bv);
- if (iod->nvmeq->qid &&
- dev->ctrl.sgls & ((1 << 0) | (1 << 1)))
- return nvme_setup_sgl_simple(dev, req,
- &cmnd->rw, &bv);
- }
+ if (iod->nvmeq->qid &&
+ dev->ctrl.sgls & ((1 << 0) | (1 << 1)))
+ return nvme_setup_sgl_simple(dev, req,
+ &cmnd->rw, &bv);
}
iod->dma_len = 0;
@@ -849,10 +846,7 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
if (!iod->nents)
goto out;
- if (is_pci_p2pdma_page(sg_page(iod->sg)))
- nr_mapped = pci_p2pdma_map_sg(dev->dev, iod->sg, iod->nents,
- rq_dma_dir(req));
- else if (blk_rq_is_dma_direct(req))
+ if (blk_rq_is_dma_direct(req))
nr_mapped = iod->nents;
else
nr_mapped = dma_map_sg_attrs(dev->dev, iod->sg, iod->nents,
@@ -2642,7 +2636,6 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = {
.name = "pcie",
.module = THIS_MODULE,
.flags = NVME_F_METADATA_SUPPORTED |
- NVME_F_PCI_P2PDMA |
NVME_F_DMA_DIRECT,
.reg_read32 = nvme_pci_reg_read32,
.reg_write32 = nvme_pci_reg_write32,
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 22/28] nvmet: Use DMA addresses instead of struct pages for P2P
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
Start using the dma-direct bios and DMA address RDMA CTX API.
This removes struct pages from all P2P transactions.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
drivers/nvme/target/core.c | 12 +++++----
drivers/nvme/target/io-cmd-bdev.c | 32 ++++++++++++++++++++---
drivers/nvme/target/nvmet.h | 5 +++-
drivers/nvme/target/rdma.c | 43 +++++++++++++++++++++++--------
4 files changed, 71 insertions(+), 21 deletions(-)
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 7734a6acff85..230e99b63320 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -420,7 +420,7 @@ static int nvmet_p2pmem_ns_enable(struct nvmet_ns *ns)
return -EINVAL;
}
- if (!blk_queue_pci_p2pdma(ns->bdev->bd_queue)) {
+ if (!blk_queue_dma_direct(ns->bdev->bd_queue)) {
pr_err("peer-to-peer DMA is not supported by the driver of %s\n",
ns->device_path);
return -EINVAL;
@@ -926,9 +926,9 @@ int nvmet_req_alloc_sgl(struct nvmet_req *req)
req->p2p_dev = NULL;
if (req->sq->qid && p2p_dev) {
- req->sg = pci_p2pmem_alloc_sgl(p2p_dev, &req->sg_cnt,
- req->transfer_len);
- if (req->sg) {
+ req->p2p_dma_buf = pci_alloc_p2pmem(p2p_dev,
+ req->transfer_len);
+ if (req->p2p_dma_buf) {
req->p2p_dev = p2p_dev;
return 0;
}
@@ -951,10 +951,12 @@ EXPORT_SYMBOL_GPL(nvmet_req_alloc_sgl);
void nvmet_req_free_sgl(struct nvmet_req *req)
{
if (req->p2p_dev)
- pci_p2pmem_free_sgl(req->p2p_dev, req->sg);
+ pci_free_p2pmem(req->p2p_dev, req->p2p_dma_buf,
+ req->transfer_len);
else
sgl_free(req->sg);
+ req->p2p_dev = NULL;
req->sg = NULL;
req->sg_cnt = 0;
}
diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index 061d40b020c7..f5621aeb1d6c 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -6,6 +6,7 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/blkdev.h>
#include <linux/module.h>
+#include <linux/pci-p2pdma.h>
#include "nvmet.h"
int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
@@ -132,6 +133,24 @@ static void nvmet_submit_sg(struct nvmet_req *req, struct bio *bio,
submit_bio(bio);
}
+static void nvmet_submit_p2p(struct nvmet_req *req, struct bio *bio)
+{
+ dma_addr_t addr;
+ int ret;
+
+ addr = pci_p2pmem_virt_to_bus(req->p2p_dev, req->p2p_dma_buf);
+
+ ret = bio_add_dma_addr(req->ns->bdev->bd_queue, bio,
+ addr, req->transfer_len);
+ if (WARN_ON_ONCE(ret != req->transfer_len)) {
+ bio->bi_status = BLK_STS_NOTSUPP;
+ nvmet_bio_done(bio);
+ return;
+ }
+
+ submit_bio(bio);
+}
+
static void nvmet_bdev_execute_rw(struct nvmet_req *req)
{
int sg_cnt = req->sg_cnt;
@@ -139,7 +158,7 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req)
sector_t sector;
int op, op_flags = 0;
- if (!req->sg_cnt) {
+ if (!req->sg_cnt && !req->p2p_dev) {
nvmet_req_complete(req, 0);
return;
}
@@ -153,8 +172,10 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req)
op = REQ_OP_READ;
}
- if (is_pci_p2pdma_page(sg_page(req->sg)))
- op_flags |= REQ_NOMERGE;
+ if (req->p2p_dev) {
+ op_flags |= REQ_DMA_DIRECT;
+ sg_cnt = 1;
+ }
sector = le64_to_cpu(req->cmd->rw.slba);
sector <<= (req->ns->blksize_shift - 9);
@@ -171,7 +192,10 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req)
bio->bi_end_io = nvmet_bio_done;
bio_set_op_attrs(bio, op, op_flags);
- nvmet_submit_sg(req, bio, sector);
+ if (req->p2p_dev)
+ nvmet_submit_p2p(req, bio);
+ else
+ nvmet_submit_sg(req, bio, sector);
}
static void nvmet_bdev_execute_flush(struct nvmet_req *req)
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index c25d88fc9dec..5714e5b5ef04 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -288,7 +288,10 @@ struct nvmet_req {
struct nvmet_sq *sq;
struct nvmet_cq *cq;
struct nvmet_ns *ns;
- struct scatterlist *sg;
+ union {
+ struct scatterlist *sg;
+ void *p2p_dma_buf;
+ };
struct bio_vec inline_bvec[NVMET_MAX_INLINE_BIOVEC];
union {
struct {
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 36d906a7f70d..92bfc7207814 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -15,6 +15,7 @@
#include <linux/string.h>
#include <linux/wait.h>
#include <linux/inet.h>
+#include <linux/pci-p2pdma.h>
#include <asm/unaligned.h>
#include <rdma/ib_verbs.h>
@@ -495,6 +496,18 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
spin_unlock(&queue->rsp_wr_wait_lock);
}
+static void nvmet_rdma_ctx_destroy(struct nvmet_rdma_rsp *rsp)
+{
+ struct nvmet_rdma_queue *queue = rsp->queue;
+
+ if (rsp->req.p2p_dev)
+ rdma_rw_ctx_dma_destroy(&rsp->rw, queue->cm_id->qp,
+ queue->cm_id->port_num);
+ else
+ rdma_rw_ctx_destroy(&rsp->rw, queue->cm_id->qp,
+ queue->cm_id->port_num, rsp->req.sg,
+ rsp->req.sg_cnt, nvmet_data_dir(&rsp->req));
+}
static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
{
@@ -502,11 +515,8 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
atomic_add(1 + rsp->n_rdma, &queue->sq_wr_avail);
- if (rsp->n_rdma) {
- rdma_rw_ctx_destroy(&rsp->rw, queue->cm_id->qp,
- queue->cm_id->port_num, rsp->req.sg,
- rsp->req.sg_cnt, nvmet_data_dir(&rsp->req));
- }
+ if (rsp->n_rdma)
+ nvmet_rdma_ctx_destroy(rsp);
if (rsp->req.sg != rsp->cmd->inline_sg)
nvmet_req_free_sgl(&rsp->req);
@@ -587,9 +597,9 @@ static void nvmet_rdma_read_data_done(struct ib_cq *cq, struct ib_wc *wc)
WARN_ON(rsp->n_rdma <= 0);
atomic_add(rsp->n_rdma, &queue->sq_wr_avail);
- rdma_rw_ctx_destroy(&rsp->rw, queue->cm_id->qp,
- queue->cm_id->port_num, rsp->req.sg,
- rsp->req.sg_cnt, nvmet_data_dir(&rsp->req));
+
+ nvmet_rdma_ctx_destroy(rsp);
+
rsp->n_rdma = 0;
if (unlikely(wc->status != IB_WC_SUCCESS)) {
@@ -663,6 +673,7 @@ static u16 nvmet_rdma_map_sgl_keyed(struct nvmet_rdma_rsp *rsp,
struct rdma_cm_id *cm_id = rsp->queue->cm_id;
u64 addr = le64_to_cpu(sgl->addr);
u32 key = get_unaligned_le32(sgl->key);
+ dma_addr_t dma_addr;
int ret;
rsp->req.transfer_len = get_unaligned_le24(sgl->length);
@@ -675,9 +686,19 @@ static u16 nvmet_rdma_map_sgl_keyed(struct nvmet_rdma_rsp *rsp,
if (ret < 0)
goto error_out;
- ret = rdma_rw_ctx_init(&rsp->rw, cm_id->qp, cm_id->port_num,
- rsp->req.sg, rsp->req.sg_cnt, 0, addr, key,
- nvmet_data_dir(&rsp->req));
+ if (rsp->req.p2p_dev) {
+ dma_addr = pci_p2pmem_virt_to_bus(rsp->req.p2p_dev,
+ rsp->req.p2p_dma_buf);
+
+ ret = rdma_rw_ctx_dma_init(&rsp->rw, cm_id->qp,
+ cm_id->port_num, dma_addr,
+ rsp->req.transfer_len, addr, key,
+ nvmet_data_dir(&rsp->req));
+ } else {
+ ret = rdma_rw_ctx_init(&rsp->rw, cm_id->qp, cm_id->port_num,
+ rsp->req.sg, rsp->req.sg_cnt, 0, addr,
+ key, nvmet_data_dir(&rsp->req));
+ }
if (ret < 0)
goto error_out;
rsp->n_rdma += ret;
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 21/28] nvmet: Split nvmet_bdev_execute_rw() into a helper function
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
Move the mapping of the SG and submission of the bio
into a static helper function to reduce the complexity.
This will be useful in the next patch which submits dma-direct bios
for P2P requests.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
drivers/nvme/target/io-cmd-bdev.c | 52 ++++++++++++++++++-------------
1 file changed, 31 insertions(+), 21 deletions(-)
diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index 7a1cf6437a6a..061d40b020c7 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -103,13 +103,41 @@ static void nvmet_bio_done(struct bio *bio)
bio_put(bio);
}
+static void nvmet_submit_sg(struct nvmet_req *req, struct bio *bio,
+ sector_t sector)
+{
+ int sg_cnt = req->sg_cnt;
+ struct scatterlist *sg;
+ int i;
+
+ for_each_sg(req->sg, sg, req->sg_cnt, i) {
+ while (bio_add_page(bio, sg_page(sg), sg->length, sg->offset)
+ != sg->length) {
+ struct bio *prev = bio;
+
+ bio = bio_alloc(GFP_KERNEL,
+ min(sg_cnt, BIO_MAX_PAGES));
+ bio_set_dev(bio, req->ns->bdev);
+ bio->bi_iter.bi_sector = sector;
+ bio->bi_opf = prev->bi_opf;
+
+ bio_chain(bio, prev);
+ submit_bio(prev);
+ }
+
+ sector += sg->length >> 9;
+ sg_cnt--;
+ }
+
+ submit_bio(bio);
+}
+
static void nvmet_bdev_execute_rw(struct nvmet_req *req)
{
int sg_cnt = req->sg_cnt;
struct bio *bio;
- struct scatterlist *sg;
sector_t sector;
- int op, op_flags = 0, i;
+ int op, op_flags = 0;
if (!req->sg_cnt) {
nvmet_req_complete(req, 0);
@@ -143,25 +171,7 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req)
bio->bi_end_io = nvmet_bio_done;
bio_set_op_attrs(bio, op, op_flags);
- for_each_sg(req->sg, sg, req->sg_cnt, i) {
- while (bio_add_page(bio, sg_page(sg), sg->length, sg->offset)
- != sg->length) {
- struct bio *prev = bio;
-
- bio = bio_alloc(GFP_KERNEL, min(sg_cnt, BIO_MAX_PAGES));
- bio_set_dev(bio, req->ns->bdev);
- bio->bi_iter.bi_sector = sector;
- bio_set_op_attrs(bio, op, op_flags);
-
- bio_chain(bio, prev);
- submit_bio(prev);
- }
-
- sector += sg->length >> 9;
- sg_cnt--;
- }
-
- submit_bio(bio);
+ nvmet_submit_sg(req, bio, sector);
}
static void nvmet_bdev_execute_flush(struct nvmet_req *req)
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 20/28] IB/core: Introduce API for initializing a RW ctx from a DMA address
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
Introduce rdma_rw_ctx_dma_init() and rdma_rw_ctx_dma_destroy() which
peform the same operation as rdma_rw_ctx_init() and
rdma_rw_ctx_destroy() respectively except they operate on a DMA
address and length instead of an SGL.
This will be used for struct page-less P2PDMA, but there's also
been opinions expressed to migrate away from SGLs and struct
pages in the RDMA APIs and this will likely fit with that
effort.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
drivers/infiniband/core/rw.c | 74 ++++++++++++++++++++++++++++++------
include/rdma/rw.h | 6 +++
2 files changed, 69 insertions(+), 11 deletions(-)
diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
index 32ca8429eaae..cefa6b930bc8 100644
--- a/drivers/infiniband/core/rw.c
+++ b/drivers/infiniband/core/rw.c
@@ -319,6 +319,39 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
}
EXPORT_SYMBOL(rdma_rw_ctx_init);
+/**
+ * rdma_rw_ctx_dma_init - initialize a RDMA READ/WRITE context from a
+ * DMA address instead of SGL
+ * @ctx: context to initialize
+ * @qp: queue pair to operate on
+ * @port_num: port num to which the connection is bound
+ * @addr: DMA address to READ/WRITE from/to
+ * @len: length of memory to operate on
+ * @remote_addr:remote address to read/write (relative to @rkey)
+ * @rkey: remote key to operate on
+ * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ
+ *
+ * Returns the number of WQEs that will be needed on the workqueue if
+ * successful, or a negative error code.
+ */
+int rdma_rw_ctx_dma_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
+ u8 port_num, dma_addr_t addr, u32 len, u64 remote_addr,
+ u32 rkey, enum dma_data_direction dir)
+{
+ struct scatterlist sg;
+
+ sg_dma_address(&sg) = addr;
+ sg_dma_len(&sg) = len;
+
+ if (rdma_rw_io_needs_mr(qp->device, port_num, dir, 1))
+ return rdma_rw_init_mr_wrs(ctx, qp, port_num, &sg, 1, 0,
+ remote_addr, rkey, dir);
+ else
+ return rdma_rw_init_single_wr(ctx, qp, &sg, 0, remote_addr,
+ rkey, dir);
+}
+EXPORT_SYMBOL(rdma_rw_ctx_dma_init);
+
/**
* rdma_rw_ctx_signature_init - initialize a RW context with signature offload
* @ctx: context to initialize
@@ -566,17 +599,7 @@ int rdma_rw_ctx_post(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
}
EXPORT_SYMBOL(rdma_rw_ctx_post);
-/**
- * rdma_rw_ctx_destroy - release all resources allocated by rdma_rw_ctx_init
- * @ctx: context to release
- * @qp: queue pair to operate on
- * @port_num: port num to which the connection is bound
- * @sg: scatterlist that was used for the READ/WRITE
- * @sg_cnt: number of entries in @sg
- * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ
- */
-void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
- struct scatterlist *sg, u32 sg_cnt, enum dma_data_direction dir)
+static void __rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp)
{
int i;
@@ -596,6 +619,21 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
BUG();
break;
}
+}
+
+/**
+ * rdma_rw_ctx_destroy - release all resources allocated by rdma_rw_ctx_init
+ * @ctx: context to release
+ * @qp: queue pair to operate on
+ * @port_num: port num to which the connection is bound
+ * @sg: scatterlist that was used for the READ/WRITE
+ * @sg_cnt: number of entries in @sg
+ * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ
+ */
+void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
+ struct scatterlist *sg, u32 sg_cnt, enum dma_data_direction dir)
+{
+ __rdma_rw_ctx_destroy(ctx, qp);
/* P2PDMA contexts do not need to be unmapped */
if (!is_pci_p2pdma_page(sg_page(sg)))
@@ -603,6 +641,20 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
}
EXPORT_SYMBOL(rdma_rw_ctx_destroy);
+/**
+ * rdma_rw_ctx_dma_destroy - release all resources allocated by
+ * rdma_rw_ctx_dma_init
+ * @ctx: context to release
+ * @qp: queue pair to operate on
+ * @port_num: port num to which the connection is bound
+ */
+void rdma_rw_ctx_dma_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
+ u8 port_num)
+{
+ __rdma_rw_ctx_destroy(ctx, qp);
+}
+EXPORT_SYMBOL(rdma_rw_ctx_dma_destroy);
+
/**
* rdma_rw_ctx_destroy_signature - release all resources allocated by
* rdma_rw_ctx_init_signature
diff --git a/include/rdma/rw.h b/include/rdma/rw.h
index 494f79ca3e62..e47f8053af6e 100644
--- a/include/rdma/rw.h
+++ b/include/rdma/rw.h
@@ -58,6 +58,12 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
struct scatterlist *sg, u32 sg_cnt,
enum dma_data_direction dir);
+int rdma_rw_ctx_dma_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
+ u8 port_num, dma_addr_t addr, u32 len, u64 remote_addr,
+ u32 rkey, enum dma_data_direction dir);
+void rdma_rw_ctx_dma_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
+ u8 port_num);
+
int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
u8 port_num, struct scatterlist *sg, u32 sg_cnt,
struct scatterlist *prot_sg, u32 prot_sg_cnt,
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 19/28] nvme-pci: Support dma-direct bios
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
Adding support for dma-direct bios only requires putting a condition
around the call to dma_map_sg() so it is skipped when the request
has the REQ_DMA_ADDR flag.
We then need to indicate support for the queue in much the same way
we did with PCI P2PDMA. Seeing this provides the same support as
PCI P2PDMA those flags will be removed in a subsequent patch.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
drivers/nvme/host/core.c | 2 ++
drivers/nvme/host/nvme.h | 1 +
drivers/nvme/host/pci.c | 10 +++++++---
3 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 120fb593d1da..8e876417c44b 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3259,6 +3259,8 @@ static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
blk_queue_flag_set(QUEUE_FLAG_NONROT, ns->queue);
if (ctrl->ops->flags & NVME_F_PCI_P2PDMA)
blk_queue_flag_set(QUEUE_FLAG_PCI_P2PDMA, ns->queue);
+ if (ctrl->ops->flags & NVME_F_DMA_DIRECT)
+ blk_queue_flag_set(QUEUE_FLAG_DMA_DIRECT, ns->queue);
ns->queue->queuedata = ns;
ns->ctrl = ctrl;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 55553d293a98..f1dddc95c6a8 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -362,6 +362,7 @@ struct nvme_ctrl_ops {
#define NVME_F_FABRICS (1 << 0)
#define NVME_F_METADATA_SUPPORTED (1 << 1)
#define NVME_F_PCI_P2PDMA (1 << 2)
+#define NVME_F_DMA_DIRECT (1 << 3)
int (*reg_read32)(struct nvme_ctrl *ctrl, u32 off, u32 *val);
int (*reg_write32)(struct nvme_ctrl *ctrl, u32 off, u32 val);
int (*reg_read64)(struct nvme_ctrl *ctrl, u32 off, u64 *val);
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 524d6bd6d095..5957f3a4f261 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -565,7 +565,8 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req)
WARN_ON_ONCE(!iod->nents);
/* P2PDMA requests do not need to be unmapped */
- if (!is_pci_p2pdma_page(sg_page(iod->sg)))
+ if (!is_pci_p2pdma_page(sg_page(iod->sg)) &&
+ !blk_rq_is_dma_direct(req))
dma_unmap_sg(dev->dev, iod->sg, iod->nents, rq_dma_dir(req));
@@ -824,7 +825,7 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
blk_status_t ret = BLK_STS_RESOURCE;
int nr_mapped;
- if (blk_rq_nr_phys_segments(req) == 1) {
+ if (blk_rq_nr_phys_segments(req) == 1 && !blk_rq_is_dma_direct(req)) {
struct bio_vec bv = req_bvec(req);
if (!is_pci_p2pdma_page(bv.bv_page)) {
@@ -851,6 +852,8 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
if (is_pci_p2pdma_page(sg_page(iod->sg)))
nr_mapped = pci_p2pdma_map_sg(dev->dev, iod->sg, iod->nents,
rq_dma_dir(req));
+ else if (blk_rq_is_dma_direct(req))
+ nr_mapped = iod->nents;
else
nr_mapped = dma_map_sg_attrs(dev->dev, iod->sg, iod->nents,
rq_dma_dir(req), DMA_ATTR_NO_WARN);
@@ -2639,7 +2642,8 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = {
.name = "pcie",
.module = THIS_MODULE,
.flags = NVME_F_METADATA_SUPPORTED |
- NVME_F_PCI_P2PDMA,
+ NVME_F_PCI_P2PDMA |
+ NVME_F_DMA_DIRECT,
.reg_read32 = nvme_pci_reg_read32,
.reg_write32 = nvme_pci_reg_write32,
.reg_read64 = nvme_pci_reg_read64,
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 18/28] block: Introduce bio_add_dma_addr()
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
bio_add_dma_addr() is analagous to bio_add_page() except it
adds a dma address to a dma-direct bio instead of a struct page.
It also checks to ensure that the queue supports dma address bios and
that we are not mixing dma addresses and struct pages.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
block/bio.c | 38 ++++++++++++++++++++++++++++++++++++++
include/linux/bio.h | 10 ++++++++++
2 files changed, 48 insertions(+)
diff --git a/block/bio.c b/block/bio.c
index 6998fceddd36..02ae72e3ccfa 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -874,6 +874,44 @@ static void bio_release_pages(struct bio *bio)
put_page(bvec->bv_page);
}
+/**
+ * bio_add_dma_addr - attempt to add a dma address to a bio
+ * @q: the target queue
+ * @bio: destination bio
+ * @dma_addr: dma address to add
+ * @len: vec entry length
+ *
+ * Attempt to add a dma address to the dma_vec maplist. This can
+ * fail for a number of reasons, such as the bio being full or
+ * target block device limitations. The target request queue must
+ * support dma-only bios and bios can not mix pages and dma_addresses.
+ */
+int bio_add_dma_addr(struct request_queue *q, struct bio *bio,
+ dma_addr_t dma_addr, unsigned int len)
+{
+ struct dma_vec *dv = &bio->bi_dma_vec[bio->bi_vcnt];
+
+ if (!blk_queue_dma_direct(q))
+ return -EINVAL;
+
+ if (!bio_is_dma_direct(bio))
+ return -EINVAL;
+
+ if (bio_dma_full(bio))
+ return 0;
+
+ WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED));
+
+ dv->dv_addr = dma_addr;
+ dv->dv_len = len;
+
+ bio->bi_iter.bi_size += len;
+ bio->bi_vcnt++;
+
+ return len;
+}
+EXPORT_SYMBOL_GPL(bio_add_dma_addr);
+
static int __bio_iov_bvec_add_pages(struct bio *bio, struct iov_iter *iter)
{
const struct bio_vec *bv = iter->bvec;
diff --git a/include/linux/bio.h b/include/linux/bio.h
index df7973932525..d775f381ae00 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -112,6 +112,13 @@ static inline bool bio_full(struct bio *bio)
return bio->bi_vcnt >= bio->bi_max_vecs;
}
+static inline bool bio_dma_full(struct bio *bio)
+{
+ size_t vec_size = bio->bi_max_vecs * sizeof(struct bio_vec);
+
+ return bio->bi_vcnt >= (vec_size / sizeof(struct dma_vec));
+}
+
static inline bool bio_next_segment(const struct bio *bio,
struct bvec_iter_all *iter)
{
@@ -438,6 +445,9 @@ void bio_chain(struct bio *, struct bio *);
extern int bio_add_page(struct bio *, struct page *, unsigned int,unsigned int);
extern int bio_add_pc_page(struct request_queue *, struct bio *, struct page *,
unsigned int, unsigned int);
+extern int bio_add_dma_addr(struct request_queue *q, struct bio *bio,
+ dma_addr_t dma_addr, unsigned int len);
+
bool __bio_try_merge_page(struct bio *bio, struct page *page,
unsigned int len, unsigned int off, bool same_page);
void __bio_add_page(struct bio *bio, struct page *page,
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 17/28] block: Introduce queue flag to indicate support for dma-direct bios
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
Queues will need to advertise support to accept dma-direct requests.
The existing PCI P2P support which will be replaced by this and thus
the P2P flag will be dropped in a subsequent patch.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
include/linux/blkdev.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index ce70d5dded5f..a5b856324276 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -616,6 +616,7 @@ struct request_queue {
#define QUEUE_FLAG_SCSI_PASSTHROUGH 23 /* queue supports SCSI commands */
#define QUEUE_FLAG_QUIESCED 24 /* queue has been quiesced */
#define QUEUE_FLAG_PCI_P2PDMA 25 /* device supports PCI p2p requests */
+#define QUEUE_FLAG_DMA_DIRECT 26 /* device supports dma-addr requests */
#define QUEUE_FLAG_MQ_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \
(1 << QUEUE_FLAG_SAME_COMP))
@@ -642,6 +643,8 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q);
test_bit(QUEUE_FLAG_SCSI_PASSTHROUGH, &(q)->queue_flags)
#define blk_queue_pci_p2pdma(q) \
test_bit(QUEUE_FLAG_PCI_P2PDMA, &(q)->queue_flags)
+#define blk_queue_dma_direct(q) \
+ test_bit(QUEUE_FLAG_DMA_DIRECT, &(q)->queue_flags)
#define blk_noretry_request(rq) \
((rq)->cmd_flags & (REQ_FAILFAST_DEV|REQ_FAILFAST_TRANSPORT| \
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 16/28] block: Implement mapping dma-direct requests to SGs in blk_rq_map_sg()
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
blk_rq_map_sg() just needs to move the dma_vec into the dma_address
of the sgl. Callers will need to ensure not to call dma_map_sg()
for dma-direct requests.
This will likely get less ugly with Christoph's proposed cleanup
to the DMA API. It will be much simpler if devices are just
calling a dma_map_bio() and don't have to worry about dma-direct
requests.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
block/blk-merge.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 65 insertions(+)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index a7a5453987f9..ccd6c44b9f6e 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -545,6 +545,69 @@ static int __blk_bios_map_sg(struct request_queue *q, struct bio *bio,
return nsegs;
}
+static unsigned blk_dvec_map_sg(struct request_queue *q,
+ struct dma_vec *dvec, struct scatterlist *sglist,
+ struct scatterlist **sg)
+{
+ unsigned nbytes = dvec->dv_len;
+ unsigned nsegs = 0, total = 0;
+
+ while (nbytes > 0) {
+ unsigned seg_size;
+
+ *sg = blk_next_sg(sg, sglist);
+
+ seg_size = get_max_segment_size(q, total);
+ seg_size = min(nbytes, seg_size);
+
+ (*sg)->dma_address = dvec->dv_addr + total;
+ sg_dma_len(*sg) = seg_size;
+
+ total += seg_size;
+ nbytes -= seg_size;
+ nsegs++;
+ }
+
+ return nsegs;
+}
+
+static inline void
+__blk_segment_dma_map_sg(struct request_queue *q, struct dma_vec *dvec,
+ struct scatterlist *sglist, struct dma_vec *dvprv,
+ struct scatterlist **sg, int *nsegs)
+{
+ int nbytes = dvec->dv_len;
+
+ if (*sg) {
+ if ((*sg)->length + nbytes > queue_max_segment_size(q))
+ goto new_segment;
+ if (!dmavec_phys_mergeable(q, dvprv, dvec))
+ goto new_segment;
+
+ (*sg)->length += nbytes;
+ } else {
+new_segment:
+ (*nsegs) += blk_dvec_map_sg(q, dvec, sglist, sg);
+ }
+ *dvprv = *dvec;
+}
+
+static int __blk_dma_bios_map_sg(struct request_queue *q, struct bio *bio,
+ struct scatterlist *sglist,
+ struct scatterlist **sg)
+{
+ struct dma_vec dvec, dvprv = {};
+ struct bvec_iter iter;
+ int nsegs = 0;
+
+ for_each_bio(bio)
+ bio_for_each_dvec(dvec, bio, iter)
+ __blk_segment_dma_map_sg(q, &dvec, sglist, &dvprv,
+ sg, &nsegs);
+
+ return nsegs;
+}
+
/*
* map a request to scatterlist, return number of sg entries setup. Caller
* must make sure sg can hold rq->nr_phys_segments entries
@@ -559,6 +622,8 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq,
nsegs = __blk_bvec_map_sg(rq->special_vec, sglist, &sg);
else if (rq->bio && bio_op(rq->bio) == REQ_OP_WRITE_SAME)
nsegs = __blk_bvec_map_sg(bio_iovec(rq->bio), sglist, &sg);
+ else if (blk_rq_is_dma_direct(rq))
+ nsegs = __blk_dma_bios_map_sg(q, rq->bio, sglist, &sg);
else if (rq->bio)
nsegs = __blk_bios_map_sg(q, rq->bio, sglist, &sg);
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 15/28] block: Support counting dma-direct bio segments
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
Change __blk_recalc_rq_segments() to loop through dma_vecs when
appropriate. It calls vec_split_segs() for each dma_vec or bio_vec.
Once this is done the bvec_split_segs() helper is no longer used.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
block/blk-merge.c | 41 ++++++++++++++++++++++++++++++-----------
1 file changed, 30 insertions(+), 11 deletions(-)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index c4c016f994f6..a7a5453987f9 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -194,13 +194,6 @@ static bool vec_split_segs(struct request_queue *q, unsigned offset,
return !!len;
}
-static bool bvec_split_segs(struct request_queue *q, struct bio_vec *bv,
- unsigned *nsegs, unsigned *sectors, unsigned max_segs)
-{
- return vec_split_segs(q, bv->bv_offset, bv->bv_len, nsegs,
- sectors, max_segs);
-}
-
struct blk_segment_split_ctx {
unsigned nsegs;
unsigned sectors;
@@ -366,12 +359,36 @@ void blk_queue_split(struct request_queue *q, struct bio **bio)
}
EXPORT_SYMBOL(blk_queue_split);
+static unsigned int bio_calc_segs(struct request_queue *q, struct bio *bio)
+{
+ unsigned int nsegs = 0;
+ struct bvec_iter iter;
+ struct bio_vec bv;
+
+ bio_for_each_bvec(bv, bio, iter)
+ vec_split_segs(q, bv.bv_offset, bv.bv_len, &nsegs,
+ NULL, UINT_MAX);
+
+ return nsegs;
+}
+
+static unsigned int bio_dma_calc_segs(struct request_queue *q, struct bio *bio)
+{
+ unsigned int nsegs = 0;
+ struct bvec_iter iter;
+ struct dma_vec dv;
+
+ bio_for_each_dvec(dv, bio, iter)
+ vec_split_segs(q, dv.dv_addr, dv.dv_len, &nsegs,
+ NULL, UINT_MAX);
+
+ return nsegs;
+}
+
static unsigned int __blk_recalc_rq_segments(struct request_queue *q,
struct bio *bio)
{
unsigned int nr_phys_segs = 0;
- struct bvec_iter iter;
- struct bio_vec bv;
if (!bio)
return 0;
@@ -386,8 +403,10 @@ static unsigned int __blk_recalc_rq_segments(struct request_queue *q,
}
for_each_bio(bio) {
- bio_for_each_bvec(bv, bio, iter)
- bvec_split_segs(q, &bv, &nr_phys_segs, NULL, UINT_MAX);
+ if (bio_is_dma_direct(bio))
+ nr_phys_segs += bio_calc_segs(q, bio);
+ else
+ nr_phys_segs += bio_dma_calc_segs(q, bio);
}
return nr_phys_segs;
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 14/28] block: Support splitting dma-direct bios
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
If the bio is a dma-direct bio, loop through the dma_vecs instead
of the bio_vecs when calling vec_should_split().
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
block/blk-merge.c | 45 +++++++++++++++++++++++++++++++++++++--------
1 file changed, 37 insertions(+), 8 deletions(-)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index 32653fca53ce..c4c016f994f6 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -257,14 +257,44 @@ static bool vec_should_split(struct request_queue *q, unsigned offset,
return false;
}
+static bool bio_should_split(struct request_queue *q, struct bio *bio,
+ struct blk_segment_split_ctx *ctx)
+{
+ struct bvec_iter iter;
+ struct bio_vec bv;
+ bool ret;
+
+ bio_for_each_bvec(bv, bio, iter) {
+ ret = vec_should_split(q, bv.bv_offset, bv.bv_len, ctx);
+ if (ret)
+ return true;
+ }
+
+ return false;
+}
+
+static bool bio_dma_should_split(struct request_queue *q, struct bio *bio,
+ struct blk_segment_split_ctx *ctx)
+{
+ struct bvec_iter iter;
+ struct dma_vec dv;
+ bool ret;
+
+ bio_for_each_dvec(dv, bio, iter) {
+ ret = vec_should_split(q, dv.dv_addr, dv.dv_len, ctx);
+ if (ret)
+ return true;
+ }
+
+ return false;
+}
+
static struct bio *blk_bio_segment_split(struct request_queue *q,
struct bio *bio,
struct bio_set *bs,
unsigned *segs)
{
- struct bio_vec bv;
- struct bvec_iter iter;
- bool do_split = false;
+ bool do_split;
struct bio *new = NULL;
struct blk_segment_split_ctx ctx = {
@@ -272,11 +302,10 @@ static struct bio *blk_bio_segment_split(struct request_queue *q,
.max_segs = queue_max_segments(q),
};
- bio_for_each_bvec(bv, bio, iter) {
- do_split = vec_should_split(q, bv.bv_offset, bv.bv_len, &ctx);
- if (do_split)
- break;
- }
+ if (bio_is_dma_direct(bio))
+ do_split = bio_dma_should_split(q, bio, &ctx);
+ else
+ do_split = bio_should_split(q, bio, &ctx);
*segs = ctx.nsegs;
--
2.20.1
^ permalink raw reply related
* [RFC PATCH 13/28] block: Generalize bvec_should_split()
From: Logan Gunthorpe @ 2019-06-20 16:12 UTC (permalink / raw)
To: linux-kernel, linux-block, linux-nvme, linux-pci, linux-rdma
Cc: Jens Axboe, Christoph Hellwig, Bjorn Helgaas, Dan Williams,
Sagi Grimberg, Keith Busch, Jason Gunthorpe, Stephen Bates,
Logan Gunthorpe
In-Reply-To: <20190620161240.22738-1-logang@deltatee.com>
bvec_should_split() will need to also operate on dma_vecs so
generalize it to take an offset and length instead of a bio_vec.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
block/blk-merge.c | 31 +++++++++++++++++--------------
1 file changed, 17 insertions(+), 14 deletions(-)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index d9e89c0ad40d..32653fca53ce 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -206,23 +206,25 @@ struct blk_segment_split_ctx {
unsigned sectors;
bool prv_valid;
- struct bio_vec bvprv;
+ unsigned prv_offset;
+ unsigned prv_len;
const unsigned max_sectors;
const unsigned max_segs;
};
-static bool bvec_should_split(struct request_queue *q, struct bio_vec *bv,
- struct blk_segment_split_ctx *ctx)
+static bool vec_should_split(struct request_queue *q, unsigned offset,
+ unsigned len, struct blk_segment_split_ctx *ctx)
{
/*
* If the queue doesn't support SG gaps and adding this
* offset would create a gap, disallow it.
*/
- if (ctx->prv_valid && bvec_gap_to_prev(q, &ctx->bvprv, bv->bv_offset))
+ if (ctx->prv_valid &&
+ vec_gap_to_prev(q, ctx->prv_offset, ctx->prv_len, offset))
return true;
- if (ctx->sectors + (bv->bv_len >> 9) > ctx->max_sectors) {
+ if (ctx->sectors + (len >> 9) > ctx->max_sectors) {
/*
* Consider this a new segment if we're splitting in
* the middle of this vector.
@@ -230,9 +232,9 @@ static bool bvec_should_split(struct request_queue *q, struct bio_vec *bv,
if (ctx->nsegs < ctx->max_segs &&
ctx->sectors < ctx->max_sectors) {
/* split in the middle of bvec */
- bv->bv_len = (ctx->max_sectors - ctx->sectors) << 9;
- bvec_split_segs(q, bv, &ctx->nsegs,
- &ctx->sectors, ctx->max_segs);
+ len = (ctx->max_sectors - ctx->sectors) << 9;
+ vec_split_segs(q, offset, len, &ctx->nsegs,
+ &ctx->sectors, ctx->max_segs);
}
return true;
}
@@ -240,14 +242,15 @@ static bool bvec_should_split(struct request_queue *q, struct bio_vec *bv,
if (ctx->nsegs == ctx->max_segs)
return true;
- ctx->bvprv = *bv;
+ ctx->prv_offset = offset;
+ ctx->prv_len = len;
ctx->prv_valid = true;
- if (bv->bv_offset + bv->bv_len <= PAGE_SIZE) {
+ if (offset + len <= PAGE_SIZE) {
ctx->nsegs++;
- ctx->sectors += bv->bv_len >> 9;
- } else if (bvec_split_segs(q, bv, &ctx->nsegs, &ctx->sectors,
- ctx->max_segs)) {
+ ctx->sectors += len >> 9;
+ } else if (vec_split_segs(q, offset, len, &ctx->nsegs, &ctx->sectors,
+ ctx->max_segs)) {
return true;
}
@@ -270,7 +273,7 @@ static struct bio *blk_bio_segment_split(struct request_queue *q,
};
bio_for_each_bvec(bv, bio, iter) {
- do_split = bvec_should_split(q, &bv, &ctx);
+ do_split = vec_should_split(q, bv.bv_offset, bv.bv_len, &ctx);
if (do_split)
break;
}
--
2.20.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox