* Re: SQ overflow seen running isert traffic
From: Jason Gunthorpe @ 2016-10-18 16:13 UTC (permalink / raw)
To: Steve Wise
Cc: 'Sagi Grimberg', 'Potnuri Bharat Teja',
target-devel, nab, linux-rdma
In-Reply-To: <005101d2294c$be5bb460$3b131d20$@opengridcomputing.com>
On Tue, Oct 18, 2016 at 09:34:33AM -0500, Steve Wise wrote:
> 1) we believe the iSER + RW API correctly sizes the SQ, yet we're seeing SQ
> overflows. So the SQ sizing needs more investigation.
NFS had this sort of problem - in that case it was because the code
was assuming that a RQ completion implied SQ space - that is not
legal, only direct completions from SQ WCs can guide available space
in the SQ..
Jason
^ permalink raw reply
* Re: ibacm cleanups
From: Leon Romanovsky @ 2016-10-18 16:25 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1476731482-26491-1-git-send-email-hch-jcswGhMUV9g@public.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 479 bytes --]
On Mon, Oct 17, 2016 at 09:11:09PM +0200, Christoph Hellwig wrote:
> This series fixed various sparse issues in ibacm, makes it use
> the common list helpers and removes various cruft.
Thanks, applied
https://github.com/linux-rdma/rdma-core/pull/21
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply
* Re: [PATCH TRIVIAL infiniband-diags] ibportstate.c: When not switch, port can be router port
From: ira.weiny @ 2016-10-18 16:25 UTC (permalink / raw)
To: Hal Rosenstock; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <f44f2a4f-15f7-6ae5-d1fb-7af0a77aeaee-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
On Fri, Sep 23, 2016 at 09:17:10AM -0400, Hal Rosenstock wrote:
>
> Indicate PortInfo is on CA/RT port rather than just CA port when not
> on Switch port. RT is what's used by ibnetdiscover for router nodes.
>
> Signed-off-by: Hal Rosenstock <hal-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>
Thanks applied,
Ira
> ---
> diff --git a/src/ibportstate.c b/src/ibportstate.c
> index cb47aa9..cfb8be7 100644
> --- a/src/ibportstate.c
> +++ b/src/ibportstate.c
> @@ -522,9 +522,9 @@ int main(int argc, char **argv)
> IBEXIT("Can't set M_Key fields on switch port != 0");
>
> if (port_op != QUERY || changed)
> - printf("Initial %s PortInfo:\n", is_switch ? "Switch" : "CA");
> + printf("Initial %s PortInfo:\n", is_switch ? "Switch" : "CA/RT");
> else
> - printf("%s PortInfo:\n", is_switch ? "Switch" : "CA");
> + printf("%s PortInfo:\n", is_switch ? "Switch" : "CA/RT");
> espeed_cap = get_port_info(&portid, data, portnum, is_switch);
> show_port_info(&portid, data, portnum, espeed_cap, is_switch);
> if (is_mlnx_ext_port_info_supported(devid)) {
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH TRIVIAL infiniband-diags] ibaddr.c: Fix LID printfs to be unsigned rather than signed
From: ira. weiny @ 2016-10-18 16:27 UTC (permalink / raw)
To: Hal Rosenstock; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <b0f11628-e2ef-15d9-5a9e-68edbc81ca77-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
On Mon, Sep 26, 2016 at 07:38:46AM -0400, Hal Rosenstock wrote:
>
> %u is what's used in other places for unicast LIDs
>
> Signed-off-by: Hal Rosenstock <hal-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Thanks applied,
Ira
> ---
> diff --git a/src/ibaddr.c b/src/ibaddr.c
> index 5845b4e..1a721d0 100644
> --- a/src/ibaddr.c
> +++ b/src/ibaddr.c
> @@ -83,7 +83,7 @@ static int ib_resolve_addr(ib_portid_t * portid, int portnum, int show_lid,
> printf("LID start 0x%x end 0x%x", portid->lid,
> portid->lid + (1 << lmc) - 1);
> else if (show_lid < 0)
> - printf("LID start %d end %d", portid->lid,
> + printf("LID start %u end %u", portid->lid,
> portid->lid + (1 << lmc) - 1);
> printf("\n");
> return 0;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: building rdma-core from travis
From: Hefty, Sean @ 2016-10-18 16:31 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky
Cc: linux-rdma (linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org)
In-Reply-To: <20161018160716.GB24189-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
> Right. rdma-core does not support precise, at all. Aside from cmake,
> the original source packages had some non-functional 'compat' for such
> an old glibc (from 2012!) that I stripped out as well.
To people who use vi to write C code with text-based email for their code review process, 2012 is spaceship futuristic.
> I can't recommend that anybody working on a c-language project should
> be using the precise images on travis for anything <shrug>
We use containers, which are supposed to be faster for processing.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: building rdma-core from travis
From: Jason Gunthorpe @ 2016-10-18 17:00 UTC (permalink / raw)
To: Hefty, Sean
Cc: Leon Romanovsky,
linux-rdma (linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org)
In-Reply-To: <1828884A29C6694DAF28B7E6B8A82373AB096CC3-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
On Tue, Oct 18, 2016 at 04:31:23PM +0000, Hefty, Sean wrote:
> > Right. rdma-core does not support precise, at all. Aside from cmake,
> > the original source packages had some non-functional 'compat' for such
> > an old glibc (from 2012!) that I stripped out as well.
>
> To people who use vi to write C code with text-based email for their
> code review process, 2012 is spaceship futuristic.
Well, those people will suffer then *shrug*
> > I can't recommend that anybody working on a c-language project should
> > be using the precise images on travis for anything <shrug>
>
> We use containers, which are supposed to be faster for processing.
I seem to recall the GCE images have faster CPUs but spend more time
booting the VM.
For rdma-core I counter balanced the VM spin up time by avoiding the
matrix feature. rdma-core has a very fast compile at 19s, I can run
all four compiler passes in less time than it takes to spin up and
configure the VM, container or not :\.
For something like libfabric you might get a win if you do the same
and run fabtests on one of the compiler's output instead of for both.
Plus, if you use trusty you don't need to download and compile
rdma-core, just install libibverbs-dev and librdmacm-dev from Ubuntu.
But at the end of the day, it depend what you expect out of
travis. rdma-core has travis setup to try and show that the code is
clean with bleeding edge rolling release distros (eg forward
looking).
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH rdma-core] libibumad/sysfs.h: Add copyright
From: Hal Rosenstock @ 2016-10-18 17:20 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Signed-off-by: Hal Rosenstock <hal-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
diff --git a/libibumad/sysfs.h b/libibumad/sysfs.h
index 7f8cbfe..489f24b 100644
--- a/libibumad/sysfs.h
+++ b/libibumad/sysfs.h
@@ -1,3 +1,35 @@
+/*
+ * Copyright (c) 2008 Voltaire Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ */
#ifndef _UMAD_SYSFS_H
#define _UMAD_SYSFS_H
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH] IB/mad: Eliminate redundant SM class version defines for OPA
From: Hal Rosenstock @ 2016-10-18 17:20 UTC (permalink / raw)
To: Doug Ledford
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Weiny, Ira
and rename class version define to indicate SM rather than SMP or SMI
Signed-off-by: Hal Rosenstock <hal-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Ira Weiny <ira.weiny-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c
index 40cbd6b..bc2a9c2 100644
--- a/drivers/infiniband/core/mad.c
+++ b/drivers/infiniband/core/mad.c
@@ -769,7 +769,7 @@ static int handle_outgoing_dr_smp(struct ib_mad_agent_private *mad_agent_priv,
* If we are at the start of the LID routed part, don't update the
* hop_ptr or hop_cnt. See section 14.2.2, Vol 1 IB spec.
*/
- if (opa && smp->class_version == OPA_SMP_CLASS_VERSION) {
+ if (opa && smp->class_version == OPA_SM_CLASS_VERSION) {
u32 opa_drslid;
if ((opa_get_smp_direction(opa_smp)
@@ -2167,7 +2167,7 @@ static bool generate_unmatched_resp(const struct ib_mad_private *recv,
struct ib_mad_hdr *mad_hdr = (struct ib_mad_hdr *)recv->mad;
if (opa && mad_hdr->base_version == OPA_MGMT_BASE_VERSION &&
- mad_hdr->class_version == OPA_SMI_CLASS_VERSION)
+ mad_hdr->class_version == OPA_SM_CLASS_VERSION)
return handle_opa_smi(port_priv, qp_info, wc, port_num, recv,
response);
diff --git a/drivers/infiniband/hw/hfi1/mad.c b/drivers/infiniband/hw/hfi1/mad.c
index 9487c9b..320e4da 100644
--- a/drivers/infiniband/hw/hfi1/mad.c
+++ b/drivers/infiniband/hw/hfi1/mad.c
@@ -128,7 +128,7 @@ static void send_trap(struct hfi1_ibport *ibp, void *data, unsigned len)
smp = send_buf->mad;
smp->base_version = OPA_MGMT_BASE_VERSION;
smp->mgmt_class = IB_MGMT_CLASS_SUBN_LID_ROUTED;
- smp->class_version = OPA_SMI_CLASS_VERSION;
+ smp->class_version = OPA_SM_CLASS_VERSION;
smp->method = IB_MGMT_METHOD_TRAP;
ibp->rvp.tid++;
smp->tid = cpu_to_be64(ibp->rvp.tid);
@@ -343,7 +343,7 @@ static int __subn_get_opa_nodeinfo(struct opa_smp *smp, u32 am, u8 *data,
ni->port_guid = cpu_to_be64(dd->pport[pidx].guid);
ni->base_version = OPA_MGMT_BASE_VERSION;
- ni->class_version = OPA_SMI_CLASS_VERSION;
+ ni->class_version = OPA_SM_CLASS_VERSION;
ni->node_type = 1; /* channel adapter */
ni->num_ports = ibdev->phys_port_cnt;
/* This is already in network order */
@@ -379,7 +379,7 @@ static int subn_get_nodeinfo(struct ib_smp *smp, struct ib_device *ibdev,
nip->port_guid = cpu_to_be64(dd->pport[pidx].guid);
nip->base_version = OPA_MGMT_BASE_VERSION;
- nip->class_version = OPA_SMI_CLASS_VERSION;
+ nip->class_version = OPA_SM_CLASS_VERSION;
nip->node_type = 1; /* channel adapter */
nip->num_ports = ibdev->phys_port_cnt;
/* This is already in network order */
@@ -2302,7 +2302,7 @@ static int pma_get_opa_classportinfo(struct opa_pma_mad *pmp,
pmp->mad_hdr.status |= IB_SMP_INVALID_FIELD;
p->base_version = OPA_MGMT_BASE_VERSION;
- p->class_version = OPA_SMI_CLASS_VERSION;
+ p->class_version = OPA_SM_CLASS_VERSION;
/*
* Expected response time is 4.096 usec. * 2^18 == 1.073741824 sec.
*/
@@ -4022,7 +4022,7 @@ static int process_subn_opa(struct ib_device *ibdev, int mad_flags,
am = be32_to_cpu(smp->attr_mod);
attr_id = smp->attr_id;
- if (smp->class_version != OPA_SMI_CLASS_VERSION) {
+ if (smp->class_version != OPA_SM_CLASS_VERSION) {
smp->status |= IB_SMP_UNSUP_VERSION;
ret = reply((struct ib_mad_hdr *)smp);
return ret;
@@ -4232,7 +4232,7 @@ static int process_perf_opa(struct ib_device *ibdev, u8 port,
*out_mad = *in_mad;
- if (pmp->mad_hdr.class_version != OPA_SMI_CLASS_VERSION) {
+ if (pmp->mad_hdr.class_version != OPA_SM_CLASS_VERSION) {
pmp->mad_hdr.status |= IB_SMP_UNSUP_VERSION;
return reply((struct ib_mad_hdr *)pmp);
}
diff --git a/include/rdma/ib_mad.h b/include/rdma/ib_mad.h
index c8a773f..981214b 100644
--- a/include/rdma/ib_mad.h
+++ b/include/rdma/ib_mad.h
@@ -46,7 +46,7 @@
#define IB_MGMT_BASE_VERSION 1
#define OPA_MGMT_BASE_VERSION 0x80
-#define OPA_SMP_CLASS_VERSION 0x80
+#define OPA_SM_CLASS_VERSION 0x80
/* Management classes */
#define IB_MGMT_CLASS_SUBN_LID_ROUTED 0x01
diff --git a/include/rdma/opa_smi.h b/include/rdma/opa_smi.h
index 4a529ef..f789611 100644
--- a/include/rdma/opa_smi.h
+++ b/include/rdma/opa_smi.h
@@ -44,8 +44,6 @@
#define OPA_MAX_SLS 32
#define OPA_MAX_SCS 32
-#define OPA_SMI_CLASS_VERSION 0x80
-
#define OPA_LID_PERMISSIVE cpu_to_be32(0xFFFFFFFF)
struct opa_smp {
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH rdma-core] libibumad/sysfs.h: Add copyright
From: Leon Romanovsky @ 2016-10-18 17:30 UTC (permalink / raw)
To: Hal Rosenstock; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <dfe69226-fb23-0bd5-c371-ad55c1356340-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 2175 bytes --]
On Tue, Oct 18, 2016 at 01:20:18PM -0400, Hal Rosenstock wrote:
>
> Signed-off-by: Hal Rosenstock <hal-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Thanks, applied.
https://github.com/linux-rdma/rdma-core/pull/22
> ---
> diff --git a/libibumad/sysfs.h b/libibumad/sysfs.h
> index 7f8cbfe..489f24b 100644
> --- a/libibumad/sysfs.h
> +++ b/libibumad/sysfs.h
> @@ -1,3 +1,35 @@
> +/*
> + * Copyright (c) 2008 Voltaire Inc. All rights reserved.
> + *
> + * This software is available to you under a choice of one of two
> + * licenses. You may choose to be licensed under the terms of the GNU
> + * General Public License (GPL) Version 2, available from the file
> + * COPYING in the main directory of this source tree, or the
> + * OpenIB.org BSD license below:
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * - Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer.
> + *
> + * - Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> + * SOFTWARE.
> + *
> + */
> #ifndef _UMAD_SYSFS_H
> #define _UMAD_SYSFS_H
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply
* [PATCH] qedr: return -EINVAL if pd is null and avoid null ptr dereference
From: Colin King @ 2016-10-18 18:39 UTC (permalink / raw)
To: Doug Ledford, Sean Hefty, Hal Rosenstock, Rajesh Borundia,
Ram Amrani, linux-rdma-u79uwXL29TY76Z2rM5mHXA
Cc: linux-kernel-u79uwXL29TY76Z2rM5mHXA
From: Colin Ian King <colin.king-Z7WLFzj8eWMS+FvcfC7Uqw@public.gmane.org>
Currently, if pd is null then we hit a null pointer derference
on accessing pd->pd_id. Instead of just printing an error message
we should also return -EINVAL immediately.
Signed-off-by: Colin Ian King <colin.king-Z7WLFzj8eWMS+FvcfC7Uqw@public.gmane.org>
---
drivers/infiniband/hw/qedr/verbs.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/qedr/verbs.c b/drivers/infiniband/hw/qedr/verbs.c
index a615142..b2a0eb8 100644
--- a/drivers/infiniband/hw/qedr/verbs.c
+++ b/drivers/infiniband/hw/qedr/verbs.c
@@ -511,8 +511,10 @@ int qedr_dealloc_pd(struct ib_pd *ibpd)
struct qedr_dev *dev = get_qedr_dev(ibpd->device);
struct qedr_pd *pd = get_qedr_pd(ibpd);
- if (!pd)
+ if (!pd) {
pr_err("Invalid PD received in dealloc_pd\n");
+ return -EINVAL;
+ }
DP_DEBUG(dev, QEDR_MSG_INIT, "Deallocating PD %d\n", pd->pd_id);
dev->ops->rdma_dealloc_pd(dev->rdma_ctx, pd->pd_id);
--
2.9.3
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH rdma-core] libibumad/sysfs.h: Add copyright
From: Christoph Hellwig @ 2016-10-18 18:43 UTC (permalink / raw)
To: Hal Rosenstock
Cc: Leon Romanovsky,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <dfe69226-fb23-0bd5-c371-ad55c1356340-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
That seems like an odd copyright statement for a file that I created..
While the delcarations came from another file the whitespaces and
oinclude headers take up just as little lines and have just as little
substance..
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH rdma-core] libibumad/sysfs.h: Add copyright
From: Hal Rosenstock @ 2016-10-18 18:53 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Leon Romanovsky,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <20161018184352.GA15413-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
On 10/18/2016 2:43 PM, Christoph Hellwig wrote:
> That seems like an odd copyright statement for a file that I created..
>
> While the delcarations came from another file the whitespaces and
> oinclude headers take up just as little lines and have just as little
> substance..
What's the best thing to do ?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: SQ overflow seen running isert traffic
From: Steve Wise @ 2016-10-18 19:03 UTC (permalink / raw)
To: 'Jason Gunthorpe'
Cc: 'Sagi Grimberg', 'Potnuri Bharat Teja',
target-devel, nab, linux-rdma
In-Reply-To: <20161018161313.GC24189@obsidianresearch.com>
> On Tue, Oct 18, 2016 at 09:34:33AM -0500, Steve Wise wrote:
>
> > 1) we believe the iSER + RW API correctly sizes the SQ, yet we're seeing SQ
> > overflows. So the SQ sizing needs more investigation.
>
> NFS had this sort of problem - in that case it was because the code
> was assuming that a RQ completion implied SQ space - that is not
> legal, only direct completions from SQ WCs can guide available space
> in the SQ..
>
Agreed. And this could be the case Bharat is hitting. (I wonder if there is a
clever way we can tell?) I don't see I the code that isert flow controls SQ WR
submission based on SQ completions.
It would be nice to have some core API, maybe in rdma-rw, that could handle this
by queuing and processing when the SQ has resources...
Steve.
^ permalink raw reply
* Re: [PATCH rdma-core] libibumad/sysfs.h: Add copyright
From: Christoph Hellwig @ 2016-10-18 19:07 UTC (permalink / raw)
To: Hal Rosenstock
Cc: Christoph Hellwig, Leon Romanovsky,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <c1796e9a-32dc-62d2-481d-7decdc87700b-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
On Tue, Oct 18, 2016 at 02:53:44PM -0400, Hal Rosenstock wrote:
> On 10/18/2016 2:43 PM, Christoph Hellwig wrote:
> > That seems like an odd copyright statement for a file that I created..
> >
> > While the delcarations came from another file the whitespaces and
> > oinclude headers take up just as little lines and have just as little
> > substance..
>
> What's the best thing to do ?
Just not bother with copyright notices for trivial files. E.g. we don't
bother with one for README.md or rdma-core.spec either.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH rdma-core 4/4] glue/redhat/spec: build split rpm packages
From: Jason Gunthorpe @ 2016-10-18 19:28 UTC (permalink / raw)
To: Jarod Wilson
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
dledford-H+wXaHxf7aLQT0dZR+AlfA
In-Reply-To: <20161018144411.GS14983-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Tue, Oct 18, 2016 at 10:44:11AM -0400, Jarod Wilson wrote:
> > I think you can just add ../README.md to the
> > Documentation/CMakeFile.txt
>
> First cut, I added a Files line to the root CMakeLists.txt. Would the
> above be preferred over that? Both seem to work.
It is nitpicky, but I'd put it in the Documentation/CMakelists as
currently the top level is all setup and doesn't build or install
anything.
eg the top level plays a role like configure.ac
Might be more familiar for people..
> $ git archive --prefix=rdma-core-11/ -o ~/rpmbuild/SOURCES/rdma-core-11.tgz --format tar.gz HEAD
Yah, that is the right thing, it should match the tar file what github
will autogenerate when we make a tag.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCHv12 0/3] rdmacg: IB/core: rdma controller support
From: Parav Pandit @ 2016-10-18 20:02 UTC (permalink / raw)
To: Tejun Heo
Cc: Leon Romanovsky, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-rdma,
Li Zefan, Johannes Weiner, Doug Ledford, Christoph Hellwig,
Liran Liss, Hefty, Sean, Jason Gunthorpe, Haggai Eran,
james.l.morris-QHcLZuEGTsvQT0dZR+AlfA, Or Gerlitz, Matan Barak
In-Reply-To: <20161013231413.GA32534-qYNAdHglDFBN0TnZuCh8vA@public.gmane.org>
Hi Tejun,
Sorry for the late response. I was traveling during weekend.
On Fri, Oct 14, 2016 at 4:44 AM, Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> Hello, Parav.
>
> On Mon, Oct 10, 2016 at 07:02:11PM +0530, Parav Pandit wrote:
>> Give me some more time, I will think more and take feeback from Leon
>> and others on
>> (a) how can we implement or want to implement weight like
>> functionality for non-work-conversing resource
>
> I think what you want is just a way to specify absolute limits using
> percentages of what's available at the time of configuration -
> e.g. being able to say "allow upto 30% of what's available in the
> parent".
Yes. I am concerned about how to configure < 1% value to avoid
floating point math in kernel as thats discouraged.
Configuring in range of 1 to 100% for a given resource limits to only
100 or less cgroup instances which I think is not desired.
> If so, the simplest way would be simply updating the
> existing knobs to accept % inputs in addition to absolute values on
> writes.
I was not sure to overload rdma.max file for accepting % inputs as
thats not done in other cgroups. So I was thinking more of weights
interface which avoids floating point problem and also allows much
wider configuration range.
>
> Thanks.
>
> --
> tejun
^ permalink raw reply
* Re: [PATCH rdma-core] libibumad/sysfs.h: Add copyright
From: Hal Rosenstock @ 2016-10-18 20:07 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Leon Romanovsky,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <20161018190745.GA26381-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
On 10/18/2016 3:07 PM, Christoph Hellwig wrote:
> On Tue, Oct 18, 2016 at 02:53:44PM -0400, Hal Rosenstock wrote:
>> On 10/18/2016 2:43 PM, Christoph Hellwig wrote:
>>> That seems like an odd copyright statement for a file that I created..
>>>
>>> While the delcarations came from another file the whitespaces and
>>> oinclude headers take up just as little lines and have just as little
>>> substance..
>>
>> What's the best thing to do ?
>
> Just not bother with copyright notices for trivial files. E.g. we don't
> bother with one for README.md or rdma-core.spec either.
My understanding is that when a new file is originated from an existing
file that it should maintain the copyright of the original file.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCHv12 1/3] rdmacg: Added rdma cgroup controller
From: Parav Pandit @ 2016-10-18 20:15 UTC (permalink / raw)
To: Tejun Heo
Cc: Leon Romanovsky, Jason Gunthorpe, Christoph Hellwig, Matan Barak,
cgroups, linux-doc, Linux Kernel Mailing List, linux-rdma,
Li Zefan, Johannes Weiner, Doug Ledford, Liran Liss, Hefty, Sean,
Haggai Eran, Jonathan Corbet, james.l.morris, serge, Or Gerlitz,
Andrew Morton, linux-security-module
In-Reply-To: <CAG53R5VqQ6UzsuzcovsD+ufv3HaEQEbUpaf3kDo4zLOrAf__8A@mail.gmail.com>
Hi Doug,
Leon has finished review as well in [7].
Christoph Acked too in [8].
Can you please advise whether
(1) I should rebase and resend PatchV12?
(2) If so for which branch - master/4.9 or?
Tejun and Christoph mentioned that it might be late for 4.9.
Can we atleast merge to linux-rdma tree, so that more features/changes
can be done on top of it?
How can we avoid merge conflict to Linus since this patchset is
applicable to two git trees. (cgroup and linux-rdma).
I was thinking to push through linux-rdma as it is currently going
through more changes, so resolving merge conflict would be simpler if
that happens.
Please provide the direction.
[7] https://lkml.org/lkml/2016/10/5/134
[8] https://lkml.org/lkml/2016/10/5/30
Regards,
Parav Pandit
On Tue, Oct 4, 2016 at 11:49 PM, Parav Pandit <pandit.parav@gmail.com> wrote:
> Hi Doug,
>
> I am still waiting for Leon to provide his comments if any on rdma cgroup.
> From other email context, he was on vacation last week.
> While we wait for his comments, I wanted to know your view of this
> patchset in 4.9 merge window.
>
> To summarize the discussion that happened in two threads.
>
> [1] Ack by Tejun, asking for review from rdma list
> [2] quick review by Christoph on patch-v11 (patch 12 has only typo corrections)
> [3] Christoph's ack on architecture of rdma cgroup and fitting it with ABI
> [4] My response on Matan's query on RSS indirection table
> [5] Response from Intel on their driver support for Matan's query
> [6] Christoph's point on architecture, which we are following in new
> ABI and current ABI
>
> I have reviewed recent patch [7] from Matan where I see IB verbs
> objects are still handled through common path as suggested by
> Christoph.
>
> I do not see any issues with rdma cgroup patchset other than it requires rebase.
> Am I missing something?
> Can you please help me - What would be required to merge it to 4.9?
>
> [1] https://lkml.org/lkml/2016/8/31/494
> [2] https://lkml.org/lkml/2016/8/25/146
> [3] https://lkml.org/lkml/2016/9/10/175
> [4] https://lkml.org/lkml/2016/9/14/221
> [5] https://lkml.org/lkml/2016/9/19/571
> [6] http://www.spinics.net/lists/linux-rdma/msg40337.html
> [7] email subject: [RFC ABI V4 0/7] SG-based RDMA ABI Proposal
>
> Regards,
> Parav Pandit
>
> On Wed, Sep 21, 2016 at 9:32 PM, Parav Pandit <pandit.parav@gmail.com> wrote:
>> Hi Tejun,
>>
>> On Wed, Sep 21, 2016 at 7:56 PM, Tejun Heo <tj@kernel.org> wrote:
>>> Hello, Parav.
>>>
>>> On Wed, Sep 21, 2016 at 10:13:38AM +0530, Parav Pandit wrote:
>>>> We have completed review from Tejun, Christoph.
>>>> HFI driver folks also provided feedback for Intel drivers.
>>>> Matan's also doesn't have any more comments.
>>>>
>>>> If possible, if you can also review, it will be helpful.
>>>>
>>>> I have some more changes unrelated to cgroup in same files in both the git tree.
>>>> Pushing them now either results into merge conflict later on for
>>>> Doug/Tejun, or requires rebase and resending patch.
>>>> If you can review, we can avoid such rework.
>>>
>>> My impression of the thread was that there doesn't seem to be enough
>>> of consensus around how rdma resources should be defined. Is that
>>> part agreed upon now?
>>>
>>
>> We ended up discussing few points on different thread [1].
>>
>> There was confusion on how some non-rdma/non-IB drivers would work
>> with rdma cgroup from Matan.
>> Christoph explained how they don't fit in the rdma subsystem and
>> therefore its not prime target to addess.
>> Intel driver maintainer Denny also acknowledged same on [2].
>> IB compliant drivers of Intel support rdma cgroup as explained in [2].
>> With that usnic and Intel psm drivers falls out of rdma cgroup support
>> as they don't fit very well in the verbs definition.
>>
>> [1] https://www.spinics.net/lists/linux-rdma/msg40340.html
>> [2] http://www.spinics.net/lists/linux-rdma/msg40717.html
>>
>> I will wait for Leon's review comments if he has different view on architecture.
>> Back in April when I met face-to-face to Leon and Haggai, Leon was in
>> support to have kernel defined the rdma resources as suggested by
>> Christoph and Tejun instead of IB/RDMA subsystem.
>> I will wait for his comments if his views have changed with new uAPI
>> taking shape.
^ permalink raw reply
* Re: [PATCH rdma-core] libibumad/sysfs.h: Add copyright
From: Christoph Hellwig @ 2016-10-18 20:37 UTC (permalink / raw)
To: Hal Rosenstock
Cc: Christoph Hellwig, Leon Romanovsky,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <f34c1ead-29ec-7968-e112-28af468c38de-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
On Tue, Oct 18, 2016 at 04:07:18PM -0400, Hal Rosenstock wrote:
> My understanding is that when a new file is originated from an existing
> file that it should maintain the copyright of the original file.
IFF there is copytightable material, yes. Either way, no need to waste
time discussing this - I think it's fairly silly, but it it allows
ticking a checkbox go ahead with it..
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] iw_cxgb4: set *bad_wr for post_send/post_recv errors
From: Steve Wise @ 2016-10-18 21:04 UTC (permalink / raw)
To: dledford-H+wXaHxf7aLQT0dZR+AlfA; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
There are a few cases in c4iw_post_send() and c4iw_post_receive()
where *bad_wr is not set when an error is returned. This can
cause a crash if the application tries to use bad_wr.
Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
---
drivers/infiniband/hw/cxgb4/qp.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/infiniband/hw/cxgb4/qp.c b/drivers/infiniband/hw/cxgb4/qp.c
index 6904352..c09fbf3 100644
--- a/drivers/infiniband/hw/cxgb4/qp.c
+++ b/drivers/infiniband/hw/cxgb4/qp.c
@@ -762,11 +762,13 @@ int c4iw_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
spin_lock_irqsave(&qhp->lock, flag);
if (t4_wq_in_error(&qhp->wq)) {
spin_unlock_irqrestore(&qhp->lock, flag);
+ *bad_wr = wr;
return -EINVAL;
}
num_wrs = t4_sq_avail(&qhp->wq);
if (num_wrs == 0) {
spin_unlock_irqrestore(&qhp->lock, flag);
+ *bad_wr = wr;
return -ENOMEM;
}
while (wr) {
@@ -885,11 +887,13 @@ int c4iw_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr,
spin_lock_irqsave(&qhp->lock, flag);
if (t4_wq_in_error(&qhp->wq)) {
spin_unlock_irqrestore(&qhp->lock, flag);
+ *bad_wr = wr;
return -EINVAL;
}
num_wrs = t4_rq_avail(&qhp->wq);
if (num_wrs == 0) {
spin_unlock_irqrestore(&qhp->lock, flag);
+ *bad_wr = wr;
return -ENOMEM;
}
while (wr) {
--
2.7.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 0/3] iopmem : A block device for PCIe memory
From: Stephen Bates @ 2016-10-18 21:42 UTC (permalink / raw)
To: linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-block-u79uwXL29TY76Z2rM5mHXA,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg
Cc: dan.j.williams-ral2JQCrhuEAvxtiuMwx3w,
ross.zwisler-VuQAYsv1563Yd54FQh9/CA, willy-VuQAYsv1563Yd54FQh9/CA,
jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/,
haggaie-VPRAkNaXOzVWk0Htik3J/w, hch-wEGCiKHe2LqWVfeAwA7xHQ,
axboe-b10kYP2dOMg, corbet-T1hC0tSOHrs,
jim.macdonald-FgSLVYC75IpWk0Htik3J/w,
sbates-Rgftl6RXld5BDgjK7y7TUQ, logang-OTvnGxWRz7hWk0Htik3J/w,
Stephen Bates
This patch follows from an RFC we did earlier this year [1]. This
patchset applies cleanly to v4.9-rc1.
Updates since RFC
-----------------
Rebased.
Included the iopmem driver in the submission.
History
-------
There have been several attempts to upstream patchsets that enable
DMAs between PCIe peers. These include Peer-Direct [2] and DMA-Buf
style patches [3]. None have been successful to date. Haggai Eran
gives a nice overview of the prior art in this space in his cover
letter [3].
Motivation and Use Cases
------------------------
PCIe IO devices are getting faster. It is not uncommon now to find PCIe
network and storage devices that can generate and consume several GB/s.
Almost always these devices have either a high performance DMA engine, a
number of exposed PCIe BARs or both.
Until this patch, any high-performance transfer of information between
two PICe devices has required the use of a staging buffer in system
memory. With this patch the bandwidth to system memory is not compromised
when high-throughput transfers occurs between PCIe devices. This means
that more system memory bandwidth is available to the CPU cores for data
processing and manipulation. In addition, in systems where the two PCIe
devices reside behind a PCIe switch the datapath avoids the CPU
entirely.
Consumers
---------
We provide a PCIe device driver in an accompanying patch that can be
used to map any PCIe BAR into a DAX capable block device. For
non-persistent BARs this simply serves as an alternative to using
system memory bounce buffers. For persistent BARs this can serve as an
additional storage device in the system.
Testing and Performance
-----------------------
We have done a moderate about of testing of this patch on a QEMU
environment and on real hardware. On real hardware we have observed
peer-to-peer writes of up to 4GB/s and reads of up to 1.2 GB/s. In
both cases these numbers are limitations of our consumer hardware. In
addition, we have observed that the CPU DRAM bandwidth is not impacted
when using IOPMEM which is not the case when a traditional path
through system memory is taken.
For more information on the testing and performance results see the
GitHub site [4].
Known Issues
------------
1. Address Translation. Suggestions have been made that in certain
architectures and topologies the dma_addr_t passed to the DMA master
in a peer-2-peer transfer will not correctly route to the IO memory
intended. However in our testing to date we have not seen this to be
an issue, even in systems with IOMMUs and PCIe switches. It is our
understanding that an IOMMU only maps system memory and would not
interfere with device memory regions. (It certainly has no opportunity
to do so if the transfer gets routed through a switch).
2. Memory Segment Spacing. This patch has the same limitations that
ZONE_DEVICE does in that memory regions must be spaces at least
SECTION_SIZE bytes part. On x86 this is 128MB and there are cases where
BARs can be placed closer together than this. Thus ZONE_DEVICE would not
be usable on neighboring BARs. For our purposes, this is not an issue as
we'd only be looking at enabling a single BAR in a given PCIe device.
More exotic use cases may have problems with this.
3. Coherency Issues. When IOMEM is written from both the CPU and a PCIe
peer there is potential for coherency issues and for writes to occur out
of order. This is something that users of this feature need to be
cognizant of. Though really, this isn't much different than the
existing situation with things like RDMA: if userspace sets up an MR
for remote use, they need to be careful about using that memory region
themselves.
4. Architecture. Currently this patch is applicable only to x86_64
architectures. The same is true for much of the code pertaining to
PMEM and ZONE_DEVICE. It is hoped that the work will be extended to other
ARCH over time.
References
----------
[1] https://patchwork.kernel.org/patch/8583221/
[2] http://comments.gmane.org/gmane.linux.drivers.rdma/21849
[3] http://www.spinics.net/lists/linux-rdma/msg38748.html
[4] https://github.com/sbates130272/zone-device
Logan Gunthorpe (1):
memremap.c : Add support for ZONE_DEVICE IO memory with struct pages.
Stephen Bates (2):
iopmem : Add a block device driver for PCIe attached IO memory.
iopmem : Add documentation for iopmem driver
Documentation/blockdev/00-INDEX | 2 +
Documentation/blockdev/iopmem.txt | 62 +++++++
MAINTAINERS | 7 +
drivers/block/Kconfig | 27 ++++
drivers/block/Makefile | 1 +
drivers/block/iopmem.c | 333 ++++++++++++++++++++++++++++++++++++++
drivers/dax/pmem.c | 4 +-
drivers/nvdimm/pmem.c | 4 +-
include/linux/memremap.h | 5 +-
kernel/memremap.c | 80 ++++++++-
tools/testing/nvdimm/test/iomap.c | 3 +-
11 files changed, 518 insertions(+), 10 deletions(-)
create mode 100644 Documentation/blockdev/iopmem.txt
create mode 100644 drivers/block/iopmem.c
--
2.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 1/3] memremap.c : Add support for ZONE_DEVICE IO memory with struct pages.
From: Stephen Bates @ 2016-10-18 21:42 UTC (permalink / raw)
To: linux-kernel, linux-nvdimm, linux-rdma, linux-block, linux-mm
Cc: dan.j.williams, ross.zwisler, willy, jgunthorpe, haggaie, hch,
axboe, corbet, jim.macdonald, sbates, logang, Stephen Bates
In-Reply-To: <1476826937-20665-1-git-send-email-sbates@raithlin.com>
From: Logan Gunthorpe <logang@deltatee.com>
We build on recent work that adds memory regions owned by a device
driver (ZONE_DEVICE) [1] and to add struct page support for these new
regions of memory [2].
1. Add an extra flags argument into dev_memremap_pages to take in a
MEMREMAP_XX argument. We update the existing calls to this function to
reflect the change.
2. For completeness, we add MEMREMAP_WT support to the memremap;
however we have no actual need for this functionality.
3. We add the static functions, add_zone_device_pages and
remove_zone_device pages. These are similar to arch_add_memory except
they don't create the memory mapping. We don't believe these need to be
made arch specific, but are open to other opinions.
4. dev_memremap_pages and devm_memremap_pages_release are updated to
treat IO memory slightly differently. For IO memory we use a combination
of the appropriate io_remap function and the zone_device pages functions
created above. A flags variable and kaddr pointer are added to struct
page_mem to facilitate this for the release function. We also set up
the page attribute tables for the mapped region correctly based on the
desired mapping.
[1] https://lists.01.org/pipermail/linux-nvdimm/2015-August/001810.html
[2] https://lists.01.org/pipermail/linux-nvdimm/2015-October/002387.html
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
drivers/dax/pmem.c | 4 +-
drivers/nvdimm/pmem.c | 4 +-
include/linux/memremap.h | 5 ++-
kernel/memremap.c | 80 +++++++++++++++++++++++++++++++++++++--
tools/testing/nvdimm/test/iomap.c | 3 +-
5 files changed, 86 insertions(+), 10 deletions(-)
diff --git a/drivers/dax/pmem.c b/drivers/dax/pmem.c
index 9630d88..58ac456 100644
--- a/drivers/dax/pmem.c
+++ b/drivers/dax/pmem.c
@@ -14,6 +14,7 @@
#include <linux/memremap.h>
#include <linux/module.h>
#include <linux/pfn_t.h>
+#include <linux/pmem.h>
#include "../nvdimm/pfn.h"
#include "../nvdimm/nd.h"
#include "dax.h"
@@ -108,7 +109,8 @@ static int dax_pmem_probe(struct device *dev)
if (rc)
return rc;
- addr = devm_memremap_pages(dev, &res, &dax_pmem->ref, altmap);
+ addr = devm_memremap_pages(dev, &res, &dax_pmem->ref, altmap,
+ ARCH_MEMREMAP_PMEM);
if (IS_ERR(addr))
return PTR_ERR(addr);
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 42b3a82..97032a1 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -278,7 +278,7 @@ static int pmem_attach_disk(struct device *dev,
pmem->pfn_flags = PFN_DEV;
if (is_nd_pfn(dev)) {
addr = devm_memremap_pages(dev, &pfn_res, &q->q_usage_counter,
- altmap);
+ altmap, ARCH_MEMREMAP_PMEM);
pfn_sb = nd_pfn->pfn_sb;
pmem->data_offset = le64_to_cpu(pfn_sb->dataoff);
pmem->pfn_pad = resource_size(res) - resource_size(&pfn_res);
@@ -287,7 +287,7 @@ static int pmem_attach_disk(struct device *dev,
res->start += pmem->data_offset;
} else if (pmem_should_map_pages(dev)) {
addr = devm_memremap_pages(dev, &nsio->res,
- &q->q_usage_counter, NULL);
+ &q->q_usage_counter, NULL, ARCH_MEMREMAP_PMEM);
pmem->pfn_flags |= PFN_MAP;
} else
addr = devm_memremap(dev, pmem->phys_addr,
diff --git a/include/linux/memremap.h b/include/linux/memremap.h
index 9341619..fc99283 100644
--- a/include/linux/memremap.h
+++ b/include/linux/memremap.h
@@ -51,12 +51,13 @@ struct dev_pagemap {
#ifdef CONFIG_ZONE_DEVICE
void *devm_memremap_pages(struct device *dev, struct resource *res,
- struct percpu_ref *ref, struct vmem_altmap *altmap);
+ struct percpu_ref *ref, struct vmem_altmap *altmap,
+ unsigned long flags);
struct dev_pagemap *find_dev_pagemap(resource_size_t phys);
#else
static inline void *devm_memremap_pages(struct device *dev,
struct resource *res, struct percpu_ref *ref,
- struct vmem_altmap *altmap)
+ struct vmem_altmap *altmap, unsigned long flags)
{
/*
* Fail attempts to call devm_memremap_pages() without
diff --git a/kernel/memremap.c b/kernel/memremap.c
index b501e39..d5f462c 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -175,13 +175,41 @@ static RADIX_TREE(pgmap_radix, GFP_KERNEL);
#define SECTION_MASK ~((1UL << PA_SECTION_SHIFT) - 1)
#define SECTION_SIZE (1UL << PA_SECTION_SHIFT)
+enum {
+ PAGEMAP_IO_MEM = 1 << 0,
+};
+
struct page_map {
struct resource res;
struct percpu_ref *ref;
struct dev_pagemap pgmap;
struct vmem_altmap altmap;
+ void *kaddr;
+ int flags;
};
+static int add_zone_device_pages(int nid, u64 start, u64 size)
+{
+ struct pglist_data *pgdat = NODE_DATA(nid);
+ struct zone *zone = pgdat->node_zones + ZONE_DEVICE;
+ unsigned long start_pfn = start >> PAGE_SHIFT;
+ unsigned long nr_pages = size >> PAGE_SHIFT;
+
+ return __add_pages(nid, zone, start_pfn, nr_pages);
+}
+
+static void remove_zone_device_pages(u64 start, u64 size)
+{
+ unsigned long start_pfn = start >> PAGE_SHIFT;
+ unsigned long nr_pages = size >> PAGE_SHIFT;
+ struct zone *zone;
+ int ret;
+
+ zone = page_zone(pfn_to_page(start_pfn));
+ ret = __remove_pages(zone, start_pfn, nr_pages);
+ WARN_ON_ONCE(ret);
+}
+
void get_zone_device_page(struct page *page)
{
percpu_ref_get(page->pgmap->ref);
@@ -246,9 +274,17 @@ static void devm_memremap_pages_release(struct device *dev, void *data)
/* pages are dead and unused, undo the arch mapping */
align_start = res->start & ~(SECTION_SIZE - 1);
align_size = ALIGN(resource_size(res), SECTION_SIZE);
- arch_remove_memory(align_start, align_size);
+
+ if (page_map->flags & PAGEMAP_IO_MEM) {
+ remove_zone_device_pages(align_start, align_size);
+ iounmap(page_map->kaddr);
+ } else {
+ arch_remove_memory(align_start, align_size);
+ }
+
untrack_pfn(NULL, PHYS_PFN(align_start), align_size);
pgmap_radix_release(res);
+
dev_WARN_ONCE(dev, pgmap->altmap && pgmap->altmap->alloc,
"%s: failed to free all reserved pages\n", __func__);
}
@@ -270,6 +306,8 @@ struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
* @res: "host memory" address range
* @ref: a live per-cpu reference count
* @altmap: optional descriptor for allocating the memmap from @res
+ * @flags: either MEMREMAP_WB, MEMREMAP_WT and MEMREMAP_WC
+ * see memremap() for a description of the flags
*
* Notes:
* 1/ @ref must be 'live' on entry and 'dead' before devm_memunmap_pages() time
@@ -280,7 +318,8 @@ struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
* this is not enforced.
*/
void *devm_memremap_pages(struct device *dev, struct resource *res,
- struct percpu_ref *ref, struct vmem_altmap *altmap)
+ struct percpu_ref *ref, struct vmem_altmap *altmap,
+ unsigned long flags)
{
resource_size_t key, align_start, align_size, align_end;
pgprot_t pgprot = PAGE_KERNEL;
@@ -288,6 +327,8 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
struct page_map *page_map;
int error, nid, is_ram;
unsigned long pfn;
+ void *addr = NULL;
+ enum page_cache_mode pcm;
align_start = res->start & ~(SECTION_SIZE - 1);
align_size = ALIGN(res->start + resource_size(res), SECTION_SIZE)
@@ -353,15 +394,44 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
if (nid < 0)
nid = numa_mem_id();
+ if (flags & MEMREMAP_WB)
+ pcm = _PAGE_CACHE_MODE_WB;
+ else if (flags & MEMREMAP_WT)
+ pcm = _PAGE_CACHE_MODE_WT;
+ else if (flags & MEMREMAP_WC)
+ pcm = _PAGE_CACHE_MODE_WC;
+ else
+ pcm = _PAGE_CACHE_MODE_WB;
+
+ pgprot = __pgprot(pgprot_val(pgprot) | cachemode2protval(pcm));
+
error = track_pfn_remap(NULL, &pgprot, PHYS_PFN(align_start), 0,
align_size);
if (error)
goto err_pfn_remap;
- error = arch_add_memory(nid, align_start, align_size, true);
+ if (flags & MEMREMAP_WB || !flags) {
+ error = arch_add_memory(nid, align_start, align_size, true);
+ addr = __va(res->start);
+ } else {
+ page_map->flags |= PAGEMAP_IO_MEM;
+ error = add_zone_device_pages(nid, align_start, align_size);
+ }
+
if (error)
goto err_add_memory;
+ if (!addr && (flags & MEMREMAP_WT))
+ addr = ioremap_wt(res->start, resource_size(res));
+
+ if (!addr && (flags & MEMREMAP_WC))
+ addr = ioremap_wc(res->start, resource_size(res));
+
+ if (!addr && page_map->flags & PAGEMAP_IO_MEM) {
+ remove_zone_device_pages(res->start, resource_size(res));
+ goto err_add_memory;
+ }
+
for_each_device_pfn(pfn, page_map) {
struct page *page = pfn_to_page(pfn);
@@ -374,8 +444,10 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
list_del(&page->lru);
page->pgmap = pgmap;
}
+
+ page_map->kaddr = addr;
devres_add(dev, page_map);
- return __va(res->start);
+ return addr;
err_add_memory:
untrack_pfn(NULL, PHYS_PFN(align_start), align_size);
diff --git a/tools/testing/nvdimm/test/iomap.c b/tools/testing/nvdimm/test/iomap.c
index 3ccef73..b82fecb 100644
--- a/tools/testing/nvdimm/test/iomap.c
+++ b/tools/testing/nvdimm/test/iomap.c
@@ -17,6 +17,7 @@
#include <linux/module.h>
#include <linux/types.h>
#include <linux/pfn_t.h>
+#include <linux/pmem.h>
#include <linux/acpi.h>
#include <linux/io.h>
#include <linux/mm.h>
@@ -109,7 +110,7 @@ void *__wrap_devm_memremap_pages(struct device *dev, struct resource *res,
if (nfit_res)
return nfit_res->buf + offset - nfit_res->res.start;
- return devm_memremap_pages(dev, res, ref, altmap);
+ return devm_memremap_pages(dev, res, ref, altmap, ARCH_MEMREMAP_PMEM);
}
EXPORT_SYMBOL(__wrap_devm_memremap_pages);
--
2.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH 2/3] iopmem : Add a block device driver for PCIe attached IO memory.
From: Stephen Bates @ 2016-10-18 21:42 UTC (permalink / raw)
To: linux-kernel, linux-nvdimm, linux-rdma, linux-block, linux-mm
Cc: dan.j.williams, ross.zwisler, willy, jgunthorpe, haggaie, hch,
axboe, corbet, jim.macdonald, sbates, logang, Stephen Bates
In-Reply-To: <1476826937-20665-1-git-send-email-sbates@raithlin.com>
Add a new block device driver that binds to PCIe devices and turns
PCIe BARs into DAX capable block devices.
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
MAINTAINERS | 7 ++
drivers/block/Kconfig | 27 ++++
drivers/block/Makefile | 1 +
drivers/block/iopmem.c | 333 +++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 368 insertions(+)
create mode 100644 drivers/block/iopmem.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 1cd38a7..c379f9d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6510,6 +6510,13 @@ S: Maintained
F: Documentation/devicetree/bindings/iommu/
F: drivers/iommu/
+IOPMEM BLOCK DEVICE DRVIER
+M: Stephen Bates <sbates@raithlin.com>
+L: linux-block@vger.kernel.org
+S: Maintained
+F: drivers/block/iopmem.c
+F: Documentation/blockdev/iopmem.txt
+
IP MASQUERADING
M: Juanjo Ciarlante <jjciarla@raiz.uncu.edu.ar>
S: Maintained
diff --git a/drivers/block/Kconfig b/drivers/block/Kconfig
index 39dd30b..13ae1e7 100644
--- a/drivers/block/Kconfig
+++ b/drivers/block/Kconfig
@@ -537,4 +537,31 @@ config BLK_DEV_RSXX
To compile this driver as a module, choose M here: the
module will be called rsxx.
+config BLK_DEV_IOPMEM
+ tristate "Persistent block device backed by PCIe Memory"
+ depends on ZONE_DEVICE
+ default n
+ help
+ Say Y here if you want to include a generic device driver
+ that can create a block device from persistent PCIe attached
+ IO memory.
+
+ To compile this driver as a module, choose M here: The
+ module will be called iopmem. A block device will be created
+ for each PCIe attached device that matches the vendor and
+ device ID as specified in the module. Alternativel this
+ driver can be bound to any aribtary PCIe function using the
+ sysfs bind entry.
+
+ This block device supports direct access (DAX) file systems
+ and supports struct page backing for the IO Memory. This
+ makes the underlying memory suitable for things like RDMA
+ Memory Regions and Direct IO which is useful for PCIe
+ peer-to-peer DMA operations.
+
+ Note that persistent is only assured if the memory on the
+ PCIe card has some form of power loss protection. This could
+ be provided via some form of battery, a supercap/NAND combo
+ or some exciting new persistent memory technology.
+
endif # BLK_DEV
diff --git a/drivers/block/Makefile b/drivers/block/Makefile
index 1e9661e..1f4f69b 100644
--- a/drivers/block/Makefile
+++ b/drivers/block/Makefile
@@ -41,6 +41,7 @@ obj-$(CONFIG_BLK_DEV_PCIESSD_MTIP32XX) += mtip32xx/
obj-$(CONFIG_BLK_DEV_RSXX) += rsxx/
obj-$(CONFIG_BLK_DEV_NULL_BLK) += null_blk.o
obj-$(CONFIG_ZRAM) += zram/
+obj-$(CONFIG_BLK_DEV_IOPMEM) += iopmem.o
skd-y := skd_main.o
swim_mod-y := swim.o swim_asm.o
diff --git a/drivers/block/iopmem.c b/drivers/block/iopmem.c
new file mode 100644
index 0000000..4a1e693
--- /dev/null
+++ b/drivers/block/iopmem.c
@@ -0,0 +1,333 @@
+/*
+ * IOPMEM Block Device Driver
+ * Copyright (c) 2016, Microsemi Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * This driver is heavily based on drivers/block/pmem.c.
+ * Copyright (c) 2014, Intel Corporation.
+ * Copyright (C) 2007 Nick Piggin
+ * Copyright (C) 2007 Novell Inc.
+ */
+
+#include <linux/blkdev.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/pfn_t.h>
+#include <linux/memremap.h>
+
+static const int BAR_ID = 4;
+
+static struct pci_device_id iopmem_id_table[] = {
+ { PCI_DEVICE(0x11f8, 0xf115) },
+ { 0, }
+};
+MODULE_DEVICE_TABLE(pci, iopmem_id_table);
+
+struct iopmem_device {
+ struct request_queue *queue;
+ struct gendisk *disk;
+ struct device *dev;
+
+ int instance;
+
+ /* One contiguous memory region per device */
+ phys_addr_t phys_addr;
+ void *virt_addr;
+ size_t size;
+};
+
+ /*
+ * We can only access the iopmem device with full 32-bit word
+ * accesses which cannot be gaurantee'd by the regular memcpy
+ */
+
+static void memcpy_from_iopmem(void *dst, const void *src, size_t sz)
+{
+ u64 *wdst = dst;
+ const u64 *wsrc = src;
+ u64 tmp;
+
+ while (sz >= sizeof(*wdst)) {
+ *wdst++ = *wsrc++;
+ sz -= sizeof(*wdst);
+ }
+
+ if (!sz)
+ return;
+
+ tmp = *wsrc;
+ memcpy(wdst, &tmp, sz);
+}
+
+static void write_iopmem(void *iopmem_addr, struct page *page,
+ unsigned int off, unsigned int len)
+{
+ void *mem = kmap_atomic(page);
+
+ memcpy(iopmem_addr, mem + off, len);
+ kunmap_atomic(mem);
+}
+
+static void read_iopmem(struct page *page, unsigned int off,
+ void *iopmem_addr, unsigned int len)
+{
+ void *mem = kmap_atomic(page);
+
+ memcpy_from_iopmem(mem + off, iopmem_addr, len);
+ kunmap_atomic(mem);
+}
+
+static void iopmem_do_bvec(struct iopmem_device *iopmem, struct page *page,
+ unsigned int len, unsigned int off, bool is_write,
+ sector_t sector)
+{
+ phys_addr_t iopmem_off = sector * 512;
+ void *iopmem_addr = iopmem->virt_addr + iopmem_off;
+
+ if (!is_write) {
+ read_iopmem(page, off, iopmem_addr, len);
+ flush_dcache_page(page);
+ } else {
+ flush_dcache_page(page);
+ write_iopmem(iopmem_addr, page, off, len);
+ }
+}
+
+static blk_qc_t iopmem_make_request(struct request_queue *q, struct bio *bio)
+{
+ struct iopmem_device *iopmem = q->queuedata;
+ struct bio_vec bvec;
+ struct bvec_iter iter;
+
+ bio_for_each_segment(bvec, bio, iter) {
+ iopmem_do_bvec(iopmem, bvec.bv_page, bvec.bv_len,
+ bvec.bv_offset, op_is_write(bio_op(bio)),
+ iter.bi_sector);
+ }
+
+ bio_endio(bio);
+ return BLK_QC_T_NONE;
+}
+
+static int iopmem_rw_page(struct block_device *bdev, sector_t sector,
+ struct page *page, bool is_write)
+{
+ struct iopmem_device *iopmem = bdev->bd_queue->queuedata;
+
+ iopmem_do_bvec(iopmem, page, PAGE_SIZE, 0, is_write, sector);
+ page_endio(page, is_write, 0);
+ return 0;
+}
+
+static long iopmem_direct_access(struct block_device *bdev, sector_t sector,
+ void **kaddr, pfn_t *pfn, long size)
+{
+ struct iopmem_device *iopmem = bdev->bd_queue->queuedata;
+ resource_size_t offset = sector * 512;
+
+ if (!iopmem)
+ return -ENODEV;
+
+ *kaddr = iopmem->virt_addr + offset;
+ *pfn = phys_to_pfn_t(iopmem->phys_addr + offset, PFN_DEV | PFN_MAP);
+
+ return iopmem->size - offset;
+}
+
+static const struct block_device_operations iopmem_fops = {
+ .owner = THIS_MODULE,
+ .rw_page = iopmem_rw_page,
+ .direct_access = iopmem_direct_access,
+};
+
+static DEFINE_IDA(iopmem_instance_ida);
+static DEFINE_SPINLOCK(ida_lock);
+
+static int iopmem_set_instance(struct iopmem_device *iopmem)
+{
+ int instance, error;
+
+ do {
+ if (!ida_pre_get(&iopmem_instance_ida, GFP_KERNEL))
+ return -ENODEV;
+
+ spin_lock(&ida_lock);
+ error = ida_get_new(&iopmem_instance_ida, &instance);
+ spin_unlock(&ida_lock);
+
+ } while (error == -EAGAIN);
+
+ if (error)
+ return -ENODEV;
+
+ iopmem->instance = instance;
+ return 0;
+}
+
+static void iopmem_release_instance(struct iopmem_device *iopmem)
+{
+ spin_lock(&ida_lock);
+ ida_remove(&iopmem_instance_ida, iopmem->instance);
+ spin_unlock(&ida_lock);
+}
+
+static int iopmem_attach_disk(struct iopmem_device *iopmem)
+{
+ struct gendisk *disk;
+ int nid = dev_to_node(iopmem->dev);
+ struct request_queue *q = iopmem->queue;
+
+ blk_queue_write_cache(q, true, true);
+ blk_queue_make_request(q, iopmem_make_request);
+ blk_queue_physical_block_size(q, PAGE_SIZE);
+ blk_queue_max_hw_sectors(q, UINT_MAX);
+ blk_queue_bounce_limit(q, BLK_BOUNCE_ANY);
+ queue_flag_set_unlocked(QUEUE_FLAG_NONROT, q);
+ queue_flag_set_unlocked(QUEUE_FLAG_DAX, q);
+ q->queuedata = iopmem;
+
+ disk = alloc_disk_node(0, nid);
+ if (unlikely(!disk))
+ return -ENOMEM;
+
+ disk->fops = &iopmem_fops;
+ disk->queue = q;
+ disk->flags = GENHD_FL_EXT_DEVT;
+ sprintf(disk->disk_name, "iopmem%d", iopmem->instance);
+ set_capacity(disk, iopmem->size / 512);
+ iopmem->disk = disk;
+
+ device_add_disk(iopmem->dev, disk);
+ revalidate_disk(disk);
+
+ return 0;
+}
+
+static void iopmem_detach_disk(struct iopmem_device *iopmem)
+{
+ del_gendisk(iopmem->disk);
+ put_disk(iopmem->disk);
+}
+
+static int iopmem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+ struct iopmem_device *iopmem;
+ struct device *dev;
+ int err = 0;
+ int nid = dev_to_node(&pdev->dev);
+
+ if (pci_enable_device_mem(pdev) < 0) {
+ dev_err(&pdev->dev, "unable to enable device!\n");
+ goto out;
+ }
+
+ iopmem = kzalloc(sizeof(*iopmem), GFP_KERNEL);
+ if (unlikely(!iopmem)) {
+ err = -ENOMEM;
+ goto out_disable_device;
+ }
+
+ iopmem->phys_addr = pci_resource_start(pdev, BAR_ID);
+ iopmem->size = pci_resource_end(pdev, BAR_ID) - iopmem->phys_addr + 1;
+ iopmem->dev = dev = get_device(&pdev->dev);
+ pci_set_drvdata(pdev, iopmem);
+
+ err = iopmem_set_instance(iopmem);
+ if (err)
+ goto out_put_device;
+
+ dev_info(dev, "bar space 0x%llx len %lld\n",
+ (unsigned long long) iopmem->phys_addr,
+ (unsigned long long) iopmem->size);
+
+ if (!devm_request_mem_region(dev, iopmem->phys_addr,
+ iopmem->size, dev_name(dev))) {
+ dev_warn(dev, "could not reserve region [0x%pa:0x%zx]\n",
+ &iopmem->phys_addr, iopmem->size);
+ err = -EBUSY;
+ goto out_release_instance;
+ }
+
+ iopmem->queue = blk_alloc_queue_node(GFP_KERNEL, nid);
+ if (!iopmem->queue) {
+ err = -ENOMEM;
+ goto out_release_instance;
+ }
+
+ iopmem->virt_addr = devm_memremap_pages(dev, &pdev->resource[BAR_ID],
+ &iopmem->queue->q_usage_counter,
+ NULL, MEMREMAP_WC);
+ if (IS_ERR(iopmem->virt_addr)) {
+ err = -ENXIO;
+ goto out_free_queue;
+ }
+
+ err = iopmem_attach_disk(iopmem);
+ if (err)
+ goto out_free_queue;
+
+ return 0;
+
+out_free_queue:
+ blk_cleanup_queue(iopmem->queue);
+out_release_instance:
+ iopmem_release_instance(iopmem);
+out_put_device:
+ put_device(&pdev->dev);
+ kfree(iopmem);
+out_disable_device:
+ pci_disable_device(pdev);
+out:
+ return err;
+}
+
+static void iopmem_remove(struct pci_dev *pdev)
+{
+ struct iopmem_device *iopmem = pci_get_drvdata(pdev);
+
+ blk_set_queue_dying(iopmem->queue);
+ iopmem_detach_disk(iopmem);
+ blk_cleanup_queue(iopmem->queue);
+ iopmem_release_instance(iopmem);
+ put_device(iopmem->dev);
+ kfree(iopmem);
+ pci_disable_device(pdev);
+}
+
+static struct pci_driver iopmem_pci_driver = {
+ .name = "iopmem",
+ .id_table = iopmem_id_table,
+ .probe = iopmem_probe,
+ .remove = iopmem_remove,
+};
+
+static int __init iopmem_init(void)
+{
+ int rc;
+
+ rc = pci_register_driver(&iopmem_pci_driver);
+ if (rc)
+ return rc;
+
+ pr_info("iopmem: module loaded\n");
+ return 0;
+}
+
+static void __exit iopmem_exit(void)
+{
+ pci_unregister_driver(&iopmem_pci_driver);
+ pr_info("iopmem: module unloaded\n");
+}
+
+MODULE_AUTHOR("Logan Gunthorpe <logang@deltatee.com>");
+MODULE_LICENSE("GPL");
+module_init(iopmem_init);
+module_exit(iopmem_exit);
--
2.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH 3/3] iopmem : Add documentation for iopmem driver
From: Stephen Bates @ 2016-10-18 21:42 UTC (permalink / raw)
To: linux-kernel, linux-nvdimm, linux-rdma, linux-block, linux-mm
Cc: dan.j.williams, ross.zwisler, willy, jgunthorpe, haggaie, hch,
axboe, corbet, jim.macdonald, sbates, logang, Stephen Bates
In-Reply-To: <1476826937-20665-1-git-send-email-sbates@raithlin.com>
Add documentation for the iopmem PCIe device driver.
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
Documentation/blockdev/00-INDEX | 2 ++
Documentation/blockdev/iopmem.txt | 62 +++++++++++++++++++++++++++++++++++++++
2 files changed, 64 insertions(+)
create mode 100644 Documentation/blockdev/iopmem.txt
diff --git a/Documentation/blockdev/00-INDEX b/Documentation/blockdev/00-INDEX
index c08df56..913e500 100644
--- a/Documentation/blockdev/00-INDEX
+++ b/Documentation/blockdev/00-INDEX
@@ -8,6 +8,8 @@ cpqarray.txt
- info on using Compaq's SMART2 Intelligent Disk Array Controllers.
floppy.txt
- notes and driver options for the floppy disk driver.
+iopmem.txt
+ - info on the iopmem block driver.
mflash.txt
- info on mGine m(g)flash driver for linux.
nbd.txt
diff --git a/Documentation/blockdev/iopmem.txt b/Documentation/blockdev/iopmem.txt
new file mode 100644
index 0000000..ba805b8
--- /dev/null
+++ b/Documentation/blockdev/iopmem.txt
@@ -0,0 +1,62 @@
+IOPMEM Block Driver
+===================
+
+Logan Gunthorpe and Stephen Bates - October 2016
+
+Introduction
+------------
+
+The iopmem module creates a DAX capable block device from a BAR on a PCIe
+device. iopmem leverages heavily from the pmem driver although it utilizes IO
+memory rather than system memory as its backing store.
+
+Usage
+-----
+
+To include the iopmem module in your kernel please set CONFIG_BLK_DEV_IOPMEM
+to either y or m. A block device will be created for each PCIe attached device
+that matches the vendor and device ID as specified in the module. Currently an
+unallocated PMC PCIe ID is used as the default. Alternatively this driver can
+be bound to any aribtary PCIe function using the sysfs bind entry.
+
+The main purpose for an iopmem block device is expected to be for peer-2-peer
+PCIe transfers. We DO NOT RECCOMEND accessing a iopmem device using the local
+CPU unless you are doing one of the three following things:
+
+1. Creating a DAX capable filesystem on the iopmem device.
+2. Creating some files on the DAX capable filesystem.
+3. Interogating the files on said filesystem to obtain pointers that can be
+ passed to other PCIe devices for p2p DMA operations.
+
+Issues
+------
+
+1. Address Translation. Suggestions have been made that in certain
+architectures and topologies the dma_addr_t passed to the DMA master
+in a peer-2-peer transfer will not correctly route to the IO memory
+intended. However in our testing to date we have not seen this to be
+an issue, even in systems with IOMMUs and PCIe switches. It is our
+understanding that an IOMMU only maps system memory and would not
+interfere with device memory regions. (It certainly has no opportunity
+to do so if the transfer gets routed through a switch).
+
+2. Memory Segment Spacing. This patch has the same limitations that
+ZONE_DEVICE does in that memory regions must be spaces at least
+SECTION_SIZE bytes part. On x86 this is 128MB and there are cases where
+BARs can be placed closer together than this. Thus ZONE_DEVICE would not
+be usable on neighboring BARs. For our purposes, this is not an issue as
+we'd only be looking at enabling a single BAR in a given PCIe device.
+More exotic use cases may have problems with this.
+
+3. Coherency Issues. When IOMEM is written from both the CPU and a PCIe
+peer there is potential for coherency issues and for writes to occur out
+of order. This is something that users of this feature need to be
+cognizant of and may necessitate the use of CONFIG_EXPERT. Though really,
+this isn't much different than the existing situation with RDMA: if
+userspace sets up an MR for remote use, they need to be careful about
+using that memory region themselves.
+
+4. Architecture. Currently this patch is applicable only to x86
+architectures. The same is true for much of the code pertaining to
+PMEM and ZONE_DEVICE. It is hoped that the work will be extended to other
+ARCH over time.
--
2.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v3 0/11] Fix race conditions related to stopping block layer queues
From: Bart Van Assche @ 2016-10-18 21:48 UTC (permalink / raw)
To: Jens Axboe
Cc: Christoph Hellwig, James Bottomley, Martin K. Petersen,
Mike Snitzer, Doug Ledford, Keith Busch, Ming Lin,
Laurence Oberman, linux-block@vger.kernel.org,
linux-scsi@vger.kernel.org, linux-rdma@vger.kernel.org,
linux-nvme@lists.infradead.org
Hello Jens,
Multiple block drivers need the functionality to stop a request queue
and to wait until all ongoing request_fn() / queue_rq() calls have
finished without waiting until all outstanding requests have finished.
Hence this patch series that introduces the blk_mq_quiesce_queue() and
blk_mq_resume_queue() functions. The dm-mq, SRP and NVMe patches in this
patch series are three examples of where these functions are useful.
These patches have been tested on top of kernel v4.9-rc1. The following
tests have been run to verify this patch series:
- My own srp-test suite that stress-tests SRP on top of dm-multipath.
- Mike's mptest suite that stress-tests dm-multipath.
- fio on top of the NVMeOF host driver that was connected to the NVMeOF
target driver on the same host.
- Laurence verified the previous version (v2) of this patch series by
running it through the Red Hat SRP test suite. Laurence also ran some
NVMe tests (thanks Laurence!).
The changes compared to the second version of this patch series are:
- Changed the order of the patches in this patch series.
- Added several new patches: a patch that avoids that .queue_rq() gets
invoked from the direct submission path if a queue has been stopped
and also a patch that introduces the helper function
blk_mq_hctx_stopped().
- blk_mq_quiesce_queue() has been reworked (thanks to Ming Lin and Sagi
for their feedback).
- A bool 'kick' argument has been added to blk_mq_requeue_request().
- As proposed by Christoph, the code that waits for queuecommand() has
been moved from the SRP transport driver to the SCSI core.
Changes between v2 and v1:
- Dropped the non-blk-mq changes from this patch series.
- Added support for harware queues with BLK_MQ_F_BLOCKING set.
- Added a call stack to the description of the dm race fix patch.
- Dropped the non-scsi-mq changes from the SRP patch.
- Added a patch that introduces blk_mq_queue_stopped() in the dm driver.
The individual patches in this series are:
0001-blk-mq-Do-not-invoke-.queue_rq-for-a-stopped-queue.patch
0002-blk-mq-Introduce-blk_mq_hctx_stopped.patch
0003-blk-mq-Introduce-blk_mq_queue_stopped.patch
0004-blk-mq-Introduce-blk_mq_quiesce_queue.patch
0005-blk-mq-Add-a-kick_requeue_list-argument-to-blk_mq_re.patch
0006-dm-Use-BLK_MQ_S_STOPPED-instead-of-QUEUE_FLAG_STOPPE.patch
0007-dm-Fix-a-race-condition-related-to-stopping-and-star.patch
0008-SRP-transport-Move-queuecommand-wait-code-to-SCSI-co.patch
0009-SRP-transport-scsi-mq-Wait-for-.queue_rq-if-necessar.patch
0010-nvme-Use-BLK_MQ_S_STOPPED-instead-of-QUEUE_FLAG_STOP.patch
0011-nvme-Fix-a-race-condition.patch
Thanks,
Bart.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox