* Re: [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.
[not found] ` <1415193919-17361-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
@ 2014-11-05 15:10 ` Burakov, Anatoly
2014-11-06 14:11 ` [PATCH v2] " lxu
` (6 subsequent siblings)
7 siblings, 0 replies; 41+ messages in thread
From: Burakov, Anatoly @ 2014-11-05 15:10 UTC (permalink / raw)
To: lxu, dev-VfR2kkLFssw@public.gmane.org
I have a slight problems with this patch.
The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever reasons.
Also,
Thanks,
Anatoly
-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of lxu
Sent: Wednesday, November 5, 2014 1:25 PM
To: dev-VfR2kkLFssw@public.gmane.org
Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..bc7ed3a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct rte_pci_addr *loc = &dev->addr;
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ static void * requested_addr = NULL;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ requested_addr = (uint8_t *) internal_config.base_virtaddr
+ + internal_config.memory;
+ }
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = (uint8_t *)mapaddr + maps[j].size;
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread* [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.
[not found] ` <1415193919-17361-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-05 15:10 ` Burakov, Anatoly
@ 2014-11-06 14:11 ` lxu
[not found] ` <1415283104-29970-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-06 14:48 ` 答复：[PATCH " 徐亮
2014-11-06 14:47 ` [PATCH v3] " lxu
` (5 subsequent siblings)
7 siblings, 2 replies; 41+ messages in thread
From: lxu @ 2014-11-06 14:11 UTC (permalink / raw)
To: dev-VfR2kkLFssw
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a591da3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */
int
pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr = NULL;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = (void *)(last->addr_64 + last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = (uint8_t *)mapaddr + maps[j].size;
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread[parent not found: <1415283104-29970-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>]
* Re: [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.
[not found] ` <1415283104-29970-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
@ 2014-11-06 14:27 ` Burakov, Anatoly
0 siblings, 0 replies; 41+ messages in thread
From: Burakov, Anatoly @ 2014-11-06 14:27 UTC (permalink / raw)
To: lxu, dev-VfR2kkLFssw@public.gmane.org
Few nitpicks.
Static variables are always initialized to 0, so "= NULL" isn't necessary, a declaration will suffice. Also, we have a macro RTE_PTR_ADD to add numbers to pointers, I think it would be better to use those. Otherwise, looks fine to me.
I still feel uneasy about depending on nothing being mapped directly after hugepages (perhaps we could do mmap(bar_size) before trying pci_map_resource, and increment requested_addr until we find a free spot?), but I imagine this case would be quite rare, so probably it's not worth the added kludge.
Thanks,
Anatoly
-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org]
Sent: Thursday, November 6, 2014 2:12 PM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly
Subject: [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a591da3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */
int
pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr = NULL;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = (void *)(last->addr_64 + last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = (uint8_t *)mapaddr + maps[j].size;
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread
* 答复：[PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.
2014-11-06 14:11 ` [PATCH v2] " lxu
[not found] ` <1415283104-29970-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
@ 2014-11-06 14:48 ` 徐亮
1 sibling, 0 replies; 41+ messages in thread
From: 徐亮 @ 2014-11-06 14:48 UTC (permalink / raw)
To: Burakov, Anatoly, dev-VfR2kkLFssw@public.gmane.org
When user configure base_virtaddr, we should believe they can take care it.
In my case, I always check /proc/xxxx/maps to find a huge free address space, such as 0x20 0000 0000, to map all the hugepages and uio resource.
------------------------------------------------------------------发件人：Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>发送时间：2014年11月6日(星期四) 22:29收件人：徐亮 <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>，dev-VfR2kkLFssw@public.gmane.org <dev-VfR2kkLFssw@public.gmane.org>主 题：RE: [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.
Few nitpicks.
Static variables are always initialized to 0, so "= NULL" isn't necessary, a declaration will suffice. Also, we have a macro RTE_PTR_ADD to add numbers to pointers, I think it would be better to use those. Otherwise, looks fine to me.
I still feel uneasy about depending on nothing being mapped directly after hugepages (perhaps we could do mmap(bar_size) before trying pci_map_resource, and increment requested_addr until we find a free spot?), but I imagine this case would be quite rare, so probably it's not worth the added kludge.
Thanks,
Anatoly
-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org]
Sent: Thursday, November 6, 2014 2:12 PM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly
Subject: [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a591da3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */
int
pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr = NULL;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = (void *)(last->addr_64 + last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = (uint8_t *)mapaddr + maps[j].size;
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v3] eal: map uio resources after hugepages when the base_virtaddr is configured.
[not found] ` <1415193919-17361-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-05 15:10 ` Burakov, Anatoly
2014-11-06 14:11 ` [PATCH v2] " lxu
@ 2014-11-06 14:47 ` lxu
[not found] ` <1415285223-7662-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-06 15:07 ` [PATCH v4] " lxu
` (4 subsequent siblings)
7 siblings, 1 reply; 41+ messages in thread
From: lxu @ 2014-11-06 14:47 UTC (permalink / raw)
To: dev-VfR2kkLFssw
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..3a218d0 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */
int
pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = RTE_PTR_ADD(last->addr, last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = (uint8_t *)mapaddr + maps[j].size;
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread* [PATCH v4] eal: map uio resources after hugepages when the base_virtaddr is configured.
[not found] ` <1415193919-17361-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
` (2 preceding siblings ...)
2014-11-06 14:47 ` [PATCH v3] " lxu
@ 2014-11-06 15:07 ` lxu
[not found] ` <1415286460-14337-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-06 15:11 ` lxu
` (3 subsequent siblings)
7 siblings, 1 reply; 41+ messages in thread
From: lxu @ 2014-11-06 15:07 UTC (permalink / raw)
To: dev-VfR2kkLFssw
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */
int
pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = RTE_PTR_ADD(last->addr, last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread* [PATCH v4] eal: map uio resources after hugepages when the base_virtaddr is configured.
[not found] ` <1415193919-17361-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
` (3 preceding siblings ...)
2014-11-06 15:07 ` [PATCH v4] " lxu
@ 2014-11-06 15:11 ` lxu
2014-11-06 15:32 ` [PATCH v5] " lxu
` (2 subsequent siblings)
7 siblings, 0 replies; 41+ messages in thread
From: lxu @ 2014-11-06 15:11 UTC (permalink / raw)
To: dev-VfR2kkLFssw
Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */
int
pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = RTE_PTR_ADD(last->addr, last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread* [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.
[not found] ` <1415193919-17361-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
` (4 preceding siblings ...)
2014-11-06 15:11 ` lxu
@ 2014-11-06 15:32 ` lxu
[not found] ` <1415287928-14513-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-07 8:01 ` [PATCH v6] " lxu
2014-11-07 14:57 ` [PATCH v7] eal: map uio resources after hugepages lxu
7 siblings, 1 reply; 41+ messages in thread
From: lxu @ 2014-11-06 15:32 UTC (permalink / raw)
To: dev-VfR2kkLFssw
Sorry, I'm learning the right way to send a patch by git.
I have a multiple processes application. When start the secondary process, I got error message "EAL: pci_map_resource(): cannot mmap(11, 0x7ffff7fba000, 0x20000, 0x0): Bad file descriptor (0x7ffff7fb9000)".
The secondary process links a lot of additional shared libraries, so the address 0x7ffff7fba000 had already be used.
I had fixed similar hugepages mmap problems by base_virtaddr. So I believe the uio resource should be mapped into base_virtaddr at this situation.
This patch try to fix it.
Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */
int
pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = RTE_PTR_ADD(last->addr, last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread* [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
[not found] ` <1415193919-17361-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
` (5 preceding siblings ...)
2014-11-06 15:32 ` [PATCH v5] " lxu
@ 2014-11-07 8:01 ` lxu
[not found] ` <1415347284-15468-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-07 9:57 ` [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured XU Liang
2014-11-07 14:57 ` [PATCH v7] eal: map uio resources after hugepages lxu
7 siblings, 2 replies; 41+ messages in thread
From: lxu @ 2014-11-07 8:01 UTC (permalink / raw)
To: dev-VfR2kkLFssw
A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.
This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.
Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */
int
pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = RTE_PTR_ADD(last->addr, last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread[parent not found: <1415347284-15468-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>]
* Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
[not found] ` <1415347284-15468-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
@ 2014-11-07 9:42 ` Bruce Richardson
2014-11-07 9:47 ` Burakov, Anatoly
2014-11-10 11:34 ` [PATCH v7] eal: map PCI memory resources after hugepages Anatoly Burakov
2 siblings, 0 replies; 41+ messages in thread
From: Bruce Richardson @ 2014-11-07 9:42 UTC (permalink / raw)
To: lxu; +Cc: dev-VfR2kkLFssw
On Fri, Nov 07, 2014 at 04:01:24PM +0800, lxu wrote:
> A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.
>
> This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.
>
> Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
> ---
> lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
> 1 file changed, 28 insertions(+), 1 deletion(-)
>
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> index 7e62266..a2c9ab6 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> @@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
> return uio_num;
> }
>
> +static inline const struct rte_memseg *
> +get_physmem_last(void)
> +{
> + const struct rte_memseg * seg = rte_eal_get_physmem_layout();
> + const struct rte_memseg * last = seg;
> + unsigned i = 0;
> +
> + for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
> + if (seg->addr == NULL)
> + break;
> +
> + if(seg->addr > last->addr)
> + last = seg;
> +
> + }
> + return last;
> +}
> +
> /* map the PCI resource of a PCI device in virtual memory */
> int
> pci_uio_map_resource(struct rte_pci_device *dev)
> @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
> struct mapped_pci_resource *uio_res;
> struct pci_map *maps;
>
> + /* map uio resource into user required virtual address */
> + static void * requested_addr;
> + if (internal_config.base_virtaddr && NULL == requested_addr) {
> + const struct rte_memseg * last = get_physmem_last();
> + requested_addr = RTE_PTR_ADD(last->addr, last->len);
> + }
> +
Could you perhaps take anatoly's suggestion and modify this patch so the checking
for a space right after the memsegs does not just depend on base-virtaddr being
set. Unless it causes other problems, there is no reason this code should not
always be used.
/Bruce
^ permalink raw reply [flat|nested] 41+ messages in thread* Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
[not found] ` <1415347284-15468-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-07 9:42 ` Bruce Richardson
@ 2014-11-07 9:47 ` Burakov, Anatoly
2014-11-10 11:34 ` [PATCH v7] eal: map PCI memory resources after hugepages Anatoly Burakov
2 siblings, 0 replies; 41+ messages in thread
From: Burakov, Anatoly @ 2014-11-07 9:47 UTC (permalink / raw)
To: lxu, dev-VfR2kkLFssw@public.gmane.org
The commit message looks fine to me, but VFIO code needs to be adjusted the same way.
Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).
My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).
My apologies for not thinking about any of this during the first 6 iterations of the patch :(
Thanks,
Anatoly
-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org]
Sent: Friday, November 7, 2014 8:01 AM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly; thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo
Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.
This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.
Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */ int pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = RTE_PTR_ADD(last->addr, last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread* [PATCH v7] eal: map PCI memory resources after hugepages
[not found] ` <1415347284-15468-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-07 9:42 ` Bruce Richardson
2014-11-07 9:47 ` Burakov, Anatoly
@ 2014-11-10 11:34 ` Anatoly Burakov
2014-11-11 3:53 ` XU Liang
[not found] ` <1415619272-8281-1-git-send-email-anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
2 siblings, 2 replies; 41+ messages in thread
From: Anatoly Burakov @ 2014-11-10 11:34 UTC (permalink / raw)
To: dev-VfR2kkLFssw
Multi-process DPDK application must mmap hugepages and pci resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.
This patch makes EAL map PCI BARs right after the hugepages (instead of
location chosen by mmap) in virtual memory.
Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci.c | 19 +++++++++++++++++++
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 13 +++++++++++--
lib/librte_eal/linuxapp/eal/include/eal_pci_init.h | 6 ++++++
4 files changed, 44 insertions(+), 3 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..dae8739 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
return -1;
}
+void *
+pci_find_max_end_va(void)
+{
+ const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg *last = seg;
+ unsigned i = 0;
+
+ for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if (seg->addr > last->addr)
+ last = seg;
+
+ }
+ return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
/* map a particular resource from a file */
void *
pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..5090bf1 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -48,6 +48,8 @@
static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
+void *pci_map_addr = NULL;
+
#define OFF_MAX ((uint64_t)(off_t)-1)
static int
@@ -371,10 +373,15 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ if (pci_map_addr == NULL)
+ pci_map_addr = pci_find_max_end_va();
+
+ mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+
+ pci_map_addr = RTE_PTR_ADD(pci_map_addr, maps[j].size);
}
if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..fb6ee7a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -720,8 +720,17 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
if (i == msix_bar)
continue;
- bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
- reg.size);
+ if (internal_config.process_type == RTE_PROC_PRIMARY) {
+ if (pci_map_addr == NULL)
+ pci_map_addr = pci_find_max_end_va();
+
+ bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+ reg.size);
+ pci_map_addr = RTE_PTR_ADD(pci_map_addr, reg.size);
+ } else {
+ bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+ reg.size);
+ }
if (bar_addr == NULL) {
RTE_LOG(ERR, EAL, " %s mapping BAR%i failed: %s\n", pci_addr, i,
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
extern struct mapped_pci_res_list *pci_res_list;
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
void *pci_map_resource(void *requested_addr, int fd, off_t offset,
size_t size);
--
1.8.1.4
^ permalink raw reply related [flat|nested] 41+ messages in thread* Re: [PATCH v7] eal: map PCI memory resources after hugepages
2014-11-10 11:34 ` [PATCH v7] eal: map PCI memory resources after hugepages Anatoly Burakov
@ 2014-11-11 3:53 ` XU Liang
[not found] ` <1415619272-8281-1-git-send-email-anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
1 sibling, 0 replies; 41+ messages in thread
From: XU Liang @ 2014-11-11 3:53 UTC (permalink / raw)
To: Burakov, Anatoly, dev-VfR2kkLFssw@public.gmane.org
I had finished some tests. The patch works fine. My tests are included :* single process + uio + vfio * single process + uio + vfio + base-virtaddr * multiple processes + uio + vfio * multiple processes + uio + vfio + base-virtaddr My unlucky multiple process application still got error without base-virtaddr when initial hugepages. See the attchments: primary.txt and secondary.txt.With base-virtaddr the patch worked, both hugepages and pci resources were mapped into base-virtaddr, My application is happy. See the attchments: base-virtaddr_primary.txt and base-virtaddr_secondary.txt. ------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Time:2014 Nov 10 (Mon) 21 : 34To:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>, dev-VfR2kkLFssw@public.gmane.org <dev-VfR2kkLFssw@public.gmane.org>Subject:Re: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages
Nak, there are issues with the patch. There is another patch already, but I'll submit it whenever Liang verifies it works with his setup.
Thanks,
Anatoly
-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Anatoly Burakov
Sent: Monday, November 10, 2014 11:35 AM
To: dev@dpdk.org
Subject: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages
Multi-process DPDK application must mmap hugepages and pci resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.
This patch makes EAL map PCI BARs right after the hugepages (instead of
location chosen by mmap) in virtual memory.
Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci.c | 19 +++++++++++++++++++
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 13 +++++++++++--
lib/librte_eal/linuxapp/eal/include/eal_pci_init.h | 6 ++++++
4 files changed, 44 insertions(+), 3 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..dae8739 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
return -1;
}
+void *
+pci_find_max_end_va(void)
+{
+ const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg *last = seg;
+ unsigned i = 0;
+
+ for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if (seg->addr > last->addr)
+ last = seg;
+
+ }
+ return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
/* map a particular resource from a file */
void *
pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..5090bf1 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -48,6 +48,8 @@
static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
+void *pci_map_addr = NULL;
+
#define OFF_MAX ((uint64_t)(off_t)-1)
static int
@@ -371,10 +373,15 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ if (pci_map_addr == NULL)
+ pci_map_addr = pci_find_max_end_va();
+
+ mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+
+ pci_map_addr = RTE_PTR_ADD(pci_map_addr, maps[j].size);
}
if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..fb6ee7a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -720,8 +720,17 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
if (i == msix_bar)
continue;
- bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
- reg.size);
+ if (internal_config.process_type == RTE_PROC_PRIMARY) {
+ if (pci_map_addr == NULL)
+ pci_map_addr = pci_find_max_end_va();
+
+ bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+ reg.size);
+ pci_map_addr = RTE_PTR_ADD(pci_map_addr, reg.size);
+ } else {
+ bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+ reg.size);
+ }
if (bar_addr == NULL) {
RTE_LOG(ERR, EAL, " %s mapping BAR%i failed: %s\n", pci_addr, i,
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
extern struct mapped_pci_res_list *pci_res_list;
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
void *pci_map_resource(void *requested_addr, int fd, off_t offset,
size_t size);
--
1.8.1.4
^ permalink raw reply related [flat|nested] 41+ messages in thread[parent not found: <1415619272-8281-1-git-send-email-anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>]
* Re: [PATCH v7] eal: map PCI memory resources after hugepages
[not found] ` <1415619272-8281-1-git-send-email-anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
@ 2014-11-10 13:33 ` Burakov, Anatoly
2014-11-11 10:09 ` [PATCH v8] " Anatoly Burakov
1 sibling, 0 replies; 41+ messages in thread
From: Burakov, Anatoly @ 2014-11-10 13:33 UTC (permalink / raw)
To: Burakov, Anatoly, dev-VfR2kkLFssw@public.gmane.org
Nak, there are issues with the patch. There is another patch already, but I'll submit it whenever Liang verifies it works with his setup.
Thanks,
Anatoly
-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Anatoly Burakov
Sent: Monday, November 10, 2014 11:35 AM
To: dev-VfR2kkLFssw@public.gmane.org
Subject: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages
Multi-process DPDK application must mmap hugepages and pci resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.
This patch makes EAL map PCI BARs right after the hugepages (instead of
location chosen by mmap) in virtual memory.
Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci.c | 19 +++++++++++++++++++
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 13 +++++++++++--
lib/librte_eal/linuxapp/eal/include/eal_pci_init.h | 6 ++++++
4 files changed, 44 insertions(+), 3 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..dae8739 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
return -1;
}
+void *
+pci_find_max_end_va(void)
+{
+ const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg *last = seg;
+ unsigned i = 0;
+
+ for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if (seg->addr > last->addr)
+ last = seg;
+
+ }
+ return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
/* map a particular resource from a file */
void *
pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..5090bf1 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -48,6 +48,8 @@
static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
+void *pci_map_addr = NULL;
+
#define OFF_MAX ((uint64_t)(off_t)-1)
static int
@@ -371,10 +373,15 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ if (pci_map_addr == NULL)
+ pci_map_addr = pci_find_max_end_va();
+
+ mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+
+ pci_map_addr = RTE_PTR_ADD(pci_map_addr, maps[j].size);
}
if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..fb6ee7a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -720,8 +720,17 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
if (i == msix_bar)
continue;
- bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
- reg.size);
+ if (internal_config.process_type == RTE_PROC_PRIMARY) {
+ if (pci_map_addr == NULL)
+ pci_map_addr = pci_find_max_end_va();
+
+ bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+ reg.size);
+ pci_map_addr = RTE_PTR_ADD(pci_map_addr, reg.size);
+ } else {
+ bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+ reg.size);
+ }
if (bar_addr == NULL) {
RTE_LOG(ERR, EAL, " %s mapping BAR%i failed: %s\n", pci_addr, i,
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
extern struct mapped_pci_res_list *pci_res_list;
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
void *pci_map_resource(void *requested_addr, int fd, off_t offset,
size_t size);
--
1.8.1.4
^ permalink raw reply related [flat|nested] 41+ messages in thread* [PATCH v8] eal: map PCI memory resources after hugepages
[not found] ` <1415619272-8281-1-git-send-email-anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
2014-11-10 13:33 ` Burakov, Anatoly
@ 2014-11-11 10:09 ` Anatoly Burakov
[not found] ` <1415700565-19157-1-git-send-email-anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
1 sibling, 1 reply; 41+ messages in thread
From: Anatoly Burakov @ 2014-11-11 10:09 UTC (permalink / raw)
To: dev-VfR2kkLFssw
Multi-process DPDK application must mmap hugepages and PCI resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.
This patch makes EAL try and map PCI BARs right after the hugepages
(instead of location chosen by mmap) in virtual memory, so that PCI BARs
have less chance of ending up in random places in virtual memory.
Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci.c | 30 ++++++++++++++++------
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 13 ++++++++--
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 19 +++++++++++---
lib/librte_eal/linuxapp/eal/include/eal_pci_init.h | 6 +++++
4 files changed, 55 insertions(+), 13 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..79fbbb8 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
return -1;
}
+void *
+pci_find_max_end_va(void)
+{
+ const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg *last = seg;
+ unsigned i = 0;
+
+ for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if (seg->addr > last->addr)
+ last = seg;
+
+ }
+ return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
/* map a particular resource from a file */
void *
pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
@@ -106,21 +125,16 @@ pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
/* Map the PCI memory resource of device */
mapaddr = mmap(requested_addr, size, PROT_READ | PROT_WRITE,
MAP_SHARED, fd, offset);
- if (mapaddr == MAP_FAILED ||
- (requested_addr != NULL && mapaddr != requested_addr)) {
+ if (mapaddr == MAP_FAILED) {
RTE_LOG(ERR, EAL, "%s(): cannot mmap(%d, %p, 0x%lx, 0x%lx): %s (%p)\n",
__func__, fd, requested_addr,
(unsigned long)size, (unsigned long)offset,
strerror(errno), mapaddr);
- goto fail;
+ } else {
+ RTE_LOG(DEBUG, EAL, " PCI memory mapped at %p\n", mapaddr);
}
- RTE_LOG(DEBUG, EAL, " PCI memory mapped at %p\n", mapaddr);
-
return mapaddr;
-
-fail:
- return NULL;
}
/* parse the "resource" sysfs file */
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e53f06b 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -35,6 +35,7 @@
#include <fcntl.h>
#include <dirent.h>
#include <sys/stat.h>
+#include <sys/mman.h>
#include <rte_log.h>
#include <rte_pci.h>
@@ -48,6 +49,8 @@
static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
+void *pci_map_addr = NULL;
+
#define OFF_MAX ((uint64_t)(off_t)-1)
static int
@@ -371,10 +374,16 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ /* try mapping somewhere close to the end of hugepages */
+ if (pci_map_addr == NULL)
+ pci_map_addr = pci_find_max_end_va();
+
+ mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
(size_t)maps[j].size);
- if (mapaddr == NULL)
+ if (mapaddr == MAP_FAILED)
fail = 1;
+
+ pci_map_addr = RTE_PTR_ADD(mapaddr, (size_t) maps[j].size);
}
if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..c1246e8 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -37,6 +37,7 @@
#include <sys/eventfd.h>
#include <sys/socket.h>
#include <sys/ioctl.h>
+#include <sys/mman.h>
#include <rte_log.h>
#include <rte_pci.h>
@@ -720,10 +721,22 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
if (i == msix_bar)
continue;
- bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
- reg.size);
+ if (internal_config.process_type == RTE_PROC_PRIMARY) {
+ /* try mapping somewhere close to the end of hugepages */
+ if (pci_map_addr == NULL)
+ pci_map_addr = pci_find_max_end_va();
+
+ bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+ reg.size);
+ pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg.size);
+ } else {
+ bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+ reg.size);
+ }
- if (bar_addr == NULL) {
+ if (bar_addr == MAP_FAILED ||
+ (internal_config.process_type == RTE_PROC_SECONDARY &&
+ bar_addr != maps[i].addr)) {
RTE_LOG(ERR, EAL, " %s mapping BAR%i failed: %s\n", pci_addr, i,
strerror(errno));
close(vfio_dev_fd);
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
extern struct mapped_pci_res_list *pci_res_list;
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
void *pci_map_resource(void *requested_addr, int fd, off_t offset,
size_t size);
--
1.8.1.4
^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
2014-11-07 8:01 ` [PATCH v6] " lxu
[not found] ` <1415347284-15468-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
@ 2014-11-07 9:57 ` XU Liang
[not found] ` <2c2ad9a4-1c04-4641-9667-249617ae2c56-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-07 14:37 ` XU Liang
1 sibling, 2 replies; 41+ messages in thread
From: XU Liang @ 2014-11-07 9:57 UTC (permalink / raw)
To: Burakov, Anatoly, dev-VfR2kkLFssw@public.gmane.org
How to find the maximum end virtual address ? I'm not the DPDK expert, but I will try to do my best.
If the segments isn't overlap, "if(seg->addr > last->addr) last = seg;" already find the segment with maximum end virtual address.
------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Time:2014 Nov 7 (Fri) 17 : 47To:徐亮 <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>, dev-VfR2kkLFssw@public.gmane.org <dev@dpdk.org>Cc:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org <thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>, De Lara Guarch, Pablo <pablo.de.lara.guarch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
The commit message looks fine to me, but VFIO code needs to be adjusted the same way.
Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).
My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).
My apologies for not thinking about any of this during the first 6 iterations of the patch :(
Thanks,
Anatoly
-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org]
Sent: Friday, November 7, 2014 8:01 AM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly; thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo
Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.
This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.
Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */ int pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = RTE_PTR_ADD(last->addr, last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread[parent not found: <2c2ad9a4-1c04-4641-9667-249617ae2c56-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>]
* Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
[not found] ` <2c2ad9a4-1c04-4641-9667-249617ae2c56-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
@ 2014-11-07 10:00 ` Burakov, Anatoly
0 siblings, 0 replies; 41+ messages in thread
From: Burakov, Anatoly @ 2014-11-07 10:00 UTC (permalink / raw)
To: XU Liang, dev-VfR2kkLFssw@public.gmane.org
Ah, yes, sorry about that, haven’t quite woke up yet ☹ You’re right. So it’s removing the dependency on --base-virtaddr, moving the function to eal_pci.c and adding similar VFIO code that’s left then.
Thanks,
Anatoly
From: XU Liang [mailto:liang.xu@cinfotech.cn]
Sent: Friday, November 7, 2014 9:57 AM
To: Burakov, Anatoly; dev@dpdk.org
Cc: thomas.monjalon@6wind.com; De Lara Guarch, Pablo
Subject: Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
How to find the maximum end virtual address ? I'm not the DPDK expert, but I will try to do my best.
If the segments isn't overlap, "if(seg->addr > last->addr) last = seg;" already find the segment with maximum end virtual address.
------------------------------------------------------------------
From:Burakov, Anatoly <anatoly.burakov@intel.com<mailto:anatoly.burakov@intel.com>>
Time:2014 Nov 7 (Fri) 17 : 47
To:徐亮 <liang.xu@cinfotech.cn<mailto:liang.xu@cinfotech.cn>>, dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
Cc:thomas.monjalon@6wind.com <thomas.monjalon@6wind.com<mailto:thomas.monjalon@6wind.com>>, De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com<mailto:pablo.de.lara.guarch@intel.com>>
Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
The commit message looks fine to me, but VFIO code needs to be adjusted the same way.
Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).
My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).
My apologies for not thinking about any of this during the first 6 iterations of the patch :(
Thanks,
Anatoly
-----Original Message-----
From: lxu [mailto:liang.xu@cinfotech.cn]
Sent: Friday, November 7, 2014 8:01 AM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Burakov, Anatoly; thomas.monjalon@6wind.com<mailto:thomas.monjalon@6wind.com>; De Lara Guarch, Pablo
Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.
This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.
Signed-off-by: lxu <liang.xu@cinfotech.cn<mailto:liang.xu@cinfotech.cn>>
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */ int pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = RTE_PTR_ADD(last->addr, last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
2014-11-07 9:57 ` [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured XU Liang
[not found] ` <2c2ad9a4-1c04-4641-9667-249617ae2c56-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
@ 2014-11-07 14:37 ` XU Liang
1 sibling, 0 replies; 41+ messages in thread
From: XU Liang @ 2014-11-07 14:37 UTC (permalink / raw)
To: Burakov, Anatoly, dev-VfR2kkLFssw@public.gmane.org
Because vfio and uio can be used at the same time, we need save the value of requested_addr at pci_vfio_map_resource() and pci_uio_map_resource().Your guys like add a global variable "void * pci_requested_addr" or a "void** requested_addr" parameter at pci_vfio_map_resource() and pci_uio_map_resource() ?------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Time:2014 Nov 7 (Fri) 18 : 03To:徐亮 <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>, dev-VfR2kkLFssw@public.gmane.org <dev@dpdk.org>Cc:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org <thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>, De Lara Guarch, Pablo <pablo.de.lara.guarch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
font-family: Wingdings;panose-1: 5 0 0 0 0 0 0 0 0 0;font-family: MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Cambria Math;panose-1: 2 4 5 3 5 4 6 3 2 4;font-family: Calibri;panose-1: 2 15 5 2 2 2 4 3 2 4;font-family: Tahoma;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;p.MsoNormal, li.MsoNormal, div.MsoNormal {margin: 0.0in;margin-bottom: 1.0E-4pt;font-size: 12.0pt;font-family: Times New Roman , serif;}
a:link, span.MsoHyperlink {mso-style-priority: 99;color: #0563c1;text-decoration: underline;}
a:visited, span.MsoHyperlinkFollowed {mso-style-priority: 99;color: #954f72;text-decoration: underline;}
span.EmailStyle17 {mso-style-type: personal-reply;font-family: Calibri , sans-serif;color: #1f497d;}
*.MsoChpDefault {mso-style-type: export-only;font-family: Calibri , sans-serif;}
size: 8.5in 11.0in;margin: 1.0in 1.0in 1.0in 1.0in;div.WordSection1 {page: WordSection1;}
Ah, yes, sorry about that, haven’t quite woke up yet
L You’re right. So it’s removing the dependency on --base-virtaddr, moving the function to eal_pci.c
and adding similar VFIO code that’s left then.
Thanks,
Anatoly
From: XU Liang [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org]
Sent: Friday, November 7, 2014 9:57 AM
To: Burakov, Anatoly; dev-VfR2kkLFssw@public.gmane.org
Cc: thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo
Subject: Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
How to find the maximum end virtual address ? I'm not the DPDK expert, but I will try to do my best.
If the segments isn't overlap, "if(seg->addr > last->addr) last = seg;" already find the segment with maximum end virtual address.
------------------------------------------------------------------
From:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Time:2014 Nov 7 (Fri) 17 : 47
To:徐亮
<liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>,
dev-VfR2kkLFssw@public.gmane.org <dev-VfR2kkLFssw@public.gmane.org>
Cc:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org <thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>, De Lara Guarch, Pablo <pablo.de.lara.guarch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
The commit message looks fine to me, but VFIO code needs to be adjusted the same way.
Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the
last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).
My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).
My apologies for not thinking about any of this during the first 6 iterations of the patch :(
Thanks,
Anatoly
-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org]
Sent: Friday, November 7, 2014 8:01 AM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly; thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo
Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary
process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If
it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.
This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.
Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */ int pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = RTE_PTR_ADD(last->addr, last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
--
1.9.1
^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v7] eal: map uio resources after hugepages.
[not found] ` <1415193919-17361-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
` (6 preceding siblings ...)
2014-11-07 8:01 ` [PATCH v6] " lxu
@ 2014-11-07 14:57 ` lxu
[not found] ` <1415372269-8723-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-07 15:19 ` XU Liang
7 siblings, 2 replies; 41+ messages in thread
From: lxu @ 2014-11-07 14:57 UTC (permalink / raw)
To: dev-VfR2kkLFssw
A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot mmap" will be got.
This patch try to map uio resources after hugepages. So the error can be resolved by set base-virtaddr into free virtual address space.
Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci.c | 25 ++++++++++++++++++++--
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 6 ++++--
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 4 +++-
lib/librte_eal/linuxapp/eal/include/eal_pci_init.h | 4 ++--
4 files changed, 32 insertions(+), 7 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..aef6f5e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -483,15 +483,36 @@ pci_config_space_set(struct rte_pci_device *dev)
}
#endif
+static void *
+pci_find_max_end_va(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return RTE_PTR_ADD(last->addr, last->len);
+}
+
static int
pci_map_device(struct rte_pci_device *dev)
{
int ret, mapped = 0;
+ static void * requested_addr;
+ if(NULL == requested_addr)
+ requested_addr = pci_find_max_end_va();
/* try mapping the NIC resources using VFIO if it exists */
#ifdef VFIO_PRESENT
if (pci_vfio_is_enabled()) {
- ret = pci_vfio_map_resource(dev);
+ ret = pci_vfio_map_resource(dev, &requested_addr);
if (ret == 0)
mapped = 1;
else if (ret < 0)
@@ -500,7 +521,7 @@ pci_map_device(struct rte_pci_device *dev)
#endif
/* map resources for devices that use igb_uio */
if (!mapped) {
- ret = pci_uio_map_resource(dev);
+ ret = pci_uio_map_resource(dev, &requested_addr);
if (ret != 0)
return ret;
}
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e92124e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -275,7 +275,7 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
/* map the PCI resource of a PCI device in virtual memory */
int
-pci_uio_map_resource(struct rte_pci_device *dev)
+pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr)
{
int i, j;
char dirname[PATH_MAX];
@@ -371,10 +371,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(*requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else
+ *requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..2102adf 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -515,7 +515,7 @@ clear_current_group(void)
* primary and secondary processes follow almost exactly the same path
*/
int
-pci_vfio_map_resource(struct rte_pci_device *dev)
+pci_vfio_map_resource(struct rte_pci_device *dev, void **requested_addr)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -720,6 +720,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
if (i == msix_bar)
continue;
+ maps[i].addr = *requested_addr;
bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
reg.size);
@@ -732,6 +733,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
return -1;
}
+ *requested_addr = bar_addr;
maps[i].addr = bar_addr;
maps[i].offset = reg.offset;
maps[i].size = reg.size;
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..e14fa36 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -63,7 +63,7 @@ void *pci_map_resource(void *requested_addr, int fd, off_t offset,
size_t size);
/* map IGB_UIO resource prototype */
-int pci_uio_map_resource(struct rte_pci_device *dev);
+int pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr);
#ifdef VFIO_PRESENT
@@ -74,7 +74,7 @@ int pci_vfio_is_enabled(void);
int pci_vfio_mp_sync_setup(void);
/* map VFIO resource prototype */
-int pci_vfio_map_resource(struct rte_pci_device *dev);
+int pci_vfio_map_resource(struct rte_pci_device *dev, void **requested_addr);
int pci_vfio_get_group_fd(int iommu_group_fd);
int pci_vfio_get_container_fd(void);
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread[parent not found: <1415372269-8723-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>]
* Re: [PATCH v7] eal: map uio resources after hugepages.
[not found] ` <1415372269-8723-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
@ 2014-11-07 15:14 ` Burakov, Anatoly
2014-11-07 15:15 ` Thomas Monjalon
1 sibling, 0 replies; 41+ messages in thread
From: Burakov, Anatoly @ 2014-11-07 15:14 UTC (permalink / raw)
To: lxu, dev-VfR2kkLFssw@public.gmane.org
Um, not sure I agree with this implementation. I think a cleaner way would be to put the requested_addr in pci_uio_map_resource and pci_vfio_map_resource (or rather, put it in include/eal_pci_init.h, like extern void *requested_addr) but make actual use of it in pci_uio/vfio_map_resource only (and leave all of this out of eal_pci.c at all). That will also rid you of the necessity to pass around pointers to pointers.
(in that case I would also rename requested_addr to pci_map_addr or something, to make it less vague)
Thanks,
Anatoly
-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org]
Sent: Friday, November 7, 2014 2:58 PM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly; thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo
Subject: [PATCH v7] eal: map uio resources after hugepages.
A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot mmap" will be got.
This patch try to map uio resources after hugepages. So the error can be resolved by set base-virtaddr into free virtual address space.
Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci.c | 25 ++++++++++++++++++++--
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 6 ++++--
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 4 +++-
lib/librte_eal/linuxapp/eal/include/eal_pci_init.h | 4 ++--
4 files changed, 32 insertions(+), 7 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..aef6f5e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -483,15 +483,36 @@ pci_config_space_set(struct rte_pci_device *dev) } #endif
+static void *
+pci_find_max_end_va(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return RTE_PTR_ADD(last->addr, last->len); }
+
static int
pci_map_device(struct rte_pci_device *dev) {
int ret, mapped = 0;
+ static void * requested_addr;
+ if(NULL == requested_addr)
+ requested_addr = pci_find_max_end_va();
/* try mapping the NIC resources using VFIO if it exists */ #ifdef VFIO_PRESENT
if (pci_vfio_is_enabled()) {
- ret = pci_vfio_map_resource(dev);
+ ret = pci_vfio_map_resource(dev, &requested_addr);
if (ret == 0)
mapped = 1;
else if (ret < 0)
@@ -500,7 +521,7 @@ pci_map_device(struct rte_pci_device *dev) #endif
/* map resources for devices that use igb_uio */
if (!mapped) {
- ret = pci_uio_map_resource(dev);
+ ret = pci_uio_map_resource(dev, &requested_addr);
if (ret != 0)
return ret;
}
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e92124e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -275,7 +275,7 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
/* map the PCI resource of a PCI device in virtual memory */ int -pci_uio_map_resource(struct rte_pci_device *dev)
+pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr)
{
int i, j;
char dirname[PATH_MAX];
@@ -371,10 +371,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(*requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else
+ *requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..2102adf 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -515,7 +515,7 @@ clear_current_group(void)
* primary and secondary processes follow almost exactly the same path
*/
int
-pci_vfio_map_resource(struct rte_pci_device *dev)
+pci_vfio_map_resource(struct rte_pci_device *dev, void
+**requested_addr)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -720,6 +720,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
if (i == msix_bar)
continue;
+ maps[i].addr = *requested_addr;
bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
reg.size);
@@ -732,6 +733,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
return -1;
}
+ *requested_addr = bar_addr;
maps[i].addr = bar_addr;
maps[i].offset = reg.offset;
maps[i].size = reg.size;
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..e14fa36 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -63,7 +63,7 @@ void *pci_map_resource(void *requested_addr, int fd, off_t offset,
size_t size);
/* map IGB_UIO resource prototype */
-int pci_uio_map_resource(struct rte_pci_device *dev);
+int pci_uio_map_resource(struct rte_pci_device *dev, void
+**requested_addr);
#ifdef VFIO_PRESENT
@@ -74,7 +74,7 @@ int pci_vfio_is_enabled(void); int pci_vfio_mp_sync_setup(void);
/* map VFIO resource prototype */
-int pci_vfio_map_resource(struct rte_pci_device *dev);
+int pci_vfio_map_resource(struct rte_pci_device *dev, void
+**requested_addr);
int pci_vfio_get_group_fd(int iommu_group_fd); int pci_vfio_get_container_fd(void);
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread* Re: [PATCH v7] eal: map uio resources after hugepages.
[not found] ` <1415372269-8723-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
2014-11-07 15:14 ` Burakov, Anatoly
@ 2014-11-07 15:15 ` Thomas Monjalon
1 sibling, 0 replies; 41+ messages in thread
From: Thomas Monjalon @ 2014-11-07 15:15 UTC (permalink / raw)
To: lxu; +Cc: dev-VfR2kkLFssw
Hi Liang,
I see you are learning how to send a patch. It's a good thing.
I don't know if the idea of your patch will be accepted but here are more tips
about formatting:
- You should use checkpatch.pl from kernel.org and fix the most important issues
- The commit log should be wrapped to avoid long lines
- The title is important. You didn't update it for vfio.
And there should be no point at the end of the title.
- You should use your real name in Signed-off-by.
- You should annotate each new version to show the changes
See http://dpdk.org/dev#send and
https://www.kernel.org/doc/Documentation/SubmittingPatches
Looking at other submissions could be inspiring also.
Thanks
--
Thomas
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v7] eal: map uio resources after hugepages.
2014-11-07 14:57 ` [PATCH v7] eal: map uio resources after hugepages lxu
[not found] ` <1415372269-8723-1-git-send-email-liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
@ 2014-11-07 15:19 ` XU Liang
1 sibling, 0 replies; 41+ messages in thread
From: XU Liang @ 2014-11-07 15:19 UTC (permalink / raw)
To: Burakov, Anatoly, dev-VfR2kkLFssw@public.gmane.org
I got it. Just some guys don't like global variable. I'm not sure what is DPDK code style.------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Time:2014 Nov 7 (Fri) 23 : 15To:徐亮 <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>, dev-VfR2kkLFssw@public.gmane.org <dev-VfR2kkLFssw@public.gmane.org>Cc:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org <thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>, De Lara Guarch, Pablo <pablo.de.lara.guarch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Subject:RE: [PATCH v7] eal: map uio resources after hugepages.
Um, not sure I agree with this implementation. I think a cleaner way would be to put the requested_addr in pci_uio_map_resource and pci_vfio_map_resource (or rather, put it in include/eal_pci_init.h, like extern void *requested_addr) but make actual use of it in pci_uio/vfio_map_resource only (and leave all of this out of eal_pci.c at all). That will also rid you of the necessity to pass around pointers to pointers.
(in that case I would also rename requested_addr to pci_map_addr or something, to make it less vague)
Thanks,
Anatoly
-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org]
Sent: Friday, November 7, 2014 2:58 PM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly; thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo
Subject: [PATCH v7] eal: map uio resources after hugepages.
A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot mmap" will be got.
This patch try to map uio resources after hugepages. So the error can be resolved by set base-virtaddr into free virtual address space.
Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
lib/librte_eal/linuxapp/eal/eal_pci.c | 25 ++++++++++++++++++++--
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 6 ++++--
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 4 +++-
lib/librte_eal/linuxapp/eal/include/eal_pci_init.h | 4 ++--
4 files changed, 32 insertions(+), 7 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..aef6f5e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -483,15 +483,36 @@ pci_config_space_set(struct rte_pci_device *dev) } #endif
+static void *
+pci_find_max_end_va(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return RTE_PTR_ADD(last->addr, last->len); }
+
static int
pci_map_device(struct rte_pci_device *dev) {
int ret, mapped = 0;
+ static void * requested_addr;
+ if(NULL == requested_addr)
+ requested_addr = pci_find_max_end_va();
/* try mapping the NIC resources using VFIO if it exists */ #ifdef VFIO_PRESENT
if (pci_vfio_is_enabled()) {
- ret = pci_vfio_map_resource(dev);
+ ret = pci_vfio_map_resource(dev, &requested_addr);
if (ret == 0)
mapped = 1;
else if (ret < 0)
@@ -500,7 +521,7 @@ pci_map_device(struct rte_pci_device *dev) #endif
/* map resources for devices that use igb_uio */
if (!mapped) {
- ret = pci_uio_map_resource(dev);
+ ret = pci_uio_map_resource(dev, &requested_addr);
if (ret != 0)
return ret;
}
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e92124e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -275,7 +275,7 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
/* map the PCI resource of a PCI device in virtual memory */ int -pci_uio_map_resource(struct rte_pci_device *dev)
+pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr)
{
int i, j;
char dirname[PATH_MAX];
@@ -371,10 +371,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(*requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else
+ *requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..2102adf 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -515,7 +515,7 @@ clear_current_group(void)
* primary and secondary processes follow almost exactly the same path
*/
int
-pci_vfio_map_resource(struct rte_pci_device *dev)
+pci_vfio_map_resource(struct rte_pci_device *dev, void
+**requested_addr)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -720,6 +720,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
if (i == msix_bar)
continue;
+ maps[i].addr = *requested_addr;
bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
reg.size);
@@ -732,6 +733,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
return -1;
}
+ *requested_addr = bar_addr;
maps[i].addr = bar_addr;
maps[i].offset = reg.offset;
maps[i].size = reg.size;
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..e14fa36 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -63,7 +63,7 @@ void *pci_map_resource(void *requested_addr, int fd, off_t offset,
size_t size);
/* map IGB_UIO resource prototype */
-int pci_uio_map_resource(struct rte_pci_device *dev);
+int pci_uio_map_resource(struct rte_pci_device *dev, void
+**requested_addr);
#ifdef VFIO_PRESENT
@@ -74,7 +74,7 @@ int pci_vfio_is_enabled(void); int pci_vfio_mp_sync_setup(void);
/* map VFIO resource prototype */
-int pci_vfio_map_resource(struct rte_pci_device *dev);
+int pci_vfio_map_resource(struct rte_pci_device *dev, void
+**requested_addr);
int pci_vfio_get_group_fd(int iommu_group_fd); int pci_vfio_get_container_fd(void);
--
1.9.1
^ permalink raw reply related [flat|nested] 41+ messages in thread