[PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..bc7ed3a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct rte_pci_addr *loc = &dev->addr;
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		requested_addr = (uint8_t *) internal_config.base_virtaddr 
+			+ internal_config.memory;
+	}
 
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

I have a slight problems with this patch.

The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever reasons.

Also, 

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of lxu
Sent: Wednesday, November 5, 2014 1:25 PM
To: dev-VfR2kkLFssw@public.gmane.org
Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..bc7ed3a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct rte_pci_addr *loc = &dev->addr;
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		requested_addr = (uint8_t *) internal_config.base_virtaddr 
+			+ internal_config.memory;
+	}
 
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
--
1.9.1

答复： [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[XU Liang]

I think the base_virtadd will be carefully selected by user when they need it. So maybe it's not a real problem.  :>The real reason is I can't find a easy way to get the end address of hugepages. Can you give me some suggestions ?------------------------------------------------------------------发件人：Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>发送时间：2014年11月5日(星期三) 23:10收件人：徐亮 <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>，dev-VfR2kkLFssw@public.gmane.org <dev-VfR2kkLFssw@public.gmane.org>主　题：RE: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the	base_virtaddr is configured.
I have a slight problems with this patch.

The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever reasons.

Also, 

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of lxu
Sent: Wednesday, November 5, 2014 1:25 PM
To: dev-VfR2kkLFssw@public.gmane.org
Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..bc7ed3a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct rte_pci_addr *loc = &dev->addr;
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		requested_addr = (uint8_t *) internal_config.base_virtaddr 
+			+ internal_config.memory;
+	}
 
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
--
1.9.1

Re: 答复： [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

Hi Liang

Yes it is a problem. Even if it was carefully selected by user, nothing stops the DPDK application from mapping something into where you’re trying to map your UIO devices. Plus, this changes the default behavior where a wrong base-virtaddr leads to a failure to initialize, rather than simply using a different address (remember that pci_map_resource fails if it cannot map the resource at the exact address you requested).

A very crude way of finding out where hugepages end would be to walk the hugepage memory (walk through memsegs and note the maximum start addr + length of that memseg).

Could you perhaps explain what is the problem that you’re trying to solve with this? I can’t think of a situation where the location of UIO maps would matter, to be honest.

Thanks,
Anatoly

From: XU Liang [mailto:liang.xu@cinfotech.cn]
Sent: Wednesday, November 5, 2014 3:49 PM
To: Burakov, Anatoly; dev@dpdk.org
Subject: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

I think the base_virtadd will be carefully selected by user when they need it. So maybe it's not a real problem.  :>

The real reason is I can't find a easy way to get the end address of hugepages. Can you give me some suggestions ?
------------------------------------------------------------------
发件人：Burakov, Anatoly <anatoly.burakov@intel.com<mailto:anatoly.burakov@intel.com>>
发送时间：2014年11月5日(星期三) 23:10
收件人：徐亮 <liang.xu@cinfotech.cn<mailto:liang.xu@cinfotech.cn>>，dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
主　题：RE: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

I have a slight problems with this patch.

The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever reasons.

Also,

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of lxu
Sent: Wednesday, November 5, 2014 1:25 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..bc7ed3a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct rte_pci_addr *loc = &dev->addr;
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ static void * requested_addr = NULL;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ requested_addr = (uint8_t *) internal_config.base_virtaddr
+ + internal_config.memory;
+ }

dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = (uint8_t *)mapaddr + maps[j].size;
}

if (fail) {
--
1.9.1

答复：答复： [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[XU Liang]

I have a multiple processes application. When start a secondary process, we got error message "EAL: pci_map_resource(): cannot mmap(11, 0x7ffff7fba000, 0x20000, 0x0): Bad file descriptor (0x7ffff7fb9000)".The secondary process link difference shared libraries, so the address 0x7ffff7fba000 is used.
------------------------------------------------------------------发件人：Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>发送时间：2014年11月5日(星期三) 23:59收件人：徐亮 <liang.xu@cinfotech.cn>，dev-VfR2kkLFssw@public.gmane.org <dev-VfR2kkLFssw@public.gmane.org>主　题：RE: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the	base_virtaddr is configured.



font-family: MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Cambria Math;panose-1: 2 4 5 3 5 4 6 3 2 4;font-family: Calibri;panose-1: 2 15 5 2 2 2 4 3 2 4;font-family: Tahoma;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Microsoft JhengHei;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@Microsoft JhengHei;panose-1: 2 11 6 4 3 5 4 4 2 4;p.MsoNormal, li.MsoNormal, div.MsoNormal {margin: 0.0cm;margin-bottom: 1.0E-4pt;font-size: 12.0pt;font-family: Times New Roman , serif;}
a:link, span.MsoHyperlink {mso-style-priority: 99;color: #0563c1;text-decoration: underline;}
a:visited, span.MsoHyperlinkFollowed {mso-style-priority: 99;color: #954f72;text-decoration: underline;}
span.EmailStyle17 {mso-style-type: personal-reply;font-family: Calibri , sans-serif;color: #1f497d;}
*.MsoChpDefault {mso-style-type: export-only;font-family: Calibri , sans-serif;}
size: 612.0pt 792.0pt;margin: 72.0pt 72.0pt 72.0pt 72.0pt;div.WordSection1 {page: WordSection1;}




Hi Liang
 
Yes it is a problem. Even if it was carefully selected by user, nothing stops the DPDK application from mapping something into where you’re trying to map your
 UIO devices. Plus, this changes the default behavior where a wrong base-virtaddr leads to a failure to initialize, rather than simply using a different address (remember that pci_map_resource fails if it cannot map the resource at the exact address you requested).
 
A very crude way of finding out where hugepages end would be to walk the hugepage memory (walk through memsegs and note the maximum start addr + length of that
 memseg).
 
Could you perhaps explain what is the problem that you’re trying to solve with this? I can’t think of a situation where the location of UIO maps would matter,
 to be honest.
 
Thanks,
Anatoly
 
From: XU Liang [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org]


Sent: Wednesday, November 5, 2014 3:49 PM

To: Burakov, Anatoly; dev@dpdk.org

Subject: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.
 


I think the base_virtadd will be carefully selected by user when they need it. So maybe it's not a real problem.  :>


 


The real reason is I can't find a easy way to get the end address of hugepages. Can you give me some suggestions ?



------------------------------------------------------------------


发件人：Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>


发送时间：2014年11月5日(星期三)
 23:10


收件人：徐亮 <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>，dev-VfR2kkLFssw@public.gmane.org
 <dev-VfR2kkLFssw@public.gmane.org>


主　题：RE:
 [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.


 

I have a slight problems with this patch.



The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will
 get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever
 reasons.



Also, 



Thanks,

Anatoly



-----Original Message-----

From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of lxu

Sent: Wednesday, November 5, 2014 1:25 PM

To: dev-VfR2kkLFssw@public.gmane.org

Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.



---

lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-

1 file changed, 8 insertions(+), 1 deletion(-)



diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

index 7e62266..bc7ed3a 100644

--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)

struct rte_pci_addr *loc = &dev->addr;

struct mapped_pci_resource *uio_res;

struct pci_map *maps;

+ static void * requested_addr = NULL;

+ if (internal_config.base_virtaddr && NULL == requested_addr) {

+ requested_addr = (uint8_t *) internal_config.base_virtaddr 

+ + internal_config.memory;

+ }



dev->intr_handle.fd = -1;

dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)

if (maps[j].addr != NULL)

fail = 1;

else {

- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,

+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,

(size_t)maps[j].size);

if (mapaddr == NULL)

fail = 1;

+ else if (NULL != requested_addr)

+ requested_addr = (uint8_t *)mapaddr + maps[j].size;

}



if (fail) {

--

1.9.1

Re: 答复：答复： [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

Ah, that makes sense. So you’re actually hitting the very issue I’m concerned about when mapping purely with base-virtaddr (a library is mapped into where you’re trying to map your UIO resources).

Well, as I said, you can try and walk the memsegs and work out the biggest end-address of hugepage memory. That’s the easiest way I can think of.

Thanks,
Anatoly

From: XU Liang [mailto:liang.xu@cinfotech.cn]
Sent: Wednesday, November 5, 2014 4:10 PM
To: Burakov, Anatoly; dev@dpdk.org
Subject: 答复：答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

I have a multiple processes application. When start a secondary process, we got error message "EAL: pci_map_resource(): cannot mmap(11, 0x7ffff7fba000, 0x20000, 0x0): Bad file descriptor (0x7ffff7fb9000)".

The secondary process link difference shared libraries, so the address 0x7ffff7fba000 is used.

------------------------------------------------------------------
发件人：Burakov, Anatoly <anatoly.burakov@intel.com<mailto:anatoly.burakov@intel.com>>
发送时间：2014年11月5日(星期三) 23:59
收件人：徐亮 <liang.xu@cinfotech.cn<mailto:liang.xu@cinfotech.cn>>，dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
主　题：RE: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

Hi Liang

Yes it is a problem. Even if it was carefully selected by user, nothing stops the DPDK application from mapping something into where you’re trying to map your UIO devices. Plus, this changes the default behavior where a wrong base-virtaddr leads to a failure to initialize, rather than simply using a different address (remember that pci_map_resource fails if it cannot map the resource at the exact address you requested).

A very crude way of finding out where hugepages end would be to walk the hugepage memory (walk through memsegs and note the maximum start addr + length of that memseg).

Could you perhaps explain what is the problem that you’re trying to solve with this? I can’t think of a situation where the location of UIO maps would matter, to be honest.

Thanks,
Anatoly

From: XU Liang [mailto:liang.xu@cinfotech.cn]
Sent: Wednesday, November 5, 2014 3:49 PM
To: Burakov, Anatoly; dev@dpdk.org<mailto:dev@dpdk.org>
Subject: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

I think the base_virtadd will be carefully selected by user when they need it. So maybe it's not a real problem.  :>

The real reason is I can't find a easy way to get the end address of hugepages. Can you give me some suggestions ?
------------------------------------------------------------------
发件人：Burakov, Anatoly <anatoly.burakov@intel.com<mailto:anatoly.burakov@intel.com>>
发送时间：2014年11月5日(星期三) 23:10
收件人：徐亮 <liang.xu@cinfotech.cn<mailto:liang.xu@cinfotech.cn>>，dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
主　题：RE: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

I have a slight problems with this patch.

The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever reasons.

Also,

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of lxu
Sent: Wednesday, November 5, 2014 1:25 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..bc7ed3a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct rte_pci_addr *loc = &dev->addr;
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ static void * requested_addr = NULL;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ requested_addr = (uint8_t *) internal_config.base_virtaddr
+ + internal_config.memory;
+ }

dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = (uint8_t *)mapaddr + maps[j].size;
}

if (fail) {
--
1.9.1

[PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a591da3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = (void *)(last->addr_64 + last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

Few nitpicks.

Static variables are always initialized to 0, so "= NULL" isn't necessary, a declaration will suffice. Also, we have a macro RTE_PTR_ADD to add numbers to pointers, I think it would be better to use those. Otherwise, looks fine to me.

I still feel uneasy about depending on nothing being mapped directly after hugepages (perhaps we could do mmap(bar_size) before trying pci_map_resource, and increment requested_addr until we find a free spot?), but I imagine this case would be quite rare, so probably it's not worth the added kludge.

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org] 
Sent: Thursday, November 6, 2014 2:12 PM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly
Subject: [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a591da3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = (void *)(last->addr_64 + last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
-- 
1.9.1

[PATCH v3] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..3a218d0 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
-- 
1.9.1

答复：[PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.

[徐亮]

When user configure base_virtaddr, we should believe they can take care it.
In my case, I always check /proc/xxxx/maps to find a huge free address space, such as 0x20 0000 0000, to map all the hugepages and uio resource. 
------------------------------------------------------------------发件人：Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>发送时间：2014年11月6日(星期四) 22:29收件人：徐亮 <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>，dev-VfR2kkLFssw@public.gmane.org <dev-VfR2kkLFssw@public.gmane.org>主　题：RE: [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.
Few nitpicks.

Static variables are always initialized to 0, so "= NULL" isn't necessary, a declaration will suffice. Also, we have a macro RTE_PTR_ADD to add numbers to pointers, I think it would be better to use those. Otherwise, looks fine to me.

I still feel uneasy about depending on nothing being mapped directly after hugepages (perhaps we could do mmap(bar_size) before trying pci_map_resource, and increment requested_addr until we find a free spot?), but I imagine this case would be quite rare, so probably it's not worth the added kludge.

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org] 
Sent: Thursday, November 6, 2014 2:12 PM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly
Subject: [PATCH v2] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a591da3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr = NULL;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = (void *)(last->addr_64 + last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [PATCH v3] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

+					requested_addr = (uint8_t *)mapaddr + maps[j].size;

RTE_PTR_ADD?

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org] 
Sent: Thursday, November 6, 2014 2:47 PM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly
Subject: [PATCH v3] eal: map uio resources after hugepages when the base_virtaddr is configured.

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..3a218d0 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */  int  pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = (uint8_t *)mapaddr + maps[j].size;
 			}
 
 			if (fail) {
--
1.9.1

Re: [PATCH v3] eal: map uio resources after hugepages when the base_virtaddr is configured.

[De Lara Guarch, Pablo]

Include at least signoff.

> -----Original Message-----
> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of lxu
> Sent: Thursday, November 06, 2014 2:47 PM
> To: dev-VfR2kkLFssw@public.gmane.org
> Subject: [dpdk-dev] [PATCH v3] eal: map uio resources after hugepages
> when the base_virtaddr is configured.
> 
> ---
>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29
> ++++++++++++++++++++++++++++-
>  1 file changed, 28 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> index 7e62266..3a218d0 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> @@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char
> *dstbuf,
>  	return uio_num;
>  }
> 
> +static inline const struct rte_memseg *
> +get_physmem_last(void)
> +{
> +	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
> +	const struct rte_memseg * last = seg;
> +	unsigned i = 0;
> +
> +	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
> +		if (seg->addr == NULL)
> +			break;
> +
> +		if(seg->addr > last->addr)
> +		 	last = seg;
> +
> +	}
> +	return last;
> +}
> +
>  /* map the PCI resource of a PCI device in virtual memory */
>  int
>  pci_uio_map_resource(struct rte_pci_device *dev)
> @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>  	struct mapped_pci_resource *uio_res;
>  	struct pci_map *maps;
> 
> +	/* map uio resource into user required virtual address */
> +	static void * requested_addr;
> +	if (internal_config.base_virtaddr && NULL == requested_addr) {
> +		const struct rte_memseg * last = get_physmem_last();
> +		requested_addr = RTE_PTR_ADD(last->addr, last->len);
> +	}
> +
>  	dev->intr_handle.fd = -1;
>  	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
> 
> @@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>  			if (maps[j].addr != NULL)
>  				fail = 1;
>  			else {
> -				mapaddr = pci_map_resource(NULL, fd,
> (off_t)offset,
> +				mapaddr =
> pci_map_resource(requested_addr, fd, (off_t)offset,
>  						(size_t)maps[j].size);
>  				if (mapaddr == NULL)
>  					fail = 1;
> +				else if (NULL != requested_addr)
> +					requested_addr = (uint8_t
> *)mapaddr + maps[j].size;
>  			}
> 
>  			if (fail) {
> --
> 1.9.1

[PATCH v4] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
-- 
1.9.1

[PATCH v4] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [PATCH v4] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Thomas Monjalon]

No explanation -> I won't review it.
No signed-off-by -> I'll ignore it.

Maybe your patch is useful, maybe not. I cannot know.

-- 
Thomas

[PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

Sorry, I'm learning the right way to send a patch by git. 
I have a multiple processes application. When start the secondary process, I got error message "EAL: pci_map_resource(): cannot mmap(11, 0x7ffff7fba000, 0x20000, 0x0): Bad file descriptor (0x7ffff7fb9000)".
The secondary process links a lot of additional shared libraries, so the address 0x7ffff7fba000 had already be used.
I had fixed similar hugepages mmap problems by base_virtaddr. So I believe the uio resource should be mapped into base_virtaddr at this situation.
This patch try to fix it.


Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

The explanation of the patch should be generic and impartial, i.e. when this and this happens, it results in such and such problem, and this patch fixes it by doing this and that. In other words, this will appear in the git history, so whoever is reading the commit log will be able to figure out what this patch does and why it's been applied.

Thomas, do we need to do similar changes to VFIO code, to keep consistency? Also, do we really need for this to depend on --base-virtaddr? Why not do it unconditionally, i.e. map PCI resources right after hugepages in memory?

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org] 
Sent: Thursday, November 6, 2014 3:32 PM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly; thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo
Subject: [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

Sorry, I'm learning the right way to send a patch by git. 
I have a multiple processes application. When start the secondary process, I got error message "EAL: pci_map_resource(): cannot mmap(11, 0x7ffff7fba000, 0x20000, 0x0): Bad file descriptor (0x7ffff7fb9000)".
The secondary process links a lot of additional shared libraries, so the address 0x7ffff7fba000 had already be used.
I had fixed similar hugepages mmap problems by base_virtaddr. So I believe the uio resource should be mapped into base_virtaddr at this situation.
This patch try to fix it.


Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */  int  pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
--
1.9.1

Re: [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Thomas Monjalon]

2014-11-06 15:41, Burakov, Anatoly:
> Thomas, do we need to do similar changes to VFIO code, to keep consistency?
> Also, do we really need for this to depend on --base-virtaddr? Why not do it
> unconditionally, i.e. map PCI resources right after hugepages in memory?

I don't really like the secondary process mechanism at all.
So I won't give good advice here ;)
But I feel this option --base-virtaddr should be improved or removed.

-- 
Thomas

Re: [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

Well, removing --base-virtaddr is not what I'm asking about.

The issue at hand here is that, given our secondary process mechanism (that you don't like :-) ), some stuff may be attempted to be mapped into space a secondary process may already have mapped something else into (some libraries, for example). This issue was originally discovered by OVDK, so we added a --base-virtaddr option to try and map hugepages at exact virtual address, rather than wherever mmap decides to do so on its own.

The issue encountered by Liang (the author of the patch) is similar, only it's not the hugepages are mapped into the occupied space, but rather the PCI resources (which are mapped with NULL by default, so can be mapped anywhere). Therefore he suggested a patch that maps the PCI resources into a space just after the last hugepage when --base-virtaddr is provided. I'm not sure we need the dependence on --base-virtaddr though, it can probably be done unconditionally. If you have no opinion on the matter, we can leave this detail of the patch as it is, then.

Also, I would suspect that if we are to modify where UIO resources are mapped, VFIO code should be modified the same way to avoid inconsistency between the two.

Thanks,
Anatoly

-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org] 
Sent: Thursday, November 6, 2014 3:58 PM
To: Burakov, Anatoly
Cc: lxu; dev-VfR2kkLFssw@public.gmane.org; De Lara Guarch, Pablo
Subject: Re: [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

2014-11-06 15:41, Burakov, Anatoly:
> Thomas, do we need to do similar changes to VFIO code, to keep consistency?
> Also, do we really need for this to depend on --base-virtaddr? Why not 
> do it unconditionally, i.e. map PCI resources right after hugepages in memory?

I don't really like the secondary process mechanism at all.
So I won't give good advice here ;)
But I feel this option --base-virtaddr should be improved or removed.

--
Thomas

Re: [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Bruce Richardson]

On Thu, Nov 06, 2014 at 04:10:15PM +0000, Burakov, Anatoly wrote:
> Well, removing --base-virtaddr is not what I'm asking about.
> 
> The issue at hand here is that, given our secondary process mechanism (that you don't like :-) ), some stuff may be attempted to be mapped into space a secondary process may already have mapped something else into (some libraries, for example). This issue was originally discovered by OVDK, so we added a --base-virtaddr option to try and map hugepages at exact virtual address, rather than wherever mmap decides to do so on its own.
> 
> The issue encountered by Liang (the author of the patch) is similar, only it's not the hugepages are mapped into the occupied space, but rather the PCI resources (which are mapped with NULL by default, so can be mapped anywhere). Therefore he suggested a patch that maps the PCI resources into a space just after the last hugepage when --base-virtaddr is provided. I'm not sure we need the dependence on --base-virtaddr though, it can probably be done unconditionally. If you have no opinion on the matter, we can leave this detail of the patch as it is, then.
> 
> Also, I would suspect that if we are to modify where UIO resources are mapped, VFIO code should be modified the same way to avoid inconsistency between the two.
> 

I find nothing wrong with your logic, Anatoly, it makes sense to me. :-)

I'm curious, however, as to what Thomas has in mind for how we might improve
the base-virtaddr flag.

/Bruce

> Thanks,
> Anatoly
> 
> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org] 
> Sent: Thursday, November 6, 2014 3:58 PM
> To: Burakov, Anatoly
> Cc: lxu; dev-VfR2kkLFssw@public.gmane.org; De Lara Guarch, Pablo
> Subject: Re: [PATCH v5] eal: map uio resources after hugepages when the base_virtaddr is configured.
> 
> 2014-11-06 15:41, Burakov, Anatoly:
> > Thomas, do we need to do similar changes to VFIO code, to keep consistency?
> > Also, do we really need for this to depend on --base-virtaddr? Why not 
> > do it unconditionally, i.e. map PCI resources right after hugepages in memory?
> 
> I don't really like the secondary process mechanism at all.
> So I won't give good advice here ;)
> But I feel this option --base-virtaddr should be improved or removed.
> 
> --
> Thomas

[PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

[lxu]

A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.

This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */
 int
 pci_uio_map_resource(struct rte_pci_device *dev)
@@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
-- 
1.9.1

Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Bruce Richardson]

On Fri, Nov 07, 2014 at 04:01:24PM +0800, lxu wrote:
> A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.
> 
> This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.
> 
> Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
> ---
>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
>  1 file changed, 28 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> index 7e62266..a2c9ab6 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> @@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
>  	return uio_num;
>  }
>  
> +static inline const struct rte_memseg *
> +get_physmem_last(void)
> +{
> +	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
> +	const struct rte_memseg * last = seg;
> +	unsigned i = 0;
> +
> +	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
> +		if (seg->addr == NULL)
> +			break;
> +
> +		if(seg->addr > last->addr)
> +		 	last = seg;
> +
> +	}
> +	return last;
> +}
> +
>  /* map the PCI resource of a PCI device in virtual memory */
>  int
>  pci_uio_map_resource(struct rte_pci_device *dev)
> @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>  	struct mapped_pci_resource *uio_res;
>  	struct pci_map *maps;
>  
> +	/* map uio resource into user required virtual address */
> +	static void * requested_addr;
> +	if (internal_config.base_virtaddr && NULL == requested_addr) {
> +		const struct rte_memseg * last = get_physmem_last();
> +		requested_addr = RTE_PTR_ADD(last->addr, last->len);
> +	}
> +

Could you perhaps take anatoly's suggestion and modify this patch so the checking
for a space right after the memsegs does not just depend on base-virtaddr being
set. Unless it causes other problems, there is no reason this code should not
always be used.

/Bruce

Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

The commit message looks fine to me, but VFIO code needs to be adjusted the same way.

Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).

My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).

My apologies for not thinking about any of this during the first 6 iterations of the patch :(

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org] 
Sent: Friday, November 7, 2014 8:01 AM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly; thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo
Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.

This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */  int  pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
--
1.9.1

Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

[XU Liang]

How to find the maximum end virtual address ? I'm not the DPDK expert, but I will try to do my best.
 If the segments isn't overlap, "if(seg->addr > last->addr) last = seg;" already find the segment with maximum end virtual address.
------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Time:2014 Nov 7 (Fri) 17 : 47To:徐亮 <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>, dev-VfR2kkLFssw@public.gmane.org <dev@dpdk.org>Cc:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org <thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>, De Lara Guarch, Pablo <pablo.de.lara.guarch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
The commit message looks fine to me, but VFIO code needs to be adjusted the same way.

Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).

My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).

My apologies for not thinking about any of this during the first 6 iterations of the patch :(

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org] 
Sent: Friday, November 7, 2014 8:01 AM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly; thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo
Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.

This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu@cinfotech.cn>
---
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 	return uio_num;
 }
 
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return last;
+}
+
 /* map the PCI resource of a PCI device in virtual memory */  int  pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 	struct mapped_pci_resource *uio_res;
 	struct pci_map *maps;
 
+	/* map uio resource into user required virtual address */
+	static void * requested_addr;
+	if (internal_config.base_virtaddr && NULL == requested_addr) {
+		const struct rte_memseg * last = get_physmem_last();
+		requested_addr = RTE_PTR_ADD(last->addr, last->len);
+	}
+
 	dev->intr_handle.fd = -1;
 	dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
 
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else if (NULL != requested_addr)
+					requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
--
1.9.1

Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

Ah, yes, sorry about that, haven’t quite woke up yet ☹ You’re right. So it’s removing the dependency on --base-virtaddr, moving the function to eal_pci.c  and adding similar VFIO code that’s left then.

Thanks,
Anatoly

From: XU Liang [mailto:liang.xu@cinfotech.cn]
Sent: Friday, November 7, 2014 9:57 AM
To: Burakov, Anatoly; dev@dpdk.org
Cc: thomas.monjalon@6wind.com; De Lara Guarch, Pablo
Subject: Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

How to find the maximum end virtual address ? I'm not the DPDK expert, but I will try to do my best.

If the segments isn't overlap, "if(seg->addr > last->addr) last = seg;" already find the segment with maximum end virtual address.

------------------------------------------------------------------
From:Burakov, Anatoly <anatoly.burakov@intel.com<mailto:anatoly.burakov@intel.com>>
Time:2014 Nov 7 (Fri) 17 : 47
To:徐亮 <liang.xu@cinfotech.cn<mailto:liang.xu@cinfotech.cn>>, dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
Cc:thomas.monjalon@6wind.com <thomas.monjalon@6wind.com<mailto:thomas.monjalon@6wind.com>>, De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com<mailto:pablo.de.lara.guarch@intel.com>>
Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

The commit message looks fine to me, but VFIO code needs to be adjusted the same way.

Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).

My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).

My apologies for not thinking about any of this during the first 6 iterations of the patch :(

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu@cinfotech.cn]
Sent: Friday, November 7, 2014 8:01 AM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Burakov, Anatoly; thomas.monjalon@6wind.com<mailto:thomas.monjalon@6wind.com>; De Lara Guarch, Pablo
Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.

This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu@cinfotech.cn<mailto:liang.xu@cinfotech.cn>>
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}

+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */ int pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;

+ /* map uio resource into user required virtual address */
+ static void * requested_addr;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = RTE_PTR_ADD(last->addr, last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;

@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}

if (fail) {
--
1.9.1

Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

[XU Liang]

Because vfio and uio can be used at the same time, we need save the value of requested_addr at pci_vfio_map_resource() and pci_uio_map_resource().Your guys like add a global variable "void * pci_requested_addr" or a "void** requested_addr" parameter at pci_vfio_map_resource() and pci_uio_map_resource() ?------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Time:2014 Nov 7 (Fri) 18 : 03To:徐亮 <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>, dev-VfR2kkLFssw@public.gmane.org <dev@dpdk.org>Cc:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org <thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>, De Lara Guarch, Pablo <pablo.de.lara.guarch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.



font-family: Wingdings;panose-1: 5 0 0 0 0 0 0 0 0 0;font-family: MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Cambria Math;panose-1: 2 4 5 3 5 4 6 3 2 4;font-family: Calibri;panose-1: 2 15 5 2 2 2 4 3 2 4;font-family: Tahoma;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;p.MsoNormal, li.MsoNormal, div.MsoNormal {margin: 0.0in;margin-bottom: 1.0E-4pt;font-size: 12.0pt;font-family: Times New Roman , serif;}
a:link, span.MsoHyperlink {mso-style-priority: 99;color: #0563c1;text-decoration: underline;}
a:visited, span.MsoHyperlinkFollowed {mso-style-priority: 99;color: #954f72;text-decoration: underline;}
span.EmailStyle17 {mso-style-type: personal-reply;font-family: Calibri , sans-serif;color: #1f497d;}
*.MsoChpDefault {mso-style-type: export-only;font-family: Calibri , sans-serif;}
size: 8.5in 11.0in;margin: 1.0in 1.0in 1.0in 1.0in;div.WordSection1 {page: WordSection1;}




Ah, yes, sorry about that, haven’t quite woke up yet
L You’re right. So it’s removing the dependency on --base-virtaddr, moving the function to eal_pci.c
  and adding similar VFIO code that’s left then.
 
Thanks,
Anatoly
 
From: XU Liang [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org]


Sent: Friday, November 7, 2014 9:57 AM

To: Burakov, Anatoly; dev-VfR2kkLFssw@public.gmane.org

Cc: thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo

Subject: Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
 


How to find the maximum end virtual address ? I'm not the DPDK expert, but I will try to do my best.


 


If the segments isn't overlap, "if(seg->addr > last->addr) last = seg;" already find the segment with maximum end virtual address.


 


------------------------------------------------------------------



From:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>


Time:2014 Nov 7 (Fri) 17 : 47


To:徐亮
 <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>, 
dev-VfR2kkLFssw@public.gmane.org <dev-VfR2kkLFssw@public.gmane.org>


Cc:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org <thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>, De Lara Guarch, Pablo <pablo.de.lara.guarch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>


Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.


 

The commit message looks fine to me, but VFIO code needs to be adjusted the same way.



Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the
 last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).



My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).



My apologies for not thinking about any of this during the first 6 iterations of the patch :(



Thanks,

Anatoly



-----Original Message-----

From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org]


Sent: Friday, November 7, 2014 8:01 AM

To: dev-VfR2kkLFssw@public.gmane.org

Cc: Burakov, Anatoly; thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo

Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.



A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary
 process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If
 it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.



This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.



Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>

---

lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-

1 file changed, 28 insertions(+), 1 deletion(-)



diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

index 7e62266..a2c9ab6 100644

--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,

return uio_num;

}



+static inline const struct rte_memseg *

+get_physmem_last(void)

+{

+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();

+ const struct rte_memseg * last = seg;

+ unsigned i = 0;

+

+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {

+ if (seg->addr == NULL)

+ break;

+

+ if(seg->addr > last->addr)

+ last = seg;

+

+ }

+ return last;

+}

+

/* map the PCI resource of a PCI device in virtual memory */ int pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)

struct mapped_pci_resource *uio_res;

struct pci_map *maps;



+ /* map uio resource into user required virtual address */

+ static void * requested_addr;

+ if (internal_config.base_virtaddr && NULL == requested_addr) {

+ const struct rte_memseg * last = get_physmem_last();

+ requested_addr = RTE_PTR_ADD(last->addr, last->len);

+ }

+

dev->intr_handle.fd = -1;

dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;



@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)

if (maps[j].addr != NULL)

fail = 1;

else {

- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,

+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,

(size_t)maps[j].size);

if (mapaddr == NULL)

fail = 1;

+ else if (NULL != requested_addr)

+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);

}



if (fail) {

--

1.9.1

[PATCH v7] eal: map uio resources after hugepages.

[lxu]

A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot mmap" will be got.

This patch try to map uio resources after hugepages. So the error can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 25 ++++++++++++++++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  6 ++++--
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         |  4 +++-
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  4 ++--
 4 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..aef6f5e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -483,15 +483,36 @@ pci_config_space_set(struct rte_pci_device *dev)
 }
 #endif
 
+static void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len);
+}
+
 static int
 pci_map_device(struct rte_pci_device *dev)
 {
 	int ret, mapped = 0;
+	static void * requested_addr;
+	if(NULL == requested_addr)
+		requested_addr = pci_find_max_end_va();
 
 	/* try mapping the NIC resources using VFIO if it exists */
 #ifdef VFIO_PRESENT
 	if (pci_vfio_is_enabled()) {
-		ret = pci_vfio_map_resource(dev);
+		ret = pci_vfio_map_resource(dev, &requested_addr);
 		if (ret == 0)
 			mapped = 1;
 		else if (ret < 0)
@@ -500,7 +521,7 @@ pci_map_device(struct rte_pci_device *dev)
 #endif
 	/* map resources for devices that use igb_uio */
 	if (!mapped) {
-		ret = pci_uio_map_resource(dev);
+		ret = pci_uio_map_resource(dev, &requested_addr);
 		if (ret != 0)
 			return ret;
 	}
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e92124e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -275,7 +275,7 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 
 /* map the PCI resource of a PCI device in virtual memory */
 int
-pci_uio_map_resource(struct rte_pci_device *dev)
+pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr)
 {
 	int i, j;
 	char dirname[PATH_MAX];
@@ -371,10 +371,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(*requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else
+					*requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..2102adf 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -515,7 +515,7 @@ clear_current_group(void)
  * primary and secondary processes follow almost exactly the same path
  */
 int
-pci_vfio_map_resource(struct rte_pci_device *dev)
+pci_vfio_map_resource(struct rte_pci_device *dev, void **requested_addr)
 {
 	struct vfio_group_status group_status = {
 			.argsz = sizeof(group_status)
@@ -720,6 +720,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
+		maps[i].addr = *requested_addr;
 		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
 				reg.size);
 
@@ -732,6 +733,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 			return -1;
 		}
 
+		*requested_addr = bar_addr;
 		maps[i].addr = bar_addr;
 		maps[i].offset = reg.offset;
 		maps[i].size = reg.size;
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..e14fa36 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -63,7 +63,7 @@ void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
 /* map IGB_UIO resource prototype */
-int pci_uio_map_resource(struct rte_pci_device *dev);
+int pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr);
 
 #ifdef VFIO_PRESENT
 
@@ -74,7 +74,7 @@ int pci_vfio_is_enabled(void);
 int pci_vfio_mp_sync_setup(void);
 
 /* map VFIO resource prototype */
-int pci_vfio_map_resource(struct rte_pci_device *dev);
+int pci_vfio_map_resource(struct rte_pci_device *dev, void **requested_addr);
 int pci_vfio_get_group_fd(int iommu_group_fd);
 int pci_vfio_get_container_fd(void);
 
-- 
1.9.1

Re: [PATCH v7] eal: map uio resources after hugepages.

[Burakov, Anatoly]

Um, not sure I agree with this implementation. I think a cleaner way would be to put the requested_addr in pci_uio_map_resource and pci_vfio_map_resource (or rather, put it in include/eal_pci_init.h, like extern void *requested_addr) but make actual use of it in pci_uio/vfio_map_resource only (and leave all of this out of eal_pci.c at all). That will also rid you of the necessity to pass around pointers to pointers.

(in that case I would also rename requested_addr to pci_map_addr or something, to make it less vague) 

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org] 
Sent: Friday, November 7, 2014 2:58 PM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly; thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo
Subject: [PATCH v7] eal: map uio resources after hugepages.

A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot mmap" will be got.

This patch try to map uio resources after hugepages. So the error can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 25 ++++++++++++++++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  6 ++++--
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         |  4 +++-
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  4 ++--
 4 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..aef6f5e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -483,15 +483,36 @@ pci_config_space_set(struct rte_pci_device *dev)  }  #endif
 
+static void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len); }
+
 static int
 pci_map_device(struct rte_pci_device *dev)  {
 	int ret, mapped = 0;
+	static void * requested_addr;
+	if(NULL == requested_addr)
+		requested_addr = pci_find_max_end_va();
 
 	/* try mapping the NIC resources using VFIO if it exists */  #ifdef VFIO_PRESENT
 	if (pci_vfio_is_enabled()) {
-		ret = pci_vfio_map_resource(dev);
+		ret = pci_vfio_map_resource(dev, &requested_addr);
 		if (ret == 0)
 			mapped = 1;
 		else if (ret < 0)
@@ -500,7 +521,7 @@ pci_map_device(struct rte_pci_device *dev)  #endif
 	/* map resources for devices that use igb_uio */
 	if (!mapped) {
-		ret = pci_uio_map_resource(dev);
+		ret = pci_uio_map_resource(dev, &requested_addr);
 		if (ret != 0)
 			return ret;
 	}
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e92124e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -275,7 +275,7 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 
 /* map the PCI resource of a PCI device in virtual memory */  int -pci_uio_map_resource(struct rte_pci_device *dev)
+pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr)
 {
 	int i, j;
 	char dirname[PATH_MAX];
@@ -371,10 +371,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(*requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else
+					*requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..2102adf 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -515,7 +515,7 @@ clear_current_group(void)
  * primary and secondary processes follow almost exactly the same path
  */
 int
-pci_vfio_map_resource(struct rte_pci_device *dev)
+pci_vfio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr)
 {
 	struct vfio_group_status group_status = {
 			.argsz = sizeof(group_status)
@@ -720,6 +720,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
+		maps[i].addr = *requested_addr;
 		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
 				reg.size);
 
@@ -732,6 +733,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 			return -1;
 		}
 
+		*requested_addr = bar_addr;
 		maps[i].addr = bar_addr;
 		maps[i].offset = reg.offset;
 		maps[i].size = reg.size;
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..e14fa36 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -63,7 +63,7 @@ void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
 /* map IGB_UIO resource prototype */
-int pci_uio_map_resource(struct rte_pci_device *dev);
+int pci_uio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr);
 
 #ifdef VFIO_PRESENT
 
@@ -74,7 +74,7 @@ int pci_vfio_is_enabled(void);  int pci_vfio_mp_sync_setup(void);
 
 /* map VFIO resource prototype */
-int pci_vfio_map_resource(struct rte_pci_device *dev);
+int pci_vfio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr);
 int pci_vfio_get_group_fd(int iommu_group_fd);  int pci_vfio_get_container_fd(void);
 
--
1.9.1

Re: [PATCH v7] eal: map uio resources after hugepages.

[Thomas Monjalon]

Hi Liang,

I see you are learning how to send a patch. It's a good thing.
I don't know if the idea of your patch will be accepted but here are more tips
about formatting:
- You should use checkpatch.pl from kernel.org and fix the most important issues
- The commit log should be wrapped to avoid long lines
- The title is important. You didn't update it for vfio.
And there should be no point at the end of the title.
- You should use your real name in Signed-off-by.
- You should annotate each new version to show the changes

See http://dpdk.org/dev#send and 
https://www.kernel.org/doc/Documentation/SubmittingPatches

Looking at other submissions could be inspiring also.

Thanks
-- 
Thomas

Re: [PATCH v7] eal: map uio resources after hugepages.

[XU Liang]

I got it. Just some guys don't like global variable. I'm not sure what is DPDK code style.------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Time:2014 Nov 7 (Fri) 23 : 15To:徐亮 <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>, dev-VfR2kkLFssw@public.gmane.org <dev-VfR2kkLFssw@public.gmane.org>Cc:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org <thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>, De Lara Guarch, Pablo <pablo.de.lara.guarch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Subject:RE: [PATCH v7] eal: map uio resources after hugepages.
Um, not sure I agree with this implementation. I think a cleaner way would be to put the requested_addr in pci_uio_map_resource and pci_vfio_map_resource (or rather, put it in include/eal_pci_init.h, like extern void *requested_addr) but make actual use of it in pci_uio/vfio_map_resource only (and leave all of this out of eal_pci.c at all). That will also rid you of the necessity to pass around pointers to pointers.

(in that case I would also rename requested_addr to pci_map_addr or something, to make it less vague) 

Thanks,
Anatoly

-----Original Message-----
From: lxu [mailto:liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org] 
Sent: Friday, November 7, 2014 2:58 PM
To: dev-VfR2kkLFssw@public.gmane.org
Cc: Burakov, Anatoly; thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org; De Lara Guarch, Pablo
Subject: [PATCH v7] eal: map uio resources after hugepages.

A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot mmap" will be got.

This patch try to map uio resources after hugepages. So the error can be resolved by set base-virtaddr into free virtual address space.

Signed-off-by: lxu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 25 ++++++++++++++++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  6 ++++--
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         |  4 +++-
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  4 ++--
 4 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..aef6f5e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -483,15 +483,36 @@ pci_config_space_set(struct rte_pci_device *dev)  }  #endif
 
+static void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg * last = seg;
+	unsigned i = 0;
+
+	for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if(seg->addr > last->addr)
+		 	last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len); }
+
 static int
 pci_map_device(struct rte_pci_device *dev)  {
 	int ret, mapped = 0;
+	static void * requested_addr;
+	if(NULL == requested_addr)
+		requested_addr = pci_find_max_end_va();
 
 	/* try mapping the NIC resources using VFIO if it exists */  #ifdef VFIO_PRESENT
 	if (pci_vfio_is_enabled()) {
-		ret = pci_vfio_map_resource(dev);
+		ret = pci_vfio_map_resource(dev, &requested_addr);
 		if (ret == 0)
 			mapped = 1;
 		else if (ret < 0)
@@ -500,7 +521,7 @@ pci_map_device(struct rte_pci_device *dev)  #endif
 	/* map resources for devices that use igb_uio */
 	if (!mapped) {
-		ret = pci_uio_map_resource(dev);
+		ret = pci_uio_map_resource(dev, &requested_addr);
 		if (ret != 0)
 			return ret;
 	}
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e92124e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -275,7 +275,7 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
 
 /* map the PCI resource of a PCI device in virtual memory */  int -pci_uio_map_resource(struct rte_pci_device *dev)
+pci_uio_map_resource(struct rte_pci_device *dev, void **requested_addr)
 {
 	int i, j;
 	char dirname[PATH_MAX];
@@ -371,10 +371,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				mapaddr = pci_map_resource(*requested_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+				else
+					*requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..2102adf 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -515,7 +515,7 @@ clear_current_group(void)
  * primary and secondary processes follow almost exactly the same path
  */
 int
-pci_vfio_map_resource(struct rte_pci_device *dev)
+pci_vfio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr)
 {
 	struct vfio_group_status group_status = {
 			.argsz = sizeof(group_status)
@@ -720,6 +720,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
+		maps[i].addr = *requested_addr;
 		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
 				reg.size);
 
@@ -732,6 +733,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 			return -1;
 		}
 
+		*requested_addr = bar_addr;
 		maps[i].addr = bar_addr;
 		maps[i].offset = reg.offset;
 		maps[i].size = reg.size;
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..e14fa36 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -63,7 +63,7 @@ void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
 /* map IGB_UIO resource prototype */
-int pci_uio_map_resource(struct rte_pci_device *dev);
+int pci_uio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr);
 
 #ifdef VFIO_PRESENT
 
@@ -74,7 +74,7 @@ int pci_vfio_is_enabled(void);  int pci_vfio_mp_sync_setup(void);
 
 /* map VFIO resource prototype */
-int pci_vfio_map_resource(struct rte_pci_device *dev);
+int pci_vfio_map_resource(struct rte_pci_device *dev, void 
+**requested_addr);
 int pci_vfio_get_group_fd(int iommu_group_fd);  int pci_vfio_get_container_fd(void);
 
--
1.9.1

[PATCH v7] eal: map PCI memory resources after hugepages

[Anatoly Burakov]

Multi-process DPDK application must mmap hugepages and pci resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.

This patch makes EAL map PCI BARs right after the hugepages (instead of
location chosen by mmap) in virtual memory.

Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 19 +++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  9 ++++++++-
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 13 +++++++++++--
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 ++++++
 4 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..dae8739 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
 	return -1;
 }
 
+void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg *last = seg;
+	unsigned i = 0;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if (seg->addr > last->addr)
+			last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
 /* map a particular resource from a file */
 void *
 pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..5090bf1 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -48,6 +48,8 @@
 
 static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
 
+void *pci_map_addr = NULL;
+
 
 #define OFF_MAX              ((uint64_t)(off_t)-1)
 static int
@@ -371,10 +373,15 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				if (pci_map_addr == NULL)
+					pci_map_addr = pci_find_max_end_va();
+
+				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+
+				pci_map_addr = RTE_PTR_ADD(pci_map_addr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..fb6ee7a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -720,8 +720,17 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
-		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
-				reg.size);
+		if (internal_config.process_type == RTE_PROC_PRIMARY) {
+			if (pci_map_addr == NULL)
+				pci_map_addr = pci_find_max_end_va();
+
+			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+					reg.size);
+			pci_map_addr = RTE_PTR_ADD(pci_map_addr, reg.size);
+		} else {
+			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+					reg.size);
+		}
 
 		if (bar_addr == NULL) {
 			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
 TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
 extern struct mapped_pci_res_list *pci_res_list;
 
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
 void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
-- 
1.8.1.4

Re: [PATCH v7] eal: map PCI memory resources after hugepages

[Burakov, Anatoly]

Nak, there are issues with the patch. There is another patch already, but I'll submit it whenever Liang verifies it works with his setup.

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Anatoly Burakov
Sent: Monday, November 10, 2014 11:35 AM
To: dev-VfR2kkLFssw@public.gmane.org
Subject: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages

Multi-process DPDK application must mmap hugepages and pci resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.

This patch makes EAL map PCI BARs right after the hugepages (instead of
location chosen by mmap) in virtual memory.

Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 19 +++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  9 ++++++++-
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 13 +++++++++++--
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 ++++++
 4 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..dae8739 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
 	return -1;
 }
 
+void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg *last = seg;
+	unsigned i = 0;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if (seg->addr > last->addr)
+			last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
 /* map a particular resource from a file */
 void *
 pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..5090bf1 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -48,6 +48,8 @@
 
 static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
 
+void *pci_map_addr = NULL;
+
 
 #define OFF_MAX              ((uint64_t)(off_t)-1)
 static int
@@ -371,10 +373,15 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				if (pci_map_addr == NULL)
+					pci_map_addr = pci_find_max_end_va();
+
+				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+
+				pci_map_addr = RTE_PTR_ADD(pci_map_addr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..fb6ee7a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -720,8 +720,17 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
-		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
-				reg.size);
+		if (internal_config.process_type == RTE_PROC_PRIMARY) {
+			if (pci_map_addr == NULL)
+				pci_map_addr = pci_find_max_end_va();
+
+			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+					reg.size);
+			pci_map_addr = RTE_PTR_ADD(pci_map_addr, reg.size);
+		} else {
+			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+					reg.size);
+		}
 
 		if (bar_addr == NULL) {
 			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
 TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
 extern struct mapped_pci_res_list *pci_res_list;
 
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
 void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
-- 
1.8.1.4

Re: [PATCH v7] eal: map PCI memory resources after hugepages

[XU Liang]

I had finished some tests. The patch works fine. My tests are included :* single process  + uio + vfio * single process  + uio + vfio + base-virtaddr * multiple processes + uio + vfio * multiple processes + uio + vfio + base-virtaddr My unlucky multiple process application still got error without base-virtaddr when initial hugepages. See the attchments: primary.txt and secondary.txt.With base-virtaddr the patch worked, both hugepages and pci resources were mapped into base-virtaddr, My application is happy. See the attchments: base-virtaddr_primary.txt and  base-virtaddr_secondary.txt. ------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>Time:2014 Nov 10 (Mon) 21 : 34To:Burakov, Anatoly <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>, dev-VfR2kkLFssw@public.gmane.org <dev-VfR2kkLFssw@public.gmane.org>Subject:Re: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages
Nak, there are issues with the patch. There is another patch already, but I'll submit it whenever Liang verifies it works with his setup.

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Anatoly Burakov
Sent: Monday, November 10, 2014 11:35 AM
To: dev@dpdk.org
Subject: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages

Multi-process DPDK application must mmap hugepages and pci resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.

This patch makes EAL map PCI BARs right after the hugepages (instead of
location chosen by mmap) in virtual memory.

Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 19 +++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          |  9 ++++++++-
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 13 +++++++++++--
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 ++++++
 4 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..dae8739 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
 	return -1;
 }
 
+void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg *last = seg;
+	unsigned i = 0;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if (seg->addr > last->addr)
+			last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
 /* map a particular resource from a file */
 void *
 pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..5090bf1 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -48,6 +48,8 @@
 
 static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
 
+void *pci_map_addr = NULL;
+
 
 #define OFF_MAX              ((uint64_t)(off_t)-1)
 static int
@@ -371,10 +373,15 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				if (pci_map_addr == NULL)
+					pci_map_addr = pci_find_max_end_va();
+
+				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
 				if (mapaddr == NULL)
 					fail = 1;
+
+				pci_map_addr = RTE_PTR_ADD(pci_map_addr, maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..fb6ee7a 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -720,8 +720,17 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
-		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
-				reg.size);
+		if (internal_config.process_type == RTE_PROC_PRIMARY) {
+			if (pci_map_addr == NULL)
+				pci_map_addr = pci_find_max_end_va();
+
+			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+					reg.size);
+			pci_map_addr = RTE_PTR_ADD(pci_map_addr, reg.size);
+		} else {
+			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+					reg.size);
+		}
 
 		if (bar_addr == NULL) {
 			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
 TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
 extern struct mapped_pci_res_list *pci_res_list;
 
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
 void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
-- 
1.8.1.4

[PATCH v8] eal: map PCI memory resources after hugepages

[Anatoly Burakov]

Multi-process DPDK application must mmap hugepages and PCI resources
into the same virtual address space. By default the virtual addresses
are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary
process - for example, secondary process is linked with more libraries
than primary process, and the library occupies the same address space
that the primary process has requested for PCI mappings.

This patch makes EAL try and map PCI BARs right after the hugepages
(instead of location chosen by mmap) in virtual memory, so that PCI BARs
have less chance of ending up in random places in virtual memory.

Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 30 ++++++++++++++++------
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          | 13 ++++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 19 +++++++++++---
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 +++++
 4 files changed, 55 insertions(+), 13 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..79fbbb8 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
 	return -1;
 }
 
+void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg *last = seg;
+	unsigned i = 0;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if (seg->addr > last->addr)
+			last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len);
+}
+
+
 /* map a particular resource from a file */
 void *
 pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
@@ -106,21 +125,16 @@ pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
 	/* Map the PCI memory resource of device */
 	mapaddr = mmap(requested_addr, size, PROT_READ | PROT_WRITE,
 			MAP_SHARED, fd, offset);
-	if (mapaddr == MAP_FAILED ||
-			(requested_addr != NULL && mapaddr != requested_addr)) {
+	if (mapaddr == MAP_FAILED) {
 		RTE_LOG(ERR, EAL, "%s(): cannot mmap(%d, %p, 0x%lx, 0x%lx): %s (%p)\n",
 			__func__, fd, requested_addr,
 			(unsigned long)size, (unsigned long)offset,
 			strerror(errno), mapaddr);
-		goto fail;
+	} else {
+		RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
 	}
 
-	RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
-
 	return mapaddr;
-
-fail:
-	return NULL;
 }
 
 /* parse the "resource" sysfs file */
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e53f06b 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -35,6 +35,7 @@
 #include <fcntl.h>
 #include <dirent.h>
 #include <sys/stat.h>
+#include <sys/mman.h>
 
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -48,6 +49,8 @@
 
 static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
 
+void *pci_map_addr = NULL;
+
 
 #define OFF_MAX              ((uint64_t)(off_t)-1)
 static int
@@ -371,10 +374,16 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				/* try mapping somewhere close to the end of hugepages */
+				if (pci_map_addr == NULL)
+					pci_map_addr = pci_find_max_end_va();
+
+				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
-				if (mapaddr == NULL)
+				if (mapaddr == MAP_FAILED)
 					fail = 1;
+
+				pci_map_addr = RTE_PTR_ADD(mapaddr, (size_t) maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..c1246e8 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -37,6 +37,7 @@
 #include <sys/eventfd.h>
 #include <sys/socket.h>
 #include <sys/ioctl.h>
+#include <sys/mman.h>
 
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -720,10 +721,22 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
-		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
-				reg.size);
+		if (internal_config.process_type == RTE_PROC_PRIMARY) {
+			/* try mapping somewhere close to the end of hugepages */
+			if (pci_map_addr == NULL)
+				pci_map_addr = pci_find_max_end_va();
+
+			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+					reg.size);
+			pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg.size);
+		} else {
+			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+					reg.size);
+		}
 
-		if (bar_addr == NULL) {
+		if (bar_addr == MAP_FAILED ||
+				(internal_config.process_type == RTE_PROC_SECONDARY &&
+						bar_addr != maps[i].addr)) {
 			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
 					strerror(errno));
 			close(vfio_dev_fd);
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {
 TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
 extern struct mapped_pci_res_list *pci_res_list;
 
+/*
+ * Helper function to map PCI resources right after hugepages in virtual memory
+ */
+extern void *pci_map_addr;
+void *pci_find_max_end_va(void);
+
 void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
-- 
1.8.1.4

Re: [PATCH v8] eal: map PCI memory resources after hugepages

[Burakov, Anatoly]

Hi Thomas and all

Are there any objections to this patch? If there are no objections to it, could someone perhaps ack it?

Thanks,
Anatoly

-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Anatoly Burakov
Sent: Tuesday, November 11, 2014 10:09 AM
To: dev-VfR2kkLFssw@public.gmane.org
Subject: [dpdk-dev] [PATCH v8] eal: map PCI memory resources after hugepages

Multi-process DPDK application must mmap hugepages and PCI resources into the same virtual address space. By default the virtual addresses are chosen by the primary process automatically when calling the mmap.
But sometimes the chosen virtual addresses aren't usable in secondary process - for example, secondary process is linked with more libraries than primary process, and the library occupies the same address space that the primary process has requested for PCI mappings.

This patch makes EAL try and map PCI BARs right after the hugepages (instead of location chosen by mmap) in virtual memory, so that PCI BARs have less chance of ending up in random places in virtual memory.

Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci.c              | 30 ++++++++++++++++------
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c          | 13 ++++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 19 +++++++++++---
 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 +++++
 4 files changed, 55 insertions(+), 13 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 5fe3961..79fbbb8 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -97,6 +97,25 @@ error:
 	return -1;
 }
 
+void *
+pci_find_max_end_va(void)
+{
+	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
+	const struct rte_memseg *last = seg;
+	unsigned i = 0;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
+		if (seg->addr == NULL)
+			break;
+
+		if (seg->addr > last->addr)
+			last = seg;
+
+	}
+	return RTE_PTR_ADD(last->addr, last->len); }
+
+
 /* map a particular resource from a file */  void *  pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size) @@ -106,21 +125,16 @@ pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
 	/* Map the PCI memory resource of device */
 	mapaddr = mmap(requested_addr, size, PROT_READ | PROT_WRITE,
 			MAP_SHARED, fd, offset);
-	if (mapaddr == MAP_FAILED ||
-			(requested_addr != NULL && mapaddr != requested_addr)) {
+	if (mapaddr == MAP_FAILED) {
 		RTE_LOG(ERR, EAL, "%s(): cannot mmap(%d, %p, 0x%lx, 0x%lx): %s (%p)\n",
 			__func__, fd, requested_addr,
 			(unsigned long)size, (unsigned long)offset,
 			strerror(errno), mapaddr);
-		goto fail;
+	} else {
+		RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
 	}
 
-	RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
-
 	return mapaddr;
-
-fail:
-	return NULL;
 }
 
 /* parse the "resource" sysfs file */
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..e53f06b 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -35,6 +35,7 @@
 #include <fcntl.h>
 #include <dirent.h>
 #include <sys/stat.h>
+#include <sys/mman.h>
 
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -48,6 +49,8 @@
 
 static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
 
+void *pci_map_addr = NULL;
+
 
 #define OFF_MAX              ((uint64_t)(off_t)-1)
 static int
@@ -371,10 +374,16 @@ pci_uio_map_resource(struct rte_pci_device *dev)
 			if (maps[j].addr != NULL)
 				fail = 1;
 			else {
-				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+				/* try mapping somewhere close to the end of hugepages */
+				if (pci_map_addr == NULL)
+					pci_map_addr = pci_find_max_end_va();
+
+				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
 						(size_t)maps[j].size);
-				if (mapaddr == NULL)
+				if (mapaddr == MAP_FAILED)
 					fail = 1;
+
+				pci_map_addr = RTE_PTR_ADD(mapaddr, (size_t) maps[j].size);
 			}
 
 			if (fail) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c776ddc..c1246e8 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -37,6 +37,7 @@
 #include <sys/eventfd.h>
 #include <sys/socket.h>
 #include <sys/ioctl.h>
+#include <sys/mman.h>
 
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -720,10 +721,22 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		if (i == msix_bar)
 			continue;
 
-		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
-				reg.size);
+		if (internal_config.process_type == RTE_PROC_PRIMARY) {
+			/* try mapping somewhere close to the end of hugepages */
+			if (pci_map_addr == NULL)
+				pci_map_addr = pci_find_max_end_va();
+
+			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
+					reg.size);
+			pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg.size);
+		} else {
+			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
+					reg.size);
+		}
 
-		if (bar_addr == NULL) {
+		if (bar_addr == MAP_FAILED ||
+				(internal_config.process_type == RTE_PROC_SECONDARY &&
+						bar_addr != maps[i].addr)) {
 			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
 					strerror(errno));
 			close(vfio_dev_fd);
diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
index d758bee..1070eb8 100644
--- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
@@ -59,6 +59,12 @@ struct mapped_pci_resource {  TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);  extern struct mapped_pci_res_list *pci_res_list;
 
+/*
+ * Helper function to map PCI resources right after hugepages in 
+virtual memory  */ extern void *pci_map_addr; void 
+*pci_find_max_end_va(void);
+
 void *pci_map_resource(void *requested_addr, int fd, off_t offset,
 		size_t size);
 
--
1.8.1.4

Re: [PATCH v8] eal: map PCI memory resources after hugepages

[Bruce Richardson]

On Thu, Nov 13, 2014 at 11:34:22AM +0000, Burakov, Anatoly wrote:
> Hi Thomas and all
> 
> Are there any objections to this patch? If there are no objections to it, could someone perhaps ack it?
> 

No objections, just a quick request for clarification below.

/Bruce

> Thanks,
> Anatoly
> 
> -----Original Message-----
> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Anatoly Burakov
> Sent: Tuesday, November 11, 2014 10:09 AM
> To: dev-VfR2kkLFssw@public.gmane.org
> Subject: [dpdk-dev] [PATCH v8] eal: map PCI memory resources after hugepages
> 
> Multi-process DPDK application must mmap hugepages and PCI resources into the same virtual address space. By default the virtual addresses are chosen by the primary process automatically when calling the mmap.
> But sometimes the chosen virtual addresses aren't usable in secondary process - for example, secondary process is linked with more libraries than primary process, and the library occupies the same address space that the primary process has requested for PCI mappings.
> 
> This patch makes EAL try and map PCI BARs right after the hugepages (instead of location chosen by mmap) in virtual memory, so that PCI BARs have less chance of ending up in random places in virtual memory.
> 
> Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
> Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>  lib/librte_eal/linuxapp/eal/eal_pci.c              | 30 ++++++++++++++++------
>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c          | 13 ++++++++--
>  lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 19 +++++++++++---
>  lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 +++++
>  4 files changed, 55 insertions(+), 13 deletions(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
> index 5fe3961..79fbbb8 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
> @@ -97,6 +97,25 @@ error:
>  	return -1;
>  }
>  
> +void *
> +pci_find_max_end_va(void)
> +{
> +	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
> +	const struct rte_memseg *last = seg;
> +	unsigned i = 0;
> +
> +	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
> +		if (seg->addr == NULL)
> +			break;
> +
> +		if (seg->addr > last->addr)
> +			last = seg;
> +
> +	}
> +	return RTE_PTR_ADD(last->addr, last->len); }
> +
> +
>  /* map a particular resource from a file */  void *  pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size) @@ -106,21 +125,16 @@ pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
>  	/* Map the PCI memory resource of device */
>  	mapaddr = mmap(requested_addr, size, PROT_READ | PROT_WRITE,
>  			MAP_SHARED, fd, offset);
> -	if (mapaddr == MAP_FAILED ||
> -			(requested_addr != NULL && mapaddr != requested_addr)) {

Why has this check been removed from here. I assume it is replaced by a new
check in secondary processes that I see added below, but perhaps you could explain
the reason for the change?

> +	if (mapaddr == MAP_FAILED) {
>  		RTE_LOG(ERR, EAL, "%s(): cannot mmap(%d, %p, 0x%lx, 0x%lx): %s (%p)\n",
>  			__func__, fd, requested_addr,
>  			(unsigned long)size, (unsigned long)offset,
>  			strerror(errno), mapaddr);
> -		goto fail;
> +	} else {
> +		RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
>  	}
>  
> -	RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
> -
>  	return mapaddr;
> -
> -fail:
> -	return NULL;
>  }
>  
>  /* parse the "resource" sysfs file */
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> index 7e62266..e53f06b 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> @@ -35,6 +35,7 @@
>  #include <fcntl.h>
>  #include <dirent.h>
>  #include <sys/stat.h>
> +#include <sys/mman.h>
>  
>  #include <rte_log.h>
>  #include <rte_pci.h>
> @@ -48,6 +49,8 @@
>  
>  static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
>  
> +void *pci_map_addr = NULL;
> +
>  
>  #define OFF_MAX              ((uint64_t)(off_t)-1)
>  static int
> @@ -371,10 +374,16 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>  			if (maps[j].addr != NULL)
>  				fail = 1;
>  			else {
> -				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
> +				/* try mapping somewhere close to the end of hugepages */
> +				if (pci_map_addr == NULL)
> +					pci_map_addr = pci_find_max_end_va();
> +
> +				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
>  						(size_t)maps[j].size);
> -				if (mapaddr == NULL)
> +				if (mapaddr == MAP_FAILED)
>  					fail = 1;
> +
> +				pci_map_addr = RTE_PTR_ADD(mapaddr, (size_t) maps[j].size);
>  			}
>  
>  			if (fail) {
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> index c776ddc..c1246e8 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> @@ -37,6 +37,7 @@
>  #include <sys/eventfd.h>
>  #include <sys/socket.h>
>  #include <sys/ioctl.h>
> +#include <sys/mman.h>
>  
>  #include <rte_log.h>
>  #include <rte_pci.h>
> @@ -720,10 +721,22 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
>  		if (i == msix_bar)
>  			continue;
>  
> -		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
> -				reg.size);
> +		if (internal_config.process_type == RTE_PROC_PRIMARY) {
> +			/* try mapping somewhere close to the end of hugepages */
> +			if (pci_map_addr == NULL)
> +				pci_map_addr = pci_find_max_end_va();
> +
> +			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
> +					reg.size);
> +			pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg.size);
> +		} else {
> +			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
> +					reg.size);
> +		}
>  
> -		if (bar_addr == NULL) {
> +		if (bar_addr == MAP_FAILED ||
> +				(internal_config.process_type == RTE_PROC_SECONDARY &&
> +						bar_addr != maps[i].addr)) {
>  			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
>  					strerror(errno));
>  			close(vfio_dev_fd);
> diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> index d758bee..1070eb8 100644
> --- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> +++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> @@ -59,6 +59,12 @@ struct mapped_pci_resource {  TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);  extern struct mapped_pci_res_list *pci_res_list;
>  
> +/*
> + * Helper function to map PCI resources right after hugepages in 
> +virtual memory  */ extern void *pci_map_addr; void 
> +*pci_find_max_end_va(void);
> +
>  void *pci_map_resource(void *requested_addr, int fd, off_t offset,
>  		size_t size);
>  
> --
> 1.8.1.4
> 

Re: [PATCH v8] eal: map PCI memory resources after hugepages

[Burakov, Anatoly]

> Why has this check been removed from here. I assume it is replaced by a
> new check in secondary processes that I see added below, but perhaps you
> could explain the reason for the change?

Sure. The reason behind that change is that we can't expect that we will get a mapping at exact same address (for whatever reasons, i.e. something else is mapped there, alignment, etc.), and in primary process, it's not an error. In other words, removing this check makes it a "best-effort" type mechanism, rather than mandates PCI resources to be mapped exactly after hugepages, exactly one after another. "Wrong" mapping will still result in failure in secondary processes, and we still are risking mapping something somewhere the secondary process can't map, but that probability is decreased because we're now asking EAL to map PCI resources closer to where we most likely have some free virtual space (as evidenced by tests being done by the original submitter of the patch).

Hope that makes sense.

Thanks,
Anatoly

Re: [PATCH v8] eal: map PCI memory resources after hugepages

[Bruce Richardson]

On Tue, Nov 11, 2014 at 10:09:25AM +0000, Anatoly Burakov wrote:
> Multi-process DPDK application must mmap hugepages and PCI resources
> into the same virtual address space. By default the virtual addresses
> are chosen by the primary process automatically when calling the mmap.
> But sometimes the chosen virtual addresses aren't usable in secondary
> process - for example, secondary process is linked with more libraries
> than primary process, and the library occupies the same address space
> that the primary process has requested for PCI mappings.
> 
> This patch makes EAL try and map PCI BARs right after the hugepages
> (instead of location chosen by mmap) in virtual memory, so that PCI BARs
> have less chance of ending up in random places in virtual memory.
> 
> Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
> Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Acked-by: Bruce Richardson <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

> ---
>  lib/librte_eal/linuxapp/eal/eal_pci.c              | 30 ++++++++++++++++------
>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c          | 13 ++++++++--
>  lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         | 19 +++++++++++---
>  lib/librte_eal/linuxapp/eal/include/eal_pci_init.h |  6 +++++
>  4 files changed, 55 insertions(+), 13 deletions(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
> index 5fe3961..79fbbb8 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
> @@ -97,6 +97,25 @@ error:
>  	return -1;
>  }
>  
> +void *
> +pci_find_max_end_va(void)
> +{
> +	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
> +	const struct rte_memseg *last = seg;
> +	unsigned i = 0;
> +
> +	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
> +		if (seg->addr == NULL)
> +			break;
> +
> +		if (seg->addr > last->addr)
> +			last = seg;
> +
> +	}
> +	return RTE_PTR_ADD(last->addr, last->len);
> +}
> +
> +
>  /* map a particular resource from a file */
>  void *
>  pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
> @@ -106,21 +125,16 @@ pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size)
>  	/* Map the PCI memory resource of device */
>  	mapaddr = mmap(requested_addr, size, PROT_READ | PROT_WRITE,
>  			MAP_SHARED, fd, offset);
> -	if (mapaddr == MAP_FAILED ||
> -			(requested_addr != NULL && mapaddr != requested_addr)) {
> +	if (mapaddr == MAP_FAILED) {
>  		RTE_LOG(ERR, EAL, "%s(): cannot mmap(%d, %p, 0x%lx, 0x%lx): %s (%p)\n",
>  			__func__, fd, requested_addr,
>  			(unsigned long)size, (unsigned long)offset,
>  			strerror(errno), mapaddr);
> -		goto fail;
> +	} else {
> +		RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
>  	}
>  
> -	RTE_LOG(DEBUG, EAL, "  PCI memory mapped at %p\n", mapaddr);
> -
>  	return mapaddr;
> -
> -fail:
> -	return NULL;
>  }
>  
>  /* parse the "resource" sysfs file */
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> index 7e62266..e53f06b 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> @@ -35,6 +35,7 @@
>  #include <fcntl.h>
>  #include <dirent.h>
>  #include <sys/stat.h>
> +#include <sys/mman.h>
>  
>  #include <rte_log.h>
>  #include <rte_pci.h>
> @@ -48,6 +49,8 @@
>  
>  static int pci_parse_sysfs_value(const char *filename, uint64_t *val);
>  
> +void *pci_map_addr = NULL;
> +
>  
>  #define OFF_MAX              ((uint64_t)(off_t)-1)
>  static int
> @@ -371,10 +374,16 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>  			if (maps[j].addr != NULL)
>  				fail = 1;
>  			else {
> -				mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
> +				/* try mapping somewhere close to the end of hugepages */
> +				if (pci_map_addr == NULL)
> +					pci_map_addr = pci_find_max_end_va();
> +
> +				mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset,
>  						(size_t)maps[j].size);
> -				if (mapaddr == NULL)
> +				if (mapaddr == MAP_FAILED)
>  					fail = 1;
> +
> +				pci_map_addr = RTE_PTR_ADD(mapaddr, (size_t) maps[j].size);
>  			}
>  
>  			if (fail) {
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> index c776ddc..c1246e8 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> @@ -37,6 +37,7 @@
>  #include <sys/eventfd.h>
>  #include <sys/socket.h>
>  #include <sys/ioctl.h>
> +#include <sys/mman.h>
>  
>  #include <rte_log.h>
>  #include <rte_pci.h>
> @@ -720,10 +721,22 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
>  		if (i == msix_bar)
>  			continue;
>  
> -		bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
> -				reg.size);
> +		if (internal_config.process_type == RTE_PROC_PRIMARY) {
> +			/* try mapping somewhere close to the end of hugepages */
> +			if (pci_map_addr == NULL)
> +				pci_map_addr = pci_find_max_end_va();
> +
> +			bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset,
> +					reg.size);
> +			pci_map_addr = RTE_PTR_ADD(bar_addr, (size_t) reg.size);
> +		} else {
> +			bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset,
> +					reg.size);
> +		}
>  
> -		if (bar_addr == NULL) {
> +		if (bar_addr == MAP_FAILED ||
> +				(internal_config.process_type == RTE_PROC_SECONDARY &&
> +						bar_addr != maps[i].addr)) {
>  			RTE_LOG(ERR, EAL, "  %s mapping BAR%i failed: %s\n", pci_addr, i,
>  					strerror(errno));
>  			close(vfio_dev_fd);
> diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> index d758bee..1070eb8 100644
> --- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> +++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
> @@ -59,6 +59,12 @@ struct mapped_pci_resource {
>  TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource);
>  extern struct mapped_pci_res_list *pci_res_list;
>  
> +/*
> + * Helper function to map PCI resources right after hugepages in virtual memory
> + */
> +extern void *pci_map_addr;
> +void *pci_find_max_end_va(void);
> +
>  void *pci_map_resource(void *requested_addr, int fd, off_t offset,
>  		size_t size);
>  
> -- 
> 1.8.1.4
> 

Re: [PATCH v8] eal: map PCI memory resources after hugepages

[Thomas Monjalon]

> > Multi-process DPDK application must mmap hugepages and PCI resources
> > into the same virtual address space. By default the virtual addresses
> > are chosen by the primary process automatically when calling the mmap.
> > But sometimes the chosen virtual addresses aren't usable in secondary
> > process - for example, secondary process is linked with more libraries
> > than primary process, and the library occupies the same address space
> > that the primary process has requested for PCI mappings.
> > 
> > This patch makes EAL try and map PCI BARs right after the hugepages
> > (instead of location chosen by mmap) in virtual memory, so that PCI BARs
> > have less chance of ending up in random places in virtual memory.
> > 
> > Signed-off-by: Liang Xu <liang.xu-uqgZSDCUmcY3TOWNJCPdyQ@public.gmane.org>
> > Signed-off-by: Anatoly Burakov <anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> 
> Acked-by: Bruce Richardson <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Applied

Thanks
-- 
Thomas

Re: 答复：答复： [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Qiu, Michael]

On 11/6/2014 12:12 AM, XU Liang wrote:
> I have a multiple processes application. When start a secondary process, we got error message "EAL: pci_map_resource(): cannot mmap(11, 0x7ffff7fba000, 0x20000, 0x0): Bad file descriptor (0x7ffff7fb9000)".The secondary process link difference shared libraries, so the address 0x7ffff7fba000 is used.

This is a known issue, and still not be solved yet.  The root cause is
exactly clear, that should be try to map an address have already used in
new process.

BTW, you should learn how to make a patch, like commit log,
signed-off-by, etc.

Thanks,
Michael

> ------------------------------------------------------------------发件人：Burakov, Anatoly <anatoly.burakov@intel.com>发送时间：2014年11月5日(星期三) 23:59收件人：徐亮 <liang.xu@cinfotech.cn>，dev@dpdk.org <dev@dpdk.org>主　题：RE: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the	base_virtaddr is configured.
>
>
>
> font-family: MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Cambria Math;panose-1: 2 4 5 3 5 4 6 3 2 4;font-family: Calibri;panose-1: 2 15 5 2 2 2 4 3 2 4;font-family: Tahoma;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Microsoft JhengHei;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@Microsoft JhengHei;panose-1: 2 11 6 4 3 5 4 4 2 4;p.MsoNormal, li.MsoNormal, div.MsoNormal {margin: 0.0cm;margin-bottom: 1.0E-4pt;font-size: 12.0pt;font-family: Times New Roman , serif;}
> a:link, span.MsoHyperlink {mso-style-priority: 99;color: #0563c1;text-decoration: underline;}
> a:visited, span.MsoHyperlinkFollowed {mso-style-priority: 99;color: #954f72;text-decoration: underline;}
> span.EmailStyle17 {mso-style-type: personal-reply;font-family: Calibri , sans-serif;color: #1f497d;}
> *.MsoChpDefault {mso-style-type: export-only;font-family: Calibri , sans-serif;}
> size: 612.0pt 792.0pt;margin: 72.0pt 72.0pt 72.0pt 72.0pt;div.WordSection1 {page: WordSection1;}
>
>
>
>
> Hi Liang
>  
> Yes it is a problem. Even if it was carefully selected by user, nothing stops the DPDK application from mapping something into where you’re trying to map your
>  UIO devices. Plus, this changes the default behavior where a wrong base-virtaddr leads to a failure to initialize, rather than simply using a different address (remember that pci_map_resource fails if it cannot map the resource at the exact address you requested).
>  
> A very crude way of finding out where hugepages end would be to walk the hugepage memory (walk through memsegs and note the maximum start addr + length of that
>  memseg).
>  
> Could you perhaps explain what is the problem that you’re trying to solve with this? I can’t think of a situation where the location of UIO maps would matter,
>  to be honest.
>  
> Thanks,
> Anatoly
>  
> From: XU Liang [mailto:liang.xu@cinfotech.cn]
>
>
> Sent: Wednesday, November 5, 2014 3:49 PM
>
> To: Burakov, Anatoly; dev@dpdk.org
>
> Subject: 答复：[dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.
>  
>
>
> I think the base_virtadd will be carefully selected by user when they need it. So maybe it's not a real problem.  :>
>
>
>  
>
>
> The real reason is I can't find a easy way to get the end address of hugepages. Can you give me some suggestions ?
>
>
>
> ------------------------------------------------------------------
>
>
> 发件人：Burakov, Anatoly <anatoly.burakov@intel.com>
>
>
> 发送时间：2014年11月5日(星期三)
>  23:10
>
>
> 收件人：徐亮 <liang.xu@cinfotech.cn>，dev@dpdk.org
>  <dev@dpdk.org>
>
>
> 主　题：RE:
>  [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.
>
>
>  
>
> I have a slight problems with this patch.
>
>
>
> The base_virtaddr doesn't necessarily correspond to an address that everything gets mapped to. It's a "hint" of sorts, that may or may not be taken into account by mmap. Therefore we can't simply assume that if we requested a base-virtaddr, everything will
>  get mapped at exactly that address. We also can't assume that hugepages will be ordered one after the other and occupy neatly all the contiguous virtual memory between base_virtaddr and base_virtaddr + internal_config.memory - there may be holes, for whatever
>  reasons.
>
>
>
> Also, 
>
>
>
> Thanks,
>
> Anatoly
>
>
>
> -----Original Message-----
>
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of lxu
>
> Sent: Wednesday, November 5, 2014 1:25 PM
>
> To: dev@dpdk.org
>
> Subject: [dpdk-dev] [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.
>
>
>
> ---
>
> lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++-
>
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
>
>
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
>
> index 7e62266..bc7ed3a 100644
>
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
>
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
>
> @@ -289,6 +289,11 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>
> struct rte_pci_addr *loc = &dev->addr;
>
> struct mapped_pci_resource *uio_res;
>
> struct pci_map *maps;
>
> + static void * requested_addr = NULL;
>
> + if (internal_config.base_virtaddr && NULL == requested_addr) {
>
> + requested_addr = (uint8_t *) internal_config.base_virtaddr 
>
> + + internal_config.memory;
>
> + }
>
>
>
> dev->intr_handle.fd = -1;
>
> dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +376,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
>
> if (maps[j].addr != NULL)
>
> fail = 1;
>
> else {
>
> - mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
>
> + mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
>
> (size_t)maps[j].size);
>
> if (mapaddr == NULL)
>
> fail = 1;
>
> + else if (NULL != requested_addr)
>
> + requested_addr = (uint8_t *)mapaddr + maps[j].size;
>
> }
>
>
>
> if (fail) {
>
> --
>
> 1.9.1
>
>
>
>

Re: 答复：答复： [PATCH] eal: map uio resources after hugepages when the base_virtaddr is configured.

[Burakov, Anatoly]

> This is a known issue, and still not be solved yet.  The root cause is exactly
> clear, that should be try to map an address have already used in new
> process.
> 
> BTW, you should learn how to make a patch, like commit log, signed-off-by,
> etc.

Hi Michael,

As far as I know, the patch that fixes this issue was integrated yesterday (thanks Thomas!), and Liang Xu verified it to be working.

Thanks,
Anatoly