All of lore.kernel.org
 help / color / mirror / Atom feed
From: bugzilla@dpdk.org
To: dev@dpdk.org
Subject: [Bug 95] eal/memory: testpmd failed to start with --socket-mem > 16384 with 2MB huge pages and with memory hotplug
Date: Tue, 16 Oct 2018 16:38:39 +0000	[thread overview]
Message-ID: <bug-95-3@http.bugs.dpdk.org/> (raw)

https://bugs.dpdk.org/show_bug.cgi?id=95

            Bug ID: 95
           Summary: eal/memory: testpmd failed to start with --socket-mem
                    >  16384 with 2MB huge pages and with memory hotplug
           Product: DPDK
           Version: 18.11
          Hardware: x86
                OS: Linux
            Status: CONFIRMED
          Severity: normal
          Priority: Normal
         Component: core
          Assignee: dev@dpdk.org
          Reporter: maxime.leroy@6wind.com
  Target Milestone: ---

With the lastest version on dpdk.org (i.e. commit id
94d31549c380ee8b9108c4621d6fad8d624a395e)

First configure 2Mb hugepages:
mkdir -p /mnt/huge
mount -t hugetlbfs nodev /mnt/huge
echo  12598 >
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

Start test pmd with 25Gb memory:
root@ubuntu1604:~/dpdk# ./build/app/testpmd -c7 --socket-mem 25196   -- -i
--nb-cores=2 - --total-num-mbufs=2048
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
PANIC in main():
Cannot init EAL
5: [./build/app/testpmd(_start+0x29) [0x48b509]]
4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f28f9adb830]]
3: [./build/app/testpmd(main+0xb0e) [0x4825ae]]
2: [./build/app/testpmd(__rte_panic+0xc3) [0x475920]]
1: [./build/app/testpmd(rte_dump_stack+0x2b) [0x5ba22b]]
Aborted (core dumped)

It failed.

With --legacy-mem option, it's work fine:
./build/app/testpmd -c7 --socket-mem 25196 --legacy-mem  -- -i --nb-cores=2 -
--total-num-mbufs=2048
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
..
Interactive-mode selected
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=2048, size=2176,
socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
testpmd> 
testpmd> dump_physmem
Segment 0-0: IOVA:0x2400000, len:2097152, virt:0x7f3491a00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:11
Segment 0-1: IOVA:0x2600000, len:2097152, virt:0x7f3491c00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:12
Segment 0-2: IOVA:0x2800000, len:2097152, virt:0x7f3491e00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:13
Segment 0-3: IOVA:0x2a00000, len:2097152, virt:0x7f3492000000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:14
Segment 0-4: IOVA:0x2c00000, len:2097152, virt:0x7f3492200000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:15
Segment 0-5: IOVA:0x2e00000, len:2097152, virt:0x7f3492400000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:16
Segment 0-6: IOVA:0x3000000, len:2097152, virt:0x7f3492600000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:17
Segment 0-7: IOVA:0x3200000, len:2097152, virt:0x7f3492800000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:18
Segment 0-8: IOVA:0x3400000, len:2097152, virt:0x7f3492a00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:19
Segment 0-9: IOVA:0x3600000, len:2097152, virt:0x7f3492c00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:20
Segment 0-10: IOVA:0x3800000, len:2097152, virt:0x7f3492e00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:21
Segment 0-11: IOVA:0x3a00000, len:2097152, virt:0x7f3493000000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:22
Segment 0-12: IOVA:0x3c00000, len:2097152, virt:0x7f3493200000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:23
Segment 0-13: IOVA:0x3e00000, len:2097152, virt:0x7f3493400000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:24
Segment 0-14: IOVA:0x4000000, len:2097152, virt:0x7f3493600000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:25
...
Segment 0-8188: IOVA:0x539200000, len:2097152, virt:0x7f3891200000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:7617
Segment 0-8189: IOVA:0x539400000, len:2097152, virt:0x7f3891400000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:7618
Segment 0-8190: IOVA:0x539600000, len:2097152, virt:0x7f3891600000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:7619
Segment 1-0: IOVA:0x514800000, len:2097152, virt:0x7f3091800000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:7335
Segment 1-1: IOVA:0x514a00000, len:2097152, virt:0x7f3091a00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:7336
Segment 1-2: IOVA:0x514c00000, len:2097152, virt:0x7f3091c00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:7337
Segment 1-3: IOVA:0x514e00000, len:2097152, virt:0x7f3091e00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:7338
Segment 1-4: IOVA:0x515000000, len:2097152, virt:0x7f3092000000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:7339
...
Segment 1-5517: IOVA:0x7f3c00000, len:2097152, virt:0x7f3343200000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12602
Segment 1-5519: IOVA:0x7f4c00000, len:2097152, virt:0x7f3343600000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12603
Segment 1-5520: IOVA:0x7f4e00000, len:2097152, virt:0x7f3343800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12604
Segment 1-5522: IOVA:0x7f7000000, len:2097152, virt:0x7f3343c00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12605
Segment 1-5524: IOVA:0x7f7a00000, len:2097152, virt:0x7f3344000000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12606
Segment 1-5526: IOVA:0x7fc600000, len:2097152, virt:0x7f3344400000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12607
Segment 1-5528: IOVA:0x800c00000, len:2097152, virt:0x7f3344800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12608

We have 2 segments. The first one has 8192 pages and the second one 5529
segment.

If we increaze the MAX_MEMSEG_LIST:
--- a/config/common_base
+++ b/config/common_base
@@ -65,7 +65,7 @@ CONFIG_RTE_MAX_HEAPS=32
 CONFIG_RTE_MAX_MEMSEG_LISTS=64
 # each memseg list will be limited to either RTE_MAX_MEMSEG_PER_LIST pages
 # or RTE_MAX_MEM_MB_PER_LIST megabytes worth of memory, whichever is smaller
-CONFIG_RTE_MAX_MEMSEG_PER_LIST=8192
+CONFIG_RTE_MAX_MEMSEG_PER_LIST=16384

./build/app/testpmd -c7 --socket-mem 25196   -- -i --nb-cores=2 -
--total-num-mbufs=2048 
Interactive-mode selected
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=2048, size=2176,
socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
testpmd> 

It's working fine.


testpmd> dump_physmem
Segment 0-0: IOVA:0x2400000, len:2097152, virt:0x7f3491a00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:11
Segment 0-1: IOVA:0x2600000, len:2097152, virt:0x7f3491c00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:12
Segment 0-2: IOVA:0x2800000, len:2097152, virt:0x7f3491e00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:13
Segment 0-3: IOVA:0x2a00000, len:2097152, virt:0x7f3492000000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:14
Segment 0-4: IOVA:0x2c00000, len:2097152, virt:0x7f3492200000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:15
Segment 0-5: IOVA:0x2e00000, len:2097152, virt:0x7f3492400000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:16
Segment 0-6: IOVA:0x3000000, len:2097152, virt:0x7f3492600000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:17
Segment 0-7: IOVA:0x3200000, len:2097152, virt:0x7f3492800000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:18
Segment 0-8: IOVA:0x3400000, len:2097152, virt:0x7f3492a00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:19
Segment 0-9: IOVA:0x3600000, len:2097152, virt:0x7f3492c00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:20
Segment 0-10: IOVA:0x3800000, len:2097152, virt:0x7f3492e00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:21
Segment 0-11: IOVA:0x3a00000, len:2097152, virt:0x7f3493000000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:22
Segment 0-12: IOVA:0x3c00000, len:2097152, virt:0x7f3493200000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:23
Segment 0-13: IOVA:0x3e00000, len:2097152, virt:0x7f3493400000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:24
Segment 0-14: IOVA:0x4000000, len:2097152, virt:0x7f3493600000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:25
Segment 0-15: IOVA:0x4200000, len:2097152, virt:0x7f3493800000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:26
..
Segment 0-12588: IOVA:0x4cd000000, len:2097152, virt:0x7fb892c00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12599
Segment 0-12589: IOVA:0x4d2e00000, len:2097152, virt:0x7fb892e00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12600
Segment 0-12590: IOVA:0x73ba00000, len:2097152, virt:0x7fb893000000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12601
Segment 0-12591: IOVA:0x4d4a00000, len:2097152, virt:0x7fb893200000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12602
Segment 0-12592: IOVA:0xc4600000, len:2097152, virt:0x7fb893400000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12603
Segment 0-12593: IOVA:0x514400000, len:2097152, virt:0x7fb893600000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12604
Segment 0-12594: IOVA:0x7e3000000, len:2097152, virt:0x7fb893800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12605
Segment 0-12595: IOVA:0x4f9800000, len:2097152, virt:0x7fb893a00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12606
Segment 0-12596: IOVA:0x294400000, len:2097152, virt:0x7fb893c00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12607
Segment 0-12597: IOVA:0x4d5000000, len:2097152, virt:0x7fb893e00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12608

But we have only one segment in this case.

Question
--------

1) It's normal that we don't support anymore to allocate more that 16GB with
hotplug memory and default config value ? It was working with previous dpdk
version.
2) Should we increase the default value of CONFIG_RTE_MAX_MEMSEG_PER_LIST ?

Note: This http://patches.dpdk.org/patch/46112/ patch doesn't fix the issue.

-- 
You are receiving this mail because:
You are the assignee for the bug.

                 reply	other threads:[~2018-10-16 16:38 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=bug-95-3@http.bugs.dpdk.org/ \
    --to=bugzilla@dpdk.org \
    --cc=dev@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.