Problem with running the load balancer example application

Problem with running the load balancer example application

[Haowei Yuan]

Hi folks,

I am new to DPDK and have been trying to run the load balancer example
application on a machine with two NUMA nodes. Somehow the program
cannot be launched correctly if I add the "--no-huge" option to the
command. I am wondering if someone had seen similar problems, or maybe
I did something wrong.

The command I used was "./load_balancer -c 0x5 -n 4 --no-huge -- --rx
"(0,0,0),(1,0,0)" --tx "(0,0),(1,0)" --w "2" --lpm "192.168.0.0/1=>1;"
--pos-lb 29"

When hugepage memory was used, everything worked fine. But when I
added the "--no-huge" option, I got the warning message of "Master
core has no memory on local socket", and eventually the program quit
as mbuf pool could not be created on socket 0. The following is the
detailed error message.

-----------------------------------------------------------------------
EAL: WARNING: Master core has no memory on local socket!
EAL: Master core 0 is ready (tid=781f4800)
EAL: Core 2 is ready (tid=72df2700)
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
EAL:   PCI memory mapped at 0x7f57723f2000
EAL:   PCI memory mapped at 0x7f57723ee000
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
EAL:   PCI memory mapped at 0x7f57721ee000
EAL:   PCI memory mapped at 0x7f57721ea000
EAL: PCI device 0000:07:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   0000:07:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:07:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   0000:07:00.1 not managed by UIO driver, skipping
Creating the mbuf pool for socket 0 ...
PANIC in app_init_mbuf_pools():
Cannot create mbuf pool on socket 0
-----------------------------------------------------------------------


Then I added one line of code to the eal.c file to print the socket
id, the edited code and the output are:
Code:
-----------------------------------------------------------------------
static void
eal_check_mem_on_local_socket(void)
{
        const struct rte_memseg *ms;
        int i, socket_id;

        socket_id = rte_lcore_to_socket_id(rte_config.master_lcore);

        ms = rte_eal_get_physmem_layout();

        for (i = 0; i < RTE_MAX_MEMSEG; i++) {
               printf("socket_id = %d, len = %lu\n", ms[i].socket_id,
ms[i].len); XXXXX Added line of code
               if (ms[i].socket_id == socket_id &&
                                ms[i].len > 0)
                        return;
        }

        RTE_LOG(WARNING, EAL, "WARNING: Master core has no "
                        "memory on local socket!\n");
}

Output:
socket_id = -1, len = 67108864
socket_id = 0, len = 0
socket_id = 0, len = 0
socket_id = 0, len = 0
socket_id = 0, len = 0
......
-----------------------------------------------------------------------

I think the first socket_id of -1 is causing the problem as Core 0 is
on socket 0....
Has someone seem similar problems, or is there anything I did incorrect?

Thanks,
Haowei

Re: Problem with running the load balancer example application

[Bruce Richardson]

On Fri, Dec 19, 2014 at 01:43:26AM -0600, Haowei Yuan wrote:
> Hi folks,
> 
> I am new to DPDK and have been trying to run the load balancer example
> application on a machine with two NUMA nodes. Somehow the program
> cannot be launched correctly if I add the "--no-huge" option to the
> command. I am wondering if someone had seen similar problems, or maybe
> I did something wrong.
> 

Hi,

Can I ask why you want to run the app with the -no-huge flag. That flag is
primarily for testing only, and is not designed for use in a real-world 
environment.

/Bruce

Re: Problem with running the load balancer example application

[Haowei Yuan]

Hi Bruce,

Thank you very much for the information. I guess I will just use hugepages then.

At the beginning, the reason for using the --no-huge flag was that
somehow hugepages used by the app was not released (I don't remember
what the exact situation was, it might be that the program failed
somewhere because something was not properly configured), and with 2K
hugepages, after running the app several times, there was no hugepage
left. So I just wanted to try the --no-huge flag as it seemed to be
easy, and I guess there are probably ways to release those hugepages.
Then the warning messages showed up after the --no-huge flag was
added.

At the moment, the app works fine with hugepages.

Thanks,
Haowei


On Fri, Dec 19, 2014 at 4:14 AM, Bruce Richardson
<bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> On Fri, Dec 19, 2014 at 01:43:26AM -0600, Haowei Yuan wrote:
>> Hi folks,
>>
>> I am new to DPDK and have been trying to run the load balancer example
>> application on a machine with two NUMA nodes. Somehow the program
>> cannot be launched correctly if I add the "--no-huge" option to the
>> command. I am wondering if someone had seen similar problems, or maybe
>> I did something wrong.
>>
>
> Hi,
>
> Can I ask why you want to run the app with the -no-huge flag. That flag is
> primarily for testing only, and is not designed for use in a real-world
> environment.
>
> /Bruce
>

Re: Problem with running the load balancer example application

[Bruce Richardson]

On Fri, Dec 19, 2014 at 12:14:27PM -0600, Haowei Yuan wrote:
> Hi Bruce,
> 
> Thank you very much for the information. I guess I will just use hugepages then.
> 
> At the beginning, the reason for using the --no-huge flag was that
> somehow hugepages used by the app was not released (I don't remember
> what the exact situation was, it might be that the program failed
> somewhere because something was not properly configured), and with 2K
> hugepages, after running the app several times, there was no hugepage
> left. So I just wanted to try the --no-huge flag as it seemed to be
> easy, and I guess there are probably ways to release those hugepages.
> Then the warning messages showed up after the --no-huge flag was
> added.
> 

At the end of the application run, the hugepages are released by the application
as the rest of the memory it uses is, ie. automatically on process termination.
However, the hugepage files used to map that memory are not automatically deleted.
Despite this, the memory is still available for reuse by a subsequent DPDK run - which
will automatically delete the old files on startup - or by another app - in which
case you may need to manually delete the files.

> At the moment, the app works fine with hugepages.
> 

Ok, good to know.

/Bruce

> Thanks,
> Haowei
> 
> 
> On Fri, Dec 19, 2014 at 4:14 AM, Bruce Richardson
> <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > On Fri, Dec 19, 2014 at 01:43:26AM -0600, Haowei Yuan wrote:
> >> Hi folks,
> >>
> >> I am new to DPDK and have been trying to run the load balancer example
> >> application on a machine with two NUMA nodes. Somehow the program
> >> cannot be launched correctly if I add the "--no-huge" option to the
> >> command. I am wondering if someone had seen similar problems, or maybe
> >> I did something wrong.
> >>
> >
> > Hi,
> >
> > Can I ask why you want to run the app with the -no-huge flag. That flag is
> > primarily for testing only, and is not designed for use in a real-world
> > environment.
> >
> > /Bruce
> >