Multi-process on the same host

Multi-process on the same host

[Walter de Donato]

Hello,

I've been using DPDK for a while and now I encountered the following issue:
when I try to run two primary processes on the same host (with --no-shconf
option enabled) respectively sending packets on one port and receiving them
on a different port (the two ports are directly connected with a CAT-6
cable), I get this error on the receiving process:

Program received signal SIGSEGV, Segmentation fault.
0x00000000004158a0 in rte_eth_rx_burst (port_id=0 '\000', queue_id=0,
rx_pkts=0x7ffff5baa8f0, nb_pkts=128) at
/home/devel/dpdk/build/include/rte_ethdev.h:1658
1658            return (*dev->rx_pkt_burst)(dev->data->rx_queues[queue_id],
rx_pkts, nb_pkts);

To give some more details:
- the options given to the two processes:
  ./receiver -c 0x3 -n 2 -m 200 --no-shconf -- -p 0x1
  ./sender -c 0xc -n 2 -m 200 --no-shconf -- -p 0x2
  where the -p option is the binary mask to select the ports to enable.
- the network card is a dualport Intel X540:
  port 0: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)
  port 1: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)
- this is the hugeadm --pool-list output:
      Size  Minimum  Current  Maximum  Default
  1073741824        2        2        2        *

My first question is: should it be possible to let separate primary
processes coexist if they use different resources (cores, ports, memory
pools)?

A second question is: there is any other workaround to let this scenario
work without merging the two processes into two lcores of the same process?

Thanks in advance,
-Walter

Walter de Donato, Ph.D.
PostDoc @ Department of Electrical Engineering and Information Technologies
University of Napoli Federico II
Via Claudio 21 -- 80125 Napoli (Italy)
Phone: +39 081 76 83821 - Fax: +39 081 76 83816
Email: walter.dedonato-7EJwN6hzp+g@public.gmane.org
WWW: http://wpage.unina.it/walter.dedonato

Re: Multi-process on the same host

[Stephen Hemminger]

On Fri, 4 Oct 2013 13:47:02 +0200
Walter de Donato <walter.dedonato-7EJwN6hzp+g@public.gmane.org> wrote:

> Hello,
> 
> I've been using DPDK for a while and now I encountered the following issue:
> when I try to run two primary processes on the same host (with --no-shconf
> option enabled) respectively sending packets on one port and receiving them
> on a different port (the two ports are directly connected with a CAT-6
> cable), I get this error on the receiving process:
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x00000000004158a0 in rte_eth_rx_burst (port_id=0 '\000', queue_id=0,
> rx_pkts=0x7ffff5baa8f0, nb_pkts=128) at
> /home/devel/dpdk/build/include/rte_ethdev.h:1658
> 1658            return (*dev->rx_pkt_burst)(dev->data->rx_queues[queue_id],
> rx_pkts, nb_pkts);
> 
> To give some more details:
> - the options given to the two processes:
>   ./receiver -c 0x3 -n 2 -m 200 --no-shconf -- -p 0x1
>   ./sender -c 0xc -n 2 -m 200 --no-shconf -- -p 0x2
>   where the -p option is the binary mask to select the ports to enable.
> - the network card is a dualport Intel X540:
>   port 0: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)
>   port 1: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)
> - this is the hugeadm --pool-list output:
>       Size  Minimum  Current  Maximum  Default
>   1073741824        2        2        2        *
> 
> My first question is: should it be possible to let separate primary
> processes coexist if they use different resources (cores, ports, memory
> pools)?
> 
> A second question is: there is any other workaround to let this scenario
> work without merging the two processes into two lcores of the same process?
> 
> Thanks in advance,
> -Walter

The problem is that huge TLB filesystem is a shared resource.
Because of that the memory pools of the two processes overlap, and memory pools
are used for packet buffers, malloc, etc.

You might be able to use no-huge, but then other things would probably break.

Re: Multi-process on the same host

[Richardson, Bruce]

> -----Original Message-----
> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Stephen
> Hemminger
> Sent: Friday, October 04, 2013 5:39 PM
> To: Walter de Donato
> Cc: dev-VfR2kkLFssw@public.gmane.org
> Subject: Re: [dpdk-dev] Multi-process on the same host
> 
> On Fri, 4 Oct 2013 13:47:02 +0200
> Walter de Donato <walter.dedonato-7EJwN6hzp+g@public.gmane.org> wrote:
> 
> > Hello,
> >
> > I've been using DPDK for a while and now I encountered the following
> issue:
> > when I try to run two primary processes on the same host (with
> > --no-shconf option enabled) respectively sending packets on one port
> > and receiving them on a different port (the two ports are directly
> > connected with a CAT-6 cable), I get this error on the receiving process:
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > 0x00000000004158a0 in rte_eth_rx_burst (port_id=0 '\000',
> queue_id=0,
> > rx_pkts=0x7ffff5baa8f0, nb_pkts=128) at
> > /home/devel/dpdk/build/include/rte_ethdev.h:1658
> > 1658            return (*dev->rx_pkt_burst)(dev->data-
> >rx_queues[queue_id],
> > rx_pkts, nb_pkts);
> >
> > To give some more details:
> > - the options given to the two processes:
> >   ./receiver -c 0x3 -n 2 -m 200 --no-shconf -- -p 0x1
> >   ./sender -c 0xc -n 2 -m 200 --no-shconf -- -p 0x2
> >   where the -p option is the binary mask to select the ports to enable.
> > - the network card is a dualport Intel X540:
> >   port 0: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev
> 01)
> >   port 1: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
> > (rev 01)
> > - this is the hugeadm --pool-list output:
> >       Size  Minimum  Current  Maximum  Default
> >   1073741824        2        2        2        *
> >
> > My first question is: should it be possible to let separate primary
> > processes coexist if they use different resources (cores, ports,
> > memory pools)?
> >
> > A second question is: there is any other workaround to let this
> > scenario work without merging the two processes into two lcores of the
> same process?
> >
> > Thanks in advance,
> > -Walter
> 
> The problem is that huge TLB filesystem is a shared resource.
> Because of that the memory pools of the two processes overlap, and
> memory pools are used for packet buffers, malloc, etc.
> 
> You might be able to use no-huge, but then other things would probably
> break.

The way to run two primary processes side by side is documented in the document "Intel(r) Data Plane Development Kit (Intel(r) DPDK): Programmer's Guide" available at: http://www.intel.com/content/www/us/en/intelligent-systems/intel-technology/intel-dpdk-programmers-guide.html and is covered in section 17.2.3. You need to pass the "--file-prefix" flag when running your application to force the processes to use different hugepage files so they are not shared among the two processes.

Regards,
/Bruce
--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Re: Multi-process on the same host

[Walter de Donato]

Thanks a lot Bruce,

I started looking at the multi-process examples - where this case is not
considered - and I missed that section in the programmer's guide.

Regards,
-Walter

Walter de Donato, Ph.D.
PostDoc @ Department of Electrical Engineering and Information Technologies
University of Napoli Federico II
Via Claudio 21 -- 80125 Napoli (Italy)
Phone: +39 081 76 83821 - Fax: +39 081 76 83816
Email: walter.dedonato-7EJwN6hzp+g@public.gmane.org
WWW: http://wpage.unina.it/walter.dedonato


2013/10/4 Richardson, Bruce <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

> > -----Original Message-----
> > From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Stephen
> > Hemminger
> > Sent: Friday, October 04, 2013 5:39 PM
> > To: Walter de Donato
> > Cc: dev-VfR2kkLFssw@public.gmane.org
> > Subject: Re: [dpdk-dev] Multi-process on the same host
> >
> > On Fri, 4 Oct 2013 13:47:02 +0200
> > Walter de Donato <walter.dedonato-7EJwN6hzp+g@public.gmane.org> wrote:
> >
> > > Hello,
> > >
> > > I've been using DPDK for a while and now I encountered the following
> > issue:
> > > when I try to run two primary processes on the same host (with
> > > --no-shconf option enabled) respectively sending packets on one port
> > > and receiving them on a different port (the two ports are directly
> > > connected with a CAT-6 cable), I get this error on the receiving
> process:
> > >
> > > Program received signal SIGSEGV, Segmentation fault.
> > > 0x00000000004158a0 in rte_eth_rx_burst (port_id=0 '\000',
> > queue_id=0,
> > > rx_pkts=0x7ffff5baa8f0, nb_pkts=128) at
> > > /home/devel/dpdk/build/include/rte_ethdev.h:1658
> > > 1658            return (*dev->rx_pkt_burst)(dev->data-
> > >rx_queues[queue_id],
> > > rx_pkts, nb_pkts);
> > >
> > > To give some more details:
> > > - the options given to the two processes:
> > >   ./receiver -c 0x3 -n 2 -m 200 --no-shconf -- -p 0x1
> > >   ./sender -c 0xc -n 2 -m 200 --no-shconf -- -p 0x2
> > >   where the -p option is the binary mask to select the ports to enable.
> > > - the network card is a dualport Intel X540:
> > >   port 0: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
> (rev
> > 01)
> > >   port 1: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
> > > (rev 01)
> > > - this is the hugeadm --pool-list output:
> > >       Size  Minimum  Current  Maximum  Default
> > >   1073741824        2        2        2        *
> > >
> > > My first question is: should it be possible to let separate primary
> > > processes coexist if they use different resources (cores, ports,
> > > memory pools)?
> > >
> > > A second question is: there is any other workaround to let this
> > > scenario work without merging the two processes into two lcores of the
> > same process?
> > >
> > > Thanks in advance,
> > > -Walter
> >
> > The problem is that huge TLB filesystem is a shared resource.
> > Because of that the memory pools of the two processes overlap, and
> > memory pools are used for packet buffers, malloc, etc.
> >
> > You might be able to use no-huge, but then other things would probably
> > break.
>
> The way to run two primary processes side by side is documented in the
> document "Intel(r) Data Plane Development Kit (Intel(r) DPDK): Programmer's
> Guide" available at:
> http://www.intel.com/content/www/us/en/intelligent-systems/intel-technology/intel-dpdk-programmers-guide.htmland is covered in section 17.2.3. You need to pass the "--file-prefix" flag
> when running your application to force the processes to use different
> hugepage files so they are not shared among the two processes.
>
> Regards,
> /Bruce
> --------------------------------------------------------------
> Intel Shannon Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
> Business address: Dromore House, East Park, Shannon, Co. Clare
>
> This e-mail and any attachments may contain confidential material for the
> sole use of the intended recipient(s). Any review or distribution by others
> is strictly prohibited. If you are not the intended recipient, please
> contact the sender and delete all copies.
>
>
>
>

Re: Multi-process on the same host

[Thomas Monjalon]

04/10/2013 16:45, Richardson, Bruce :
> This e-mail and any attachments may contain confidential material for the
> sole use of the intended recipient(s). Any review or distribution by others
> is strictly prohibited. If you are not the intended recipient, please
> contact the sender and delete all copies.

side note: this type of email is not compatible with a public mailing-list.
Maybe you've done an error by setting this footer or by sending this email 
publicly. We cannot know.

To prevent this type of error, a filter could be set which would 
moderate/reject messages containing such footer.

-- 
Thomas