From mboxrd@z Thu Jan 1 00:00:00 1970 From: Iain Barker Subject: Question about DPDK hugepage fd change Date: Tue, 5 Feb 2019 18:56:55 +0000 (UTC) Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: edwin.leung@oracle.com To: dev@dpdk.org Return-path: Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by dpdk.org (Postfix) with ESMTP id 215755F19 for ; Tue, 5 Feb 2019 19:57:02 +0100 (CET) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x15IrtXs061686 for ; Tue, 5 Feb 2019 18:57:02 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2130.oracle.com with ESMTP id 2qd9arcx72-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 05 Feb 2019 18:57:02 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x15IuuHx010952 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 5 Feb 2019 18:56:56 GMT Received: from abhmp0012.oracle.com (abhmp0012.oracle.com [141.146.116.18]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x15IuuiQ003915 for ; Tue, 5 Feb 2019 18:56:56 GMT List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi everyone, We just updated our application from DPDK 17.11.4 (LTS) to DPDK 18.11 (LTS)= and we noticed a regression. Our host platform is providing 2MB huge pages, so for 8GB reservation this = means 4000 pages are allocated. This worked fine in the prior LTS, but after upgrading DPDK what we are see= ing is that select() on an fd is failing. select() works fine when the process starts up, but does not work after DPD= K has been initialized. We did some investigation and found in the DPDK patches linked below, the h= ugepage tracking mechanism was changed from mmap to an array of file descri= ptors, and the rlimit for fd's is raised from the default to allow more fd'= s to be open. https://mails.dpdk.org/archives/dev/2018-September/110890.html https://mails.dpdk.org/archives/dev/2018-September/110889.html The problem is that the GNU C library (glibc) has a limit for the maximum f= d passed to select(), and is hard-coded in the POSIX header file and libc a= t 1024 (and probably many other OS libraries too as a result). Raising the rlimit for fd >1024 has undefined results, per the manpage: http://man7.org/linux/man-pages/man2/select.2.html An fd_set is a fixed size buffer. Executing FD_CLR() or FD_SET() with a value of fd that is negative or is equal to or larger than FD_SETSIZE will result in undefined behavior. Moreover, POSIX requires fd to be a valid file descriptor. The Linux kernel allows file descriptor sets of arbitrary size, determining the length of the sets to be checked from the value of nfds. However, in the glibc implementation, the fd_set type is fixed in size. Specifically, libc's header include/sys/select.h has an array of fd's which= is FD_SETSIZE deep. __fd_mask fds_bits[__FD_SETSIZE / __NFDBITS]; and usr/include/linux/posix_types.h is hard-coded with #define __FD_SETSIZE=A0 1024 As this define and array are in libc, they are used in many libraries on a = Linux system. So to use setsize >1024 means recompiling OS libraries and an= y other package that needs to use FDs, or ensuring that no library used by = the application ever calls select() on an fd set. That seems an unreasonabl= e burden... Any thoughts? thanks, Iain