From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============4171378332679536695==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] Question about open files Date: Wed, 28 Aug 2019 16:46:51 +0000 Message-ID: In-Reply-To: CABnqofzQ+xXJ7Bx2+7Ys2tHOJ-fE_LmknSHG+cbcdtQ4a-5SZw@mail.gmail.com List-ID: To: spdk@lists.01.org --===============4171378332679536695== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Sudhe, On your system, was 32768 the new limit or the old limit? If it was the ne= w limit, what was the old limit set as on your system? In general if you're using this large of an IOUnitSize, it will consume mor= e memory. More memory will equal more huge pages, and each hugepage has a = file handle associated with it. I would expect 1GB hugepages to alleviate this problem since you'd have one= file handle per 1GB instead of 512 file handles per 1GB (1GB / 2MB). Regards, -Jim =EF=BB=BFOn 8/28/19, 9:06 AM, "SPDK on behalf of Sudheendra Sampath" wrote: Hello All, = Resending this email as I did not see any response. = Appreciate any inputs. = On Wed, Aug 21, 2019 at 9:35 AM Sudhe (Sudheendra) Sampath wrote: > > We ran into an interesting problem yesterday where SPDK ended up runn= ing out of open files. We determined that when we set IOUnitSize and MaxIO= Size to 1 MB and use some number of cores (around 20+), we hit the open fil= es issue below. > > EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many ope= n files > EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many ope= n files > EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many ope= n files > EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many ope= n files > > When using fewer core, we did not run into this issue. > > > We worked around this by upping the limit of open files (ulimit -r 32= 768), but we want to understand better going forward. We are assuming that= SPDK is grabbing more huge page memory based off the IOUnitSize and MaxIOS= ize * number of cores. What is the correlation between open files and numb= er of cores so that we can calculate or needs in the future? > > We are using 2 MB huge pages, so we are assuming that if we use 1 GB = huge pages, this will not be an issue. Do you have any other insights in o= rder to avoid this issue when scaling up? > > Thanks, > > -Sudhe > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk = = = -- = Regards = Sudheendra Sampath _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk = --===============4171378332679536695==--