[SPDK] Question about open files

[SPDK] Question about open files

[Sudhe Sampath]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

We ran into an interesting problem yesterday where SPDK ended up running out of open files.  We determined that when we set IOUnitSize and MaxIOSize to 1 MB and use some number of cores (around 20+), we hit the open files issue below.

EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files

When using fewer core, we did not run into this issue.


We worked around this by upping the limit of open files (ulimit -r 32768), but we want to understand better going forward.  We are assuming that SPDK is grabbing more huge page memory based off the IOUnitSize and MaxIOSize * number of cores.  What is the correlation between open files and number of cores so that we can calculate or needs in the future?

We are using 2 MB huge pages, so we are assuming that if we use 1 GB huge pages, this will not be an issue.  Do you have any other insights in order to avoid this issue when scaling up?

Thanks,

-Sudhe

Re: [SPDK] Question about open files

[Sudheendra Sampath]

[-- Attachment #1: Type: text/plain, Size: 1578 bytes --]

Hello All,

Resending this email as I did not see any response.

Appreciate any inputs.

On Wed, Aug 21, 2019 at 9:35 AM Sudhe (Sudheendra) Sampath
<sudhe.s(a)samsung.com> wrote:
>
> We ran into an interesting problem yesterday where SPDK ended up running out of open files.  We determined that when we set IOUnitSize and MaxIOSize to 1 MB and use some number of cores (around 20+), we hit the open files issue below.
>
> EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
> EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
> EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
> EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
>
> When using fewer core, we did not run into this issue.
>
>
> We worked around this by upping the limit of open files (ulimit -r 32768), but we want to understand better going forward.  We are assuming that SPDK is grabbing more huge page memory based off the IOUnitSize and MaxIOSize * number of cores.  What is the correlation between open files and number of cores so that we can calculate or needs in the future?
>
> We are using 2 MB huge pages, so we are assuming that if we use 1 GB huge pages, this will not be an issue.  Do you have any other insights in order to avoid this issue when scaling up?
>
> Thanks,
>
> -Sudhe
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk



-- 
Regards

Sudheendra Sampath

Re: [SPDK] Question about open files

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 2531 bytes --]

Hi Sudhe,

On your system, was 32768 the new limit or the old limit?  If it was the new limit, what was the old limit set as on your system?

In general if you're using this large of an IOUnitSize, it will consume more memory.  More memory will equal more huge pages, and each hugepage has a file handle associated with it.

I would expect 1GB hugepages to alleviate this problem since you'd have one file handle per 1GB instead of 512 file handles per 1GB (1GB / 2MB).

Regards,

-Jim


On 8/28/19, 9:06 AM, "SPDK on behalf of Sudheendra Sampath" <spdk-bounces(a)lists.01.org on behalf of sudheendra.sampath(a)gmail.com> wrote:

    Hello All,
    
    Resending this email as I did not see any response.
    
    Appreciate any inputs.
    
    On Wed, Aug 21, 2019 at 9:35 AM Sudhe (Sudheendra) Sampath
    <sudhe.s(a)samsung.com> wrote:
    >
    > We ran into an interesting problem yesterday where SPDK ended up running out of open files.  We determined that when we set IOUnitSize and MaxIOSize to 1 MB and use some number of cores (around 20+), we hit the open files issue below.
    >
    > EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
    > EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
    > EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
    > EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Too many open files
    >
    > When using fewer core, we did not run into this issue.
    >
    >
    > We worked around this by upping the limit of open files (ulimit -r 32768), but we want to understand better going forward.  We are assuming that SPDK is grabbing more huge page memory based off the IOUnitSize and MaxIOSize * number of cores.  What is the correlation between open files and number of cores so that we can calculate or needs in the future?
    >
    > We are using 2 MB huge pages, so we are assuming that if we use 1 GB huge pages, this will not be an issue.  Do you have any other insights in order to avoid this issue when scaling up?
    >
    > Thanks,
    >
    > -Sudhe
    > _______________________________________________
    > SPDK mailing list
    > SPDK(a)lists.01.org
    > https://lists.01.org/mailman/listinfo/spdk
    
    
    
    -- 
    Regards
    
    Sudheendra Sampath
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk