extremely-high-unreasonable number of threads and memory usage on the client side using librados C++ api

extremely-high-unreasonable number of threads and memory usage on the client side using librados C++ api

[Amit Tiwary]

Ceph Developers, we are using C++ librados (ceph 0.67.9) to interact with 
two ceph clusters. cluster1(test) has 30 osds whereas cluster2(production) 
has 936 osds. We are hitting extremely-high-unreasonable number of threads 
and memory usage on the client side while interacting with cluster2 for 
performing even basic operations like stat() and read().

Scenario: consider a C++ app that performs simple-operations using librados:
1. cluster.init( ceph_client_id );
2. cluster.conf_read_file( ceph_conf_path );
3. cluster.connect();
4. cluster.ioctx_create( pool_name, ioctx);
5. while( in_file >> object_id )
     ioctx.stat( object_id, &size, &mtime);
6. ioctx.close();
7. cluster.shutdown();

When stat-ing or reading 710 objects on cluster 2, ps -o thcnt,size,rss,vsz 
pid_c++_app gives:
THCNT  SIZE    RSS    VSZ
    1 14656   8564   73148 (start of program)
   11 149032  11844  214844 (after step 1,2,3,4)
  867 1029668 15536 1095480 (during step 5)
  867 1029668 15536 1095480 (after ioctx.close)
    1 187592  13844 253404 (after cluster.shutdown)

Similarly, when stat-ing 2000 non-existent objects on cluster 2, the 
measures are given below:
THCNT  SIZE(KB)  RSS(KB)  VSZ(KB)
    1  14656     8564  73152
   11  149024   11924 214840
 1199  1371908 16672 1437724
 1199  1371908 16672 1437724
    1  188536    14212 254352 

On cluster1, the thread-count for same operations never goes above 25. Is is 
related to number of osds (are threads being created per osd and cleaned up 
only during shutdown?)

Is there any workaround that could be of any help? Any configurable 
parameter that disallows librados to cache/persist threads and release only 
at shutdown?

Note: We were hit with assertion and pthread_create failures few months ago 
and had to increase the ulimit on number of open files (ulimit -n)  
http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/19225

Regards,
Amit Tiwary

Re: extremely-high-unreasonable number of threads and memory usage on the client side using librados C++ api

[Sage Weil]

On Wed, 16 Jul 2014, Amit Tiwary wrote:
> Ceph Developers, we are using C++ librados (ceph 0.67.9) to interact with 
> two ceph clusters. cluster1(test) has 30 osds whereas cluster2(production) 
> has 936 osds. We are hitting extremely-high-unreasonable number of threads 
> and memory usage on the client side while interacting with cluster2 for 
> performing even basic operations like stat() and read().
> 
> Scenario: consider a C++ app that performs simple-operations using librados:
> 1. cluster.init( ceph_client_id );
> 2. cluster.conf_read_file( ceph_conf_path );
> 3. cluster.connect();
> 4. cluster.ioctx_create( pool_name, ioctx);
> 5. while( in_file >> object_id )
>      ioctx.stat( object_id, &size, &mtime);
> 6. ioctx.close();
> 7. cluster.shutdown();
> 
> When stat-ing or reading 710 objects on cluster 2, ps -o thcnt,size,rss,vsz 
> pid_c++_app gives:
> THCNT  SIZE    RSS    VSZ
>     1 14656   8564   73148 (start of program)
>    11 149032  11844  214844 (after step 1,2,3,4)
>   867 1029668 15536 1095480 (during step 5)
>   867 1029668 15536 1095480 (after ioctx.close)
>     1 187592  13844 253404 (after cluster.shutdown)
> 
> Similarly, when stat-ing 2000 non-existent objects on cluster 2, the 
> measures are given below:
> THCNT  SIZE(KB)  RSS(KB)  VSZ(KB)
>     1  14656     8564  73152
>    11  149024   11924 214840
>  1199  1371908 16672 1437724
>  1199  1371908 16672 1437724
>     1  188536    14212 254352 
> 
> On cluster1, the thread-count for same operations never goes above 25. Is is 
> related to number of osds (are threads being created per osd and cleaned up 
> only during shutdown?)
> 
> Is there any workaround that could be of any help? Any configurable 
> parameter that disallows librados to cache/persist threads and release only 
> at shutdown?
> 
> Note: We were hit with assertion and pthread_create failures few months ago 
> and had to increase the ulimit on number of open files (ulimit -n)  
> http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/19225

Increasing the open file limit is the way to address this currently.  The 
underlying problem is that we are creating threads to service each 
connection and there isn't a max open connection limit.  The real fix is 
to reimplement the SimpleMessenger, but the workaround is simple enough 
(ulimit -n) that we haven't addressed that yet.  Yes, it will consume more 
resources than it probably should.. sorry about that!

sage

Re: extremely-high-unreasonable number of threads and memory usage on the client side using librados C++ api

[Amit Tiwary]

Sage Weil <sweil <at> redhat.com> writes:
> Increasing the open file limit is the way to address this currently.  The 
> underlying problem is that we are creating threads to service each 
> connection and there isn't a max open connection limit.  The real fix is 
> to reimplement the SimpleMessenger, but the workaround is simple enough 
> (ulimit -n) that we haven't addressed that yet.  Yes, it will consume more 
> resources than it probably should.. sorry about that!
> 
Thanks Sage. Would eagerly await work on SimpleMessenger and other items 
from this blueprint to improve threading. 
https://wiki.ceph.com/Planning/Blueprints/Giant/librados%2F%2Fobjecter%3A_im
prove_threading
--
Amit Tiwary