extremely-high-unreasonable number of threads and memory usage on the client side using librados C++ api

[Amit Tiwary <tiwaryamt@gmail.com>]

Ceph Developers, we are using C++ librados (ceph 0.67.9) to interact with 
two ceph clusters. cluster1(test) has 30 osds whereas cluster2(production) 
has 936 osds. We are hitting extremely-high-unreasonable number of threads 
and memory usage on the client side while interacting with cluster2 for 
performing even basic operations like stat() and read().

Scenario: consider a C++ app that performs simple-operations using librados:
1. cluster.init( ceph_client_id );
2. cluster.conf_read_file( ceph_conf_path );
3. cluster.connect();
4. cluster.ioctx_create( pool_name, ioctx);
5. while( in_file >> object_id )
     ioctx.stat( object_id, &size, &mtime);
6. ioctx.close();
7. cluster.shutdown();

When stat-ing or reading 710 objects on cluster 2, ps -o thcnt,size,rss,vsz 
pid_c++_app gives:
THCNT  SIZE    RSS    VSZ
    1 14656   8564   73148 (start of program)
   11 149032  11844  214844 (after step 1,2,3,4)
  867 1029668 15536 1095480 (during step 5)
  867 1029668 15536 1095480 (after ioctx.close)
    1 187592  13844 253404 (after cluster.shutdown)

Similarly, when stat-ing 2000 non-existent objects on cluster 2, the 
measures are given below:
THCNT  SIZE(KB)  RSS(KB)  VSZ(KB)
    1  14656     8564  73152
   11  149024   11924 214840
 1199  1371908 16672 1437724
 1199  1371908 16672 1437724
    1  188536    14212 254352 

On cluster1, the thread-count for same operations never goes above 25. Is is 
related to number of osds (are threads being created per osd and cleaned up 
only during shutdown?)

Is there any workaround that could be of any help? Any configurable 
parameter that disallows librados to cache/persist threads and release only 
at shutdown?

Note: We were hit with assertion and pthread_create failures few months ago 
and had to increase the ulimit on number of open files (ulimit -n)  
http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/19225

Regards,
Amit Tiwary