civetweb threads

civetweb threads

[Mustafa Muhammad]

Hi,
I am using civetweb in my radosgw, if I use "rgw thread pool size"
that is more than 1024, civetweb doesn't work.
e.g.
rgw thread pool size = 1024
rgw frontends = "civetweb port=80"

#ps aux -L | grep rados | wc -l
1096

everything works fine


If I use:
rgw thread pool size = 1025
rgw frontends = "civetweb port=80"

# ps aux -L | grep rados | wc -l
43

And http server is not listening.

If I don't use civetweb:
rgw thread pool size = 10240

# ps aux -L | grep rados | wc -l
10278

Regards

Mustafa Muhammad

Re: civetweb threads

[Mustafa Muhammad]

On Sat, Nov 15, 2014 at 10:28 AM, Mustafa Muhammad
<mustafaa.alhamdaani@gmail.com> wrote:
> Hi,
> I am using civetweb in my radosgw, if I use "rgw thread pool size"
> that is more than 1024, civetweb doesn't work.
> e.g.
> rgw thread pool size = 1024
> rgw frontends = "civetweb port=80"
>
> #ps aux -L | grep rados | wc -l
> 1096
>
> everything works fine
>
>
> If I use:
> rgw thread pool size = 1025
> rgw frontends = "civetweb port=80"
>
> # ps aux -L | grep rados | wc -l
> 43
>
> And http server is not listening.
>
> If I don't use civetweb:
> rgw thread pool size = 10240
>
> # ps aux -L | grep rados | wc -l
> 10278
>
> Regards
>
> Mustafa Muhammad

I found the problem, it is hardcoded here:
https://github.com/ceph/civetweb/blob/master/src/civetweb.c
as:
#define MAX_WORKER_THREADS 1024

I increased it to 20480 an compiled from source, problem solved.
I should we make a patch, right?

Re: civetweb threads

[Yehuda Sadeh]

On Sat, Nov 15, 2014 at 3:55 AM, Mustafa Muhammad
<mustafaa.alhamdaani@gmail.com> wrote:
> On Sat, Nov 15, 2014 at 10:28 AM, Mustafa Muhammad
> <mustafaa.alhamdaani@gmail.com> wrote:
>> Hi,
>> I am using civetweb in my radosgw, if I use "rgw thread pool size"
>> that is more than 1024, civetweb doesn't work.
>> e.g.
>> rgw thread pool size = 1024
>> rgw frontends = "civetweb port=80"
>>
>> #ps aux -L | grep rados | wc -l
>> 1096
>>
>> everything works fine
>>
>>
>> If I use:
>> rgw thread pool size = 1025
>> rgw frontends = "civetweb port=80"
>>
>> # ps aux -L | grep rados | wc -l
>> 43
>>
>> And http server is not listening.
>>
>> If I don't use civetweb:
>> rgw thread pool size = 10240
>>
>> # ps aux -L | grep rados | wc -l
>> 10278
>>
>> Regards
>>
>> Mustafa Muhammad
>
> I found the problem, it is hardcoded here:
> https://github.com/ceph/civetweb/blob/master/src/civetweb.c
> as:
> #define MAX_WORKER_THREADS 1024
>
> I increased it to 20480 an compiled from source, problem solved.
> I should we make a patch, right?

Please do, preferably a github pull request. Also, if you could open a
ceph tracker issue with the specific would be great.

Thanks,
Yehuda

Re: civetweb threads

[pushpesh sharma]

Yehuda,

I believe it would not be wise to increase the value of this
parameter; till the underlying threading model of RGW is fixed.
I tried with various  'rgw_thread_pool_size' (128,256,512,1024), and
didn't observe any better throughput , instead response time
increased. Which means RGW is not able to handled increased
concurrency.

I have another observation regarding 'rgw_max _chunk_size'. I was
doing some experiment with this parameter. I have seen the following
behavior on a test deployment and then verified on development
environment :-

1. Based on the value we set for 'rgw_max_chunk_size', every Swift
object gets sliced in to equal sized chunks and each chunk got created
as a RAODS Object in the pool (.rgw.buckets)

2. While reading the same Swift Object, sum of all RADOS objects provided.

3. However setting this 'rgw_max_chunk_size' value up to 4MB only
works after which GET request will fail and only the first RADOS
object will be returned.

#######################
$./ceph --admin-daemon out/client.admin.9660.asok config set
rgw_max_chunk_size 4195000
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{ "success": "rgw_max_chunk_size = '4195000' "}


swift -A http://localhost:8000/auth -U tester:testing -K asdf upload
my4+MB_chunked ceph-0.85.tar.gz
ceph-0.85.tar.gz
ceph@Ubuntu14:~$ cd copy/
ceph@Ubuntu14:~/copy$ swift -A http://localhost:8000/auth -U
tester:testing -K asdf download my4+MB_chunked ceph-0.85.tar.gz
ceph-0.85.tar.gz: md5sum != etag, ea27d35584bae391e9f3e67c86849a66 !=
fd9237fa01803398bd8012c4d59975d8
ceph-0.85.tar.gz [auth 0.021s, headers 0.040s, total 30.045s, 0.140 MB/s]
ceph-0.85.tar.gz: read_length != content_length, 4194304 != 7384934
######################

Is this behavior sane? Why the Swift Object is chunked  by default
based on this value. Should the decision to chunk the object be coming
from end-user? (Like Native Swift)

On Sat, Nov 15, 2014 at 9:48 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> On Sat, Nov 15, 2014 at 3:55 AM, Mustafa Muhammad
> <mustafaa.alhamdaani@gmail.com> wrote:
>> On Sat, Nov 15, 2014 at 10:28 AM, Mustafa Muhammad
>> <mustafaa.alhamdaani@gmail.com> wrote:
>>> Hi,
>>> I am using civetweb in my radosgw, if I use "rgw thread pool size"
>>> that is more than 1024, civetweb doesn't work.
>>> e.g.
>>> rgw thread pool size = 1024
>>> rgw frontends = "civetweb port=80"
>>>
>>> #ps aux -L | grep rados | wc -l
>>> 1096
>>>
>>> everything works fine
>>>
>>>
>>> If I use:
>>> rgw thread pool size = 1025
>>> rgw frontends = "civetweb port=80"
>>>
>>> # ps aux -L | grep rados | wc -l
>>> 43
>>>
>>> And http server is not listening.
>>>
>>> If I don't use civetweb:
>>> rgw thread pool size = 10240
>>>
>>> # ps aux -L | grep rados | wc -l
>>> 10278
>>>
>>> Regards
>>>
>>> Mustafa Muhammad
>>
>> I found the problem, it is hardcoded here:
>> https://github.com/ceph/civetweb/blob/master/src/civetweb.c
>> as:
>> #define MAX_WORKER_THREADS 1024
>>
>> I increased it to 20480 an compiled from source, problem solved.
>> I should we make a patch, right?
>
> Please do, preferably a github pull request. Also, if you could open a
> ceph tracker issue with the specific would be great.
>
> Thanks,
> Yehuda
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
-Pushpesh

Re: civetweb threads

[Yehuda Sadeh]

On Sat, Nov 15, 2014 at 8:57 PM, pushpesh sharma <pushpesh.eck@gmail.com> wrote:
> Yehuda,
>
> I believe it would not be wise to increase the value of this
> parameter; till the underlying threading model of RGW is fixed.
> I tried with various  'rgw_thread_pool_size' (128,256,512,1024), and
> didn't observe any better throughput , instead response time
> increased. Which means RGW is not able to handled increased
> concurrency.
>
> I have another observation regarding 'rgw_max _chunk_size'. I was
> doing some experiment with this parameter. I have seen the following
> behavior on a test deployment and then verified on development
> environment :-
>
> 1. Based on the value we set for 'rgw_max_chunk_size', every Swift
> object gets sliced in to equal sized chunks and each chunk got created
> as a RAODS Object in the pool (.rgw.buckets)
>
> 2. While reading the same Swift Object, sum of all RADOS objects provided.
>
> 3. However setting this 'rgw_max_chunk_size' value up to 4MB only
> works after which GET request will fail and only the first RADOS
> object will be returned.
>
> #######################
> $./ceph --admin-daemon out/client.admin.9660.asok config set
> rgw_max_chunk_size 4195000
> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
> { "success": "rgw_max_chunk_size = '4195000' "}
>
>
> swift -A http://localhost:8000/auth -U tester:testing -K asdf upload
> my4+MB_chunked ceph-0.85.tar.gz
> ceph-0.85.tar.gz
> ceph@Ubuntu14:~$ cd copy/
> ceph@Ubuntu14:~/copy$ swift -A http://localhost:8000/auth -U
> tester:testing -K asdf download my4+MB_chunked ceph-0.85.tar.gz
> ceph-0.85.tar.gz: md5sum != etag, ea27d35584bae391e9f3e67c86849a66 !=
> fd9237fa01803398bd8012c4d59975d8
> ceph-0.85.tar.gz [auth 0.021s, headers 0.040s, total 30.045s, 0.140 MB/s]
> ceph-0.85.tar.gz: read_length != content_length, 4194304 != 7384934
> ######################
>
> Is this behavior sane? Why the Swift Object is chunked  by default

I need to see the logs for it to maybe determine what was going on.
There's another param that limits max request size
(rgw_get_obj_max_req_size), so you probably hit this one. Note that
it's not advisable to have the chunk size too big.

> based on this value. Should the decision to chunk the object be coming
> from end-user? (Like Native Swift)

Does native swift even support striping? Not having striping or having
variable stripe sizes could have severe impact on data balancing.
Also, the gateway insures atomicity of certain operations by limiting
the object head's size.

Yehuda

>
> On Sat, Nov 15, 2014 at 9:48 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> On Sat, Nov 15, 2014 at 3:55 AM, Mustafa Muhammad
>> <mustafaa.alhamdaani@gmail.com> wrote:
>>> On Sat, Nov 15, 2014 at 10:28 AM, Mustafa Muhammad
>>> <mustafaa.alhamdaani@gmail.com> wrote:
>>>> Hi,
>>>> I am using civetweb in my radosgw, if I use "rgw thread pool size"
>>>> that is more than 1024, civetweb doesn't work.
>>>> e.g.
>>>> rgw thread pool size = 1024
>>>> rgw frontends = "civetweb port=80"
>>>>
>>>> #ps aux -L | grep rados | wc -l
>>>> 1096
>>>>
>>>> everything works fine
>>>>
>>>>
>>>> If I use:
>>>> rgw thread pool size = 1025
>>>> rgw frontends = "civetweb port=80"
>>>>
>>>> # ps aux -L | grep rados | wc -l
>>>> 43
>>>>
>>>> And http server is not listening.
>>>>
>>>> If I don't use civetweb:
>>>> rgw thread pool size = 10240
>>>>
>>>> # ps aux -L | grep rados | wc -l
>>>> 10278
>>>>
>>>> Regards
>>>>
>>>> Mustafa Muhammad
>>>
>>> I found the problem, it is hardcoded here:
>>> https://github.com/ceph/civetweb/blob/master/src/civetweb.c
>>> as:
>>> #define MAX_WORKER_THREADS 1024
>>>
>>> I increased it to 20480 an compiled from source, problem solved.
>>> I should we make a patch, right?
>>
>> Please do, preferably a github pull request. Also, if you could open a
>> ceph tracker issue with the specific would be great.
>>
>> Thanks,
>> Yehuda
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> -Pushpesh

Re: civetweb threads

[Mustafa Muhammad]

On Sat, Nov 15, 2014 at 7:18 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> On Sat, Nov 15, 2014 at 3:55 AM, Mustafa Muhammad
> <mustafaa.alhamdaani@gmail.com> wrote:
>> On Sat, Nov 15, 2014 at 10:28 AM, Mustafa Muhammad
>> <mustafaa.alhamdaani@gmail.com> wrote:
>>> Hi,
>>> I am using civetweb in my radosgw, if I use "rgw thread pool size"
>>> that is more than 1024, civetweb doesn't work.
>>> e.g.
>>> rgw thread pool size = 1024
>>> rgw frontends = "civetweb port=80"
>>>
>>> #ps aux -L | grep rados | wc -l
>>> 1096
>>>
>>> everything works fine
>>>
>>>
>>> If I use:
>>> rgw thread pool size = 1025
>>> rgw frontends = "civetweb port=80"
>>>
>>> # ps aux -L | grep rados | wc -l
>>> 43
>>>
>>> And http server is not listening.
>>>
>>> If I don't use civetweb:
>>> rgw thread pool size = 10240
>>>
>>> # ps aux -L | grep rados | wc -l
>>> 10278
>>>
>>> Regards
>>>
>>> Mustafa Muhammad
>>
>> I found the problem, it is hardcoded here:
>> https://github.com/ceph/civetweb/blob/master/src/civetweb.c
>> as:
>> #define MAX_WORKER_THREADS 1024
>>
>> I increased it to 20480 an compiled from source, problem solved.
>> I should we make a patch, right?
>
> Please do, preferably a github pull request. Also, if you could open a
> ceph tracker issue with the specific would be great.

I wanted to do create a github pull request with this value set to
20480, but thought I should ask if you want it to be hardcoded, I
understood that 'rgw_thread_pool_size' maps to 'num_op_thread' in
civetweb, is there a hardcoded max for rgw_thread_pool_size so I can
set the same for MAX_WORKER_THREADS? What do you suggest?

Regards
Mustafa

>
> Thanks,
> Yehuda

Re: civetweb threads

[Mustafa Muhammad]

On Sun, Nov 16, 2014 at 7:57 AM, pushpesh sharma <pushpesh.eck@gmail.com> wrote:
> Yehuda,
>
> I believe it would not be wise to increase the value of this
> parameter; till the underlying threading model of RGW is fixed.
> I tried with various  'rgw_thread_pool_size' (128,256,512,1024), and
> didn't observe any better throughput , instead response time
> increased. Which means RGW is not able to handled increased
> concurrency.
I needed to increase this number to increase the number of concurrent
connections to civetweb, I need high number of threads (>7000
connections, large files)
I had my throughput going from 200 MB/s to 1200 MB/s (10G interface is
full, I could've gone higher)  due to increasing civetweb connections.

> I have another observation regarding 'rgw_max _chunk_size'. I was
> doing some experiment with this parameter. I have seen the following
> behavior on a test deployment and then verified on development
> environment :-
>
> 1. Based on the value we set for 'rgw_max_chunk_size', every Swift
> object gets sliced in to equal sized chunks and each chunk got created
> as a RAODS Object in the pool (.rgw.buckets)
>
> 2. While reading the same Swift Object, sum of all RADOS objects provided.
>
> 3. However setting this 'rgw_max_chunk_size' value up to 4MB only
> works after which GET request will fail and only the first RADOS
> object will be returned.
>
> #######################
> $./ceph --admin-daemon out/client.admin.9660.asok config set
> rgw_max_chunk_size 4195000
> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
> { "success": "rgw_max_chunk_size = '4195000' "}
>
>
> swift -A http://localhost:8000/auth -U tester:testing -K asdf upload
> my4+MB_chunked ceph-0.85.tar.gz
> ceph-0.85.tar.gz
> ceph@Ubuntu14:~$ cd copy/
> ceph@Ubuntu14:~/copy$ swift -A http://localhost:8000/auth -U
> tester:testing -K asdf download my4+MB_chunked ceph-0.85.tar.gz
> ceph-0.85.tar.gz: md5sum != etag, ea27d35584bae391e9f3e67c86849a66 !=
> fd9237fa01803398bd8012c4d59975d8
> ceph-0.85.tar.gz [auth 0.021s, headers 0.040s, total 30.045s, 0.140 MB/s]
> ceph-0.85.tar.gz: read_length != content_length, 4194304 != 7384934
> ######################
>
> Is this behavior sane? Why the Swift Object is chunked  by default
> based on this value. Should the decision to chunk the object be coming
> from end-user? (Like Native Swift)
>
> On Sat, Nov 15, 2014 at 9:48 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> On Sat, Nov 15, 2014 at 3:55 AM, Mustafa Muhammad
>> <mustafaa.alhamdaani@gmail.com> wrote:
>>> On Sat, Nov 15, 2014 at 10:28 AM, Mustafa Muhammad
>>> <mustafaa.alhamdaani@gmail.com> wrote:
>>>> Hi,
>>>> I am using civetweb in my radosgw, if I use "rgw thread pool size"
>>>> that is more than 1024, civetweb doesn't work.
>>>> e.g.
>>>> rgw thread pool size = 1024
>>>> rgw frontends = "civetweb port=80"
>>>>
>>>> #ps aux -L | grep rados | wc -l
>>>> 1096
>>>>
>>>> everything works fine
>>>>
>>>>
>>>> If I use:
>>>> rgw thread pool size = 1025
>>>> rgw frontends = "civetweb port=80"
>>>>
>>>> # ps aux -L | grep rados | wc -l
>>>> 43
>>>>
>>>> And http server is not listening.
>>>>
>>>> If I don't use civetweb:
>>>> rgw thread pool size = 10240
>>>>
>>>> # ps aux -L | grep rados | wc -l
>>>> 10278
>>>>
>>>> Regards
>>>>
>>>> Mustafa Muhammad
>>>
>>> I found the problem, it is hardcoded here:
>>> https://github.com/ceph/civetweb/blob/master/src/civetweb.c
>>> as:
>>> #define MAX_WORKER_THREADS 1024
>>>
>>> I increased it to 20480 an compiled from source, problem solved.
>>> I should we make a patch, right?
>>
>> Please do, preferably a github pull request. Also, if you could open a
>> ceph tracker issue with the specific would be great.
>>
>> Thanks,
>> Yehuda
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> -Pushpesh

Re: civetweb threads

[Yehuda Sadeh]

On Sun, Nov 16, 2014 at 2:51 AM, Mustafa Muhammad
<mustafaa.alhamdaani@gmail.com> wrote:
> On Sat, Nov 15, 2014 at 7:18 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> On Sat, Nov 15, 2014 at 3:55 AM, Mustafa Muhammad
>> <mustafaa.alhamdaani@gmail.com> wrote:
>>> On Sat, Nov 15, 2014 at 10:28 AM, Mustafa Muhammad
>>> <mustafaa.alhamdaani@gmail.com> wrote:
>>>> Hi,
>>>> I am using civetweb in my radosgw, if I use "rgw thread pool size"
>>>> that is more than 1024, civetweb doesn't work.
>>>> e.g.
>>>> rgw thread pool size = 1024
>>>> rgw frontends = "civetweb port=80"
>>>>
>>>> #ps aux -L | grep rados | wc -l
>>>> 1096
>>>>
>>>> everything works fine
>>>>
>>>>
>>>> If I use:
>>>> rgw thread pool size = 1025
>>>> rgw frontends = "civetweb port=80"
>>>>
>>>> # ps aux -L | grep rados | wc -l
>>>> 43
>>>>
>>>> And http server is not listening.
>>>>
>>>> If I don't use civetweb:
>>>> rgw thread pool size = 10240
>>>>
>>>> # ps aux -L | grep rados | wc -l
>>>> 10278
>>>>
>>>> Regards
>>>>
>>>> Mustafa Muhammad
>>>
>>> I found the problem, it is hardcoded here:
>>> https://github.com/ceph/civetweb/blob/master/src/civetweb.c
>>> as:
>>> #define MAX_WORKER_THREADS 1024
>>>
>>> I increased it to 20480 an compiled from source, problem solved.
>>> I should we make a patch, right?
>>
>> Please do, preferably a github pull request. Also, if you could open a
>> ceph tracker issue with the specific would be great.
>
> I wanted to do create a github pull request with this value set to
> 20480, but thought I should ask if you want it to be hardcoded, I
> understood that 'rgw_thread_pool_size' maps to 'num_op_thread' in
> civetweb, is there a hardcoded max for rgw_thread_pool_size so I can
> set the same for MAX_WORKER_THREADS? What do you suggest?

I don't think there's a hard coded max for the rgw thread pool size. A
configurable setting would be nicer, but I'm not really tied into it,
depending on what are the ramifications. Whichever route you're taking
it'd be nice to get it upstream to civetweb, so it needs to make sense
in that context.

Yehuda

Re: civetweb threads

[pushpesh sharma]

Mustafa,

I am not positive about the results you posted here(200MB/s ==>
1200MB/s, by just increasing number of 'rgw_thread_pool_size'). These
are results I observed on my setup:

1. Workload : 100% Read, Object_size=1MB, ClientWorkers: 512,
rgw_object_chunk_size= 1048576, rgw_thread_pool_size=128

General Report

Op-Type    Op-Count       Byte-Count      Avg-ResTime   Avg-ProTime
  Throughput        Bandwidth    Succ-Ratio
Read         515.99 kops   515.99 GB         74.37 ms          72.2 ms
           1721.04 op/s      1.72 GB/S     99.33%

ResTime (RT) Details
Op-Type     60%-RT    80%-RT      90%-RT   95%-RT    99%-RT        100%-RT
 read          < 30 ms   < 80 ms        < 200 ms       < 290 ms
  < 670 ms              < 6,530 ms


2.  Workload : 100% Read, Object_size=1MB, ClientWorkers: 512,
rgw_object_chunk_size= 1048576, rgw_thread_pool_size=512

General Report

Op-Type    Op-Count         Byte-Count    Avg-ResTime
Avg-ProcTime     Throughput    Bandwidth    Succ-Ratio
read          521.21 kops      521.21 GB      294.17 ms
291.7 ms           1740.46 op/s  1.74 GB/S      100%


ResTime (RT) Details
Op-Type     60%-RT     80%-RT      90%-RT      95%-RT     99%-RT        100%-RT
read          < 280 ms     < 360 ms     < 450 ms   < 570 ms  <1,040 ms
  <3,790 ms


You can checkout more setup details
http://pushpeshsharma.blogspot.in/2014/11/openstack-swift-vs-ceph-rgw-read.html
 .

 It is certainly possible I am missing some other configs, please
enlighten me.

On Sun, Nov 16, 2014 at 10:41 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> On Sun, Nov 16, 2014 at 2:51 AM, Mustafa Muhammad
> <mustafaa.alhamdaani@gmail.com> wrote:
>> On Sat, Nov 15, 2014 at 7:18 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>> On Sat, Nov 15, 2014 at 3:55 AM, Mustafa Muhammad
>>> <mustafaa.alhamdaani@gmail.com> wrote:
>>>> On Sat, Nov 15, 2014 at 10:28 AM, Mustafa Muhammad
>>>> <mustafaa.alhamdaani@gmail.com> wrote:
>>>>> Hi,
>>>>> I am using civetweb in my radosgw, if I use "rgw thread pool size"
>>>>> that is more than 1024, civetweb doesn't work.
>>>>> e.g.
>>>>> rgw thread pool size = 1024
>>>>> rgw frontends = "civetweb port=80"
>>>>>
>>>>> #ps aux -L | grep rados | wc -l
>>>>> 1096
>>>>>
>>>>> everything works fine
>>>>>
>>>>>
>>>>> If I use:
>>>>> rgw thread pool size = 1025
>>>>> rgw frontends = "civetweb port=80"
>>>>>
>>>>> # ps aux -L | grep rados | wc -l
>>>>> 43
>>>>>
>>>>> And http server is not listening.
>>>>>
>>>>> If I don't use civetweb:
>>>>> rgw thread pool size = 10240
>>>>>
>>>>> # ps aux -L | grep rados | wc -l
>>>>> 10278
>>>>>
>>>>> Regards
>>>>>
>>>>> Mustafa Muhammad
>>>>
>>>> I found the problem, it is hardcoded here:
>>>> https://github.com/ceph/civetweb/blob/master/src/civetweb.c
>>>> as:
>>>> #define MAX_WORKER_THREADS 1024
>>>>
>>>> I increased it to 20480 an compiled from source, problem solved.
>>>> I should we make a patch, right?
>>>
>>> Please do, preferably a github pull request. Also, if you could open a
>>> ceph tracker issue with the specific would be great.
>>
>> I wanted to do create a github pull request with this value set to
>> 20480, but thought I should ask if you want it to be hardcoded, I
>> understood that 'rgw_thread_pool_size' maps to 'num_op_thread' in
>> civetweb, is there a hardcoded max for rgw_thread_pool_size so I can
>> set the same for MAX_WORKER_THREADS? What do you suggest?
>
> I don't think there's a hard coded max for the rgw thread pool size. A
> configurable setting would be nicer, but I'm not really tied into it,
> depending on what are the ramifications. Whichever route you're taking
> it'd be nice to get it upstream to civetweb, so it needs to make sense
> in that context.
>
> Yehuda
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
-Pushpesh

Re: civetweb threads

[Mustafa Muhammad]

On Mon, Nov 17, 2014 at 3:03 PM, pushpesh sharma <pushpesh.eck@gmail.com> wrote:
> Mustafa,
>
> I am not positive about the results you posted here(200MB/s ==> 1200MB/s, by
> just increasing number of 'rgw_thread_pool_size'). These are results I
> observed on my setup:
>
> 1. Workload : 100% Read, Object_size=1MB, ClientWorkers: 512,
> rgw_object_chunk_size= 1048576, rgw_thread_pool_size=128
>
> General Report
>
> Op-Type    Op-Count     Byte-Count              Avg-Res  TimeAvg-Proc
> TimeThroughput   Bandwidth    Succ-Ratio
> Read         515.99 kops 515.99 GB  74.37 ms 72.2 ms 1721.04 op/s 1.72 GB/S
> 99.33%
>
> ResTime (RT) Details
> Op-Type     60%-RT    80%-RT      90%-RT   95%-RT    99%-RT        100%-RT
>  read          < 30 ms < 80 ms < 200 ms < 290 ms < 670 ms  < 6,530 ms
>
>
> 2.  Workload : 100% Read, Object_size=1MB, ClientWorkers: 512,
> rgw_object_chunk_size= 1048576, rgw_thread_pool_size=512
>
> General Report
>
> Op-Type    Op-Count         Byte-Count    Avg-Res       TimeAvg-Proc
> TimeThroughput    Bandwidth    Succ-Ratio
> read          521.21 kops 521.21 GB 294.17 ms 291.7 ms 1740.46 op/s 1.74
> GB/S 100%
>
>
> ResTime (RT) Details
> Op-Type     60%-RT     80%-RT      90%-RT      95%-RT     99%-RT
> 100%-RT
> read          < 280 ms     < 360 ms     < 450 ms   < 570 ms  <1,040 ms
> <3,790 ms
>
>
> You can checkout more setup details here  . It is certainly possible I am
> missing some other configs, please enlighten me.
>

In my case, I have a very large files (multiple gigs), I have a very
large number of connections, so I hit the "MAX_WORKER_THREADS" that is
1024 in civetweb code, so increasing the "rgw thread pool size" for me
was to increase "num_op_thread" to handle more than 1024 connection, I
could handle more than 7000 connections and I got the 1200 MB/s

In your blog, you said you used civetweb in default configuration
"‘num_op_thread’, which is set to 128", but the number is mapped from
rgw_thread_pool_size, so when you have more clients, they will wait.

Regards
Mustafa

>
> On Sun, Nov 16, 2014 at 4:33 PM, Mustafa Muhammad
> <mustafaa.alhamdaani@gmail.com> wrote:
>> On Sun, Nov 16, 2014 at 7:57 AM, pushpesh sharma <pushpesh.eck@gmail.com>
>> wrote:
>>> Yehuda,
>>>
>>> I believe it would not be wise to increase the value of this
>>> parameter; till the underlying threading model of RGW is fixed.
>>> I tried with various  'rgw_thread_pool_size' (128,256,512,1024), and
>>> didn't observe any better throughput , instead response time
>>> increased. Which means RGW is not able to handled increased
>>> concurrency.
>> I needed to increase this number to increase the number of concurrent
>> connections to civetweb, I need high number of threads (>7000
>> connections, large files)
>> I had my throughput going from 200 MB/s to 1200 MB/s (10G interface is
>> full, I could've gone higher)  due to increasing civetweb connections.
>>
>>> I have another observation regarding 'rgw_max _chunk_size'. I was
>>> doing some experiment with this parameter. I have seen the following
>>> behavior on a test deployment and then verified on development
>>> environment :-
>>>
>>> 1. Based on the value we set for 'rgw_max_chunk_size', every Swift
>>> object gets sliced in to equal sized chunks and each chunk got created
>>> as a RAODS Object in the pool (.rgw.buckets)
>>>
>>> 2. While reading the same Swift Object, sum of all RADOS objects
>>> provided.
>>>
>>> 3. However setting this 'rgw_max_chunk_size' value up to 4MB only
>>> works after which GET request will fail and only the first RADOS
>>> object will be returned.
>>>
>>> #######################
>>> $./ceph --admin-daemon out/client.admin.9660.asok config set
>>> rgw_max_chunk_size 4195000
>>> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
>>> { "success": "rgw_max_chunk_size = '4195000' "}
>>>
>>>
>>> swift -A http://localhost:8000/auth -U tester:testing -K asdf upload
>>> my4+MB_chunked ceph-0.85.tar.gz
>>> ceph-0.85.tar.gz
>>> ceph@Ubuntu14:~$ cd copy/
>>> ceph@Ubuntu14:~/copy$ swift -A http://localhost:8000/auth -U
>>> tester:testing -K asdf download my4+MB_chunked ceph-0.85.tar.gz
>>> ceph-0.85.tar.gz: md5sum != etag, ea27d35584bae391e9f3e67c86849a66 !=
>>> fd9237fa01803398bd8012c4d59975d8
>>> ceph-0.85.tar.gz [auth 0.021s, headers 0.040s, total 30.045s, 0.140 MB/s]
>>> ceph-0.85.tar.gz: read_length != content_length, 4194304 != 7384934
>>> ######################
>>>
>>> Is this behavior sane? Why the Swift Object is chunked  by default
>>> based on this value. Should the decision to chunk the object be coming
>>> from end-user? (Like Native Swift)
>>>
>>> On Sat, Nov 15, 2014 at 9:48 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>>> On Sat, Nov 15, 2014 at 3:55 AM, Mustafa Muhammad
>>>> <mustafaa.alhamdaani@gmail.com> wrote:
>>>>> On Sat, Nov 15, 2014 at 10:28 AM, Mustafa Muhammad
>>>>> <mustafaa.alhamdaani@gmail.com> wrote:
>>>>>> Hi,
>>>>>> I am using civetweb in my radosgw, if I use "rgw thread pool size"
>>>>>> that is more than 1024, civetweb doesn't work.
>>>>>> e.g.
>>>>>> rgw thread pool size = 1024
>>>>>> rgw frontends = "civetweb port=80"
>>>>>>
>>>>>> #ps aux -L | grep rados | wc -l
>>>>>> 1096
>>>>>>
>>>>>> everything works fine
>>>>>>
>>>>>>
>>>>>> If I use:
>>>>>> rgw thread pool size = 1025
>>>>>> rgw frontends = "civetweb port=80"
>>>>>>
>>>>>> # ps aux -L | grep rados | wc -l
>>>>>> 43
>>>>>>
>>>>>> And http server is not listening.
>>>>>>
>>>>>> If I don't use civetweb:
>>>>>> rgw thread pool size = 10240
>>>>>>
>>>>>> # ps aux -L | grep rados | wc -l
>>>>>> 10278
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Mustafa Muhammad
>>>>>
>>>>> I found the problem, it is hardcoded here:
>>>>> https://github.com/ceph/civetweb/blob/master/src/civetweb.c
>>>>> as:
>>>>> #define MAX_WORKER_THREADS 1024
>>>>>
>>>>> I increased it to 20480 an compiled from source, problem solved.
>>>>> I should we make a patch, right?
>>>>
>>>> Please do, preferably a github pull request. Also, if you could open a
>>>> ceph tracker issue with the specific would be great.
>>>>
>>>> Thanks,
>>>> Yehuda
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>
>>> --
>>> -Pushpesh
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> -Pushpesh
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: civetweb threads

[Mustafa Muhammad]

On Sat, Nov 15, 2014 at 7:18 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> On Sat, Nov 15, 2014 at 3:55 AM, Mustafa Muhammad
> <mustafaa.alhamdaani@gmail.com> wrote:
>> On Sat, Nov 15, 2014 at 10:28 AM, Mustafa Muhammad
>> <mustafaa.alhamdaani@gmail.com> wrote:
>>> Hi,
>>> I am using civetweb in my radosgw, if I use "rgw thread pool size"
>>> that is more than 1024, civetweb doesn't work.
>>> e.g.
>>> rgw thread pool size = 1024
>>> rgw frontends = "civetweb port=80"
>>>
>>> #ps aux -L | grep rados | wc -l
>>> 1096
>>>
>>> everything works fine
>>>
>>>
>>> If I use:
>>> rgw thread pool size = 1025
>>> rgw frontends = "civetweb port=80"
>>>
>>> # ps aux -L | grep rados | wc -l
>>> 43
>>>
>>> And http server is not listening.
>>>
>>> If I don't use civetweb:
>>> rgw thread pool size = 10240
>>>
>>> # ps aux -L | grep rados | wc -l
>>> 10278
>>>
>>> Regards
>>>
>>> Mustafa Muhammad
>>
>> I found the problem, it is hardcoded here:
>> https://github.com/ceph/civetweb/blob/master/src/civetweb.c
>> as:
>> #define MAX_WORKER_THREADS 1024
>>
>> I increased it to 20480 an compiled from source, problem solved.
>> I should we make a patch, right?
>
> Please do, preferably a github pull request. Also, if you could open a
> ceph tracker issue with the specific would be great.
Hi,
I opened an issue upstream:
https://github.com/bel2125/civetweb/issues/50
And this default is going to change to 64k using this pull request:
https://github.com/bel2125/civetweb/commit/23bdcc98dd72ac421ed0a9e82d66977089fbfe63

I should make a pull request and issue here too, right?
I tried to register with ceph issue tracker, but misspelled the email
address (wrote @gmai.com), now I can't take the name "mustafa" again
(I waited the request to expire, but until now I can't register).
Who can delete that wrong registration request so I can take this
username again?

Regards
Mustafa
>
> Thanks,
> Yehuda

Re: civetweb threads

[Dan Mick]

> I tried to register with ceph issue tracker, but misspelled the email
> address (wrote @gmai.com), now I can't take the name "mustafa" again
> (I waited the request to expire, but until now I can't register).
> Who can delete that wrong registration request so I can take this
> username again?

Done; please contact me if reregistration doesn't work

Re: civetweb threads

[Mustafa Muhammad]

On Tue, Dec 2, 2014 at 12:56 AM, Dan Mick <dan.mick@inktank.com> wrote:
>
>> I tried to register with ceph issue tracker, but misspelled the email
>> address (wrote @gmai.com), now I can't take the name "mustafa" again
>> (I waited the request to expire, but until now I can't register).
>> Who can delete that wrong registration request so I can take this
>> username again?
>
> Done; please contact me if reregistration doesn't work

Thank you very much, it worked.

I created an issue:
http://tracker.ceph.com/issues/10243

I tried to make a pull request for your copy but after forking, I
noticed that the pull request is for the deprecated repo:
https://github.com/sunsetbrew/civetweb
Not the active upstream:
https://github.com/bel2125/civetweb
I think this should be fixed too.

Regrads
Mustafa