Re: teuthology : 70 workers need more than 8GB RAM / 2 CPUS

[Loic Dachary <loic@dachary.org>]

[-- Attachment #1: Type: text/plain, Size: 2693 bytes --]

Thanks for the feedback. I'll try with postgresql as it seems the sqlite modifications did nothing really significant.

On 20/07/2015 17:38, Zack Cerza wrote:
> Hi Loic,
> 
> This is definitely something to keep an eye on. It's actually a bit surprising to me, though - I haven't seen ansible-playbook use any significant resources in sepia.
> 
> I wouldn't really recommend running paddles on the same host as teuthology though, to do any serious amount of testing; some teuthology tasks do use large amounts of RAM and/or CPU, and severe load issues could feasibly cause requests to time out, affecting other jobs.
> 
> That's all theory though, as I've always used separate hosts for the two services.
> 
> Zack
> 
> ----- Original Message -----
>> From: "Loic Dachary" <loic@dachary.org>
>> To: "Zack Cerza" <zack@redhat.com>, "Andrew Schoen" <aschoen@redhat.com>
>> Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
>> Sent: Sunday, July 19, 2015 9:06:41 AM
>> Subject: Re: teuthology : 70 workers need more than 8GB RAM / 2 CPUS
>>
>> Hi again,
>>
>> I had the same problem when 50 workers kick in at the same time. I've lowered
>> the number of workers down to 25 and it went well. During a few minutes (~8
>> minutes) the load average stayed around 25 (CPU bound, mainly the ansible
>> playbook competing, see the screenshot of htop). But did not see any error /
>> timeout. then I added 15 workers, wait for the load to go back to < 2 (10
>> minutes), then 15 more (10 minutes) to get to 55.
>>
>> That sound like a log of CPU used by a single playbook run. Is there a known
>> way to reduce that ? If not I'll just upgrade the machine. Just want to make
>> sure I'm not missing a simple solution ;-)
>>
>> Cheers
>>
>> On 19/07/2015 14:22, Loic Dachary wrote:
>>> Hi,
>>>
>>> For the record, I launched a rados suite on an idle teuthology cluster,
>>> with 70 workers running on a 8GB RAM / 2 CPUS / 40GB SSD disk. The load
>>> average reached 40 within a minute or two and some jobs started failing /
>>> timeouting. I had pulpito running on the same machine and it failed one
>>> time out of two because of the load (see the top image).
>>>
>>> On friday I was able to run 70 workers because I gradually added them. The
>>> load peak is when a job starts and all workers kick in a the same time.
>>>
>>> Cheers
>>>
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]