* Ceph Community Lab: Understanding where the QA suites spend time @ 2015-08-19 19:46 Mark Nelson 2015-08-19 22:05 ` [Cbt] " Loic Dachary 0 siblings, 1 reply; 5+ messages in thread From: Mark Nelson @ 2015-08-19 19:46 UTC (permalink / raw) To: ceph-devel, cbt Hi Guys, About a month ago we were going through the process of trying to figure out how to replace some of the hardware in the community laboratory that runs all of the nightly Teuthology tests. Given a limited budget to replace the existing nodes, we wanted to understand how the current QA suites actually spend time on the hardware. To do this, we investigated the composition of suites, the amount of time that is spent in each suite, and then a deeper dive into how the most resource intensive jobs spend their time. We then wrote a new benchmark for CBT to run ceph-test-rados in a rather naive reproduction of what the ceph task in teuthology does and tested it against several different storage device configurations to see how much benefit SSDs in the nodes may provide. A couple of folks at the Hackathon were interested in the paper we wrote. I thought I would share it with the community as well in case any folks ever wondered what Teuthology actually spends it's time doing. http://nhm.ceph.com/community/Ceph_Community_Lab_Performance_Investigation.pdf Thanks, Mark ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Cbt] Ceph Community Lab: Understanding where the QA suites spend time 2015-08-19 19:46 Ceph Community Lab: Understanding where the QA suites spend time Mark Nelson @ 2015-08-19 22:05 ` Loic Dachary 2015-08-19 22:08 ` Mark Nelson 0 siblings, 1 reply; 5+ messages in thread From: Loic Dachary @ 2015-08-19 22:05 UTC (permalink / raw) To: Mark Nelson, ceph-devel, cbt [-- Attachment #1: Type: text/plain, Size: 2329 bytes --] Hi, To help put things in perspective (please forgive me if it's a fact already known to everyone), > 95% of the teuthology jobs can run on virtual machines with 8GB RAM, 40GB disk, 2 processors and no attached disks. The cost of running such virtual machines in the cloud currently is 0.033 euros per hour. Running 40 teuthology jobs at all times would require an average of 80 such virtual machines (most jobs use 2 machines, some 1 or 3), that is 0.033 * 80 vms * 24 hours * 31 days == ~2000 euros per month (also note that at least one cloud provider offers 50% discount if such virtual machines are reserved full time). That's 3 to 6 times cheaper than AWS. I'm not sure how much it costs to run these 40 teuthology jobs on a hand made lab but I'd be surprised if it was cheaper, all included. Whatever hardware already exists in the sepia lab can easily handle the remaining 5% of jobs that require actual hardware for one reason or the other. My 3.3cts ;-) On 19/08/2015 21:46, Mark Nelson wrote: > Hi Guys, > > About a month ago we were going through the process of trying to figure out how to replace some of the hardware in the community laboratory that runs all of the nightly Teuthology tests. Given a limited budget to replace the existing nodes, we wanted to understand how the current QA suites actually spend time on the hardware. To do this, we investigated the composition of suites, the amount of time that is spent in each suite, and then a deeper dive into how the most resource intensive jobs spend their time. We then wrote a new benchmark for CBT to run ceph-test-rados in a rather naive reproduction of what the ceph task in teuthology does and tested it against several different storage device configurations to see how much benefit SSDs in the nodes may provide. > > A couple of folks at the Hackathon were interested in the paper we wrote. I thought I would share it with the community as well in case any folks ever wondered what Teuthology actually spends it's time doing. > > http://nhm.ceph.com/community/Ceph_Community_Lab_Performance_Investigation.pdf > > Thanks, > Mark > _______________________________________________ > Cbt mailing list > Cbt@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/cbt-ceph.com -- Loïc Dachary, Artisan Logiciel Libre [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Cbt] Ceph Community Lab: Understanding where the QA suites spend time 2015-08-19 22:05 ` [Cbt] " Loic Dachary @ 2015-08-19 22:08 ` Mark Nelson 2015-08-19 22:22 ` Loic Dachary 0 siblings, 1 reply; 5+ messages in thread From: Mark Nelson @ 2015-08-19 22:08 UTC (permalink / raw) To: Loic Dachary, ceph-devel, cbt I have no idea about the price of cloud computing services, but if you can find setups that give you access to SSDs you may be able to run through thrasher tests much faster (cheaper?). :) Mark On 08/19/2015 05:05 PM, Loic Dachary wrote: > Hi, > > To help put things in perspective (please forgive me if it's a fact already known to everyone), > 95% of the teuthology jobs can run on virtual machines with 8GB RAM, 40GB disk, 2 processors and no attached disks. The cost of running such virtual machines in the cloud currently is 0.033 euros per hour. Running 40 teuthology jobs at all times would require an average of 80 such virtual machines (most jobs use 2 machines, some 1 or 3), that is 0.033 * 80 vms * 24 hours * 31 days == ~2000 euros per month (also note that at least one cloud provider offers 50% discount if such virtual machines are reserved full time). That's 3 to 6 times cheaper than AWS. > > I'm not sure how much it costs to run these 40 teuthology jobs on a hand made lab but I'd be surprised if it was cheaper, all included. Whatever hardware already exists in the sepia lab can easily handle the remaining 5% of jobs that require actual hardware for one reason or the other. > > My 3.3cts ;-) > > On 19/08/2015 21:46, Mark Nelson wrote: >> Hi Guys, >> >> About a month ago we were going through the process of trying to figure out how to replace some of the hardware in the community laboratory that runs all of the nightly Teuthology tests. Given a limited budget to replace the existing nodes, we wanted to understand how the current QA suites actually spend time on the hardware. To do this, we investigated the composition of suites, the amount of time that is spent in each suite, and then a deeper dive into how the most resource intensive jobs spend their time. We then wrote a new benchmark for CBT to run ceph-test-rados in a rather naive reproduction of what the ceph task in teuthology does and tested it against several different storage device configurations to see how much benefit SSDs in the nodes may provide. >> >> A couple of folks at the Hackathon were interested in the paper we wrote. I thought I would share it with the community as well in case any folks ever wondered what Teuthology actually spends it's time doing. >> >> http://nhm.ceph.com/community/Ceph_Community_Lab_Performance_Investigation.pdf >> >> Thanks, >> Mark >> _______________________________________________ >> Cbt mailing list >> Cbt@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/cbt-ceph.com > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Cbt] Ceph Community Lab: Understanding where the QA suites spend time 2015-08-19 22:08 ` Mark Nelson @ 2015-08-19 22:22 ` Loic Dachary 2015-08-19 22:36 ` Shinobu Kinjo 0 siblings, 1 reply; 5+ messages in thread From: Loic Dachary @ 2015-08-19 22:22 UTC (permalink / raw) To: Mark Nelson, ceph-devel, cbt [-- Attachment #1: Type: text/plain, Size: 3001 bytes --] On 20/08/2015 00:08, Mark Nelson wrote:> I have no idea about the price of cloud computing services, but if you can find setups that give you access to SSDs you may be able to run through thrasher tests much faster (cheaper?). :) In my experience (~2 month experience, not much really ;-), VMs are twice slower than bare metal on average (sometime it's only sightly slower, sometime it's 3 times slower). The network variance also counts : I've seen providers take ~20 minutes to install the Ceph packages (it's ~2GB altogether). My 3.3 * ~2 cts ;-) > Mark > > On 08/19/2015 05:05 PM, Loic Dachary wrote: >> Hi, >> >> To help put things in perspective (please forgive me if it's a fact already known to everyone), > 95% of the teuthology jobs can run on virtual machines with 8GB RAM, 40GB disk, 2 processors and no attached disks. The cost of running such virtual machines in the cloud currently is 0.033 euros per hour. Running 40 teuthology jobs at all times would require an average of 80 such virtual machines (most jobs use 2 machines, some 1 or 3), that is 0.033 * 80 vms * 24 hours * 31 days == ~2000 euros per month (also note that at least one cloud provider offers 50% discount if such virtual machines are reserved full time). That's 3 to 6 times cheaper than AWS. >> >> I'm not sure how much it costs to run these 40 teuthology jobs on a hand made lab but I'd be surprised if it was cheaper, all included. Whatever hardware already exists in the sepia lab can easily handle the remaining 5% of jobs that require actual hardware for one reason or the other. >> >> My 3.3cts ;-) >> >> On 19/08/2015 21:46, Mark Nelson wrote: >>> Hi Guys, >>> >>> About a month ago we were going through the process of trying to figure out how to replace some of the hardware in the community laboratory that runs all of the nightly Teuthology tests. Given a limited budget to replace the existing nodes, we wanted to understand how the current QA suites actually spend time on the hardware. To do this, we investigated the composition of suites, the amount of time that is spent in each suite, and then a deeper dive into how the most resource intensive jobs spend their time. We then wrote a new benchmark for CBT to run ceph-test-rados in a rather naive reproduction of what the ceph task in teuthology does and tested it against several different storage device configurations to see how much benefit SSDs in the nodes may provide. >>> >>> A couple of folks at the Hackathon were interested in the paper we wrote. I thought I would share it with the community as well in case any folks ever wondered what Teuthology actually spends it's time doing. >>> >>> http://nhm.ceph.com/community/Ceph_Community_Lab_Performance_Investigation.pdf >>> >>> Thanks, >>> Mark >>> _______________________________________________ >>> Cbt mailing list >>> Cbt@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/cbt-ceph.com >> -- Loïc Dachary, Artisan Logiciel Libre [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Cbt] Ceph Community Lab: Understanding where the QA suites spend time 2015-08-19 22:22 ` Loic Dachary @ 2015-08-19 22:36 ` Shinobu Kinjo 0 siblings, 0 replies; 5+ messages in thread From: Shinobu Kinjo @ 2015-08-19 22:36 UTC (permalink / raw) To: Loic Dachary; +Cc: Mark Nelson, ceph-devel, cbt > The network variance also counts : Yes, it's quite true. We have to consider of that part. Shinobu ----- Original Message ----- From: "Loic Dachary" <loic@dachary.org> To: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>, cbt@lists.ceph.com Sent: Thursday, August 20, 2015 7:22:40 AM Subject: Re: [Cbt] Ceph Community Lab: Understanding where the QA suites spend time On 20/08/2015 00:08, Mark Nelson wrote:> I have no idea about the price of cloud computing services, but if you can find setups that give you access to SSDs you may be able to run through thrasher tests much faster (cheaper?). :) In my experience (~2 month experience, not much really ;-), VMs are twice slower than bare metal on average (sometime it's only sightly slower, sometime it's 3 times slower). The network variance also counts : I've seen providers take ~20 minutes to install the Ceph packages (it's ~2GB altogether). My 3.3 * ~2 cts ;-) > Mark > > On 08/19/2015 05:05 PM, Loic Dachary wrote: >> Hi, >> >> To help put things in perspective (please forgive me if it's a fact already known to everyone), > 95% of the teuthology jobs can run on virtual machines with 8GB RAM, 40GB disk, 2 processors and no attached disks. The cost of running such virtual machines in the cloud currently is 0.033 euros per hour. Running 40 teuthology jobs at all times would require an average of 80 such virtual machines (most jobs use 2 machines, some 1 or 3), that is 0.033 * 80 vms * 24 hours * 31 days == ~2000 euros per month (also note that at least one cloud provider offers 50% discount if such virtual machines are reserved full time). That's 3 to 6 times cheaper than AWS. >> >> I'm not sure how much it costs to run these 40 teuthology jobs on a hand made lab but I'd be surprised if it was cheaper, all included. Whatever hardware already exists in the sepia lab can easily handle the remaining 5% of jobs that require actual hardware for one reason or the other. >> >> My 3.3cts ;-) >> >> On 19/08/2015 21:46, Mark Nelson wrote: >>> Hi Guys, >>> >>> About a month ago we were going through the process of trying to figure out how to replace some of the hardware in the community laboratory that runs all of the nightly Teuthology tests. Given a limited budget to replace the existing nodes, we wanted to understand how the current QA suites actually spend time on the hardware. To do this, we investigated the composition of suites, the amount of time that is spent in each suite, and then a deeper dive into how the most resource intensive jobs spend their time. We then wrote a new benchmark for CBT to run ceph-test-rados in a rather naive reproduction of what the ceph task in teuthology does and tested it against several different storage device configurations to see how much benefit SSDs in the nodes may provide. >>> >>> A couple of folks at the Hackathon were interested in the paper we wrote. I thought I would share it with the community as well in case any folks ever wondered what Teuthology actually spends it's time doing. >>> >>> http://nhm.ceph.com/community/Ceph_Community_Lab_Performance_Investigation.pdf >>> >>> Thanks, >>> Mark >>> _______________________________________________ >>> Cbt mailing list >>> Cbt@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/cbt-ceph.com >> -- Loïc Dachary, Artisan Logiciel Libre -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-08-19 22:37 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-08-19 19:46 Ceph Community Lab: Understanding where the QA suites spend time Mark Nelson 2015-08-19 22:05 ` [Cbt] " Loic Dachary 2015-08-19 22:08 ` Mark Nelson 2015-08-19 22:22 ` Loic Dachary 2015-08-19 22:36 ` Shinobu Kinjo
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.