From mboxrd@z Thu Jan 1 00:00:00 1970 From: Loic Dachary Subject: Re: upgrade:firefly-x:stress-split related workload Date: Tue, 05 Aug 2014 21:29:09 +0200 Message-ID: <53E13085.8060602@dachary.org> References: <53DF8672.2050109@dachary.org> <53DFAFA5.8040606@dachary.org> <53E07964.1080809@dachary.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="deG4eRMMXb14uJH0jq10xWN3SHPgPIAVA" Return-path: Received: from mail2.dachary.org ([91.121.57.175]:54370 "EHLO smtp.dmail.dachary.org" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753559AbaHET3Q (ORCPT ); Tue, 5 Aug 2014 15:29:16 -0400 In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Yuri Weinstein Cc: Ceph Development This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --deG4eRMMXb14uJH0jq10xWN3SHPgPIAVA Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Yuri, It already has an ec_pool workload at https://github.com/ceph/ceph-qa-suite/blob/master/suites/upgrade/firef= ly-x/stress-split/5-workload/ec-readwrite.yaml Cheers On 05/08/2014 19:25, Yuri Weinstein wrote: > BTW, I have a suite upgrade/dumpling-firefly-x, which all passed yester= day. >=20 > How can EC be enabled for it? Do you want to try EC on it? >=20 > Thx > YuriW >=20 > On Tue, Aug 5, 2014 at 10:05 AM, Yuri Weinstein > wrote: >> What is really disappointing when failed tests pass - 396983 passed. >> >> ??? >> >> On Tue, Aug 5, 2014 at 8:26 AM, Yuri Weinstein >> wrote: >>> Yeah, that's possible. >>> >>> I am re-trying manually now 396893 >>> (http://qa-proxy.ceph.com/teuthology/loic-2014-08-04_15:06:02-upgrade= :firefly-x:stress-split-wip-8475-testing-basic-plana/396893/orig.config.y= aml) >>> >>> On Mon, Aug 4, 2014 at 11:27 PM, Loic Dachary wrot= e: >>>> Hi, >>>> >>>> The test suite finished at produced 5 errors, 4 of them in the "SWIF= T_TEST_CONFIG_FILE" category >>>> >>>> http://pulpito.ceph.com/loic-2014-08-04_15:06:02-upgrade:firefly-x:s= tress-split-wip-8475-testing-basic-plana/396882/ >>>> INFO:teuthology.orchestra.run.plana28.stderr:-----------------------= ----------------------------------------------- >>>> 2014-08-04T10:54:48.138 INFO:teuthology.orchestra.run.plana28.stderr= :Traceback (most recent call last): >>>> 2014-08-04T10:54:48.138 INFO:teuthology.orchestra.run.plana28.stderr= : File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104,= in setUp >>>> 2014-08-04T10:54:48.138 INFO:teuthology.orchestra.run.plana28.stderr= : cls.env.setUp() >>>> 2014-08-04T10:54:48.139 INFO:teuthology.orchestra.run.plana28.stderr= : File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 140,= in setUp >>>> 2014-08-04T10:54:48.139 INFO:teuthology.orchestra.run.plana28.stderr= : raise ResponseError(cls.conn.response) >>>> 2014-08-04T10:54:48.139 INFO:teuthology.orchestra.run.plana28.stderr= :ResponseError: 500: Internal Server Error >>>> 2014-08-04T10:54:48.139 INFO:teuthology.orchestra.run.plana28.stderr= : >>>> 2014-08-04T10:54:48.139 INFO:teuthology.orchestra.run.plana28.stderr= :=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>> 2014-08-04T10:54:48.139 INFO:teuthology.orchestra.run.plana28.stderr= :FAIL: testGetRequest (test.functional.tests.TestAccountNoContainers) >>>> 2014-08-04T10:54:48.139 INFO:teuthology.orchestra.run.plana28.stderr= :---------------------------------------------------------------------- >>>> 2014-08-04T10:54:48.139 INFO:teuthology.orchestra.run.plana28.stderr= :Traceback (most recent call last): >>>> 2014-08-04T10:54:48.139 INFO:teuthology.orchestra.run.plana28.stderr= : File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 325,= in testGetRequest >>>> 2014-08-04T10:54:48.140 INFO:teuthology.orchestra.run.plana28.stderr= : parms=3D{'format':format})) >>>> 2014-08-04T10:54:48.140 INFO:teuthology.orchestra.run.plana28.stderr= :AssertionError: False is not true >>>> 2014-08-04T10:54:48.140 INFO:teuthology.orchestra.run.plana28.stderr= : >>>> 2014-08-04T10:54:48.140 >>>> >>>> http://pulpito.ceph.com/loic-2014-08-04_15:06:02-upgrade:firefly-x:s= tress-split-wip-8475-testing-basic-plana/396890/ >>>> INFO:teuthology.orchestra.run.plana64.stderr:=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>> 2014-08-04T12:51:21.582 INFO:teuthology.orchestra.run.plana64.stderr= :FAIL: testUtf8Container (test.functional.tests.TestContainerUTF8) >>>> 2014-08-04T12:51:21.582 INFO:teuthology.orchestra.run.plana64.stderr= :---------------------------------------------------------------------- >>>> 2014-08-04T12:51:21.582 INFO:teuthology.orchestra.run.plana64.stderr= :Traceback (most recent call last): >>>> 2014-08-04T12:51:21.583 INFO:teuthology.orchestra.run.plana64.stderr= : File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 462,= in testUtf8Container >>>> 2014-08-04T12:51:21.583 INFO:teuthology.orchestra.run.plana64.stderr= : self.assert_(container.create(cfg=3D{'no_path_quote':True})) >>>> 2014-08-04T12:51:21.583 INFO:teuthology.orchestra.run.plana64.stderr= :AssertionError: False is not true >>>> 2014-08-04T12:51:21.583 INFO:teuthology.orchestra.run.plana64.stderr= : >>>> 2014-08-04T12:51:21.583 >>>> >>>> http://pulpito.ceph.com/loic-2014-08-04_15:06:02-upgrade:firefly-x:s= tress-split-wip-8475-testing-basic-plana/396891/ >>>> INFO:teuthology.orchestra.run.plana12.stderr:=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>> 2014-08-04T13:36:17.551 INFO:teuthology.orchestra.run.plana12.stderr= :FAIL: testSerialization (test.functional.tests.TestFile) >>>> 2014-08-04T13:36:17.552 INFO:teuthology.orchestra.run.plana12.stderr= :---------------------------------------------------------------------- >>>> 2014-08-04T13:36:17.552 INFO:teuthology.orchestra.run.plana12.stderr= :Traceback (most recent call last): >>>> 2014-08-04T13:36:17.552 INFO:teuthology.orchestra.run.plana12.stderr= : File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 1315= , in testSerialization >>>> 2014-08-04T13:36:17.552 INFO:teuthology.orchestra.run.plana12.stderr= : self.assert_(container.create()) >>>> 2014-08-04T13:36:17.552 INFO:teuthology.orchestra.run.plana12.stderr= :AssertionError: False is not true >>>> 2014-08-04T13:36:17.553 INFO:teuthology.orchestra.run.plana12.stderr= : >>>> 2014-08-04T13:36:17.553 >>>> >>>> http://pulpito.ceph.com/loic-2014-08-04_15:06:02-upgrade:firefly-x:s= tress-split-wip-8475-testing-basic-plana/396893/ >>>> INFO:teuthology.orchestra.run.plana34.stderr:=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>> 2014-08-04T12:59:44.078 INFO:teuthology.orchestra.run.plana34.stderr= :FAIL: testPrefixAndLimit (test.functional.tests.TestContainer) >>>> 2014-08-04T12:59:44.078 INFO:teuthology.orchestra.run.plana34.stderr= :---------------------------------------------------------------------- >>>> 2014-08-04T12:59:44.078 INFO:teuthology.orchestra.run.plana34.stderr= :Traceback (most recent call last): >>>> 2014-08-04T12:59:44.078 INFO:teuthology.orchestra.run.plana34.stderr= : File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 415,= in testPrefixAndLimit >>>> 2014-08-04T12:59:44.079 INFO:teuthology.orchestra.run.plana34.stderr= : self.assert_(cont.create()) >>>> 2014-08-04T12:59:44.079 INFO:teuthology.orchestra.run.plana34.stderr= :AssertionError: False is not true >>>> 2014-08-04T12:59:44.079 INFO:teuthology.orchestra.run.plana34.stderr= : >>>> 2014-08-04T12:59:44.079 >>>> >>>> one of them is because of a failed kill ceph-osd -f -i 5 >>>> >>>> http://pulpito.ceph.com/loic-2014-08-04_15:06:02-upgrade:firefly-x:s= tress-split-wip-8475-testing-basic-plana/396887/ >>>> >>>> which I suspect to be related to http://tracker.ceph.com/issues/8736= >>>> >>>> It strikes me as odd that all radosgw failures are different. As if = something was preventing, randomly, the test to complete. Maybe a combina= tion of running the test against a cluster that is stopped before the tes= t can complete ? >>>> >>>> Cheers >>>> >>>> On 04/08/2014 18:13, Yuri Weinstein wrote: >>>>> Not yet. I was wondering if you cold take a look after your run is= >>>>> completed (we had mane environment related and backport related >>>>> issues, so we need to get trustworthy runs)? >>>>> >>>>> But I did talk to Yehuda and logged >>>>> http://tracker.ceph.com/issues/8988 which maybe covered a couple of= >>>>> them. >>>>> >>>>> Thx >>>>> YuriW >>>>> >>>>> On Mon, Aug 4, 2014 at 9:07 AM, Loic Dachary wro= te: >>>>>> >>>>>> >>>>>> On 04/08/2014 16:54, Yuri Weinstein wrote: >>>>>>> Sounds good, Loic. >>>>>>> >>>>>>> I ran 60 tests on vps - >>>>>>> http://pulpito.front.sepia.ceph.com/teuthology-2014-08-03_16:05:0= 1-upgrade:firefly-x:stress-split-next---basic-vps/ >>>>>>> >>>>>>> I will need your help to see what to do about on ## >>>>>>> http://tracker.ceph.com/issues/8736, >>>>>>> http://tracker.ceph.com/issues/8737 and >>>>>>> http://tracker.ceph.com/issues/8740 >>>>>> >>>>>> Sure. Did you match them to errors from the 19 failures of http://= pulpito.front.sepia.ceph.com/teuthology-2014-08-03_16:05:01-upgrade:firef= ly-x:stress-split-next---basic-vps/ ? >>>>>>> >>>>>>> Thx >>>>>>> YuriW >>>>>>> >>>>>>> On Mon, Aug 4, 2014 at 6:11 AM, Loic Dachary w= rote: >>>>>>>> Hi Yuri, >>>>>>>> >>>>>>>> FYI I scheduled a upgrade:firefly-x:stress-split to check if htt= p://tracker.ceph.com/issues/8475#note-14 behaves as expected http://pulpi= to.ceph.com/loic-2014-08-04_15:06:02-upgrade:firefly-x:stress-split-wip-8= 475-testing-basic-plana/ >>>>>>>> >>>>>>>> I'm hoping to see the same failures you have on master on next h= ttp://pulpito.ceph.com/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:s= tress-split-next---basic-plana/ >>>>>>>> >>>>>>>> Cheers >>>>>>>> -- >>>>>>>> Lo=C3=AFc Dachary, Artisan Logiciel Libre >>>>>>>> >>>>>> >>>>>> -- >>>>>> Lo=C3=AFc Dachary, Artisan Logiciel Libre >>>>>> >>>> >>>> -- >>>> Lo=C3=AFc Dachary, Artisan Logiciel Libre >>>> --=20 Lo=C3=AFc Dachary, Artisan Logiciel Libre --deG4eRMMXb14uJH0jq10xWN3SHPgPIAVA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlPhMIUACgkQ8dLMyEl6F20frgCcCNnNtpKyQa7nn8hRLQHhTDsW U3YAn2nWpN0KUNjc3661BABVrXW6oZ8J =uwIH -----END PGP SIGNATURE----- --deG4eRMMXb14uJH0jq10xWN3SHPgPIAVA--