ceph v10.2.4 QE validation status

ceph v10.2.4 QE validation status

[Yuri Weinstein]

Detailed summary of the QE Validation can be found here
http://tracker.ceph.com/issues/17487#note-32

The following suites were in scope of this point release validation:

rados (subset 35/50 297 jobs)
rbd
rgw
fs
krbd
kcephfs
knfs
rest
hadoop
samba
ceph-deploy
ceph-disk
upgrade/client-upgrade
upgrade/hammer-x (jewel)
upgrade/infernalis-x (jewel) - deprecated
powercycle
(please let me know if any suites are missing from this ^ list)

==============================
Issues requiring approval/decision:

fs - http://tracker.ceph.com/issues/17832 (Greg pls review/approve)

krbd - http://tracker.ceph.com/issues/17221 (Jason, Ilya pls review/approve)

knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
pls review/approve)

upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
pls review/approve)

upgrade/infernalis-x (jewel) - deprecated, but Nathan is still working
to make it pass (Sage pls review/approve)

powercycle - in progress

Thx
YuriW

Re: ceph v10.2.4 QE validation status

[Ilya Dryomov]

On Fri, Nov 11, 2016 at 5:47 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Detailed summary of the QE Validation can be found here
> http://tracker.ceph.com/issues/17487#note-32
>
> The following suites were in scope of this point release validation:
>
> rados (subset 35/50 297 jobs)
> rbd
> rgw
> fs
> krbd
> kcephfs
> knfs
> rest
> hadoop
> samba
> ceph-deploy
> ceph-disk
> upgrade/client-upgrade
> upgrade/hammer-x (jewel)
> upgrade/infernalis-x (jewel) - deprecated
> powercycle
> (please let me know if any suites are missing from this ^ list)
>
> ==============================
> Issues requiring approval/decision:
>
> fs - http://tracker.ceph.com/issues/17832 (Greg pls review/approve)
>
> krbd - http://tracker.ceph.com/issues/17221 (Jason, Ilya pls review/approve)

Approved.

Thanks,

                Ilya

Re: ceph v10.2.4 QE validation status

[John Spray]

On Fri, Nov 11, 2016 at 4:47 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Detailed summary of the QE Validation can be found here
> http://tracker.ceph.com/issues/17487#note-32
>
> The following suites were in scope of this point release validation:
>
> rados (subset 35/50 297 jobs)
> rbd
> rgw
> fs
> krbd
> kcephfs
> knfs
> rest
> hadoop
> samba
> ceph-deploy
> ceph-disk
> upgrade/client-upgrade
> upgrade/hammer-x (jewel)
> upgrade/infernalis-x (jewel) - deprecated
> powercycle
> (please let me know if any suites are missing from this ^ list)
>
> ==============================
> Issues requiring approval/decision:
>
> fs - http://tracker.ceph.com/issues/17832 (Greg pls review/approve)

OK.  BTW feel free to backport the patch from master that disables the
unstable flock tests.

John

> krbd - http://tracker.ceph.com/issues/17221 (Jason, Ilya pls review/approve)
>
> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
> pls review/approve)
>
> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
> pls review/approve)
>
> upgrade/infernalis-x (jewel) - deprecated, but Nathan is still working
> to make it pass (Sage pls review/approve)
>
> powercycle - in progress
>
> Thx
> YuriW

Re: ceph v10.2.4 QE validation status

[Nathan Cutler]

>> fs - http://tracker.ceph.com/issues/17832 (Greg pls review/approve)
>
> OK.  BTW feel free to backport the patch from master that disables the
> unstable flock tests.

Here's the backport:

* http://tracker.ceph.com/issues/17885
* https://github.com/ceph/ceph/pull/11953

Nathan

Re: ceph v10.2.4 QE validation status

[Yuri Weinstein]

See updated status - http://tracker.ceph.com/issues/17487#note-32

Outstanding issues:

knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
pls review/approve, assumed Approved ?)

upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
pls review/approve, seems persistent, but maybe not a showstopper)

upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
to make it pass, see issues in the tacker summary above)

Sage, jewel 10.2.4 can be released as soon as you agree with the
findings/summary.

Thx
YuriW

On Fri, Nov 11, 2016 at 8:47 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Detailed summary of the QE Validation can be found here
> http://tracker.ceph.com/issues/17487#note-32
>
> The following suites were in scope of this point release validation:
>
> rados (subset 35/50 297 jobs)
> rbd
> rgw
> fs
> krbd
> kcephfs
> knfs
> rest
> hadoop
> samba
> ceph-deploy
> ceph-disk
> upgrade/client-upgrade
> upgrade/hammer-x (jewel)
> upgrade/infernalis-x (jewel) - deprecated
> powercycle
> (please let me know if any suites are missing from this ^ list)
>
> ==============================
> Issues requiring approval/decision:
>
> fs - http://tracker.ceph.com/issues/17832 (Greg pls review/approve)
>
> krbd - http://tracker.ceph.com/issues/17221 (Jason, Ilya pls review/approve)
>
> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
> pls review/approve)
>
> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
> pls review/approve)
>
> upgrade/infernalis-x (jewel) - deprecated, but Nathan is still working
> to make it pass (Sage pls review/approve)
>
> powercycle - in progress
>
> Thx
> YuriW

Re: ceph v10.2.4 QE validation status

[Nathan Cutler]

> upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
> to make it pass, see issues in the tacker summary above)

Actually, I'm no longer working on it. Two fixes were merged that 
brought the run from 100% fail to 8/11 passing. This is the latest run:

http://pulpito.ceph.com/yuriw-2016-11-14_16:21:41-upgrade:infernalis-x-jewel-distro-basic-vps/

I didn't look, but based on previous runs the two fails are MON crashes 
and the dead job is an unknown (there tends to be one dead job in every 
run, but not always in the same test).

Perhaps a developer could look at those three?

Nathan

Re: ceph v10.2.4 QE validation status

[Abhishek L]

Hi Sage, Greg,


Yuri Weinstein writes:

> See updated status - http://tracker.ceph.com/issues/17487#note-32
>
> Outstanding issues:
>
> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
> pls review/approve, assumed Approved ?)
>
> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
> pls review/approve, seems persistent, but maybe not a showstopper)
>
> upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
> to make it pass, see issues in the tacker summary above)
>
> Sage, jewel 10.2.4 can be released as soon as you agree with the
> findings/summary.

Do you think we're ready to release 10.2.4 yet?


Best,
Abhishek
>
> Thx
> YuriW
>
> On Fri, Nov 11, 2016 at 8:47 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
>> Detailed summary of the QE Validation can be found here
>> http://tracker.ceph.com/issues/17487#note-32
>>
>> The following suites were in scope of this point release validation:
>>
>> rados (subset 35/50 297 jobs)
>> rbd
>> rgw
>> fs
>> krbd
>> kcephfs
>> knfs
>> rest
>> hadoop
>> samba
>> ceph-deploy
>> ceph-disk
>> upgrade/client-upgrade
>> upgrade/hammer-x (jewel)
>> upgrade/infernalis-x (jewel) - deprecated
>> powercycle
>> (please let me know if any suites are missing from this ^ list)
>>
>> ==============================
>> Issues requiring approval/decision:
>>
>> fs - http://tracker.ceph.com/issues/17832 (Greg pls review/approve)
>>
>> krbd - http://tracker.ceph.com/issues/17221 (Jason, Ilya pls review/approve)
>>
>> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>> pls review/approve)
>>
>> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>> pls review/approve)
>>
>> upgrade/infernalis-x (jewel) - deprecated, but Nathan is still working
>> to make it pass (Sage pls review/approve)
>>
>> powercycle - in progress
>>
>> Thx
>> YuriW


--
Abhishek Lekshmanan
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)

Re: ceph v10.2.4 QE validation status

[Gregory Farnum]

On Wed, Nov 23, 2016 at 5:39 AM, Abhishek L <abhishek@suse.com> wrote:
>
> Hi Sage, Greg,

John signed off on the FS stuff and he's the one who does that now.
Looks good to me. ;)
-Greg

>
>
> Yuri Weinstein writes:
>
>> See updated status - http://tracker.ceph.com/issues/17487#note-32
>>
>> Outstanding issues:
>>
>> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>> pls review/approve, assumed Approved ?)
>>
>> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>> pls review/approve, seems persistent, but maybe not a showstopper)
>>
>> upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>> to make it pass, see issues in the tacker summary above)
>>
>> Sage, jewel 10.2.4 can be released as soon as you agree with the
>> findings/summary.
>
> Do you think we're ready to release 10.2.4 yet?
>
>
> Best,
> Abhishek
>>
>> Thx
>> YuriW
>>
>> On Fri, Nov 11, 2016 at 8:47 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
>>> Detailed summary of the QE Validation can be found here
>>> http://tracker.ceph.com/issues/17487#note-32
>>>
>>> The following suites were in scope of this point release validation:
>>>
>>> rados (subset 35/50 297 jobs)
>>> rbd
>>> rgw
>>> fs
>>> krbd
>>> kcephfs
>>> knfs
>>> rest
>>> hadoop
>>> samba
>>> ceph-deploy
>>> ceph-disk
>>> upgrade/client-upgrade
>>> upgrade/hammer-x (jewel)
>>> upgrade/infernalis-x (jewel) - deprecated
>>> powercycle
>>> (please let me know if any suites are missing from this ^ list)
>>>
>>> ==============================
>>> Issues requiring approval/decision:
>>>
>>> fs - http://tracker.ceph.com/issues/17832 (Greg pls review/approve)
>>>
>>> krbd - http://tracker.ceph.com/issues/17221 (Jason, Ilya pls review/approve)
>>>
>>> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>>> pls review/approve)
>>>
>>> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>>> pls review/approve)
>>>
>>> upgrade/infernalis-x (jewel) - deprecated, but Nathan is still working
>>> to make it pass (Sage pls review/approve)
>>>
>>> powercycle - in progress
>>>
>>> Thx
>>> YuriW
>
>
> --
> Abhishek Lekshmanan
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)

Re: ceph v10.2.4 QE validation status

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2636 bytes --]

On Wed, 23 Nov 2016, Abhishek L wrote:
> 
> Hi Sage, Greg,
> 
> 
> Yuri Weinstein writes:
> 
> > See updated status - http://tracker.ceph.com/issues/17487#note-32
> >
> > Outstanding issues:
> >
> > knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
> > pls review/approve, assumed Approved ?)
> >
> > upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
> > pls review/approve, seems persistent, but maybe not a showstopper)
> >
> > upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
> > to make it pass, see issues in the tacker summary above)
> >
> > Sage, jewel 10.2.4 can be released as soon as you agree with the
> > findings/summary.
> 
> Do you think we're ready to release 10.2.4 yet?

I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make 
sure this isn't a regression.

We can ignore the infernalis runs.

I think we can ignore the knfs selinux issue too.. Greg, can you confirm?

Thanks!
sage

> 
> 
> Best,
> Abhishek
> >
> > Thx
> > YuriW
> >
> > On Fri, Nov 11, 2016 at 8:47 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
> >> Detailed summary of the QE Validation can be found here
> >> http://tracker.ceph.com/issues/17487#note-32
> >>
> >> The following suites were in scope of this point release validation:
> >>
> >> rados (subset 35/50 297 jobs)
> >> rbd
> >> rgw
> >> fs
> >> krbd
> >> kcephfs
> >> knfs
> >> rest
> >> hadoop
> >> samba
> >> ceph-deploy
> >> ceph-disk
> >> upgrade/client-upgrade
> >> upgrade/hammer-x (jewel)
> >> upgrade/infernalis-x (jewel) - deprecated
> >> powercycle
> >> (please let me know if any suites are missing from this ^ list)
> >>
> >> ==============================
> >> Issues requiring approval/decision:
> >>
> >> fs - http://tracker.ceph.com/issues/17832 (Greg pls review/approve)
> >>
> >> krbd - http://tracker.ceph.com/issues/17221 (Jason, Ilya pls review/approve)
> >>
> >> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
> >> pls review/approve)
> >>
> >> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
> >> pls review/approve)
> >>
> >> upgrade/infernalis-x (jewel) - deprecated, but Nathan is still working
> >> to make it pass (Sage pls review/approve)
> >>
> >> powercycle - in progress
> >>
> >> Thx
> >> YuriW
> 
> 
> --
> Abhishek Lekshmanan
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: ceph v10.2.4 QE validation status

[John Spray]

On Wed, Nov 23, 2016 at 2:42 PM, Sage Weil <sweil@redhat.com> wrote:
> On Wed, 23 Nov 2016, Abhishek L wrote:
>>
>> Hi Sage, Greg,
>>
>>
>> Yuri Weinstein writes:
>>
>> > See updated status - http://tracker.ceph.com/issues/17487#note-32
>> >
>> > Outstanding issues:
>> >
>> > knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>> > pls review/approve, assumed Approved ?)
>> >
>> > upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>> > pls review/approve, seems persistent, but maybe not a showstopper)
>> >
>> > upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>> > to make it pass, see issues in the tacker summary above)
>> >
>> > Sage, jewel 10.2.4 can be released as soon as you agree with the
>> > findings/summary.
>>
>> Do you think we're ready to release 10.2.4 yet?
>
> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
> sure this isn't a regression.
>
> We can ignore the infernalis runs.
>
> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?

The knfs selinux issue can be ignored, yes.  Last investigation
indicated that it probably wasn't cephfs-specific.

John

> Thanks!
> sage
>
>>
>>
>> Best,
>> Abhishek
>> >
>> > Thx
>> > YuriW
>> >
>> > On Fri, Nov 11, 2016 at 8:47 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
>> >> Detailed summary of the QE Validation can be found here
>> >> http://tracker.ceph.com/issues/17487#note-32
>> >>
>> >> The following suites were in scope of this point release validation:
>> >>
>> >> rados (subset 35/50 297 jobs)
>> >> rbd
>> >> rgw
>> >> fs
>> >> krbd
>> >> kcephfs
>> >> knfs
>> >> rest
>> >> hadoop
>> >> samba
>> >> ceph-deploy
>> >> ceph-disk
>> >> upgrade/client-upgrade
>> >> upgrade/hammer-x (jewel)
>> >> upgrade/infernalis-x (jewel) - deprecated
>> >> powercycle
>> >> (please let me know if any suites are missing from this ^ list)
>> >>
>> >> ==============================
>> >> Issues requiring approval/decision:
>> >>
>> >> fs - http://tracker.ceph.com/issues/17832 (Greg pls review/approve)
>> >>
>> >> krbd - http://tracker.ceph.com/issues/17221 (Jason, Ilya pls review/approve)
>> >>
>> >> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>> >> pls review/approve)
>> >>
>> >> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>> >> pls review/approve)
>> >>
>> >> upgrade/infernalis-x (jewel) - deprecated, but Nathan is still working
>> >> to make it pass (Sage pls review/approve)
>> >>
>> >> powercycle - in progress
>> >>
>> >> Thx
>> >> YuriW
>>
>>
>> --
>> Abhishek Lekshmanan
>> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>

Re: ceph v10.2.4 QE validation status

[Abhishek L]

Sage Weil writes:

> On Wed, 23 Nov 2016, Abhishek L wrote:
>>
>> Hi Sage, Greg,
>>
>>
>> Yuri Weinstein writes:
>>
>> > See updated status - http://tracker.ceph.com/issues/17487#note-32
>> >
>> > Outstanding issues:
>> >
>> > knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>> > pls review/approve, assumed Approved ?)
>> >
>> > upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>> > pls review/approve, seems persistent, but maybe not a showstopper)
>> >
>> > upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>> > to make it pass, see issues in the tacker summary above)
>> >
>> > Sage, jewel 10.2.4 can be released as soon as you agree with the
>> > findings/summary.
>>
>> Do you think we're ready to release 10.2.4 yet?
>
> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
> sure this isn't a regression.
>
> We can ignore the infernalis runs.
>
> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
>
Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
at
http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &

updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
with details of the test. I'll update the progress once the suite goes
through

Best,
Abhishek

Re: ceph v10.2.4 QE validation status

[Abhishek L]

Abhishek L writes:

> Sage Weil writes:
>
>> On Wed, 23 Nov 2016, Abhishek L wrote:
>>>
>>> Hi Sage, Greg,
>>>
>>>
>>> Yuri Weinstein writes:
>>>
>>> > See updated status - http://tracker.ceph.com/issues/17487#note-32
>>> >
>>> > Outstanding issues:
>>> >
>>> > knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>>> > pls review/approve, assumed Approved ?)
>>> >
>>> > upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>>> > pls review/approve, seems persistent, but maybe not a showstopper)
>>> >
>>> > upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>>> > to make it pass, see issues in the tacker summary above)
>>> >
>>> > Sage, jewel 10.2.4 can be released as soon as you agree with the
>>> > findings/summary.
>>>
>>> Do you think we're ready to release 10.2.4 yet?
>>
>> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
>> sure this isn't a regression.
>>
>> We can ignore the infernalis runs.
>>
>> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
>>
> Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
> at
> http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &
>
> updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
> with details of the test. I'll update the progress once the suite goes
> through

Had around 11 tests fail from 296 scheduled, there were a couple of
valgrind issues on ceph-mon (which were seen at earlier runs on jewel as
well) and an s3test failure, rest of the issues were looking related to
infrastructure as they were failing to get specific version numbers from
gitbuilders.

Reported this issue as:

http://tracker.ceph.com/issues/18089

subsequent re runs are still failing with similar errors. The details
are updated at

http://tracker.ceph.com/issues/17487#note-37

The upgrade suite has also failed with similar errors.

Overall it looks like we're close to cutting the release, as soon as we
can get through these infra errors

Cheers,
Abhishek

Re: ceph v10.2.4 QE validation status

[Sage Weil]

On Wed, 30 Nov 2016, Abhishek L wrote:
> 
> Abhishek L writes:
> 
> > Sage Weil writes:
> >
> >> On Wed, 23 Nov 2016, Abhishek L wrote:
> >>>
> >>> Hi Sage, Greg,
> >>>
> >>>
> >>> Yuri Weinstein writes:
> >>>
> >>> > See updated status - http://tracker.ceph.com/issues/17487#note-32
> >>> >
> >>> > Outstanding issues:
> >>> >
> >>> > knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
> >>> > pls review/approve, assumed Approved ?)
> >>> >
> >>> > upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
> >>> > pls review/approve, seems persistent, but maybe not a showstopper)
> >>> >
> >>> > upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
> >>> > to make it pass, see issues in the tacker summary above)
> >>> >
> >>> > Sage, jewel 10.2.4 can be released as soon as you agree with the
> >>> > findings/summary.
> >>>
> >>> Do you think we're ready to release 10.2.4 yet?
> >>
> >> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
> >> sure this isn't a regression.
> >>
> >> We can ignore the infernalis runs.
> >>
> >> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
> >>
> > Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
> > at
> > http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &

https://github.com/ceph/ceph-qa-suite/pull/1292

avoids xenial for the rados upgrade tests (in jewel branch).

The upgrade tests already explicitly call out trusty, so they should be 
fine.

> > updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
> > with details of the test. I'll update the progress once the suite goes
> > through
> 
> Had around 11 tests fail from 296 scheduled, there were a couple of
> valgrind issues on ceph-mon (which were seen at earlier runs on jewel as
> well) and an s3test failure, rest of the issues were looking related to
> infrastructure as they were failing to get specific version numbers from
> gitbuilders.
> 
> Reported this issue as:
> 
> http://tracker.ceph.com/issues/18089
> 
> subsequent re runs are still failing with similar errors. The details
> are updated at
> 
> http://tracker.ceph.com/issues/17487#note-37
> 
> The upgrade suite has also failed with similar errors.

Which one?

Thanks!
sage

> 
> Overall it looks like we're close to cutting the release, as soon as we
> can get through these infra errors
> 
> Cheers,
> Abhishek
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: ceph v10.2.4 QE validation status

[Josh Durgin]

On 11/30/2016 06:57 AM, Abhishek L wrote:
>
> Abhishek L writes:
>
>> Sage Weil writes:
>>
>>> On Wed, 23 Nov 2016, Abhishek L wrote:
>>>>
>>>> Hi Sage, Greg,
>>>>
>>>>
>>>> Yuri Weinstein writes:
>>>>
>>>>> See updated status - http://tracker.ceph.com/issues/17487#note-32
>>>>>
>>>>> Outstanding issues:
>>>>>
>>>>> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>>>>> pls review/approve, assumed Approved ?)
>>>>>
>>>>> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>>>>> pls review/approve, seems persistent, but maybe not a showstopper)
>>>>>
>>>>> upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>>>>> to make it pass, see issues in the tacker summary above)
>>>>>
>>>>> Sage, jewel 10.2.4 can be released as soon as you agree with the
>>>>> findings/summary.
>>>>
>>>> Do you think we're ready to release 10.2.4 yet?
>>>
>>> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
>>> sure this isn't a regression.
>>>
>>> We can ignore the infernalis runs.
>>>
>>> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
>>>
>> Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
>> at
>> http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &
>>
>> updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
>> with details of the test. I'll update the progress once the suite goes
>> through
>
> Had around 11 tests fail from 296 scheduled, there were a couple of
> valgrind issues on ceph-mon (which were seen at earlier runs on jewel as
> well) and an s3test failure, rest of the issues were looking related to
> infrastructure as they were failing to get specific version numbers from
> gitbuilders.
>
> Reported this issue as:
>
> http://tracker.ceph.com/issues/18089
>
> subsequent re runs are still failing with similar errors. The details
> are updated at
>
> http://tracker.ceph.com/issues/17487#note-37
>
> The upgrade suite has also failed with similar errors.
>
> Overall it looks like we're close to cutting the release, as soon as we
> can get through these infra errors

We should also get https://github.com/ceph/ceph/pull/12207 in once the
master version (https://github.com/ceph/ceph/pull/12206) is merged -
likely later today.

Josh

Re: ceph v10.2.4 QE validation status

[Abhishek L]

Sage Weil writes:

> On Wed, 30 Nov 2016, Abhishek L wrote:
>>
>> Abhishek L writes:
>>
>> > Sage Weil writes:
>> >
>> >> On Wed, 23 Nov 2016, Abhishek L wrote:
>> >>>
>> >>> Hi Sage, Greg,
>> >>>
>> >>>
>> >>> Yuri Weinstein writes:
>> >>>
>> >>> > See updated status - http://tracker.ceph.com/issues/17487#note-32
>> >>> >
>> >>> > Outstanding issues:
>> >>> >
>> >>> > knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>> >>> > pls review/approve, assumed Approved ?)
>> >>> >
>> >>> > upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>> >>> > pls review/approve, seems persistent, but maybe not a showstopper)
>> >>> >
>> >>> > upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>> >>> > to make it pass, see issues in the tacker summary above)
>> >>> >
>> >>> > Sage, jewel 10.2.4 can be released as soon as you agree with the
>> >>> > findings/summary.
>> >>>
>> >>> Do you think we're ready to release 10.2.4 yet?
>> >>
>> >> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
>> >> sure this isn't a regression.
>> >>
>> >> We can ignore the infernalis runs.
>> >>
>> >> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
>> >>
>> > Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
>> > at
>> > http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &
>
> https://github.com/ceph/ceph-qa-suite/pull/1292
>
> avoids xenial for the rados upgrade tests (in jewel branch).
>
> The upgrade tests already explicitly call out trusty, so they should be
> fine.
>
>> > updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
>> > with details of the test. I'll update the progress once the suite goes
>> > through
>>
>> Had around 11 tests fail from 296 scheduled, there were a couple of
>> valgrind issues on ceph-mon (which were seen at earlier runs on jewel as
>> well) and an s3test failure, rest of the issues were looking related to
>> infrastructure as they were failing to get specific version numbers from
>> gitbuilders.
>>
>> Reported this issue as:
>>
>> http://tracker.ceph.com/issues/18089
>>
>> subsequent re runs are still failing with similar errors. The details
>> are updated at
>>
>> http://tracker.ceph.com/issues/17487#note-37
>>
>> The upgrade suite has also failed with similar errors.
>
> Which one?
The upgrade/client-upgrade suite (I hope this is the right upgrade
suite), the errors are the same failed to fetch package version errors
seen for the rados suite, so not actual test run errors yet.

http://pulpito.ceph.com/abhi-2016-11-30_09:58:50-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/
is the run

Best,
Abhishek

Re: ceph v10.2.4 QE validation status

[Abhishek L]

Josh Durgin writes:

> On 11/30/2016 06:57 AM, Abhishek L wrote:
>>
>> Abhishek L writes:
>>
>>> Sage Weil writes:
>>>
>>>> On Wed, 23 Nov 2016, Abhishek L wrote:
>>>>>
>>>>> Hi Sage, Greg,
>>>>>
>>>>>
>>>>> Yuri Weinstein writes:
>>>>>
>>>>>> See updated status - http://tracker.ceph.com/issues/17487#note-32
>>>>>>
>>>>>> Outstanding issues:
>>>>>>
>>>>>> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>>>>>> pls review/approve, assumed Approved ?)
>>>>>>
>>>>>> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>>>>>> pls review/approve, seems persistent, but maybe not a showstopper)
>>>>>>
>>>>>> upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>>>>>> to make it pass, see issues in the tacker summary above)
>>>>>>
>>>>>> Sage, jewel 10.2.4 can be released as soon as you agree with the
>>>>>> findings/summary.
>>>>>
>>>>> Do you think we're ready to release 10.2.4 yet?
>>>>
>>>> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
>>>> sure this isn't a regression.
>>>>
>>>> We can ignore the infernalis runs.
>>>>
>>>> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
>>>>
>>> Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
>>> at
>>> http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &
>>>
>>> updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
>>> with details of the test. I'll update the progress once the suite goes
>>> through
>>
>> Had around 11 tests fail from 296 scheduled, there were a couple of
>> valgrind issues on ceph-mon (which were seen at earlier runs on jewel as
>> well) and an s3test failure, rest of the issues were looking related to
>> infrastructure as they were failing to get specific version numbers from
>> gitbuilders.
>>
>> Reported this issue as:
>>
>> http://tracker.ceph.com/issues/18089
>>
>> subsequent re runs are still failing with similar errors. The details
>> are updated at
>>
>> http://tracker.ceph.com/issues/17487#note-37
>>
>> The upgrade suite has also failed with similar errors.
>>
>> Overall it looks like we're close to cutting the release, as soon as we
>> can get through these infra errors
>
> We should also get https://github.com/ceph/ceph/pull/12207 in once the
> master version (https://github.com/ceph/ceph/pull/12206) is merged -
> likely later today.

Do you want it in 10.2.4 itself? what all qe suites would you recommend if so?

Re: ceph v10.2.4 QE validation status

[Abhishek L]

Abhishek L writes:

> Sage Weil writes:
>
>> On Wed, 30 Nov 2016, Abhishek L wrote:
>>>
>>> Abhishek L writes:
>>>
>>> > Sage Weil writes:
>>> >
>>> >> On Wed, 23 Nov 2016, Abhishek L wrote:
>>> >>>
>>> >>> Hi Sage, Greg,
>>> >>>
>>> >>>
>>> >>> Yuri Weinstein writes:
>>> >>>
>>> >>> > See updated status - http://tracker.ceph.com/issues/17487#note-32
>>> >>> >
>>> >>> > Outstanding issues:
>>> >>> >
>>> >>> > knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>>> >>> > pls review/approve, assumed Approved ?)
>>> >>> >
>>> >>> > upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>>> >>> > pls review/approve, seems persistent, but maybe not a showstopper)
>>> >>> >
>>> >>> > upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>>> >>> > to make it pass, see issues in the tacker summary above)
>>> >>> >
>>> >>> > Sage, jewel 10.2.4 can be released as soon as you agree with the
>>> >>> > findings/summary.
>>> >>>
>>> >>> Do you think we're ready to release 10.2.4 yet?
>>> >>
>>> >> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
>>> >> sure this isn't a regression.
>>> >>
>>> >> We can ignore the infernalis runs.
>>> >>
>>> >> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
>>> >>
>>> > Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
>>> > at
>>> > http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &
>>
>> https://github.com/ceph/ceph-qa-suite/pull/1292
>>
>> avoids xenial for the rados upgrade tests (in jewel branch).

Ah alright, maybe this was the cause after all, let's get this in (or
push a branch to ceph-qa-suite so that I can schedule against that)
>>
>> The upgrade tests already explicitly call out trusty, so they should be
>> fine.
>>
>>> > updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
>>> > with details of the test. I'll update the progress once the suite goes
>>> > through
>>>
>>> Had around 11 tests fail from 296 scheduled, there were a couple of
>>> valgrind issues on ceph-mon (which were seen at earlier runs on jewel as
>>> well) and an s3test failure, rest of the issues were looking related to
>>> infrastructure as they were failing to get specific version numbers from
>>> gitbuilders.
>>>
>>> Reported this issue as:
>>>
>>> http://tracker.ceph.com/issues/18089
>>>
>>> subsequent re runs are still failing with similar errors. The details
>>> are updated at
>>>
>>> http://tracker.ceph.com/issues/17487#note-37
>>>
>>> The upgrade suite has also failed with similar errors.
>>
>> Which one?
> The upgrade/client-upgrade suite (I hope this is the right upgrade
> suite), the errors are the same failed to fetch package version errors
> seen for the rados suite, so not actual test run errors yet.
>
> http://pulpito.ceph.com/abhi-2016-11-30_09:58:50-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/
> is the run
>
> Best,
> Abhishek


--
Abhishek Lekshmanan
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)

Re: ceph v10.2.4 QE validation status

[Josh Durgin]

On 11/30/2016 08:23 AM, Abhishek L wrote:
>
> Josh Durgin writes:
>
>> On 11/30/2016 06:57 AM, Abhishek L wrote:
>>>
>>> Abhishek L writes:
>>>
>>>> Sage Weil writes:
>>>>
>>>>> On Wed, 23 Nov 2016, Abhishek L wrote:
>>>>>>
>>>>>> Hi Sage, Greg,
>>>>>>
>>>>>>
>>>>>> Yuri Weinstein writes:
>>>>>>
>>>>>>> See updated status - http://tracker.ceph.com/issues/17487#note-32
>>>>>>>
>>>>>>> Outstanding issues:
>>>>>>>
>>>>>>> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>>>>>>> pls review/approve, assumed Approved ?)
>>>>>>>
>>>>>>> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>>>>>>> pls review/approve, seems persistent, but maybe not a showstopper)
>>>>>>>
>>>>>>> upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>>>>>>> to make it pass, see issues in the tacker summary above)
>>>>>>>
>>>>>>> Sage, jewel 10.2.4 can be released as soon as you agree with the
>>>>>>> findings/summary.
>>>>>>
>>>>>> Do you think we're ready to release 10.2.4 yet?
>>>>>
>>>>> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
>>>>> sure this isn't a regression.
>>>>>
>>>>> We can ignore the infernalis runs.
>>>>>
>>>>> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
>>>>>
>>>> Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
>>>> at
>>>> http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &
>>>>
>>>> updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
>>>> with details of the test. I'll update the progress once the suite goes
>>>> through
>>>
>>> Had around 11 tests fail from 296 scheduled, there were a couple of
>>> valgrind issues on ceph-mon (which were seen at earlier runs on jewel as
>>> well) and an s3test failure, rest of the issues were looking related to
>>> infrastructure as they were failing to get specific version numbers from
>>> gitbuilders.
>>>
>>> Reported this issue as:
>>>
>>> http://tracker.ceph.com/issues/18089
>>>
>>> subsequent re runs are still failing with similar errors. The details
>>> are updated at
>>>
>>> http://tracker.ceph.com/issues/17487#note-37
>>>
>>> The upgrade suite has also failed with similar errors.
>>>
>>> Overall it looks like we're close to cutting the release, as soon as we
>>> can get through these infra errors
>>
>> We should also get https://github.com/ceph/ceph/pull/12207 in once the
>> master version (https://github.com/ceph/ceph/pull/12206) is merged -
>> likely later today.
>
> Do you want it in 10.2.4 itself? what all qe suites would you recommend if so?

Yes, we need to not break the librados API or ABI in stable series 
especially.

The rgw suite would be enough - it's the only thing that really uses 
setxattr.

Josh

Re: ceph v10.2.4 QE validation status

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3773 bytes --]

On Wed, 30 Nov 2016, Abhishek L wrote:
> Abhishek L writes:
> 
> > Sage Weil writes:
> >
> >> On Wed, 30 Nov 2016, Abhishek L wrote:
> >>>
> >>> Abhishek L writes:
> >>>
> >>> > Sage Weil writes:
> >>> >
> >>> >> On Wed, 23 Nov 2016, Abhishek L wrote:
> >>> >>>
> >>> >>> Hi Sage, Greg,
> >>> >>>
> >>> >>>
> >>> >>> Yuri Weinstein writes:
> >>> >>>
> >>> >>> > See updated status - http://tracker.ceph.com/issues/17487#note-32
> >>> >>> >
> >>> >>> > Outstanding issues:
> >>> >>> >
> >>> >>> > knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
> >>> >>> > pls review/approve, assumed Approved ?)
> >>> >>> >
> >>> >>> > upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
> >>> >>> > pls review/approve, seems persistent, but maybe not a showstopper)
> >>> >>> >
> >>> >>> > upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
> >>> >>> > to make it pass, see issues in the tacker summary above)
> >>> >>> >
> >>> >>> > Sage, jewel 10.2.4 can be released as soon as you agree with the
> >>> >>> > findings/summary.
> >>> >>>
> >>> >>> Do you think we're ready to release 10.2.4 yet?
> >>> >>
> >>> >> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
> >>> >> sure this isn't a regression.
> >>> >>
> >>> >> We can ignore the infernalis runs.
> >>> >>
> >>> >> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
> >>> >>
> >>> > Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
> >>> > at
> >>> > http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &
> >>
> >> https://github.com/ceph/ceph-qa-suite/pull/1292
> >>
> >> avoids xenial for the rados upgrade tests (in jewel branch).
> 
> Ah alright, maybe this was the cause after all, let's get this in (or
> push a branch to ceph-qa-suite so that I can schedule against that)

I added a patch for the client-upgrade tests to the same branch. You 
should be able to just schedule with --suite-branch jewel-avoid-xenial 
instead of --suite-branch jewel (for both rados and 
upgrade/client-upgrade).

sage


> >>
> >> The upgrade tests already explicitly call out trusty, so they should be
> >> fine.
> >>
> >>> > updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
> >>> > with details of the test. I'll update the progress once the suite goes
> >>> > through
> >>>
> >>> Had around 11 tests fail from 296 scheduled, there were a couple of
> >>> valgrind issues on ceph-mon (which were seen at earlier runs on jewel as
> >>> well) and an s3test failure, rest of the issues were looking related to
> >>> infrastructure as they were failing to get specific version numbers from
> >>> gitbuilders.
> >>>
> >>> Reported this issue as:
> >>>
> >>> http://tracker.ceph.com/issues/18089
> >>>
> >>> subsequent re runs are still failing with similar errors. The details
> >>> are updated at
> >>>
> >>> http://tracker.ceph.com/issues/17487#note-37
> >>>
> >>> The upgrade suite has also failed with similar errors.
> >>
> >> Which one?
> > The upgrade/client-upgrade suite (I hope this is the right upgrade
> > suite), the errors are the same failed to fetch package version errors
> > seen for the rados suite, so not actual test run errors yet.
> >
> > http://pulpito.ceph.com/abhi-2016-11-30_09:58:50-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/
> > is the run
> >
> > Best,
> > Abhishek
> 
> 
> --
> Abhishek Lekshmanan
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: ceph v10.2.4 QE validation status

[Ken Dreyer]

On Wed, Nov 23, 2016 at 3:39 AM, Abhishek L <abhishek@suse.com> wrote:
> Do you think we're ready to release 10.2.4 yet?

It would be great to get https://github.com/ceph/ceph/pull/12001 in
before v10.2.4 as well.

- Ken

Re: ceph v10.2.4 QE validation status

[Abhishek L]

On Wed, Nov 30, 2016 at 10:56 PM, Ken Dreyer <kdreyer@redhat.com> wrote:
> On Wed, Nov 23, 2016 at 3:39 AM, Abhishek L <abhishek@suse.com> wrote:
>> Do you think we're ready to release 10.2.4 yet?
>
> It would be great to get https://github.com/ceph/ceph/pull/12001 in
> before v10.2.4 as well.

The current test runs already include 12001,
once we have a green run we should be having it in 10.2.4

Best,
Abhishek

Re: ceph v10.2.4 QE validation status

[Yoann Moulin]

Hello,

>> It would be great to get https://github.com/ceph/ceph/pull/12001 in
>> before v10.2.4 as well.
> 
> The current test runs already include 12001,
> once we have a green run we should be having it in 10.2.4

I don't see the pull request 12001 for 10.2.4[1] release, it's in the 10.2.5[2]

1. http://tracker.ceph.com/issues/17487
2. http://tracker.ceph.com/issues/17851

Do you plan to include it in the 10.2.4 ?

Best regards,

-- 
Yoann Moulin
EPFL IC-IT

Re: ceph v10.2.4 QE validation status

[Alfredo Deza]

On Wed, Nov 23, 2016 at 5:39 AM, Abhishek L <abhishek@suse.com> wrote:
>
> Hi Sage, Greg,
>
>
> Yuri Weinstein writes:
>
>> See updated status - http://tracker.ceph.com/issues/17487#note-32
>>
>> Outstanding issues:
>>
>> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>> pls review/approve, assumed Approved ?)
>>
>> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>> pls review/approve, seems persistent, but maybe not a showstopper)
>>
>> upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>> to make it pass, see issues in the tacker summary above)
>>
>> Sage, jewel 10.2.4 can be released as soon as you agree with the
>> findings/summary.
>
> Do you think we're ready to release 10.2.4 yet?

I thought this PR had made it https://github.com/ceph/ceph/pull/11749
but apparently it didn't for Jewel

The backport PR for Jewel is https://github.com/ceph/ceph/pull/11884

It fixes quite a bit of issues we've had for deployment. It is only
relevant when bootstrapping, when keys are needed for the first time.

Hopefully we can include it as well.
>
>
> Best,
> Abhishek
>>
>> Thx
>> YuriW
>>
>> On Fri, Nov 11, 2016 at 8:47 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
>>> Detailed summary of the QE Validation can be found here
>>> http://tracker.ceph.com/issues/17487#note-32
>>>
>>> The following suites were in scope of this point release validation:
>>>
>>> rados (subset 35/50 297 jobs)
>>> rbd
>>> rgw
>>> fs
>>> krbd
>>> kcephfs
>>> knfs
>>> rest
>>> hadoop
>>> samba
>>> ceph-deploy
>>> ceph-disk
>>> upgrade/client-upgrade
>>> upgrade/hammer-x (jewel)
>>> upgrade/infernalis-x (jewel) - deprecated
>>> powercycle
>>> (please let me know if any suites are missing from this ^ list)
>>>
>>> ==============================
>>> Issues requiring approval/decision:
>>>
>>> fs - http://tracker.ceph.com/issues/17832 (Greg pls review/approve)
>>>
>>> krbd - http://tracker.ceph.com/issues/17221 (Jason, Ilya pls review/approve)
>>>
>>> knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>>> pls review/approve)
>>>
>>> upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>>> pls review/approve)
>>>
>>> upgrade/infernalis-x (jewel) - deprecated, but Nathan is still working
>>> to make it pass (Sage pls review/approve)
>>>
>>> powercycle - in progress
>>>
>>> Thx
>>> YuriW
>
>
> --
> Abhishek Lekshmanan
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph v10.2.4 QE validation status

[Abhishek L]

Sage Weil writes:

> On Wed, 30 Nov 2016, Abhishek L wrote:
>> Abhishek L writes:
>>
>> > Sage Weil writes:
>> >
>> >> On Wed, 30 Nov 2016, Abhishek L wrote:
>> >>>
>> >>> Abhishek L writes:
>> >>>
>> >>> > Sage Weil writes:
>> >>> >
>> >>> >> On Wed, 23 Nov 2016, Abhishek L wrote:
>> >>> >>>
>> >>> >>> Hi Sage, Greg,
>> >>> >>>
>> >>> >>>
>> >>> >>> Yuri Weinstein writes:
>> >>> >>>
>> >>> >>> > See updated status - http://tracker.ceph.com/issues/17487#note-32
>> >>> >>> >
>> >>> >>> > Outstanding issues:
>> >>> >>> >
>> >>> >>> > knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>> >>> >>> > pls review/approve, assumed Approved ?)
>> >>> >>> >
>> >>> >>> > upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>> >>> >>> > pls review/approve, seems persistent, but maybe not a showstopper)
>> >>> >>> >
>> >>> >>> > upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>> >>> >>> > to make it pass, see issues in the tacker summary above)
>> >>> >>> >
>> >>> >>> > Sage, jewel 10.2.4 can be released as soon as you agree with the
>> >>> >>> > findings/summary.
>> >>> >>>
>> >>> >>> Do you think we're ready to release 10.2.4 yet?
>> >>> >>
>> >>> >> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
>> >>> >> sure this isn't a regression.
>> >>> >>
>> >>> >> We can ignore the infernalis runs.
>> >>> >>
>> >>> >> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
>> >>> >>
>> >>> > Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
>> >>> > at
>> >>> > http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &
>> >>
>> >> https://github.com/ceph/ceph-qa-suite/pull/1292
>> >>
>> >> avoids xenial for the rados upgrade tests (in jewel branch).
>>
>> Ah alright, maybe this was the cause after all, let's get this in (or
>> push a branch to ceph-qa-suite so that I can schedule against that)
>
> I added a patch for the client-upgrade tests to the same branch. You
> should be able to just schedule with --suite-branch jewel-avoid-xenial
> instead of --suite-branch jewel (for both rados and
> upgrade/client-upgrade).
>
> sage
>
>
>> >>
>> >> The upgrade tests already explicitly call out trusty, so they should be
>> >> fine.
>> >>
>> >>> > updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
>> >>> > with details of the test. I'll update the progress once the suite goes
>> >>> > through
>> >>>
>> >>> Had around 11 tests fail from 296 scheduled, there were a couple of
>> >>> valgrind issues on ceph-mon (which were seen at earlier runs on jewel as
>> >>> well) and an s3test failure, rest of the issues were looking related to
>> >>> infrastructure as they were failing to get specific version numbers from
>> >>> gitbuilders.
>> >>>
>> >>> Reported this issue as:
>> >>>
>> >>> http://tracker.ceph.com/issues/18089
>> >>>
>> >>> subsequent re runs are still failing with similar errors. The details
>> >>> are updated at
>> >>>
>> >>> http://tracker.ceph.com/issues/17487#note-37
>> >>>
>> >>> The upgrade suite has also failed with similar errors.
>> >>
>> >> Which one?
>> > The upgrade/client-upgrade suite (I hope this is the right upgrade
>> > suite), the errors are the same failed to fetch package version errors
>> > seen for the rados suite, so not actual test run errors yet.
>> >
>> > http://pulpito.ceph.com/abhi-2016-11-30_09:58:50-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/
>> > is the run
>> >
Update on the current status,
For prs 12001 & 12167: (sage/Sam)

The upgrade passed with the exception of infernalis/jewel which failed
with RBD (test_librbd_api.sh) (jdillaman please approve)

http://pulpito.ceph.com/abhi-2016-12-02_09:45:37-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/

the rados run (with the exception of the valgrind issue) are still
waiting for completion for 4 of the 296 jobs, this one suite always
seems to die when scheduled, this is the current url of the rados jobs
(still waiting)
http://pulpito.ceph.com/abhi-2016-12-02_10:06:39-rados-wip-jewel-10-2-4-rc-distro-basic-smithi/


for pr 12207: (josh)
the rgw suite also saw the same valgrind issue
(http://pulpito.ceph.com/abhi-2016-12-01_22:19:07-rgw-wip-jewel-10-2-4-rc-distro-basic-smithi/)

https://github.com/ceph/ceph/pull/12267, (mon cleanup)  was merged on Sam's approval.

prs 11884 & 12067 (Sam, Alfredo):
the ceph create-keys  pr #11884 hasn't been merged yet into jewel, is
there any suite that must be run to validate this. Same applies for
https://github.com/ceph/ceph/pull/12067 (OSDMon: MOSDBoot )


Further the details of all the runs so far are captured at http://tracker.ceph.com/issues/17487#note-37

I'll update soon once the rados jobs complete.

Best,
Abhishek

Re: ceph v10.2.4 QE validation status

[Abhishek L]

Abhishek L writes:

> Sage Weil writes:
>
>> On Wed, 30 Nov 2016, Abhishek L wrote:
>>> Abhishek L writes:
>>>
>>> > Sage Weil writes:
>>> >
>>> >> On Wed, 30 Nov 2016, Abhishek L wrote:
>>> >>>
>>> >>> Abhishek L writes:
>>> >>>
>>> >>> > Sage Weil writes:
>>> >>> >
>>> >>> >> On Wed, 23 Nov 2016, Abhishek L wrote:
>>> >>> >>>
>>> >>> >>> Hi Sage, Greg,
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> Yuri Weinstein writes:
>>> >>> >>>
>>> >>> >>> > See updated status - http://tracker.ceph.com/issues/17487#note-32
>>> >>> >>> >
>>> >>> >>> > Outstanding issues:
>>> >>> >>> >
>>> >>> >>> > knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>>> >>> >>> > pls review/approve, assumed Approved ?)
>>> >>> >>> >
>>> >>> >>> > upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>>> >>> >>> > pls review/approve, seems persistent, but maybe not a showstopper)
>>> >>> >>> >
>>> >>> >>> > upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>>> >>> >>> > to make it pass, see issues in the tacker summary above)
>>> >>> >>> >
>>> >>> >>> > Sage, jewel 10.2.4 can be released as soon as you agree with the
>>> >>> >>> > findings/summary.
>>> >>> >>>
>>> >>> >>> Do you think we're ready to release 10.2.4 yet?
>>> >>> >>
>>> >>> >> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
>>> >>> >> sure this isn't a regression.
>>> >>> >>
>>> >>> >> We can ignore the infernalis runs.
>>> >>> >>
>>> >>> >> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
>>> >>> >>
>>> >>> > Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
>>> >>> > at
>>> >>> > http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &
>>> >>
>>> >> https://github.com/ceph/ceph-qa-suite/pull/1292
>>> >>
>>> >> avoids xenial for the rados upgrade tests (in jewel branch).
>>>
>>> Ah alright, maybe this was the cause after all, let's get this in (or
>>> push a branch to ceph-qa-suite so that I can schedule against that)
>>
>> I added a patch for the client-upgrade tests to the same branch. You
>> should be able to just schedule with --suite-branch jewel-avoid-xenial
>> instead of --suite-branch jewel (for both rados and
>> upgrade/client-upgrade).
>>
>> sage
>>
>>
>>> >>
>>> >> The upgrade tests already explicitly call out trusty, so they should be
>>> >> fine.
>>> >>
>>> >>> > updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
>>> >>> > with details of the test. I'll update the progress once the suite goes
>>> >>> > through
>>> >>>
>>> >>> Had around 11 tests fail from 296 scheduled, there were a couple of
>>> >>> valgrind issues on ceph-mon (which were seen at earlier runs on jewel as
>>> >>> well) and an s3test failure, rest of the issues were looking related to
>>> >>> infrastructure as they were failing to get specific version numbers from
>>> >>> gitbuilders.
>>> >>>
>>> >>> Reported this issue as:
>>> >>>
>>> >>> http://tracker.ceph.com/issues/18089
>>> >>>
>>> >>> subsequent re runs are still failing with similar errors. The details
>>> >>> are updated at
>>> >>>
>>> >>> http://tracker.ceph.com/issues/17487#note-37
>>> >>>
>>> >>> The upgrade suite has also failed with similar errors.
>>> >>
>>> >> Which one?
>>> > The upgrade/client-upgrade suite (I hope this is the right upgrade
>>> > suite), the errors are the same failed to fetch package version errors
>>> > seen for the rados suite, so not actual test run errors yet.
>>> >
>>> > http://pulpito.ceph.com/abhi-2016-11-30_09:58:50-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/
>>> > is the run
>>> >
> Update on the current status,
> For prs 12001 & 12167: (sage/Sam)
>
> The upgrade passed with the exception of infernalis/jewel which failed
> with RBD (test_librbd_api.sh) (jdillaman please approve)
>
> http://pulpito.ceph.com/abhi-2016-12-02_09:45:37-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/
>
> the rados run (with the exception of the valgrind issue) are still
> waiting for completion for 4 of the 296 jobs, this one suite always
> seems to die when scheduled, this is the current url of the rados jobs
> (still waiting)
> http://pulpito.ceph.com/abhi-2016-12-02_10:06:39-rados-wip-jewel-10-2-4-rc-distro-basic-smithi/

Loic was able to spot the error in this run (which was a regression
introduced by f95ed3e, this was later fixed in
https://github.com/ceph/ceph-qa-suite/pull/1297, so running this suite
on the latest branch came out green,

http://pulpito.front.sepia.ceph.com/abhi-2016-12-05_10:25:26-rados-wip-jewel-10-2-4-rc-distro-basic-smithi/

Which means 12167 is also good to go.
>
>
> for pr 12207: (josh)
> the rgw suite also saw the same valgrind issue
> (http://pulpito.ceph.com/abhi-2016-12-01_22:19:07-rgw-wip-jewel-10-2-4-rc-distro-basic-smithi/)
>
> https://github.com/ceph/ceph/pull/12267, (mon cleanup)  was merged on Sam's approval.
>
> prs 11884 & 12067 (Sam, Alfredo):
> the ceph create-keys  pr #11884 hasn't been merged yet into jewel, is
> there any suite that must be run to validate this. Same applies for
> https://github.com/ceph/ceph/pull/12067 (OSDMon: MOSDBoot )

12067 was merged on Sam's approval as well. Which only leaves us with
11884.

Best,
Abhishek Lekshmanan
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)

Re: ceph v10.2.4 QE validation status

[Alfredo Deza]

On Mon, Dec 5, 2016 at 6:53 AM, Abhishek L <abhishek@suse.com> wrote:
>
> Abhishek L writes:
>
>> Sage Weil writes:
>>
>>> On Wed, 30 Nov 2016, Abhishek L wrote:
>>>> Abhishek L writes:
>>>>
>>>> > Sage Weil writes:
>>>> >
>>>> >> On Wed, 30 Nov 2016, Abhishek L wrote:
>>>> >>>
>>>> >>> Abhishek L writes:
>>>> >>>
>>>> >>> > Sage Weil writes:
>>>> >>> >
>>>> >>> >> On Wed, 23 Nov 2016, Abhishek L wrote:
>>>> >>> >>>
>>>> >>> >>> Hi Sage, Greg,
>>>> >>> >>>
>>>> >>> >>>
>>>> >>> >>> Yuri Weinstein writes:
>>>> >>> >>>
>>>> >>> >>> > See updated status - http://tracker.ceph.com/issues/17487#note-32
>>>> >>> >>> >
>>>> >>> >>> > Outstanding issues:
>>>> >>> >>> >
>>>> >>> >>> > knfs - http://tracker.ceph.com/issues/16397 (same as in v10.2.3, Greg
>>>> >>> >>> > pls review/approve, assumed Approved ?)
>>>> >>> >>> >
>>>> >>> >>> > upgrade/hammer-x (jewel) - http://tracker.ceph.com/issues/17847 ((Sage
>>>> >>> >>> > pls review/approve, seems persistent, but maybe not a showstopper)
>>>> >>> >>> >
>>>> >>> >>> > upgrade/infernalis-x (jewel) - deprecated (Nathan is still working
>>>> >>> >>> > to make it pass, see issues in the tacker summary above)
>>>> >>> >>> >
>>>> >>> >>> > Sage, jewel 10.2.4 can be released as soon as you agree with the
>>>> >>> >>> > findings/summary.
>>>> >>> >>>
>>>> >>> >>> Do you think we're ready to release 10.2.4 yet?
>>>> >>> >>
>>>> >>> >> I'm reproducing http://tracker.ceph.com/issues/17847 with logs to make
>>>> >>> >> sure this isn't a regression.
>>>> >>> >>
>>>> >>> >> We can ignore the infernalis runs.
>>>> >>> >>
>>>> >>> >> I think we can ignore the knfs selinux issue too.. Greg, can you confirm?
>>>> >>> >>
>>>> >>> > Added the prs 12001 & 12167 on top of the jewel branch and scheduled rados runs
>>>> >>> > at
>>>> >>> > http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/ &
>>>> >>
>>>> >> https://github.com/ceph/ceph-qa-suite/pull/1292
>>>> >>
>>>> >> avoids xenial for the rados upgrade tests (in jewel branch).
>>>>
>>>> Ah alright, maybe this was the cause after all, let's get this in (or
>>>> push a branch to ceph-qa-suite so that I can schedule against that)
>>>
>>> I added a patch for the client-upgrade tests to the same branch. You
>>> should be able to just schedule with --suite-branch jewel-avoid-xenial
>>> instead of --suite-branch jewel (for both rados and
>>> upgrade/client-upgrade).
>>>
>>> sage
>>>
>>>
>>>> >>
>>>> >> The upgrade tests already explicitly call out trusty, so they should be
>>>> >> fine.
>>>> >>
>>>> >>> > updated the tracker at http://tracker.ceph.com/issues/17851#note-17,
>>>> >>> > with details of the test. I'll update the progress once the suite goes
>>>> >>> > through
>>>> >>>
>>>> >>> Had around 11 tests fail from 296 scheduled, there were a couple of
>>>> >>> valgrind issues on ceph-mon (which were seen at earlier runs on jewel as
>>>> >>> well) and an s3test failure, rest of the issues were looking related to
>>>> >>> infrastructure as they were failing to get specific version numbers from
>>>> >>> gitbuilders.
>>>> >>>
>>>> >>> Reported this issue as:
>>>> >>>
>>>> >>> http://tracker.ceph.com/issues/18089
>>>> >>>
>>>> >>> subsequent re runs are still failing with similar errors. The details
>>>> >>> are updated at
>>>> >>>
>>>> >>> http://tracker.ceph.com/issues/17487#note-37
>>>> >>>
>>>> >>> The upgrade suite has also failed with similar errors.
>>>> >>
>>>> >> Which one?
>>>> > The upgrade/client-upgrade suite (I hope this is the right upgrade
>>>> > suite), the errors are the same failed to fetch package version errors
>>>> > seen for the rados suite, so not actual test run errors yet.
>>>> >
>>>> > http://pulpito.ceph.com/abhi-2016-11-30_09:58:50-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/
>>>> > is the run
>>>> >
>> Update on the current status,
>> For prs 12001 & 12167: (sage/Sam)
>>
>> The upgrade passed with the exception of infernalis/jewel which failed
>> with RBD (test_librbd_api.sh) (jdillaman please approve)
>>
>> http://pulpito.ceph.com/abhi-2016-12-02_09:45:37-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/
>>
>> the rados run (with the exception of the valgrind issue) are still
>> waiting for completion for 4 of the 296 jobs, this one suite always
>> seems to die when scheduled, this is the current url of the rados jobs
>> (still waiting)
>> http://pulpito.ceph.com/abhi-2016-12-02_10:06:39-rados-wip-jewel-10-2-4-rc-distro-basic-smithi/
>
> Loic was able to spot the error in this run (which was a regression
> introduced by f95ed3e, this was later fixed in
> https://github.com/ceph/ceph-qa-suite/pull/1297, so running this suite
> on the latest branch came out green,
>
> http://pulpito.front.sepia.ceph.com/abhi-2016-12-05_10:25:26-rados-wip-jewel-10-2-4-rc-distro-basic-smithi/
>
> Which means 12167 is also good to go.
>>
>>
>> for pr 12207: (josh)
>> the rgw suite also saw the same valgrind issue
>> (http://pulpito.ceph.com/abhi-2016-12-01_22:19:07-rgw-wip-jewel-10-2-4-rc-distro-basic-smithi/)
>>
>> https://github.com/ceph/ceph/pull/12267, (mon cleanup)  was merged on Sam's approval.
>>
>> prs 11884 & 12067 (Sam, Alfredo):
>> the ceph create-keys  pr #11884 hasn't been merged yet into jewel, is
>> there any suite that must be run to validate this. Same applies for
>> https://github.com/ceph/ceph/pull/12067 (OSDMon: MOSDBoot )
>
> 12067 was merged on Sam's approval as well. Which only leaves us with
> 11884.

Bummed that 11884 didn't make it. It fixes issues where it hangs for
ever for any tool that tries to bootstrap and encounters problems
during deployment.

There isn't a specific suite that must be run to validate this.

>
> Best,
> Abhishek Lekshmanan
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)