CephFS and the next jewel release v10.2.3

CephFS and the next jewel release v10.2.3

[Loic Dachary]

Hi John,

The next jewel release as found at https://github.com/ceph/ceph/tree/jewel passed the fs suite (http://tracker.ceph.com/issues/16344#note-30 except for three valgrind failures). Do you think the jewel branch is ready for QE to start their own round of testing ?

Cheers

P.S. CC'ing Greg because John is on vacation, in case a decision is to be made before he returns.

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: CephFS and the next jewel release v10.2.3

[Gregory Farnum]

On Tue, Aug 23, 2016 at 1:59 AM, Loic Dachary <loic@dachary.org> wrote:
> Hi John,
>
> The next jewel release as found at https://github.com/ceph/ceph/tree/jewel passed the fs suite (http://tracker.ceph.com/issues/16344#note-30 except for three valgrind failures). Do you think the jewel branch is ready for QE to start their own round of testing ?
>
> Cheers
>
> P.S. CC'ing Greg because John is on vacation, in case a decision is to be made before he returns.

Ugh, I went to check these valgrind failures and they're a failed
mutex assert on shutdown. Let me dig into what's happening. :(
-Greg

Re: CephFS and the next jewel release v10.2.3

[Gregory Farnum]

On Wed, Aug 24, 2016 at 11:11 AM, Gregory Farnum <gfarnum@redhat.com> wrote:
> On Tue, Aug 23, 2016 at 1:59 AM, Loic Dachary <loic@dachary.org> wrote:
>> Hi John,
>>
>> The next jewel release as found at https://github.com/ceph/ceph/tree/jewel passed the fs suite (http://tracker.ceph.com/issues/16344#note-30 except for three valgrind failures). Do you think the jewel branch is ready for QE to start their own round of testing ?
>>
>> Cheers
>>
>> P.S. CC'ing Greg because John is on vacation, in case a decision is to be made before he returns.
>
> Ugh, I went to check these valgrind failures and they're a failed
> mutex assert on shutdown. Let me dig into what's happening. :(
> -Greg

Hmm, we seem to have some badly-colliding backports. If you look at
commit ac449472df2978123192d5f22247d8b8bc5efb28 and MDSRank.cc line
242, you'll see it does mds_lock.Unlock() twice in quick succession.
:(

One of them came in from 7c2eab19, and the other from 1d3a8168. I
think maybe the patches got reordered in their backport order or
something.

Patrick, can you give https://github.com/ceph/ceph/pull/10847 a quick
review? Loic, can you then run the failed tests against a backport
branch which includes that patch?
-Greg

Re: CephFS and the next jewel release v10.2.3

[Loic Dachary]

On 24/08/2016 22:46, Gregory Farnum wrote:
> On Wed, Aug 24, 2016 at 11:11 AM, Gregory Farnum <gfarnum@redhat.com> wrote:
>> On Tue, Aug 23, 2016 at 1:59 AM, Loic Dachary <loic@dachary.org> wrote:
>>> Hi John,
>>>
>>> The next jewel release as found at https://github.com/ceph/ceph/tree/jewel passed the fs suite (http://tracker.ceph.com/issues/16344#note-30 except for three valgrind failures). Do you think the jewel branch is ready for QE to start their own round of testing ?
>>>
>>> Cheers
>>>
>>> P.S. CC'ing Greg because John is on vacation, in case a decision is to be made before he returns.
>>
>> Ugh, I went to check these valgrind failures and they're a failed
>> mutex assert on shutdown. Let me dig into what's happening. :(
>> -Greg
> 
> Hmm, we seem to have some badly-colliding backports. If you look at
> commit ac449472df2978123192d5f22247d8b8bc5efb28 and MDSRank.cc line
> 242, you'll see it does mds_lock.Unlock() twice in quick succession.
> :(
> 
> One of them came in from 7c2eab19, and the other from 1d3a8168. I
> think maybe the patches got reordered in their backport order or
> something.
> 
> Patrick, can you give https://github.com/ceph/ceph/pull/10847 a quick
> review? Loic, can you then run the failed tests against a backport
> branch which includes that patch?

https://github.com/ceph/ceph/pull/10847 has been pushed as part of the jewel-backports integration branch right now and I'll schedule a fs run as soon as it finishes building. Thanks for the quick fix !

> -Greg
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: CephFS and the next jewel release v10.2.3

[Gregory Farnum]

On Wed, Aug 24, 2016 at 2:02 PM, Loic Dachary <loic@dachary.org> wrote:
>
>
> On 24/08/2016 22:46, Gregory Farnum wrote:
>> On Wed, Aug 24, 2016 at 11:11 AM, Gregory Farnum <gfarnum@redhat.com> wrote:
>>> On Tue, Aug 23, 2016 at 1:59 AM, Loic Dachary <loic@dachary.org> wrote:
>>>> Hi John,
>>>>
>>>> The next jewel release as found at https://github.com/ceph/ceph/tree/jewel passed the fs suite (http://tracker.ceph.com/issues/16344#note-30 except for three valgrind failures). Do you think the jewel branch is ready for QE to start their own round of testing ?
>>>>
>>>> Cheers
>>>>
>>>> P.S. CC'ing Greg because John is on vacation, in case a decision is to be made before he returns.
>>>
>>> Ugh, I went to check these valgrind failures and they're a failed
>>> mutex assert on shutdown. Let me dig into what's happening. :(
>>> -Greg
>>
>> Hmm, we seem to have some badly-colliding backports. If you look at
>> commit ac449472df2978123192d5f22247d8b8bc5efb28 and MDSRank.cc line
>> 242, you'll see it does mds_lock.Unlock() twice in quick succession.
>> :(
>>
>> One of them came in from 7c2eab19, and the other from 1d3a8168. I
>> think maybe the patches got reordered in their backport order or
>> something.
>>
>> Patrick, can you give https://github.com/ceph/ceph/pull/10847 a quick
>> review? Loic, can you then run the failed tests against a backport
>> branch which includes that patch?
>
> https://github.com/ceph/ceph/pull/10847 has been pushed as part of the jewel-backports integration branch right now and I'll schedule a fs run as soon as it finishes building. Thanks for the quick fix !

Sure. Assuming that works out everything else looked good to me. :)

Re: CephFS and the next jewel release v10.2.3

[Loic Dachary]

Hi,

The run completed at http://pulpito.ceph.com/loic-2016-08-25_06:39:08-fs-jewel-backports-distro-basic-smithi/ with no valgrind error but a few other failures. Note that it is based on the jewel-backports branch which contains a few more backports in addition to https://github.com/ceph/ceph/pull/10847 and I'm not sure what to blame yet. I'll look into it tomorrow unless you beat me to it :-)

Cheers

On 24/08/2016 23:03, Gregory Farnum wrote:
> On Wed, Aug 24, 2016 at 2:02 PM, Loic Dachary <loic@dachary.org> wrote:
>>
>>
>> On 24/08/2016 22:46, Gregory Farnum wrote:
>>> On Wed, Aug 24, 2016 at 11:11 AM, Gregory Farnum <gfarnum@redhat.com> wrote:
>>>> On Tue, Aug 23, 2016 at 1:59 AM, Loic Dachary <loic@dachary.org> wrote:
>>>>> Hi John,
>>>>>
>>>>> The next jewel release as found at https://github.com/ceph/ceph/tree/jewel passed the fs suite (http://tracker.ceph.com/issues/16344#note-30 except for three valgrind failures). Do you think the jewel branch is ready for QE to start their own round of testing ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> P.S. CC'ing Greg because John is on vacation, in case a decision is to be made before he returns.
>>>>
>>>> Ugh, I went to check these valgrind failures and they're a failed
>>>> mutex assert on shutdown. Let me dig into what's happening. :(
>>>> -Greg
>>>
>>> Hmm, we seem to have some badly-colliding backports. If you look at
>>> commit ac449472df2978123192d5f22247d8b8bc5efb28 and MDSRank.cc line
>>> 242, you'll see it does mds_lock.Unlock() twice in quick succession.
>>> :(
>>>
>>> One of them came in from 7c2eab19, and the other from 1d3a8168. I
>>> think maybe the patches got reordered in their backport order or
>>> something.
>>>
>>> Patrick, can you give https://github.com/ceph/ceph/pull/10847 a quick
>>> review? Loic, can you then run the failed tests against a backport
>>> branch which includes that patch?
>>
>> https://github.com/ceph/ceph/pull/10847 has been pushed as part of the jewel-backports integration branch right now and I'll schedule a fs run as soon as it finishes building. Thanks for the quick fix !
> 
> Sure. Assuming that works out everything else looked good to me. :)
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: CephFS and the next jewel release v10.2.3

[Gregory Farnum]

You've got two jobs never getting locks, 2 dead on Java linker issues
(which probably never worked; they aren't any kind of release
blocker), and one OSD crash. The crash is

2016-08-25 09:21:45.616945 7fa22623a700 -1 os/filestore/FileStore.cc:
In function 'void
FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
ThreadPool::TPHandle*)' thread 7fa22623a700 time 2016-08-25
09:21:45.612229
os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error")

That most often is an fs/disk issue of some kind, but the kernel logs
are sadly empty so I can't guarantee it.


On Thu, Aug 25, 2016 at 2:13 PM, Loic Dachary <loic@dachary.org> wrote:
> Hi,
>
> The run completed at http://pulpito.ceph.com/loic-2016-08-25_06:39:08-fs-jewel-backports-distro-basic-smithi/ with no valgrind error but a few other failures. Note that it is based on the jewel-backports branch which contains a few more backports in addition to https://github.com/ceph/ceph/pull/10847 and I'm not sure what to blame yet. I'll look into it tomorrow unless you beat me to it :-)
>
> Cheers
>
> On 24/08/2016 23:03, Gregory Farnum wrote:
>> On Wed, Aug 24, 2016 at 2:02 PM, Loic Dachary <loic@dachary.org> wrote:
>>>
>>>
>>> On 24/08/2016 22:46, Gregory Farnum wrote:
>>>> On Wed, Aug 24, 2016 at 11:11 AM, Gregory Farnum <gfarnum@redhat.com> wrote:
>>>>> On Tue, Aug 23, 2016 at 1:59 AM, Loic Dachary <loic@dachary.org> wrote:
>>>>>> Hi John,
>>>>>>
>>>>>> The next jewel release as found at https://github.com/ceph/ceph/tree/jewel passed the fs suite (http://tracker.ceph.com/issues/16344#note-30 except for three valgrind failures). Do you think the jewel branch is ready for QE to start their own round of testing ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> P.S. CC'ing Greg because John is on vacation, in case a decision is to be made before he returns.
>>>>>
>>>>> Ugh, I went to check these valgrind failures and they're a failed
>>>>> mutex assert on shutdown. Let me dig into what's happening. :(
>>>>> -Greg
>>>>
>>>> Hmm, we seem to have some badly-colliding backports. If you look at
>>>> commit ac449472df2978123192d5f22247d8b8bc5efb28 and MDSRank.cc line
>>>> 242, you'll see it does mds_lock.Unlock() twice in quick succession.
>>>> :(
>>>>
>>>> One of them came in from 7c2eab19, and the other from 1d3a8168. I
>>>> think maybe the patches got reordered in their backport order or
>>>> something.
>>>>
>>>> Patrick, can you give https://github.com/ceph/ceph/pull/10847 a quick
>>>> review? Loic, can you then run the failed tests against a backport
>>>> branch which includes that patch?
>>>
>>> https://github.com/ceph/ceph/pull/10847 has been pushed as part of the jewel-backports integration branch right now and I'll schedule a fs run as soon as it finishes building. Thanks for the quick fix !
>>
>> Sure. Assuming that works out everything else looked good to me. :)
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> --
> Loïc Dachary, Artisan Logiciel Libre

Re: CephFS and the next jewel release v10.2.3

[Loic Dachary]

https://github.com/ceph/ceph/pull/10847 is merged

On 25/08/2016 23:33, Gregory Farnum wrote:
> You've got two jobs never getting locks, 2 dead on Java linker issues
> (which probably never worked; they aren't any kind of release
> blocker), and one OSD crash. The crash is
> 
> 2016-08-25 09:21:45.616945 7fa22623a700 -1 os/filestore/FileStore.cc:
> In function 'void
> FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
> ThreadPool::TPHandle*)' thread 7fa22623a700 time 2016-08-25
> 09:21:45.612229
> os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error")
> 
> That most often is an fs/disk issue of some kind, but the kernel logs
> are sadly empty so I can't guarantee it.
> 
> 
> On Thu, Aug 25, 2016 at 2:13 PM, Loic Dachary <loic@dachary.org> wrote:
>> Hi,
>>
>> The run completed at http://pulpito.ceph.com/loic-2016-08-25_06:39:08-fs-jewel-backports-distro-basic-smithi/ with no valgrind error but a few other failures. Note that it is based on the jewel-backports branch which contains a few more backports in addition to https://github.com/ceph/ceph/pull/10847 and I'm not sure what to blame yet. I'll look into it tomorrow unless you beat me to it :-)
>>
>> Cheers
>>
>> On 24/08/2016 23:03, Gregory Farnum wrote:
>>> On Wed, Aug 24, 2016 at 2:02 PM, Loic Dachary <loic@dachary.org> wrote:
>>>>
>>>>
>>>> On 24/08/2016 22:46, Gregory Farnum wrote:
>>>>> On Wed, Aug 24, 2016 at 11:11 AM, Gregory Farnum <gfarnum@redhat.com> wrote:
>>>>>> On Tue, Aug 23, 2016 at 1:59 AM, Loic Dachary <loic@dachary.org> wrote:
>>>>>>> Hi John,
>>>>>>>
>>>>>>> The next jewel release as found at https://github.com/ceph/ceph/tree/jewel passed the fs suite (http://tracker.ceph.com/issues/16344#note-30 except for three valgrind failures). Do you think the jewel branch is ready for QE to start their own round of testing ?
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> P.S. CC'ing Greg because John is on vacation, in case a decision is to be made before he returns.
>>>>>>
>>>>>> Ugh, I went to check these valgrind failures and they're a failed
>>>>>> mutex assert on shutdown. Let me dig into what's happening. :(
>>>>>> -Greg
>>>>>
>>>>> Hmm, we seem to have some badly-colliding backports. If you look at
>>>>> commit ac449472df2978123192d5f22247d8b8bc5efb28 and MDSRank.cc line
>>>>> 242, you'll see it does mds_lock.Unlock() twice in quick succession.
>>>>> :(
>>>>>
>>>>> One of them came in from 7c2eab19, and the other from 1d3a8168. I
>>>>> think maybe the patches got reordered in their backport order or
>>>>> something.
>>>>>
>>>>> Patrick, can you give https://github.com/ceph/ceph/pull/10847 a quick
>>>>> review? Loic, can you then run the failed tests against a backport
>>>>> branch which includes that patch?
>>>>
>>>> https://github.com/ceph/ceph/pull/10847 has been pushed as part of the jewel-backports integration branch right now and I'll schedule a fs run as soon as it finishes building. Thanks for the quick fix !
>>>
>>> Sure. Assuming that works out everything else looked good to me. :)
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre